* fix: chunk IN clauses to stay within DB parameter limits
psycopg caps query parameters at 65 535 and SQLite defaults to 999.
assign_buckets, prune_workstreams, and count_skill_resources_bulk were
passing unbounded lists into single IN(...) clauses, causing
OperationalError during rebalancer runs on full-size hash rings.
Chunk sizes: 10 000 (PostgreSQL), 500 (SQLite).
* fix: deduplicate assign_buckets input, add chunking regression tests
Address review feedback: deduplicate bucket list before chunking to
prevent inflated rowcount from cross-chunk duplicates. Add tests that
exercise the multi-chunk path (1200 buckets > SQLite chunk_size of 500)
and verify dedup preserves accurate counts.
Multiple CI completions for the same commit (tag push + branch push)
caused duplicate publish and docker runs. Concurrency group keyed on
head_sha ensures only one publish runs per commit.
* chore: release infrastructure for dual-track stable/experimental
CI/CD changes for the 1.0 release:
- Gate PyPI publish and Docker publish on CI success via workflow_run
- Add docker-publish.yml: builds and pushes to GHCR with smart tagging
(stable gets :X.Y.Z/:X.Y/:stable/:latest, pre-release gets :experimental)
- Add stable/* and v* tags to CI and docker-scan triggers
- Remove stale [mq] extra and types-redis from CI (Redis MQ deleted)
- Remove stale redis from Renovate package rules
Release tooling:
- scripts/release.sh: bump version, uv lock, commit, tag (with --push)
- docs/releasing.md: documents stable/experimental workflow
Docker:
- Add /workspace mount point (WORKSPACE_MOUNT env var, defaults to empty volume)
- Update .env.example: remove stale Redis/auth-token refs, add workspace/model/discord
README:
- Remove beta warning, add hero image and release tracks table
* fix: derive release tag from git instead of workflow_run.head_branch
Use git tag --points-at HEAD after checkout to resolve the release
tag instead of relying on workflow_run.head_branch, which may not
reliably be the tag name for tag-triggered CI runs. Both publish
and docker-publish workflows now skip cleanly when no v* tag exists
at the checked-out commit.
* fix: replay plan review prompt on SSE reconnection
Plan approval prompts were lost when a user navigated to the server
web UI from the console dashboard (triggering a new SSE connection).
Tool approvals stored pending state in _pending_approval and replayed
it on reconnection, but plan reviews used fire-and-forget _enqueue
with no persistent state.
Mirror the _pending_approval pattern: store _pending_plan_review
before blocking, replay it in events_sse for new SSE clients, and
clear it on resolution. Without this fix, plan reviews silently
timed out after 1 hour and were treated as approval.
* test: add plan review SSE replay regression tests
Covers pending state lifecycle: stored during on_plan_review, cleared
on resolve_plan, available for SSE reconnection replay.
The async variants accepted token_factory for auto-rotating JWTs via
ServiceTokenManager, but the sync wrappers did not expose or forward
the parameter. External SDK users calling the sync clients with
token_factory got a TypeError.
Settings rows and admin table rows used align-items: center, which
caused inputs and source badges to drift away from their labels when
descriptions wrapped to multiple lines. Switch to align-items: start
so controls stay next to their label names regardless of row height.
Add 2px top margin on settings toggles to pixel-align with text input
top padding in start-aligned rows.
* fix(examples): rewrite mcp-cluster-ops to use console SDK for cluster routing
The example MCP server was broken after the direct HTTP transport
refactor — it used TurnstoneServer (single-node) for cluster ops that
require TurnstoneConsole (cluster gateway). Rewrites dispatch flow to:
route via console → SSE stream from node → cleanup via console.
- Switch from TurnstoneServer to TurnstoneConsole for node listing and
workstream routing (TURNSTONE_CONSOLE_URL replaces TURNSTONE_SERVER_URL)
- Add proper workstream lifecycle: create via routing proxy, stream from
node, close in finally block with leak-safe ws_id guard
- Catch dispatch exceptions in run_on_node for structured JSON errors
- Extract _extract_node_ids helper, remove dead n.get("id") fallback
- Normalise _console_kwargs to always include token key
- Rewrite tests against Console+Server mocks (36 → 44 tests)
* fix(examples): paginate node listing and clarify auth in README
Address Copilot review feedback on #278:
- _list_nodes_sync now paginates via offset/limit loop so clusters
with >100 nodes are fully discovered
- README step 2 now mentions token passthrough for authenticated clusters
- New test_paginates_large_clusters verifies multi-page fetch (45 tests)
* fix: remove non-auth support from bootstrap wizard
Auth is now mandatory for all deployments. Remove the
TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN
required in the wizard's system prompt.
* fix: remove auth disable support from runtime and infra
Remove AuthConfig.enabled field — auth is always on. Drop
TURNSTONE_AUTH_ENABLED env var, config toggle, and the
check_request bypass. Update compose.yaml, Helm chart,
Terraform, docs, and tests to match.
* feat: deprecate config tokens, require JWT secret, prefer JWT auth
Phase 1 of config-token removal:
- load_jwt_secret() now exits with error if no secret is configured
(was: silently auto-generated ephemeral secret)
- _authenticate_token() logs deprecation warning on config token use
- CLI /cluster commands use ServiceTokenManager when JWT secret is set
- turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set
- Update bootstrap wizard, docker.md, security.md to mark
TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required
- Console test fixtures use auth token + headers (auth always enforced)
* feat: add service scope for inter-service JWT auth
Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens
bypass require_permission() RBAC checks, replacing the old
empty-user-id bypass that config tokens relied on.
All ServiceTokenManager instances that need admin access now include
"service" in their scopes (console proxy, channel gateway, CLI,
admin CLI). Read-only services (collector, notification) unchanged.
* feat: phase 2 config token deprecation
- SDK doc examples now show API tokens (ts_) instead of config tokens
- Remove _get_config_token() from admin CLI (dead code)
- Block config token exchange in handle_auth_login — only password
and API token login allowed
- Update login tests to use password-based auth instead of config
token exchange
* feat: phase 3 — remove config tokens entirely
Complete removal of config-file token authentication:
- Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch
branch, and config token loading from load_auth_config()
- Remove auth_config parameter from _authenticate_token() and
check_request() — callers updated throughout
- Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts,
Terraform, turnstone.example.toml
- Remove --auth-token CLI flags from turnstone, turnstone-admin,
and turnstone-console
- Simplify console main() — always use ServiceTokenManager
(no fallback to static tokens)
- Delete config-token-specific tests, rewrite check_request and
integration tests to use JWT auth with proper audience claims
- Remove all config token references from docs (security.md,
docker.md, sdk.md, console.md, architecture.md, bootstrap prompt)
* fix: address code review findings
- Fix 33 broken tests: add JWT auth to test_api_versioning,
test_console_routing_proxy, test_tls_admin, test_tls_manager,
test_server_live (jwt_secret + audience-scoped auth headers)
- Add TestRequirePermissionServiceScope: 4 tests covering the
service scope RBAC bypass path
- Remove stale comments referencing config tokens in auth.py and
console/server.py
- Remove dead proxy_auth_token parameter from console create_app()
and static token fallback in _proxy_auth_headers()
- Remove TURNSTONE_AUTH_TOKEN from env.py scrub list
* fix: address Copilot review — JWT audience, compose require secret
- CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager
(console validates audience, JWTs without it were rejected)
- Admin CLI tls-list: same audience fix
- compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset
- SDK console: fix default port from 8081 to 8090
* test: add auth enforcement tests for TLS admin endpoints
5 new tests: unauthenticated requests return 401 (list, renew,
delete), read-only-scoped requests return 403 (renew, delete).
Closes the TLS auth enforcement test gap noted in PROGRESS.md.
* fix: address remaining Copilot review feedback
- Fix token_source="config" → "test" in TLS test fixtures
- Fix AuthResult.token_source docstring to include service origins
- Require TURNSTONE_JWT_SECRET in cluster compose profile (:?)
- Helm: add auth.jwtSecret + auth.existingSecret values, wire
TURNSTONE_JWT_SECRET into secret.yaml and both deployments
- Terraform: replace auth_token with jwt_secret variable + secret,
remove orphaned auth_token resources and IAM reference
- Remove [[auth.tokens]] from security.md config example
* fix: address full code review — 10 findings
Critical:
- Terraform: replace concat(common_env, auth_env) with common_env
(auth_env local was removed but still referenced)
- Channel gateway: remove hmac static token auth from _check_auth(),
use JWT-only validation. Remove --auth-token CLI arg from channel
- Rebalancer: add token_manager support so migration requests carry
JWT auth (was sending unauthenticated POST to /internal/migrate)
Major:
- Guard _permissions_to_scopes() against "service" privilege
escalation from DB role permissions
- Remove dead AuthConfig class, load_auth_config(), and all
auth_config parameters from create_app() signatures
- Helm: inject JWT secret for both inline and existingSecret paths
Minor:
- Remove dead auth_token param from ClusterCollector
- Remove empty TestLoadAuthConfig class
- Short JWT secret now exits instead of warning
- Compose: add generation command comment above JWT_SECRET
- Clean stale config token references from 6 doc files
- Clean stale AUTH_TOKEN reference from bootstrap wizard prompt
* fix: remove remaining stale config token references from docs
- channels.md: remove --auth-token from options table
- oidc.md: remove "config-file tokens still work" claim
- security.md: remove config token section, fix JWT secret docs
(now required/exits, no ephemeral fallback), remove hmac from
ASCII diagram, remove --auth-token reference
* fix: populate model in _last_usage so usage-by-model records correctly
_last_usage was built purely from UsageInfo token counts, never
including a "model" key. server.py's on_status() fell back to
model="" for every record_usage_event call, so GROUP BY model
collapsed all rows into a single empty-key bucket.
* fix: inject model at emission time, preserve dict[str, int] typing
Address Copilot review: keep _last_usage as dict[str, int] for type
safety, inject "model" from self.model when passing to on_status().
This also fixes stale model after /model switch since the value is
read fresh each time.
* fix: MCP tools not surfacing after Sync to Nodes, update Anthropic tool search
Three fixes:
1. session_factory closure captured mcp_client=None when no --mcp-config
was passed at startup. internal_mcp_reload created a new MCPClientManager
on app.state but the factory never saw it. New workstreams got 0 MCP tools.
Fix: mutable _mcp_ref list shared between factory and reload handler.
2. Anthropic dropped the date suffix from tool_search_tool_bm25_20251119
and now requires name == type. Updated constant and tool definition.
3. Add diagnostic logging around API errors (provider, model, base_url,
message counts, full exception chain) and workstream resume (pre/post
provider state, alias resolution warnings).
Also adds Node.js 24 LTS to Dockerfile via multi-stage copy for npx-based
MCP servers.
* fix: address Copilot review — set_storage on reload, sanitize log output
- Call mcp_mgr.set_storage(storage) when internal_mcp_reload creates a
new MCPClientManager so prompt sync works for post-startup servers
- Strip query params from base_url before logging (may contain API keys
in some vLLM deployments)
- Split API error logging: concise warning (type names only) + separate
debug with exc_info=True for full traceback when needed
* chore: remove DDG MCP sidecar, web_search uses built-in ddgs client
The DuckDuckGo MCP server container is redundant — the built-in
DuckDuckGoClient (via ddgs package, included in all extras) auto-detects
when no Tavily key is configured. Removes the ddg-search service,
ddgCluster profile, and mcp-ddg.json config file.
* fix: materialize skill resources to disk for subprocess access
Skill-bundled scripts stored in skill_resources were loaded into memory
but never written to disk, causing FileNotFoundError when the model
tried to execute them. Write resources to a per-workstream temp directory
on skill load, expose via SKILL_RESOURCES_DIR env var and PATH, clean up
on skill change or session close.
* fix: pre-flight validation warns when skill references missing resources
Scan rendered skill content for path references (scripts/foo.py, etc.)
and compare against bundled skill_resources. Warn via on_info if any
referenced paths are not bundled, so operators see the gap at skill
activation rather than at runtime FileNotFoundError.
* fix: address PR #271 review feedback
- Fix trailing colon in PATH when $PATH is empty (cwd-on-PATH risk)
- Move try/except inside per-resource loop so one bad write doesn't
abort all resources
- Explicit encoding="utf-8" for deterministic writes across locales
* fix: address PR #271 review round 2
- Normalize available paths in _validate_skill_resources() to match
referenced paths (both sides use os.path.normpath now)
- Fix flaky traversal test: assert inside base dir, not escaped path
Multiple browser tabs open to the same server could see workstream
names, states, and content mixed up between workstreams when creating,
closing, and switching tabs rapidly.
Root causes and fixes:
- Global SSE ws_created events were never handled — other tabs never
learned about new workstreams, causing blank names and stale tab bars
- SSE reconnection assigned all stale panes to the first workstream
instead of deduplicating; now uses two-pass assignment with tracking
- switchTab left the old EventSource open while reassigning pane.wsId,
creating a window for events to leak; now disconnects SSE first
- Per-workstream events carried no ws_id — server now stamps ws_id on
all events via _enqueue (shallow copy); client handleEvent drops
events with mismatched ws_id as defense-in-depth
- Plan dialog used pane.wsId at resolve time (could drift after tab
switch); now captures ws_id when the dialog opens
- Global ws_closed could reassign panes before per-ws SSE finished
draining; now disconnects per-ws SSE immediately on close
The 5 prompt policy admin endpoints required "admin.prompt_policies"
but the builtin-admin role only grants "admin.policies". Changed to
match the existing permission used by tool policy endpoints.
* fix: harden Discord bot against gateway disconnects and SSE failures
- Isolate Discord API failures from SSE stream — _on_ws_event exceptions
no longer kill the SSE connection and cause missed events
- Fix broken exponential backoff on 4xx/5xx (delay was reset on every
attempt); skip aiter_sse() on error responses
- Add read timeout (90s) to SSE httpx client so half-open TCP
connections are detected and recovered
- Re-resolve node URL on each SSE reconnect attempt
- Add on_resumed handler to recover SSE tasks that died during brief
gateway disconnects (on_ready is not called on session resume)
- Sync slash commands only on first on_ready to avoid Discord rate limits
* fix: SSE backoff on 4xx/5xx and retrieve dead task exceptions
- Replace `continue` with raise+catch so 4xx/5xx errors hit the
exponential backoff path instead of tight-looping
- Retrieve task exceptions in _purge_dead_sse_tasks to suppress
"Task exception was never retrieved" warnings and log the cause
* feat: modular system message composition with admin prompt policies (#267)
Replace the monolithic persona+tools block in _init_system_messages()
with a modular composition harness (turnstone/prompts/). System messages
are now assembled from five typed layers: BASE (persona), ENV (client
surface — web/cli/chat), CONTEXT (datetime, timezone, username), TOOLS
(usage patterns), and POLICIES (behavioral rules with tool gating).
Prompt policies are admin-managed via a new Prompts tab in the Governance
group (CRUD with modal forms, tool gating, priority ordering, enable/disable).
DB policies override file-based defaults by name; file-based policies serve
as deployment defaults. Migration 031 adds the prompt_policies table.
ClientType is threaded end-to-end from channel adapters through the SDK,
HTTP API, WorkstreamManager, and session factory to ChatSession. Discord
sessions now receive chat-optimized formatting (no tables, no Mermaid,
concise output) instead of the web UI's rich markdown instructions.
* fix: address CI failures and Copilot review feedback
- Add client_type param to CLI session_factory (mypy protocol match)
- Add prompt policy CRUD to PostgreSQL backend (test-postgres CI)
- Fix ClientType resolution: compare against enum values, not members
- Fix null client_type coercion (body.get returns None, not "")
- Use local time with astimezone() instead of UTC with local tz name
- Sanitize tool_gate in update endpoint (coerce null to empty string)
* feat: replace console HTTP polling with persistent SSE streams
Console collector now subscribes to each server node's /v1/api/events/global
SSE stream for real-time state updates instead of polling /v1/api/dashboard
and /health every 15 seconds.
Server changes:
- Emit ws_created/ws_closed events on global queue from create/close handlers
- Add node_snapshot on SSE connect (workstreams, health, aggregate)
- Add ?expected_node_id= identity verification (409 on mismatch)
- Add health_changed callback to BackendHealthMonitor circuit breaker
- Add periodic aggregate emitter thread (10s)
Console collector changes:
- Single asyncio event loop on one thread multiplexes all SSE connections
(scales to 1000+ nodes vs thread-per-node)
- Discovery loop spawns/cancels async SSE tasks per node
- Snapshot reconciliation on connect, delta application for live events
- Fix ws_state→cluster_state event type mismatch
- Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI)
SDK changes:
- Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses
- Add stream_node_events() method (async + sync)
* fix: address review feedback on node event streams
- Fix stop() to let SSE manager exit naturally instead of force-stopping
the event loop (ensures finally cleanup runs)
- Guard against empty/invalid SSE data from ping frames
- Treat missing node_id as identity mismatch (409) when expected_node_id
is provided
- Fix stale docstring on _update_metrics
* feat: show thinking indicators, tool calls, and results in Discord threads
Discord threads now surface real-time activity during multi-tool chains
instead of appearing idle. ThinkingStart/Stop events display a transient
italic status message. ToolInfoEvent sends a per-tool "running" embed
that ToolResultEvent edits in-place with the result (FIFO matching by
tool name, fallback to new message). Includes backtick-injection escaping
in tool output. Visibility respects auto-approve config so tool calls
always appear somewhere.
* fix: address review feedback on Discord action visibility
Delete thinking messages in unsubscribe/stale-route cleanup (not just
pop state). Sanitize tool-call previews (escape backticks, strip
mentions). Fix format_tool_result docstring re ellipsis line count.
Add regression test for triple-backtick escaping.
* fix: disable approval buttons on server-side resolution (timeout)
Handle ApprovalResolvedEvent in _on_ws_event to disable buttons and
grey out the approval embed when the server resolves the approval
externally (timeout, auto-approve from another client). Extract
disable_message_buttons helper from views.py so it works on a plain
Message (not just an Interaction).
* fix: reply with guidance when user DMs the bot directly
Non-reply DMs were silently ignored. Now sends a message directing the
user to /ask or @mention in a server channel.
* fix: address round 2 review feedback
- Show error items (policy-denied) in ToolInfoEvent unconditionally
- Match ToolResultEvent to ToolInfoEvent by call_id (deterministic),
fall back to name-based FIFO when call_id is absent
- Escape triple backticks before truncating in format_tool_result so
the 500-char limit holds after expansion
* fix: edit thinking message in-place instead of delete-and-recreate
ThinkingStopEvent now preserves the message for the next event to reuse.
ContentEvent seeds StreamingMessage with the thinking message so the
first flush edits it. ToolInfoEvent edits the thinking message into the
first tool embed. Eliminates the visible delete → gap → new message
flicker during thinking → tool call transitions.
* feat: separate tool call and result into distinct Discord messages
ToolInfoEvent sends a "running" embed (light grey, tool name + preview).
ToolResultEvent marks it "Done"/"Error" (color + title update) and sends
the result as a separate message. This gives clear lifecycle tracing in
chat-style threads where verbosity aids readability.
* fix: show running embed for all tools and remove redundant name prefix
ToolInfoEvent now shows a running embed for every tool regardless of
needs_approval — the running indicator and approval dialog serve
different purposes. Removes the needs_approval/auto_approve filter
that caused missing running embeds when tools were approved via
"Always Approve" or server-side auto-approve.
Also drops the redundant **name** prefix from format_tool_result since
the embed title already carries the tool name.
* fix: concise logging for SSE connection failures
Catch httpx.ConnectError/ConnectTimeout separately from the generic
exception handler. Logs url and error string instead of the full
httpx/httpcore stack trace, which is noise for expected transient
connection failures during node restarts.
* fix: address round 3 review feedback
- Pop _pending_approval_msgs on button click so ApprovalResolvedEvent
doesn't double-update the embed title (e.g. "Approved - Approved")
- Remove unused name/is_error params from format_tool_result — embed
title carries the name, embed color carries the error status
- Fix _disable_buttons docstring to mention title update
- Fix missing /v1 prefix on SSE endpoint URL (caused all SSE connections
to get text/plain 404 responses instead of event streams)
- Stop treating StreamEndEvent as session-terminal (it fires per-segment,
not per-workstream) so multi-turn conversations work in Discord
- Bail on 404 instead of retrying forever for gone workstreams, and clean
up stale routes from storage
- Check response status before iterating SSE events to avoid retrying
non-retryable upstream errors
- Default rebalancer.enabled to True so hash ring routing works without
manual ConfigStore setup
- Add one-shot cache refresh fallback on route endpoints to handle the
startup race between rebalancer and first routed request
- Remove stale "message queues" language from tagline
- Remove duplicated content covered by docs (governance details,
judge config, config.toml reference, health/rate-limit details,
monitoring metrics, tool table, multi-model config)
- Replace tool table with summary + link to docs/tools.md
- Add documentation index table linking to all doc pages
- Add architecture summary (single-node vs multi-node routing)
- Add component table for entry points
- Trim diagram table to most useful subset
- Consolidate quickstart section
README is now a concise landing page that directs to docs for
details, not a duplicated reference manual.
When TURNSTONE_ADVERTISE_URL is set (Docker deployments), the
_advertise_host variable was never assigned. The TLS upgrade path
tried to use it to construct the https:// URL, causing an
UnboundLocalError that made TLS init fail silently.
Fix: derive the TLS URL from _advertise_url (replace http → https)
instead of reconstructing from _advertise_host.
Address Copilot PR feedback:
- Exit with clear error if neither console_url nor server_url is
available after discovery (prevents cryptic failures downstream)
- Fix log field names: console → console_url, server → server_url
for consistency with other channel log events
Add token_factory parameter to SDK clients (_BaseClient, server,
console) — a Callable[[], str] invoked before each request to get
the current auth token. Supports ServiceTokenManager for auto-rotating
JWTs that re-mint transparently before expiry.
Channel gateway creates dual token managers:
- console-audience JWT for routing proxy calls (via AsyncTurnstoneConsole)
- server-audience JWT for direct SSE connections to server nodes
Both _request() and _stream_sse() inject the factory header per-call,
so long-lived connections get fresh tokens on reconnect.
Also adds TURNSTONE_CONSOLE_URL to console compose service for
DNS-resolvable service discovery.
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Log discovery progress (discovering, discovered_console, discovered_server)
and warn on timeout or failure
- Wrap discovery in try/except so storage init failures don't crash startup
- Default --server-url is now empty (not localhost:8080) to avoid
unreachable fallback URLs inside Docker containers
- Auto-discovery retries for up to 30s waiting for console or server
to register in the services table (handles startup ordering)
- Both console_url and server_url are discovered from DB when not
explicitly set via CLI flags or env vars
- 404 retry: use blocking lock acquire so retry waits for cache refresh
to complete instead of skipping on contention
- 404 retry: surface httpx.HTTPError as 502 instead of suppressing it
and returning the original 404
- channel router: pass auto_approve_tools to create_workstream calls
(was silently dropped for console-routed creates)
- api-reference.md: document all /v1/api/route/* console routing proxy
endpoints and console /metrics
- router.route(): validate ws_id length and hex format before bucket
extraction, raise NoAvailableNodeError instead of ValueError
- router: expose version as public property, collector uses it instead
of accessing _version directly
- memory.py: deduplicate _bucket_of with canonical bucket_of from
hash_ring module
- architecture SVG: reroute direct/SSE lines below console to avoid
crossing over the console box
The collector's _apply_poll only detected workstream additions and
removals (set diff on ws_ids). State changes within existing
workstreams (idle → running, running → attention, etc.) were not
emitted to the SSE stream, so the dashboard only updated on manual
page refresh.
Now _apply_poll compares state and name fields between old and new
poll snapshots and emits ws_state and ws_rename events for any
changes. These flow through _fanout to the browser SSE stream,
giving real-time dashboard updates without page refresh.
Bug: Server nodes registered with container ID hostnames (e.g.,
http://a236323a92f6:8080) which aren't DNS-resolvable by other
containers. The console collector failed to poll nodes, causing
stale health/error status on the dashboard.
Fix: Add TURNSTONE_ADVERTISE_URL env var support. In compose, each
server sets it to the Docker service name (http://server-1:8080 etc).
Falls back to socket.getfqdn() when not set.
Also: remove the 100-node stress cluster (ddgStressCluster profile)
from compose.yaml. It was 720 lines of boilerplate from the old
simulator era. The simulator is being rebuilt separately (task #5).
Compose goes from 1028 to 304 lines.
Bug 1: Server's create_workstream handler ignored initial_message from
the request body. The old bridge sent it as a follow-up SendMessage
via Redis, but with direct HTTP nobody was sending it. Now the server
spawns a worker thread to send the initial message after creation,
matching the bridge's behavior.
Bug 2: Channel gateway compose config used --server-url=http://server:8080
which doesn't exist in cluster/ddgCluster profiles. Removed the hardcoded
URL — the channel gateway auto-discovers the console from the services
table via shared PostgreSQL. Added TURNSTONE_DB_URL and auth token to
the channel environment so DB-based service discovery works.
Server SDK create_workstream: add initial_message, auto_approve_tools,
user_id, ws_id params (all optional, omitted when empty).
Console SDK: add auto_approve, auto_approve_tools, user_id to
create_workstream. Add 8 route_* methods for the routing proxy path
(/api/route/*): route_create_workstream, route_send, route_approve,
route_plan_feedback, route_close, route_cancel, route_command,
route_lookup. Sync mirrors for all.
Prepares for channel gateway and scheduler to use SDK clients instead
of raw httpx calls.
Move the consistent hash ring implementation (FNV-1a, virtual nodes,
bisect lookup) from code to docs/design/consistent-hash-ring.md as a
forward-looking reference for future scalability work.
The current rebalancer uses weight-proportional distribution (simpler,
exact splits, no hash variance). The ring algorithm is documented with
test vectors, stability properties, and a comparison table for when
the ring approach becomes advantageous (large clusters, decentralized
routing, cross-language determinism).
hash_ring.py retains: RING_SIZE, bucket_of(), RingNode, NoAvailableNodeError
(all actively used by router and rebalancer).
Replace the full-rehash algorithm (diff ideal vs current across all
65536 buckets) with a donor/recipient algorithm that only moves
buckets from overloaded nodes to underloaded nodes.
Key improvements:
- Adding node C to {A, B} only moves buckets TO C, never between
A and B. Previously the HashRing rehash could shuffle between
existing nodes.
- Seeding uses weight-proportional distribution instead of HashRing
virtual nodes, producing an exact split that doesn't trigger
immediate correction on the next cycle.
- Dead-node buckets are redistributed to the most underloaded
survivors, not rehashed across the whole ring.
- HashRing class is no longer used by the rebalancer (still
available for other uses like the Go rewrite reference).
The threshold check still gates live-to-live moves. Dead-node
recovery remains unconditional.
set_bucket_stat: single-upsert storage method replacing the N-loop
reconciliation in the rebalancer. Reduces DB round-trips from
|ws_delta| per bucket to exactly 1.
Console metrics: /metrics endpoint on the console exposing 6 routing
and ring metrics in Prometheus text format:
- turnstone_router_requests_total (method, status)
- turnstone_router_request_duration_seconds (method)
- turnstone_ring_membership_size
- turnstone_ring_version
- turnstone_ring_rebalance_total (result)
- turnstone_ring_migrations_total
Instrumented in route_create, route_proxy, route_lookup handlers.
Ring gauges updated on collector discovery loop. Rebalance/migration
counters recorded after each rebalancer pass.
When rebalancer.eager_migrate is enabled, the rebalancer POSTs
/_internal/migrate to source nodes after reassigning buckets,
triggering immediate workstream eviction instead of waiting for
lazy resume on the next request.
Only idle workstreams are eagerly migrated — active ones (running,
thinking, attention) are left alone to avoid disrupting in-flight
work. Failed migrations are logged and skipped (the lazy path
handles them eventually).
Rebalancer: daemon thread in the console process that maintains
bucket-to-node assignments in hash_ring_buckets. Seeds the ring on
first run (empty table → 65536 rows via consistent hash). Periodically
checks for membership changes and rebalances: moves cheapest buckets
first (empty > idle > active), respects imbalance threshold, reconciles
bucket_stats against actual workstream counts before each pass.
Uses DB-based leader election (rebalancer_lock in system_settings) for
multi-console deployments. Increments rebalancer_version after writes
so console routers refresh their caches.
Add 6 settings: ring.vnodes_per_unit, rebalancer.enabled/interval/
threshold/eager_migrate, node.weight.
Add /_internal/migrate endpoint on server for eager workstream eviction.
Add routing proxy endpoints to the console server:
- POST /v1/api/route/workstreams/new — hash-ring-routed create with
503 retry, target_node pinning, and node_url injection
- POST /v1/api/route/{send,approve,cancel,command,close} — generic
proxy to workstream owner via O(1) bucket lookup
- GET /v1/api/route?ws_id=X — node URL lookup for direct SSE
Wire ConsoleRouter into console lifespan (cache refresh on startup)
and collector discovery loop (version-based cache invalidation).
Add --console-url to channel gateway CLI for multi-node routing.
ChannelRouter routes control-plane through console when set, SSE
connections go direct to server nodes via node_url from create response.
HashRing: FNV-1a virtual nodes, immutable, computes ideal bucket-to-node
distribution. Used by the rebalancer (next commit) to seed and maintain
the assignment table.
ConsoleRouter: in-memory flat array of 65536 NodeRef entries loaded from
hash_ring_buckets table. O(1) routing via ws_id prefix. Supports
per-workstream overrides, version-based cache refresh, and targeted
ws_id generation.
Both are pure library code with no server integration yet.
Delete the entire turnstone/mq/ package (broker, bridge, protocol,
client) and turnstone/sim/ package. Remove Redis as a dependency.
Channel gateway and console now communicate with server nodes via
direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues.
Single-node deployments work with zero infrastructure beyond the
database.
Key changes:
- Channel adapters use httpx POST for create/send/approve/close
and httpx-sse for per-workstream event streaming
- Console collector discovers nodes via services table instead of
Redis SCAN
- Console scheduler dispatches tasks via HTTP POST with DB-based
leader election
- Server registers in services table with 30s heartbeat
- Server accepts optional ws_id in create request (for Phase 2
console-generated routing)
- SDK events gain IntentVerdictEvent and OutputWarningEvent types
- All docs, examples, bootstrap wizard updated
63 files changed, -5968 net lines (Redis transport fully removed)
* fix: sync actual TLS state to ConfigStore on console startup
The console writes tls.enabled to the DB but never clears it when TLS
init fails or isn't configured. Server nodes read the stale DB value
and attempt TLS negotiation with a non-TLS console, producing noisy
SSL errors on every startup.
Console now syncs the actual TLS state after init: if TLS succeeded,
tls.enabled=true; if it failed or wasn't attempted, tls.enabled=false.
Server TLS failure log reduced from full traceback to one-line warning.
* fix: sync TLS state to ConfigStore on console startup
Console now writes the definitive TLS state to ConfigStore so server
nodes don't attempt TLS against a non-TLS console:
- TLS init succeeded → write true
- TLS not configured (DB false/unset) → write false (definitive)
- TLS configured (DB true) but init failed → don't overwrite
(transient failure shouldn't permanently disable)
Server TLS warning reduced to one line with exception type, full
traceback available at debug level.
* feat: auto-detect model changes when LLM backend swaps models
The BackendHealthMonitor already probes /v1/models every 30s but
discarded the response. Now compares the detected model against the
last known one and triggers a registry reload when it changes.
- Extract _extract_context_window() helper for reuse across
detect_model, probe_model_endpoint, and the health monitor
- BackendHealthMonitor: new provider/initial_model/on_model_changed
params; _check_model_change() fires callback on model swap
- Server: wire _handle_model_change callback that updates cli_model_args
and calls registry.reload(); guarded by _user_specified_model flag
so --model overrides are never auto-replaced
- Session: _refresh_model_from_registry() called at top of send();
two string compares when nothing changed, full re-resolve on swap
- 7 new tests for _extract_context_window and model change detection
* fix: address Copilot review on model re-detection
- server: update cli_model_args only after successful reload (not
before), add finally block for new_reg.shutdown(), guard against
cli_model_args not yet initialized
- session: wrap registry lookup in try/except for concurrent reload
race, reset judge on model change, recompute tool_truncation when
context_window changes in auto mode
Replace "You are an expert software engineer" with a grounded
narrative persona: a resident engineer on a focused team with real
tools, real code, and real consequences. Sets expectations about
boundaries, judgment calls, and working within constraints.
* fix: scope-filter memory list/search to current workstream and user
Unscoped memory(action='list') and memory(action='search') returned all
memories across all workstreams. Now applies the same 3-query pattern
(global + current workstream + current user) used by system prompt
injection.
* fix: validate user scope on memory search/list for unauthenticated sessions
Adds _validate_scope guard to search and list prepare paths, matching
save/get/delete. Prevents explicit scope='user' from returning all
user-scoped memories when session is unauthenticated.
* fix: update _get_visible_memories references to _list_visible_memories
* fix: defense-in-depth guard for empty scope_id on search/list
Copilot review: if scope is 'user' or 'workstream' with empty
scope_id, the storage query returns all memories in that scope
across all users/workstreams. The prepare step already validates
via _validate_scope, but add exec-level guard to reject scoped
queries with empty scope_id as defense-in-depth.
* fix: detect context window from vLLM max_model_len field
vLLM exposes the context window as max_model_len on the model object,
not meta.n_ctx_train (llama.cpp format). Both detect_model() and
probe_model_endpoint() now check max_model_len first, falling back
to meta.n_ctx_train for llama.cpp. Fixes 32768 fallback on vLLM
servers that report 262144+ token context windows.
* test: add vLLM max_model_len detection tests
Copilot review: new vLLM context window path had no test coverage.
Add tests for probe_model_endpoint (max_model_len detected, preferred
over meta.n_ctx_train) and detect_model (vLLM model object with
max_model_len).
The first message sent from Discord was silently dropped because the
cog delegated the initial message to the bridge via CreateWorkstream-
Message, but the bridge published response events to the per-workstream
Redis pub/sub channel before the Discord bot had subscribed to it.
Redis pub/sub is fire-and-forget — events with no subscribers are lost.
Fix: create the workstream with initial_message="" (no delegation),
subscribe to the per-workstream event channel, then send the message
through router.send_message() — the same path the second message
already uses successfully.
Applied to both @mention handler and /ask slash command.
* feat: per-workstream status bar above input
Move the global token counter and model name from the header into a
per-pane telemetry strip between messages and the text input. Each
workstream pane now independently shows model name, token usage with
context percentage, tool calls this turn, and turn count.
Backend: add _ws_turn_tool_calls counter (reset per user turn, emitted
in SSE status event alongside turn_count). MQ bridge forwards the new
fields. SDK and TypeScript types updated.
Frontend: build .ws-status-bar DOM in _createDOM, rewrite updateStatus
to target per-pane elements, update SSE connect/disconnect handlers.
Remove #model-name and #status-bar from global header. Restore console
#status-bar CSS in its own stylesheet.
Accessibility: aria-atomic, aria-labels on each field, warning symbols
(▲/⚠) at 80%/95% context for color-blind users, placeholder text
before first status event. Disconnect state uses 2px red border with
dimmed stale fields.
* fix: emit status event on SSE connect so status bar populates on resume
When resuming a workstream, the event_generator only sent connected +
history events. The status bar stayed at placeholder values until the
next LLM response. Now replays session._last_usage as a synthetic
status event right after connected, so token count, tool calls, and
turn count render immediately.
* fix: address Copilot review — remove dead function, clarify locals
Remove updateHeaderForFocusedPane() and its call site (no-op since
status moved per-pane). Rename ambiguous ttc/tc locals to
turn_tool_calls/turn_count in the status replay block.
* feat: add memory get action, reduce search/list preview to 200 chars
search and list truncated memory content to 500 chars with no way to
read the full value. Two changes:
- New 'get' action retrieves a single memory by name with complete
untruncated content. Searches scopes narrowest-first (workstream
→ user → global).
- search/list previews reduced from 500 to 200 chars now that get
exists for full content. Both append a hint:
"Use memory(action='get', name='...') for full content."
Includes get_structured_memory_by_name wrapper in memory.py and
4 tests.
* Update turnstone/tools/memory.json
* fix: include 'get' in _prepare_memory docstring and invalid-action error
PostgreSQL text fields cannot store NUL (0x00) bytes, and SQLite
stores them but they cause downstream issues (API payloads, web UI).
Add sanitize_text() to _utils.py and apply it in both backends'
save_message to content and provider_data fields.
* fix: drop orphaned tool_results with no matching tool_use in _convert_messages
The context window increase from 200K to 1M for Claude 4.6 means
conversations that previously triggered auto-compaction now send their
full history. Older messages with orphaned tool_results (from
pre-fix cancels or compaction boundaries) are now visible to the API,
causing "unexpected tool_use_id in tool_result blocks" errors.
The existing repair code handles orphaned tool_use (synthesizes
missing results), but not the reverse. Now validates each
tool_result against the preceding assistant message's tool_use IDs
and silently drops results with no match.
* fix: filter empty IDs from prev_tool_use_ids, document pass-through
Code review: empty-ID tool_use blocks were added to the filter set,
and the intentional pass-through when prev_tool_use_ids is empty
needed documentation.
* fix: block math sandbox escape via getattr/setattr/type reflection
getattr() with runtime-constructed strings bypassed the AST validator,
allowing full os/subprocess access from the sandboxed math tool via
module.__builtins__['__import__']('os').
Three-layer fix:
- Block getattr, setattr, delattr, type, __import__ in
_MATH_BLOCKED_BUILTINS (prevents direct calls)
- Add AST validation for getattr/setattr/delattr call nodes
(catches them even if builtins dict is bypassed)
- Strip __builtins__ from all pre-imported modules in the execution
namespace (runtime defense — even if AST is somehow bypassed,
module.__builtins__ returns empty dict)
Normal math, sympy, numpy, scipy operations unaffected.
* fix: harden _safe_import to strip __builtins__ from runtime imports
Copilot review: modules imported at runtime via _safe_import still
had their original __builtins__ dict, accessible via
operator.attrgetter('__builtins__'). Now _safe_import strips
__builtins__ from every module it returns. Also blocks
operator.attrgetter/itemgetter at the AST level, and removes the
redundant duplicate getattr check in visit_Call.
* fix: add type ignore for module __builtins__ assignment
* fix: block /proc/*/environ access in bash filter and judge heuristic
/proc/1/environ leaks the full server environment including DB
credentials, API keys, and JWT secrets. Env scrubbing in env.py
only affects subprocess calls, not procfs reads.
- Add /proc/1/environ and /proc/self/environ to BLOCKED_PATTERNS
in safety.py (hard block)
- Add proc-environ-exfil heuristic rule at critical severity with
deny recommendation (catches /proc/<pid>/environ patterns)
* fix: move proc-environ-exfil rule to _CRITICAL_RULES list
Copilot review: rule had risk_level=critical but was placed in
_HIGH_RULES. Move to _CRITICAL_RULES for consistency with the
first-match-wins severity ordering.
The intent judge was receiving up to 50% of the context window in
conversation history (FIFO from end), which grows linearly with
conversation length and causes increasing latency. The judge only
needs the immediate request context to evaluate a tool call's safety.
Now trims to messages from the last user message onward before
applying the FIFO budget cap. Keeps the user's request, the
assistant's response with tool calls, and any recent tool results
while discarding earlier conversation that isn't relevant to the
current intent evaluation.
Claude 4.6 (Opus + Sonnet) unified on 1M token context windows.
Update capabilities table from 200K to 1M for both models. Remove
claude-opus-4 and claude-sonnet-4 entries (end of life). 4.5 models
remain at 200K. Default fallback stays at 200K for unknown models.
* fix: distinguish user cancel from crash in bash tool results
When a user cancels a running bash command, the process is killed
with SIGKILL (exit code -9). Previously this showed as an error,
causing the model to retry. Now checks cancel.is_set() after proc
exit and returns "Cancelled by user." as a non-error result so the
model knows to stop rather than retry.
* fix: use -signal.SIGKILL instead of magic -9
Copilot review: replace hard-coded -9 with -signal.SIGKILL for
clarity. Popen.returncode is negative of signal number when killed.
* feat: add stop_on_error param to bash tool for set -e behavior
New boolean parameter enables 'set -e' in the bash preamble so
multi-step scripts exit on the first command failure instead of
silently continuing. Default false (existing behavior preserved).
pipefail remains always-on.
* fix: strict bool parsing for stop_on_error, treat exit 1 as error with set -e
Copilot review: bool("false") is True — use `is True` for strict
JSON boolean parsing. Also, with stop_on_error enabled, any non-zero
exit code is now treated as an error (set -e means the script halted
on failure), whereas without it exit code 1 remains benign.
* fix: synthesize cancelled tool results instead of stripping turns
When a user cancels during tool execution, the model previously lost
all context about what was attempted (assistant message + tool_calls
stripped entirely). Now synthesizes tool_result messages with
is_error=true and "Cancelled by user." content for any tool_calls
that lack matching results. This keeps the conversation valid for
both providers while preserving the full tool call structure so the
model knows what was tried.
Also applies to KeyboardInterrupt with "Interrupted by user." text.
* fix: persist synthesized cancel results to DB, assert is_error in test
Copilot review: synthesized tool messages were in-memory only,
creating a mismatch with DB that could break rewind/retry. Now
calls save_message() for each synthesized result. Also adds
is_error=True assertion to the cancel test.
* feat: add pagination and longer content to recall tool
- New offset parameter for paginating through recall results
- Content preview increased from 500 to 2000 chars per match with
total length indicator when truncated
- Output passed through _truncate_output for consistency
- OFFSET clause added to SQLite (FTS5 + LIKE) and PostgreSQL
(tsvector + ILIKE) search queries
* fix: defensive int coercion for recall offset/limit
Copilot review: offset/limit could arrive as null, float, or other
non-int types from JSON. Coerce with int() + try/except in prepare,
and int() at the storage layer before binding into SQL OFFSET/LIMIT.
* feat: add diff_file tool for comparing files and content
New read-only tool that shows unified diffs between two files or
between a file and provided content. Useful for verifying edit_file
changes and comparing file versions. Auto-approved (no side effects).
Configurable context lines (default 3). Available to task agents.
* refactor: extract _read_text_lines helper, share across read_file and diff_file
Copilot review: diff_file duplicated file-loading and lacked binary
detection. Extract _read_text_lines() that handles realpath
resolution, null-byte binary detection, and error handling. Used by
both _exec_read_file and _exec_diff for consistent behavior.
* fix: address code review — agent flag, resolved shadowing, read_files
- Add agent: true to diff_file schema so plan agents can use it
- Fix resolved variable shadowing in _exec_read_file (use _ for
unused return from _read_text_lines)
- Register diffed files in _read_files so edit_file read guard
is satisfied after diff_file
- Move difflib import to module level (stdlib, no lazy-load needed)
- Fix description wording ("provided string" not "previous version")
* fix: stream diff with early cutoff, expand paths before header
- Stream difflib output and stop collecting after tool_truncation
chars to avoid large intermediate allocations on big diffs
- Expand paths with expanduser before building the approval header
so display matches actual execution paths
* docs: tool descriptions, bash timeout param, multi-line preview
- task_agent/plan_agent: document the tool subset limitation (no
memory, recall, watch, skill, or further delegation)
- bash: add per-call timeout parameter (1-600s, defaults to 120s),
shown in approval header when specified
- bash: show full command in preview for multi-line scripts so the
approval flow displays the complete command, not just the first line
- bash: document 256KB output cap and stderr prefix in description
* fix: address Copilot review on tool descriptions
- bash: say "truncated" not "256KB" (limit is configurable), document
timeout clamping range (1-600) and global fallback
- bash preview: fix "1 more lines" → "1 more line" singular
- plan_agent: remove bash from listed tools (not in AGENT_TOOLS)
* fix: improve memory save error message, narrow dd command filter
Two minor fixes from harness shakedown:
- memory save: split "both name and content required" into separate
errors for missing name vs empty content
- bash safety: replace blanket "dd if=" block with targeted patterns
for writes to block devices (of=/dev/sd*, /dev/nvme*, /dev/disk/,
etc.) and redirects to the same. Legitimate dd use like generating
test data or benchmarking reads is no longer blocked.
* fix: generalize > /dev/sda redirect pattern to > /dev/sd
Copilot review: only /dev/sda was blocked for redirects while
/dev/sdb, /dev/sdc etc were not. Generalize to match any /dev/sd*
device, consistent with the of= patterns.
* feat: edit_file replace_all, write_file append mode, search match count
Three tool enhancements from harness shakedown feedback:
- edit_file: new replace_all parameter replaces all occurrences of
old_string instead of requiring a unique match. Cannot combine with
near_line or edits array.
- write_file: new mode parameter with "append" option. Appends
content to end of file instead of truncating.
- search: output now includes a summary footer showing total match
count and file count (e.g. "47 matches across 12 files").
* fix: address Copilot review on tool enhancements
- replace_all: skip multi-occurrence rejection in pre-validation so
the feature actually works; show occurrence count in preview
- write_file mode: coerce non-string types safely via str()
- search footer: append before truncation to respect output limits
- edit_file error: mention replace_all as alternative to near_line
read_file silently converted null bytes to spaces, showing corrupted
content with no warning. Now samples the first 8KB for null bytes and
returns a clear error directing the user to bash for binary inspection.
* fix: memory delete searches all scopes when scope not specified
Previously delete defaulted to scope=global, so deleting a
workstream-scoped memory without explicitly passing scope=workstream
silently failed. Now tries narrowest scope first (workstream → user
→ global) and deletes the first match. Explicit scope still honored
when provided.
* fix: reject invalid scope on memory delete instead of silent fallback
Copilot review: invalid scope values were silently treated as
unspecified, which could cause accidental deletion from the wrong
scope. Now returns a clear error listing valid scopes.
* fix: exclude build/vendor/VCS directories from search tool
grep -rn recursed into .git, node_modules, target, __pycache__, etc.
producing hundreds of noise hits from generated content. Add
--exclude-dir flags for common directories that should never appear
in search results.
* fix: glob egg-info pattern and add vendor exclude
Copilot review: .egg-info misses turnstone.egg-info (named dirs),
use *.egg-info glob. Also add vendor to the exclude list.
Agent workflows need git for version control, curl for raw HTTP
requests, jq for JSON processing, and man/info for documentation
lookup. All were missing from the slim base image, leaving the man
tool non-functional and standard dev workflows broken.
* fix: block IPv6 loopback/link-local/private in SSRF filter
check_ssrf used gethostbyname which only resolves IPv4. IPv6 addresses
like ::1, fe80::, fd00:: bypassed the filter entirely. Switch to
getaddrinfo which resolves both address families and check all results.
* fix: handle IPv4-mapped IPv6 and zone IDs in SSRF filter
Copilot review caught two bypasses: ::ffff:127.0.0.1 (IPv4-mapped
IPv6) wasn't normalized before private/loopback checks, and fe80::1%lo0
(zone ID suffix) caused a ValueError that was silently swallowed.
Now normalizes IPv4-mapped addresses and strips zone IDs before parsing.
* fix: resolve symlinks before file I/O to prevent path-based bypass
write_file and edit_file followed symlinks silently — a symlink at
/data/link → /etc/passwd would show the /data path in the approval
header while writing to the real target. Three changes:
- open() calls in _exec_write_file, _exec_edit_file, _exec_read_file
now use the resolved (realpath) path instead of the raw symlink
- Approval headers show both paths when a symlink is detected
(e.g. "⚙ write_file: /data/link → /etc/passwd")
- Judge _get_arg_text includes the resolved path so heuristic rules
like write-system-path fire even through symlinks
* fix: address Copilot review — expanduser in fallback, pre-read, image paths
- edit_file exec fallback: add expanduser before realpath (tilde bypass)
- judge _get_arg_text: compare resolved against abspath(expanduser(path))
so ~/ paths don't false-positive as symlinks
- edit_file pre-read: use resolved path instead of raw symlink path
- _exec_read_image: use resolved path for getsize and binary open
* fix: clear dedup sigs after write tools to avoid false repeat warnings
The read→edit→read workflow triggered "identical repeat" warnings
because the dedup tracker compared (tool_name, args) without
considering intervening state changes. Now clears the signature set
when write_file, edit_file, or bash executes successfully, so
subsequent reads of the same file are not flagged.
* fix: use shared error prefixes for write-success detection in dedup
Copilot review: the error detection for write tools only checked
"Error" prefix, missing "Command timed out", "Blocked:", "Denied",
etc. Now shares the same _error_prefixes tuple used by the repeat
detection below, ensuring consistent classification.
The judge pre-converted tool schemas via convert_tools() before
passing them to create_completion(), which internally calls
convert_tools() again. The second conversion tried to extract
function.name from already-converted Anthropic-format tools,
producing empty tool names that the API rejected with
"tools.0.custom.name: String should have at least 1 character".
Fix: pass raw OpenAI-format schemas directly — create_completion
handles the provider-specific conversion.
* feat: add /retry and /rewind commands for conversation history navigation
Allow users to re-send the last message for a new response (/retry) or
drop the last N turns to restore an earlier conversation state (/rewind N).
Both operations sync in-memory state with the persistent database.
Server path includes conversation.modify permission gate, audit trail
(conversation.rewind / conversation.retry events), and thread-safe retry
dispatch. Migration 029 grants the permission to admin and operator roles.
* feat: add message action controls for retry, edit, and rewind in web UI
Hover toolbar on messages with CSS-only icons matching instrument panel
aesthetic. User messages get edit (pencil) and rewind (chevrons) buttons;
last assistant message gets retry (circular arrow). Edit flow uses
event-driven coordination — rewind completes via SSE history event before
send fires. Includes ARIA labels, keyboard nav, touch device support,
reduced motion, and busy-state gating.
Addresses Copilot review feedback on #219:
1. Anthropic _convert_messages: collect tool_use IDs in order (list
not set), filter empty IDs, defer synthetic results until after
real tool results so _merge_consecutive produces correct ordering.
2. Universal repair in reconstruct_messages: synthesize tool results
for mid-conversation orphaned tool calls on DB load. Benefits all
providers (OpenAI is lenient today but may tighten).
3. Test improvements: assert on is_error flag instead of "cancelled"
substring, verify real-before-synthetic ordering in partial results.
When a cancel interrupts tool execution, the assistant message with
tool_use blocks is saved to DB before tools run, but
GenerationCancelled prevents tool results from being created. The
in-memory rollback removes the orphaned message, but the DB row
persists. On resume, Anthropic rejects the conversation with
"tool_use ids were found without tool_result blocks".
Fix: _convert_messages now peeks ahead after each assistant message
with tool_use blocks. If any tool_use IDs lack matching tool_result
messages, synthetic error results are injected (is_error: true,
"Tool execution was cancelled."). Transparent to all callers,
provider-specific (OpenAI is lenient about this).
5 new tests covering: single orphan, multiple orphans, partial
results, complete results (no synthesis), and trailing orphan.
plan_agent and task_agent fail on Anthropic models with "Streaming is
required for operations that may take longer than 10 minutes" from
the SDK. The non-streaming create_completion path used
client.messages.create() which the SDK rejects for thinking-enabled
models.
Fix: use client.messages.stream() internally and call
get_final_message() to get the same Message object. Transparent to
all callers — fixes sub-agents, title generation, summarization,
web fetch, and judge create_completion calls.
Five improvements from Opus self-evaluation of the turnstone harness:
1. Batch edit_file: edits array parameter for atomic multi-edit in a
single tool call. Overlap detection, reverse-order application,
mutual exclusivity with single-edit params.
2. Sandbox packages: new [sandbox] extras group with sympy, numpy,
scipy, pytest — the sandbox already had graceful ImportError
fallbacks, now the packages are actually installed.
3. Stderr labeling: bash tool output prefixes stderr lines with
[stderr] so the model can distinguish errors from stdout.
4. JSON secret redaction: output guard now detects and redacts secrets
in JSON format ("api_key": "...", "password": "...", etc.) with
18 key patterns and 8-char minimum value length.
5. Model persisted on resume: workstream config now saves model and
model_alias. Resume restores the original model via registry
(same path as /model command), falling back to raw model name
if the alias is no longer available.
24 new tests (23 in test_edit_file.py, 1 in test_sessions.py).
stream_end fires per-segment (between tool calls), not per-turn.
The UI was using stream_end to transition to idle, causing a window
where the Send button appeared but the server worker thread was still
alive. User messages submitted during this window were silently
dropped. No Stop button was visible, so the user had no cancel path.
Root cause: state_change events (idle/thinking/running/error) were
only broadcast to the global SSE stream (console dashboard), never
to the per-workstream SSE that the browser UI listens to.
Fix: (1) server.py: on_state_change now also enqueues to the
per-workstream SSE listeners. (2) app.js: stream_end no longer
calls setBusy(false) — it only finalizes markdown rendering.
New state_change handler manages busy transitions: idle/error
set busy=false, thinking/running set busy=true.
* feat: model detect button, capabilities API, and model dropdowns
Admin Models tab: add Detect button that probes a model endpoint to
verify reachability, list available models, detect context_window, and
identify server type (llama.cpp/vLLM/SGLang/OpenAI/Anthropic). Add
static capability lookup endpoint for auto-filling form fields when
a known model name is entered. Add known-models endpoint for datalist
autocomplete suggestions.
Add "openai-compatible" as a third provider option for local servers,
keeping the OpenAI SDK under the hood but suppressing capability
auto-fill and known-model suggestions.
Replace free-text model input with a select dropdown in both console
and server new-workstream modals, populated from a new lightweight
GET /v1/api/models endpoint.
New endpoints:
- POST /v1/api/admin/model-definitions/detect
- GET /v1/api/admin/model-capabilities
- GET /v1/api/admin/model-capabilities/known
- GET /v1/api/models (both console and server)
* fix: accumulate signature_delta for Anthropic thinking blocks (#214)
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
* fix: address PR review — empty base_url, capability leak, response schemas
- Don't pass empty base_url to OpenAI client (falls back to SDK default)
- Return None from lookup_model_capabilities for openai-compatible provider
- Only use static capability table for known models in _detect_openai_compat,
avoiding misleading 200k default for unknown local models
- Add AvailableModelInfo + ListAvailableModelsResponse schemas to both
console_spec and server_spec
- Regenerate TypeScript SDK OpenAPI snapshots
* fix: apply same known-model guard to Anthropic context_window detection
Only report context_window from the static capability table when the
Anthropic model is actually known, matching the OpenAI path fix.
* ui: add autocomplete hint to Model ID label in admin modal
* fix: use explicit kwargs for OpenAI() to satisfy strict mypy
The streaming path captured thinking_delta events but not
signature_delta, leaving the signature empty on round-trip and
causing 400 errors on multi-turn conversations with thinking enabled.
When a local model generates tool calls that are silently dropped
(truncation, missing tool-call-parser), there was zero server-side
logging — making it impossible to diagnose from docker compose logs.
- OpenAI provider: log request params and response summary (debug)
- Session: log stream completion, tool call presence (info), and
tool call discard with names when truncated (warning)
CLI unaffected — log level is WARNING there.
Add model_definitions table (migration 028) enabling model management
via the admin console without SSH access or server restarts. Models
defined in the database coexist with config.toml models through a
per-node merge strategy — config.toml overrides DB for the same alias,
DB-only models coexist alongside, no cross-node contamination.
Storage layer:
- model_definitions table with CRUD (SQLite + PostgreSQL)
- MODEL_DEFINITION_MUTABLE allowlist, admin.models permission
ModelRegistry integration:
- load_model_registry() merges DB + config.toml + CLI models
- context_window=0 auto-detects from provider capability table
or inherits CLI-detected value (same as config.toml behavior)
- ModelRegistry.reload() with validation, TOCTOU-safe accessors
- internal_model_reload + internal_model_status server endpoints
Admin API + UI:
- 6 console endpoints (list, create, get, update, delete, reload)
with admin.models permission, audit trail, provider validation
- Models tab in System group with sky blue (--blue) accent color
- Provider badges (openai/anthropic), source badges (config/db)
- Write-only API keys (never readable, "***" sentinel on update)
- Sync-pending indicator, mobile responsive, focus-trapped modal
Also changes is_secret settings from write-blocked (403) to write-only
across all settings, making judge.api_key configurable via admin UI.
* fix: watch dispatch error handler missing stream_end and state cleanup
The watch dispatch run() closure was missing GenerationCancelled
handling, stream_end emission, on_state_change calls, and the
worker_thread identity guard that the send_message path has. This
left the web UI in a stale state when watch-dispatched sends failed.
* fix: address review feedback - on_stream_end, put_nowait, ws._lock, tests
* fix: ruff lint (unused pytest import)
* fix: send_message() use on_stream_end() instead of raw _enqueue
* refactor: add is_error to on_tool_result protocol, remove text heuristics
Add is_error keyword arg to SessionUI.on_tool_result() so tools
report errors structurally. Server and JS client no longer guess
from output text prefixes — each tool sets the flag at the source.
Bash tool: exit code >= 2 is error, exit code 1 is ambiguous (grep
no-match). History reconstruction keeps text heuristic as fallback
for pre-migration data.
Update SDKs (Python + TypeScript), test mocks, docs, and diagrams.
* fix: infinite recursion in _report_tool_result, signal exits, stale docs
* fix: add _tool_error_flags to test_load_skill ChatSession stubs
Add vendor-js workflow that triggers on Renovate PRs touching
pyproject.toml — detects version changes, runs update-vendored-js.sh,
and commits the actual files back to the PR branch. Supports manual
dispatch via pr_number input for one-off runs.
Providers now register the SDK stream handle eagerly (before returning
the iterator) instead of lazily inside a generator body. This closes
the window where cancel() couldn't abort a blocked HTTP read because
the stream handle wasn't populated yet.
- OpenAI: yield from → return (HTTP call + cancel_ref happen eagerly)
- Anthropic: split into eager __enter__ + _iter_with_cleanup generator
with defensive __exit__ on __enter__ failure
- Console SDK: delete_setting() return type fixed from StatusResponse
to DeleteSettingResponse (matching actual endpoint response)
- 2 new threaded force-cancel integration tests verifying orphaned
threads don't mutate messages and new generations succeed
- Updated _CancelRef docstring, test robustness (assert on wait)
The cancel endpoint emitted a 'cancelled' SSE event before the worker
thread terminated. The frontend transitioned to "send" mode prematurely,
so the next send got rejected with "Already processing a request."
Backend:
- Providers expose SDK stream handle via cancel_ref parameter so
cancel() can close the HTTP connection and unblock iteration
- Generation counter prevents orphaned threads from mutating messages
or clearing cancel state after force cancel
- _check_cancelled() added between retry attempts in _try_stream
- Server polls (async, non-blocking) for cancelled worker to exit
- Force cancel (force:true) abandons stuck worker, keeps cancel event
set so subprocesses are killed, guards against spurious SSE events
Frontend:
- 'cancelled' shows "Cancelling..." then escalates to "Force Stop"
after 2s for a harder cancel that abandons the worker immediately
- 10s safety timeout auto-recovers if stream_end never arrives
- busy_error re-enables stop button instead of showing send
- Timeout cleanup in disconnectSSE, stream_end, and force .then()
- Layout shift prevention (min-width, white-space: nowrap)
- aria-label updates for accessibility
Tests:
- 7 new tests: stream close, error suppression, cancel_ref population,
transport error conversion, non-cancel exception propagation, retry
cancellation check
When a model calls the same tool with identical arguments as a previous
call, append a warning to the tool result and inject a metacognitive
nudge. This breaks loops where small local models get stuck repeating
the same action (e.g. running the same bash command 3+ times).
The repeat signature set is cleared after a warning fires, giving the
model a clean slate. Also cleared on conversation compaction.
Ref: #186
* fix: harden tool call handling for local model servers
Local models (Qwen 3.5 9B, etc.) via llama.cpp produce tool calls with
empty IDs, whitespace-padded names, and malformed JSON arguments. These
defensive gaps caused cascading conversation corruption and silent
failures.
- Strip whitespace from tool names in both main and agent paths
- Generate synthetic UUIDs when tool call IDs are empty/null
- Surface malformed tool call errors to the user via on_error
- Give the model actionable hints (expected JSON format, available tools)
so it can self-correct on retry
- Surface metacognition nudge types to UI via on_info
Ref: #186, #117
* refactor: extract _ensure_tool_call_ids helper, include MCP tools in error
Address Copilot review feedback on PR #200:
- Extract duplicated ID fixup into _ensure_tool_call_ids() static method
- Tests now exercise the actual helper instead of reimplementing the logic
- Unknown tool error now includes MCP tool names alongside builtins
When an HTTP MCP server is unreachable and TCP connect fails immediately
(ECONNREFUSED, DNS failure), the anyio task group inside
streamablehttp_client produces a CancelledError that escapes
asyncio.wait_for and leaves orphaned cancel-scope tasks in an infinite
_deliver_cancellation loop (~800K callbacks/sec).
Three-part fix:
- TCP pre-flight probe (5s timeout) before entering the anyio transport
context — fails fast on unreachable servers, avoiding the bug entirely
- Catch CancelledError in _connect_one with current_task().cancelling()
check to distinguish stray anyio cancels from real shutdown
- _safe_close_stack helper with bounded timeout that never raises,
preventing cleanup errors from masking the original exception
These built-in tools were always sent to the LLM even when no MCP
servers were connected, wasting model turns on calls that would always
return errors. Now gated per-request in _get_active_tools() — same
pattern as the existing web_search gating — using resource_count and
prompt_count for granular filtering.
- TLS: suppress no-any-return + unused-ignore on lacme mtls helpers
(lacme stubs typed locally but not in CI)
- TLS: catch duplicate Prometheus metric registration specifically
(not blanket ValueError) so real setup errors still propagate
- Discord: add missing storage=None to _make_bot mock (policy evaluation
path accesses self.storage)
- Web search: patch _ddg_available in gating test so ddgs availability
doesn't mask the Tavily-only test path
- Bridge stress: close real httpx client before replacing with mock so
daemon threads don't make real HTTP calls or leak connections
- README: comment out tavily_key placeholder in example config
* fix: stream tool errors in real-time with visual error indicator
Tool executors that hit errors (file not found, timeout, write failure,
etc.) were returning error strings without calling ui.on_tool_result(),
so no SSE event was emitted to the browser during streaming. Errors only
appeared after page refresh via history rebuild. Now all error paths
call on_tool_result() so errors stream in real-time.
Added visual error state: tool blocks with errors get a red left border,
red "✗ error" badge, and red error text — matching the existing denied
state pattern but distinct from it. Error detection uses prefix matching
("Error", "Command timed out", "Search timed out") both server-side
(is_error flag on SSE event + _build_history) and client-side (regex
fallback).
Affected tools: bash, read_file, write_file, edit_file, search, math,
man, memory, web_fetch, web_search, plus the run_one catch-all and
prepare-error paths.
* review: expand error prefix detection per copilot feedback
Add Unknown tool, JSON parse error, MCP prompt timed out, and MCP
prompt error to the is_error prefix list in on_tool_result, _build_history,
and the frontend regex. These error messages were confirmed in the
codebase but missing from the detection heuristic.
Replayed or cancelled conversations could produce assistant messages
with content=None and no tool_calls, which OpenAI-compatible APIs
reject with a 400. Fix at three layers for defense in depth:
- session.py: use empty string instead of None when building assistant
messages (streaming + cancellation paths)
- _utils.py: normalise content on DB load in reconstruct_messages()
- _openai.py: add _sanitize_messages() catch-all at provider boundary
Closes#194
When the LLM judge is enabled and the user makes rapid approval
decisions, judge daemon threads pile up competing for the inference
server, causing timeouts. Pass a threading.Event from session to the
judge — set on approval — so the daemon abandons remaining work
within ~1s, including shutting down the executor to kill in-flight
API calls.
* fix: TLS Docker end-to-end testing fixes
Fixes discovered during Docker Compose TLS integration testing:
- Dockerfile: use --extra all (prevents missing optional deps)
- lacme 1.0.3: fixes CACertificateIssued event logging crash
- chmod PermissionError: guard for Docker volume mounts
- socket import: moved to top of main() (was inside TLS conditional,
caused NameError in _default_node_id)
- redis.SSLConnection: ConnectionPool needs explicit connection_class,
not ssl=True (which only works on Redis() directly)
- Empty redis password: pass None instead of "" to avoid AUTH error
- TURNSTONE_CONSOLE_URL: env var for Docker service discovery
(0.0.0.0 bind address isn't reachable from other containers)
- HTTP01Handler: ACME client needs a challenge handler even when
server auto-approves
- Docker overlay: tls-init as root with chmod, Redis conditional
password, console Redis TLS flags, TURNSTONE_CONSOLE_URL
* feat: full mTLS end-to-end with lacme 1.0.4
Completes the mTLS chain across all services:
lacme 1.0.4:
- Dual EKU certs (serverAuth + clientAuth) — fixes mTLS rejection
- Configurable CA name (name="turnstone") — consistent store key
Bootstrap CA import:
- Console imports bootstrap CA from /certs volume on first boot
- Single trust root: bootstrap CA → console → all service certs
Bridge mTLS:
- TLSClient init when TURNSTONE_TLS_ENABLED set
- Auto-upgrades server URL from http:// to https://
- SSLContext passed to all 3 httpx clients via verify=
Console collector mTLS:
- upgrade_tls() method replaces httpx client with mTLS context
- Called in lifespan after cert issuance alongside proxy upgrade
- Fixes "Failed to poll node" when server serves HTTPS
Docker overlay:
- TURNSTONE_TLS_SANS on all services (Docker service names as SANs)
- TURNSTONE_TLS_ENABLED on bridge
- Channel service with Redis TLS flags
- TURNSTONE_CONSOLE_URL for service discovery
- Server healthcheck disabled (mTLS healthcheck deferred)
- Redis conditional password from env
Verified end-to-end: bootstrap → console CA → server HTTPS →
bridge mTLS → Redis TLS → channel Redis TLS → console collector
polls server over mTLS → workstream creation works through bridge
* fix: lint + copilot feedback on TLS Docker e2e
- SIM105: contextlib.suppress(PermissionError) for chmod
- F401: remove unused get_storage import in bridge
- Redis healthcheck: pass password when REDIS_PASSWORD is set
* fix: sort imports in admin.py and bridge.py
* fix: tls-init key permissions, healthcheck env, collector race
- tls-init: add set -e, chown to turnstone:turnstone with restrictive
perms (keys 0600, certs 0640, dirs 0750) instead of world-readable
- Redis healthcheck: use container runtime $$REDIS_PASSWORD instead of
Compose-time interpolation for consistency with --requirepass block
- collector upgrade_tls(): don't close old httpx client while concurrent
poll threads may still be using it — let GC handle cleanup
TLSManager class owns the internal CA, ACME responder, and cert
lifecycle:
- CertificateAuthority with DB-backed storage (StorageStore adapter)
- ACMEResponder mounted at /acme (auto_approve for internal network)
- Dual cert issuance: internal CA for mTLS, optional external ACME CA
for frontend HTTPS (Let's Encrypt via acme_directory setting)
- Auto-renewal via RenewalManager with clean async shutdown
- Expired cert detection on reload (re-issues instead of loading stale)
- EventDispatcher wired to structlog + lacme Prometheus metrics
- GET /v1/api/admin/tls/ca.pem — root cert download
- GET /v1/api/admin/tls/ca — CA status + cert inventory
- Console lifespan: init CA, issue certs, start renewal, stop on shutdown
- 11 async tests covering CA lifecycle, cert persistence, SSL contexts,
endpoints, and event wiring
Update plan_agent tool pattern to show codebase exploration before
delegating to the planning agent. This pattern achieved 100% pass
rate (160/160 runs) on the eval suite — up from 98% on the previous
prompt.
Remove console-proxy from trusted_sources — end-user tokens via the
console proxy already carry the real user_id in the JWT, so they must
not be able to override it via the request body (impersonation risk).
Only bridge and console service identities are trusted to forward
user_id on behalf of users. Add 5 tests covering the trust boundary.
Usage tab grouped by user showed raw hex user_id instead of username.
Audit tab showed truncated hex. Now both API endpoints resolve user_id
to username via list_users() lookup before returning the response.
Update security.md with trusted service user_id forwarding. Update
MQ protocol diagram to include user_id field on CreateWorkstreamMessage.
Update console data flow diagram to show user_id in message and bridge
forwarding.
Console create_workstream was constructing CreateWorkstreamMessage
without setting user_id, so workstreams created via console→MQ→bridge
→server had empty user_id in usage events. Now the console extracts
user_id from auth_result and passes it through the MQ message. The
bridge forwards it in the HTTP payload, and the server accepts it
from trusted service callers (bridge, console-proxy).
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
- Record prompt_variant in parallel execution path (was serial-only)
- Handle "Subprocess error:" and "Skipped (fast-fail)" in failure
classifier instead of mis-bucketing as missing_tool
- Build case_id->case_def dict once in _build_failure_analysis,
_run_analyst, _propose_prompt_modification (was O(n²) linear scan)
- Enforce original prompt at slot 0 of cached user_prompts variants
- Use wall clock time for TSV elapsed_s instead of sum of run times
_suppress_stdout() was setting sys.stdout = StringIO() which is
process-global. When send_headless runs in a ThreadPoolExecutor and
blocks on an API call inside the suppression context, the main
thread's print() calls silently go to the StringIO and vanish.
Switch to os.dup2 fd-level redirect which is thread-safe.
Replace linear optimization chain with UCB1 evolution tree. Each
iteration selects the most promising node to extend, preventing
irrecoverable collapse from bad edits. Also adds improvement-based
delta feedback to the optimizer and optional holdout set separation.
Inspired by "Learning to Self-Evolve" (arXiv:2603.18620).
Migrations 014, 021-024 used bare op.add_column/alter_column/drop_column
which fails on SQLite (no ALTER of constraints). Switch to
batch_alter_table and enable render_as_batch in env.py. Migration 014
also moved UniqueConstraint inline into create_table.
Hotfix: DDG web search returning empty results.
- Switch dependency from duckduckgo-search (deprecated shim, empty
results) to ddgs>=9.0 (actively maintained successor)
- Include ddg extra in Docker image for free web search fallback
- Update all user-facing install instructions to reference ddgs
duckduckgo-search 8.x is a deprecated shim that returns empty results
(upstream temporarily disabled HTML/Lite backends, Bing backend broken).
The package was renamed to ddgs in v9.x which works correctly.
Address Copilot + code review feedback:
- Fix mypy: rename tool_names → _policy_names in CLI to avoid type clash
- Tighten env scrub from substring to suffix matching (_KEY, _TOKEN, etc.)
to avoid false positives on MONKEYTYPE, KEYBOARD_LAYOUT
- Add DATABASE_URL/TURNSTONE_DB_URL to explicit scrub list
- Bridge: use _storage directly instead of get_storage() which
auto-initializes a local SQLite DB with no admin policies
- Discord bot: add self.storage None guard
- Add tests for suffix-only matching and false positive avoidance
New turnstone/core/env.py provides scrubbed_env() that strips API keys,
tokens, passwords, and credentials from os.environ before passing to
subprocesses. Applied to all 5 subprocess call sites: _exec_bash,
_exec_search, _exec_man, watch _run_command, and MCP stdio servers.
Pattern-based scrubbing (KEY, SECRET, TOKEN, PASSWORD, CREDENTIAL
substrings) plus explicit blocklist for known secrets. Safe vars
(PATH, HOME, locale, etc.) always preserved. Passthrough list for
operator overrides.
evaluate_tool_policies_batch() was only called in the server WebUI.
CLI, bridge, and channel entry points now evaluate admin-defined tool
policies before auto-approve checks. Deny policies block tools, allow
policies auto-approve. Best-effort: gracefully skipped if storage is
unavailable.
_run_agent (plan + task agents) now passes tool results through
_evaluate_output() before appending to context — same as the main
session loop. Runs before truncation so the guard sees full output.
Catches prompt injection, credential leakage, and encoded payloads
in agent tool results that were previously unscanned.
* fix: bridge approval & plan review TOCTOU races (#158, #159)
Replace "pop on completion" with a tombstone pattern — pending entries
are marked resolved=True instead of being removed, eliminating the
window where stale SSE reconnect events bypass the duplicate guard.
Resolved tombstones are cleaned up on ws_state events and ws_closed.
Stress tests now pass reliably (previously ~12-16% failure rate).
* review: extract _mark_resolved helper, use real ws_closed path in test, add refinement loop test
Address code review suggestions:
- Extract _mark_resolved() helper in _wait_plan to reduce duplication
- Document cross-stream ordering assumption for plan review refinement
- Race 6 test now calls _handle_global_event instead of manual dict pops
- New Race 7 test validates plan review refinement loop (tombstone → cleanup → re-entry)
* fix: add TTL fallback for tombstone cleanup when global SSE lags
If the global SSE stream is temporarily down while per-WS SSE continues,
resolved tombstones would block legitimate new approvals/plan reviews.
Add a 30s TTL so stale tombstones are expired in the duplicate guard
as a fallback to the normal ws_state-based cleanup.
Also changes tombstone type from (request_id, bool) to
(request_id, float) where 0.0 = active, >0 = resolved_at monotonic time.
* fix: use 3x approval_timeout for tombstone TTL instead of hardcoded 30s
Tie the TTL to the configurable approval_timeout (default 300s = 900s TTL)
rather than a short hardcoded value. The TTL is only a fallback for when
the global SSE stream is completely down — a conservative value is safer.
* feat: pluggable web search backends (DDG, Tavily, MCP)
web_search is now an abstract capability with swappable backends:
- DuckDuckGoClient — free, no API key, uses duckduckgo-search library
- TavilyClient — existing behavior, requires API key
- MCPSearchClient — delegates to any MCP server tool
New tools.web_search_backend setting (ConfigStore + --web-search-backend
CLI flag): '' (auto), 'tavily', 'ddg', or 'mcp:server:tool'.
Auto-detection (default): Tavily if key present, else DDG if installed,
else disabled. This means users with duckduckgo-search installed get
web search for free with local models — no API key needed.
Closes#131
* fix: address Copilot review on pluggable web search
- Unknown backend values now log warning + return None (not silent
fallthrough to auto-detect)
- Pass timeout to DDGS constructor
- MCP: use math.ceil for timeout, forward topic kwarg
- Fix agent mode web_search gating to use _resolve_search_client()
instead of get_tavily_key() (was still using old check)
- Update docstring to reflect new backend resolution
- Fix DDG test to patch DDGS import properly
- Add test for unknown backend rejection
The script detected the old version from pyproject.toml, which Renovate
had already updated. This caused OLD_DIR == NEW_DIR, so the script
downloaded files then immediately deleted them.
Fix: detect old version from the actual directory on disk. Add a guard
that errors if old == new version to prevent silent data loss.
Also: run the fixed script to vendor katex 0.16.40 (fonts + css + js).
* feat: --config flag and $TURNSTONE_CONFIG env var for config.toml path
Add set_config_path() to config.py with three-tier resolution:
1. --config CLI flag (via set_config_path)
2. $TURNSTONE_CONFIG environment variable
3. ~/.config/turnstone/config.toml (default)
--config added to all 5 entry points that load config.toml: CLI,
server, console, bridge, eval. Uses parse_known_args pre-parse so
the path is resolved before apply_config reads the file.
Closes#130
* fix: centralize --config pre-parse, fix help and docstrings
- Add add_config_arg() helper with separate pre-parser (add_help=False)
so --help still shows config-derived defaults
- Replace duplicated pre-parse blocks in all 5 entry points
- Fix set_config_path docstring (works after load_config too)
- Fix module docstring precedence description
- Remove redundant import os in get_tavily_key
* feat: add PostgreSQL CI integration tests
Add --storage-backend pytest option and shared storage_backend fixture
in conftest.py that creates SQLiteBackend or PostgreSQLBackend based
on the flag. Migrate 13 storage test files to use shared fixture
instead of local SQLiteBackend fixtures.
Add test-postgres CI job with PostgreSQL 17 service container that
runs the full test suite against real PostgreSQL.
* fix: use TRUNCATE CASCADE for PG cleanup, wrap in try/finally
TRUNCATE is faster than per-table DELETE and resets autoincrement
sequences. try/except ensures reset_storage() always runs even if
cleanup fails due to a corrupted connection from a failing test.
* fix: document _engine coupling in PG cleanup comment
* feat: live session config via ConfigStore point-of-use reads
Existing sessions now pick up admin settings changes without requiring
workstream recreation. ChatSession reads MemoryConfig and JudgeConfig
behavioral flags from ConfigStore at point-of-use via _mem_cfg and
_judge_cfg properties. Judge LLM client config (model, provider,
base_url, api_key) stays frozen from creation time.
Falls back to frozen dataclasses when ConfigStore is absent (CLI mode).
* fix: type config_store param, clarify _ensure_judge guard comment
* fix: re-check live judge.enabled on every _ensure_judge call
Copilot correctly identified that the cached judge was returned without
re-checking the live enabled flag. Move the enabled check before the
cache check so disabling the judge via admin takes immediate effect.
Also defer JudgeConfig import to local scope in _judge_cfg property
to avoid pulling in the full judge module at import time.
Add test for disable-after-init scenario.
* fix: medium reliability — SQLite WAL + timeout, eviction cancel, title retry
M1: Increase SQLite busy timeout to 30s and enable WAL journal mode
for better concurrent read/write. Prevents OperationalError under
multi-workstream write contention.
M2: Call session.cancel() during workstream eviction cleanup so
in-flight worker threads stop promptly instead of running to
completion on an evicted workstream.
M3: Reset _title_generated flag on exception so title generation
retries on the next successful exchange instead of permanently
giving up after one failure.
* fix: address review — WAL pragma error handling, title retry ws_id guard
Wrap WAL pragma in try/except and verify returned mode. Log warning if
WAL is not enabled (e.g., filesystem permissions) instead of aborting
connection.
Guard title retry flag reset with ws_id comparison to prevent
re-enabling titling for a different workstream after /resume.
* fix: address review — add title retry tests
Two tests verifying _title_generated flag behavior: reset on failure
(allows retry on next turn), stays True on success. Covers the new
retry logic added in this PR.
* fix: guard title update success path against ws_id change during resume
Use captured ws_id on success path (not just failure path) so a
concurrent resume() can't cause the background title thread to rename
the wrong workstream. Add test for the race scenario.
The probe loop continued probing while the circuit was OPEN but never
transitioned to HALF_OPEN — that only happened inside
acquire_request_permit() which requires a user request. If no user
sends a message during the cooldown, the circuit never recovers.
Now the probe loop checks cooldown elapsed and transitions to HALF_OPEN
before probing, so recovery happens automatically without user
interaction.
* fix: align ConfigStore implementation with spec
- Add cluster + skills sections to admin UI settings order and labels
- Return default value in DELETE /v1/api/admin/settings response per spec
- Document 4 missing settings in docs/settings.md (trusted_proxies,
output_guard, redact_secrets, discovery_url) and correct count to 48
- Wire ConfigStore into console server replacing 4 raw
get_system_setting() calls with validated/cached config_store.get()
- Reload console ConfigStore on settings mutations via
_publish_config_change()
- Update registry URL tests for ConfigStore-based resolution
* fix: address Copilot review feedback on ConfigStore PR
- Move config_store.reload() before collector guard in
_publish_config_change() so cache refreshes even without collector
- Add DeleteSettingResponse schema and update OpenAPI spec to match
the actual delete response (status + key + default)
- Add test asserting default field in delete response
- Fix stale docstring in test helper
* perf: add conversations.timestamp index, batch config saves, cache capabilities
P1: Add idx_conversations_timestamp index (migration 025) to eliminate
full table scans on search_history_recent ORDER BY timestamp DESC.
P2: Batch save_workstream_config — replace N separate SQL statements
with single executemany call. SQLite uses INSERT OR REPLACE,
PostgreSQL uses INSERT ON CONFLICT DO UPDATE.
P3: Cache _get_capabilities() result on ChatSession — called 4-6x per
turn but deterministic for session lifetime. Invalidated on model
switch.
* fix: address review — capabilities cache bypassed for fallback models
Cache only applies to the primary session model. Fallback models
(different provider/model passed to _get_capabilities) resolve fresh
to avoid stale capability flags affecting tool selection and web search.
Replace bare `import logging` / `logging.getLogger(__name__)` with
`from turnstone.core.log import get_logger` / `get_logger(__name__)`
across all core modules and server.py. This enables structured log
context injection (node_id, ws_id, request_id) in modules that
previously used plain stdlib logging.
Also adds _ensure_stdlib_factory() to log.py for pytest caplog
compatibility when configure_logging() hasn't been called.
Renames `logger` to `log` in skill_sources.py for naming consistency.
Mark platform as experimental beta with explicit disclaimers: no
guarantees of determinism, reliability, or backward compatibility.
Advise thorough evaluation before deployment.
* fix: critical reliability fixes for production readiness
C1: Add 1-hour timeout to _approval_event.wait() and _plan_event.wait()
to prevent permanent worker thread hangs when users disconnect.
C2: Atomically check-and-start worker thread under Workstream._lock to
prevent race condition where two concurrent send_message requests
spawn duplicate workers on the same non-thread-safe ChatSession.
C3: Bound _watch_pending queue to maxsize=20 to prevent OOM under
heavy watch load with busy workstreams.
H1: Add timeout to proc.wait() (10s) and stderr_thread.join() (5s)
after SIGKILL to prevent indefinite hang on D-state processes.
H2: Protect _pending_verdicts with _ws_lock at all three mutation sites
(reset in approve_tools, append in on_intent_verdict, swap-and-clear
in resolve_approval) to prevent lost verdicts from concurrent
judge daemon and approval threads.
H3: Bound global SSE queue to maxsize=10000 with put_nowait() and
contextlib.suppress(queue.Full) for backpressure. Prevents
unbounded memory growth when fanout thread is overloaded.
H4: Bridge SSE threads for closed workstreams now check ws_id membership
in _ws_threads before reconnecting, preventing thread leak on
workstream close.
* fix: address review — verdict lock consistency, watch queue non-blocking, SSE drop logging
- Move _last_verdict_decision set inside _ws_lock in resolve_approval()
so swap+decision is atomic with on_intent_verdict() reads
- Read _last_verdict_decision under _ws_lock in on_intent_verdict()
- Build heuristic_verdicts locally then assign under lock in approve_tools()
- Use resolve_approval() for timeout path so verdicts are updated consistently
- Watch queue producer uses put_nowait with log on Full (prevents WatchRunner hang)
- Global SSE state broadcasts log on queue.Full instead of silent suppress
- Plan event wait also gets 1-hour timeout (same class of bug as approval)
On Python 3.14, Future.exception() raises CancelledError on cancelled
futures instead of returning None. Check future.cancelled() before
calling exception() to prevent crash when the MCP event loop shuts down
before _connect_all completes.
* feat: add priority column for skill ordering control
Add priority INTEGER DEFAULT 0 column to prompt_templates (migration
024). Skills with activation="default" are now ordered by priority ASC,
name ASC instead of name-only. Admins can set priority via create/update
API. Lower values run first. Priority is editable on readonly/installed
skills. 4 new tests. Python SDK, TypeScript SDK, and Pydantic models
updated.
* fix: address review — apply priority ordering to list_default_templates
list_default_templates() still ordered by name only, so priority had
no effect on default skill execution order. Update both SQLite and
PostgreSQL backends to order by (priority, name).
* fix: address review — regenerate OpenAPI snapshot, add default template ordering test
Regenerate openapi-console.json to include priority field on skill
models. Add test_list_default_templates_ordered_by_priority to verify
the execution path for default skills respects priority ordering.
* fix: use approval_label for per-tool always-approve in CLI and bridge
The server stores approval_label (e.g. mcp__server__tool) for per-tool
auto-approve, but CLI and bridge extracted only func_name (bare tool
name). This caused always-approve decisions to not carry over across
access paths. Align CLI and bridge to prefer approval_label with
func_name fallback, matching the server's WebUI.approve_tools() pattern.
* fix: address review — exclude errored items from bridge auto-approve check
Filter out items with error set from the auto-approve subset check,
matching the server's WebUI.approve_tools() behavior. Prevents
policy-denied items from affecting auto-approve decisions.
The workstreams.skill_id and skill_version columns were already being
populated correctly (wired in the skills unification PR #106). Add two
tests confirming: lineage columns set when skill is applied, and
defaults when no skill is used. Check off the PROGRESS.md item.
* test: add governance SDK integration tests against real Starlette app
24 TestClient-based tests verifying round-trip serialization of SDK
governance methods (roles, policies, orgs) against actual route
handlers with SQLite storage. Covers create/list/update/delete
lifecycles, error cases, and Pydantic model field validation.
* fix: address review — close AsyncClient in sdk_client fixture teardown
Convert sdk_client fixture to async context manager so the httpx
AsyncClient is properly closed after tests, avoiding resource leak
warnings.
* test: add MCP reload and reconcile endpoint integration tests
11 new tests covering POST /v1/api/admin/mcp-servers/reload (console)
and POST /v1/api/_internal/mcp-reload (node). Verifies reconcile_sync
invocation, fan-out results, permission checks, missing storage
handling, and mixed node error propagation.
* fix: address review — lazy-import internal_mcp_reload to avoid heavy module load
Move turnstone.server import inside _routes_with_internal() helper so
the full server module (which reads UI static assets) is only loaded
when node-side endpoint tests actually run, not during test collection.
* fix: validate OIDC issuer URLs against SSRF before discovery fetch
Add validate_issuer_url() that rejects private/loopback/link-local IPs,
non-HTTPS (except localhost for dev), embedded credentials, and
unresolvable hostnames. Called before the HTTP fetch in discover_oidc()
so the request is never made for invalid URLs. 17 new tests.
* fix: address review — use is_global, redact userinfo, catch ValueError
Use `not addr.is_global` instead of individual range checks to cover
all non-routable addresses (CGNAT, unspecified, multicast). Redact
credentials from error messages to prevent log leakage. Catch ValueError
from ip_address() for zone-indexed IPv6 addresses.
* test: skill session config application to workstreams
13 TestClient-based integration tests verifying that skill session
config fields (model, temperature, token_budget, auto_approve,
allowed_tools, reasoning_effort, agent_max_turns) are correctly applied
to ChatSession and WebUI when creating a workstream with a skill.
Covers: individual fields, combined application, disabled/unknown skill
rejection, zero-value no-ops.
* fix: address review — pass skill kwarg, clarify no-op test assertions
Pass skill=kwargs.get("skill") into ChatSession in test factory to
match production behavior. Clarify zero-value no-op test docstrings
to document they verify the handler's guard conditions, not observable
state changes.
* fix: memory access tracking and BM25 context caching
Add touch_structured_memory/touch_structured_memories to storage
protocol + SQLite/PostgreSQL backends. Bumps last_accessed and
access_count on memory retrieval (BM25 injection + search results).
9 new storage tests.
Cache the scored BM25 memory context string on ChatSession, invalidated
on memory save/delete. Eliminates ~12 redundant storage queries + index
rebuilds per session lifecycle.
* fix: address review — deduplicate keys in touch facade, clarify contract
Deduplicate keys in the memory.py facade before calling storage so each
distinct memory is touched at most once. Update protocol docstring to
clarify per-call increment semantics. Add deduplication unit test.
* fix: replace unused-import test with real batch duplicate test
Replace facade dedup test (which only tested Python set logic) with a
real storage-level test that verifies duplicate keys each increment
access_count. Fixes ruff F401 lint failure.
Skill methods on TurnstoneConsole and AsyncTurnstoneConsole returned
dict[str, Any] instead of validated Pydantic models. Update list_skills,
create_skill, get_skill, update_skill, list_skill_resources,
create_skill_resource, and install_skill to use response_model= with
ListSkillsResponse, SkillInfo, SkillResourceInfo, and
SkillInstallResponse.
8 tests covering the DB setting → config.toml → default URL resolution
chain, including storage errors, empty values, malformed JSON, and
documenting that RuntimeError propagates uncaught through the except
clause.
* fix: add split pane button to tab bar for discoverability
The split pane feature was only accessible via right-click context menu
or Ctrl+\ keyboard shortcut. Add a subtle split icon (⧉) to the tab
bar that appears at low opacity in single-pane mode. Hidden in
multi-pane mode where pane headers already provide split/close controls.
* fix: address review — change tab-bar from tablist to toolbar role
The tab bar contains both tabs and action buttons (new workstream,
split pane), which is invalid for role=tablist. Change to role=toolbar
which correctly describes a container of mixed interactive controls.
* fix: address design review — WCAG contrast, ARIA structure, mobile
- Drop opacity approach, use border: dashed var(--border) matching
#new-tab-btn pattern (fixes WCAG contrast failure at 35% opacity)
- Nest tabs in #tab-list[role=tablist] inside toolbar (fixes invalid
role=tab children inside role=toolbar)
- Hide split button on mobile (<600px) where splits can't work
- Add aria-keyshortcuts to both action buttons
* fix: enable output guard in CLI mode
The heuristic output guard (credential redaction, prompt injection
detection) only ran in server mode because cli.py never constructed a
JudgeConfig. Additionally, the guard condition in session.py required
enabled=True, coupling the zero-cost heuristic (<5ms) to the full LLM
judge.
Wire JudgeConfig from existing CLI args into the session factory.
Decouple the output_guard condition from the enabled flag so the
heuristic guard runs even when the LLM judge is disabled via --no-judge.
* fix: address review — pass config.toml judge fields to CLI JudgeConfig
apply_config() merges [judge] section from config.toml into args as
judge_base_url and judge_api_key. Pass these through to JudgeConfig so
the CLI respects config.toml judge settings (e.g. separate judge
endpoint).
* fix: validate URL scheme after MCP registry template substitution
resolve_install_config() substitutes user-provided values into URL
templates via string replacement without validating the resulting URL.
Add urlparse check after substitution to reject non-HTTP(S) schemes,
preventing SSRF-style redirection through crafted template variables.
* fix: address review — reject empty hostname and embedded credentials
Add hostname presence check and userinfo rejection after URL scheme
validation. Prevents URLs like https:///path (no host) and
https://user:pass@host (credential leakage in config). Two new tests.
* fix: server startup stampede — timeout model detection, non-fatal PG migrations
detect_model() blocked the main thread for up to 400s when the LLM backend
was unreachable (OpenAI SDK default: 600s read timeout × 2 retries × TCP
retransmit). Cap startup detection at 10s with no retries — the
BackendHealthMonitor handles ongoing availability probing after startup.
PostgreSQL migrations via _run_with_pg_lock() crashed the server on lock
contention when 10 containers stampeded the advisory lock simultaneously.
Wrap in try/except matching the SQLite path — the entrypoint script already
runs migrations before the server process starts.
Health check start_period increased from 15s to 60s to accommodate the
startup sequence under load.
* fix: address review — narrow PG migration except, add detect_model test
Narrow the PG migration except clause to (OSError, EOFError) so DDL
errors still propagate. Add two unit tests for detect_model() verifying
with_options(timeout=10, max_retries=0) is called and that connection
errors in non-fatal mode return (None, None).
* feat: raise scaling limits for 1000-node clusters
Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.
Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers
Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling
Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster
New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)
Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.
* fix: add image tag to compose anchors to avoid redundant builds
All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.
* fix: address Copilot review feedback on scaling PR
- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor
* fix: add PostgreSQL env vars to cluster bridge anchor
Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.
* fix: address Copilot round 2 + PG connection exhaustion at startup
Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change
Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service
* fix readme
* fix: startup resilience for large clusters
Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.
Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.
Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).
Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service
* fix: replace silent error suppression with structured logging
Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.
memory.py (26 changes):
Every storage operation now logs on failure. Previously the entire
persistence facade had zero logging — messages, workstream state,
and structured memories could silently stop being saved.
server.py:
Usage recording failures now log at warning (was pass).
Global SSE fan-out errors log at debug (was pass).
console/server.py:
Config reload notification logs per-node failures at warning.
Settings read fallbacks log at warning with the default value used.
auth.py:
User existence check logs at warning (was pass).
Setup rollback failures log at error (was suppress).
OIDC state cleanup logs at debug (was suppress).
mcp_client.py:
DB-managed MCP server list failure logs at warning (was pass).
collector.py:
Node poll failure upgraded from debug to warning with exc_info.
Health fetch failure logs at debug with exc_info (was silent).
bridge.py:
Best-effort plan rejection logs at warning (was suppress).
Malformed SSE data logs at debug (was suppress).
session.py:
Tool output UI callback failure logs at debug (was suppress).
* fix: stagger collector poll with deterministic per-node jitter
Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.
Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.
* fix: add startup jitter to bridge heartbeat and health monitor probe
Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.
Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.
Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.
* fix: split collector httpx timeout and raise keepalive pool
Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.
* fix: narrow detect_model return type for CLI and eval callers
detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
Drag ratio bounds were hardcoded at 0.1/0.9 which allowed panes to be
resized below their CSS min-width (200px) / min-height (150px), causing
input areas and text to overflow and clip. Now compute bounds dynamically
from the container size and CSS minimums.
* feat: split-pane layout for chat UI
Refactor the server UI from a single-pane global-state design to a
multi-pane architecture with per-workstream Pane instances and a binary
layout tree. Each pane has its own SSE connection, message area, input,
and state (busy, approval, streaming).
Phase 1 — Pane class with 25 prototype methods encapsulating all
per-workstream state. Phase 2 — binary split tree (leaf/split nodes)
with recursive flexbox rendering and drag-to-resize handles. Phase 3 —
keyboard shortcuts (Ctrl+\, Ctrl+Shift+\, Ctrl+Shift+W, Ctrl+Alt+Arrow)
and right-click context menu. Phase 4 — layout persistence via
localStorage.
Key design decisions:
- No duplicate workstreams across panes (split refused if no unused ws,
auto-close redundant pane on ws deletion)
- Max 6 panes to avoid exhausting browser SSE connections
- Viewport guard prevents splitting below min-width/min-height
- Only focused pane refreshes workstream list on SSE reconnect (prevents
race when multiple panes disconnect simultaneously)
- Tab click focuses existing pane showing that ws in multi-pane mode
- Pointer events on drag handles for mouse + touch support
- Full a11y: ARIA roles/labels, keyboard nav in context menu, focus
restoration, prefers-reduced-motion coverage
* fix: address PR #127 review feedback
- Add focusin handler so keyboard focus (Tab) updates focusedPaneId
- Context menu skips interactive elements (textarea, input, links,
buttons) so native copy/paste and link context menus work
- Split handles get ARIA role=separator, aria-orientation, aria-valuenow,
keyboard resizing (arrow keys, Home/End), and tabindex=0
- Enforce MAX_PANES limit in deserializeLayout to prevent corrupted
localStorage from creating too many panes/SSE connections
- Update architecture.md to document split-pane layout
* fix: collector JWT expiry causes silent workstream data wipe
The console collector baked a one-time JWT snapshot into its httpx
client headers at startup. After 1 hour (JWT expiry), every poll to
server nodes returned 401. The error JSON was silently parsed as valid
empty data, wiping all workstream state while nodes still appeared
reachable — the cluster showed "10 nodes, 0 workstreams."
Root causes fixed:
- Collector: no auth baked into httpx.Client; per-request headers
from ServiceTokenManager.token (auto-rotating) or static fallback
- Proxy: same pattern — proxy_client/proxy_sse_client created without
auth headers; _proxy_auth_headers() injects fresh token per-request
- main(): static token snapshot only passed when no token_manager
exists, preventing stale JWT from being stored anywhere
- _fetch_node: raise_for_status() before .json() so 401s throw
instead of returning error JSON as "0 workstreams"
- Auth errors (401/403) logged at warning level for operator visibility
* fix: address PR #126 review — type annotation, regression tests, log messages
Tighten token_manager type from Any to ServiceTokenManager | None.
Add two regression tests verifying 401/403 poll responses preserve
existing workstream data and mark nodes unreachable. Fix misleading
log messages: "jwt_minted" → "token_manager_created" since
ServiceTokenManager mints lazily on first .token access.
* fix: auto-titler SSE event + SSE reconnection after restart
_generate_title() now calls self.ui.on_rename() after persisting the
title, so the tab bar, bridge, and console all update in real time.
Also handles multi-part (vision) content and replaces silent except
with log.debug.
SSE onerror handler now parses the workstreams response, replaces the
stale workstreams map, and switches to the first available workstream
if the current ws_id no longer exists (e.g. after server restart).
Previously it retried the stale ws_id forever.
* fix: address PR #125 review — avoid double reconnect + sync tab bar
Return immediately after switchTab/showDashboard on stale ws_id to
prevent scheduling a redundant connectContentSSE via setTimeout.
Always re-render tab bar after replacing the workstreams map so DOM
stays in sync even when currentWsId is still valid.
* fix: wire resume_ws through console + expose max_ws in heartbeat
Console create_workstream handler now reads resume_ws from the request
body and passes it to CreateWorkstreamMessage on all three dispatch paths
(pool, auto, explicit). Previously resume only worked via channel router
and direct CLI — the console layer never plumbed it through.
Server /health now includes max_ws from WorkstreamManager. Bridge reads
it on startup and includes it in heartbeat metadata so the console's
_pick_best_node gets accurate capacity instead of always defaulting to 10.
Collector also updates max_ws on subsequent heartbeats (not just discovery).
Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks
fixed for new max_workstreams property access in /health.
* fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id
Add _fetch_server_metadata() so bridge reads max_ws from /health even
when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats
would advertise max_ws=10 regardless of actual server config.
Add 3 test cases verifying resume_ws flows through all three console
dispatch paths (directed, pool, auto-select).
Extract _resolve_capabilities() shared helper so _get_capabilities()
and _run_agent() use the same config-override logic instead of
duplicating inline. Add _without_tool() module-level helper to
deduplicate the tool-filtering listcomp.
Add UI error notification and exc_info logging to run_one() exception
handler so tool failures are visible in the frontend. Apply config.toml
capability overrides when gating web_search in _run_agent(), matching
the pattern used by _get_capabilities().
Two bugs: (1) an uncaught exception in one parallel tool call killed the
entire batch via pool.map(), losing all results including successful ones.
Wrap run_one() in try/except so failures return error strings instead of
propagating. (2) web_search was offered to local models even without a
Tavily API key — the model would attempt it, only to fail at execution
time. Filter web_search from _get_active_tools() and _run_agent() when
neither native support nor Tavily is available.
Closes https://github.com/turnstonelabs/turnstone/issues/117
- Null-safe extraction for description, license, and compatibility in
skill_parser.py — YAML bare keys (e.g. `description:`) no longer
produce the literal string "None"
- Log warning on skill catalog storage failure instead of silent swallow
- Use `enabled == 1` in list_skills_by_activation for consistency with
other prompt_templates queries in both storage backends
- Add parser tests for YAML null description, license, and compatibility
- Update governance.md: document runtime config editing on installed
skills, two-column modal layout, SPDX license dropdown, origin badge
- Regenerate OpenAPI snapshots (openapi-console.json) to include license
and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest
- Omit version from create/update payloads when blank so server applies
default "1.0.0" instead of storing empty string
- Push enabled_only + limit filters into list_skills_by_activation storage
query (protocol, SQLite, PostgreSQL) instead of loading all rows and
filtering in Python; session.py now passes enabled_only=True, limit=30
- License length cap ([:128]) was already applied in previous commit
Redesigns the create/edit/view skill modal into a two-column spec manifest
layout (Identity/Manifest/Deployment | Skill Content) matching the Agent
Skills spec structure. Installed (readonly) skills can now have their runtime
config (model, temperature, token limits, enabled) edited independently of
the locked spec/content fields.
- Two-column spec layout with section headings (Identity, Manifest, Deployment,
Skill Content); content textarea uses monospace font and fills the column
- h3 section headings for screen-reader nav; h3 UA stylesheet reset in CSS
- Runtime Config collapsible uses 3-column grid; license field is now a select
of SPDX identifiers (MIT, Apache-2.0, GPL-3.0, AGPL-3.0, etc.)
- Origin badge (cyan) shows source URL for installed skills in view mode
- server.py: _SKILL_RUNTIME_CONFIG_FIELDS frozenset; readonly skills filter
updates to config-only fields (spec fields silently dropped); audit action
distinguishes skill.update.config from skill.update; license field capped
at 128 chars in both create and update paths
- governance.js: spec fields disabled for readonly; config fields always
editable; Save button shown for all skills (labeled "Save Config" when
readonly); collapsible state reset between modal opens prevents state leak;
esk-allowed-tools disabled state driven by auto_approve not readonly
- Tests: spec-only body on readonly skill → 400; config-only → 200 with
spec fields unchanged; mixed body → config fields applied, spec dropped
* fix: output guard detects single secret-bearing env lines
The credential leak check required 3+ env-style lines before flagging.
A single AWS_SECRET_ACCESS_KEY=... line was missed. Now flags whenever
any env line has a secret-bearing key name (SECRET, KEY, TOKEN,
PASSWORD, CREDENTIAL), regardless of how many total env lines exist.
* fix: tighten env secret key matching, add tests
Tighten _RE_ENV_SECRET_KEY to word-boundary segments so MONKEY/TURKEY
don't false-positive. Use any() for short-circuit. Add test for single
secret line detection and substring false-positive prevention.
Add tool_error nudge type that fires when a tool returns an error,
prompting the model to search memories for prior feedback about the
tool or error pattern before retrying.
- Gated on nudges config (respects nudges=false)
- Only fires when memories exist (no noise on fresh workstreams)
- Broad error detection: Error*, *error:*, Command timed out, Unknown tool
- Nudge wording aligned to memory(action='search') convention
- Respects existing cooldown (5 min) and rate limiting
- 4 new tests
Readonly guard should prevent editing content, not uninstalling.
Remove readonly check from admin_delete_skill so batch-installed
skills can be individually deleted. Enable delete button in UI
for all skills regardless of readonly flag.
When a GitHub repo URL has no root SKILL.md (monorepo pattern like
anthropics/skills), automatically scan the repo tree for all SKILL.md
files and install every discovered skill in one operation.
- Add fetch_skills_from_github_repo() — scans recursive tree, parses
each SKILL.md, collects per-skill resources via shared helpers
- Extract _find_resource_files() and _fetch_resource_contents() to
eliminate duplication between single and batch fetch paths
- Extend admin_skill_install to fall back to batch scanning when
single-skill fetch returns 404
- Each skill gets a specific source_url pointing to its subdirectory
- Backward compatible: single-skill repos return same response shape
- Frontend handles both shapes with contextual toast messages
- Filter tree scan to URL path subtree when path is provided
- Cap at 50 skills per repo scan
Also addresses review feedback:
- Fix path prefix check (scripts/ not scriptsX/)
- Use count_skill_resources_bulk for single skill GET
- Add content field to SkillResourceInfo schema
- Fix OpenAPI spec paths ({path} not {path:path})
- Check r.ok on resource upload promises
- Preserve / in URL-encoded paths (split/map/join pattern)
- URL-encode path in Python SDK delete method
- Fix test_install_not_found to mock batch fallback
- Fix test_search_empty_results for required q param
- Narrow except clause to ValueError in batch parser
The heuristic engine was seeing empty {} for web_fetch, web_search,
watch, notify, task, and load_skill — only bash, file ops, and MCP
tools had their arguments forwarded. The judge could not pattern-match
on URLs, queries, commands, or messages for these tools.
* feat: load_skill built-in tool — model-driven skill discovery and activation
Two-action tool: 'search' finds skills by multi-word query with substring
matching on name/description/tags/category (auto-approved, read-only);
'load' activates a skill by name via set_skill() (requires approval).
Guards: filters disabled skills from search + load; short-circuits when
skill is already active; approval_label includes skill name for granular
tool policies (load_skill__<name>); main session only (excluded from
sub-agents). Logs storage errors in search path.
25 tests covering registration, preparer validation, executor logic,
disabled/already-active edge cases, multi-word queries, approval labels.
* refactor: use BM25 relevance ranking for load_skill search
Replace substring matching with BM25Index from turnstone/core/bm25.py,
matching the pattern used by memory relevance and tool search. Handles
multi-word queries, term frequency, and document length normalization.
* fix: address copilot review — BM25 tags parsing, primary_key, test cleanup
- Parse JSON tags into space-separated text before BM25 indexing so
individual tag terms match queries (was passing raw '["foo","bar"]')
- Add primary_key: "name" to load_skill.json for PRIMARY_KEY_MAP
- Remove dead resolve_workstream patch from test helper
- Update diagram: "substring match" → "BM25 ranking"
* feat: skill discovery — search and install skills from external sources
Add discovery UI and API for finding and installing skills from
skills.sh registries and GitHub repositories with one-click install,
SKILL.md frontmatter parsing, and security scan integration.
Core modules:
- skill_parser.py: ParsedSkill dataclass, parse_skill_md() with YAML
frontmatter support (Anthropic + Hermes tag formats), name validation
- skill_sources.py: SkillsShClient (async search + resolve),
fetch_skill_from_github (SKILL.md + bundled resource fetching with
256KB cap, text extension filter, GitHub API tree traversal)
API:
- GET /v1/api/admin/skills/discover — search with installed annotation
and scan_status for installed skills
- POST /v1/api/admin/skills/install — fetch, parse, duplicate check,
create with origin="source" readonly=true, store resources, audit
Also fixes pre-existing bug where _skill_to_response omitted scan_status,
scan_report, scan_version fields — scan tier badges in the installed
skills table were silently empty despite data existing in storage.
Admin UI: pill toggle (Installed/Discover), discovery cards with scan
tier badges, GitHub import modal with proper focus trap/Escape/backdrop,
scoped selectors preventing MCP↔Skills cross-tab state corruption.
SDK: discover_skills() + install_skill() on Python (async+sync) and
TypeScript console clients.
48 new tests across 3 test files. All 2632 tests pass.
* fix: address copilot review — 404 vs 502, O(n) lookups, branch fallback
- SkillNotFoundError subclass: install returns 404 when SKILL.md is
missing, 502 only for connectivity/upstream errors
- get_skill_by_source_url() + list_installed_skill_urls(): indexed
storage lookups replace O(n) full-table scans with content blobs
- Default branch fallback: tries main then master when URL doesn't
specify a branch
- Path normalization: strip trailing slash once, remove redundant
candidate
- SDK install_skill() returns typed SkillInfo with response_model
- Tree size guard: skip resource tree if response >2MB
* feat: output guard data pipeline — persist assessments, SSE events, admin UI
Complete the output guard pipeline: persist assessments for v2 calibration,
surface warnings in every UI layer, and add scan badges to admin skills tab.
Storage: migration 022 adds output_assessments table (flags, risk_level,
annotations, output_length, redacted — raw output never stored) and
scan_version column on prompt_templates. Three new protocol methods with
SQLite + PostgreSQL implementations.
Server/CLI: on_output_warning now persists assessments fire-and-forget.
CLI shows flags, annotations, and redaction notice. Session emits on_info
warning when high/critical scan_status skill is loaded.
MQ: OutputWarningEvent dataclass + bridge SSE forwarding.
Web UI: output_warning SSE handler with inline warning rendering
(role="alert" for accessibility), semantic risk colors.
Console admin: scan badges on skills list (dedicated scope-scan-* CSS with
green/yellow/red risk vocabulary), scan report breakdown in edit modal with
4-axis scores, POST /admin/skills/{id}/rescan endpoint, GET
/admin/output-assessments endpoint with date-filtered pagination.
Security fixes: ReDoS in connection string regex ([^@\s]+ → [^:@\s]+),
negative limit bypass in all admin endpoints (max(1, ...)), to_dict()
excludes sanitized output by default.
False-positive fixes: credentials pattern anchored to path context,
env secret key check restricted to key portion only.
* fix: address PR #110 review — list redaction, test fixture, OpenAPI snapshot
Per-part redaction: evaluate each text part independently in structured
output instead of joining all parts and replacing only the first one.
Fix test annotations default from "{}" to "[]" matching schema.
Regenerate TypeScript OpenAPI snapshots for new admin endpoints.
Add turnstone/core/skill_scanner.py — a production content scanner
that evaluates skill risk across four axes:
1. Content risk: command execution, external downloads, credential
handling, data exfiltration, eval/exec, sudo, browser automation
2. Supply chain risk: pipe-to-shell, transitive installs, obfuscation,
download-exec chains, executable URLs from untrusted domains
3. Vulnerability risk: prompt injection (E004), insecure credential
handling (W007), third-party content exposure (W011)
4. Declared capability risk: parsed from allowed_tools field —
Bash(*) is high, Bash(git:*) is low, read-only tools are safe
Composite score with equal 25% weights per axis. Floor rule: any
single axis at critical forces composite to at least medium tier.
Wired into both SQLite and PostgreSQL storage backends:
- scan_skill() runs at create_prompt_template time
- Re-scan triggers on update when content or allowed_tools change
- Results populate the existing scan_status and scan_report columns
- Silent failure on scanner errors (never blocks skill creation)
Scanner helper factored into _utils.py (shared across backends).
23 unit tests covering tier classification, capability scoring,
negation filtering, floor rule, serialization, and trusted domains.
* feat(judge): enrich heuristic rules from 23 to 36
Add 13 new pattern-based rules to the intent validation heuristic,
calibrated from analysis of 25K public agent skill security audits
across three independent auditors.
New critical: download-then-execute chains.
New high: browser+data export, transitive installs from untrusted
sources, control plane mutations (crontab, systemctl).
New medium: content ingestion pipelines (curl|python3), interpreter
execution (python3 script.py), cloud CLI mutations (az/gcloud/aws/
kubectl/terraform create/delete/destroy).
New low: tool_search, read_resource, web_search.
Fixes: crontab -l no longer false-positives, systemctl stop/disable
now flagged, az/gcloud subcommand patterns work correctly.
* fix(judge): address PR #107 review feedback
- content-ingestion: narrow second pattern to specific interpreters/
processors (python3, node, ruby, perl, php, jq) instead of any word.
Prevents false positives on read-only downstream (wget -O - | head).
- cloud-infra-mutation: split kubectl into its own pattern with specific
verbs (apply, create, delete, scale, rollout, drain, cordon) to avoid
false positive on resource types (kubectl get deploy).
- cloud-infra-mutation: split terraform/pulumi to specific verbs only
(apply, destroy, import) — terraform plan no longer matches.
- control-plane-mutation: exclude -h and -V flags from crontab pattern
alongside existing -l exclusion.
- Add 35 heuristic rule tests covering all 13 new rules with positive
matches and negative (false-positive prevention) cases.
* feat: unified skills system — merge prompt templates + workstream templates
Evolves prompt_templates into a first-class skills entity and merges
workstream templates into the same model, collapsing two concepts into
one.
Migration 021: 21 new columns on prompt_templates (skills metadata,
security scan fields, session config from WS templates), skill_resources
table for bundled files, skill_versions table for auto-snapshot version
history. Data migration converts existing WS templates into skills with
name collision handling, migrates version history, renames workstreams
and scheduled_tasks columns, cleans orphaned permissions, drops old
tables.
Key changes:
- All public interfaces renamed: templates → skills (API, CLI, SDK, UI)
- Session config (model, temperature, token_budget, auto_approve, etc.)
now lives on the skill and is applied at workstream creation
- /skill slash command, set_skill() API, --skill CLI flag
- BM25 skill search via SkillSearchManager for activation="search" skills
- Admin UI: Skills tab with collapsible Session Config section,
description subtitles, activation/origin/MCP badges, pagination
- Shared validation helper (_parse_skill_session_config) for DRY CRUD
- Version history with auto-snapshot on every edit + API endpoint
- Cascade delete (resources + versions) on skill removal
- Security: range validation, activation allowlist, fail-closed enabled
check, duplicate name 409, readonly guard, JSON validation
- 77 new tests across storage, runtime, search, API integration, and
migration behavior verification (2521 total)
* fix: address Copilot review + rename admin.templates → admin.skills
- Skip skill lookup when resume_ws is set (avoids spurious 400)
- Fix _applied_skill_version mismatch (1 in both workstreams table and session)
- Remove stale template field from MQ protocol diagram
- Rename admin.templates permission to admin.skills everywhere (runtime,
frontend, tests, docs) with migration step for persisted role data
- Fix stale /api/templates references in docs and diagrams
- Update docstrings/comments for skills terminology
* fix: address Copilot round 2 — skill version lineage + stale doc refs
- Compute actual skill version from skill_versions count (not hardcoded 1)
- Use same version in both workstreams table and session metadata
- Fix response payload example: "templates" → "skills" key
- Fix "Each template summary" → "Each skill summary"
stack.aclose() on a stuck streamable-http transport hangs indefinitely,
causing 50% CPU on all nodes when removing a broken remote server via
reconcile_sync. Wrap with asyncio.wait_for(timeout=10s) so cleanup
proceeds even if the transport refuses to close cleanly.
Review fixes:
- Rename query param from `q` to `search` across endpoint, frontend,
SDKs, OpenAPI spec, docs, and tests to match upstream registry API
- Validate variables/env/headers are dicts in install endpoint (400 on
malformed input instead of 500)
- Block javascript: and unsafe URL schemes on repo and website links
rendered from registry data (XSS prevention)
- Add roving tabindex to Servers/Registry pill toggle for correct
keyboard focus behavior
- Add noreferrer to website link in detail modal
Sync-pending indicator:
- "Sync to Nodes" button pulses yellow after create/edit/delete/import
to alert admin that nodes have unseen changes
- Clears after successful sync
- Reduced-motion safe
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.
Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.
Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.
71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
uv lock --check fails after version bump because the lockfile is stale.
pip-audit --strict fails because turnstone 0.7.0 isn't on PyPI yet.
Fix: regenerate uv.lock, and audit only third-party deps via
uv export --no-emit-project piped to pip-audit -r.
* fix: surface MCP server errors in admin UI instead of silent logging
get_server_status() hardcoded error="" — connection and refresh failures
were logged but never surfaced to the admin panel.
Added _last_error dict to MCPClientManager: set on failure (connect,
refresh, periodic refresh, notification handler), cleared on success,
cleaned up on remove. Read in get_server_status().
Admin UI: error tooltip on list row status span, error text in red
in detail modal per-node list. Schema already had the field.
6 new tests for error tracking lifecycle.
* feat: add turnstone_mcp_server_errors Prometheus gauge
Exposes the count of MCP servers currently in error state via
/metrics for alerting and reliability tracking.
* fix: address copilot review — sanitize error strings, clear on notification success
- Add _set_error() helper: strips newlines, truncates to 256 chars
- All error-setting sites now use _set_error() for consistent sanitization
- Notification handler clears _last_error on successful refresh (fixes
stale error for push-notification servers that skip _periodic_refresh)
TestClient-based integration tests for the 4 OIDC HTTP endpoints: authorize, callback, admin list identities, admin delete identity.
Uses real SQLite storage with mocked external OIDC calls (exchange_code, validate_id_token, provision_oidc_user) to exercise the full handler→module→storage contract. Covers happy paths, error flows, rate limiting, JWKS key rotation retry, and state expiration.
Trivy scan fails on HIGH for libc-bin/libc6 (2.41-12+deb13u1).
The fix (2.41-12+deb13u2) is available in Debian repos but the
base python:3.14-slim image hasn't been rebuilt yet. Adding
apt-get upgrade pulls in all pending security patches at build time.
Replaces postgres:18-alpine with pgautoupgrade/pgautoupgrade:18-alpine
in compose.yaml. Sets PGDATA=/var/lib/postgresql/data so pgautoupgrade
detects existing pg17 data and runs pg_upgrade automatically on first
start. No manual migration needed.
Also increases healthcheck start_period to 30s to accommodate the
one-time upgrade process.
* feat: per-tool "Always" approve instead of blanket auto-approve
Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.
Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.
UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.
Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.
16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.
* fix: address copilot review — filter errored items, hide Always on budget-only
- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
Eliminate dual accumulation by piggybacking assistant response text on
the server's ws_state:idle SSE event. The bridge no longer maintains
its own _ws_content_buffer — it reads content directly from the idle
event and passes it through to TurnCompleteEvent unchanged.
Server-side: WebUI accumulates tokens in on_content_token(), joins and
includes in the idle broadcast, then resets (with 256 KB cap).
Downstream consumers (Discord bidi DM forwarding, catch-up) are
unaffected — TurnCompleteEvent.content is still populated.
* fix: validate scope_id requires scope in memory API
Prevent misleading scope_id usage: reject scope_id with global scope,
require scope when scope_id is provided, require scope_id for
workstream/user scopes on writes. Belt-and-suspenders guard in storage
backends ignores scope_id when scope is empty.
* fix: strip whitespace in scope validation, relax user scope_id requirement
Address Copilot review: .strip() whitespace-only values in all three
validation helpers; SaveMemoryRequest no longer requires scope_id for
user scope since the server auto-resolves it from auth context.
* fix: inject prompt template guardrails into plan agent system message
Safety/behavioral templates were silently bypassed by the plan agent,
which only used _PLAN_IDENTITY. Now _plan_system_content() prepends
_template_content (when present) so admin-configured guardrails apply
to both _exec_plan and _refine_plan, matching the task agent pattern.
* fix: address Copilot review — log truncation, comment clarity, test robustness
- Log warning on template truncation in _plan_system_content() for
consistency with _init_system_messages()
- Clarify comment that prior plan pairs (not general history) are forwarded
- Use ChatSession._PLAN_IDENTITY for index assertions instead of substring
* fix: reorder new-workstream modal so Task is the primary field
Users were typing their prompt into the Name field (first text input,
auto-focused) and leaving Task empty, creating idle workstreams. Move
Task textarea to the top of the form, auto-focus it, and add
Ctrl/Cmd+Enter submit shortcut. Accessibility fixes: cancel button
focus-visible, label-hint contrast raised to WCAG AA, platform-aware
keyboard hint, Ctrl+Enter added to shortcuts overlay.
* fix: Enter on Cancel button no longer triggers submit
Copilot review caught that pressing Enter while focused on the Cancel
button bypassed native click and called submitNewWs(). Skip the
Enter-to-submit handler for BUTTON elements so native activation fires.
Also make keyboard shortcuts overlay platform-aware (Ctrl vs ⌘).
Every append site immediately drains via _init_system_messages(), so this
is defensive — ensures multiple nudges survive if the drain flow is ever
refactored to batch calls.
* perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload with asyncio.gather
Both functions queried cluster nodes sequentially, making latency
O(N × timeout). Use asyncio.gather to query all nodes concurrently,
matching the existing admin_list_watches pattern. Also reuse the
shared proxy_client instead of creating throwaway httpx clients per
node, and add debug logging on MCP status fetch failures.
* perf: bound node fan-out concurrency and improve debug logging
Add _NODE_FAN_OUT_LIMIT (50) semaphore to all three gather fan-out
sites (_collect_mcp_status, _notify_nodes_mcp_reload, admin_list_watches)
to cap concurrent outbound connections below the httpx pool limit,
leaving headroom for other proxy traffic at 1000-node scale.
Add exc_info=True to all debug log calls for actionable diagnostics.
* test: add unit tests for _collect_mcp_status and _notify_nodes_mcp_reload
11 tests covering success, non-200, missing URL, exceptions, empty
cluster, and mixed multi-node scenarios for both fan-out helpers.
* fix: reduce metacognition false positives with strong/weak pattern tiers
Correction detection: split "no" handling — "no," and "no." are strong
(always fire), "no <word>" uses an allowlist of correction-context words
(pronouns, demonstratives, verbs) instead of a blocklist. Phrases like
"no problem", "no worries", "no rush" are excluded automatically.
Completion detection: move most patterns to weak tier, gated by message
length (<80 chars) and absence of continuation markers ("?", "can you",
"but", "now", "please", etc.). "thanks for X" excluded at regex level.
Strong tier (always fire): "that's all", "lgtm".
* fix: align allowlist comment with implementation (include articles)
* fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var
OIDC redirect_uri was derived from the request Host header, which is
unreliable behind reverse proxies. Add TURNSTONE_OIDC_REDIRECT_BASE
(env var / config.toml) to pin the externally-reachable origin.
Extract _build_oidc_redirect_uri() helper to deduplicate the authorize
and callback handlers. Validate redirect_base at load time (must be
scheme://host[:port], rejects paths/query strings/invalid schemes).
* fix(oidc): reject redirect_base with missing hostname
Addresses Copilot review: values like `https://` or `https://:443`
passed validation but would produce invalid redirect URIs.
* fix(oidc): reject redirect_base with userinfo or invalid port
Addresses Copilot round 2: urlparse silently accepts user:pass@host
and non-numeric ports. Now explicitly rejects both.
* test: add scope coverage for internal MCP/config reload endpoints
Verify required_scope() returns "approve" for _internal endpoints
across all access patterns (bare, /v1/-prefixed, console proxy with
and without /v1/), plus a GET negative test confirming only POST is
elevated. Closes the "internal endpoints accept read scope" item in
PROGRESS.md — the endpoints were already in APPROVE_PATHS.
* test: add config-reload v1/proxy scope tests per review feedback
Add /v1/-prefixed and console proxy variants for config-reload to
match the mcp-reload coverage, as flagged by Copilot review.
Expandable user rows in the console Users tab reveal OIDC identities
linked to each user. Issuer badge, truncated subject, email, relative
last-login time, and unlink action with confirmation modal + audit trail.
Keyboard accessible (tabindex, Enter/Space, aria-expanded, focus-visible).
In-place refresh after unlink (no close/reopen flicker). Audit captures
user_id before delete. Mobile responsive (3-column at <700px).
Reduced-motion support. 2 new admin API endpoints reusing admin.users
permission and existing storage methods.
* fix: restore safe HTML element rendering and suppress plantuml warning
- Add safe HTML tag allowlist in inlineMarkdown: br, hr, kbd, mark,
sub, sup, ins, wbr, details, summary, abbr, small, u, s
(attribute-free only — XSS safe, tags with attributes stay escaped)
- Add <details>/<summary> block-level protection pass with recursive
markdown rendering of inner content
- Add plantuml to _NO_HIGHLIGHT_LANGS (suppresses highlight.js warning
for unsupported language)
- CSS for details (collapsible, overflow hidden), kbd (mono font,
key style), mark (yellow-glow token for theme adaptation)
* fix: restrict safe tags to inline-only, broaden details regex
- Remove hr, details, summary from inline _SAFE_TAGS allowlist (they
are block-level and produce invalid HTML inside <p> wrappers)
- Make <details> regex newline-optional so same-line
<details><summary>Title</summary> patterns are captured
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal
Close test coverage gaps for prompt templates:
- Resume with deleted template: verifies graceful degradation (template_content=None, warning logged)
- Threading safety: concurrent set_template/init_system_messages with no race conditions
- Factory passthrough: template kwarg propagation through WorkstreamManager.create()
Add read-only template listing endpoints (read scope, no content exposed):
- GET /v1/api/templates — prompt template summaries (name, category, is_default, origin)
- GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model)
- Available on both server and console; Python + TypeScript SDK methods added
- Console creation modal switched from admin endpoint to read-scope endpoint
Eliminate double-load inefficiency in workstream creation:
- Template validation moved before mgr.create() (no create-then-rollback on invalid template)
- template kwarg plumbed through WorkstreamManager.create() and session factory
- _SessionFactory Protocol added for proper mypy typing
Add workstream creation modal to server web UI:
- Name, model, template dropdown, ws_template/profile dropdown
- Instrument panel aesthetic: gradient top border, blur backdrop, amber accent
- Focus trap, Escape/Enter keyboard handling, loading state, error display
- WCAG AA contrast compliance, reduced-motion support
* fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots
Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates()
to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint.
Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types.
Regenerate openapi-server.json and openapi-console.json snapshots.
Addresses Copilot review feedback on PR #67.
* fix: skip template pre-validation when resuming a workstream
When resume_ws is set, the request's template field is irrelevant —
resume() restores the template from workstream_config. Pre-validating
a stale template name would incorrectly return 400 before the resume
even runs.
Addresses Copilot review feedback on PR #67.
* feat: mermaid diagram rendering with lazy loading and theme integration
Integrate mermaid.js 11.13.0 (self-hosted, MIT, ~2.9MB) for rendering
```mermaid code blocks as inline SVG diagrams. Covers flowcharts,
sequence, class, ER, state, gantt, pie, timeline, and mindmap.
- Lazy-loaded via dynamic script injection on first mermaid block
detection (not eagerly loaded on every page view)
- 3-state loader (idle/loading/ready) with callback queue
- Serialized rendering to avoid mermaid internal state corruption
- Theme integration via getComputedStyle reading CSS design tokens;
re-renders all diagrams on dark/light theme toggle
- Source preserved in data-mermaid-source for theme re-rendering
- Error handling with source code fallback display
- securityLevel: "strict" (DOMPurify) for SVG XSS prevention
- THIRD-PARTY-NOTICES updated with mermaid MIT license
* fix: mermaid render fixes from Copilot review
- Call result.bindFunctions(container) after SVG insertion for
interactive diagram elements (click handlers, links, tooltips)
- Clear mermaid-error class on successful render (fixes stale error
styling after theme toggle re-render)
- Clear mermaid-error in reRenderAllMermaid before re-render sequence
- Restructure postRenderMarkdown so mermaid rendering runs even when
highlight.js is unavailable (hljs guard changed from early return
to conditional block)
- Regex changed from (\w*) to ([^\s`]*) to capture language names with
special chars (c++, c#, objective-c, shell-session)
- Alias map normalizes c++ → cpp, c# → csharp, f# → fsharp for CSS
class names
- Empty language no longer emits class="language-", preventing
highlight.js auto-detect across all 37 bundled languages on
unlabeled code blocks (performance fix for large blocks)
* feat: GFM extended syntax renderers (callouts, footnotes, definition lists)
Add three GFM extended syntax features to the server web UI markdown
renderer, with no external library dependencies (pure JS/CSS):
- Callouts/Alerts: > [!NOTE], [!TIP], [!IMPORTANT], [!WARNING], [!CAUTION]
with color-coded left borders, icons, and recursive markdown body
- Definition Lists: Term + `: Definition` pattern with multi-term support
- Footnotes: [^id] inline superscript references, [^id]: definitions
collected into a numbered section with bidirectional navigation
Design review fixes: scoped footnote IDs (prevent collisions across
messages), aria-hidden on callout icons, aria-label on callout containers,
focus-visible on footnote links, smooth-scroll footnote navigation.
* fix: use getElementById for footnote scroll to handle special chars in IDs
querySelector throws on fragment IDs containing &, . or : characters
(produced by escapeHtml on footnote labels). getElementById accepts any
string and is the correct API for ID-based element lookup.
* feat: rich markdown renderer with LaTeX support for server web UI
Extract markdown rendering from app.js into dedicated renderer.js with
full GFM support: tables (alignment, hover, striping), nested lists,
task list checkboxes, nested blockquotes, images (click-to-load for
privacy), and inline/display LaTeX math via self-hosted KaTeX 0.16.38.
Security: escape image/link URLs to prevent attribute injection, block
javascript: scheme in links, add rel="noopener noreferrer", images
require explicit click to load (no automatic external requests).
Accessibility: scope="col" on table headers, tabindex on scrollable
table containers, aria-labels on task checkboxes and image placeholders,
KaTeX error color override for WCAG AA contrast, reduced-motion support.
* fix: address code review — XSS hardening and list type splitting
- Escape all text through escapeHtml() at start of inlineMarkdown()
so only renderer-generated tags appear in innerHTML (prevents raw
HTML/script injection from LLM output)
- Replace inline onclick handler on image placeholders with data-*
attributes and delegated DOM event listeners (prevents entity
decoding XSS in event handler attributes)
- Split list blocks into separate <ul>/<ol> when marker type changes
at the same indent level (mixed ordered/unordered sequences)
* feat: Discord content catch-up + bidirectional notification replies (#64)
Two improvements to the Discord channel adapter:
1. Fix intermittent dropped responses caused by a race between the
bridge's two independent SSE connections (global SSE detects idle
before per-ws SSE delivers all content tokens). The bridge now
accumulates content in _ws_content_buffer and attaches it to
TurnCompleteEvent.content. The Discord bot uses this as a catch-up
when streaming events were missed.
2. Bidirectional notification replies — when the notify tool sends a DM,
the message is tracked with the originating ws_id. Users can reply to
the DM and the reply is routed to the workstream. The response is
forwarded back to the DM, with the response itself tracked for
multi-turn conversations. Includes user identity verification,
stale notification feedback, and FIFO-capped tracking (100 entries).
* fix: address Copilot review — re-insert on unlinked user, deque buffer
- Re-insert _notify_ws_map entry when resolve_user returns None so the
user can retry after linking (same pattern as user-mismatch re-insert)
- Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len()
returns characters, not bytes)
- Use deque + running total for O(1) popleft instead of list.pop(0)
Migrations 011-016 each appended a permission to the builtin-admin role
via conditional UPDATE, but on some deployments these never applied.
Migration 017 idempotently sets the complete permission string rather
than appending incrementally.
Must be merged after feat/admin-mcp-servers (migration 016).
* feat: admin Settings tab — form-based editor replacing "coming soon" stub
Section-grouped layout with collapsible headers for all ~40 ConfigStore
settings (model, session, tools, server, mcp, ratelimit, health, judge,
memory). Type-appropriate inputs: CSS toggle for bools, number with
min/max/step, select for choices, text for strings. Secret fields shown
read-only. Source badge (storage/default), amber restart indicator.
Inline save per field with dirty detection, row flash on success, reset
to default via styled confirm modal. Full WCAG keyboard accessibility
(Enter/Space on section headers, aria-labels, focus-visible). Mobile
responsive single-column at <700px. Reduced-motion safe.
* fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes
Settings UI:
- Help tooltips: ? button on ~25 settings with plain-English explanations
and optional reference links (arXiv, Fowler, MCP spec). Click to toggle
popover, Escape to dismiss, aria-expanded for accessibility.
- Sections start collapsed for scannable overview.
- Restart badge: hidden by default, shows when dirty, persists after save
with amber glow. Positioned left of source badge.
- Secret row alignment fixed (transparent border matches input box model).
- Docs link in toolbar → Swagger UI Settings section.
- Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard,
numeric dirty detection (0.1 vs 0.10 no longer false positive).
- Secret reset button enabled when source=storage (clear legacy overrides).
- Space key repeat guard on section headers.
- Sidebar: sticky + max-height:100vh, no longer stretches with content.
Backend:
- context_window default changed from 131072 to 0 (auto-detect). Fallback
lowered from 131K to 32K (realistic for local models when detection fails).
Session normalizes 0→32768 defensively.
- Settings registry: help + reference_url fields on SettingDef, richer
descriptions for model/session/tools/judge/memory settings.
- Schema API includes help + reference_url.
- Bootstrap system prompt: added Runtime Settings section.
Docs: tab counts updated to 13 across README, architecture, console, governance.
* feat: database-backed settings (ConfigStore) with admin API
Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore. ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API. CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).
Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides. ON CONFLICT upsert in both
SQLite and PostgreSQL. admin.settings permission granted to
builtin-admin role.
Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.
ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init. Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.
Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.
warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.
Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default). Audit trail on mutations.
MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).
Python + TypeScript SDK methods. 63 new tests. Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.
* fix: address PR review — config-reload scope, registry defaults, doc alignment
- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
* feat: [memory] admin panel Memories tab — browse, search, inspect, delete
Add 13th admin tab in the Observe group for cluster-wide memory
management. List view with type/scope filter dropdowns and debounced
search input. Detail modal shows full metadata grid and scrollable
content block. Delete from both list row and detail modal with
confirmation and audit trail.
Permission-gated behind admin.memories. Escape key, backdrop click,
and focus trap wired for the detail modal. Mobile responsive: hides
description and updated columns below 700px.
* fix: memory detail modal — focus, delete safety, CSS shorthand order
Address Copilot review feedback: move focus to close button on modal
open for keyboard accessibility, disable delete button and clear stale
handler during loading/error states to prevent wrong-memory deletion,
and fix font shorthand/font-size ordering in toolbar filter styles.
* feat: [memory] REST API endpoints + SDK methods + docs
Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope
Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit
Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.
Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.
TypeScript SDK: matching methods + types on both clients.
Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.
Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.
Also fixes stray `total: int` on CreateChannelUserRequest.
* fix: [memory] address PR review — cross-user scope, schema types, snapshots
Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity. Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.
Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.
Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.
6 new security tests for user-scope access control.
* feat: MCP cluster-ops example — reference MCP server + SDK implementation
Standalone MCP server under examples/mcp-cluster-ops/ that exposes
tools for executing commands across a Turnstone cluster via the MQ
client SDK. Serves as a reference implementation for both MCP server
patterns (FastMCP, lifespan, tool handlers) and TurnstoneClient usage.
4 tools: list_nodes, run_on_node, run_on_nodes, run_on_all_nodes.
Parallel dispatch via asyncio.gather, raw ToolResultEvent output
capture, UTF-8 safe truncation, input validation, concurrency caps.
35 tests, ruff clean, mypy --strict clean.
* fix: address review feedback on MCP cluster-ops example
- Remove REDIS_SSL support (RedisBroker doesn't accept ssl kwarg)
- Move max-nodes check from _dispatch_parallel into tool handlers
for consistent error shape (always returns {"error": ...} object)
- Propagate KeyboardInterrupt/SystemExit from asyncio.gather instead
of swallowing them as per-node failures
- Fix _truncate omitted bytes count to reflect actual bytes dropped
after multi-byte boundary adjustment
- Apply strip/dedup to node IDs in run_on_all_nodes (matching
run_on_nodes behavior)
- Add __name__ guard to __main__.py
- Fix misleading UTF-8 byte count comment in tests
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting
Replace flat key-value memories table with structured_memories (migration 014).
Four memory types (user/project/feedback/reference), three scopes
(global/workstream/user). Consolidate remember/recall/forget into two tools:
memory (action-based: save/search/delete/list) and recall (conversation
history only).
BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5
memories for system message injection based on conversation context.
Metacognitive prompting injects ephemeral nudges after corrections, tool
denials, workstream resume, and completion signals.
Scope isolation enforced: system message injection and nudge counts filtered
to visible memories only (global + current workstream + authenticated user).
User scope requires authentication. Content capped at 32KB. ILIKE/LIKE
metacharacters escaped in both backends.
113 new tests (2053 total).
* fix: CI failure + copilot review feedback
- Fix time.monotonic() cooldown: use None sentinel instead of 0.0
default (monotonic clock starts at boot, not epoch — fresh CI
runners have uptime < 300s so cooldown check always triggered)
- Catch sa.exc.IntegrityError specifically in upsert instead of
broad Exception (copilot review)
- Preserve existing description/type on upsert when caller doesn't
explicitly set them (copilot review)
- Add last_accessed + access_count columns to schema/migration for
future LRU/LFU eviction support
* feat: admin panel — right-aligned sidebar navigation with two-column modals
Replace the horizontal tab bar (11 tabs, overflowing on standard monitors)
with a grouped sidebar on the right side, matching the admin button's
position in the header for natural spatial flow.
Sidebar: 5 groups (Identity, Automation, Governance, Observe, System) with
12 nav items including new Settings stub. Always visible on desktop (180px),
off-canvas drawer on mobile (<700px) sliding from right with backdrop.
Admin button: toggle behavior (click again to return to overview), active
state with amber highlight + top accent line, aria-expanded management.
Breadcrumb: shows active tab ("Admin / Users", "Admin / Audit", etc).
Modals: WS Template and Schedule create/edit forms restructured into
two-column grid (820px) with "Identity"/"Model Config" and
"Schedule"/"Execution" column headings. All modals gain max-height: 85vh
+ overflow-y: auto safety net. Modal z-index bumped to 600 (above sidebar).
Also: "Tokens" renamed to "API Tokens", redundant "Server default"
placeholders removed from model config fields, view fade-in transition,
comprehensive ARIA (grouped sidebar, aria-hidden on mobile, focus return
on drawer close), reduced-motion support.
* fix: address Copilot review — aria-orientation, settings permission gate, inert sidebar
- Add aria-orientation="vertical" to sidebar tablist for assistive tech
- Gate Settings tab behind admin.users permission so empty-state logic
works correctly when user has no admin permissions
- Use inert attribute on mobile sidebar when closed to prevent keyboard
focus from reaching off-canvas controls
- Add resize listener to sync aria-hidden/inert when crossing the
700px mobile breakpoint
* fix: simplify conversation storage — atomic assistant rows with tool_calls JSON
Replace the denormalized storage model (separate rows for assistant
content, tool_call, tool_result) with atomic assistant rows carrying
tool_calls as a JSON column. Eliminates the 100-line heuristic
reconstruct_messages function and its cross-turn merge bug.
Schema: add tool_calls TEXT column to conversations (migration 013).
Migration backfills existing data — merges tool_call rows into their
parent assistant row as JSON, renames tool_result to tool, deletes
consumed tool_call rows.
Session save path: assistant content + tool_calls saved in one
save_message call before tool execution (crash resilient). Tool
results saved as role="tool".
Extract shared storage utilities to _utils.py: row_to_dict, mutable
field frozensets, reconstruct_messages. Both backends import from
_utils — PostgreSQL no longer depends on _sqlite.py.
Includes denied/blocked tool call badge fix on resume: _build_history
detects denied results and propagates flag to parent assistant entry.
Frontend uses flag for correct badge-denied rendering. Denied tools
visually muted. role="status" on badges for accessibility.
Net -45 lines. 8 new tests for reconstruction, all 1914 tests pass.
* fix: migration 013 uses parameterized deletes and ordered downgrade
- DELETE of consumed tool_call rows now uses parameterized batches
(chunks of 500) instead of string interpolation
- Downgrade rebuilds via temp table to preserve chronological id
ordering when re-inserting tool_call rows
* feat: intent validation v1 — advisory LLM judge for tool approvals (#50)
Two-tier evaluation pipeline for non-auto-approved tool calls:
- Heuristic tier (instant): 23 pattern-based rules across 4 severity
levels (critical/high/medium/low) with first-match-wins priority
- LLM judge tier (async): multi-turn evaluation with read_file/
list_directory tool access, security-hardened path blocking, forcing
message on final turn, four-stage JSON parsing with retry nudge
Progressive UI: heuristic verdict badge + judge spinner, LLM verdict
upgrade via intent_verdict SSE event, glow on action buttons. Verdict
persisted to intent_verdicts table for audit. Prometheus metrics for
verdict counts and LLM latency. Enabled by default (--no-judge to opt
out). 132 new tests (1938 total).
Integration: session, server/WebUI, CLI, MQ bridge, console admin API,
Discord channel adapter. Config via [judge] in config.toml or CLI flags.
* fix: address PR #50 Copilot review feedback
- Fix double JSON encoding of func_args in both heuristic and LLM
verdict persistence paths — use pre-serialized string from verdict
- Fix confidence 0.0 treated as falsy in channel verdict formatter
- Fix timestamp format inconsistency in storage backends (isoformat
vs strftime) — now uses strftime consistently
- Add on_intent_verdict to eval.py NullUI (mypy fix)
- Fix late verdict after approval resolved — store last decision and
apply immediately to late-arriving verdicts
- Add permission rollback to migration 012 downgrade
- Update docs to reflect judge enabled by default
- Document confidence_threshold as reserved for v2
* fix: judge per-call timeout and credential recon heuristic
- Wrap create_completion() in ThreadPoolExecutor with per-call timeout
to prevent indefinite hangs on slow local models. On timeout, replace
the executor so subsequent batch items don't queue behind lingering
API calls
- Add IntentJudge.shutdown() and wire into session.close() for cleanup
- Add credential-recon heuristic rule: /etc/passwd, /etc/shadow,
/etc/master.passwd access flagged as HIGH/review (reconnaissance
pattern even though the command itself is read-only)
- 3 new tests for credential file access patterns
* fix: denied/blocked tool calls show correct badge on resume
- _build_history() detects denied results ("Denied by user") and
blocked results ("Blocked") and propagates denied flag to parent
assistant entry for frontend consumption
- Frontend history replay uses denied flag for badge-denied class
instead of hardcoding badge-approved for all historical tool calls
- Denial feedback always prefixed with "Denied by user:" so content
detection works with custom user feedback
- Denied tools visually muted (opacity 0.55, muted tool name)
- role="status" on all approval badge elements (accessibility)
- Broadened "Blocked" prefix match (catches "Blocked by tool policy")
* feat: wire prompt templates into session startup with full creation-path support
Prompt templates (prompt_templates table) now have runtime effect:
- is_default=true templates auto-apply as system message content,
concatenated in name order before user instructions
- Per-workstream template selection via --template CLI flag, template
field on POST /v1/api/workstreams/new, console creation modal dropdown,
scheduled task config, and channel adapter config
- {{model}}, {{ws_id}}, {{node_id}} variable substitution via single-pass
regex (prevents cross-variable injection)
- /template slash command for runtime switching, persisted across resume
- set_template() public API on ChatSession
Security hardening:
- MCP sync resets is_default=False on content update (prevents compromised
server from injecting defaults)
- 32KB content cap on template create/update + defensive truncation
- Template existence validation returns 400 before workstream creation
- Single-pass regex eliminates cross-variable expansion
Template field plumbed through all creation paths: CLI, server API, MQ
protocol/bridge, console backend, scheduler dispatch, channel router,
MQ client. Migration 010 adds template column to scheduled_tasks.
Frontend: console workstream modal template dropdown, scheduler
create/edit template field, governance template UI variables auto-detected
from content (read-only display replaces editable input). Focus trap and
Enter-key accessibility fixes in workstream modal.
Docs: governance.md template runtime section, api-reference.md template
field, governance + MCP architecture diagrams updated.
Python + TypeScript SDKs, Pydantic schemas all updated. 29 new tests.
* fix: address PR #47 review feedback
- Defer template validation until after resume_ws — a bad template name
no longer 400s when resume would have ignored it anyway
- Add template field to OpenAPI JSON specs (openapi-server.json,
openapi-console.json) for SDK/docs consistency
- Validate template existence in schedule create and update endpoints —
reject unknown template names with 400 instead of allowing schedules
that would silently fail at dispatch time
Add ddgCluster profile extending the 10-node cluster with a DuckDuckGo
Search MCP sidecar. All cluster nodes connect via streamable-http and
gain duckduckgo_web_search + duckduckgo_fetch_content tools. No API
key required.
Key implementation details learned during testing:
- MCP SDK DNS rebinding protection must be disabled for Docker
internal networking (Host header uses container names)
- FastMCP server binds to 127.0.0.1 by default; must set
mcp.settings.host='0.0.0.0' for cross-container access
- DDG CLI lacks --host/--port flags; settings configured via Python
entry point that patches FastMCP.settings directly
- Safe search disabled by default
Also adds MCP_CONFIG env var support to all server commands (shell
conditional, no-op when empty) and moves default server/bridge to
production profile for cleaner profile separation.
* fix: MCP resource template URI expansion via prefix matching
Resource templates (RFC 6570 URI patterns like `db://tables/{table}/rows/{id}`)
were discovered from MCP servers but non-functional — `read_resource_sync()`
only accepted exact URIs from `_resource_map`, which excludes templates.
Add prefix-based fallback: extract the static prefix from each template
(everything before the first `{`), store a prefix→server mapping, and
fall back to longest-prefix matching when exact URI lookup fails. MCP
servers handle URI routing internally so we just need to route the
expanded URI to the correct server.
Also surface templates in the system message catalog and `/mcp` command
so the model knows they exist and can construct expanded URIs.
* fix: address PR #46 review feedback
- Template prefix collision now keeps more specific (longer) template
URI instead of blindly overriding
- Fix _match_template docstring to accurately describe startswith
matching on static prefixes (not full template matching)
- Add missing loop.close() in integration test finally block
- Rewrite test_template_longest_prefix_wins with genuinely different
prefix lengths to avoid brittle collision-order dependency
* feat: MCP resource and prompt discovery with read_resource tool
Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.
New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.
Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).
/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.
* feat: MCP prompt governance sync with origin tracking and readonly guards
Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".
Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).
Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.
Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.
* feat: use_prompt tool, prompt catalog, and PR review hardening
New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).
Prompt listener registered in session for catalog rebuild on changes.
Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
assign atomically) so background thread callbacks never see
partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references
Docs and diagrams updated for 18 built-in tools.
* feat: granular tool policies for MCP resources, prompts, and tools
Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")
URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.
TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.
* feat: MCP visibility in server and console UIs
Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.
Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).
New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).
* fix: McpStatus in health schema, count properties, catalog name fidelity
Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.
Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
/health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
HTML-escaped (model must use exact strings in use_prompt calls);
only descriptions are escaped
* fix: OpenAPI spec McpStatus + diagram approval column accuracy
Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.
Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).
* fix: channel bidirectional routing — emit TurnCompleteEvent on all idle transitions
Bridge previously only emitted TurnCompleteEvent for MQ-initiated turns
(those with a correlation_id in _active_sends). Server-UI-initiated turns
went idle without emitting TurnCompleteEvent, so the Discord bot's
StreamingMessage never finalized — content accumulated in the buffer and
collided with the next Discord-triggered response.
Now TurnCompleteEvent is emitted unconditionally on every idle transition.
correlation_id is empty for non-MQ turns; SDK client filters by
correlation_id so existing consumers are unaffected.
* fix: remove unused variable flagged by ruff
* fix: approval timeout UI state and content flush before tool calls
Two bug fixes:
1. Approval timeout now shows denied state in UI — resolve_approval()
emits an approval_resolved SSE event so the browser transitions
from pending to denied (red border + badge). Also fixes the cancel-
during-approval path. Frontend resolveInlineApproval() gains a
skipPost parameter to avoid redundant POST when server-initiated.
ApprovalResolvedEvent added to Python and TypeScript SDKs.
2. Content streaming flushes pending buffer before tool call deltas —
_stream_response() held up to 13 trailing chars in the pending
buffer (for <think> tag detection) when transitioning to tool calls.
Now flushed eagerly when tool_call_deltas arrive, before clearing
in_think so reasoning text is correctly categorized.
* fix: address Copilot review feedback on PR #42
Patch _execute_tools in stream flush test to prevent real bash execution,
simplify confusing nested comprehension, and update resolve_approval()
docstring to reflect cancel/timeout call paths.
* feat: robust plan quality gate, iterative refinement, and amend UX
Plan agent output from weak models often produced garbage (11-char plans
that echo the prompt). Two fixes:
1. Quality validation (_validate_plan) checks length, section structure,
echo detection, and refusal patterns. Fails trigger one automatic
retry with a coaching message injected into the agent's existing
conversation, preserving all prior exploration context.
2. Iterative feedback loop — user feedback at plan review re-runs the
plan agent via _refine_plan() instead of appending text to the tool
result. Up to 5 refinement rounds. The plan file path is always
included in the tool result so the outer model knows where it lives.
UI improvements:
- Web: Reject button dynamically becomes "Amend" (amber) when feedback
is typed. Key hint badges (Esc/Enter) on plan buttons. Main input
disabled during review. Light-theme contrast fix via --on-color var.
- CLI: Prompt shows all three actions (approve/amend/reject).
- Bridge: Race condition fix — clear pending entry before HTTP POST so
sequential plan reviews from the refinement loop aren't skipped.
15 new tests covering validation, retry, and refinement.
* fix: address PR 41 review feedback
- Escape key in plan dialog now mirrors the Amend button: if feedback is
typed, Esc sends the feedback (amend); if empty, Esc rejects. Previously
Esc always hard-coded "reject", discarding typed feedback.
- Coaching message for plan retry now says "should include at least two of"
instead of "MUST include these", matching the actual validation rule
(_MIN_PLAN_SECTIONS = 2).
* feat: render plan inline in chat after approval
After the plan review dialog closes, the plan content is now rendered
as a collapsible inline block in the chat stream — styled with a
status header (approved/rejected/amending), markdown-rendered body,
and feedback note when amending. Uses the same makeCollapsible pattern
as tool output blocks.
* fix: prevent plan approval hang when inline render fails
The authFetch call that unblocks the server must fire before the
cosmetic inline plan rendering. Previously _addInlinePlan ran first
and any JS error (e.g. from renderMarkdown) prevented the API call,
leaving the session thread blocked forever.
- Move authFetch before _addInlinePlan
- Wrap _addInlinePlan in try-catch
- Guard against empty content
- Only auto-collapse plans longer than 12 lines
* fix: address PR 41 review feedback (round 2)
- Max refinement rounds no longer implicitly approve: the loop now
shows the final plan for explicit approve/reject before proceeding.
Previously exhausting 5 rounds silently accepted the last revision.
- Plan inline block: correct aria-label from "Tool output" to
"Plan content" when makeCollapsible is applied.
- XSS concern (not applicable): renderMarkdown is used for all
assistant messages — plan content follows the same trust model.
- Test loop concern (acknowledged): refinement tests verify component
logic; full _execute_tools integration would require extensive
mocking for marginal coverage gain.
* feat: thinking spinner + inline plan hardening
* fix lint
Add `turnstone-bootstrap`, a new entry point that uses any LLM (OpenAI,
Anthropic, or local/vLLM) to conversationally walk users through
configuring a Turnstone deployment. Generates .env files, setup.sh
scripts, and optional docker-compose overrides.
- Fully interactive startup (zero CLI args) with provider/model selection
- Auto-detects available models on local OpenAI-compatible endpoints
- 7 tools: read_file, write_file, generate_secret, check_port,
validate_api_key, check_docker, finish
- Path traversal protection on file read/write
- Duplicate write detection (skips identical content)
- Bounded retry loop (3 attempts) on LLM errors
- Anthropic message conversion with consecutive-role merging
* feat: generation cancellation — stop button, cancel API, cooperative cancel
Add cooperative cancellation via threading.Event on ChatSession. The cancel
signal is set from outside the worker thread (HTTP handler, MQ bridge, or
Escape key) and checked at defined checkpoints: per streaming chunk, before
tool execution, inside bash commands, and at each sub-agent turn.
Core: GenerationCancelled(BaseException) exception, cancel()/_check_cancelled()
methods, partial content preservation in _stream_response, clean rollback in
send() with idle state emission (no re-raise).
Server: POST /v1/api/cancel endpoint, CancelledEvent SSE emission, worker
thread safety net.
Frontend: Stop button (■ Stop) with send/stop swap via setBusy(), Escape key
shortcut, cancelled event handler. Accessible: aria-label, focus-visible
override, light theme contrast, non-color differentiation.
MQ: CancelMessage inbound type, bridge _handle_cancel routed handler.
SDK: cancel() on Python async+sync clients, CancelledEvent in Python+TypeScript
event registries, isCancelledEvent type guard.
OpenAPI: CancelRequest schema + endpoint spec.
Docs: API reference, architecture, SDK docs updated. Diagrams: conversation
turn, tool pipeline, MQ protocol, workstream states, SDK architecture.
* fix: address PR #40 review feedback
- setBusy() now resets stopBtn.disabled so stop button is re-enabled on
next generation after a successful cancel
- Gate cancel side effects (resolve_approval, resolve_plan, cancelled SSE
event) on worker_thread.is_alive() to avoid spurious events when idle
- Add /v1/api/cancel endpoint and CancelRequest schema to TypeScript
openapi-server.json to keep it in sync with Python-generated spec
* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging
Add comprehensive governance layer for the admin console:
- RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer),
custom role CRUD, user-role assignment with privilege escalation prevention
- Tool policies with glob pattern matching, priority-ordered evaluation
(allow/deny/ask), enforced before auto-approve in WebUI.approve_tools()
- Prompt templates with variable substitution, categories, default flag
- Usage tracking: per-LLM-request token/tool metrics, aggregated queries
(group by day/model/user), automatic 90-day pruning via scheduler
- Audit logging: append-only event trail for all admin mutations,
filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For
aware IP extraction
- require_permission() enforced on all 35+ admin endpoints (users, tokens,
channels, schedules, watches, roles, orgs, policies, templates, usage, audit)
- Field allowlists on storage update methods prevent mass-assignment bugs
- Self-deletion guard on admin_delete_user, delete_user cascades user_roles
- _row_to_dict helper eliminates ~400 lines of fragile positional row mapping
- _audit_context helper deduplicates 18 instances of audit boilerplate
- Migration 008: 7 new tables, 3 builtin roles, org_id on users
- Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit)
with permission-gated visibility, 7 modal dialogs, full keyboard accessibility
- Python + TypeScript SDK methods for all governance endpoints
- 120+ new tests (1554 total)
* fix: address PR #39 review feedback
- Rebuild serialized items after policy evaluation so denied/allowed
verdicts are reflected in tool_info/approve_request SSE payloads
- Make `since` query param optional in usage OpenAPI spec (handler
already defaults to last 7 days)
- Add response_model=StatusResponse to DELETE role/policy/template
and POST/DELETE role assignment endpoints in OpenAPI spec
- Add missing org_id/created/updated fields to UserRoleInfo schema
- Add missing created field to AuditEventInfo schema
- Show "no permissions" empty state instead of loading inaccessible
tab when all admin tabs are permission-gated
- Fix "13 permissions" → "15 permissions" in architecture.md and
security.md
- Fix import sorting in test_audit.py and test_tool_policy.py
* fix: address PR #39 round 2 review feedback
- Clear stale permissions from sessionStorage on config-token login
(auth.js _storePermissions)
- Only trust X-Forwarded-For when behind a proxy that sets
X-Forwarded-Proto (conditional on is_secure_request trust model)
- Thread user_id from auth into WebUI.on_status for usage events
- Add created field to TS AuditEventInfo type
- Return typed Pydantic models from all SDK governance methods instead
of dict[str, Any] — both async and sync clients
- Validate group_by param against allowed enum in admin_usage handler
- Add deterministic secondary sort (event_id DESC) to
list_audit_events in both SQLite and PostgreSQL backends
Agent tool outputs are now truncated to 16k chars to prevent search
results (14M+ chars observed) from blowing past the model's context
limit. On context-exceeded API errors, the agent returns its last
content instead of crashing.
Plan agent: own identity only (no base system prompt needed).
Task agent: base system prompt merged with task identity into a single
system message (needs tool patterns for tool execution).
Neither agent receives conversation history.
Fixes Jinja template error on Qwen models that reject system messages
appearing after non-system messages.
* fix: per-workstream SSE fan-out — multiple consumers no longer steal each other's tokens
After 4d665a5 removed the single-consumer SSE lock, the shared
_event_queue let concurrent consumers (browser, bridge, console proxy)
race on Queue.get(), each receiving ~1/N of content tokens and producing
garbled streaming text.
Replace the single queue with per-client fan-out: each SSE connection
registers its own bounded queue (maxsize=500) on WebUI._listeners, and
_enqueue() copies every event to all registered queues. On eviction or
close, a ws_closed sentinel is injected so SSE generators exit promptly.
* fix: address CI failures and Copilot review feedback
- Handle ws_closed sentinel in events_sse generator (break on close)
- Guarantee sentinel delivery by evicting one item when queue is full
- Clear listeners list after injecting sentinels on cleanup
- Fix test_slow_consumer to fill only slow queue directly
- Fix ruff SIM117 (nested with), unused import, mypy unused-ignore
* ci: add GitHub Release creation on tag push
* refactor: rename plan tool to create_plan
Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns
* feat: eval harness improvements inspired by autoresearch patterns
Major enhancements to turnstone-eval:
- Per-test timeout (--test-timeout, default 300s) and suite timeout
(--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
models and providers (--optimizer-model, --observer-model, etc.)
with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)
* fix: address Copilot review feedback
- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
* feat: watch tool — periodic command polling within workstreams
Add a new `watch` tool that lets the model (or user) set up periodic
polling of a shell command. Results inject as synthetic user messages
that trigger LLM turns, enabling reactive workflows like PR monitoring,
CI/CD status tracking, and deployment health checks.
Key design:
- Single tool with create/list/cancel actions
- Python expression DSL for stop conditions (restricted eval)
- Server-owned WatchRunner daemon (DB-persisted, survives eviction + restart)
- Three dispatch paths: idle, busy, and evicted workstream restore
- REST API for console visibility (GET /v1/api/watches, POST cancel)
- Migration 007, 8 storage CRUD methods, 75 new tests (1383 total)
* fix: address Copilot review — condition errors, restore deadlock, docs
- Condition eval errors now deactivate the watch immediately instead
of silently looping until max_polls
- Restored (evicted) workstreams set auto_approve=True to prevent
approval deadlocks with no connected user
- Tool description clarifies first-poll baseline behavior for change
detection mode
- Diagram updated: DELETE → POST /v1/api/watches/{id}/cancel
The _sse_generation mechanism assumed one SSE consumer per workstream,
but the bridge also maintains an SSE connection to each workstream.
When a new client connected (browser, proxy, or test), it incremented
the generation counter, killing the bridge's connection. The bridge
reconnected, killing the new client's connection — creating a
mutual-kill cascade that closed every SSE connection after one ping
cycle (5s).
Fix: remove _sse_generation entirely. sse-starlette handles disconnect
detection via its own ASGI task. Also remove the redundant
request.is_disconnected() check which raced with sse-starlette's
disconnect listener in Starlette 0.52.
Root cause confirmed via raw socket test: the server was sending
a zero-length chunked terminator (0\r\n\r\n) at exactly 5s,
cleanly ending the HTTP response body.
* fix: recovered workstreams invisible in console UI
Bridge startup recovery (_recover_workstreams) re-registered workstream
ownership but never published WorkstreamCreatedEvent to the cluster
channel. The collector's poll loop would pick up the workstream in its
internal state, but _apply_poll never fanned out SSE events to connected
browsers. Combined, this made channel-resumed workstreams invisible in
the console while remaining accessible through the proxied node UI.
- Bridge: emit WorkstreamCreatedEvent for each recovered workstream
- Collector: diff poll results and fan out synthetic ws_created/ws_closed
events for workstream additions and removals
- Skip workstreams with empty IDs in poll processing
- Add 4 tests for poll-diff fanout behavior
- Update console data-flow diagram and architecture docs
* fix: address PR review — filter empty ws IDs, stable event ordering
- Filter empty-string keys from old_ids to avoid phantom ws_closed
events if a previous poll inserted a workstream under key "".
- Sort set diffs before iterating so ws_created/ws_closed fanout
order is deterministic across poll cycles.
* feat: add ClusterSnapshot for instant console UI state rebuild
The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.
Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).
Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.
SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.
* fix: address review feedback and SSE proxy reconnect bug
Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
both state and listener locks, eliminating the event gap between
snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
console schema models (ClusterNodeInfo, NodeDetailResponse,
ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
and ClusterOverviewResponse types instead of Record<string, unknown>.
SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
when no upstream data arrives, preventing the browser EventSource
from dropping idle connections. The raw byte passthrough refactor
(4d11078) removed the proxy's independent keepalive — this restores
it without reverting to EventSourceResponse.
* feat: add vision/image support to read_file tool
read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).
* fix: address PR review feedback
- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
* refactor: use raw streaming for SSE proxy to preserve event framing
- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service
* Refactor SSE proxy to raw byte passthrough
- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
events, and comments now flow through verbatim. Added per-request timeout
override (read=None, pool=None) for long-lived SSE streams.
- tests/test_console.py: Add 3 new tests for SSE proxy:
- Ping and event preservation
- Upstream error status handling
- Client disconnect handling
- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
approach.
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI
When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:
- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool
New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.
Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.
43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.
* Fix Copilot review feedback on tool search
- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
Multiple containers starting simultaneously race on Alembic migrations
against shared PostgreSQL. Use pg_advisory_lock so they wait in line.
Also update SQLite bootstrap to detect post-migration databases.
* Normalize session_id into ws_id as sole persistent identity
Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).
Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id
Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days
Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.
* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec
- docs/channels.md: "resumes the session" → "resumes the workstream",
"Session resumed:" → "Resumed:", "old session was pruned" → "old
workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
models — removes all stale session_id properties from WorkstreamInfo,
DashboardWorkstream, CreateWorkstreamResponse schemas
* Add GPT-5.3, GPT-5.4, and pro model capabilities
Add capability entries for gpt-5-pro (272k output, high-only reasoning),
gpt-5.2-pro, gpt-5.3, gpt-5.4 (1.05M context), and gpt-5.4-pro.
* Validate reasoning_effort against model capabilities
_apply_model_params now falls back to caps.default_reasoning_effort when
the requested value is not in caps.reasoning_effort_values. Prevents
sending unsupported effort levels to models like gpt-5-pro (high only).
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
Two runtime bugs:
1. Channel gateway advertised http://127.0.0.1:8091 which is
unreachable from other Docker containers. Add
TURNSTONE_CHANNEL_ADVERTISE_URL env var override for Docker/K8s
environments, set to http://channel:8091 in compose.yaml, and
pass --http-host=0.0.0.0 so the gateway listens on all interfaces.
2. Console proxy service JWT had only "write" scope but the approval
endpoint requires "approve". Tool approval buttons in the server
web UI silently failed when accessed through the console proxy.
Changed proxy token scopes to read+write+approve.
Replace single-node Mermaid diagram with a 3-node cluster layout
showing bridge+server pairs per node, shared Redis MQ, console, and
channel gateway. Remove the verbose directory tree listing.
GitHub renders Mermaid natively as an interactive SVG. The new diagram
shows all client entry points (CLI, browser, SDK, Discord), the full
cluster topology including the channel gateway and notify path, and
the LLM provider layer.
* Add channel notification tool with security hardening
Implements the `notify` tool allowing the LLM to send notifications to
Discord channels/users via the channel gateway. Includes fixes for 11
review findings: JWT auth on the gateway endpoint, first-healthy gateway
delivery with retry+backoff, rate limiting only on success, SSRF URL
scheme validation, Discord mention sanitization, SQLite ON CONFLICT
upsert preserving created timestamps, advertise URL resolution for
0.0.0.0 bind, randomized service IDs, generic error messages to prevent
internal state leakage, and partial direct-target validation.
Service registry with heartbeat-based health filtering (migration 005).
Channel gateway registers on startup, heartbeats every 30s, deregisters
on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint
auth (static + JWT), storage CRUD, and retry behavior.
* Add notify documentation, diagrams, and review fixes
Documentation:
- New sequence diagram 17-notify-flow.puml showing end-to-end delivery
- Updated 16-channel-architecture.puml with services table, notify HTTP
path, and Notification Flow note
- channels.md: Notifications section (targeting, delivery flow, service
registry, security) and new config table entries
- tools.md: notify tool reference, updated counts/tables (14→15 tools)
- security.md: channel gateway row in service-to-service auth table
- architecture.md: notification subsystem paragraph
Review fixes (copilot):
- _http.py: fail closed when auth unconfigured (401 instead of pass-
through), strip whitespace on message/title, generic error messages
for user-not-found vs no-linked-channels
- session.py: parse gateway response JSON and require at least one
result with status=="sent" before counting as success
- _postgresql.py: use index_elements instead of constraint for upsert
* Add scheduled task system with cron/at scheduling, admin API, and console UI
Console-integrated background scheduler dispatches workstreams on recurring
cron expressions or one-shot ISO8601 timestamps. Four target modes: auto
(best node by headroom), pool (shared queue), all (fan-out), or specific
node. Redis distributed lock with unique owner + Lua conditional release
prevents duplicate dispatch in multi-console deployments.
Storage: scheduled_tasks + scheduled_task_runs tables (migration 004),
9 protocol methods on both SQLite and PostgreSQL backends, field allowlist
on updates, run history auto-pruned at 90 days.
API: 6 CRUD endpoints under /v1/api/admin/schedules with croniter
validation, ISO8601 future-time checks, field length bounds, schedule
count cap (200), and OpenAPI spec entries with Pydantic models.
UI: Schedules tab in admin panel with create/edit/delete modals, run
history modal, cron/at type toggle, target mode select, status dots for
accessibility, responsive grid, keyboard navigation, and focus management.
Security: fan-out capped at 20 nodes/task/tick, auto-approve dispatches
logged at WARNING with created_by attribution, user_id propagated in
CreateWorkstreamMessage for audit trail.
46 tests across storage, scheduler engine, and API endpoints.
* Add croniter to test extras for CI compatibility
CI installs [test] extras but not [console], so croniter was missing
when schedule API tests import console/server.py validation functions.
* Address Copilot review: timezone validation, focus trap, enabled flag
- Reject naive at_time timestamps — require timezone offset (e.g. +00:00 or Z)
- UI appends +00:00 to datetime-local values for explicit UTC
- Fix datetime-local normalization: check length before appending seconds
- Add textarea to modal focus trap selector (prevents focus escape)
- Fix _normalize_task_dict not called in update response
- Persist enabled=false on create (storage defaults to enabled=1)
- Validate at_time is still in future when re-enabling a one-shot task
- broker._redis coupling acknowledged as tracked tech debt
* Add JWT auth security hardening (6 fixes)
- Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h
- Login brute-force protection: LoginRateLimiter (5 attempts/5min per key)
- JWT aud/iss claims: create_jwt/validate_jwt support audience validation
- Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh
- CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var
- JWT secret strength: warning on secrets shorter than 32 chars
- Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing
* Refactor duplicated code into shared utilities and fix 3 UI bugs
Code deduplication (~235 net lines removed):
- Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py
- Create core/web_helpers.py (require_storage_or_503, read_json_or_400,
parse_cors_origins, cors_middleware)
- Extract add_redis_args/broker_from_args to mq/broker.py
- Extract add_log_args/configure_logging_from_args to core/log.py
- Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper,
unused required_role(), duplicate detect_model() wrapper
Bug fixes:
- Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server
nodes (use ServiceTokenManager with JWT_AUD_SERVER instead)
- Fix login form autofill: wrap inputs in <form>, add name attributes,
set type=submit on button
- Fix SSE reconnecting flash: add onopen handler to clear status
immediately on connection (not waiting for first message)
- Fix chat scroll: add min-height:0 to flex containers, overflow:hidden
on body to constrain viewport height
* Address CI typecheck failure and Copilot review feedback
- Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary)
- Bridge SSE loops: use event_hooks for auth header refresh on reconnect
instead of static headers that go stale after token rotation
- Login form: remove javascript:void(0) action (CSP anti-pattern)
- Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals
in middleware builder calls to prevent drift
* Add user identity, JWT auth, and admin console UI (#23)
JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.
New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.
Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.
Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).
Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().
New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.
* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review
Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
Replace ad-hoc logging.basicConfig() calls across all 6 entry points with
a centralized configure_logging() function backed by structlog. JSON output
when stderr is not a TTY (production/Docker), colored console output otherwise.
- New turnstone/core/log.py: configure_logging(), get_logger(), contextvars
for node_id/ws_id/user_id/request_id auto-injected into every log event
- All entry points (server, bridge, console, sim, cli, migrate) call
configure_logging() with --log-level and --log-format CLI flags
- Server operational print() calls replaced with structured log.info()
- LogContextMiddleware sets request_id + ws_id per HTTP request with
token-based reset to prevent context leaking across requests
- Bridge _run_in_context() helper propagates ctx_node_id to child threads
- Env var overrides: TURNSTONE_LOG_LEVEL, TURNSTONE_LOG_FORMAT
- 18 new tests (959 total passing)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner
Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery
Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links
SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary
Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).
* Address PR #22 review feedback
- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
* Fix console proxy regressions and add workstream task field (#21)
Bug fixes:
- Fix collector polling unversioned /api/dashboard (404 after API
versioning PR) — nodes showed red/unreachable, no workstreams
- Fix SSE proxy dropping all data events — upstream sends \r\n line
endings but proxy split on \n\n only; normalize before parsing
- Fix workstream state stuck on idle — on_state_change() only
broadcasted via SSE but never updated ws.state on the Workstream
object; dashboard polling now sees correct attention/running states
- Fix deep-link switchTab early return — when ?ws_id matched the
only workstream, switchTab bailed (wsId === currentWsId) before
establishing SSE connection; inline init instead of delegating
- Fix console banner covering dashboard overlay — inject <style>
offsetting .dashboard-overlay below the 32px banner
Enhancements:
- Add turnstone branding to console proxy banner (turnstone │ Console │ node-id)
- Add initial_message field to CreateWorkstreamMessage protocol and
console "New Workstream" modal (Task textarea, sent as first message)
- Refactor SSE proxy to use shared httpx client with 30s read timeout
instead of per-request client creation
- Increase approval timeout default from 300s to 3600s (1 hour)
Updated: Python SDK, TypeScript SDK, OpenAPI specs, MQ client,
API schemas, MQ protocol diagram, SDK docs.
* Address PR #21 review feedback (4 items)
- Log unknown state strings in on_state_change instead of silently
swallowing; remove unnecessary KeyError catch
- Wrap initial_message POST in _handle_create_ws with error handling
so workstream creation success isn't masked by send failure
- Strip all \r from SSE chunks instead of replacing \r\n, fixing
chunk-boundary split edge case
- Add tests for initial_message wiring in directed and pool targeting
* Refactor SSE proxy to use httpx-sse aconnect_sse
Replace manual SSE chunk buffering/parsing with httpx_sse.aconnect_sse()
which handles line endings, event types, and all SSE spec edge cases.
Eliminates the \r\n chunk-boundary bug class entirely. Event types are
now always forwarded (sse.event defaults to "message" per spec).
str(engine.url) in SQLAlchemy 2.x masks the password as '***',
causing SCRAM-SHA-256 authentication to fail when Alembic creates
its own engine from the config URL.
* Add Python and TypeScript client SDKs for server and console APIs
Python SDK (turnstone/sdk/) with sync + async clients for both server
and console APIs. Returns Pydantic models directly, streams SSE events
as typed dataclasses. 27 event types with registry-based deserialization.
High-level send_and_wait() for request-response patterns.
TypeScript SDK (sdk/typescript/) with zero browser dependencies. Uses
fetch + ReadableStream for SSE parsing. Discriminated union event types
with type guards. Same API surface as Python SDK.
63 Python tests, 21 TypeScript tests (vitest). Comprehensive docs at
docs/sdk.md with SDK architecture diagram.
* Address PR #19 review feedback + fix lint
- Fix consume_task leak in send_and_wait when send() raises (try/finally)
- Fix TS sendAndWait: open SSE before send, plumb AbortSignal for timeout
- Add signal param to TS streamSSE for cancellation support
- Fix SSE parser: join multi-line data: fields with \n per spec, handle CRLF
- Fix generate-types.py sys.path (parents[3] not parents[2])
- Document token ignored when httpx_client provided
- Document TS timeout units as milliseconds
- Fix stale docstring in test_sdk_sse.py
- Fix import sorting (ruff I001)
* Extract shared frontend design system into turnstone/shared_static/
The server UI and console UI had ~60% CSS overlap and significant JS
duplication. Extract shared assets into a new turnstone/shared_static/
package mounted at /shared/ in both servers:
- base.css: design tokens, reset, typography, login/toast/kb overlays,
dashboard table, state dots, health bar, scrollbar, reduced motion
- auth.js: authFetch, login overlay with focus trap, logout (hooks for
page-specific post-login/logout callbacks)
- theme.js: dark/light toggle with system preference detection
- toast.js: notification queue with configurable timeout
- utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount
- kb.js: keyboard shortcuts overlay with configurable content, focus
management, and focus restore on dismiss
Console proxy updated: JS shim injection moved from proxy_static (app.js
prepend) to proxy_index (inline <script> in HTML) so it runs before any
external scripts. New /shared/ path rewriting and proxy_shared_static
route added. ~1540 lines removed from page-specific files, 775 lines in
shared package. 13 new tests (788 total).
* Fix /shared/ auth and remove __init__.py from shared_static
Address PR #17 review feedback:
1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS
loads before authentication (required for login overlay to render)
2. Remove turnstone/shared_static/__init__.py to prevent exposing
Python package internals (__init__.py, __pycache__) via the
StaticFiles mount. Not needed for packaging since pyproject.toml
uses explicit glob includes.
3 new auth tests for /shared/ public path access.
* Add node version tracking and drift detection to console dashboard
Surface the version field from each node's /health endpoint in the
console dashboard. Collector extracts version into get_overview()
(version_drift + versions fields), promotes it to top-level in
get_nodes(), and adds get_version_info() for per-node detail. Console
/health endpoint includes drift fields.
Frontend adds a VER column to the 7-column node table grid, shows
per-node version strings, tracks versions per group with "mixed" +
yellow drift badge when nodes disagree, and displays a DRIFT warning
or single version in the status bar. Column hidden on mobile (<700px).
ARIA labels include version info for accessibility.
10 new tests (745 total). Docs and diagram updated.
* Fix drift tooltip text: show 'Versions detected' not 'Nodes running'
* Fix circuit breaker, rate limiter, and Anthropic web search correctness (#15)
Three tech debt items addressing correctness and security gaps:
Circuit breaker HALF_OPEN single-request permit:
- Rename should_allow_request property to acquire_request_permit() method
to make the side-effecting, non-idempotent nature explicit
- Add _half_open_permit flag: exactly one probe request in HALF_OPEN,
subsequent callers blocked until probe completes
- Explicitly reset permit on all state transitions (record_success,
record_failure) for clean state machine invariants
- Session uses BaseException catch to ensure record_failure always fires,
preventing permanent circuit deadlock on probe crash
Rate limiter X-Forwarded-For support:
- Add resolve_client_ip() with rightmost-untrusted XFF parsing
- Configurable trusted_proxies via --ratelimit-trusted-proxies CLI flag
and [ratelimit] trusted_proxies config (comma-separated CIDRs)
- IPv4-mapped IPv6 normalization (::ffff:x.x.x.x → IPv4) for dual-stack
- Clientless requests (request.client is None) pass through instead of
sharing a single "unknown" bucket
- Log warning for invalid CIDR entries in trusted_proxies config
- Show trusted proxies in startup log when enabled
Anthropic web search multi-turn encrypted content:
- Capture raw provider content blocks during streaming via _block_to_dict()
using model_dump(exclude_none=True) to avoid Anthropic API rejection
- Accumulate thinking_delta into raw_blocks (was silently empty on replay)
- Store _provider_content on assistant messages, pass through verbatim in
_convert_messages() so encrypted_content/encrypted_index survive turns
- Persist to SQLite via new provider_data column (auto-migrated)
- Add thinking/signature to _block_to_dict fallback attribute list
23 new tests (735 total), ruff + mypy clean.
* Fix Copilot PR #15 review issues: provider data, circuit breaker, IP normalization
- Persist assistant message when provider_data exists even if text
content is empty — prevents losing Anthropic web search encrypted
content needed for multi-turn replay (session.py)
- Re-raise KeyboardInterrupt/SystemExit immediately after recording
failure instead of attempting fallback models (session.py)
- Consume HALF_OPEN permit for the transition caller — prevents two
concurrent probe requests when only one should be allowed
(healthcheck.py)
- Normalize IPv4-mapped IPv6 addresses consistently in
resolve_client_ip() — prevents duplicate rate-limit buckets for
::ffff:x.x.x.x vs x.x.x.x (ratelimit.py)
* Add console workstream creation + server reverse proxy (#14)
Enable the console dashboard to create workstreams and proxy server UIs,
so users only need network access to the console port.
Workstream creation via MQ:
- POST /api/cluster/workstreams/new with three targeting modes:
specific node (directed queue), auto (best node by capacity),
or general pool (shared queue, any bridge picks up)
- Console pushes CreateWorkstreamMessage to Redis; bridge handles
the rest (server creation, ownership registration, SSE events)
Reverse proxy for server UIs:
- /node/{node_id}/ serves the server's HTML with static path rewriting
and a console-return banner injected after <body>
- JS proxy shim prepended to app.js overrides fetch() and EventSource()
to route root-relative URLs through /node/{id}/api/...
- SSE streams proxied via httpx.AsyncClient(timeout=None) with per-
connection clients for long-lived streams
- GET/POST API requests forwarded with body and auth token
Security:
- Proxy write paths checked against WRITE_PATHS to prevent read-token
escalation (read tokens cannot POST /api/send through proxy)
- html.escape() on node_id in banner HTML to prevent XSS
- String length limits on name/model inputs
Frontend:
- "+ new" button in header opens creation modal with node dropdown
(Auto / General pool / specific nodes with capacity display)
- Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling
- Workstream rows and node links deep-link via proxy paths
- Custom select arrow, Instrument Panel modal styling
Documentation:
- docs/console.md rewritten with proxy and creation API docs
- docs/architecture.md console section updated
- PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered
- README.md updated
28 new tests (741 total), ruff + mypy clean.
* Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness
- Normalize trailing slashes in required_role() to prevent write-role
bypass via /api/send/ or /node/{id}/api/send/ (auth.py)
- Validate node_id format in proxy handlers (alphanumeric, dot, dash,
underscore only) to prevent injection vectors
- Use json.dumps() for JS proxy shim prefix to prevent script injection
- URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static)
- Check upstream status in _proxy_sse() — emit error event on non-200
instead of keeping a dead SSE connection open
- Check upstream status in proxy_index() — propagate non-2xx errors
- Forward query string in _proxy_post() (consistency with _proxy_get)
- Handle JSON null values in create_workstream() — treat null as empty,
reject non-string types with 400
- Fix docs/diagram LPUSH → RPUSH to match actual broker implementation
* Add provider-native web search with Tavily fallback
Replace client-side Tavily web search with provider-native implementations:
- Anthropic: inject web_search_20250305 server-side tool, handle
server_tool_use / web_search_tool_result streaming blocks, emit
info_delta for search status display
- OpenAI: inject web_search_options for gpt-5-search-api, format
url_citation annotations as footnote sources
- Local/vLLM: preserve existing Tavily-based web_search tool as fallback
Add supports_web_search to ModelCapabilities and info_delta to StreamChunk.
Remove end-of-life GPT-4o model entries from capability tables.
Update docs, diagrams, and README. 88 provider tests (32 new).
* Fix Copilot PR #13 review: capture streaming url_citation annotations
Accumulate url_citation annotations during OpenAI streaming and emit
formatted citations as a final info_delta chunk after the stream ends.
Previously annotations were only captured in non-streaming mode, so
search model users in the interactive path never saw citation sources.
* Add multi-provider LLM adapter with model capability flags
Introduce a provider abstraction layer between ChatSession and LLM SDK
clients, enabling native support for Anthropic alongside OpenAI-compatible
APIs. Each provider translates at the API boundary while the internal
message format remains OpenAI-like throughout session history and persistence.
- LLMProvider protocol with StreamChunk/CompletionResult normalized types
- OpenAIProvider: GPT-4o, GPT-5.x, O-series capability tables with
conditional temperature, reasoning_effort, and token param handling
- AnthropicProvider: native streaming, message/tool format conversion,
adaptive vs manual thinking modes, effort parameter for 4.6 models
- ModelCapabilities per-model flags: temperature support, token param name,
thinking mode, effort levels, context window, max output
- Smart auto-detect: latest Opus for Anthropic, latest base GPT for OpenAI
- --provider CLI flag for both turnstone and turnstone-server
- anthropic SDK as optional dependency (pip install turnstone[anthropic])
- 56 new provider tests, 672 total passing
- Updated architecture docs and 4 PlantUML diagrams
* Fix Copilot PR #12 review: reasoning_effort gating, Anthropic thinking, provider factory
- Default reasoning_effort_values to () so unknown/local models don't
receive unsupported top-level reasoning_effort param. Models that need
it (GPT-5.x, search models) have explicit capability declarations.
- Fix Anthropic _reasoning_params: "none" and "" effort now return {}
instead of enabling thinking with 4096 budget.
- Use create_provider("openai") singleton instead of OpenAIProvider()
in ChatSession fallback for consistency with registry path.
- Add 8 parameter gating tests: unknown model no reasoning_effort,
GPT-5 no temperature, GPT-5.1 conditional temperature, O-series
no temperature, Anthropic none/empty/low effort.
* Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn
Replace Python stdlib http.server (ThreadedHTTPServer, BaseHTTPRequestHandler)
with Starlette ASGI applications served by uvicorn across all three HTTP
entry points. SSE endpoints use sse-starlette EventSourceResponse with
async generators that bridge sync queue.Queue via run_in_executor().
Bridge SSE parser replaced with httpx-sse EventSource.
- turnstone/server.py: Starlette app factory with create_app(), pure ASGI
middleware (auth, rate limit, metrics, CORS), async route handlers,
lifespan context manager for startup/shutdown. WebUI and ChatSession
remain fully synchronous — worker threads unchanged.
- turnstone/console/server.py: Same pattern, simpler (no ChatSession).
Path params replace manual string slicing for node detail route.
- turnstone/mq/bridge.py: _iter_sse_data() uses httpx_sse.EventSource
instead of hand-rolled line parser.
- Tests: All ThreadedHTTPServer fixtures replaced with
starlette.testclient.TestClient via create_app() factories.
- Docs: Updated architecture.md, api-reference.md, README.md, and
PlantUML diagrams (03, 11) + regenerated PNGs.
* Fix Copilot PR #11 review: TestClient cleanup, JSON error handling, SSE timeout
- Close TestClient in teardown for TestConsoleAuth and TestConsoleLogin to avoid lifespan/resource leaks
- Close TestClient via yield/finally in TestConsoleHTTPEndpoints fixture
- Add _read_json() helper for safe JSON body parsing (returns {} on invalid JSON instead of 500, matching old stdlib handler behavior)
- Apply same try/except pattern to console auth_login endpoint
- Increase SSE queue.get timeout from 1s to 5s to align with sse-starlette ping interval, reducing executor task churn
Move detect_model() to turnstone.core.model_registry as a single
implementation replacing duplicates in cli.py, server.py, and eval.py.
Auto-detects context window from backend metadata (meta.n_ctx_train)
when available, falling back to the 131072 default otherwise.
Also replace vLLM-specific references in help text and comments with
generic "OpenAI-compatible API" / "model server" language.
The developer message role is not supported by all chat templates
(e.g. Qwen). Switch to the standard system role which is universally
supported by OpenAI-compatible APIs.
2026-03-02 23:22:30 -08:00
511 changed files with 148240 additions and 13336 deletions
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers, driven by message queues or interactive interfaces.
Multi-node AI orchestration platform. Deploy tool-using AI agents across a cluster of servers with direct HTTP routing, interactive interfaces, and enterprise governance.
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) — a bird that flips rocks to expose what's hiding underneath.
<p align="center">
<img src="docs/assets/hero.png" alt="Turnstone console — multi-workstream AI orchestration with mermaid diagrams" width="960"/>
</p>
Named after the [Ruddy Turnstone](https://en.wikipedia.org/wiki/Ruddy_turnstone) (*Arenaria interpres*) — a shorebird that flips stones to discover what's hiding underneath.
| **Experimental** | `pip install turnstone --pre` | `ghcr.io/turnstonelabs/turnstone:experimental` | New features. May have rough edges. |
See [docs/releasing.md](docs/releasing.md) for the full release process.
## What it does
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports. It runs as:
Turnstone gives LLMs tools — shell, files, search, web, planning — and orchestrates multi-turn conversations where the model investigates, acts, and reports.
- **Interactive sessions** — terminal CLI or browser UI with parallel workstreams
- **Queue-driven agents** — trigger workstreams via message queue, stream progress, approve or auto-approve tool use
- **Multi-node clusters** — generic work load-balances across nodes, directed work routes to a specific server
- **Cluster dashboard** — real-time view of all nodes, workstreams, and resource utilization
- **Cluster simulator** — test the stack at scale (up to 1000 nodes) without an LLM backend
- **Cluster dashboard** — real-time view of all nodes and workstreams with console routing proxy
- **Intent validation** — LLM judge evaluates every tool call with risk assessments and evidence
| `turnstone:events:global` | Global event pub/sub |
| `turnstone:events:cluster` | Cluster-wide state changes (for turnstone-console) |
**Routing rules:**
1. Message has `target_node` → routes to that node's queue
2. Message has `ws_id` → looks up owner, routes to owning node
3. Neither → shared queue, next available bridge picks it up
Bridges BLPOP from their per-node queue (priority) then the shared queue. Directed work always takes precedence.
## Tools
14 built-in tools, 2 agent tools, plus external tools via MCP:
Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration.
| Tool | Description | Auto-approved |
|------|-------------|:---:|
| `bash` | Execute shell commands | |
| `read_file` | Read file contents | yes |
| `write_file` | Write/create files | |
| `edit_file` | Fuzzy-match file editing | |
| `search` | Search files by name/content | yes |
| `math` | Sandboxed Python evaluation | |
| `man` | Read man pages | yes |
| `web_fetch` | Fetch URL content | |
| `web_search` | Search via Tavily API | |
| `remember` | Save persistent facts | yes |
| `recall` | Search memories and history | yes |
| `forget` | Remove a memory | yes |
| `task` | Spawn autonomous sub-agent | |
| `plan` | Explore codebase, write .plan.md | |
| `mcp__*` | External tools from MCP servers | |
## Architecture
### MCP Tool Servers
**Single-node**: Client → Server (direct HTTP + SSE). No external dependencies beyond the database.
Turnstone supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) for connecting external tool servers. MCP tools are discovered at startup, converted to OpenAI function-calling format, and merged with built-in tools. Each MCP tool is prefixed with `mcp__{server}__{tool}` to avoid name collisions.
**Multi-node**: Client → Console (hash ring routing proxy) → Server nodes. The console maintains a 65536-entry bucket cache for O(1) workstream routing. A rebalancer daemon redistributes buckets when nodes join or leave.
Configure via `config.toml` or `--mcp-config`:
| Component | Purpose |
|-----------|---------|
| `turnstone` | Terminal CLI (REPL) |
| `turnstone-server` | Web UI + REST API + SSE events |
Use `/mcp` in the REPL to list connected tools. MCP tools require user approval by default (overridden by `--skip-permissions` or UI auto-approve).
### Multi-Model Support
Turnstone supports multiple model backends per server instance. Define named models in `config.toml` and select per-workstream or switch mid-session with `/model <alias>`.
```toml
[models.local]
base_url="http://localhost:8000/v1"
model="qwen3-32b"
[models.openai]
base_url="https://api.openai.com/v1"
api_key="sk-..."
model="gpt-4o"
context_window=128000
[model]
default="local"# which model to use by default
fallback=["openai"]# try these if the primary is unreachable
agent_model="local"# optional: cheaper model for plan/task sub-agents
```
Use `/model` to show available models, `/model openai` to switch. Workstreams created via the API accept an optional `model` parameter.
## Configuration
All entry points read `~/.config/turnstone/config.toml`. CLI flags override config values.
```toml
[api]
base_url="http://localhost:8000/v1"
api_key=""
tavily_key=""
[model]
name=""# empty = auto-detect
temperature=0.5
reasoning_effort="medium"
default="default"# model alias for new workstreams
fallback=[]# ordered list of fallback model aliases
agent_model=""# model alias for plan/task sub-agents
[tools]
timeout=30
skip_permissions=false
[server]
host="0.0.0.0"
port=8080
max_workstreams=10# auto-evicts oldest idle when full
[redis]
host="localhost"
port=6379
password=""
[bridge]
server_url="http://localhost:8080"
node_id=""# empty = hostname_xxxx
[console]
host="0.0.0.0"
port=8090
url="http://localhost:8090"# used by CLI /cluster commands
poll_interval=10
[health]
backend_probe_interval=30
backend_probe_timeout=5
circuit_breaker_threshold=5
circuit_breaker_cooldown=60
[ratelimit]
enabled=true
requests_per_second=10.0
burst=20
[mcp]
config_path=""# path to MCP JSON config file (alternative to TOML sections)
Parallel independent conversations, each with its own session and state:
| Symbol | State | Meaning |
|--------|-------|---------|
| `·` | idle | Waiting for input |
| `◌` | thinking | Model is generating |
| `▸` | running | Tool execution in progress |
| `◆` | attention | Waiting for approval |
| `✖` | error | Something went wrong |
Idle workstreams are automatically cleaned up after 2 hours (configurable). In multi-node deployments, workstream ownership is tracked in Redis — follow-up messages auto-route to the owning node.
-`turnstone_circuit_state` — circuit breaker state (0=closed, 1=open, 2=half_open)
-`turnstone_workstreams_evicted_total` — workstreams auto-evicted at capacity
Per-workstream metrics are labeled by `ws_id` (bounded to 10 max workstreams).
### Health & Rate Limiting
**Health degradation.** A background `BackendHealthMonitor` probes the LLM backend every `backend_probe_interval` seconds. When the backend is unreachable, `/health` reports `"status": "degraded"` (HTTP 200) and the `turnstone_backend_up` gauge drops to 0.
**Circuit breaker.** After `circuit_breaker_threshold` consecutive probe failures the circuit opens (CLOSED -> OPEN). While open, `ChatSession._create_stream_with_retry` skips the backend entirely and returns an error. After `circuit_breaker_cooldown` seconds the circuit enters HALF_OPEN, allowing a single probe. A successful probe closes the circuit; a failure re-opens it.
**Per-IP rate limiting.** When `[ratelimit].enabled` is true, each client IP is tracked with a token-bucket limiter (`requests_per_second` / `burst`). Rate limiting is applied in `do_GET`/`do_POST` after authentication but before route dispatch. `/health` and `/metrics` are exempt. Requests that exceed the limit receive HTTP 429 with a `Retry-After` header.
**Workstream eviction.** When `WorkstreamManager.create()` would exceed `max_workstreams`, the oldest IDLE workstream is automatically evicted and the `turnstone_workstreams_evicted_total` counter is incremented. Configure via `[server].max_workstreams` (default 10).
- An OpenAI-compatible API endpoint ([vLLM](https://github.com/vllm-project/vllm), [NVIDIA NIM](https://build.nvidia.com/), [llama.cpp](https://github.com/ggml-org/llama.cpp), etc.)
`turnstone-console` is a standalone monitoring service that provides cluster-wide visibility across all turnstone nodes. It connects to the shared Redis broker, discovers nodes via heartbeat keys, polls each node's HTTP API for workstream data, and subscribes to a cluster event channel for real-time state changes.
`turnstone-console` is a cluster management service that provides cluster-wide visibility and control across all turnstone nodes. It discovers nodes via the`services` database table and subscribes to each node's SSE event stream for real-time workstream, health, and metric updates.
The console is read-only — it observes but does not own workstreams or drive LLM sessions.
The console also supports **workstream creation** (dispatched via HTTP proxy to target nodes) and a **reverse proxy** that serves each node's server UI through the console port — so users only need network access to the console, not to individual server nodes.
## Architecture
> See also: [Console Data Flow diagram](diagrams/png/11-console-data-flow.png)
Each bridge publishes state changes to `{prefix}:events:cluster` on Redis pub/sub. The console subscribes once to that channel for real-time updates and periodically polls each node's `GET /api/dashboard` for full workstream snapshots.
Data flows in two directions:
- **Inbound (monitoring):** The console discovers nodes via the `services` database table (nodes register on startup and send periodic heartbeats). It opens a persistent SSE connection to each node's `GET /v1/api/events/global` endpoint, receiving a full snapshot on connect followed by real-time delta events (state changes, health transitions, aggregate metrics).
- **Outbound (control):** The console proxies workstream creation requests to target nodes via HTTP.
- **Proxy (pass-through):** The console reverse-proxies each node's server UI at `/node/{node_id}/`, forwarding HTTP and SSE traffic so the browser never contacts server nodes directly.
| Node HTTP API | `GET {server_url}/api/dashboard` | Every 10s | Full workstream list with tokens, context, activity |
| Node HTTP API | `GET {server_url}/health` | Every 10s | Node health status |
### Redis Key: Cluster Event Channel
Bridges publish to `{prefix}:events:cluster` whenever a workstream state change, creation, closure, or rename occurs. Events include `node_id` so the console can attribute them to the correct node.
| Node HTTP API | `GET/POST {server_url}/*` | Proxy | Server UI, API requests, SSE streams |
---
## ClusterCollector
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Three daemon threads handle data acquisition:
The collector (`turnstone/console/collector.py`) maintains an in-memory snapshot of all nodes and workstreams. Two daemon threads handle data acquisition:
1. **Event subscriber** — subscribes to`{prefix}:events:cluster` via `RedisBroker.subscribe_cluster()`. Applies state changes, creates, closes, and renames to the in-memory model immediately.
1. **Node discovery** — queries the`services` database table every 60 seconds. Adds newly discovered nodes, removes expired ones (stale heartbeats), emits `node_joined` / `node_lost` events to SSE listeners, and spawns/cancels SSE tasks for new/lost nodes.
2. **Node discovery** — scans heartbeat keys every 15 seconds via `broker.list_nodes()`. Adds newly discovered nodes, removes expired ones, emits `node_joined` / `node_lost` events to SSE listeners.
2. **SSE manager** — a single asyncio event loop on one thread multiplexes persistent SSE connections to all discovered nodes via `GET /v1/api/events/global`. Each connection receives a `node_snapshot` on connect (workstreams, health, aggregate) followed by real-time delta events (`ws_state`, `ws_created`, `ws_closed`, `ws_rename`, `health_changed`, `aggregate`). On disconnect, the node is marked unreachable and the connection is retried with exponential backoff (1s–30s). An `?expected_node_id=` query parameter provides identity verification against IP reuse (server returns 409 on mismatch).
3. **Poll loop** — fetches `GET /api/dashboard` and `GET /health` from each known node every 10 seconds. Uses `ThreadPoolExecutor(max_workers=50)` for parallelism. Each poll replaces the node's workstream list with the authoritative server data.
A `get_snapshot()` method builds the full cluster state under a single lock acquisition — overview aggregates and per-node workstream lists in one atomic read. This is served both as a REST endpoint and as the initial SSE event on client connect.
### Thread Safety
@@ -55,16 +52,17 @@ All reads and writes to the node/workstream map are protected by a single `threa
### Scale Considerations
- **10,000 workstreams** at ~500 bytes each = ~5 MB in memory
- **1,000 nodes**polled in parallel with 50 threads at ~100ms each = ~2 second poll cycle
- **50,000 workstreams** (1,000 nodes × 50 per node) at ~500 bytes each = ~25 MB in memory
- **1,000 nodes**connected via persistent SSE — a single asyncio event loop multiplexes all connections with negligible overhead. Ensure `ulimit -n` >= 4096 for fd headroom
- **Filtering and pagination** run in-memory on the full workstream list — sub-millisecond at this scale
- **SSE fan-out** uses the same per-client queue pattern as the per-node server — backed-up clients get events dropped, not blocking
- **SSE fan-out** uses per-client queues (2,000 events) — backed-up clients get events dropped, not blocking
- **Database** — for clusters sharing PostgreSQL, use [PgBouncer](pgbouncer.md) in transaction pooling mode
---
## HTTP API
### `GET /api/cluster/overview`
### `GET /v1/api/cluster/overview`
Cluster-wide state counts and aggregate metrics.
@@ -73,11 +71,15 @@ Cluster-wide state counts and aggregate metrics.
Filtered, paginated workstream list. All query parameters are optional. `per_page` is capped at 200.
@@ -115,7 +118,7 @@ Filtered, paginated workstream list. All query parameters are optional. `per_pag
}
```
### `GET /api/cluster/node/{node_id}`
### `GET /v1/api/cluster/node/{node_id}`
Single node detail with all its workstreams.
@@ -129,9 +132,75 @@ Single node detail with all its workstreams.
}
```
### `GET /api/cluster/events`
### `GET /v1/api/cluster/snapshot`
Server-Sent Events stream for real-time cluster updates.
Full cluster state in a single response — all nodes with their workstreams plus overview aggregates. Built under a single lock for internal consistency. Used by the browser on initial load and SSE reconnect.
Create a new workstream on a target node. The console proxies the creation request to the target node's HTTP API. Requires `write` scope.
Request:
```json
{
"node_id": "db-west-04",
"name": "perf-analysis",
"model": "gpt-5"
}
```
All fields are optional:
- `node_id` — targeting mode:
- **omitted or `"auto"`** — console picks the reachable node with the most available capacity (max_ws - ws_total) and proxies the request to it.
- **`"pool"`** — console picks a reachable node with available capacity using round-robin selection.
- **specific node ID** — proxies the request to that node directly.
- `name` — workstream display name. Auto-generated if omitted.
- `model` — model alias from the target node's registry. Uses the node's default model if omitted.
Response:
```json
{
"status": "ok",
"correlation_id": "a1b2c3d4e5f6",
"target_node": "db-west-04"
}
```
The response confirms the workstream creation request was proxied to the target node. A `ws_created` event on the cluster SSE stream confirms the workstream was actually created.
### `GET /v1/api/cluster/events`
Server-Sent Events stream for real-time cluster updates. The first event is always a `snapshot` containing the full cluster state (same shape as `GET /v1/api/cluster/snapshot` with an added `type: "snapshot"` field), followed by incremental events:
User and token management endpoints. All admin endpoints require `approve` scope, except for the setup endpoint which is public.
#### `POST /v1/api/auth/setup`
Creates the first admin user when no users exist. Public endpoint (no auth required). Returns a JWT and sets a session cookie. Returns `409` if users already exist. See [Security: First-time setup](security.md#first-time-setup) for full details.
Create an API token for the given user. Returns a `ts_`-prefixed token string that can be used for Bearer auth or passed to `client.login(token="ts_xxx")`.
```json
{
"name": "CI pipeline",
"scopes": ["read", "write"]
}
```
#### `GET /v1/api/admin/users/{user_id}/tokens`
List active tokens for a user (token strings are not returned, only metadata).
#### `DELETE /v1/api/admin/tokens/{token_id}`
Revoke a specific API token.
### Channel links
| Method | Path | Description |
|--------|------|-------------|
| GET | `/v1/api/admin/users/{user_id}/channels` | List channel links for a user |
| POST | `/v1/api/admin/users/{user_id}/channels` | Link a channel account (channel_type, channel_user_id) |
| DELETE | `/v1/api/admin/channels/{channel_type}/{channel_user_id}` | Unlink a channel account |
These endpoints manage the `channel_users` table mappings that connect external platform identities (e.g. Discord user IDs) to turnstone users. See [Channel Integrations](channels.md) for details on the linking flow.
#### `GET /v1/api/auth/status`
Public endpoint for login UI state detection. Returns auth configuration, not
current-user identity.
```json
{
"auth_enabled": true,
"has_users": true,
"setup_required": false
}
```
### Auth Scopes
The auth system uses three scopes instead of the earlier read/full role model:
| `approve` | Admin operations: manage users and API tokens |
Scopes are cumulative — a user with `approve` scope can also perform `write` and `read` operations.
---
## Reverse Proxy
The console reverse-proxies each node's server UI at `/node/{node_id}/`. This allows users to interact with any node's workstreams through the console port alone — individual server ports do not need to be exposed to the office network.
### Proxy Routes
| Route | Behavior |
|-------|----------|
| `GET /node/{node_id}/` | Fetches the server's `index.html`, rewrites static and shared asset paths, injects a console-return banner and an inline JS proxy shim |
The server UI uses root-relative URLs (`/v1/api/send`, `/static/app.js`, `/shared/base.css`, etc.). Since `<base>` tags cannot rewrite root-relative URLs, the console uses a JS shim approach:
1. **HTML rewriting** — when serving `index.html`, replaces `href=` and `src=` references to both `/static/` and `/shared/` with the proxy prefix (`/node/{node_id}/static/` and `/node/{node_id}/shared/` respectively).
2. **Inline JS shim** — injects an inline `<script>` block into the proxied HTML (after the console-return banner, before any external scripts) that overrides `window.fetch()` and `window.EventSource()` to prepend the proxy prefix to any root-relative URL. Running the shim inline ensures it executes before any external scripts load, so all API calls and SSE connections are intercepted transparently.
3. **Console-return banner** — injects a thin inline-styled `<div>` after `<body>` with a "← Console" link and the node ID, providing navigation back to the dashboard.
### SSE Proxy
SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte passthrough — the console opens an `httpx.AsyncClient.stream()` to the upstream server (with `read=None` and `pool=None` timeouts since SSE connections are long-lived) and relays every byte via `StreamingResponse`. This preserves server-side ping comments, event framing, and keepalives verbatim without parsing or re-encoding.
### Authentication
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
## Browser Dashboard
The web UI has three views, toggled client-side:
The web UI has five views, toggled client-side:
### 1. Cluster Overview (landing)
- **State cards** — 5 clickable cards (running, thinking, attention, idle, error) with count and colored top border. Clicking filters to that state.
- **Aggregate bar** — total tokens and tool calls across the cluster.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, HEALTH. Sorted by activity. Clickable rows drill down to node detail.
- **Node table** — columns: NODE, WS, RUN, ATTN, TOKENS, VER, LOAD. Sorted by activity. Clickable rows drill down to node detail. Version column shows per-node version; hidden on mobile.
- **Version drift indicator** — when nodes report different versions, the status bar shows a yellow "DRIFT" warning with a tooltip listing all versions. Node groups show "mixed" with a yellow badge when their members disagree.
- **"+ new" button** — opens the workstream creation modal (see below).
### 2. Node Drill-down
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's own dashboard (`http://{server_url}/`).
Breadcrumb: `Cluster > db-west-04`. Shows the node's workstreams in a table matching the per-node dashboard layout (STATE, NAME, MODEL, NODE, TASK, TOKENS, CTX) with activity sub-lines. Includes a link to the node's proxied server UI.
**Deeplinking:** Clicking a workstream row opens the node's server UI in a new tab with `?ws_id=<id>`, which auto-selects that workstream. A `↗` indicator appears on hover to signal the external navigation. Rows without a `server_url` are non-interactive.
**Proxy deep-linking:** Clicking a workstream row opens the node's server UI in a new tab via the proxy at `/node/{node_id}/?ws_id=<id>`, which auto-selects that workstream. Users do not need direct network access to the server node.
### 3. Filtered Workstreams
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows are deep-linkable when `server_url` is available (injected by the collector from the parent node).
Breadcrumb: `Cluster > Running` or `Cluster > db-west-04`. Server-side paginated workstream table. NODE column values are clickable to filter further. Pagination controls at bottom. Workstream rows use proxy deep-links.
All three views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
### 4. Workstream Creation Modal
Triggered by the "+ new" header button. A modal dialog with:
- **Node selector** — dropdown with three targeting modes: "Auto (best available)" picks the node with the most headroom, "General pool (any node)" picks a node with available capacity using round-robin, or a specific node from the list (showing capacity).
- **Profile** — optional dropdown listing enabled skills. Applies the skill's model, auto-approve policy, token budget, and other behavioral settings at creation time.
- **Name** — optional text input. Auto-generated if left empty.
- **Model** — optional text input for a model alias from the target node's registry.
On submit, `POST /v1/api/cluster/workstreams/new` dispatches the creation request. A toast confirms success; the SSE stream delivers the `ws_created` event to update the dashboard.
All five views receive live updates via SSE — state cards update counts, node rows update metrics, workstream rows update state indicators.
The browser maintains a local `clusterState` object that mirrors the cluster snapshot. It is initialized from the SSE `snapshot` event on connect (or via `GET /v1/api/cluster/snapshot` on initial page load) and updated incrementally by SSE events. View navigation reads from local state — no API round-trips needed after the initial snapshot.
### 5. Admin Panel
Accessed via the "admin" button in the header (visible when authenticated
with `approve` scope). Provides user, API token, channel link, MCP server,
and skill management with 13 tabs (see also
[Governance](governance.md) for
the Roles, Policies, Skills, Usage, and Audit tabs, and
[Settings](settings.md) for the database-backed configuration editor):
- "Create User" button opens a modal with fields for username, display name,
and password (validated: username 1-64 ASCII, password min 8 characters)
- Delete button on each row opens a styled confirmation modal before
removing the user and cascading to revoke all their tokens
**Tokens tab:**
- User selector dropdown to pick which user's tokens to manage
- Grid table listing tokens for the selected user (name, prefix, scopes,
creation date)
- Scope badges rendered as colored pills for visual clarity
- "Create Token" button opens a modal with fields for token name and scope
checkboxes
- On creation, a "Token Created" modal displays the raw `ts_`-prefixed
token with a copy button. The token is shown once and cannot be retrieved
again.
- Revoke button on each row opens a styled confirmation modal before
deleting the token
**Channels tab:**
- User selector dropdown to pick which user's channel links to manage
- Grid table listing linked channel accounts for the selected user
(channel type, channel user ID, creation date)
- "Link Channel" button opens a modal with fields for channel type
(e.g. `discord`) and the platform user ID
- Unlink button on each row opens a styled confirmation modal before
removing the channel mapping
- Admins can force-link users who have not self-linked via `/link` in
Discord
**MCP Servers tab:**
The tab has two views toggled via a pill control: **Servers** and
**Registry**.
- **Servers view** -- lists all installed MCP servers with source badges
(CONFIG, MANUAL, REGISTRY), transport badges, tool/resource/prompt
counts, per-node connection status, and CRUD actions for DB-managed
servers
- **Registry view** -- search the official MCP Registry to discover and
install servers. Results show server name, description, version, source
type badges (remote/npm/pypi), and Install/Installed/Update buttons.
Remote servers without required configuration are installed with one
click; servers needing env vars, headers, or URL variables open an
install modal for configuration
**Accessibility:**
- Full keyboard navigation: focus traps in modals, Escape to close, arrow
keys for tab switching
- Responsive layout with column hiding at 700px breakpoint
**First-time setup:**
The console also exposes `POST /v1/api/auth/setup` for first-time
bootstrap. When no users exist, the setup wizard calls this public endpoint
to create the initial admin user and receive a JWT in one step. See
[Security: First-time setup](security.md#first-time-setup) for details.
---
## Scheduled Tasks
The console includes a background **TaskScheduler** daemon that creates workstreams on a timed basis via HTTP proxy to target nodes. It supports cron-based recurring schedules and one-shot `at` schedules.
### Architecture
The scheduler runs as a daemon thread inside the console process. Every `check_interval` seconds (default 15) it:
1. Acquires a distributed lock via the `system_settings` table (prevents duplicate dispatch in multi-console deployments)
2. Queries the storage backend for tasks whose `next_run <= now` and `enabled = true`
3. Dispatches each due task as one or more workstream creation requests via HTTP proxy
4. Updates `last_run` and computes the next `next_run` (or disables one-shot `at` tasks)
5. Releases the lock
Run history is automatically pruned (runs older than 90 days) approximately once per hour.
### Schedule Types
| Type | Field | Behavior |
|------|-------|----------|
| `cron` | `cron_expr` | Recurring schedule using standard 5-field cron syntax. Requires `croniter`. |
| `at` | `at_time` | One-shot: fires once at the given ISO 8601 timestamp (must include timezone), then auto-disables. |
### Target Modes
| Mode | Behavior |
|------|----------|
| `auto` | Picks the reachable node with the most available capacity |
| `pool` | Picks a reachable node with available capacity using round-robin |
| `all` | Fan-out to all reachable nodes (capped at `max_fan_out`, default 20) |
| `<node_id>` | Targets a specific node by ID |
### Configuration
| Parameter | Default | Description |
|-----------|---------|-------------|
| `check_interval` | `15.0` | Seconds between scheduler ticks |
| `max_fan_out` | `20` | Maximum nodes for `all` target mode |
Dependency: `croniter` (installed with turnstone).
### Schedule API
All schedule endpoints require `approve` scope. Maximum 200 schedules.
#### `GET /v1/api/admin/schedules`
List all scheduled tasks.
```json
{
"schedules": [
{
"task_id": "a1b2c3d4",
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 ** *",
"at_time": "",
"target_mode": "auto",
"model": "",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"auto_approve_tools": [],
"enabled": true,
"created_by": "u_admin",
"last_run": "2026-03-05T02:00:00Z",
"next_run": "2026-03-06T02:00:00Z",
"created": "2026-03-01T12:00:00Z",
"updated": "2026-03-05T02:00:01Z"
}
]
}
```
#### `POST /v1/api/admin/schedules`
Create a scheduled task.
Request:
```json
{
"name": "nightly-checks",
"description": "Run nightly health checks",
"schedule_type": "cron",
"cron_expr": "0 2 ** *",
"target_mode": "auto",
"initial_message": "Run the nightly health check suite.",
"auto_approve": false,
"enabled": true
}
```
Required fields: `name`, `schedule_type`, `initial_message`. For `cron` schedules provide `cron_expr`; for `at` schedules provide `at_time` (ISO 8601 with timezone, must be in the future).
Response: `ScheduleInfo` (same shape as list items above). Returns `400` for invalid cron syntax, naive timestamps, or past `at_time`. Returns `409` if the 200-schedule cap is reached.
#### `GET /v1/api/admin/schedules/{task_id}`
Get a single scheduled task. Returns `ScheduleInfo` or `404`.
#### `PUT /v1/api/admin/schedules/{task_id}`
Partial update — only include fields to change. If `schedule_type`, `cron_expr`, or `at_time` change, `next_run` is recomputed automatically.
```json
{
"enabled": false
}
```
Response: updated `ScheduleInfo`. Returns `400` for validation errors, `404` if not found.
#### `DELETE /v1/api/admin/schedules/{task_id}`
Delete a scheduled task and all its run history. Returns `{"status": "ok"}` or `404`.
Open `http://localhost:8090` for the cluster dashboard.
Open `http://localhost:8090` for the cluster dashboard. Create workstreams via the "+ new" button. Click any workstream to open the proxied server UI — no direct access to server ports required.
component [**turnstone:node:{node_id}**\n\nNode heartbeat + metadata.\nValue: JSON {server_url, started, ...}\nTTL: 60s (refreshed every 30s)\n\nOps: SET with EX, GET, SCAN] as node_hb <<STRING>>
}
package "Event Channels (Redis PUBSUB)" #FFF3E0 {
component [**turnstone:events:global**\n\nGlobal event broadcast.\nAll state changes, ws lifecycle.\n\nOps: PUBLISH, SUBSCRIBE] as evt_global <<PUBSUB>>
component [**turnstone:events:{ws_id}**\n\nPer-workstream events.\nContent, tools, status.\n\nOps: PUBLISH, SUBSCRIBE] as evt_ws <<PUBSUB>>
component [**turnstone:events:cluster**\n\nCluster-wide state changes.\nUsed by Console dashboard.\n\nOps: PUBLISH, SUBSCRIBE] as evt_cluster <<PUBSUB>>
oid sha256:98ba80fa1dab4d37299e61be079a6fbc8740fc3ab92196f828a765f74caf4556
size 200720
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Patrick Buckley
