mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-16 08:52:27 -06:00
376da3d084471fa8ea0e307ffedccc660409cd4e
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
8895bf07eb |
feat: admin Settings tab — form-based editor replacing "coming soon" … (#60)
* feat: admin Settings tab — form-based editor replacing "coming soon" stub Section-grouped layout with collapsible headers for all ~40 ConfigStore settings (model, session, tools, server, mcp, ratelimit, health, judge, memory). Type-appropriate inputs: CSS toggle for bools, number with min/max/step, select for choices, text for strings. Secret fields shown read-only. Source badge (storage/default), amber restart indicator. Inline save per field with dirty detection, row flash on success, reset to default via styled confirm modal. Full WCAG keyboard accessibility (Enter/Space on section headers, aria-labels, focus-visible). Mobile responsive single-column at <700px. Reduced-motion safe. * fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes Settings UI: - Help tooltips: ? button on ~25 settings with plain-English explanations and optional reference links (arXiv, Fowler, MCP spec). Click to toggle popover, Escape to dismiss, aria-expanded for accessibility. - Sections start collapsed for scannable overview. - Restart badge: hidden by default, shows when dirty, persists after save with amber glow. Positioned left of source badge. - Secret row alignment fixed (transparent border matches input box model). - Docs link in toolbar → Swagger UI Settings section. - Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard, numeric dirty detection (0.1 vs 0.10 no longer false positive). - Secret reset button enabled when source=storage (clear legacy overrides). - Space key repeat guard on section headers. - Sidebar: sticky + max-height:100vh, no longer stretches with content. Backend: - context_window default changed from 131072 to 0 (auto-detect). Fallback lowered from 131K to 32K (realistic for local models when detection fails). Session normalizes 0→32768 defensively. - Settings registry: help + reference_url fields on SettingDef, richer descriptions for model/session/tools/judge/memory settings. - Schema API includes help + reference_url. - Bootstrap system prompt: added Runtime Settings section. Docs: tab counts updated to 13 across README, architecture, console, governance. |
||
|
Patrick Buckley
|
02d9c5c797 |
feat: workstream templates — behavioral profiles for workstream creation (#49)
* feat: workstream templates — behavioral profiles for workstream creation Workstream templates define the complete configuration for workstream creation: system prompt, model, auto-approve policy, per-tool auto-approve, temperature, reasoning effort, max tokens, agent max turns, token budget, and completion notifications. Applied once at creation time (snapshot, not live binding). Auto-versioning captures pre-update state on every edit. Schema & storage: - workstream_templates + workstream_template_versions tables (migration 011) - ws_template_id/ws_template_version columns on workstreams table - ws_template column on scheduled_tasks table - Full CRUD + versioning on SQLite and PostgreSQL backends - prompt_template_hash (SHA-256) for drift detection Runtime: - Template resolution before mgr.create() for model override - Post-creation settings application (prompt, temperature, approval, budget) - Token budget enforcement in session.send() — 80% warning, approval gate at 100% via __budget_override__ synthetic tool - WebUI.auto_approve_tools server-side per-tool auto-approve - Prompt template drift detection (hash comparison, log warning on mismatch) Integration: - ws_template field on CreateWorkstreamMessage, bridge, channel router, scheduler dispatch, MQ client - Console admin "WS Templates" tab (11th) with CRUD, version history modal - Profile dropdown on workstream creation modal - WS template dropdown on scheduler create/edit modals - Prompt template name validation on ws_template create/update - 7 console admin API endpoints + read-only summary endpoint - Full OpenAPI spec entries in console_spec.py - Python SDK (sync + async) and TypeScript SDK methods - Pydantic schemas for all request/response models Docs & diagrams: - New 21-ws-template-architecture.puml sequence diagram - Updated governance, storage, MQ protocol diagrams + PNGs - Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md 48 new tests (1788 total). mypy clean. ruff clean. * fix: address PR #49 review feedback - auto_approve_tools uses approval_label (not just func_name) for consistency with tool policy evaluation - inline system_prompt from ws_template persisted as _ws_template_system_prompt in workstream_config, restored on resume (previously lost because _template_content wasn't persisted) - budget gate (__budget_override__) no longer bypassed by blanket auto_approve — requires explicit approval or tool policy allow - diagram 21 field list corrected (removed tool_search/threshold, added prompt_template_hash/notify_on_complete) * fix: address PR #49 review feedback (round 2) - Grant admin.ws_templates permission in migration 011 (tab was hidden) - Center WS template modals and fix radio button alignment - Skip template validation when ws_template overrides prompt - Guard against empty version snapshots on no-op updates - Replace setTimeout race with Promise chain in schedule ws_template select - Validate numeric fields in admin create/update handlers (400 not 500) - Add ws_template to TypeScript OpenAPI specs - Use typed Pydantic response models in SDK ws_template methods |
||
|
Patrick Buckley
|
5f0004dc91 |
feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild
The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.
Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).
Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.
SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.
* fix: address review feedback and SSE proxy reconnect bug
Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
both state and listener locks, eliminating the event gap between
snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
console schema models (ClusterNodeInfo, NodeDetailResponse,
ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
and ClusterOverviewResponse types instead of Record<string, unknown>.
SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
when no upstream data arrives, preventing the browser EventSource
from dropping idle connections. The raw byte passthrough refactor
(
|
||
|
Patrick Buckley
|
4d1107839b |
refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing - Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy - Stream bytes verbatim to preserve server-side ping comments and event framing - Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering) - Update compose.yaml to add 'cluster' profile to the service * Refactor SSE proxy to raw byte passthrough - turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with httpx.stream() + StreamingResponse for raw byte passthrough. Server pings, events, and comments now flow through verbatim. Added per-request timeout override (read=None, pool=None) for long-lived SSE streams. - tests/test_console.py: Add 3 new tests for SSE proxy: - Ping and event preservation - Upstream error status handling - Client disconnect handling - docs/console.md: Update SSE Proxy section to reflect raw byte passthrough approach. |
||
|
Patrick Buckley
|
924b976f1f |
Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system (cron/at scheduling via console API). Includes Python SDK methods (async + sync), TypeScript SDK methods, console.md API reference, sdk.md table update, and architecture.md module map entry. |
||
|
Patrick Buckley
|
a6e929b0a0 |
Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24) Bidirectional channel adapter framework connecting external messaging platforms to turnstone workstreams via Redis MQ. Discord ships as the first adapter; the protocol supports future Slack/Teams integrations. Channel framework: - ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping - AsyncRedisBroker with single dispatch loop and per-channel ordered workers - channel_routes table (migration 003) for persistent route storage - 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD) - Unified turnstone-channel gateway entry point, loads adapters by config - Message chunking, approval formatting, plan review formatting Discord adapter: - discord.py v2.4+ bot with thread-per-@mention model - Slash commands: /link (modal), /unlink, /ask, /status, /close - Persistent button views for tool approval and plan review - Streaming responses via edit-in-place (1.5s interval) - Stale route detection and atomic session resume via resume_session field - SessionResumedEvent confirmation back to channel - Auto-approve support (blanket + per-tool list) Atomic session resume: - resume_session field on CreateWorkstreamMessage for single-request resume - Server resumes session during POST /v1/api/workstreams/new atomically - Bridge emits SessionResumedEvent to per-workstream channel - WorkstreamCreatedEvent extended with resumed/session_id/message_count - Server UI dashboardResumeSession simplified to single request - Pruned sessions fall back gracefully to fresh start Service auth: - Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET - Bridge: approve scope (1 week). Console collector: read. Proxy: write. Console admin: - Channels tab with per-user view, force-link modal, unlink - 3 admin API endpoints for channel user management - Styled confirm modals replacing browser confirm() dialogs Bug fixes: - AsyncRedisBroker: replaced per-channel listener tasks with single dispatch loop + per-channel queue workers (fixes message stealing race) - Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates - Bridge: _active_sends tracked for initial messages (fixes missing TurnCompleteEvent and unfinalized streaming messages) - Bridge: HTTP calls moved outside lock scope in approval handlers - Bridge: _handle_send cleans up _active_sends on HTTP/server errors - Formatter: reads server SSE format (func_name/preview) with fallback Docs, SDK, tests: - docs/channels.md setup guide, architecture diagram 16 - Updated api-reference.md, architecture.md, console.md, docker.md - Python SDK: resume_session param on create_workstream (async + sync) - TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces - OpenAPI schema: resume_session request, resumed/message_count response - 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing, 18 discord, 12 resume flow) — 1120 total passing * Fix CI lint/typecheck failures and address Copilot review feedback (#24) Lint: fix import ordering, remove unused imports, use contextlib.suppress. Mypy: explicit postgresql dialect import, add discord module overrides for optional-dependency CI environments. Copilot: fix double-escaping in admin confirm modals, return resolved session_id from server resume response, fix channel_routes diagram schema, use atomic setdefault for routing locks, add post-insert race guard in admin channel create, support SSE format in auto-approve check, update identity linking note in architecture diagram. * Fix remaining mypy call-arg errors for discord.py optional dependency Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class definitions that fail when discord.py is not installed in CI. |
||
|
Patrick Buckley
|
047680d669 |
Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23) JWT-based authentication with three token types: config-file (hmac, backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs (HS256, 24h expiry). Username:password login via bcrypt. Hierarchical scopes: read < write < approve. New tables: users (username, password_hash), api_tokens (token_hash, scopes, expires), channel_users (future channel integrations). user_id column added to sessions and workstreams for attribution. Console owns admin CRUD (6 endpoints under /api/admin/). Server validates JWTs locally with shared signing secret. Public /api/auth/setup endpoint for first-time admin creation (atomic, only works with zero users). turnstone-admin CLI for user/token management. Admin console UI: Users and Tokens tabs with full CRUD modals, scope badges, token show-once with clipboard copy, keyboard accessibility (focus traps, Escape, arrow key tabs, ARIA roles). Login UI redesigned: username:password primary, token toggle for legacy, setup wizard auto-detected via /api/auth/status. Python + TypeScript SDKs updated with login(username, password), authStatus(), setup(). New docs/security.md + diagram 15-auth-architecture.puml. All existing docs updated. OpenAPI specs include all new endpoints. 64 new tests (1023 total). Dependencies: PyJWT, bcrypt. * Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review Address Copilot review feedback: escape double quotes in escapeHtml() to prevent XSS in HTML attributes, add JWT validation fallback so config tokens containing dots still work, add user_id to AuthLoginResponse schema, return created field from admin_create_user, and correct five documentation files to match actual API behavior. |
||
|
Patrick Buckley
|
62a4ceac96 |
Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec
All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.
New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI
Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
(PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
EXEMPT_PATHS
Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).
42 new tests (830 total). All frontend JS, docs, and diagrams updated.
* Fix mypy type errors in turnstone/api/ package
- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules
* Address PR #18 review feedback + fix mypy errors
Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
(was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
parameter for air-gapped deployments
Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
|
||
|
Patrick Buckley
|
29c00c0cdf |
Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/ The server UI and console UI had ~60% CSS overlap and significant JS duplication. Extract shared assets into a new turnstone/shared_static/ package mounted at /shared/ in both servers: - base.css: design tokens, reset, typography, login/toast/kb overlays, dashboard table, state dots, health bar, scrollbar, reduced motion - auth.js: authFetch, login overlay with focus trap, logout (hooks for page-specific post-login/logout callbacks) - theme.js: dark/light toggle with system preference detection - toast.js: notification queue with configurable timeout - utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount - kb.js: keyboard shortcuts overlay with configurable content, focus management, and focus restore on dismiss Console proxy updated: JS shim injection moved from proxy_static (app.js prepend) to proxy_index (inline <script> in HTML) so it runs before any external scripts. New /shared/ path rewriting and proxy_shared_static route added. ~1540 lines removed from page-specific files, 775 lines in shared package. 13 new tests (788 total). * Fix /shared/ auth and remove __init__.py from shared_static Address PR #17 review feedback: 1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS loads before authentication (required for login overlay to render) 2. Remove turnstone/shared_static/__init__.py to prevent exposing Python package internals (__init__.py, __pycache__) via the StaticFiles mount. Not needed for packaging since pyproject.toml uses explicit glob includes. 3 new auth tests for /shared/ public path access. |
||
|
Patrick Buckley
|
b6e0f0fcca |
Add node version tracking and drift detection to console dashboard (#16)
* Add node version tracking and drift detection to console dashboard Surface the version field from each node's /health endpoint in the console dashboard. Collector extracts version into get_overview() (version_drift + versions fields), promotes it to top-level in get_nodes(), and adds get_version_info() for per-node detail. Console /health endpoint includes drift fields. Frontend adds a VER column to the 7-column node table grid, shows per-node version strings, tracks versions per group with "mixed" + yellow drift badge when nodes disagree, and displays a DRIFT warning or single version in the status bar. Column hidden on mobile (<700px). ARIA labels include version info for accessibility. 10 new tests (745 total). Docs and diagram updated. * Fix drift tooltip text: show 'Versions detected' not 'Nodes running' |
||
|
Patrick Buckley
|
6c5441435b |
Add console workstream creation + server reverse proxy (#14)
* Add console workstream creation + server reverse proxy (#14) Enable the console dashboard to create workstreams and proxy server UIs, so users only need network access to the console port. Workstream creation via MQ: - POST /api/cluster/workstreams/new with three targeting modes: specific node (directed queue), auto (best node by capacity), or general pool (shared queue, any bridge picks up) - Console pushes CreateWorkstreamMessage to Redis; bridge handles the rest (server creation, ownership registration, SSE events) Reverse proxy for server UIs: - /node/{node_id}/ serves the server's HTML with static path rewriting and a console-return banner injected after <body> - JS proxy shim prepended to app.js overrides fetch() and EventSource() to route root-relative URLs through /node/{id}/api/... - SSE streams proxied via httpx.AsyncClient(timeout=None) with per- connection clients for long-lived streams - GET/POST API requests forwarded with body and auth token Security: - Proxy write paths checked against WRITE_PATHS to prevent read-token escalation (read tokens cannot POST /api/send through proxy) - html.escape() on node_id in banner HTML to prevent XSS - String length limits on name/model inputs Frontend: - "+ new" button in header opens creation modal with node dropdown (Auto / General pool / specific nodes with capacity display) - Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling - Workstream rows and node links deep-link via proxy paths - Custom select arrow, Instrument Panel modal styling Documentation: - docs/console.md rewritten with proxy and creation API docs - docs/architecture.md console section updated - PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered - README.md updated 28 new tests (741 total), ruff + mypy clean. * Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness - Normalize trailing slashes in required_role() to prevent write-role bypass via /api/send/ or /node/{id}/api/send/ (auth.py) - Validate node_id format in proxy handlers (alphanumeric, dot, dash, underscore only) to prevent injection vectors - Use json.dumps() for JS proxy shim prefix to prevent script injection - URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static) - Check upstream status in _proxy_sse() — emit error event on non-200 instead of keeping a dead SSE connection open - Check upstream status in proxy_index() — propagate non-2xx errors - Forward query string in _proxy_post() (consistency with _proxy_get) - Handle JSON null values in create_workstream() — treat null as empty, reject non-string types with 400 - Fix docs/diagram LPUSH → RPUSH to match actual broker implementation |
||
|
Patrick Buckley
|
b6c1c3a676 |
Add console-to-server deep linking via ?ws_id= query parameter (#7)
* Add console-to-server deep linking via ?ws_id= query parameter Server UI parses ?ws_id= on load (both direct init and post-login) and auto-selects the matching workstream instead of showing the dashboard. URL is cleaned from the address bar via history.replaceState after navigation. Defers initial SSE connection to avoid redundant connect when deep-linking switches tabs immediately. Console workstream rows are now clickable — opens the node's server UI in a new tab with ?ws_id= targeting that workstream. Uses URL constructor for safe URL building. External-link indicator (↗) appears on hover. Rows without server_url have role/tabindex removed to avoid broken affordance. currentServerUrl reset on showOverview() to prevent stale fallback across views. Collector injects server_url into workstream dicts in both the poll path and ws_created event path so deep links work immediately. * Fix Copilot PR #7 review: deep-link duplicate history entry, server_url test coverage - Suppress history.pushState in switchTab() during deep-link navigation by setting _historyNavigation=true around both call sites (post-login and direct init). Fixes Back button appearing to do nothing on first press. - Add server_url assertions to poll and ws_created collector tests to prevent regressions of deep-link functionality. |
||
|
Patrick Buckley
|
9be155b97a |
Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup - Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files - Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow - Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems - Refresh README and docs with badges, diagram links, and current descriptions - Refactor test_server_live.py with mock streaming helpers for deterministic CI testing - Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2) Console dashboard: - Move state indicators from top cards to fixed bottom status bar with cluster metrics - Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000) - Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent, LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance - Add render cache, stale indicator, active filter highlight, loading states Server web UI: - Apply matching Instrument Panel aesthetic for visual consistency with console - Fix branding (pcode → turnstone), extract inline styles to CSS classes - Rename pcode localStorage keys and history state to turnstone Legacy cleanup: - Remove persona-model-specific --persona flag and /persona slash command - Remove model_identity from chat_template_kwargs (vLLM-specific mechanism) - Refactor plan agent to use standard developer message instead of model_identity - Remove dead code (unused date/has_tools variables, noqa suppressions) * Fix CI typecheck: add mypy overrides for optional sympy/numpy imports The math sandbox optionally imports sympy and numpy at runtime (try/except ImportError). In CI these packages are not installed, so mypy raises import-not-found rather than import-untyped. Add mypy overrides to ignore missing imports for these optional dependencies. * Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity - Change #node-table from role="tree" to role="list" and group elements from role="treeitem" to role="listitem" (proper ARIA semantics) - Include currentView and currentFilter.state in renderStatusBar cache key so active pill highlight updates when switching views - Align pulse animation to 0.35 opacity (already applied in CSS) |
||
|
Patrick Buckley
|
0d6252dd7d | Initial commit — turnstone multi-node AI orchestration platform. |