mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-27 22:34:51 -06:00
fa8cd4bb0b41feb7597e5ff3cd950992ac505d30
19 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
58c81b2b46 | fix: resolve CodeQL double-import findings in test files (#331) | ||
|
Patrick Buckley
|
66c856eb6e |
fix: post-merge follow-ups for PRs #312-#316 (#319)
Security: - Add write scope rules for 4 new workstream POST endpoints (delete, open, refresh-title, title) in required_scope() — both direct and console-proxied paths Judge: - Restore cancel_event check in inner poll loop (was removed) - Fix fallback delivery off-by-one: items[idx+1:] not items[idx:] - Skip empty-response retry when finish_reason=="length" - Reset empty_retries counter after non-empty response - Document per-turn timeout semantics in JudgeConfig Google provider: - Add default base_url for Gemini endpoint in create_client() - Bump max_output_tokens 8192→65536, set token_param="max_tokens" - Add api_key detection for googleapis.com in console detect - Add provider badge CSS (green) and openai-compatible (dim) Theme: - Fix POST→PUT for settings persistence (was silently 405-ing) - Consolidate dual localStorage keys with backwards-compat read - Lower banner z-index 9999→200, raise login overlay to 10001 - Fix undefined --bg-input, banner contrast for WCAG AA - Add smooth theme transition with prefers-reduced-motion override - Console onThemeChange: add title + aria-label updates Workstream backend: - Restore close_workstream 400 for last-ws case (was changed to 404) - Thread-safe _llm_verdicts via _ws_lock on all mutation sites - Fork: persist tool_calls + provider_data in save_message - Add get_workstream_metadata to StorageBackend protocol - Add ChatSession.request_title_refresh() public API - Use cs.stored_keys() instead of cs._cache - Redact exception text in delete 500 response - web_helpers: catch-all logs and returns 500 not 400 - Live-stream ws_created SSE includes title field Workstream UI: - Focus traps + Escape on edit-title and delete-ws modals - Tab close aria-label, mobile breakpoint for action buttons - Restore name priority (live SSE over stale API) - Fix double-delete, fork button text, batch delete handler leak - Optimistic title update, close-last-tab error toast - ws_id badge show-on-hover, hover states, aria-live, emoji a11y Console admin: - Banner aria-labels, judge dropdown wording, detect button class - New-ws modal Escape handler, provider defaults cross-reference |
||
|
Patrick Buckley
|
7968f1b361 |
feat: auto-invalidate JWT and static assets on version upgrade (#307)
* feat: auto-invalidate JWT and static assets on version upgrade
Add a `ver` claim (major.minor) to user-facing JWTs so tokens from
previous versions are rejected after upgrade, triggering re-login.
Service tokens are excluded for rolling-deployment safety. Tokens
without a `ver` claim (pre-upgrade) are accepted for backward compat.
Inject `?v={__version__}` query strings into static asset URLs at
startup so browsers fetch fresh JS/CSS after any release. Vendored
libraries (KaTeX, Highlight.js, etc.) are skipped since they already
carry version numbers in directory paths. HTML responses now include
`Cache-Control: no-cache` to ensure browsers always revalidate.
Frontend detects upgrade-specific 401s and shows a contextual subtitle
("The server was updated — please sign in again"), then performs a full
page reload after re-auth to load the new versioned assets.
* refactor: address PR review — public API name, single decode, idempotent regex
Rename _version_slot() → jwt_version_slot() to make the cross-module
import explicit rather than relying on a private name.
Move version gating from validate_jwt() into check_request() via a new
AuthResult.token_version field. This eliminates the double JWT decode
that occurred on version-mismatch detection — the token is now decoded
once and the version compared afterward.
Guard version_html() regex against double-apply by excluding URLs that
already contain a query string ([^"?]+ instead of [^"]+).
* feat: structured version_mismatch code, ETag, cross-tab auth sync
Add structured "code": "version_mismatch" field to the 401 response
so the frontend detects upgrade-triggered re-auth without string
matching on the error message.
Add ETag headers to HTML index responses (server, console, and proxied
node UI). Combined with Cache-Control: no-cache, browsers send
conditional GETs and receive 304 between upgrades, saving bandwidth.
Add BroadcastChannel-based cross-tab auth sync so logging in on one
tab dismisses the login modal on all other tabs (and vice-versa for
logout).
Add a reminder to the vendored JS update script about the
version_html() regex lookahead.
* fix: remove unused import in test_web_helpers
|
||
|
Patrick Buckley
|
62d2a0fe6a |
fix: remove non-auth support from bootstrap wizard (#274)
* fix: remove non-auth support from bootstrap wizard Auth is now mandatory for all deployments. Remove the TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN required in the wizard's system prompt. * fix: remove auth disable support from runtime and infra Remove AuthConfig.enabled field — auth is always on. Drop TURNSTONE_AUTH_ENABLED env var, config toggle, and the check_request bypass. Update compose.yaml, Helm chart, Terraform, docs, and tests to match. * feat: deprecate config tokens, require JWT secret, prefer JWT auth Phase 1 of config-token removal: - load_jwt_secret() now exits with error if no secret is configured (was: silently auto-generated ephemeral secret) - _authenticate_token() logs deprecation warning on config token use - CLI /cluster commands use ServiceTokenManager when JWT secret is set - turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set - Update bootstrap wizard, docker.md, security.md to mark TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required - Console test fixtures use auth token + headers (auth always enforced) * feat: add service scope for inter-service JWT auth Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens bypass require_permission() RBAC checks, replacing the old empty-user-id bypass that config tokens relied on. All ServiceTokenManager instances that need admin access now include "service" in their scopes (console proxy, channel gateway, CLI, admin CLI). Read-only services (collector, notification) unchanged. * feat: phase 2 config token deprecation - SDK doc examples now show API tokens (ts_) instead of config tokens - Remove _get_config_token() from admin CLI (dead code) - Block config token exchange in handle_auth_login — only password and API token login allowed - Update login tests to use password-based auth instead of config token exchange * feat: phase 3 — remove config tokens entirely Complete removal of config-file token authentication: - Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch branch, and config token loading from load_auth_config() - Remove auth_config parameter from _authenticate_token() and check_request() — callers updated throughout - Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts, Terraform, turnstone.example.toml - Remove --auth-token CLI flags from turnstone, turnstone-admin, and turnstone-console - Simplify console main() — always use ServiceTokenManager (no fallback to static tokens) - Delete config-token-specific tests, rewrite check_request and integration tests to use JWT auth with proper audience claims - Remove all config token references from docs (security.md, docker.md, sdk.md, console.md, architecture.md, bootstrap prompt) * fix: address code review findings - Fix 33 broken tests: add JWT auth to test_api_versioning, test_console_routing_proxy, test_tls_admin, test_tls_manager, test_server_live (jwt_secret + audience-scoped auth headers) - Add TestRequirePermissionServiceScope: 4 tests covering the service scope RBAC bypass path - Remove stale comments referencing config tokens in auth.py and console/server.py - Remove dead proxy_auth_token parameter from console create_app() and static token fallback in _proxy_auth_headers() - Remove TURNSTONE_AUTH_TOKEN from env.py scrub list * fix: address Copilot review — JWT audience, compose require secret - CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager (console validates audience, JWTs without it were rejected) - Admin CLI tls-list: same audience fix - compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset - SDK console: fix default port from 8081 to 8090 * test: add auth enforcement tests for TLS admin endpoints 5 new tests: unauthenticated requests return 401 (list, renew, delete), read-only-scoped requests return 403 (renew, delete). Closes the TLS auth enforcement test gap noted in PROGRESS.md. * fix: address remaining Copilot review feedback - Fix token_source="config" → "test" in TLS test fixtures - Fix AuthResult.token_source docstring to include service origins - Require TURNSTONE_JWT_SECRET in cluster compose profile (:?) - Helm: add auth.jwtSecret + auth.existingSecret values, wire TURNSTONE_JWT_SECRET into secret.yaml and both deployments - Terraform: replace auth_token with jwt_secret variable + secret, remove orphaned auth_token resources and IAM reference - Remove [[auth.tokens]] from security.md config example * fix: address full code review — 10 findings Critical: - Terraform: replace concat(common_env, auth_env) with common_env (auth_env local was removed but still referenced) - Channel gateway: remove hmac static token auth from _check_auth(), use JWT-only validation. Remove --auth-token CLI arg from channel - Rebalancer: add token_manager support so migration requests carry JWT auth (was sending unauthenticated POST to /internal/migrate) Major: - Guard _permissions_to_scopes() against "service" privilege escalation from DB role permissions - Remove dead AuthConfig class, load_auth_config(), and all auth_config parameters from create_app() signatures - Helm: inject JWT secret for both inline and existingSecret paths Minor: - Remove dead auth_token param from ClusterCollector - Remove empty TestLoadAuthConfig class - Short JWT secret now exits instead of warning - Compose: add generation command comment above JWT_SECRET - Clean stale config token references from 6 doc files - Clean stale AUTH_TOKEN reference from bootstrap wizard prompt * fix: remove remaining stale config token references from docs - channels.md: remove --auth-token from options table - oidc.md: remove "config-file tokens still work" claim - security.md: remove config token section, fix JWT secret docs (now required/exits, no ephemeral fallback), remove hmac from ASCII diagram, remove --auth-token reference |
||
|
Patrick Buckley
|
2bb55590bf |
feat: replace Redis MQ with direct HTTP transport (Phase 1)
Delete the entire turnstone/mq/ package (broker, bridge, protocol, client) and turnstone/sim/ package. Remove Redis as a dependency. Channel gateway and console now communicate with server nodes via direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues. Single-node deployments work with zero infrastructure beyond the database. Key changes: - Channel adapters use httpx POST for create/send/approve/close and httpx-sse for per-workstream event streaming - Console collector discovers nodes via services table instead of Redis SCAN - Console scheduler dispatches tasks via HTTP POST with DB-based leader election - Server registers in services table with 30s heartbeat - Server accepts optional ws_id in create request (for Phase 2 console-generated routing) - SDK events gain IntentVerdictEvent and OutputWarningEvent types - All docs, examples, bootstrap wizard updated 63 files changed, -5968 net lines (Redis transport fully removed) |
||
|
Patrick Buckley
|
037308f3b1 |
fix: propagate user identity through console proxy
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
|
||
|
Patrick Buckley
|
ec3454ee2e |
fix: wire resume_ws through console + expose max_ws in heartbeat (#124)
* fix: wire resume_ws through console + expose max_ws in heartbeat Console create_workstream handler now reads resume_ws from the request body and passes it to CreateWorkstreamMessage on all three dispatch paths (pool, auto, explicit). Previously resume only worked via channel router and direct CLI — the console layer never plumbed it through. Server /health now includes max_ws from WorkstreamManager. Bridge reads it on startup and includes it in heartbeat metadata so the console's _pick_best_node gets accurate capacity instead of always defaulting to 10. Collector also updates max_ws on subsequent heartbeats (not just discovery). Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks fixed for new max_workstreams property access in /health. * fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id Add _fetch_server_metadata() so bridge reads max_ws from /health even when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats would advertise max_ws=10 regardless of actual server config. Add 3 test cases verifying resume_ws flows through all three console dispatch paths (directed, pool, auto-select). |
||
|
Patrick Buckley
|
2e95f2ac73 |
test: add scope coverage for internal MCP/config reload endpoints (#75)
* test: add scope coverage for internal MCP/config reload endpoints Verify required_scope() returns "approve" for _internal endpoints across all access patterns (bare, /v1/-prefixed, console proxy with and without /v1/), plus a GET negative test confirming only POST is elevated. Closes the "internal endpoints accept read scope" item in PROGRESS.md — the endpoints were already in APPROVE_PATHS. * test: add config-reload v1/proxy scope tests per review feedback Add /v1/-prefixed and console proxy variants for config-reload to match the mcp-reload coverage, as flagged by Copilot review. |
||
|
Patrick Buckley
|
20df7b3034 |
feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71)
* feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping Add OpenID Connect as a fourth authentication method, enabling single sign-on via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars (TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET). Security: - Authorization Code Flow with PKCE (S256) - State/nonce parameters with database-backed pending store (multi-node safe) - JWKS signature validation with async fetch + key rotation retry - Algorithm allowlist from JWKS key (not token header) prevents confusion - Identity matching exclusively by (issuer, sub) — prevents account takeover - password_enabled=false enforced server-side, not just UI - Rate limiting on both authorize and callback endpoints - OIDC users get "!oidc" password sentinel (bcrypt rejects naturally) - ID token validated for iss, aud, exp, nonce Features: - Auto-provisioning with username deduplication on first login - Claim-based role mapping with IdP demotion propagation (revokes stale roles) - "Continue with [Provider]" SSO button on login page - OIDC-only mode hides password form - Setup wizard required before OIDC login (admin bootstrap) Storage: migration 018 (oidc_identities + oidc_pending_states tables), 8 new protocol methods on both SQLite and PostgreSQL backends. 66 new tests (2273 total). * fix: address PR #71 review feedback (18 items) Bugs fixed: - OIDC success redirect now fetches permissions via new /auth/whoami endpoint before completing login (fixes permission-gating in UI) - Remove double decodeURIComponent on oidc_error (URLSearchParams already decodes; extra call throws on stray %) - Authorize rate limiter returns redirect instead of JSON 429 (endpoint reached via browser navigation, not fetch) - Lazy JWKS fetch in callback when startup discovery failed (IdP recovery without restart) - Startup exception handlers now log with exc_info=True - PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for true atomicity (eliminates TOCTOU) Behavior: - New OIDC users without role mapping get builtin-viewer by default (assigned_by="oidc-default", not revoked by role sync) Documentation fixes: - Role mapping: sync semantics (add + revoke stale), not "additive only" - PASSWORD_ENABLED=false blocks ALL password logins including admin - Algorithm: asymmetric allowlist, not per-key derivation - PlantUML diagram updated for role revocation API spec fixes: - Removed error_codes=[302] from callback (302 is success redirect) - Added /auth/whoami to both server + console specs - Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths) * fix: address PR #71 round 2 review feedback (10 items) Rate limiting: - Authorize endpoint now calls record() after check() so the rate limiter actually counts attempts (was a no-op before) OIDC resilience: - Split startup try/except: discovery failure disables OIDC, JWKS prefetch failure leaves OIDC enabled for lazy retry on first login - JWKS unavailable message changed to "temporarily unavailable" (was misleadingly "not configured") - create_oidc_pending_state raises on collision instead of OR IGNORE (prevents silent insert drop on state collision) - SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock (eliminates TOCTOU race) Frontend: - OIDC error display deferred 300ms so showLogin()'s async status fetch doesn't clear it via _switchMode → _clearError API spec: - OIDC authorize/callback endpoints now declare response_code=302 - Added AuthWhoamiResponse Pydantic model for /auth/whoami - Regenerated TypeScript SDK OpenAPI snapshots Documentation: - Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly") - Added TODO(tech-debt) comments on Host header redirect_uri sites |
||
|
Patrick Buckley
|
67f43a7ee0 |
feat: [memory] REST API endpoints + SDK methods + docs (#56)
* feat: [memory] REST API endpoints + SDK methods + docs
Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope
Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit
Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.
Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.
TypeScript SDK: matching methods + types on both clients.
Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.
Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.
Also fixes stray `total: int` on CreateChannelUserRequest.
* fix: [memory] address PR review — cross-user scope, schema types, snapshots
Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity. Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.
Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.
Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.
6 new security tests for user-scope access control.
|
||
|
Patrick Buckley
|
fb190f8977 |
Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity Eliminate the separate session_id concept. The workstream ID (ws_id) is now the single identity used for both real-time routing and conversation persistence, removing a layer of indirection that was 1:1 in practice and buggy on resume (stale pointers, orphaned rows). Schema changes (migration 006): - Drop sessions table; add alias/title columns to workstreams - Rename conversations.session_id → ws_id - Rename session_config table → workstream_config (ws_id column) - Data migration remaps existing conversations to ws_id Storage/API renames: - register_session → register_workstream (already existed, merged) - save_message/load_messages now keyed by ws_id - resolve_session → resolve_workstream - ChatSession.session_id property → ws_id - ChatSession.resume_session() → resume() - resume_session field → resume_ws - SessionResumedEvent → WorkstreamResumedEvent - /api/sessions → /api/workstreams/saved - /sessions slash command → /workstreams - --session-retention-days → --retention-days Channel eviction recovery simplified: reuses old ws_id directly instead of get_session_id_by_ws() reverse lookup. * Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec - docs/channels.md: "resumes the session" → "resumes the workstream", "Session resumed:" → "Resumed:", "old session was pruned" → "old workstream was pruned" - docs/api-reference.md: "Each session object" → "Each saved workstream object", field descriptions updated, removed stale node_id field - sdk/typescript/openapi-server.json: fully regenerated from Python models — removes all stale session_id properties from WorkstreamInfo, DashboardWorkstream, CreateWorkstreamResponse schemas |
||
|
Patrick Buckley
|
872e1770e6 |
Feature/code dedup (#25)
* Add JWT auth security hardening (6 fixes) - Secure cookie flag: make_set_cookie defaults Secure=True, max_age=24h - Login brute-force protection: LoginRateLimiter (5 attempts/5min per key) - JWT aud/iss claims: create_jwt/validate_jwt support audience validation - Service JWT auto-rotation: ServiceTokenManager with 1h expiry, 80% refresh - CORS restriction: configurable via TURNSTONE_CORS_ORIGINS env var - JWT secret strength: warning on secrets shorter than 32 chars - Hard fail for bridge/console when TURNSTONE_JWT_SECRET is missing * Refactor duplicated code into shared utilities and fix 3 UI bugs Code deduplication (~235 net lines removed): - Extract AuthMiddleware + 4 auth endpoint handlers to core/auth.py - Create core/web_helpers.py (require_storage_or_503, read_json_or_400, parse_cors_origins, cors_middleware) - Extract add_redis_args/broker_from_args to mq/broker.py - Extract add_log_args/configure_logging_from_args to core/log.py - Remove dead _CSS/_JS loads, duplicate states dict, _read_json helper, unused required_role(), duplicate detect_model() wrapper Bug fixes: - Fix console proxy forwarding user's JWT_AUD_CONSOLE token to server nodes (use ServiceTokenManager with JWT_AUD_SERVER instead) - Fix login form autofill: wrap inputs in <form>, add name attributes, set type=submit on button - Fix SSE reconnecting flash: add onopen handler to clear status immediately on connection (not waiting for first message) - Fix chat scroll: add min-height:0 to flex containers, overflow:hidden on body to constrain viewport height * Address CI typecheck failure and Copilot review feedback - Fix mypy arg-type: use Any for jwt.decode options (PyJWT stubs vary) - Bridge SSE loops: use event_hooks for auth header refresh on reconnect instead of static headers that go stale after token rotation - Login form: remove javascript:void(0) action (CSP anti-pattern) - Use JWT_AUD_SERVER/JWT_AUD_CONSOLE constants instead of string literals in middleware builder calls to prevent drift |
||
|
Patrick Buckley
|
047680d669 |
Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23) JWT-based authentication with three token types: config-file (hmac, backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs (HS256, 24h expiry). Username:password login via bcrypt. Hierarchical scopes: read < write < approve. New tables: users (username, password_hash), api_tokens (token_hash, scopes, expires), channel_users (future channel integrations). user_id column added to sessions and workstreams for attribution. Console owns admin CRUD (6 endpoints under /api/admin/). Server validates JWTs locally with shared signing secret. Public /api/auth/setup endpoint for first-time admin creation (atomic, only works with zero users). turnstone-admin CLI for user/token management. Admin console UI: Users and Tokens tabs with full CRUD modals, scope badges, token show-once with clipboard copy, keyboard accessibility (focus traps, Escape, arrow key tabs, ARIA roles). Login UI redesigned: username:password primary, token toggle for legacy, setup wizard auto-detected via /api/auth/status. Python + TypeScript SDKs updated with login(username, password), authStatus(), setup(). New docs/security.md + diagram 15-auth-architecture.puml. All existing docs updated. OpenAPI specs include all new endpoints. 64 new tests (1023 total). Dependencies: PyJWT, bcrypt. * Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review Address Copilot review feedback: escape double quotes in escapeHtml() to prevent XSS in HTML attributes, add JWT validation fallback so config tokens containing dots still work, add user_id to AuthLoginResponse schema, return created field from admin_create_user, and correct five documentation files to match actual API behavior. |
||
|
Patrick Buckley
|
62a4ceac96 |
Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec
All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.
New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI
Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
(PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
EXEMPT_PATHS
Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).
42 new tests (830 total). All frontend JS, docs, and diagrams updated.
* Fix mypy type errors in turnstone/api/ package
- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules
* Address PR #18 review feedback + fix mypy errors
Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
(was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
parameter for air-gapped deployments
Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
|
||
|
Patrick Buckley
|
29c00c0cdf |
Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/ The server UI and console UI had ~60% CSS overlap and significant JS duplication. Extract shared assets into a new turnstone/shared_static/ package mounted at /shared/ in both servers: - base.css: design tokens, reset, typography, login/toast/kb overlays, dashboard table, state dots, health bar, scrollbar, reduced motion - auth.js: authFetch, login overlay with focus trap, logout (hooks for page-specific post-login/logout callbacks) - theme.js: dark/light toggle with system preference detection - toast.js: notification queue with configurable timeout - utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount - kb.js: keyboard shortcuts overlay with configurable content, focus management, and focus restore on dismiss Console proxy updated: JS shim injection moved from proxy_static (app.js prepend) to proxy_index (inline <script> in HTML) so it runs before any external scripts. New /shared/ path rewriting and proxy_shared_static route added. ~1540 lines removed from page-specific files, 775 lines in shared package. 13 new tests (788 total). * Fix /shared/ auth and remove __init__.py from shared_static Address PR #17 review feedback: 1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS loads before authentication (required for login overlay to render) 2. Remove turnstone/shared_static/__init__.py to prevent exposing Python package internals (__init__.py, __pycache__) via the StaticFiles mount. Not needed for packaging since pyproject.toml uses explicit glob includes. 3 new auth tests for /shared/ public path access. |
||
|
Patrick Buckley
|
6c5441435b |
Add console workstream creation + server reverse proxy (#14)
* Add console workstream creation + server reverse proxy (#14) Enable the console dashboard to create workstreams and proxy server UIs, so users only need network access to the console port. Workstream creation via MQ: - POST /api/cluster/workstreams/new with three targeting modes: specific node (directed queue), auto (best node by capacity), or general pool (shared queue, any bridge picks up) - Console pushes CreateWorkstreamMessage to Redis; bridge handles the rest (server creation, ownership registration, SSE events) Reverse proxy for server UIs: - /node/{node_id}/ serves the server's HTML with static path rewriting and a console-return banner injected after <body> - JS proxy shim prepended to app.js overrides fetch() and EventSource() to route root-relative URLs through /node/{id}/api/... - SSE streams proxied via httpx.AsyncClient(timeout=None) with per- connection clients for long-lived streams - GET/POST API requests forwarded with body and auth token Security: - Proxy write paths checked against WRITE_PATHS to prevent read-token escalation (read tokens cannot POST /api/send through proxy) - html.escape() on node_id in banner HTML to prevent XSS - String length limits on name/model inputs Frontend: - "+ new" button in header opens creation modal with node dropdown (Auto / General pool / specific nodes with capacity display) - Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling - Workstream rows and node links deep-link via proxy paths - Custom select arrow, Instrument Panel modal styling Documentation: - docs/console.md rewritten with proxy and creation API docs - docs/architecture.md console section updated - PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered - README.md updated 28 new tests (741 total), ruff + mypy clean. * Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness - Normalize trailing slashes in required_role() to prevent write-role bypass via /api/send/ or /node/{id}/api/send/ (auth.py) - Validate node_id format in proxy handlers (alphanumeric, dot, dash, underscore only) to prevent injection vectors - Use json.dumps() for JS proxy shim prefix to prevent script injection - URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static) - Check upstream status in _proxy_sse() — emit error event on non-200 instead of keeping a dead SSE connection open - Check upstream status in proxy_index() — propagate non-2xx errors - Forward query string in _proxy_post() (consistency with _proxy_get) - Handle JSON null values in create_workstream() — treat null as empty, reject non-string types with 400 - Fix docs/diagram LPUSH → RPUSH to match actual broker implementation |
||
|
Patrick Buckley
|
a1f00092f5 |
Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn (#11)
* Migrate HTTP servers from stdlib to Starlette/ASGI + uvicorn Replace Python stdlib http.server (ThreadedHTTPServer, BaseHTTPRequestHandler) with Starlette ASGI applications served by uvicorn across all three HTTP entry points. SSE endpoints use sse-starlette EventSourceResponse with async generators that bridge sync queue.Queue via run_in_executor(). Bridge SSE parser replaced with httpx-sse EventSource. - turnstone/server.py: Starlette app factory with create_app(), pure ASGI middleware (auth, rate limit, metrics, CORS), async route handlers, lifespan context manager for startup/shutdown. WebUI and ChatSession remain fully synchronous — worker threads unchanged. - turnstone/console/server.py: Same pattern, simpler (no ChatSession). Path params replace manual string slicing for node detail route. - turnstone/mq/bridge.py: _iter_sse_data() uses httpx_sse.EventSource instead of hand-rolled line parser. - Tests: All ThreadedHTTPServer fixtures replaced with starlette.testclient.TestClient via create_app() factories. - Docs: Updated architecture.md, api-reference.md, README.md, and PlantUML diagrams (03, 11) + regenerated PNGs. * Fix Copilot PR #11 review: TestClient cleanup, JSON error handling, SSE timeout - Close TestClient in teardown for TestConsoleAuth and TestConsoleLogin to avoid lifespan/resource leaks - Close TestClient via yield/finally in TestConsoleHTTPEndpoints fixture - Add _read_json() helper for safe JSON body parsing (returns {} on invalid JSON instead of 500, matching old stdlib handler behavior) - Apply same try/except pattern to console auth_login endpoint - Increase SSE queue.get timeout from 1s to 5s to align with sse-starlette ping interval, reducing executor task churn |
||
|
Patrick Buckley
|
9be155b97a |
Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup - Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files - Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow - Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems - Refresh README and docs with badges, diagram links, and current descriptions - Refactor test_server_live.py with mock streaming helpers for deterministic CI testing - Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2) Console dashboard: - Move state indicators from top cards to fixed bottom status bar with cluster metrics - Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000) - Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent, LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance - Add render cache, stale indicator, active filter highlight, loading states Server web UI: - Apply matching Instrument Panel aesthetic for visual consistency with console - Fix branding (pcode → turnstone), extract inline styles to CSS classes - Rename pcode localStorage keys and history state to turnstone Legacy cleanup: - Remove persona-model-specific --persona flag and /persona slash command - Remove model_identity from chat_template_kwargs (vLLM-specific mechanism) - Refactor plan agent to use standard developer message instead of model_identity - Remove dead code (unused date/has_tools variables, noqa suppressions) * Fix CI typecheck: add mypy overrides for optional sympy/numpy imports The math sandbox optionally imports sympy and numpy at runtime (try/except ImportError). In CI these packages are not installed, so mypy raises import-not-found rather than import-untyped. Add mypy overrides to ignore missing imports for these optional dependencies. * Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity - Change #node-table from role="tree" to role="list" and group elements from role="treeitem" to role="listitem" (proper ARIA semantics) - Include currentView and currentFilter.state in renderStatusBar cache key so active pill highlight updates when switching views - Align pulse animation to 0.35 opacity (already applied in CSS) |
||
|
Patrick Buckley
|
0d6252dd7d | Initial commit — turnstone multi-node AI orchestration platform. |