28 Commits

Author SHA1 Message Date
Patrick Buckley 998271b016 feat: convert large pastes to attachments 2026-08-12 00:27:38 -07:00
Patrick Buckley 98e96ab5f3 Add per-alias model concurrency admission (#990)
* feat(models): add per-alias concurrency admission

Add registry-backed FIFO admission limits with queue-aware deadlines and full-stream leases. Expose max_concurrency through storage, admin configuration, OpenAPI, documentation, and diagrams, with role and live backend count coverage.

* fix(api): omit null concurrency schema default

Keep max_concurrency optional for presence-keyed updates without advertising a null default for its non-null integer OpenAPI shape.
2026-08-08 22:01:04 -07:00
Patrick Buckley 7a06f5e8bc refactor(session): make ModelLane the provider boundary (#979) (#989)
* refactor(session): make ModelLane the provider boundary (#979)

## Summary

This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot.

- Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding.
- Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call.
- Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references.
- Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results.
- Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts.

## Validation

- SQLite suite: 11,188 passed, 9 skipped, 10 deselected
- PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected
- Live backend: 3 passed
- SSE recovery: 6 passed; browser recovery harness passed all scenarios
- Ruff: clean; 595 files correctly formatted
- mypy: 243 source files clean
- TypeScript: typecheck/build and 35 tests passed
- OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte
- `git diff --check` and Git LFS integrity clean

Closes #979.

* fix(deps): update nanoid for GHSA-2v37-7h3g-55p8

Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation.

Validation:
- npm ci
- npm audit --audit-level=moderate: 0 vulnerabilities
- TypeScript typecheck and build
- TypeScript tests: 35 passed

* fix(test): assert canonical model registry URLs

Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation.

Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy.
2026-08-08 16:13:35 -07:00
Patrick Buckley b65e5cae0e docs(personas): accuracy sweep — spec models, protocol contracts, page corrections
Spec models now describe what the endpoints do: ListPersonasResponse
declares the tool_inventory the shelf depends on, both console create
models declare persona, CreatePersonaRequest declares org_id, and
UpdatePersonaRequest documents the null-vs-absent split (null clears
base_prompt/tool_allowlist, null on flags/kinds is ignored). Console
OpenAPI regenerated.

Protocol contracts match the implementations: update_persona's return
covers the no-op case, create_persona's raises-list is complete, and
both extended row-shape docstrings gain their tail columns plus the
append-only rule. The workstreams.persona comments say slug, not
display name.

Page corrections from the docs review: personas.md documents the
creative_mode-to-writer migration conversion, the mid-session /resume
MCP-lever behavior, visibility-based nudge gating, the soft-set
prompt-cache cost, and the executive tool list — and drops internal
jargon. The changelog entry moves under [Unreleased] with the house
breaking-marker style and the auto-conversion note. coordinator-skills
and the API tour stop using persona to mean framing; governance,
api-reference, sdk, console, tools, and memory pick up the new
permission family, endpoints, kwargs, picker, and lever caveats.
2026-07-03 00:29:26 -07:00
Patrick Buckley ad0e7ce6eb docs: mark 1.5.0 legacy URL surface removal
CHANGELOG [Unreleased] / Removed (BREAKING — 1.5.0) block calling out
the legacy URL family removal with the swap table. Doc passes on
api-reference.md (per-endpoint sections rewritten with path
parameters and slimmer body shapes), architecture.md (handler-list
diagram and console-proxy URL example), console.md (URL-rewriting
JS shim docstring + SSE proxy example), and the two PlantUML
diagrams (11-console-data-flow, 16-channel-architecture).

Also picks up two test-side stragglers from step 5 that referenced
the legacy adapters in a docstring + a stale /v1/api/events SSE
test: turn into path-keyed equivalents. OpenAPI JSON dump regenerated
to reflect the catalog edits from step 3.

After this commit:
- 4557 tests passing under -m "not live"
- ruff + mypy clean on turnstone/ tests/ sdk/
- grep for "/v1/api/send", "/v1/api/approve", "/v1/api/cancel",
  "/v1/api/workstreams/close" returns zero hits across turnstone/
  sdk/ docs/ tests/ (excluding CHANGELOG.md, which intentionally
  documents the old shape).
- grep for make_legacy_body_keyed_adapter, make_legacy_query_keyed_adapter,
  _make_method_dispatch, close_legacy returns zero hits.
2026-04-26 22:14:22 -07:00
Patrick Buckley 471d1a3311 docs: audit documentation for 1.4 / 1.5 state
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams.  Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.

User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
  gateway entry reflects shipped Discord + Slack adapters instead of
  "Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
  wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
  we actually expect before push.

Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
  turnstone-bootstrap added to entry-points table, OpenAI provider
  file split (chat/responses/common) documented, 38 SDK event
  dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
  adapter + multi-adapter gateway, plan_agent/task_agent naming,
  governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
  DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
  multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
  scopes, tokens), Slack CLI/env reference in config table, combined-
  adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
  descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
  includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
  example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
  promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
  overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
  Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
  AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
  section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
  pointer redirected to architecture.md.

Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
  and bootstrap, add slack/bot.py, rename channels/gateway.py →
  channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
  SlackBot class and the slack-bolt Socket Mode edges; wire the new
  bot into ChannelService.  PNGs regenerated from both puml sources.
2026-04-16 16:01:24 -07:00
Patrick Buckley a3140da3a5 docs: update documentation for PRs #312-#316 (#324)
- README: add Google Gemini to multi-provider feature list and requirements
- architecture.md: add GoogleProvider, update supported provider values,
  file listing, config example
- judge.md: document cancel_on_approval, fresh-client lifecycle, fallback
  delivery, Google compatibility
- settings.md: add judge.cancel_on_approval, new interface.* section
  (close_tab_action, theme), update total count
- api-reference.md: document 6 new workstream/settings endpoints,
  add judge_model to workstreams/new
- console.md: add judge model to modal fields, add keyboard shortcuts
- console_schemas.py: add judge_model field to ConsoleCreateWsRequest
- server_spec.py: add 6 new EndpointSpec entries
- diagrams: add GoogleProvider to package structure and class diagram
2026-04-06 03:43:12 -07:00
Patrick Buckley 62d2a0fe6a fix: remove non-auth support from bootstrap wizard (#274)
* fix: remove non-auth support from bootstrap wizard

Auth is now mandatory for all deployments. Remove the
TURNSTONE_AUTH_ENABLED toggle and make JWT_SECRET and AUTH_TOKEN
required in the wizard's system prompt.

* fix: remove auth disable support from runtime and infra

Remove AuthConfig.enabled field — auth is always on. Drop
TURNSTONE_AUTH_ENABLED env var, config toggle, and the
check_request bypass. Update compose.yaml, Helm chart,
Terraform, docs, and tests to match.

* feat: deprecate config tokens, require JWT secret, prefer JWT auth

Phase 1 of config-token removal:

- load_jwt_secret() now exits with error if no secret is configured
  (was: silently auto-generated ephemeral secret)
- _authenticate_token() logs deprecation warning on config token use
- CLI /cluster commands use ServiceTokenManager when JWT secret is set
- turnstone-admin tls-list uses ServiceTokenManager when JWT secret is set
- Update bootstrap wizard, docker.md, security.md to mark
  TURNSTONE_AUTH_TOKEN as deprecated and JWT_SECRET as required
- Console test fixtures use auth token + headers (auth always enforced)

* feat: add service scope for inter-service JWT auth

Add "service" to VALID_SCOPES and SCOPE_HIERARCHY. Service tokens
bypass require_permission() RBAC checks, replacing the old
empty-user-id bypass that config tokens relied on.

All ServiceTokenManager instances that need admin access now include
"service" in their scopes (console proxy, channel gateway, CLI,
admin CLI). Read-only services (collector, notification) unchanged.

* feat: phase 2 config token deprecation

- SDK doc examples now show API tokens (ts_) instead of config tokens
- Remove _get_config_token() from admin CLI (dead code)
- Block config token exchange in handle_auth_login — only password
  and API token login allowed
- Update login tests to use password-based auth instead of config
  token exchange

* feat: phase 3 — remove config tokens entirely

Complete removal of config-file token authentication:

- Delete AuthConfig.tokens, check(), _ROLE_TO_SCOPES, hmac dispatch
  branch, and config token loading from load_auth_config()
- Remove auth_config parameter from _authenticate_token() and
  check_request() — callers updated throughout
- Remove TURNSTONE_AUTH_TOKEN from compose.yaml, Helm charts,
  Terraform, turnstone.example.toml
- Remove --auth-token CLI flags from turnstone, turnstone-admin,
  and turnstone-console
- Simplify console main() — always use ServiceTokenManager
  (no fallback to static tokens)
- Delete config-token-specific tests, rewrite check_request and
  integration tests to use JWT auth with proper audience claims
- Remove all config token references from docs (security.md,
  docker.md, sdk.md, console.md, architecture.md, bootstrap prompt)

* fix: address code review findings

- Fix 33 broken tests: add JWT auth to test_api_versioning,
  test_console_routing_proxy, test_tls_admin, test_tls_manager,
  test_server_live (jwt_secret + audience-scoped auth headers)
- Add TestRequirePermissionServiceScope: 4 tests covering the
  service scope RBAC bypass path
- Remove stale comments referencing config tokens in auth.py and
  console/server.py
- Remove dead proxy_auth_token parameter from console create_app()
  and static token fallback in _proxy_auth_headers()
- Remove TURNSTONE_AUTH_TOKEN from env.py scrub list

* fix: address Copilot review — JWT audience, compose require secret

- CLI /cluster: add audience=JWT_AUD_CONSOLE to ServiceTokenManager
  (console validates audience, JWTs without it were rejected)
- Admin CLI tls-list: same audience fix
- compose.yaml: TURNSTONE_JWT_SECRET now uses :? to fail fast if unset
- SDK console: fix default port from 8081 to 8090

* test: add auth enforcement tests for TLS admin endpoints

5 new tests: unauthenticated requests return 401 (list, renew,
delete), read-only-scoped requests return 403 (renew, delete).
Closes the TLS auth enforcement test gap noted in PROGRESS.md.

* fix: address remaining Copilot review feedback

- Fix token_source="config" → "test" in TLS test fixtures
- Fix AuthResult.token_source docstring to include service origins
- Require TURNSTONE_JWT_SECRET in cluster compose profile (:?)
- Helm: add auth.jwtSecret + auth.existingSecret values, wire
  TURNSTONE_JWT_SECRET into secret.yaml and both deployments
- Terraform: replace auth_token with jwt_secret variable + secret,
  remove orphaned auth_token resources and IAM reference
- Remove [[auth.tokens]] from security.md config example

* fix: address full code review — 10 findings

Critical:
- Terraform: replace concat(common_env, auth_env) with common_env
  (auth_env local was removed but still referenced)
- Channel gateway: remove hmac static token auth from _check_auth(),
  use JWT-only validation. Remove --auth-token CLI arg from channel
- Rebalancer: add token_manager support so migration requests carry
  JWT auth (was sending unauthenticated POST to /internal/migrate)

Major:
- Guard _permissions_to_scopes() against "service" privilege
  escalation from DB role permissions
- Remove dead AuthConfig class, load_auth_config(), and all
  auth_config parameters from create_app() signatures
- Helm: inject JWT secret for both inline and existingSecret paths

Minor:
- Remove dead auth_token param from ClusterCollector
- Remove empty TestLoadAuthConfig class
- Short JWT secret now exits instead of warning
- Compose: add generation command comment above JWT_SECRET
- Clean stale config token references from 6 doc files
- Clean stale AUTH_TOKEN reference from bootstrap wizard prompt

* fix: remove remaining stale config token references from docs

- channels.md: remove --auth-token from options table
- oidc.md: remove "config-file tokens still work" claim
- security.md: remove config token section, fix JWT secret docs
  (now required/exits, no ephemeral fallback), remove hmac from
  ASCII diagram, remove --auth-token reference
2026-04-01 19:38:24 -07:00
Patrick Buckley 9a518657a3 feat: replace console HTTP polling with persistent SSE streams (#266)
* feat: replace console HTTP polling with persistent SSE streams

Console collector now subscribes to each server node's /v1/api/events/global
SSE stream for real-time state updates instead of polling /v1/api/dashboard
and /health every 15 seconds.

Server changes:
- Emit ws_created/ws_closed events on global queue from create/close handlers
- Add node_snapshot on SSE connect (workstreams, health, aggregate)
- Add ?expected_node_id= identity verification (409 on mismatch)
- Add health_changed callback to BackendHealthMonitor circuit breaker
- Add periodic aggregate emitter thread (10s)

Console collector changes:
- Single asyncio event loop on one thread multiplexes all SSE connections
  (scales to 1000+ nodes vs thread-per-node)
- Discovery loop spawns/cancels async SSE tasks per node
- Snapshot reconciliation on connect, delta application for live events
- Fix ws_state→cluster_state event type mismatch
- Remove polling code (poll_interval, max_poll_workers, --poll-interval CLI)

SDK changes:
- Add NodeSnapshotEvent, HealthChangedEvent, AggregateEvent dataclasses
- Add stream_node_events() method (async + sync)

* fix: address review feedback on node event streams

- Fix stop() to let SSE manager exit naturally instead of force-stopping
  the event loop (ensures finally cleanup runs)
- Guard against empty/invalid SSE data from ping frames
- Treat missing node_id as identity mismatch (409) when expected_node_id
  is provided
- Fix stale docstring on _update_metrics
2026-03-31 11:28:39 -07:00
Patrick Buckley a7d9461735 refactor: channel router + scheduler use SDK clients
ChannelRouter: replace raw httpx with AsyncTurnstoneServer (single-node)
and AsyncTurnstoneConsole route methods (multi-node). Remove _post()
helper, _route_path(), and manual JSON construction.

Scheduler: replace raw httpx.Client with TurnstoneServer (sync). Lazy
per-node client cache with token rotation and stale client pruning.

Clean remaining Redis/MQ references from tests, docs, and config:
- test_tls_admin: redis.internal -> app.internal
- test_config: [redis] test data -> [database]
- docs/channels.md, console.md: rewrite for HTTP architecture
- docs/api-reference.md, openshell.md: remove stale diagram/Redis refs
- turnstone.example.toml: remove [redis] section
- .pre-commit-config.yaml: remove types-redis dependency
- QUICKSTART.md: remove bridge/Redis from deployment descriptions
2026-03-30 20:30:05 -07:00
Patrick Buckley 037308f3b1 fix: propagate user identity through console proxy
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
2026-03-24 03:13:23 -07:00
Patrick Buckley 414eb52d67 feat: raise scaling limits for 1000-node clusters (#129)
* feat: raise scaling limits for 1000-node clusters

Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.

Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers

Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling

Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster

New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)

Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.

* fix: add image tag to compose anchors to avoid redundant builds

All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.

* fix: address Copilot review feedback on scaling PR

- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor

* fix: add PostgreSQL env vars to cluster bridge anchor

Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.

* fix: address Copilot round 2 + PG connection exhaustion at startup

Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
  per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change

Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
  to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service

* fix readme

* fix: startup resilience for large clusters

Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.

Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.

Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).

Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service

* fix: replace silent error suppression with structured logging

Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.

memory.py (26 changes):
  Every storage operation now logs on failure. Previously the entire
  persistence facade had zero logging — messages, workstream state,
  and structured memories could silently stop being saved.

server.py:
  Usage recording failures now log at warning (was pass).
  Global SSE fan-out errors log at debug (was pass).

console/server.py:
  Config reload notification logs per-node failures at warning.
  Settings read fallbacks log at warning with the default value used.

auth.py:
  User existence check logs at warning (was pass).
  Setup rollback failures log at error (was suppress).
  OIDC state cleanup logs at debug (was suppress).

mcp_client.py:
  DB-managed MCP server list failure logs at warning (was pass).

collector.py:
  Node poll failure upgraded from debug to warning with exc_info.
  Health fetch failure logs at debug with exc_info (was silent).

bridge.py:
  Best-effort plan rejection logs at warning (was suppress).
  Malformed SSE data logs at debug (was suppress).

session.py:
  Tool output UI callback failure logs at debug (was suppress).

* fix: stagger collector poll with deterministic per-node jitter

Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.

Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.

* fix: add startup jitter to bridge heartbeat and health monitor probe

Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.

Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.

Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.

* fix: split collector httpx timeout and raise keepalive pool

Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.

* fix: narrow detect_model return type for CLI and eval callers

detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
2026-03-19 04:53:11 -07:00
Patrick Buckley 75eda9a096 feat: unified skills system — merge prompt templates + workstream tem… (#106)
* feat: unified skills system — merge prompt templates + workstream templates

Evolves prompt_templates into a first-class skills entity and merges
workstream templates into the same model, collapsing two concepts into
one.

Migration 021: 21 new columns on prompt_templates (skills metadata,
security scan fields, session config from WS templates), skill_resources
table for bundled files, skill_versions table for auto-snapshot version
history. Data migration converts existing WS templates into skills with
name collision handling, migrates version history, renames workstreams
and scheduled_tasks columns, cleans orphaned permissions, drops old
tables.

Key changes:
- All public interfaces renamed: templates → skills (API, CLI, SDK, UI)
- Session config (model, temperature, token_budget, auto_approve, etc.)
  now lives on the skill and is applied at workstream creation
- /skill slash command, set_skill() API, --skill CLI flag
- BM25 skill search via SkillSearchManager for activation="search" skills
- Admin UI: Skills tab with collapsible Session Config section,
  description subtitles, activation/origin/MCP badges, pagination
- Shared validation helper (_parse_skill_session_config) for DRY CRUD
- Version history with auto-snapshot on every edit + API endpoint
- Cascade delete (resources + versions) on skill removal
- Security: range validation, activation allowlist, fail-closed enabled
  check, duplicate name 409, readonly guard, JSON validation
- 77 new tests across storage, runtime, search, API integration, and
  migration behavior verification (2521 total)

* fix: address Copilot review + rename admin.templates → admin.skills

- Skip skill lookup when resume_ws is set (avoids spurious 400)
- Fix _applied_skill_version mismatch (1 in both workstreams table and session)
- Remove stale template field from MQ protocol diagram
- Rename admin.templates permission to admin.skills everywhere (runtime,
  frontend, tests, docs) with migration step for persisted role data
- Fix stale /api/templates references in docs and diagrams
- Update docstrings/comments for skills terminology

* fix: address Copilot round 2 — skill version lineage + stale doc refs

- Compute actual skill version from skill_versions count (not hardcoded 1)
- Use same version in both workstreams table and session metadata
- Fix response payload example: "templates" → "skills" key
- Fix "Each template summary" → "Each skill summary"
2026-03-16 14:47:06 -07:00
Patrick Buckley 50544c0d1b feat: MCP Registry integration — discover and install servers from the official registry
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.

Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.

Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.

71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
2026-03-15 21:09:27 -07:00
Patrick Buckley 8895bf07eb feat: admin Settings tab — form-based editor replacing "coming soon" … (#60)
* feat: admin Settings tab — form-based editor replacing "coming soon" stub

Section-grouped layout with collapsible headers for all ~40 ConfigStore
settings (model, session, tools, server, mcp, ratelimit, health, judge,
memory). Type-appropriate inputs: CSS toggle for bools, number with
min/max/step, select for choices, text for strings. Secret fields shown
read-only. Source badge (storage/default), amber restart indicator.

Inline save per field with dirty detection, row flash on success, reset
to default via styled confirm modal. Full WCAG keyboard accessibility
(Enter/Space on section headers, aria-labels, focus-visible). Mobile
responsive single-column at <700px. Reduced-motion safe.

* fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes

Settings UI:
- Help tooltips: ? button on ~25 settings with plain-English explanations
  and optional reference links (arXiv, Fowler, MCP spec). Click to toggle
  popover, Escape to dismiss, aria-expanded for accessibility.
- Sections start collapsed for scannable overview.
- Restart badge: hidden by default, shows when dirty, persists after save
  with amber glow. Positioned left of source badge.
- Secret row alignment fixed (transparent border matches input box model).
- Docs link in toolbar → Swagger UI Settings section.
- Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard,
  numeric dirty detection (0.1 vs 0.10 no longer false positive).
- Secret reset button enabled when source=storage (clear legacy overrides).
- Space key repeat guard on section headers.
- Sidebar: sticky + max-height:100vh, no longer stretches with content.

Backend:
- context_window default changed from 131072 to 0 (auto-detect). Fallback
  lowered from 131K to 32K (realistic for local models when detection fails).
  Session normalizes 0→32768 defensively.
- Settings registry: help + reference_url fields on SettingDef, richer
  descriptions for model/session/tools/judge/memory settings.
- Schema API includes help + reference_url.
- Bootstrap system prompt: added Runtime Settings section.

Docs: tab counts updated to 13 across README, architecture, console, governance.
2026-03-14 13:12:40 -07:00
Patrick Buckley 02d9c5c797 feat: workstream templates — behavioral profiles for workstream creation (#49)
* feat: workstream templates — behavioral profiles for workstream creation

Workstream templates define the complete configuration for workstream
creation: system prompt, model, auto-approve policy, per-tool
auto-approve, temperature, reasoning effort, max tokens, agent max
turns, token budget, and completion notifications. Applied once at
creation time (snapshot, not live binding). Auto-versioning captures
pre-update state on every edit.

Schema & storage:
- workstream_templates + workstream_template_versions tables (migration 011)
- ws_template_id/ws_template_version columns on workstreams table
- ws_template column on scheduled_tasks table
- Full CRUD + versioning on SQLite and PostgreSQL backends
- prompt_template_hash (SHA-256) for drift detection

Runtime:
- Template resolution before mgr.create() for model override
- Post-creation settings application (prompt, temperature, approval, budget)
- Token budget enforcement in session.send() — 80% warning, approval gate
  at 100% via __budget_override__ synthetic tool
- WebUI.auto_approve_tools server-side per-tool auto-approve
- Prompt template drift detection (hash comparison, log warning on mismatch)

Integration:
- ws_template field on CreateWorkstreamMessage, bridge, channel router,
  scheduler dispatch, MQ client
- Console admin "WS Templates" tab (11th) with CRUD, version history modal
- Profile dropdown on workstream creation modal
- WS template dropdown on scheduler create/edit modals
- Prompt template name validation on ws_template create/update
- 7 console admin API endpoints + read-only summary endpoint
- Full OpenAPI spec entries in console_spec.py
- Python SDK (sync + async) and TypeScript SDK methods
- Pydantic schemas for all request/response models

Docs & diagrams:
- New 21-ws-template-architecture.puml sequence diagram
- Updated governance, storage, MQ protocol diagrams + PNGs
- Updated architecture.md, governance.md, api-reference.md, console.md, sdk.md

48 new tests (1788 total). mypy clean. ruff clean.

* fix: address PR #49 review feedback

- auto_approve_tools uses approval_label (not just func_name) for
  consistency with tool policy evaluation
- inline system_prompt from ws_template persisted as
  _ws_template_system_prompt in workstream_config, restored on resume
  (previously lost because _template_content wasn't persisted)
- budget gate (__budget_override__) no longer bypassed by blanket
  auto_approve — requires explicit approval or tool policy allow
- diagram 21 field list corrected (removed tool_search/threshold,
  added prompt_template_hash/notify_on_complete)

* fix: address PR #49 review feedback (round 2)

- Grant admin.ws_templates permission in migration 011 (tab was hidden)
- Center WS template modals and fix radio button alignment
- Skip template validation when ws_template overrides prompt
- Guard against empty version snapshots on no-op updates
- Replace setTimeout race with Promise chain in schedule ws_template select
- Validate numeric fields in admin create/update handlers (400 not 500)
- Add ws_template to TypeScript OpenAPI specs
- Use typed Pydantic response models in SDK ws_template methods
2026-03-12 21:06:51 -07:00
Patrick Buckley 5f0004dc91 feat: add ClusterSnapshot for instant console UI state rebuild (#34)
* feat: add ClusterSnapshot for instant console UI state rebuild

The console web UI was SSE-driven with no initial state — reloads and
navigation caused blank/loading gaps while waiting for API re-fetches.

Server-side: GET /v1/api/cluster/snapshot returns the full cluster state
(all nodes with workstreams + overview aggregates) built under a single
lock. The SSE stream now emits this snapshot as the first event on
connect (snapshot taken before listener registration to avoid race).

Frontend: local clusterState object mirrors the snapshot, patched
incrementally by SSE events. View navigation renders from local state
with no API round-trips. Fixes popstate/pushState history corruption
on Back/Forward navigation (pre-existing bug). Stable node sorting
with node_id tie-breaker on both server and client.

SDK: snapshot() method on Python (sync + async) and TypeScript console
clients. ClusterSnapshotEvent in event registries.

* fix: address review feedback and SSE proxy reconnect bug

Copilot review fixes:
- Atomic snapshot+register: new get_snapshot_and_register() acquires
  both state and listener locks, eliminating the event gap between
  snapshot read and listener registration.
- Debounce patch renders: patchClusterState uses requestAnimationFrame
  to batch rapid SSE events into a single recompute+render cycle.
- Fix health type: dict[str, str] → dict[str, Any] on all three
  console schema models (ClusterNodeInfo, NodeDetailResponse,
  ClusterSnapshotNode) since /health payloads contain nested objects.
- TypeScript ClusterSnapshotEvent: use concrete ClusterSnapshotNode[]
  and ClusterOverviewResponse types instead of Record<string, unknown>.

SSE proxy reconnect fix:
- _proxy_sse raw_stream now emits `: proxy-ping` comments every 3s
  when no upstream data arrives, preventing the browser EventSource
  from dropping idle connections. The raw byte passthrough refactor
  (4d11078) removed the proxy's independent keepalive — this restores
  it without reverting to EventSourceResponse.
2026-03-09 01:13:35 -07:00
Patrick Buckley 4d1107839b refactor: use raw streaming for SSE proxy to preserve event framing (#32)
* refactor: use raw streaming for SSE proxy to preserve event framing

- Replace httpx_sse aconnect_sse with raw httpx.stream for SSE proxy
- Stream bytes verbatim to preserve server-side ping comments and event framing
- Add StreamingResponse with proper headers (Cache-Control, X-Accel-Buffering)
- Update compose.yaml to add 'cluster' profile to the service

* Refactor SSE proxy to raw byte passthrough

- turnstone/console/server.py: Replace aconnect_sse + EventSourceResponse with
  httpx.stream() + StreamingResponse for raw byte passthrough. Server pings,
  events, and comments now flow through verbatim. Added per-request timeout
  override (read=None, pool=None) for long-lived SSE streams.

- tests/test_console.py: Add 3 new tests for SSE proxy:
  - Ping and event preservation
  - Upstream error status handling
  - Client disconnect handling

- docs/console.md: Update SSE Proxy section to reflect raw byte passthrough
  approach.
2026-03-08 04:46:34 -07:00
Patrick Buckley 924b976f1f Add scheduled task docs and SDK client methods
Add documentation and SDK support for the scheduled task system
(cron/at scheduling via console API). Includes Python SDK methods
(async + sync), TypeScript SDK methods, console.md API reference,
sdk.md table update, and architecture.md module map entry.
2026-03-05 18:11:36 -08:00
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00
Patrick Buckley 047680d669 Add user identity, JWT auth, and admin console UI (#23)
* Add user identity, JWT auth, and admin console UI (#23)

JWT-based authentication with three token types: config-file (hmac,
backward-compat), API tokens (ts_ prefix, SHA-256 hashed), and JWTs
(HS256, 24h expiry). Username:password login via bcrypt. Hierarchical
scopes: read < write < approve.

New tables: users (username, password_hash), api_tokens (token_hash,
scopes, expires), channel_users (future channel integrations). user_id
column added to sessions and workstreams for attribution.

Console owns admin CRUD (6 endpoints under /api/admin/). Server
validates JWTs locally with shared signing secret. Public /api/auth/setup
endpoint for first-time admin creation (atomic, only works with zero
users). turnstone-admin CLI for user/token management.

Admin console UI: Users and Tokens tabs with full CRUD modals, scope
badges, token show-once with clipboard copy, keyboard accessibility
(focus traps, Escape, arrow key tabs, ARIA roles).

Login UI redesigned: username:password primary, token toggle for legacy,
setup wizard auto-detected via /api/auth/status. Python + TypeScript
SDKs updated with login(username, password), authStatus(), setup().

New docs/security.md + diagram 15-auth-architecture.puml. All existing
docs updated. OpenAPI specs include all new endpoints. 64 new tests
(1023 total). Dependencies: PyJWT, bcrypt.

* Fix auth bugs, XSS vector, and doc inaccuracies from PR #23 review

Address Copilot review feedback: escape double quotes in escapeHtml()
to prevent XSS in HTML attributes, add JWT validation fallback so
config tokens containing dots still work, add user_id to
AuthLoginResponse schema, return created field from admin_create_user,
and correct five documentation files to match actual API behavior.
2026-03-04 09:12:18 -08:00
Patrick Buckley 62a4ceac96 Dev/api versioning openapi (#18)
* Add API versioning under /v1/ prefix with OpenAPI 3.1 spec

All API endpoints move to /v1/api/* (clean break, no unversioned
aliases). Non-API routes (/, /health, /metrics, /static, /shared,
/node proxy) stay unversioned.

New turnstone/api/ package:
- Pydantic v2 models for all request/response schemas (server +
  console) used for OpenAPI spec generation
- Programmatic OpenAPI 3.1 spec builder with EndpointSpec catalog
- /openapi.json serves machine-readable spec, /docs serves Swagger UI

Route changes:
- Both servers use Mount("/v1", routes=[...API routes...])
- Auth middleware strips /v1/ prefix before path classification
  (PUBLIC_PATHS/WRITE_PATHS stay unversioned internally)
- Console proxy handles /node/{id}/v1/api/ upstream forwarding
- Bridge and CLI HTTP clients updated to /v1/api/ paths
- /openapi.json and /docs added to PUBLIC_PATHS and rate limiter
  EXEMPT_PATHS

Security fix from review: required_role() now correctly handles
/node/{id}/v1/api/{path} proxy routes (previously the v1 segment
caused write-path detection to fail, allowing read-only token
escalation).

42 new tests (830 total). All frontend JS, docs, and diagrams updated.

* Fix mypy type errors in turnstone/api/ package

- Add generic type params to dict fields in console_schemas.py
- Add return type annotations to docs.py handler factories
- Move type-only imports (BaseModel, Callable, Awaitable) into
  TYPE_CHECKING blocks to satisfy TC002/TC003 ruff rules

* Address PR #18 review feedback + fix mypy errors

Review fixes:
- Add pydantic>=2.0 as explicit dependency in pyproject.toml
  (was only transitively available via openai/mcp)
- Auto-detect path parameters from {param} segments in OpenAPI
  spec builder (fixes missing required path params)
- Use startswith() with concrete prefix for proxy version
  detection instead of fragile substring check
- Make Swagger UI base URL configurable via swagger_ui_base_url
  parameter for air-gapped deployments

Mypy fixes:
- Add generic type params to dict fields in console_schemas
- Add return type annotations to docs.py handler factories
- Move type-only imports into TYPE_CHECKING blocks
2026-03-03 20:28:49 -08:00
Patrick Buckley 29c00c0cdf Extract shared frontend design system into turnstone/shared_static/ (#17)
* Extract shared frontend design system into turnstone/shared_static/

The server UI and console UI had ~60% CSS overlap and significant JS
duplication. Extract shared assets into a new turnstone/shared_static/
package mounted at /shared/ in both servers:

- base.css: design tokens, reset, typography, login/toast/kb overlays,
  dashboard table, state dots, health bar, scrollbar, reduced motion
- auth.js: authFetch, login overlay with focus trap, logout (hooks for
  page-specific post-login/logout callbacks)
- theme.js: dark/light toggle with system preference detection
- toast.js: notification queue with configurable timeout
- utils.js: escapeHtml, formatTokens, ctxClass, formatUptime, formatCount
- kb.js: keyboard shortcuts overlay with configurable content, focus
  management, and focus restore on dismiss

Console proxy updated: JS shim injection moved from proxy_static (app.js
prepend) to proxy_index (inline <script> in HTML) so it runs before any
external scripts. New /shared/ path rewriting and proxy_shared_static
route added. ~1540 lines removed from page-specific files, 775 lines in
shared package. 13 new tests (788 total).

* Fix /shared/ auth and remove __init__.py from shared_static

Address PR #17 review feedback:

1. Add /shared/ to PUBLIC_PREFIXES in auth.py so shared CSS/JS
   loads before authentication (required for login overlay to render)

2. Remove turnstone/shared_static/__init__.py to prevent exposing
   Python package internals (__init__.py, __pycache__) via the
   StaticFiles mount. Not needed for packaging since pyproject.toml
   uses explicit glob includes.

3 new auth tests for /shared/ public path access.
2026-03-03 20:11:49 -08:00
Patrick Buckley b6e0f0fcca Add node version tracking and drift detection to console dashboard (#16)
* Add node version tracking and drift detection to console dashboard

Surface the version field from each node's /health endpoint in the
console dashboard. Collector extracts version into get_overview()
(version_drift + versions fields), promotes it to top-level in
get_nodes(), and adds get_version_info() for per-node detail. Console
/health endpoint includes drift fields.

Frontend adds a VER column to the 7-column node table grid, shows
per-node version strings, tracks versions per group with "mixed" +
yellow drift badge when nodes disagree, and displays a DRIFT warning
or single version in the status bar. Column hidden on mobile (<700px).
ARIA labels include version info for accessibility.

10 new tests (745 total). Docs and diagram updated.

* Fix drift tooltip text: show 'Versions detected' not 'Nodes running'
2026-03-03 18:57:30 -08:00
Patrick Buckley 6c5441435b Add console workstream creation + server reverse proxy (#14)
* Add console workstream creation + server reverse proxy (#14)

Enable the console dashboard to create workstreams and proxy server UIs,
so users only need network access to the console port.

Workstream creation via MQ:
- POST /api/cluster/workstreams/new with three targeting modes:
  specific node (directed queue), auto (best node by capacity),
  or general pool (shared queue, any bridge picks up)
- Console pushes CreateWorkstreamMessage to Redis; bridge handles
  the rest (server creation, ownership registration, SSE events)

Reverse proxy for server UIs:
- /node/{node_id}/ serves the server's HTML with static path rewriting
  and a console-return banner injected after <body>
- JS proxy shim prepended to app.js overrides fetch() and EventSource()
  to route root-relative URLs through /node/{id}/api/...
- SSE streams proxied via httpx.AsyncClient(timeout=None) with per-
  connection clients for long-lived streams
- GET/POST API requests forwarded with body and auth token

Security:
- Proxy write paths checked against WRITE_PATHS to prevent read-token
  escalation (read tokens cannot POST /api/send through proxy)
- html.escape() on node_id in banner HTML to prevent XSS
- String length limits on name/model inputs

Frontend:
- "+ new" button in header opens creation modal with node dropdown
  (Auto / General pool / specific nodes with capacity display)
- Modal has focus trap, backdrop dismiss, scroll lock, keyboard handling
- Workstream rows and node links deep-link via proxy paths
- Custom select arrow, Instrument Panel modal styling

Documentation:
- docs/console.md rewritten with proxy and creation API docs
- docs/architecture.md console section updated
- PlantUML diagrams 01, 11, 12 updated + PNGs re-rendered
- README.md updated

28 new tests (741 total), ruff + mypy clean.

* Fix Copilot PR #14 review issues: auth bypass, XSS, proxy robustness

- Normalize trailing slashes in required_role() to prevent write-role
  bypass via /api/send/ or /node/{id}/api/send/ (auth.py)
- Validate node_id format in proxy handlers (alphanumeric, dot, dash,
  underscore only) to prevent injection vectors
- Use json.dumps() for JS proxy shim prefix to prevent script injection
- URL-quote node_id in HTML attribute contexts (proxy_index, proxy_static)
- Check upstream status in _proxy_sse() — emit error event on non-200
  instead of keeping a dead SSE connection open
- Check upstream status in proxy_index() — propagate non-2xx errors
- Forward query string in _proxy_post() (consistency with _proxy_get)
- Handle JSON null values in create_workstream() — treat null as empty,
  reject non-string types with 400
- Fix docs/diagram LPUSH → RPUSH to match actual broker implementation
2026-03-03 18:25:18 -08:00
Patrick Buckley b6c1c3a676 Add console-to-server deep linking via ?ws_id= query parameter (#7)
* Add console-to-server deep linking via ?ws_id= query parameter

Server UI parses ?ws_id= on load (both direct init and post-login) and
auto-selects the matching workstream instead of showing the dashboard.
URL is cleaned from the address bar via history.replaceState after
navigation. Defers initial SSE connection to avoid redundant connect
when deep-linking switches tabs immediately.

Console workstream rows are now clickable — opens the node's server UI
in a new tab with ?ws_id= targeting that workstream. Uses URL constructor
for safe URL building. External-link indicator (↗) appears on hover.
Rows without server_url have role/tabindex removed to avoid broken
affordance. currentServerUrl reset on showOverview() to prevent stale
fallback across views.

Collector injects server_url into workstream dicts in both the poll path
and ws_created event path so deep links work immediately.

* Fix Copilot PR #7 review: deep-link duplicate history entry, server_url test coverage

- Suppress history.pushState in switchTab() during deep-link navigation by
  setting _historyNavigation=true around both call sites (post-login and
  direct init). Fixes Back button appearing to do nothing on first press.
- Add server_url assertions to poll and ws_created collector tests to
  prevent regressions of deep-link functionality.
2026-03-02 19:46:35 -08:00
Patrick Buckley 9be155b97a Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup

- Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files
- Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow
- Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems
- Refresh README and docs with badges, diagram links, and current descriptions
- Refactor test_server_live.py with mock streaming helpers for deterministic CI testing
- Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2)

Console dashboard:
- Move state indicators from top cards to fixed bottom status bar with cluster metrics
- Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000)
- Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent,
  LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance
- Add render cache, stale indicator, active filter highlight, loading states

Server web UI:
- Apply matching Instrument Panel aesthetic for visual consistency with console
- Fix branding (pcode → turnstone), extract inline styles to CSS classes
- Rename pcode localStorage keys and history state to turnstone

Legacy cleanup:
- Remove persona-model-specific --persona flag and /persona slash command
- Remove model_identity from chat_template_kwargs (vLLM-specific mechanism)
- Refactor plan agent to use standard developer message instead of model_identity
- Remove dead code (unused date/has_tools variables, noqa suppressions)

* Fix CI typecheck: add mypy overrides for optional sympy/numpy imports

The math sandbox optionally imports sympy and numpy at runtime (try/except
ImportError). In CI these packages are not installed, so mypy raises
import-not-found rather than import-untyped. Add mypy overrides to
ignore missing imports for these optional dependencies.

* Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity

- Change #node-table from role="tree" to role="list" and group elements
  from role="treeitem" to role="listitem" (proper ARIA semantics)
- Include currentView and currentFilter.state in renderStatusBar cache key
  so active pill highlight updates when switching views
- Align pulse animation to 0.35 opacity (already applied in CSS)
2026-03-02 16:55:12 -08:00
Patrick Buckley 0d6252dd7d Initial commit — turnstone multi-node AI orchestration platform. 2026-03-02 00:33:37 -08:00