Commit Graph

150 Commits

Author SHA1 Message Date
Patrick Buckley df8a36ced4 fix: add timeout to MCP server disconnect to prevent hung removals
stack.aclose() on a stuck streamable-http transport hangs indefinitely,
causing 50% CPU on all nodes when removing a broken remote server via
reconcile_sync. Wrap with asyncio.wait_for(timeout=10s) so cleanup
proceeds even if the transport refuses to close cleanly.
2026-03-15 21:42:05 -07:00
Patrick Buckley ef6cac6428 fix: address review feedback and add sync-pending indicator
Review fixes:
- Rename query param from `q` to `search` across endpoint, frontend,
  SDKs, OpenAPI spec, docs, and tests to match upstream registry API
- Validate variables/env/headers are dicts in install endpoint (400 on
  malformed input instead of 500)
- Block javascript: and unsafe URL schemes on repo and website links
  rendered from registry data (XSS prevention)
- Add roving tabindex to Servers/Registry pill toggle for correct
  keyboard focus behavior
- Add noreferrer to website link in detail modal

Sync-pending indicator:
- "Sync to Nodes" button pulses yellow after create/edit/delete/import
  to alert admin that nodes have unseen changes
- Clears after successful sync
- Reduced-motion safe
2026-03-15 21:41:02 -07:00
Patrick Buckley 50544c0d1b feat: MCP Registry integration — discover and install servers from the official registry
Backend: standalone MCPRegistryClient (httpx async) queries the official MCP
Registry API (registry.modelcontextprotocol.io, v0.1). Two new console admin
endpoints: GET /v1/api/admin/mcp-registry/search (proxy with installed-status
annotation, dedup, uninstallable server filtering) and POST
/v1/api/admin/mcp-registry/install (auto-reloads all cluster nodes). Migration
019 adds registry_name/version/meta columns to mcp_servers with partial unique
index. Configurable registry URL via mcp.registry_url setting for
enterprise/private registries. resolve_install_config() handles both remote
(streamable-http) and package (npm→npx, pypi→uvx) installs. Pydantic models,
OpenAPI spec, Python + TypeScript SDK methods.

Frontend: unified MCP admin tab with Servers/Registry pill toggle (ARIA
tablist). Servers view: tri-state source badges (CONFIG/MANUAL/REGISTRY).
Registry view: search bar with type filter (remote/npm/pypi), auto-browse on
tab switch, result cards with source-type badges and repo links, one-click
install for zero-config remotes, install modal with dynamic form for servers
needing env vars/headers/URL variables. Package install warning banner.
Post-install status polling with connection/error feedback toasts. Trust
notice banner linking to the official registry.

Safety: 30s connect timeout on streamablehttp_client and session.initialize()
prevents hung connections from blocking the MCP event loop indefinitely.
Required-only headers in install config prevents empty auth headers from
causing silent 401s.

71 new tests (registry client, API endpoints, storage columns). Docs:
dedicated docs/mcp-registry.md, updated api-reference, architecture, console,
sdk, settings docs. Updated MCP architecture diagram.
2026-03-15 21:09:27 -07:00
Patrick Buckley d1c484737f fix(ci): regenerate lockfile for v0.7.0 and exclude local package from pip-audit
uv lock --check fails after version bump because the lockfile is stale.
pip-audit --strict fails because turnstone 0.7.0 isn't on PyPI yet.

Fix: regenerate uv.lock, and audit only third-party deps via
uv export --no-emit-project piped to pip-audit -r.
2026-03-15 17:28:13 -07:00
Patrick Buckley c2c6689a8e chore: bump version to 0.7.0 v0.7.0 2026-03-15 17:15:27 -07:00
Patrick Buckley f5af4875ba fix: surface MCP server errors in admin UI instead of silent logging (#100)
* fix: surface MCP server errors in admin UI instead of silent logging

get_server_status() hardcoded error="" — connection and refresh failures
were logged but never surfaced to the admin panel.

Added _last_error dict to MCPClientManager: set on failure (connect,
refresh, periodic refresh, notification handler), cleared on success,
cleaned up on remove. Read in get_server_status().

Admin UI: error tooltip on list row status span, error text in red
in detail modal per-node list. Schema already had the field.

6 new tests for error tracking lifecycle.

* feat: add turnstone_mcp_server_errors Prometheus gauge

Exposes the count of MCP servers currently in error state via
/metrics for alerting and reliability tracking.

* fix: address copilot review — sanitize error strings, clear on notification success

- Add _set_error() helper: strips newlines, truncates to 256 chars
- All error-setting sites now use _set_error() for consistent sanitization
- Notification handler clears _last_error on successful refresh (fixes
  stale error for push-notification servers that skip _periodic_refresh)
2026-03-15 17:12:13 -07:00
Patrick Buckley 0d77d65266 test: add OIDC handler integration tests (22 tests) (#99)
TestClient-based integration tests for the 4 OIDC HTTP endpoints: authorize, callback, admin list identities, admin delete identity.

Uses real SQLite storage with mocked external OIDC calls (exchange_code, validate_id_token, provision_oidc_user) to exercise the full handler→module→storage contract. Covers happy paths, error flows, rate limiting, JWKS key rotation retry, and state expiration.
2026-03-15 16:44:44 -07:00
Patrick Buckley c11991819e fix: apt-get upgrade in Dockerfile to resolve CVE-2026-0861
Trivy scan fails on HIGH for libc-bin/libc6 (2.41-12+deb13u1).
The fix (2.41-12+deb13u2) is available in Debian repos but the
base python:3.14-slim image hasn't been rebuilt yet. Adding
apt-get upgrade pulls in all pending security patches at build time.
2026-03-15 16:40:03 -07:00
Patrick Buckley fc948d711d fix: use pgautoupgrade for seamless postgres major version upgrades
Replaces postgres:18-alpine with pgautoupgrade/pgautoupgrade:18-alpine
in compose.yaml. Sets PGDATA=/var/lib/postgresql/data so pgautoupgrade
detects existing pg17 data and runs pg_upgrade automatically on first
start. No manual migration needed.

Also increases healthcheck start_period to 30s to accommodate the
one-time upgrade process.
2026-03-15 16:26:19 -07:00
renovate[bot] d9722f3578 chore(config): migrate config .github/renovate.json (#97)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 16:24:39 -07:00
renovate[bot] f494553020 chore(deps): update helm release redis to v25 (#93)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:41 -07:00
renovate[bot] c766c81f25 chore(deps): update helm release postgresql to v18 (#92)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:38 -07:00
renovate[bot] 3b746fb28d chore(deps): update docker images (#90)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:34 -07:00
renovate[bot] b82fa4923c chore(deps): lock file maintenance (#94)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:18 -07:00
renovate[bot] 4ac316dc0e chore(deps): update github actions (#91)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:16 -07:00
renovate[bot] 387ef06da7 chore(deps): update helm release redis to ~20.13.0 (#89)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:13 -07:00
renovate[bot] e4e2200c33 chore(deps): update helm release postgresql to ~16.7.0 (#88)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:11 -07:00
renovate[bot] 8b94d553e4 chore(deps): update docker images (#87)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:55:53 -07:00
renovate[bot] cf16724137 chore(deps): pin dependencies (#86)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:53:03 -07:00
Patrick Buckley 22402e89de feat: add dependency management with Renovate, uv.lock, and security … (#83)
* feat: add dependency management with Renovate, uv.lock, and security scanning

Adds automated dependency update detection and vulnerability scanning
across all dependency layers (Python, vendored JS, TypeScript SDK, Docker,
GitHub Actions).

- Renovate config with 10 package groups and custom regex managers for
  vendored JS (KaTeX, Highlight.js, Mermaid) tracking via npm registry
- uv.lock for reproducible builds (80 packages)
- Dockerfile switched to uv sync --frozen with layer caching
- CI: pip-audit (via lock file), npm audit, lock-check jobs
- CI: lint job uses pre-commit for ruff version consistency
- Docker security scan workflow (weekly Trivy, HIGH/CRITICAL)
- Helper script for vendored JS library updates

* fix: resolve CI failures and address review feedback

- Update pre-commit hooks: ruff v0.9.10 -> v0.15.6 (fixes deprecated
  UP038 rule), mypy v1.14.1 -> v1.19.1
- Add per-file-ignore for N802 on sandbox.py (ast visitor convention)
- Fix pip-audit: install into uv venv so uv run can find it
- Pin uv-version in CI to match lock file generator (0.9.18)
- Upgrade vitest ^2.0 -> ^4.1 to fix esbuild GHSA-67mh-4wv8-2f99
- Vendored JS script: use grep -rl for auto-discovery of version refs
  (catches docs/architecture.md), fix LICENSE comment, portable grep
2026-03-15 15:46:48 -07:00
Patrick Buckley e7743fd079 feat: per-tool "Always" approve instead of blanket auto-approve (#82)
* feat: per-tool "Always" approve instead of blanket auto-approve

Interactive "Always" button now adds specific tool names to
auto_approve_tools instead of setting blanket auto_approve=True.
Only the tool types in the current batch are auto-approved going
forward — new tool types still prompt for approval.

Server uses approval_label (with func_name fallback) matching the
existing approve_tools() lookup. CLI and bridge use func_name.
Budget override excluded from all paths.

UI: dashed border on Always button signals persistent action,
dynamic tooltip/badge show tool names, aria-label for screen
readers, focus-visible outline fix, overflow-wrap on badge.

Bridge: seeds with DEFAULT_SAFE_TOOLS on first "always" to avoid
losing existing safe-tool auto-approvals.

16 new tests (10 unit + 6 TestClient integration). Updated tool
pipeline diagram and docs.

* fix: address copilot review — filter errored items, hide Always on budget-only

- Server/bridge/JS: add `not it.get("error")` filter so policy-denied
  items aren't added to auto_approve_tools
- Hide Always button when no eligible tools (budget-override-only batch)
- Docs: clarify CLI/bridge use func_name (coarser MCP granularity)
2026-03-15 15:25:40 -07:00
Patrick Buckley 27349e1c13 refactor: move bridge content buffer to server-side single source of truth
Eliminate dual accumulation by piggybacking assistant response text on
the server's ws_state:idle SSE event.  The bridge no longer maintains
its own _ws_content_buffer — it reads content directly from the idle
event and passes it through to TurnCompleteEvent unchanged.

Server-side: WebUI accumulates tokens in on_content_token(), joins and
includes in the idle broadcast, then resets (with 256 KB cap).

Downstream consumers (Discord bidi DM forwarding, catch-up) are
unaffected — TurnCompleteEvent.content is still populated.
2026-03-15 15:04:22 -07:00
Patrick Buckley 37a48bb30d fix: validate scope_id requires scope in memory API (#80)
* fix: validate scope_id requires scope in memory API

Prevent misleading scope_id usage: reject scope_id with global scope,
require scope when scope_id is provided, require scope_id for
workstream/user scopes on writes. Belt-and-suspenders guard in storage
backends ignores scope_id when scope is empty.

* fix: strip whitespace in scope validation, relax user scope_id requirement

Address Copilot review: .strip() whitespace-only values in all three
validation helpers; SaveMemoryRequest no longer requires scope_id for
user scope since the server auto-resolves it from auth context.
2026-03-15 14:28:03 -07:00
Patrick Buckley 730f5704ff fix: inject prompt template guardrails into plan agent system message (#79)
* fix: inject prompt template guardrails into plan agent system message

Safety/behavioral templates were silently bypassed by the plan agent,
which only used _PLAN_IDENTITY. Now _plan_system_content() prepends
_template_content (when present) so admin-configured guardrails apply
to both _exec_plan and _refine_plan, matching the task agent pattern.

* fix: address Copilot review — log truncation, comment clarity, test robustness

- Log warning on template truncation in _plan_system_content() for
  consistency with _init_system_messages()
- Clarify comment that prior plan pairs (not general history) are forwarded
- Use ChatSession._PLAN_IDENTITY for index assertions instead of substring
2026-03-15 14:24:10 -07:00
Patrick Buckley 9b3b1c1ddd fix: reorder new-workstream modal so Task is the primary field (#77)
* fix: reorder new-workstream modal so Task is the primary field

Users were typing their prompt into the Name field (first text input,
auto-focused) and leaving Task empty, creating idle workstreams. Move
Task textarea to the top of the form, auto-focus it, and add
Ctrl/Cmd+Enter submit shortcut. Accessibility fixes: cancel button
focus-visible, label-hint contrast raised to WCAG AA, platform-aware
keyboard hint, Ctrl+Enter added to shortcuts overlay.

* fix: Enter on Cancel button no longer triggers submit

Copilot review caught that pressing Enter while focused on the Cancel
button bypassed native click and called submitNewWs(). Skip the
Enter-to-submit handler for BUTTON elements so native activation fires.
Also make keyboard shortcuts overlay platform-aware (Ctrl vs ⌘).
2026-03-15 14:23:57 -07:00
Patrick Buckley a9ed8a954b fix: convert _pending_nudge from single-slot to list for defensive correctness
Every append site immediately drains via _init_system_messages(), so this
is defensive — ensures multiple nudges survive if the drain flow is ever
refactored to batch calls.
2026-03-15 14:17:34 -07:00
Patrick Buckley 1efcbcf2ba perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload wi… (#73)
* perf: parallelize _collect_mcp_status and _notify_nodes_mcp_reload with asyncio.gather

Both functions queried cluster nodes sequentially, making latency
O(N × timeout). Use asyncio.gather to query all nodes concurrently,
matching the existing admin_list_watches pattern. Also reuse the
shared proxy_client instead of creating throwaway httpx clients per
node, and add debug logging on MCP status fetch failures.

* perf: bound node fan-out concurrency and improve debug logging

Add _NODE_FAN_OUT_LIMIT (50) semaphore to all three gather fan-out
sites (_collect_mcp_status, _notify_nodes_mcp_reload, admin_list_watches)
to cap concurrent outbound connections below the httpx pool limit,
leaving headroom for other proxy traffic at 1000-node scale.

Add exc_info=True to all debug log calls for actionable diagnostics.

* test: add unit tests for _collect_mcp_status and _notify_nodes_mcp_reload

11 tests covering success, non-200, missing URL, exceptions, empty
cluster, and mixed multi-node scenarios for both fan-out helpers.
2026-03-15 13:54:49 -07:00
Patrick Buckley 1d36d80fe5 fix: reduce metacognition false positives with strong/weak pattern tiers (#76)
* fix: reduce metacognition false positives with strong/weak pattern tiers

Correction detection: split "no" handling — "no," and "no." are strong
(always fire), "no <word>" uses an allowlist of correction-context words
(pronouns, demonstratives, verbs) instead of a blocklist. Phrases like
"no problem", "no worries", "no rush" are excluded automatically.

Completion detection: move most patterns to weak tier, gated by message
length (<80 chars) and absence of continuation markers ("?", "can you",
"but", "now", "please", etc.). "thanks for X" excluded at regex level.
Strong tier (always fire): "that's all", "lgtm".

* fix: align allowlist comment with implementation (include articles)
2026-03-15 13:53:39 -07:00
Patrick Buckley e603a6a7d1 fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var (#74)
* fix(oidc): pin redirect URI via TURNSTONE_OIDC_REDIRECT_BASE env var

OIDC redirect_uri was derived from the request Host header, which is
unreliable behind reverse proxies. Add TURNSTONE_OIDC_REDIRECT_BASE
(env var / config.toml) to pin the externally-reachable origin.

Extract _build_oidc_redirect_uri() helper to deduplicate the authorize
and callback handlers. Validate redirect_base at load time (must be
scheme://host[:port], rejects paths/query strings/invalid schemes).

* fix(oidc): reject redirect_base with missing hostname

Addresses Copilot review: values like `https://` or `https://:443`
passed validation but would produce invalid redirect URIs.

* fix(oidc): reject redirect_base with userinfo or invalid port

Addresses Copilot round 2: urlparse silently accepts user:pass@host
and non-numeric ports. Now explicitly rejects both.
2026-03-15 13:52:05 -07:00
Patrick Buckley 2e95f2ac73 test: add scope coverage for internal MCP/config reload endpoints (#75)
* test: add scope coverage for internal MCP/config reload endpoints

Verify required_scope() returns "approve" for _internal endpoints
across all access patterns (bare, /v1/-prefixed, console proxy with
and without /v1/), plus a GET negative test confirming only POST is
elevated. Closes the "internal endpoints accept read scope" item in
PROGRESS.md — the endpoints were already in APPROVE_PATHS.

* test: add config-reload v1/proxy scope tests per review feedback

Add /v1/-prefixed and console proxy variants for config-reload to
match the mcp-reload coverage, as flagged by Copilot review.
2026-03-15 13:45:35 -07:00
Patrick Buckley 5f27ed9fca feat: OIDC identity management inline in Users admin tab (#72)
Expandable user rows in the console Users tab reveal OIDC identities
linked to each user. Issuer badge, truncated subject, email, relative
last-login time, and unlink action with confirmation modal + audit trail.

Keyboard accessible (tabindex, Enter/Space, aria-expanded, focus-visible).
In-place refresh after unlink (no close/reopen flicker). Audit captures
user_id before delete. Mobile responsive (3-column at <700px).
Reduced-motion support. 2 new admin API endpoints reusing admin.users
permission and existing storage methods.
2026-03-15 03:52:42 -07:00
Patrick Buckley 20df7b3034 feat: OIDC SSO authentication with PKCE, auto-provisioning, and role … (#71)
* feat: OIDC SSO authentication with PKCE, auto-provisioning, and role mapping

Add OpenID Connect as a fourth authentication method, enabling single sign-on
via any OIDC provider (Okta, Azure AD, Google, Keycloak). Opt-in via env vars
(TURNSTONE_OIDC_ISSUER, CLIENT_ID, CLIENT_SECRET).

Security:
- Authorization Code Flow with PKCE (S256)
- State/nonce parameters with database-backed pending store (multi-node safe)
- JWKS signature validation with async fetch + key rotation retry
- Algorithm allowlist from JWKS key (not token header) prevents confusion
- Identity matching exclusively by (issuer, sub) — prevents account takeover
- password_enabled=false enforced server-side, not just UI
- Rate limiting on both authorize and callback endpoints
- OIDC users get "!oidc" password sentinel (bcrypt rejects naturally)
- ID token validated for iss, aud, exp, nonce

Features:
- Auto-provisioning with username deduplication on first login
- Claim-based role mapping with IdP demotion propagation (revokes stale roles)
- "Continue with [Provider]" SSO button on login page
- OIDC-only mode hides password form
- Setup wizard required before OIDC login (admin bootstrap)

Storage: migration 018 (oidc_identities + oidc_pending_states tables),
8 new protocol methods on both SQLite and PostgreSQL backends.
66 new tests (2273 total).

* fix: address PR #71 review feedback (18 items)

Bugs fixed:
- OIDC success redirect now fetches permissions via new /auth/whoami
  endpoint before completing login (fixes permission-gating in UI)
- Remove double decodeURIComponent on oidc_error (URLSearchParams
  already decodes; extra call throws on stray %)
- Authorize rate limiter returns redirect instead of JSON 429
  (endpoint reached via browser navigation, not fetch)
- Lazy JWKS fetch in callback when startup discovery failed (IdP
  recovery without restart)
- Startup exception handlers now log with exc_info=True
- PostgreSQL pop_oidc_pending_state uses DELETE...RETURNING for
  true atomicity (eliminates TOCTOU)

Behavior:
- New OIDC users without role mapping get builtin-viewer by default
  (assigned_by="oidc-default", not revoked by role sync)

Documentation fixes:
- Role mapping: sync semantics (add + revoke stale), not "additive only"
- PASSWORD_ENABLED=false blocks ALL password logins including admin
- Algorithm: asymmetric allowlist, not per-key derivation
- PlantUML diagram updated for role revocation

API spec fixes:
- Removed error_codes=[302] from callback (302 is success redirect)
- Added /auth/whoami to both server + console specs
- Regenerated TypeScript SDK OpenAPI snapshots (23 + 51 paths)

* fix: address PR #71 round 2 review feedback (10 items)

Rate limiting:
- Authorize endpoint now calls record() after check() so the rate
  limiter actually counts attempts (was a no-op before)

OIDC resilience:
- Split startup try/except: discovery failure disables OIDC, JWKS
  prefetch failure leaves OIDC enabled for lazy retry on first login
- JWKS unavailable message changed to "temporarily unavailable"
  (was misleadingly "not configured")
- create_oidc_pending_state raises on collision instead of OR IGNORE
  (prevents silent insert drop on state collision)
- SQLite pop_oidc_pending_state uses BEGIN IMMEDIATE for write lock
  (eliminates TOCTOU race)

Frontend:
- OIDC error display deferred 300ms so showLogin()'s async status
  fetch doesn't clear it via _switchMode → _clearError

API spec:
- OIDC authorize/callback endpoints now declare response_code=302
- Added AuthWhoamiResponse Pydantic model for /auth/whoami
- Regenerated TypeScript SDK OpenAPI snapshots

Documentation:
- Diagram: JWKS "cached at startup, refreshed on-demand" (was "hourly")
- Added TODO(tech-debt) comments on Host header redirect_uri sites
2026-03-15 03:44:18 -07:00
Patrick Buckley 68c991fbdd fix: restore safe HTML element rendering and suppress plantuml warning (#70)
* fix: restore safe HTML element rendering and suppress plantuml warning

- Add safe HTML tag allowlist in inlineMarkdown: br, hr, kbd, mark,
  sub, sup, ins, wbr, details, summary, abbr, small, u, s
  (attribute-free only — XSS safe, tags with attributes stay escaped)
- Add <details>/<summary> block-level protection pass with recursive
  markdown rendering of inner content
- Add plantuml to _NO_HIGHLIGHT_LANGS (suppresses highlight.js warning
  for unsupported language)
- CSS for details (collapsible, overflow hidden), kbd (mono font,
  key style), mark (yellow-glow token for theme adaptation)

* fix: restrict safe tags to inline-only, broaden details regex

- Remove hr, details, summary from inline _SAFE_TAGS allowlist (they
  are block-level and produce invalid HTML inside <p> wrappers)
- Make <details> regex newline-optional so same-line
  <details><summary>Title</summary> patterns are captured
2026-03-15 02:34:03 -07:00
Patrick Buckley 376da3d084 feat: prompt template tech debt — tests, read-only endpoints, double-… (#67)
* feat: prompt template tech debt — tests, read-only endpoints, double-load fix, server creation modal

Close test coverage gaps for prompt templates:
- Resume with deleted template: verifies graceful degradation (template_content=None, warning logged)
- Threading safety: concurrent set_template/init_system_messages with no race conditions
- Factory passthrough: template kwarg propagation through WorkstreamManager.create()

Add read-only template listing endpoints (read scope, no content exposed):
- GET /v1/api/templates — prompt template summaries (name, category, is_default, origin)
- GET /v1/api/ws-templates — enabled workstream template summaries (name, description, model)
- Available on both server and console; Python + TypeScript SDK methods added
- Console creation modal switched from admin endpoint to read-scope endpoint

Eliminate double-load inefficiency in workstream creation:
- Template validation moved before mgr.create() (no create-then-rollback on invalid template)
- template kwarg plumbed through WorkstreamManager.create() and session factory
- _SessionFactory Protocol added for proper mypy typing

Add workstream creation modal to server web UI:
- Name, model, template dropdown, ws_template/profile dropdown
- Instrument panel aesthetic: gradient top border, blur backdrop, amber accent
- Focus trap, Escape/Enter keyboard handling, loading state, error display
- WCAG AA contrast compliance, reduced-motion support

* fix: add list_ws_templates SDK methods + regenerate OpenAPI snapshots

Add list_ws_templates() to Python SDK (async + sync) and listWsTemplates()
to TypeScript SDK for the new GET /v1/api/ws-templates server endpoint.
Add WsTemplateSummary + ListWsTemplateSummaryResponse TypeScript types.
Regenerate openapi-server.json and openapi-console.json snapshots.

Addresses Copilot review feedback on PR #67.

* fix: skip template pre-validation when resuming a workstream

When resume_ws is set, the request's template field is irrelevant —
resume() restores the template from workstream_config. Pre-validating
a stale template name would incorrectly return 400 before the resume
even runs.

Addresses Copilot review feedback on PR #67.
2026-03-15 02:09:31 -07:00
Patrick Buckley e2a199c9c3 feat: mermaid diagram rendering with lazy loading and theme integration (#69)
* feat: mermaid diagram rendering with lazy loading and theme integration

Integrate mermaid.js 11.13.0 (self-hosted, MIT, ~2.9MB) for rendering
```mermaid code blocks as inline SVG diagrams. Covers flowcharts,
sequence, class, ER, state, gantt, pie, timeline, and mindmap.

- Lazy-loaded via dynamic script injection on first mermaid block
  detection (not eagerly loaded on every page view)
- 3-state loader (idle/loading/ready) with callback queue
- Serialized rendering to avoid mermaid internal state corruption
- Theme integration via getComputedStyle reading CSS design tokens;
  re-renders all diagrams on dark/light theme toggle
- Source preserved in data-mermaid-source for theme re-rendering
- Error handling with source code fallback display
- securityLevel: "strict" (DOMPurify) for SVG XSS prevention
- THIRD-PARTY-NOTICES updated with mermaid MIT license

* fix: mermaid render fixes from Copilot review

- Call result.bindFunctions(container) after SVG insertion for
  interactive diagram elements (click handlers, links, tooltips)
- Clear mermaid-error class on successful render (fixes stale error
  styling after theme toggle re-render)
- Clear mermaid-error in reRenderAllMermaid before re-render sequence
- Restructure postRenderMarkdown so mermaid rendering runs even when
  highlight.js is unavailable (hljs guard changed from early return
  to conditional block)
2026-03-15 02:09:10 -07:00
Patrick Buckley 4152ea2352 fix: widen code fence regex and skip auto-detect on unlabeled blocks
- Regex changed from (\w*) to ([^\s`]*) to capture language names with
  special chars (c++, c#, objective-c, shell-session)
- Alias map normalizes c++ → cpp, c# → csharp, f# → fsharp for CSS
  class names
- Empty language no longer emits class="language-", preventing
  highlight.js auto-detect across all 37 bundled languages on
  unlabeled code blocks (performance fix for large blocks)
2026-03-15 02:04:58 -07:00
Patrick Buckley 44cc14b46f feat: syntax highlighting via highlight.js with code block variants (#68)
Integrate highlight.js 11.11.1 (self-hosted, BSD-3-Clause, ~125KB) for
language-aware syntax highlighting on fenced code blocks.

- postRenderMarkdown() hook applies highlighting at stream_end and
  history load — not during streaming (innerHTML replaced per token)
- Custom theme using CSS design tokens (auto-adapts dark/light)
- Code block variants: diff (green/red line coloring), bash/shell
  (terminal left-border), ascii/text/plaintext (no highlighting)
- Class prefix changed from lang- to language- (CommonMark standard)
- Graceful degradation when highlight.js unavailable
- THIRD-PARTY-NOTICES file for bundled dependency attribution
- pyproject.toml package-data glob for vendored hljs directory
2026-03-15 01:36:57 -07:00
Patrick Buckley 83b0cde32f feat: GFM extended syntax renderers (callouts, footnotes, definition … (#66)
* feat: GFM extended syntax renderers (callouts, footnotes, definition lists)

Add three GFM extended syntax features to the server web UI markdown
renderer, with no external library dependencies (pure JS/CSS):

- Callouts/Alerts: > [!NOTE], [!TIP], [!IMPORTANT], [!WARNING], [!CAUTION]
  with color-coded left borders, icons, and recursive markdown body
- Definition Lists: Term + `: Definition` pattern with multi-term support
- Footnotes: [^id] inline superscript references, [^id]: definitions
  collected into a numbered section with bidirectional navigation

Design review fixes: scoped footnote IDs (prevent collisions across
messages), aria-hidden on callout icons, aria-label on callout containers,
focus-visible on footnote links, smooth-scroll footnote navigation.

* fix: use getElementById for footnote scroll to handle special chars in IDs

querySelector throws on fragment IDs containing &, . or : characters
(produced by escapeHtml on footnote labels). getElementById accepts any
string and is the correct API for ID-based element lookup.
2026-03-15 01:33:54 -07:00
Patrick Buckley 2ef8a8711b feat: rich markdown renderer with LaTeX support for server web UI (#65)
* feat: rich markdown renderer with LaTeX support for server web UI

Extract markdown rendering from app.js into dedicated renderer.js with
full GFM support: tables (alignment, hover, striping), nested lists,
task list checkboxes, nested blockquotes, images (click-to-load for
privacy), and inline/display LaTeX math via self-hosted KaTeX 0.16.38.

Security: escape image/link URLs to prevent attribute injection, block
javascript: scheme in links, add rel="noopener noreferrer", images
require explicit click to load (no automatic external requests).

Accessibility: scope="col" on table headers, tabindex on scrollable
table containers, aria-labels on task checkboxes and image placeholders,
KaTeX error color override for WCAG AA contrast, reduced-motion support.

* fix: address code review — XSS hardening and list type splitting

- Escape all text through escapeHtml() at start of inlineMarkdown()
  so only renderer-generated tags appear in innerHTML (prevents raw
  HTML/script injection from LLM output)
- Replace inline onclick handler on image placeholders with data-*
  attributes and delegated DOM event listeners (prevents entity
  decoding XSS in event handler attributes)
- Split list blocks into separate <ul>/<ol> when marker type changes
  at the same indent level (mixed ordered/unordered sequences)
2026-03-15 00:04:01 -07:00
Patrick Buckley 3658b77de8 feat: Discord content catch-up + bidirectional notification replies (… (#64)
* feat: Discord content catch-up + bidirectional notification replies (#64)

Two improvements to the Discord channel adapter:

1. Fix intermittent dropped responses caused by a race between the
   bridge's two independent SSE connections (global SSE detects idle
   before per-ws SSE delivers all content tokens). The bridge now
   accumulates content in _ws_content_buffer and attaches it to
   TurnCompleteEvent.content. The Discord bot uses this as a catch-up
   when streaming events were missed.

2. Bidirectional notification replies — when the notify tool sends a DM,
   the message is tracked with the originating ws_id. Users can reply to
   the DM and the reply is routed to the workstream. The response is
   forwarded back to the DM, with the response itself tracked for
   multi-turn conversations. Includes user identity verification,
   stale notification feedback, and FIFO-capped tracking (100 entries).

* fix: address Copilot review — re-insert on unlinked user, deque buffer

- Re-insert _notify_ws_map entry when resolve_user returns None so the
  user can retry after linking (same pattern as user-mismatch re-insert)
- Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len()
  returns characters, not bytes)
- Use deque + running total for O(1) popleft instead of list.pop(0)
2026-03-14 23:58:27 -07:00
Patrick Buckley 83577739e0 chore: bump version to 0.6.2
- MCP admin tab: database-backed server management, hot-reload,
  reconcile, unified config view, paste-based import
- Catch-up migration for builtin-admin permissions (017)
- `[all]` optional dependency group (@Burhan-Q)
v0.6.2
2026-03-14 17:25:58 -07:00
Burhan 71d13936fe add "all" optional dep (#61) 2026-03-14 17:05:49 -07:00
Patrick Buckley 0cd061196c fix: catch-up migration ensuring builtin-admin has all 20 permissions (#63)
Migrations 011-016 each appended a permission to the builtin-admin role
via conditional UPDATE, but on some deployments these never applied.
Migration 017 idempotently sets the complete permission string rather
than appending incrementally.

Must be merged after feat/admin-mcp-servers (migration 016).
2026-03-14 17:03:21 -07:00
Patrick Buckley 19abc0cc65 feat: admin MCP Servers tab — database-backed MCP server management w… (#62)
* feat: admin MCP Servers tab — database-backed MCP server management with live status

Add MCP Servers admin tab (14th tab, System group) for managing MCP server
definitions via the database instead of static JSON config files.

Storage: `mcp_servers` table (migration 016), 6 CRUD methods on both SQLite
and PostgreSQL backends, `MCP_SERVER_MUTABLE` field allowlist.

Config priority chain: DB rows (if any enabled) → CLI `--mcp-config` →
`mcp.config_path` setting → none. Nodes auto-load from DB on startup via
`load_mcp_config(storage=)`.

Hot-reload: `reconcile_sync(storage)` diffs running servers against DB —
adds missing, removes stale, reconnects changed. `_db_managed` set tracks
DB-sourced servers so config-file servers (MCP_CONFIG env) are never removed
by reconcile. Per-server `AsyncExitStack` for clean teardown.

Reload pattern: console writes to DB then signals nodes via
`POST /_internal/mcp-reload` (update by reference, no config payload).

Console admin API: 7 endpoints under `/v1/api/admin/mcp-servers` (CRUD +
reload + import), `admin.mcp` permission, secret masking (env/headers
replaced with *** unless ?reveal=true), audit log sanitization.

Unified view: tab merges DB-managed servers with config-sourced servers
detected on nodes. Config servers shown as read-only rows with "config"
badge — no edit/delete.

Admin UI: 7-column grid with magenta status dots, transport badges,
single-column create/edit modal, paste-based JSON import (mcpServers format),
detail modal with per-node status. Mobile 3-column collapse, reduced-motion
support, backdrop-click dismiss, focus trapping.

SDKs: 7 methods on Python (async+sync) and TypeScript SDKs.

Also fixes: Settings tab permission gate (admin.users → admin.settings),
_ALL_PERMISSIONS list in governance.js (5 missing permissions added),
_internal/mcp-reload added to APPROVE_PATHS.

Docs: architecture.md (14 tabs), api-reference.md (7 endpoints),
20-mcp-architecture.puml updated with admin-driven lifecycle.

66 new tests (2232 total).

* fix: address Copilot review feedback on MCP admin PR

- Docs: fix "merges both sources" → "first-match-wins priority" (architecture.md)
- Validation: require command for stdio, url for streamable-http transport
- Validation: check args/headers/env types in import handler before storing
- Schema: add transport/command/url to McpServerStatus, source to McpServerDetail
- Thread safety: move all remove_server_sync mutations onto MCP event loop thread
- Regenerate OpenAPI JSON snapshots for TypeScript SDK
2026-03-14 17:02:50 -07:00
Patrick Buckley c5cdfc8f44 chore: bump version to 0.6.1 v0.6.1 2026-03-14 13:19:40 -07:00
Patrick Buckley 8895bf07eb feat: admin Settings tab — form-based editor replacing "coming soon" … (#60)
* feat: admin Settings tab — form-based editor replacing "coming soon" stub

Section-grouped layout with collapsible headers for all ~40 ConfigStore
settings (model, session, tools, server, mcp, ratelimit, health, judge,
memory). Type-appropriate inputs: CSS toggle for bools, number with
min/max/step, select for choices, text for strings. Secret fields shown
read-only. Source badge (storage/default), amber restart indicator.

Inline save per field with dirty detection, row flash on success, reset
to default via styled confirm modal. Full WCAG keyboard accessibility
(Enter/Space on section headers, aria-labels, focus-visible). Mobile
responsive single-column at <700px. Reduced-motion safe.

* fix: Settings tab polish — help tooltips, context_window auto-detect, UX fixes

Settings UI:
- Help tooltips: ? button on ~25 settings with plain-English explanations
  and optional reference links (arXiv, Fowler, MCP spec). Click to toggle
  popover, Escape to dismiss, aria-expanded for accessibility.
- Sections start collapsed for scannable overview.
- Restart badge: hidden by default, shows when dirty, persists after save
  with amber glow. Positioned left of source badge.
- Secret row alignment fixed (transparent border matches input box model).
- Docs link in toolbar → Swagger UI Settings section.
- Number inputs: spin buttons hidden (Firefox/WebKit), empty value guard,
  numeric dirty detection (0.1 vs 0.10 no longer false positive).
- Secret reset button enabled when source=storage (clear legacy overrides).
- Space key repeat guard on section headers.
- Sidebar: sticky + max-height:100vh, no longer stretches with content.

Backend:
- context_window default changed from 131072 to 0 (auto-detect). Fallback
  lowered from 131K to 32K (realistic for local models when detection fails).
  Session normalizes 0→32768 defensively.
- Settings registry: help + reference_url fields on SettingDef, richer
  descriptions for model/session/tools/judge/memory settings.
- Schema API includes help + reference_url.
- Bootstrap system prompt: added Runtime Settings section.

Docs: tab counts updated to 13 across README, architecture, console, governance.
2026-03-14 13:12:40 -07:00
Patrick Buckley 101afd84da feat: database-backed settings (ConfigStore) with admin API (#59)
* feat: database-backed settings (ConfigStore) with admin API

Replace config.toml for non-bootstrap settings on the server with a
database-backed ConfigStore.  ~40 settings across model, session,
tools, server, mcp, ratelimit, health, judge, and memory sections are
now managed via the admin Settings API.  CLI flags for these settings
removed from the server entry point (CLI standalone tool unchanged).

Storage: system_settings table (migration 015) with composite PK
(key, node_id) for per-node overrides.  ON CONFLICT upsert in both
SQLite and PostgreSQL.  admin.settings permission granted to
builtin-admin role.

Settings registry (settings_registry.py): code-defined catalog of
all known settings with types, defaults, validation, descriptions.
Registry defaults aligned with previous argparse defaults.

ConfigStore (config_store.py): thread-safe in-memory cache loaded
from storage on init.  Lock-free reads via dict snapshot swap.
reload() for hot-reload via internal endpoint.

Secret settings (judge.api_key) blocked from write via admin API
(403) — must be configured via config.toml or env vars.

warn_migrated_settings() logs warnings for config.toml keys that
overlap with ConfigStore-managed settings.

Console admin API: GET /v1/api/admin/settings (list with effective
values), GET .../schema (registry catalog), PUT .../{key} (update),
DELETE .../{key} (reset to default).  Audit trail on mutations.

MQ: ConfigChangeEvent for cross-node cache invalidation (emission
from console deferred to bridge integration).

Python + TypeScript SDK methods.  63 new tests.  Feature docs at
docs/settings.md, PlantUML diagram 24-settings-architecture.

* fix: address PR review — config-reload scope, registry defaults, doc alignment

- config-reload endpoint requires approve scope (was write)
- config-reload handler is sync def (avoids blocking event loop)
- reasoning_effort passes empty string through (removes `or "medium"`)
- ratelimit.requests_per_second changed to float (matches RateLimiter)
- session.retention_days allows 0 (disable pruning)
- ratelimit.trusted_proxies added to registry + wired in server
- admin_list_settings filters to global settings only (no node_id ambiguity)
- admin_update_setting validates "value" key presence (400 if missing)
- Console config change fans out reload to nodes directly (no MQ dep)
- Docs aligned with actual API response shapes and masking ("***")
2026-03-14 11:42:18 -07:00
Patrick Buckley efd98712e9 feat: [memory] admin panel Memories tab — browse, search, inspect, de… (#57)
* feat: [memory] admin panel Memories tab — browse, search, inspect, delete

Add 13th admin tab in the Observe group for cluster-wide memory
management.  List view with type/scope filter dropdowns and debounced
search input.  Detail modal shows full metadata grid and scrollable
content block.  Delete from both list row and detail modal with
confirmation and audit trail.

Permission-gated behind admin.memories.  Escape key, backdrop click,
and focus trap wired for the detail modal.  Mobile responsive: hides
description and updated columns below 700px.

* fix: memory detail modal — focus, delete safety, CSS shorthand order

Address Copilot review feedback: move focus to close button on modal
open for keyboard accessibility, disable delete button and clear stale
handler during loading/error states to prevent wrong-memory deletion,
and fix font shorthand/font-size ordering in toolbar filter styles.
2026-03-14 02:42:43 -07:00
Patrick Buckley 67f43a7ee0 feat: [memory] REST API endpoints + SDK methods + docs (#56)
* feat: [memory] REST API endpoints + SDK methods + docs

Server API (4 endpoints):
- GET /v1/api/memories — list with type/scope/scope_id/limit filters
- POST /v1/api/memories — save (upsert) with validation
- POST /v1/api/memories/search — search by query (read scope)
- DELETE /v1/api/memories/{name} — delete by name+scope

Console admin API (4 endpoints):
- GET /v1/api/admin/memories — list all memories
- GET /v1/api/admin/memories/search — search with ?q= param
- GET /v1/api/admin/memories/{memory_id} — get by ID
- DELETE /v1/api/admin/memories/{memory_id} — delete by ID with audit

Storage: add delete_structured_memory_by_id, add mem_type filter to
count_structured_memories. Auth: memory DELETE requires write scope,
admin.memories permission added to valid set + builtin-admin role.

Python SDK: list_memories, save_memory, search_memories, delete_memory
on both server (async+sync) and console (async+sync) clients.

TypeScript SDK: matching methods + types on both clients.

Pydantic schemas with Literal type/scope validation, OpenAPI endpoint
specs on both servers. 33 endpoint tests + 8 auth scope tests.

Docs: docs/memory.md feature guide, api-reference.md endpoint docs,
23-memory-architecture.puml diagram.

Also fixes stray `total: int` on CreateChannelUserRequest.

* fix: [memory] address PR review — cross-user scope, schema types, snapshots

Security: user-scoped memory endpoints now bind scope_id to the
authenticated user's identity.  Providing a mismatched scope_id
returns 403, preventing cross-user memory access on all 4 server
endpoints.

Schema: MemoryInfo response uses MemoryType/MemoryScope Literals.
SearchMemoriesRequest uses filter Literals (empty string allowed).
Limit query params declare schema_type="integer" for correct OpenAPI.

Regenerate sdk/typescript/openapi-{server,console}.json snapshots.
Update count_structured_memories docstring for mem_type param.
Fix fallback response to use normalized name after save.

6 new security tests for user-scope access control.
2026-03-14 02:28:47 -07:00
Patrick Buckley 2888e8ce0a feat: MCP cluster-ops example — reference MCP server + SDK implementa… (#55)
* feat: MCP cluster-ops example — reference MCP server + SDK implementation

Standalone MCP server under examples/mcp-cluster-ops/ that exposes
tools for executing commands across a Turnstone cluster via the MQ
client SDK. Serves as a reference implementation for both MCP server
patterns (FastMCP, lifespan, tool handlers) and TurnstoneClient usage.

4 tools: list_nodes, run_on_node, run_on_nodes, run_on_all_nodes.
Parallel dispatch via asyncio.gather, raw ToolResultEvent output
capture, UTF-8 safe truncation, input validation, concurrency caps.

35 tests, ruff clean, mypy --strict clean.

* fix: address review feedback on MCP cluster-ops example

- Remove REDIS_SSL support (RedisBroker doesn't accept ssl kwarg)
- Move max-nodes check from _dispatch_parallel into tool handlers
  for consistent error shape (always returns {"error": ...} object)
- Propagate KeyboardInterrupt/SystemExit from asyncio.gather instead
  of swallowing them as per-node failures
- Fix _truncate omitted bytes count to reflect actual bytes dropped
  after multi-byte boundary adjustment
- Apply strip/dedup to node IDs in run_on_all_nodes (matching
  run_on_nodes behavior)
- Add __name__ guard to __main__.py
- Fix misleading UTF-8 byte count comment in tests
2026-03-14 02:23:43 -07:00