1818 Commits

Author SHA1 Message Date
Patrick Buckley 6d0dc6df94 chore: standardize logging to structlog get_logger across 19 modules (#150)
Replace bare `import logging` / `logging.getLogger(__name__)` with
`from turnstone.core.log import get_logger` / `get_logger(__name__)`
across all core modules and server.py. This enables structured log
context injection (node_id, ws_id, request_id) in modules that
previously used plain stdlib logging.

Also adds _ensure_stdlib_factory() to log.py for pytest caplog
compatibility when configure_logging() hasn't been called.

Renames `logger` to `log` in skill_sources.py for naming consistency.
2026-03-21 04:29:40 -07:00
Patrick Buckley 7e680ee883 chore: remove dead code — chat.py, singular touch, unused vars, inline imports (#148)
* chore: remove dead code — chat.py shim, singular touch method, unused vars

- Delete turnstone/chat.py (backward-compat re-export shim, zero importers)
- Remove touch_structured_memory() singular method from protocol + both
  backends + 6 tests (only plural batch form is used)
- Remove unused _last_err variable in _compact_messages
- Remove redundant _AGENT_AUTO_TOOLS / _TASK_AUTO_TOOLS class aliases,
  use module-level constants directly
- Consolidate ~76 inline schema imports to top-level in both storage
  backends (channel_users, channel_routes, oidc_*, scheduled_tasks,
  watches, services)

Net: -219 lines

* fix: address review — remove stale inline timedelta imports in prune_task_runs

timedelta is already imported at module scope in both backends.
2026-03-21 04:29:37 -07:00
Patrick Buckley e950219246 docs: add beta status warning to README
Mark platform as experimental beta with explicit disclaimers: no
guarantees of determinism, reliability, or backward compatibility.
Advise thorough evaluation before deployment.
2026-03-21 03:43:13 -07:00
Patrick Buckley b3764a8035 fix: critical reliability fixes for production readiness (#147)
* fix: critical reliability fixes for production readiness

C1: Add 1-hour timeout to _approval_event.wait() and _plan_event.wait()
    to prevent permanent worker thread hangs when users disconnect.

C2: Atomically check-and-start worker thread under Workstream._lock to
    prevent race condition where two concurrent send_message requests
    spawn duplicate workers on the same non-thread-safe ChatSession.

C3: Bound _watch_pending queue to maxsize=20 to prevent OOM under
    heavy watch load with busy workstreams.

H1: Add timeout to proc.wait() (10s) and stderr_thread.join() (5s)
    after SIGKILL to prevent indefinite hang on D-state processes.

H2: Protect _pending_verdicts with _ws_lock at all three mutation sites
    (reset in approve_tools, append in on_intent_verdict, swap-and-clear
    in resolve_approval) to prevent lost verdicts from concurrent
    judge daemon and approval threads.

H3: Bound global SSE queue to maxsize=10000 with put_nowait() and
    contextlib.suppress(queue.Full) for backpressure. Prevents
    unbounded memory growth when fanout thread is overloaded.

H4: Bridge SSE threads for closed workstreams now check ws_id membership
    in _ws_threads before reconnecting, preventing thread leak on
    workstream close.

* fix: address review — verdict lock consistency, watch queue non-blocking, SSE drop logging

- Move _last_verdict_decision set inside _ws_lock in resolve_approval()
  so swap+decision is atomic with on_intent_verdict() reads
- Read _last_verdict_decision under _ws_lock in on_intent_verdict()
- Build heuristic_verdicts locally then assign under lock in approve_tools()
- Use resolve_approval() for timeout path so verdicts are updated consistently
- Watch queue producer uses put_nowait with log on Full (prevents WatchRunner hang)
- Global SSE state broadcasts log on queue.Full instead of silent suppress
- Plan event wait also gets 1-hour timeout (same class of bug as approval)
2026-03-21 03:28:01 -07:00
Patrick Buckley 756c4d8929 fix: guard against CancelledError on MCP startup future (Python 3.14) (#146)
On Python 3.14, Future.exception() raises CancelledError on cancelled
futures instead of returning None. Check future.cancelled() before
calling exception() to prevent crash when the MCP event loop shuts down
before _connect_all completes.
2026-03-21 01:06:11 -07:00
Patrick Buckley 04c50568e9 feat: add priority column for skill ordering control (#144)
* feat: add priority column for skill ordering control

Add priority INTEGER DEFAULT 0 column to prompt_templates (migration
024). Skills with activation="default" are now ordered by priority ASC,
name ASC instead of name-only. Admins can set priority via create/update
API. Lower values run first. Priority is editable on readonly/installed
skills. 4 new tests. Python SDK, TypeScript SDK, and Pydantic models
updated.

* fix: address review — apply priority ordering to list_default_templates

list_default_templates() still ordered by name only, so priority had
no effect on default skill execution order. Update both SQLite and
PostgreSQL backends to order by (priority, name).

* fix: address review — regenerate OpenAPI snapshot, add default template ordering test

Regenerate openapi-console.json to include priority field on skill
models. Add test_list_default_templates_ordered_by_priority to verify
the execution path for default skills respects priority ordering.
2026-03-21 00:53:51 -07:00
Patrick Buckley 3bf220c503 fix: use approval_label for per-tool always-approve in CLI and bridge (#143)
* fix: use approval_label for per-tool always-approve in CLI and bridge

The server stores approval_label (e.g. mcp__server__tool) for per-tool
auto-approve, but CLI and bridge extracted only func_name (bare tool
name). This caused always-approve decisions to not carry over across
access paths. Align CLI and bridge to prefer approval_label with
func_name fallback, matching the server's WebUI.approve_tools() pattern.

* fix: address review — exclude errored items from bridge auto-approve check

Filter out items with error set from the auto-approve subset check,
matching the server's WebUI.approve_tools() behavior. Prevents
policy-denied items from affecting auto-approve decisions.
2026-03-21 00:42:33 -07:00
Patrick Buckley 4b853e329e test: verify skill_id/skill_version populated in workstreams table (#145)
The workstreams.skill_id and skill_version columns were already being
populated correctly (wired in the skills unification PR #106). Add two
tests confirming: lineage columns set when skill is applied, and
defaults when no skill is used. Check off the PROGRESS.md item.
2026-03-21 00:40:30 -07:00
Patrick Buckley 29ffdc36d0 test: add governance SDK integration tests against real Starlette app (#142)
* test: add governance SDK integration tests against real Starlette app

24 TestClient-based tests verifying round-trip serialization of SDK
governance methods (roles, policies, orgs) against actual route
handlers with SQLite storage. Covers create/list/update/delete
lifecycles, error cases, and Pydantic model field validation.

* fix: address review — close AsyncClient in sdk_client fixture teardown

Convert sdk_client fixture to async context manager so the httpx
AsyncClient is properly closed after tests, avoiding resource leak
warnings.
2026-03-21 00:31:53 -07:00
Patrick Buckley ada8b80509 test: add MCP reload and reconcile endpoint integration tests (#141)
* test: add MCP reload and reconcile endpoint integration tests

11 new tests covering POST /v1/api/admin/mcp-servers/reload (console)
and POST /v1/api/_internal/mcp-reload (node). Verifies reconcile_sync
invocation, fan-out results, permission checks, missing storage
handling, and mixed node error propagation.

* fix: address review — lazy-import internal_mcp_reload to avoid heavy module load

Move turnstone.server import inside _routes_with_internal() helper so
the full server module (which reads UI static assets) is only loaded
when node-side endpoint tests actually run, not during test collection.
2026-03-21 00:14:50 -07:00
Patrick Buckley 1f47ca62de fix: validate OIDC issuer URLs against SSRF before discovery fetch (#140)
* fix: validate OIDC issuer URLs against SSRF before discovery fetch

Add validate_issuer_url() that rejects private/loopback/link-local IPs,
non-HTTPS (except localhost for dev), embedded credentials, and
unresolvable hostnames. Called before the HTTP fetch in discover_oidc()
so the request is never made for invalid URLs. 17 new tests.

* fix: address review — use is_global, redact userinfo, catch ValueError

Use `not addr.is_global` instead of individual range checks to cover
all non-routable addresses (CGNAT, unspecified, multicast). Redact
credentials from error messages to prevent log leakage. Catch ValueError
from ip_address() for zone-indexed IPv6 addresses.
2026-03-21 00:14:46 -07:00
Patrick Buckley 83d9233304 test: skill session config application to workstreams (#139)
* test: skill session config application to workstreams

13 TestClient-based integration tests verifying that skill session
config fields (model, temperature, token_budget, auto_approve,
allowed_tools, reasoning_effort, agent_max_turns) are correctly applied
to ChatSession and WebUI when creating a workstream with a skill.

Covers: individual fields, combined application, disabled/unknown skill
rejection, zero-value no-ops.

* fix: address review — pass skill kwarg, clarify no-op test assertions

Pass skill=kwargs.get("skill") into ChatSession in test factory to
match production behavior. Clarify zero-value no-op test docstrings
to document they verify the handler's guard conditions, not observable
state changes.
2026-03-20 23:09:59 -07:00
Patrick Buckley bf06102d37 fix: memory access tracking and BM25 context caching (#138)
* fix: memory access tracking and BM25 context caching

Add touch_structured_memory/touch_structured_memories to storage
protocol + SQLite/PostgreSQL backends. Bumps last_accessed and
access_count on memory retrieval (BM25 injection + search results).
9 new storage tests.

Cache the scored BM25 memory context string on ChatSession, invalidated
on memory save/delete. Eliminates ~12 redundant storage queries + index
rebuilds per session lifecycle.

* fix: address review — deduplicate keys in touch facade, clarify contract

Deduplicate keys in the memory.py facade before calling storage so each
distinct memory is touched at most once. Update protocol docstring to
clarify per-call increment semantics. Add deduplication unit test.

* fix: replace unused-import test with real batch duplicate test

Replace facade dedup test (which only tested Python set logic) with a
real storage-level test that verifies duplicate keys each increment
access_count. Fixes ruff F401 lint failure.
2026-03-20 23:08:32 -07:00
Patrick Buckley e015b4512d fix: return typed Pydantic models from SDK skill methods (#137)
Skill methods on TurnstoneConsole and AsyncTurnstoneConsole returned
dict[str, Any] instead of validated Pydantic models. Update list_skills,
create_skill, get_skill, update_skill, list_skill_resources,
create_skill_resource, and install_skill to use response_model= with
ListSkillsResponse, SkillInfo, SkillResourceInfo, and
SkillInstallResponse.
2026-03-20 20:02:06 -07:00
Patrick Buckley 0c1afff7fc test: add _get_registry_url three-tier fallback tests (#136)
8 tests covering the DB setting → config.toml → default URL resolution
chain, including storage errors, empty values, malformed JSON, and
documenting that RuntimeError propagates uncaught through the except
clause.
2026-03-20 20:01:58 -07:00
Patrick Buckley a94051a995 fix: add split pane button to tab bar for discoverability (#135)
* fix: add split pane button to tab bar for discoverability

The split pane feature was only accessible via right-click context menu
or Ctrl+\ keyboard shortcut. Add a subtle split icon (⧉) to the tab
bar that appears at low opacity in single-pane mode. Hidden in
multi-pane mode where pane headers already provide split/close controls.

* fix: address review — change tab-bar from tablist to toolbar role

The tab bar contains both tabs and action buttons (new workstream,
split pane), which is invalid for role=tablist. Change to role=toolbar
which correctly describes a container of mixed interactive controls.

* fix: address design review — WCAG contrast, ARIA structure, mobile

- Drop opacity approach, use border: dashed var(--border) matching
  #new-tab-btn pattern (fixes WCAG contrast failure at 35% opacity)
- Nest tabs in #tab-list[role=tablist] inside toolbar (fixes invalid
  role=tab children inside role=toolbar)
- Hide split button on mobile (<600px) where splits can't work
- Add aria-keyshortcuts to both action buttons
2026-03-20 20:01:35 -07:00
Patrick Buckley 2f906ea1f9 fix: enable output guard in CLI mode (#134)
* fix: enable output guard in CLI mode

The heuristic output guard (credential redaction, prompt injection
detection) only ran in server mode because cli.py never constructed a
JudgeConfig. Additionally, the guard condition in session.py required
enabled=True, coupling the zero-cost heuristic (<5ms) to the full LLM
judge.

Wire JudgeConfig from existing CLI args into the session factory.
Decouple the output_guard condition from the enabled flag so the
heuristic guard runs even when the LLM judge is disabled via --no-judge.

* fix: address review — pass config.toml judge fields to CLI JudgeConfig

apply_config() merges [judge] section from config.toml into args as
judge_base_url and judge_api_key. Pass these through to JudgeConfig so
the CLI respects config.toml judge settings (e.g. separate judge
endpoint).
2026-03-20 20:01:31 -07:00
Patrick Buckley b61bfd1aa6 fix: validate URL scheme after MCP registry template substitution (#133)
* fix: validate URL scheme after MCP registry template substitution

resolve_install_config() substitutes user-provided values into URL
templates via string replacement without validating the resulting URL.
Add urlparse check after substitution to reject non-HTTP(S) schemes,
preventing SSRF-style redirection through crafted template variables.

* fix: address review — reject empty hostname and embedded credentials

Add hostname presence check and userinfo rejection after URL scheme
validation. Prevents URLs like https:///path (no host) and
https://user:pass@host (credential leakage in config). Two new tests.
2026-03-20 20:01:26 -07:00
Patrick Buckley 19c3a48b10 fix: server startup stampede — timeout model detection, non-fatal PG … (#132)
* fix: server startup stampede — timeout model detection, non-fatal PG migrations

detect_model() blocked the main thread for up to 400s when the LLM backend
was unreachable (OpenAI SDK default: 600s read timeout × 2 retries × TCP
retransmit). Cap startup detection at 10s with no retries — the
BackendHealthMonitor handles ongoing availability probing after startup.

PostgreSQL migrations via _run_with_pg_lock() crashed the server on lock
contention when 10 containers stampeded the advisory lock simultaneously.
Wrap in try/except matching the SQLite path — the entrypoint script already
runs migrations before the server process starts.

Health check start_period increased from 15s to 60s to accommodate the
startup sequence under load.

* fix: address review — narrow PG migration except, add detect_model test

Narrow the PG migration except clause to (OSError, EOFError) so DDL
errors still propagate. Add two unit tests for detect_model() verifying
with_options(timeout=10, max_retries=0) is called and that connection
errors in non-fatal mode return (None, None).
2026-03-20 20:01:21 -07:00
Patrick Buckley 414eb52d67 feat: raise scaling limits for 1000-node clusters (#129)
* feat: raise scaling limits for 1000-node clusters

Raise hardcoded limits throughout the codebase so clusters up to 1000
nodes work without configuration changes.

Scaling limits:
- max_workstreams default 10 → 50 (configurable via settings)
- Console fan-out concurrency 50 → 200 (configurable: cluster.node_fan_out_limit)
- MCP max servers 50 → 200 (configurable: cluster.mcp_max_servers)
- Console SSE queue 500 → 2000, server global SSE queue 500 → 1000
- httpx proxy pool: explicit max_connections on both proxy clients
- PostgreSQL pool 5+10 → 2+3 per process (right-sized for short-burst queries)
- Redis pool: explicit max_connections=200 on both sync and async brokers

Performance optimizations:
- Redis list_nodes(): replace N+1 SCAN+GET with SCAN+MGET
- Collector poll: raise thread pool to 200 (matches fan-out limit)
- Server SSE: dedicated ThreadPoolExecutor(200) for queue polling
- Fan-out: new get_all_nodes() removes hardcoded limit=1000 ceiling

Bug fixes:
- Settings reload notification was silently failing (called .get() on tuple)
- Watch fan-out only queried 500 nodes instead of full cluster

New cluster settings (configurable via admin Settings tab):
- cluster.node_fan_out_limit (default 200, range 10-1000)
- cluster.mcp_max_servers (default 200, range 1-2000)

Adds docs/pgbouncer.md for PostgreSQL connection pooling at scale.
Adds ddgStressCluster compose profile (100 nodes, 10 groups of 10).
Updates architecture, console, docker, settings, and API reference docs.

* fix: add image tag to compose anchors to avoid redundant builds

All cluster/stress services inherit `build:` from the anchor, causing
Docker to attempt 200+ separate builds. Adding `image: turnstone:local`
means Docker builds once and all services reuse the cached image.

* fix: address Copilot review feedback on scaling PR

- Remove magic number in get_all_nodes (limit=None instead of 2**31)
- Size httpx proxy pool from fan-out limit setting (not hardcoded 250)
- Cap cluster.node_fan_out_limit max_value to 500, mark restart_required
- Convert _publish_config_change from sync to async (was blocking event loop)
- Use shutdown(wait=True, cancel_futures=True) for SSE executor

* fix: add PostgreSQL env vars to cluster bridge anchor

Bridges initialize storage for auth/migrations but the bridge anchor
was missing TURNSTONE_DB_BACKEND and TURNSTONE_DB_URL, causing all
bridges to fall back to SQLite. With 100 bridges sharing the same
volume, concurrent SQLite migrations corrupt the database.

* fix: address Copilot round 2 + PG connection exhaustion at startup

Copilot feedback:
- Raise cluster.node_fan_out_limit max_value to 1000 (matches target)
- Cache fan-out limit on app.state at startup instead of re-reading DB
  per request (pool and semaphore now use the same value consistently)
- Remove unused params from _publish_config_change

Stress cluster fix:
- Raise PG max_connections to 300 (configurable via POSTGRES_MAX_CONNECTIONS)
  to handle 200 processes connecting simultaneously at startup
- Bump PG shared_buffers to 128MB and memory limit to 1G to match
- Add DB env vars to production bridge service

* fix readme

* fix: startup resilience for large clusters

Server no longer crashes when LLM backend is unreachable at startup.
detect_model() accepts fatal=False, returning (None, None) so the
server starts in degraded mode with circuit breaker open. The health
monitor will detect when the backend becomes available.

Migration runner retries with jittered exponential backoff (up to 10
attempts) when PostgreSQL rejects connections during startup stampedes.

Collector httpx pool sized to match poll workers (was using default of
100 connections with 200 workers).

Also addresses Copilot round 2:
- Raise cluster.node_fan_out_limit max_value to 1000
- Cache fan-out limit on app.state at startup
- Remove unused params from _publish_config_change
- Add DB env vars to production bridge service

* fix: replace silent error suppression with structured logging

Audit and fix 30+ instances of silently swallowed exceptions across 8
files. No-raise contracts are preserved — all changes add logging
while keeping the same return-value behavior.

memory.py (26 changes):
  Every storage operation now logs on failure. Previously the entire
  persistence facade had zero logging — messages, workstream state,
  and structured memories could silently stop being saved.

server.py:
  Usage recording failures now log at warning (was pass).
  Global SSE fan-out errors log at debug (was pass).

console/server.py:
  Config reload notification logs per-node failures at warning.
  Settings read fallbacks log at warning with the default value used.

auth.py:
  User existence check logs at warning (was pass).
  Setup rollback failures log at error (was suppress).
  OIDC state cleanup logs at debug (was suppress).

mcp_client.py:
  DB-managed MCP server list failure logs at warning (was pass).

collector.py:
  Node poll failure upgraded from debug to warning with exc_info.
  Health fetch failure logs at debug with exc_info (was silent).

bridge.py:
  Best-effort plan rejection logs at warning (was suppress).
  Malformed SSE data logs at debug (was suppress).

session.py:
  Tool output UI callback failure logs at debug (was suppress).

* fix: stagger collector poll with deterministic per-node jitter

Each node gets a stable offset within the first half of the poll
interval, derived from hashing the node_id against a Mersenne prime
(2^31 - 1). This spreads HTTP requests across the cycle instead of
firing all 100+ at the same instant.

Also raises poll interval from 10s to 15s and HTTP timeout from 5s
to 30s for large-cluster resilience.

* fix: add startup jitter to bridge heartbeat and health monitor probe

Bridge heartbeat: deterministic per-node jitter (from node_id hash)
spreads initial registration across the first quarter of the heartbeat
TTL. At 100 bridges with 60s TTL, heartbeats spread across 15s instead
of all firing at T=0.

Health monitor probe: deterministic per-process jitter (from PID hash)
spreads initial LLM backend probes across half the probe interval. At
100 servers with 30s interval, probes spread across 15s instead of all
hitting the LLM at T=30.

Both use the same Mersenne prime hashing approach as the collector poll
jitter for consistency.

* fix: split collector httpx timeout and raise keepalive pool

Use separate connect/read/write/pool timeouts instead of a single 30s
for all phases. Raise keepalive connections from 50 to 200 so the
collector reuses TCP connections across poll cycles instead of
constantly tearing down and re-establishing them.

* fix: narrow detect_model return type for CLI and eval callers

detect_model() now returns tuple[str | None, int | None] to support
fatal=False. CLI and eval always use fatal=True (the default), which
guarantees a non-None model or SystemExit. Add assert to narrow the
type for mypy.
2026-03-19 04:53:11 -07:00
Patrick Buckley 86b404177b chore: bump version to 0.8.4
- feat: split-pane layout for chat UI (#127)
- fix: enforce CSS min dimensions during split handle drag
- feat: add OpenShell sandbox policy for turnstone-server (#128)
- fix: collector JWT expiry causes silent workstream data wipe (#126)
- fix: auto-titler SSE event + SSE reconnection after restart (#125)
- fix: wire resume_ws through console + expose max_ws in heartbeat (#124)
v0.8.4
2026-03-18 18:13:18 -07:00
Patrick Buckley 10165bb8a1 feat: add OpenShell sandbox policy for turnstone-server (#128)
* feat: add OpenShell sandbox policy for turnstone-server

Curated policy for running turnstone-server inside an OpenShell sandbox
with kernel-enforced security boundaries (Landlock, netns, seccomp).

- Filesystem: workdir read-write, /usr+/etc read-only, /tmp+/dev/null
  read-write, Landlock best_effort compatibility
- Network: default-deny with allowlisted LLM APIs (OpenAI, Anthropic),
  Tavily, skills.sh, GitHub (read-only L7), MCP registry (read-only L7),
  Redis localhost, package registries, curated web_fetch domains
- Git: L7-enforced read-only (info/refs + git-upload-pack only)
- Process: privilege drop to sandbox:sandbox
- Inference routing template for credential isolation (real API keys
  never enter the sandbox, resolved at proxy layer)

* fix: address PR #128 review feedback + add integration guide

Review fixes:
- Use python3 (not python) in usage examples to match binary allowlist
- Fix network_policy → network_policies in comment
- Remove /usr/bin/git from github_api (git uses github.com not
  api.github.com; already covered by git_operations policy)
- Remove pip/uv from bash_network_tools (package_registries already
  covers their PyPI access; no need for StackOverflow/Wikipedia reach)
- Restructure routes.yaml so commented blocks are indented under
  routes: key (uncomment without restructuring YAML)

New: docs/openshell.md covering policy customization, inference routing,
domain allowlisting, MCP subprocess inheritance, and the dual-layer
security model.
2026-03-18 18:09:47 -07:00
Patrick Buckley 5d478573cc fix: enforce CSS min dimensions during split handle drag
Drag ratio bounds were hardcoded at 0.1/0.9 which allowed panes to be
resized below their CSS min-width (200px) / min-height (150px), causing
input areas and text to overflow and clip. Now compute bounds dynamically
from the container size and CSS minimums.
2026-03-18 18:08:35 -07:00
Patrick Buckley 1b24e4717f feat: split-pane layout for chat UI (#127)
* feat: split-pane layout for chat UI

Refactor the server UI from a single-pane global-state design to a
multi-pane architecture with per-workstream Pane instances and a binary
layout tree. Each pane has its own SSE connection, message area, input,
and state (busy, approval, streaming).

Phase 1 — Pane class with 25 prototype methods encapsulating all
per-workstream state. Phase 2 — binary split tree (leaf/split nodes)
with recursive flexbox rendering and drag-to-resize handles. Phase 3 —
keyboard shortcuts (Ctrl+\, Ctrl+Shift+\, Ctrl+Shift+W, Ctrl+Alt+Arrow)
and right-click context menu. Phase 4 — layout persistence via
localStorage.

Key design decisions:
- No duplicate workstreams across panes (split refused if no unused ws,
  auto-close redundant pane on ws deletion)
- Max 6 panes to avoid exhausting browser SSE connections
- Viewport guard prevents splitting below min-width/min-height
- Only focused pane refreshes workstream list on SSE reconnect (prevents
  race when multiple panes disconnect simultaneously)
- Tab click focuses existing pane showing that ws in multi-pane mode
- Pointer events on drag handles for mouse + touch support
- Full a11y: ARIA roles/labels, keyboard nav in context menu, focus
  restoration, prefers-reduced-motion coverage

* fix: address PR #127 review feedback

- Add focusin handler so keyboard focus (Tab) updates focusedPaneId
- Context menu skips interactive elements (textarea, input, links,
  buttons) so native copy/paste and link context menus work
- Split handles get ARIA role=separator, aria-orientation, aria-valuenow,
  keyboard resizing (arrow keys, Home/End), and tabindex=0
- Enforce MAX_PANES limit in deserializeLayout to prevent corrupted
  localStorage from creating too many panes/SSE connections
- Update architecture.md to document split-pane layout
2026-03-18 18:03:46 -07:00
Patrick Buckley 9a2db63c07 fix: collector JWT expiry causes silent workstream data wipe (#126)
* fix: collector JWT expiry causes silent workstream data wipe

The console collector baked a one-time JWT snapshot into its httpx
client headers at startup. After 1 hour (JWT expiry), every poll to
server nodes returned 401. The error JSON was silently parsed as valid
empty data, wiping all workstream state while nodes still appeared
reachable — the cluster showed "10 nodes, 0 workstreams."

Root causes fixed:
- Collector: no auth baked into httpx.Client; per-request headers
  from ServiceTokenManager.token (auto-rotating) or static fallback
- Proxy: same pattern — proxy_client/proxy_sse_client created without
  auth headers; _proxy_auth_headers() injects fresh token per-request
- main(): static token snapshot only passed when no token_manager
  exists, preventing stale JWT from being stored anywhere
- _fetch_node: raise_for_status() before .json() so 401s throw
  instead of returning error JSON as "0 workstreams"
- Auth errors (401/403) logged at warning level for operator visibility

* fix: address PR #126 review — type annotation, regression tests, log messages

Tighten token_manager type from Any to ServiceTokenManager | None.
Add two regression tests verifying 401/403 poll responses preserve
existing workstream data and mark nodes unreachable. Fix misleading
log messages: "jwt_minted" → "token_manager_created" since
ServiceTokenManager mints lazily on first .token access.
2026-03-18 15:45:53 -07:00
Patrick Buckley e159837b74 fix: auto-titler SSE event + SSE reconnection after restart (#125)
* fix: auto-titler SSE event + SSE reconnection after restart

_generate_title() now calls self.ui.on_rename() after persisting the
title, so the tab bar, bridge, and console all update in real time.
Also handles multi-part (vision) content and replaces silent except
with log.debug.

SSE onerror handler now parses the workstreams response, replaces the
stale workstreams map, and switches to the first available workstream
if the current ws_id no longer exists (e.g. after server restart).
Previously it retried the stale ws_id forever.

* fix: address PR #125 review — avoid double reconnect + sync tab bar

Return immediately after switchTab/showDashboard on stale ws_id to
prevent scheduling a redundant connectContentSSE via setTimeout.
Always re-render tab bar after replacing the workstreams map so DOM
stays in sync even when currentWsId is still valid.
2026-03-18 15:14:14 -07:00
Patrick Buckley ec3454ee2e fix: wire resume_ws through console + expose max_ws in heartbeat (#124)
* fix: wire resume_ws through console + expose max_ws in heartbeat

Console create_workstream handler now reads resume_ws from the request
body and passes it to CreateWorkstreamMessage on all three dispatch paths
(pool, auto, explicit). Previously resume only worked via channel router
and direct CLI — the console layer never plumbed it through.

Server /health now includes max_ws from WorkstreamManager. Bridge reads
it on startup and includes it in heartbeat metadata so the console's
_pick_best_node gets accurate capacity instead of always defaulting to 10.
Collector also updates max_ws on subsequent heartbeats (not just discovery).

Schemas, Python SDK, TypeScript SDK, and OpenAPI specs updated. Test mocks
fixed for new max_workstreams property access in /health.

* fix: address PR #124 review — resume_ws tests + max_ws fetch on pre-set node_id

Add _fetch_server_metadata() so bridge reads max_ws from /health even
when node_id is pre-set (skipping _fetch_node_id). Without this, heartbeats
would advertise max_ws=10 regardless of actual server config.

Add 3 test cases verifying resume_ws flows through all three console
dispatch paths (directed, pool, auto-select).
2026-03-18 14:24:10 -07:00
renovate[bot] 5cbb832162 chore(deps): lock file maintenance 2026-03-18 13:20:49 -07:00
renovate[bot] 4e4ae2a91d chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.10.11 2026-03-18 13:20:47 -07:00
renovate[bot] c7d0bac638 chore(deps): update astral-sh/setup-uv digest to 37802ad 2026-03-18 13:20:44 -07:00
Patrick Buckley e86305c143 chore: bump version to 0.8.3 v0.8.3 2026-03-17 17:06:00 -07:00
Patrick Buckley d0fc42195a chore: remove dead code and fix noisy JWT test warnings
Remove unused methods (ToolSearchManager.should_activate, get_all_tools),
dead attributes (_all_tools, _threshold), unused constant (DEFAULT_INTERVAL),
unused Scenario protocol class, and vestigial parameters (judge._evaluate_single
heuristic, SimEngine.simulate_llm_response turn_number). Lengthen JWT test
secrets to >= 32 bytes to suppress InsecureKeyLengthWarning from PyJWT.
2026-03-17 17:02:25 -07:00
Patrick Buckley 760321f7ee refactor: extract _resolve_capabilities and _without_tool helpers
Extract _resolve_capabilities() shared helper so _get_capabilities()
and _run_agent() use the same config-override logic instead of
duplicating inline. Add _without_tool() module-level helper to
deduplicate the tool-filtering listcomp.
2026-03-17 16:49:11 -07:00
Patrick Buckley 693e51f782 fix: address PR #119 review feedback
Add UI error notification and exc_info logging to run_one() exception
handler so tool failures are visible in the frontend. Apply config.toml
capability overrides when gating web_search in _run_agent(), matching
the pattern used by _get_capabilities().
2026-03-17 16:49:11 -07:00
Patrick Buckley ba07409724 fix: isolate parallel tool exceptions + gate web_search without backend
Two bugs: (1) an uncaught exception in one parallel tool call killed the
entire batch via pool.map(), losing all results including successful ones.
Wrap run_one() in try/except so failures return error strings instead of
propagating. (2) web_search was offered to local models even without a
Tavily API key — the model would attempt it, only to fail at execution
time. Filter web_search from _get_active_tools() and _run_agent() when
neither native support nor Tavily is available.

Closes https://github.com/turnstonelabs/turnstone/issues/117
2026-03-17 16:49:11 -07:00
Patrick Buckley c76a61841e fix: PR #118 round 2 — null-safe parser, docs, consistency
- Null-safe extraction for description, license, and compatibility in
  skill_parser.py — YAML bare keys (e.g. `description:`) no longer
  produce the literal string "None"
- Log warning on skill catalog storage failure instead of silent swallow
- Use `enabled == 1` in list_skills_by_activation for consistency with
  other prompt_templates queries in both storage backends
- Add parser tests for YAML null description, license, and compatibility
- Update governance.md: document runtime config editing on installed
  skills, two-column modal layout, SPDX license dropdown, origin badge
2026-03-17 16:09:06 -07:00
Patrick Buckley 341d2f604f fix: address PR #118 review feedback
- Regenerate OpenAPI snapshots (openapi-console.json) to include license
  and compatibility fields in SkillInfo/CreateSkillRequest/UpdateSkillRequest
- Omit version from create/update payloads when blank so server applies
  default "1.0.0" instead of storing empty string
- Push enabled_only + limit filters into list_skills_by_activation storage
  query (protocol, SQLite, PostgreSQL) instead of loading all rows and
  filtering in Python; session.py now passes enabled_only=True, limit=30
- License length cap ([:128]) was already applied in previous commit
2026-03-17 16:09:06 -07:00
Patrick Buckley 3f7f8495d6 feat: skills modal redesign + runtime config editing for installed skills
Redesigns the create/edit/view skill modal into a two-column spec manifest
layout (Identity/Manifest/Deployment | Skill Content) matching the Agent
Skills spec structure. Installed (readonly) skills can now have their runtime
config (model, temperature, token limits, enabled) edited independently of
the locked spec/content fields.

- Two-column spec layout with section headings (Identity, Manifest, Deployment,
  Skill Content); content textarea uses monospace font and fills the column
- h3 section headings for screen-reader nav; h3 UA stylesheet reset in CSS
- Runtime Config collapsible uses 3-column grid; license field is now a select
  of SPDX identifiers (MIT, Apache-2.0, GPL-3.0, AGPL-3.0, etc.)
- Origin badge (cyan) shows source URL for installed skills in view mode
- server.py: _SKILL_RUNTIME_CONFIG_FIELDS frozenset; readonly skills filter
  updates to config-only fields (spec fields silently dropped); audit action
  distinguishes skill.update.config from skill.update; license field capped
  at 128 chars in both create and update paths
- governance.js: spec fields disabled for readonly; config fields always
  editable; Save button shown for all skills (labeled "Save Config" when
  readonly); collapsible state reset between modal opens prevents state leak;
  esk-allowed-tools disabled state driven by auto_approve not readonly
- Tests: spec-only body on readonly skill → 400; config-only → 200 with
  spec fields unchanged; mixed body → config fields applied, spec dropped
2026-03-17 16:09:06 -07:00
Patrick Buckley dc464ac313 feat: Agent Skills standard compliance + frontend spec fields
Brings skills implementation into full compliance with agentskills.io:

Parser:
- Read `allowed-tools` (hyphenated, standard) only; stored as
  `allowed_tools` internally — no underscore fallback
- Reject consecutive hyphens in skill names
- Extract author/version from standard `metadata:` map with top-level
  fallback; null-safe (no "None" string for bare YAML keys)
- Truncate description at 1024 chars, compatibility at 500 chars (spec
  caps) with log warnings
- Lenient parsing mode (lenient=True) for cross-client import: sanitizes
  names, returns None on skip, malformed-YAML colon-value retry
- Type overloads: strict mode returns ParsedSkill, lenient returns
  ParsedSkill | None

Session:
- `<available-skills>` XML catalog in system messages for
  activation="search" skills (disabled ones filtered out, capped at 30)

Tool rename:
- `load_skill` tool → `skill` (JSON, session preparers/executors,
  approval labels, tests, docs)

Storage (migration 023):
- Add `license` and `compatibility` columns to prompt_templates
- skill_license / compatibility params on create_prompt_template across
  protocol, SQLite, PostgreSQL backends
- Add to SKILL_MUTABLE for update_prompt_template

API + server:
- SkillInfo, CreateSkillRequest, UpdateSkillRequest: license +
  compatibility fields
- Create/update/install endpoints extract and persist both fields
- Install endpoint maps parsed.license + parsed.compatibility from
  imported SKILL.md (previously discarded)
- _skill_to_response() includes both fields

Admin UI:
- Create + edit modals: version, license, compatibility fields
- Readonly (imported) skills: "edit" → "view" button, modal title
  "View Skill", all fields disabled, Save hidden, Cancel → "Close",
  collapsibles auto-expand, focus on Close button
- :disabled CSS for dark-theme modal inputs (bg-highlight, cursor
  not-allowed, dimmed text)
- Fix addEventListener stacking on auto-approve checkboxes → .onchange

SDK: license + compatibility on SkillInfo, CreateSkillRequest,
UpdateSkillRequest TypeScript interfaces

Docs: governance.md, judge.md, tools.md, README, diagram updated
2026-03-17 16:09:06 -07:00
Patrick Buckley 52d59cf7b7 chore: bump version to 0.8.2 v0.8.2 2026-03-17 02:20:44 -07:00
Patrick Buckley 2dc885ab4d fix: output guard detects single secret-bearing env lines (#115)
* fix: output guard detects single secret-bearing env lines

The credential leak check required 3+ env-style lines before flagging.
A single AWS_SECRET_ACCESS_KEY=... line was missed. Now flags whenever
any env line has a secret-bearing key name (SECRET, KEY, TOKEN,
PASSWORD, CREDENTIAL), regardless of how many total env lines exist.

* fix: tighten env secret key matching, add tests

Tighten _RE_ENV_SECRET_KEY to word-boundary segments so MONKEY/TURKEY
don't false-positive. Use any() for short-circuit. Add test for single
secret line detection and substring false-positive prevention.
2026-03-17 02:19:22 -07:00
Patrick Buckley 14488f43e0 feat: metacognitive nudge on tool error — search memories for guidance
Add tool_error nudge type that fires when a tool returns an error,
prompting the model to search memories for prior feedback about the
tool or error pattern before retrying.

- Gated on nudges config (respects nudges=false)
- Only fires when memories exist (no noise on fresh workstreams)
- Broad error detection: Error*, *error:*, Command timed out, Unknown tool
- Nudge wording aligned to memory(action='search') convention
- Respects existing cooldown (5 min) and rate limiting
- 4 new tests
2026-03-17 02:06:10 -07:00
Patrick Buckley 90f2070146 chore: bump version to 0.8.1 v0.8.1 2026-03-17 01:28:14 -07:00
Patrick Buckley 1e551830ea fix: allow deleting installed (readonly) skills
Readonly guard should prevent editing content, not uninstalling.
Remove readonly check from admin_delete_skill so batch-installed
skills can be individually deleted. Enable delete button in UI
for all skills regardless of readonly flag.
2026-03-17 01:26:44 -07:00
Patrick Buckley 84cc212ecd ui: tighten category and risk columns (100px -> 80px) 2026-03-17 01:26:44 -07:00
Patrick Buckley da4025d338 ui: skills table — category first, risk column, remove variables
- Move category column before name
- Remove variables column (rarely useful in table view)
- Add dedicated RISK column with scan badge, unicode shape indicators
  (checkmark/triangle/diamond/warning), and multi-line tooltip showing
  composite score and flagged axes from scan report
- Risk badge is keyboard-focusable (tabindex=0) with aria-label
- Unscanned skills show em-dash placeholder at 40% opacity
- Balanced grid: 100px 1.5fr 100px 120px
- Risk + category hidden on mobile (<700px)
2026-03-17 01:26:44 -07:00
Patrick Buckley 88085c29ff fix: normalize install response + review fixes
Address 5 Copilot review items + code review findings:

- Normalize install endpoint to always return envelope response:
  {installed: [...], skipped: [...], total: N} — eliminates dual
  response shape (single SkillInfo vs batch). Breaking change to
  install endpoint response, SDKs and OpenAPI spec updated.
- Add SkillInstallResponse + SkillInstallSkipped Pydantic models
- POST /resources spec now correctly documents response_code=201
- SQLite count_skill_resources_bulk chunks IN clause at 900 to stay
  under SQLITE_MAX_VARIABLE_NUMBER (999)
- Fix installDiscoveredSkill() JS handler for envelope response
- Add error key to 409 duplicate response for error handler compat
- Update Python SDK install_skill return type (dict, not SkillInfo)
- Add TypeScript SkillInstallResponse + SkillInstallSkipped types
- Regenerate openapi-console.json
- Update all install tests for envelope response shape
2026-03-17 01:26:44 -07:00
Patrick Buckley 3152667a0c fix: update test_skill_sources for 5-tuple _parse_github_url
_parse_github_url now returns (owner, repo, branch, path, branch_explicit).
Update all test unpackings and add assertions for branch_explicit.
2026-03-17 01:26:44 -07:00
Patrick Buckley 4b44d88401 fix: harden batch skill install — 7 review items + OpenAPI snapshot
- Race condition: wrap create_prompt_template in try/except, append
  to skipped on conflict instead of crashing
- HTTP timeout: per-request timeout (10s+5s connect) instead of shared
  15s pool; parallelize SKILL.md and resource fetches with semaphore
  (5 concurrent)
- Branch detection: return branch_explicit from _parse_github_url(),
  eliminate duplicated regex matching and type: ignore comments
- Content-length: check len(resp.content) after fetch instead of
  unreliable content-length header; add size check in batch path
- Rate limits: _check_rate_limit() inspects x-ratelimit-remaining,
  raises actionable error on 403, warns when remaining < 10
- Root resources: fix _find_resource_files skipping root-level
  resources like scripts/foo.sh for root SKILL.md
- resource_count: pass accurate count in update and install responses
- Regenerate openapi-console.json with new resource endpoints
2026-03-17 01:26:44 -07:00
Patrick Buckley 8957b9ce0e feat: batch install skills from multi-skill GitHub repos
When a GitHub repo URL has no root SKILL.md (monorepo pattern like
anthropics/skills), automatically scan the repo tree for all SKILL.md
files and install every discovered skill in one operation.

- Add fetch_skills_from_github_repo() — scans recursive tree, parses
  each SKILL.md, collects per-skill resources via shared helpers
- Extract _find_resource_files() and _fetch_resource_contents() to
  eliminate duplication between single and batch fetch paths
- Extend admin_skill_install to fall back to batch scanning when
  single-skill fetch returns 404
- Each skill gets a specific source_url pointing to its subdirectory
- Backward compatible: single-skill repos return same response shape
- Frontend handles both shapes with contextual toast messages
- Filter tree scan to URL path subtree when path is provided
- Cap at 50 skills per repo scan

Also addresses review feedback:
- Fix path prefix check (scripts/ not scriptsX/)
- Use count_skill_resources_bulk for single skill GET
- Add content field to SkillResourceInfo schema
- Fix OpenAPI spec paths ({path} not {path:path})
- Check r.ok on resource upload promises
- Preserve / in URL-encoded paths (split/map/join pattern)
- URL-encode path in Python SDK delete method
- Fix test_install_not_found to mock batch fallback
- Fix test_search_empty_results for required q param
- Narrow except clause to ValueError in batch parser
2026-03-17 01:26:44 -07:00