mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
28ef63a10cdb7323f0aee461ffaeddd9b302d596
7 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Patrick Buckley
|
7a06f5e8bc |
refactor(session): make ModelLane the provider boundary (#979) (#989)
* refactor(session): make ModelLane the provider boundary (#979) ## Summary This closes the model-lane ownership gap left by #832: `ChatSession` no longer stores raw provider/client handles. `ResolvedModelBinding` now carries the provider, client, model, capabilities, registry generation, and backend-auth configuration as one coherent snapshot. - Atomically rebind existing sessions after model-registry changes while pinning each in-flight send, fallback, judge, output guard, task agent, title, compaction, perception, and voice operation to its initiating principal and binding. - Fence UI publication, canonical trajectory folds, durable writes, streams, retries, child scopes, and judge work by generation. Stop can hand off to a successor without accepting late state; cancelled tools retain typed effect receipts, and concurrent approval batches resolve by exact cycle or call. - Make create, fork, open, close, and delete race-safe with hidden `creating` reservations, incarnation-aware state tails, and an ACL-rechecked transaction that clones checkpoint-bounded history, configuration, project/persona state, and attachment references. - Extend REST/OpenAPI and Python/TypeScript SDK contracts for create/fork inputs, routed-create metadata, live-workstream probes, targeted approvals, and structured cancellation results. - Update architecture, storage, authentication, judge, channel, console, API, and SDK documentation, including regenerated architecture diagrams and OpenAPI artifacts. ## Validation - SQLite suite: 11,188 passed, 9 skipped, 10 deselected - PostgreSQL suite: 11,195 passed, 2 skipped, 10 deselected - Live backend: 3 passed - SSE recovery: 6 passed; browser recovery harness passed all scenarios - Ruff: clean; 595 files correctly formatted - mypy: 243 source files clean - TypeScript: typecheck/build and 35 tests passed - OpenAPI artifacts fresh; all 14 changed diagrams reproduce byte-for-byte - `git diff --check` and Git LFS integrity clean Closes #979. * fix(deps): update nanoid for GHSA-2v37-7h3g-55p8 Refresh the transitive lock entry admitted by PostCSS so the TypeScript security gate no longer resolves the vulnerable custom-generator implementation. Validation: - npm ci - npm audit --audit-level=moderate: 0 vulnerabilities - TypeScript typecheck and build - TypeScript tests: 35 passed * fix(test): assert canonical model registry URLs Replace prefix checks with exact canonical base URL assertions so the tests do not model incomplete URL validation. Validation: tests/test_model_registry.py (185 passed); Ruff check/format; mypy. |
||
|
Patrick Buckley
|
471d1a3311 |
docs: audit documentation for 1.4 / 1.5 state
Systematic pass over every doc under docs/, the root-level README /
QUICKSTART / CONTRIBUTING, and the PlantUML diagrams. Memory and docs
had drifted against the code since 1.2 — this catches them up to the
1.4.0 release and the 1.5.0a1 experimental line.
User-facing fixes
- README: fix broken docs/mcp.md link (→ mcp-registry.md); channel
gateway entry reflects shipped Discord + Slack adapters instead of
"Slack/Teams planned"; diagrams table mentions both.
- QUICKSTART: docs/*.md relative links were wrong from the repo root;
wizard version bumped from 0.5.4.
- CONTRIBUTING: add dev extra plus the ruff / mypy / pytest commands
we actually expect before push.
Reference docs
- architecture.md: 19 tool schemas (was 15), 18 admin tabs (was 14),
turnstone-bootstrap added to entry-points table, OpenAI provider
file split (chat/responses/common) documented, 38 SDK event
dataclasses (was 27 and referenced deleted mq/protocol.py), Slack
adapter + multi-adapter gateway, plan_agent/task_agent naming,
governance admin-panel rewrite.
- api-reference.md: full attachment endpoints (POST/GET/content/
DELETE on /v1/api/workstreams/{ws_id}/attachments) plus the
multipart mode on POST /v1/api/workstreams/new.
- channels.md: Slack Setup section (Socket Mode app creation, OAuth
scopes, tokens), Slack CLI/env reference in config table, combined-
adapter architecture diagram.
- console.md: 18-tab listing (was 13) with Channels/Models/Nodes/TLS
descriptions and ConfigStore live-edit note.
- docker.md: Slack env vars block; image entry-point list now
includes turnstone / turnstone-bootstrap.
- sdk.md: attachments methods on the server client, attachments
example (upload-then-send and at-creation), event count fixed.
- releasing.md: four-track table (stable/1.0, 1.3, 1.4 + main 1.5);
promotion workflow uses 1.5 / 1.6 numbering.
- settings.md: plan_model / task_model / plan_effort / task_effort
overrides section.
- governance.md: skill naming (/skill, `skill` field — not /template),
Prompts/Judge tabs called out.
- security.md: two-token-types wording; src claim values match the
AuthResult source strings actually emitted.
- mcp-registry.md: SDK package name is @turnstone/sdk.
- tools.md: plan / task renamed to plan_agent / task_agent in the
section headings and summary table; primary-key table matched.
- design/consistent-hash-ring.md: dead direct-http-transport.md
pointer redirected to architecture.md.
Diagrams
- 02-package-structure: drop phantom chat.py entry point, add admin
and bootstrap, add slack/bot.py, rename channels/gateway.py →
channels/cli.py.
- 16-channel-architecture: Slack is no longer "(future)", add a
SlackBot class and the slack-bolt Socket Mode edges; wire the new
bot into ChannelService. PNGs regenerated from both puml sources.
|
||
|
Patrick Buckley
|
a315cabe71 |
chore: polish — remove dead code, update diagrams and docs
Remove stale Redis/Bridge/MQ references found via vulture scan and manual grep: - bot.py docstring: remove Redis MQ reference - server.py trusted_sources: remove "bridge" - tls.py docstring: remove "bridge" from service list Delete 4 obsolete diagram pairs (puml + png): - 06-mq-protocol, 07-message-routing, 08-redis-key-schema, 10-simulator-architecture Update 7 diagrams to reflect direct HTTP architecture: - system-context, package-structure, workstream-states, console-data-flow, deployment, channel-architecture, settings-architecture Redraw architecture-overview.svg: Console router replaces Redis MQ, direct SSE data plane, hash ring routing. |
||
|
Patrick Buckley
|
3658b77de8 |
feat: Discord content catch-up + bidirectional notification replies (… (#64)
* feat: Discord content catch-up + bidirectional notification replies (#64) Two improvements to the Discord channel adapter: 1. Fix intermittent dropped responses caused by a race between the bridge's two independent SSE connections (global SSE detects idle before per-ws SSE delivers all content tokens). The bridge now accumulates content in _ws_content_buffer and attaches it to TurnCompleteEvent.content. The Discord bot uses this as a catch-up when streaming events were missed. 2. Bidirectional notification replies — when the notify tool sends a DM, the message is tracked with the originating ws_id. Users can reply to the DM and the reply is routed to the workstream. The response is forwarded back to the DM, with the response itself tracked for multi-turn conversations. Includes user identity verification, stale notification feedback, and FIFO-capped tracking (100 entries). * fix: address Copilot review — re-insert on unlinked user, deque buffer - Re-insert _notify_ws_map entry when resolve_user returns None so the user can retry after linking (same pattern as user-mismatch re-insert) - Rename _MAX_CONTENT_BUFFER_BYTES → _MAX_CONTENT_BUFFER_CHARS (len() returns characters, not bytes) - Use deque + running total for O(1) popleft instead of list.pop(0) |
||
|
Patrick Buckley
|
c7586abd0a |
Add dynamic tool search with native defer_loading for Anthropic/OpenAI (#30)
* Add dynamic tool search with native defer_loading for Anthropic/OpenAI
When MCP tools push the total tool count past a configurable threshold
(default 20), tool definitions are deferred to reduce token overhead and
improve tool selection accuracy. Three-tier approach mirrors the existing
web search pattern:
- Anthropic (Claude 4.x): native defer_loading + server-side BM25 search
- OpenAI (GPT-5.4+): native defer_loading + hosted search
- vLLM/llama/NIM: client-side BM25 fallback via synthetic tool_search tool
New module turnstone/core/tool_search.py with BM25Index (pure-Python,
zero deps) and ToolSearchManager (session-scoped visibility, expansion,
server hint generation). Discovered tools persist for the session lifetime
so the model only searches once per capability needed.
Config: [tools] search/search_threshold/search_max_results
CLI: --tool-search {auto,on,off}, --tool-search-threshold, --tool-search-max-results
Agents (plan/task) exempt — their scoped tool sets are always small.
43 new tests (1253 total). All diagrams regenerated with PlantUML 1.2025.2.
* Fix Copilot review feedback on tool search
- Fix _MCP_PREFIX_RE to handle underscores in server names (non-greedy match)
- Use ordered dict for _expanded to preserve tool discovery order
- Avoid constructing ToolSearchManager when below threshold in auto mode
- Return empty string from _mcp_server_summary when no servers (not "none")
- Fix CLI help text to reference threshold generically, not hardcoded "20"
- Fix agent exemption docs to accurately describe scoped tool sets
- Fix README to not hardcode "30+" threshold number
|
||
|
Patrick Buckley
|
e7fe8fca9d |
Add channel notification tool with security hardening (#27)
* Add channel notification tool with security hardening Implements the `notify` tool allowing the LLM to send notifications to Discord channels/users via the channel gateway. Includes fixes for 11 review findings: JWT auth on the gateway endpoint, first-healthy gateway delivery with retry+backoff, rate limiting only on success, SSRF URL scheme validation, Discord mention sanitization, SQLite ON CONFLICT upsert preserving created timestamps, advertise URL resolution for 0.0.0.0 bind, randomized service IDs, generic error messages to prevent internal state leakage, and partial direct-target validation. Service registry with heartbeat-based health filtering (migration 005). Channel gateway registers on startup, heartbeats every 30s, deregisters on shutdown. 70 new tests covering tool prepare/execute, HTTP endpoint auth (static + JWT), storage CRUD, and retry behavior. * Add notify documentation, diagrams, and review fixes Documentation: - New sequence diagram 17-notify-flow.puml showing end-to-end delivery - Updated 16-channel-architecture.puml with services table, notify HTTP path, and Notification Flow note - channels.md: Notifications section (targeting, delivery flow, service registry, security) and new config table entries - tools.md: notify tool reference, updated counts/tables (14→15 tools) - security.md: channel gateway row in service-to-service auth table - architecture.md: notification subsystem paragraph Review fixes (copilot): - _http.py: fail closed when auth unconfigured (401 instead of pass- through), strip whitespace on message/title, generic error messages for user-not-found vs no-linked-channels - session.py: parse gateway response JSON and require at least one result with status=="sent" before counting as success - _postgresql.py: use index_elements instead of constraint for upsert |
||
|
Patrick Buckley
|
a6e929b0a0 |
Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24) Bidirectional channel adapter framework connecting external messaging platforms to turnstone workstreams via Redis MQ. Discord ships as the first adapter; the protocol supports future Slack/Teams integrations. Channel framework: - ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping - AsyncRedisBroker with single dispatch loop and per-channel ordered workers - channel_routes table (migration 003) for persistent route storage - 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD) - Unified turnstone-channel gateway entry point, loads adapters by config - Message chunking, approval formatting, plan review formatting Discord adapter: - discord.py v2.4+ bot with thread-per-@mention model - Slash commands: /link (modal), /unlink, /ask, /status, /close - Persistent button views for tool approval and plan review - Streaming responses via edit-in-place (1.5s interval) - Stale route detection and atomic session resume via resume_session field - SessionResumedEvent confirmation back to channel - Auto-approve support (blanket + per-tool list) Atomic session resume: - resume_session field on CreateWorkstreamMessage for single-request resume - Server resumes session during POST /v1/api/workstreams/new atomically - Bridge emits SessionResumedEvent to per-workstream channel - WorkstreamCreatedEvent extended with resumed/session_id/message_count - Server UI dashboardResumeSession simplified to single request - Pruned sessions fall back gracefully to fresh start Service auth: - Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET - Bridge: approve scope (1 week). Console collector: read. Proxy: write. Console admin: - Channels tab with per-user view, force-link modal, unlink - 3 admin API endpoints for channel user management - Styled confirm modals replacing browser confirm() dialogs Bug fixes: - AsyncRedisBroker: replaced per-channel listener tasks with single dispatch loop + per-channel queue workers (fixes message stealing race) - Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates - Bridge: _active_sends tracked for initial messages (fixes missing TurnCompleteEvent and unfinalized streaming messages) - Bridge: HTTP calls moved outside lock scope in approval handlers - Bridge: _handle_send cleans up _active_sends on HTTP/server errors - Formatter: reads server SSE format (func_name/preview) with fallback Docs, SDK, tests: - docs/channels.md setup guide, architecture diagram 16 - Updated api-reference.md, architecture.md, console.md, docker.md - Python SDK: resume_session param on create_workstream (async + sync) - TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces - OpenAPI schema: resume_session request, resumed/message_count response - 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing, 18 discord, 12 resume flow) — 1120 total passing * Fix CI lint/typecheck failures and address Copilot review feedback (#24) Lint: fix import ordering, remove unused imports, use contextlib.suppress. Mypy: explicit postgresql dialect import, add discord module overrides for optional-dependency CI environments. Copilot: fix double-escaping in admin confirm modals, return resolved session_id from server resume response, fix channel_routes diagram schema, use atomic setdefault for routing locks, add post-insert race guard in admin channel create, support SSE format in auto-approve check, update identity linking note in architecture diagram. * Fix remaining mypy call-arg errors for discord.py optional dependency Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class definitions that fail when discord.py is not installed in CI. |
