diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index db2e518e..3becc781 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,15 @@ transferring ownership. ``` python -m venv .venv source .venv/bin/activate -pip install -e ".[test]" +pip install -e ".[test,dev]" +``` + +The `dev` extra installs `ruff` and `mypy`. Before pushing, run: + +``` +ruff check turnstone tests +mypy turnstone +pytest ``` ## Guidelines diff --git a/QUICKSTART.md b/QUICKSTART.md index 0c4661ab..5e9f5a2e 100644 --- a/QUICKSTART.md +++ b/QUICKSTART.md @@ -55,7 +55,7 @@ The wizard supports two deployment modes: ``` $ turnstone-bootstrap - Turnstone Bootstrap Wizard v0.5.4 + Turnstone Bootstrap Wizard v1.5.0 ──────────────────────────────────────────────── Which provider for this wizard? @@ -87,6 +87,6 @@ $ turnstone-bootstrap ## See Also -- [Docker Deployment](docker.md) — manual compose setup and profiles -- [Security](security.md) — auth architecture and token types -- [Governance](governance.md) — roles, policies, and templates +- [Docker Deployment](docs/docker.md) — manual compose setup and profiles +- [Security](docs/security.md) — auth architecture and token types +- [Governance](docs/governance.md) — roles, policies, and templates diff --git a/README.md b/README.md index 3116d9f6..79c9c886 100644 --- a/README.md +++ b/README.md @@ -84,7 +84,7 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client: ## Tools -Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp.md](docs/mcp.md) for MCP configuration. +Built-in tools for shell, files, search, web, memory, notifications, and autonomous sub-agents — plus external tools via [MCP](https://modelcontextprotocol.io/) with native deferred loading. See [docs/tools.md](docs/tools.md) for the full reference and [docs/mcp-registry.md](docs/mcp-registry.md) for MCP configuration. ## Architecture @@ -97,7 +97,7 @@ Built-in tools for shell, files, search, web, memory, notifications, and autonom | `turnstone` | Terminal CLI (REPL) | | `turnstone-server` | Web UI + REST API + SSE events | | `turnstone-console` | Cluster dashboard + routing proxy + admin panel | -| `turnstone-channel` | Channel gateway (Discord, with adapters for Slack/Teams planned) | +| `turnstone-channel` | Channel gateway (Discord and Slack adapters) | | `turnstone-admin` | User/token management CLI | | `turnstone-eval` | Eval harness for prompt/tool optimization | | `turnstone-bootstrap` | LLM-guided setup wizard | @@ -117,7 +117,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/): | [Console Data Flow](docs/diagrams/png/11-console-data-flow.png) | Dashboard data collection | | [Deployment](docs/diagrams/png/12-deployment.png) | Docker Compose topology | | [Auth](docs/diagrams/png/15-auth-architecture.png) | JWT, scopes, login flows | -| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord adapter + routing | +| [Channels](docs/diagrams/png/16-channel-architecture.png) | Discord / Slack adapters + routing | | [Judge](docs/diagrams/png/22-judge-architecture.png) | Intent validation pipeline | | [OIDC](docs/diagrams/png/25-oidc-architecture.png) | SSO authorization code flow | @@ -136,7 +136,7 @@ UML diagrams in [`docs/diagrams/`](docs/diagrams/): | Console dashboard | [docs/console.md](docs/console.md) | | Eval harness | [docs/eval.md](docs/eval.md) | | Tools reference | [docs/tools.md](docs/tools.md) | -| MCP integration | [docs/mcp.md](docs/mcp.md) | +| MCP integration | [docs/mcp-registry.md](docs/mcp-registry.md) | ## Requirements diff --git a/docs/api-reference.md b/docs/api-reference.md index f0d6a8df..6390e3c0 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -842,6 +842,15 @@ button automatically. Creates a new workstream. The server supports up to 10 concurrent workstreams. +The endpoint accepts **either** `application/json` (legacy shape) **or** +`multipart/form-data` when you want to upload attachments at creation +time. Multipart requests carry one `meta` field containing the JSON body +shown below plus zero-or-more `file` parts; each file is validated and +reserved onto the new workstream's first turn before the dispatch worker +runs, so queued multimodal turns cannot lose files to racing sends. If +validation fails the fresh workstream is rolled back so no orphan rows +leak. + **Request body:** ```json @@ -915,6 +924,100 @@ Status code: `400` --- +### `POST /v1/api/workstreams/{ws_id}/attachments` + +Upload an image or text document and attach it to the caller's next user +turn on this workstream. + +- Images (png/jpeg/gif/webp) are capped at **4 MiB** and validated via + magic-byte sniff on upload. +- Text documents (any `text/*` MIME, allow-listed application MIMEs, or + known text extensions) are capped at **512 KiB** and must be UTF-8. +- Per-(workstream, user) pending cap is **10** attachments. + +The attachment moves through three states: `pending → reserved → +consumed`. Reservation tokens are threaded through +`POST /v1/api/send` so a queued multimodal turn cannot lose its file to +an overlapping send. + +Ownership failures are masked as `404` so non-owners cannot enumerate +workstream existence. + +**Content-Type:** `multipart/form-data` with a single `file` field. + +**Response (success):** `200` + +```json +{ + "attachment_id": "att_abc123", + "kind": "image", + "mime_type": "image/png", + "size_bytes": 73240, + "filename": "screenshot.png", + "state": "pending" +} +``` + +**Errors:** + +| Code | Meaning | +|------|---------------------------------------------------------| +| 400 | Missing/invalid form, unsupported MIME, not UTF-8, etc. | +| 403 | Auth/scope failure | +| 404 | Workstream not found / not owned by caller | +| 409 | Pending-cap reached | +| 413 | Payload exceeds size cap | + +--- + +### `GET /v1/api/workstreams/{ws_id}/attachments` + +List the caller's **pending** (unconsumed) attachments for this +workstream. Ownership failures are masked as `404`. + +**Response:** `200` + +```json +{ + "attachments": [ + { + "attachment_id": "att_abc123", + "kind": "image", + "mime_type": "image/png", + "size_bytes": 73240, + "filename": "screenshot.png", + "state": "pending" + } + ] +} +``` + +--- + +### `GET /v1/api/workstreams/{ws_id}/attachments/{attachment_id}/content` + +Returns the raw bytes of an attachment with its stored `Content-Type`. +Useful for previewing an image or replaying a document. Ownership +failures are masked as `404`. + +**Response:** `200` — binary body, original `Content-Type`. + +--- + +### `DELETE /v1/api/workstreams/{ws_id}/attachments/{attachment_id}` + +Remove a pending attachment. Consumed attachments return `404` (they +are part of a committed conversation turn). Ownership failures are also +masked as `404`. + +**Response:** `200` + +```json +{"deleted": "att_abc123"} +``` + +--- + ### `POST /v1/api/workstreams/{ws_id}/delete` Permanently delete a saved workstream and all its messages from storage. diff --git a/docs/architecture.md b/docs/architecture.md index a62bfa0a..d17fa1d2 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -21,7 +21,8 @@ plugs in. | `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) | | `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization | | `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) | -| `turnstone-admin` | `turnstone.core.admin_cli` | — | Offline user and API token management | +| `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management | +| `turnstone-bootstrap` | `turnstone.bootstrap` | — | LLM-guided setup wizard | --- @@ -36,7 +37,10 @@ turnstone/ session.py ChatSession engine, SessionUI protocol, tool dispatch providers/ LLM provider adapters (pluggable backend layer) _protocol.py LLMProvider protocol, ModelCapabilities, StreamChunk, CompletionResult - _openai.py OpenAIProvider — OpenAI, vLLM, llama.cpp, any compatible API + _openai.py OpenAIProvider facade (re-exports Chat/Responses providers) + _openai_chat.py OpenAIChatCompletionsProvider — vLLM, llama.cpp, local compatible APIs + _openai_responses.py OpenAIResponsesProvider — commercial OpenAI Responses API + _openai_common.py Shared ModelCapabilities table + helpers _anthropic.py AnthropicProvider — Anthropic Messages API, native streaming, thinking _google.py GoogleProvider — Google Gemini via OpenAI-compat endpoint __init__.py create_provider() + create_client() factory functions @@ -85,6 +89,7 @@ turnstone/ _routing.py ChannelRouter — channel/thread ↔ workstream mapping via HTTP _config.py Base ChannelConfig dataclass discord/ Discord adapter (bot, cog, views, streaming, config) + slack/ Slack adapter (Socket Mode bot, DM routing, approval buttons) shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js) katex-0.16.45/ Vendored KaTeX math rendering library (MIT, woff2 fonts) ui/ @@ -97,7 +102,7 @@ turnstone/ renderer.js Markdown + LaTeX renderer (tables, nested lists, blockquotes, KaTeX math) app.js Split-pane UI (Pane class, binary layout tree, SSE, tool approval) tools/ - *.json 15 tool schemas (OpenAI function-calling format + turnstone metadata) + *.json 19 tool schemas (OpenAI function-calling format + turnstone metadata) ``` Both UIs share a common design system extracted into `turnstone/shared_static/`: design tokens, login overlay, toast notifications, theme toggle, keyboard shortcuts, and utility functions. Each UI imports `base.css` and the shared JS modules at `/shared/`, then adds only page-specific code at `/static/`. @@ -443,13 +448,15 @@ from each schema and builds: - `PRIMARY_KEY_MAP` -- `{name: primary_key}` for JSON fallback recovery - `merge_mcp_tools(builtin, mcp_tools)` -- merges built-in + MCP tools at session init -### 13 Tools by Category +### 19 Tools by Category **Read-only (auto-approve)**: - `read_file` -- read file contents with optional offset/limit +- `diff_file` -- show diff between two files / versions - `search` -- ripgrep-based codebase search - `man` -- read man pages - `recall` -- search conversation history +- `read_resource` -- read an MCP resource by URI **Write (requires approval)**: - `bash` -- execute shell commands (with safety checks via `turnstone.core.safety`) @@ -458,13 +465,20 @@ from each schema and builds: - `math` -- execute Python in sandboxed subprocess (via `turnstone.core.sandbox`) - `web_fetch` -- fetch a URL (with SSRF protection via `turnstone.core.web`) - `web_search` -- search the web (provider-native for Anthropic/OpenAI, Tavily fallback for local models) +- `notify` -- send a user-facing notification (Discord/Slack, optional reply routing) +- `watch` -- schedule a recurring poll with condition DSL **Agent (delegated sub-sessions)**: -- `task` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`) -- `plan` -- explore codebase and write a structured plan (`AGENT_TOOLS`) +- `task_agent` -- delegate to a sub-agent with full tool access (`TASK_AGENT_TOOLS`) +- `plan_agent` -- explore codebase and write a structured plan (`AGENT_TOOLS`) -**Memory (structured persistent store)**: +**Memory / skills / prompts**: - `memory` -- save, search, delete, or list memories (typed and scoped) +- `skill` -- invoke a skill (governed, versioned procedure) +- `use_prompt` -- fetch and apply a prompt template + +Tool names are `plan_agent` / `task_agent` (not `plan` / `task`); bare words +collide with chat-template channels on some local models. ### Prepare / Execute Pattern @@ -483,12 +497,12 @@ separation allows the UI to show previews before any side effects occur. ### Agent Tools -`task` and `plan` invoke `_run_agent()`, which runs a multi-turn loop with -a subset of tools and its own system prompt. The sub-agent runs +`task_agent` and `plan_agent` invoke `_run_agent()`, which runs a multi-turn +loop with a subset of tools and its own system prompt. The sub-agent runs independently, then returns the final content as the tool result. -- **task**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools) -- **plan**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output +- **task_agent**: uses `self._task_tools` (`TASK_AGENT_TOOLS` + MCP tools) +- **plan_agent**: uses `self._agent_tools` (`AGENT_TOOLS` + MCP tools). Writes output to `.plan-.md` — unique per `ChatSession` so concurrent workstreams don't collide. On repeat invocations the prior `plan` tool call and its result are forwarded from `self.messages` so the agent refines the existing plan rather @@ -1111,8 +1125,9 @@ Three hierarchical scopes control endpoint access: - **Console** is the auth management hub — it hosts the admin endpoints for creating users, issuing API tokens, and managing channel mappings. User records and token hashes live in the shared storage backend. The console - dashboard includes an **admin panel** (14 tabs) for managing - credentials, governance, MCP servers, and runtime settings through the browser. + dashboard includes an **admin panel** (18 tabs) for managing + credentials, governance, MCP servers, models, node metadata, and runtime + settings through the browser. - **Server** is a JWT validator only — it validates tokens on each request but never creates users or tokens. Both processes share the same `jwt_secret` (via `TURNSTONE_JWT_SECRET` env var or `[auth].jwt_secret` config). @@ -1347,9 +1362,10 @@ setup, auth headers, `_request()` (REST) and `_stream_sse()` (SSE). Sync clients delegate through `_SyncRunner` which maintains a persistent background event loop on a daemon thread. -**Event types**: 27 standalone dataclasses in `events.py` with a type-registry -pattern matching `OutboundEvent.from_json()` from `mq/protocol.py`. Events are -decoupled from server internals. +**Event types**: 38 standalone dataclasses in `events.py` with a type-registry +dispatch (`from_json()` on each event). Events are decoupled from server +internals — the SDK parses SSE frames directly from the `/v1/api/events` +streams. **TypeScript SDK**: `sdk/typescript/` — separate npm package with the same API surface. Zero browser dependencies, SSE via `fetch` + `ReadableStream` parsing. @@ -1371,7 +1387,8 @@ with TurnstoneServer("http://localhost:8080", token="tok_xxx") as client: > See also: [Channel Integrations guide](channels.md) The `turnstone-channel` gateway connects external messaging platforms -(Discord, Slack, Teams) to the turnstone cluster via HTTP. Each +(Discord and Slack today, with an adapter protocol for future platforms) to +the turnstone cluster via HTTP. Each platform adapter implements the `ChannelAdapter` protocol and translates between platform-native events and turnstone server API calls. @@ -1384,7 +1401,7 @@ workstream is reactivated, the router uses atomic resume via the the old workstream's conversation during creation in a single HTTP request, eliminating ordering fragility. -Discord ships as the first adapter. See [channels.md](channels.md) for +Discord and Slack adapters ship today. See [channels.md](channels.md) for setup instructions, configuration reference, and the adapter development guide. @@ -1405,11 +1422,11 @@ retries up to 3 times with backoff, re-querying the service registry on each attempt. See [Notification Flow diagram](diagrams/png/17-notify-flow.png). **Bidirectional replies:** When a user replies to a notification DM, the -Discord bot looks up the originating `ws_id` from the tracked message ID, -verifies the replying user matches the notification recipient, and routes -the reply to the workstream via `router.send_message()`. The workstream's -response is forwarded back to the DM via a temporary entry in -`_notify_reply_channels`. On `TurnCompleteEvent`, the response message is +channel adapter (Discord or Slack) looks up the originating `ws_id` from the +tracked message ID, verifies the replying user matches the notification +recipient, and routes the reply to the workstream via `router.send_message()`. +The workstream's response is forwarded back to the DM via a temporary entry +in `_notify_reply_channels`. On `TurnCompleteEvent`, the response message is itself tracked for further replies, enabling multi-turn DM conversations without requiring the user to open the web UI. Tracking entries are capped at 100 (FIFO eviction) and cleaned up on workstream close. @@ -1442,11 +1459,10 @@ and workstreams record which skill and version spawned them. Token budget enforcement tracks consumption in `session.send()` with 80% warning and 100% approval gate via the `__budget_override__` synthetic tool name. -The console admin panel adds 5 governance tabs (Roles, Policies, Skills, -Usage, Audit), a Memories tab, a Settings tab (form-based editor for all -ConfigStore settings), and an MCP Servers tab (database-backed server -definitions with live connection status and cluster-wide reload) for a -total of 13 tabs, all permission-gated. +The console admin panel exposes these capabilities as 18 permission-gated +tabs: Users, API Tokens, Channels, Schedules, Watches, Roles, Policies, +Prompts, Judge, Skills, MCP Servers, Usage, Audit, Memories, Models, Nodes, +Settings, and TLS. Both Python and TypeScript SDKs expose governance methods on the console client. diff --git a/docs/channels.md b/docs/channels.md index 876f85f0..05dd1c64 100644 --- a/docs/channels.md +++ b/docs/channels.md @@ -7,25 +7,30 @@ platform-native events (messages, button clicks, slash commands) into turnstone API calls, and renders workstream output back into the platform's UI. -Discord ships as the first adapter. The adapter protocol is designed for -future Slack and Teams integrations. +Discord and Slack adapters ship today. The adapter protocol is designed +so new platforms can be added with only a new package under +`turnstone/channels//`. --- ## Architecture ``` -Discord Gateway - | - v -turnstone-channel (Discord adapter) - | - v -turnstone-server (direct HTTP) - or -turnstone-console (routing proxy, multi-node) +Discord Gateway Slack (Socket Mode WebSocket) + \ / + v v + turnstone-channel (one or more adapters) + | + v + turnstone-server (direct HTTP) + or + turnstone-console (routing proxy, multi-node) ``` +A single `turnstone-channel` process can run multiple adapters +simultaneously (e.g. Discord + Slack) — pass the tokens for each +platform you want to enable. + Key components: - **ChannelAdapter protocol** (`turnstone/channels/_protocol.py`) — generic @@ -120,6 +125,68 @@ An admin can also force-link or unlink users via the console admin panel --- +## Slack Setup + +Slack uses **Socket Mode**, so no public URL or API Gateway is required — Slack +connects outbound to the bot via a WebSocket. Install with: + +```bash +pip install 'turnstone[slack]' +``` + +### 1. Create a Slack App + +1. Go to https://api.slack.com/apps and click **Create New App** +2. Under **Settings > Socket Mode**, enable Socket Mode. This generates an + **App-Level Token** (prefix `xapp-`) — copy it. +3. Under **OAuth & Permissions**, add these **Bot Token Scopes**: + `chat:write`, `chat:write.public`, `channels:history`, `im:history`, + `groups:history`, `mpim:history`, `reactions:write`, `commands` +4. Under **Event Subscriptions** (Socket Mode delivers events), subscribe + to bot events: `message.channels`, `message.im`, `message.groups` +5. Under **Slash Commands**, create a command (default `/turnstone`) +6. Install the app to your workspace to generate the **Bot User OAuth + Token** (prefix `xoxb-`). + +### 2. Configure Turnstone + +**Environment variables** (recommended for Docker): + +```bash +TURNSTONE_SLACK_TOKEN=xoxb-... # Bot User OAuth Token +TURNSTONE_SLACK_APP_TOKEN=xapp-... # App-Level Token (Socket Mode) +TURNSTONE_SLACK_CHANNELS= # optional, comma-separated channel IDs +TURNSTONE_SLACK_SLASH_COMMAND=/turnstone +``` + +**CLI flags** (bare-metal): + +```bash +turnstone-channel \ + --slack-token "xoxb-..." \ + --slack-app-token "xapp-..." \ + --slack-slash-command /turnstone \ + --server-url http://localhost:8080 +``` + +The Slack and Discord adapters can be enabled together — pass tokens for +both and the gateway hosts both adapters in one process. + +### 3. Usage + +- **DM the bot**: messages sent directly to the bot create a workstream + scoped to that DM; the slash command is not required. +- **Slash command**: `/turnstone ` in any channel the bot can + see starts a per-user channel session. +- Tool approvals render as Slack **Block Kit** buttons; only the user + who owns the workstream can approve/reject. +- Plan reviews render as a modal with approve / request-changes actions. +- Notifications and reply routing work identically to Discord. +- Session recovery: persisted channel routes are re-subscribed when the + bot restarts, so existing Slack conversations keep flowing. + +--- + ## Usage ### Conversations @@ -184,9 +251,13 @@ Plan review requests are displayed as a blue embed with: | CLI Flag | Env Var | Default | Description | |----------|---------|---------|-------------| -| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Bot token (required to enable Discord) | +| `--discord-token` | `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord) | | `--discord-guild` | — | `0` (all guilds) | Restrict to a single Discord guild | -| `--discord-channels` | — | empty (all) | Comma-separated channel IDs to allow | +| `--discord-channels` | — | empty (all) | Comma-separated Discord channel IDs to allow | +| `--slack-token` | `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token (`xoxb-…`, required to enable Slack) | +| `--slack-app-token` | `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token (`xapp-…`, required with `--slack-token`) | +| `--slack-channels` | `TURNSTONE_SLACK_CHANNELS` | empty (all) | Comma-separated Slack channel IDs to allow | +| `--slack-slash-command` | `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command name registered in the Slack app | | `--server-url` | `TURNSTONE_SERVER_URL` | `http://localhost:8080` | Server URL (single-node) | | `--console-url` | `TURNSTONE_CONSOLE_URL` | — | Console URL (multi-node routing proxy) | | `--model` | — | server default | Default model for new workstreams | @@ -196,6 +267,9 @@ Plan review requests are displayed as a blue embed with: | `--log-level` | `TURNSTONE_LOG_LEVEL` | `INFO` | Log level | | `--log-format` | `TURNSTONE_LOG_FORMAT` | `auto` | Log format (`auto`/`json`/`text`) | +At least one of `--discord-token` or `--slack-token` must be supplied. +Passing both starts both adapters in the same process. + --- ## User Identity @@ -249,8 +323,8 @@ waiting for them to check in. Two modes: - **Username** — provide a turnstone `username`. The gateway resolves - it via the `channel_users` table and sends to all linked channels - (e.g. Discord + future Slack). + it via the `channel_users` table and sends to every linked platform + the user has (e.g. Discord + Slack). - **Direct** — provide `channel_type` + `channel_id` to target a specific platform channel or user DM. diff --git a/docs/console.md b/docs/console.md index d1b4c367..96f4c3dd 100644 --- a/docs/console.md +++ b/docs/console.md @@ -396,10 +396,19 @@ The browser maintains a local `clusterState` object that mirrors the cluster sna Accessed via the "admin" button in the header (visible when authenticated with `approve` scope). Provides user, API token, channel link, MCP server, -and skill management with 13 tabs (see also -[Governance](governance.md) for -the Roles, Policies, Skills, Usage, and Audit tabs, and -[Settings](settings.md) for the database-backed configuration editor): +and skill management with 18 tabs (Users, API Tokens, Channels, Schedules, +Watches, Roles, Policies, Prompts, Judge, Skills, MCP Servers, Usage, +Audit, Memories, Models, Nodes, Settings, TLS). See also +[Governance](governance.md) for the Roles, Policies, Skills, Usage, and +Audit tabs, and [Settings](settings.md) for the database-backed +configuration editor. + +The **Channels** tab links users to either a Discord or Slack account +via a per-row channel-type selector. The **Models** tab is a CRUD +editor for `model_definitions`, the **Nodes** tab edits per-node +metadata, and the **TLS** tab manages CA and leaf certificates for the +internal mTLS fabric. The **Settings** tab edits ConfigStore values +live; edits apply without restart. **Users tab:** diff --git a/docs/design/consistent-hash-ring.md b/docs/design/consistent-hash-ring.md index b6aa4e14..02b00d7b 100644 --- a/docs/design/consistent-hash-ring.md +++ b/docs/design/consistent-hash-ring.md @@ -8,7 +8,8 @@ This document describes a consistent hash ring algorithm evaluated during the design of the direct HTTP transport routing system. The current implementation uses weight-proportional bucket assignment with a -donor/recipient rebalancing algorithm (see `direct-http-transport.md`). +donor/recipient rebalancing algorithm (see the routing section of +[../architecture.md](../architecture.md)). The consistent hash ring is documented here as a reference for future scalability work — if the cluster grows beyond the point where the weight-proportional approach is sufficient, the ring provides a diff --git a/docs/diagrams/02-package-structure.puml b/docs/diagrams/02-package-structure.puml index 51d59551..61ec8512 100644 --- a/docs/diagrams/02-package-structure.puml +++ b/docs/diagrams/02-package-structure.puml @@ -19,7 +19,8 @@ package "Entry Points" <> { component [cli.py\nturnstone] as cli <> component [server.py\nturnstone-server] as server <> component [eval.py\nturnstone-eval] as eval <> - component [chat.py\n(re-exports)] as chat <> + component [admin.py\nturnstone-admin] as admin <> + component [bootstrap.py\nturnstone-bootstrap] as bootstrap <> } ' Core engine @@ -48,7 +49,8 @@ package "turnstone/core/" <> { package "turnstone/channels/" <> { component [_routing.py\nChannelRouter] as router <> component [discord/bot.py\nDiscordBot] as discordbot <> - component [gateway.py\nturnstone-channel] as gateway <> + component [slack/bot.py\nSlackBot (Socket Mode)] as slackbot <> + component [cli.py\nturnstone-channel] as gateway <> } ' Console @@ -112,7 +114,8 @@ eval --> memory eval --> config eval --> tools -chat --> session +admin --> auth +bootstrap --> providers ' Core internal deps session --> providers @@ -135,8 +138,10 @@ tools --> schemas ' Channel dependencies gateway --> discordbot +gateway --> slackbot gateway --> router discordbot --> sdkserver : HTTP + SSE +slackbot --> sdkserver : HTTP + SSE router --> storage : channel_routes ' Console dependencies diff --git a/docs/diagrams/16-channel-architecture.puml b/docs/diagrams/16-channel-architecture.puml index d1f05855..eb4280b1 100644 --- a/docs/diagrams/16-channel-architecture.puml +++ b/docs/diagrams/16-channel-architecture.puml @@ -20,11 +20,14 @@ class "Discord" as Discord <> { asyncio event loop } -class "Slack (future)" as Slack <> { - Socket Mode / Events API +class "Slack" as Slack <> { + Socket Mode WebSocket Block Kit messages + Slash command (default /turnstone) + DM + channel events -- - Planned integration + slack-bolt (Python) + asyncio event loop } class "Teams (future)" as Teams <> { @@ -38,7 +41,7 @@ class "Teams (future)" as Teams <> { class "turnstone-channel" as ChannelService <> { entry point: turnstone-channel -- - One process per platform + One process — hosts one or more adapters asyncio event loop Structured logging (structlog) --log-level, --log-format @@ -47,6 +50,19 @@ class "turnstone-channel" as ChannelService <> { GET /health } +class "SlackBot" as SlackBot <> { + +on_message(event) + +on_action(action) (Block Kit buttons) + +send(channel_id, content) + +send_notification(channel_id, content, ws_id) + +run(bot_token, app_token) + -- + slack-bolt AsyncApp + Socket Mode client + Per-user channel sessions via slash command + DM routing without slash command +} + class "DiscordBot" as Bot <> { +on_message(msg) +on_interaction(interaction) @@ -138,10 +154,15 @@ Server --> Bot : SSE event stream Bot --> Discord : reply / embed\nbutton callback -Slack .[hidden]. Discord +Slack --> SlackBot : socket-mode\nevents +SlackBot --> Router : on_message / on_action +SlackBot --> Server : POST /v1/api/send\nGET /v1/api/events?ws_id= +SlackBot --> Slack : post / update\nBlock Kit button callbacks + Teams .[hidden]. Slack ChannelService --> Bot : creates + runs +ChannelService --> SlackBot : creates + runs ChannelService --> Router : creates ChannelService --> SVC : register / heartbeat /\nderegister diff --git a/docs/diagrams/png/02-package-structure.png b/docs/diagrams/png/02-package-structure.png index 2c390852..d31111ce 100644 --- a/docs/diagrams/png/02-package-structure.png +++ b/docs/diagrams/png/02-package-structure.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:75da80e6bd205e45b9fe48aa2f87197110c908f44e1d98c52eee69f956274711 -size 400402 +oid sha256:a3b5c59403a6febd81667fc8fd2a7d22bc59da6130eba0dea5449c42668d0ede +size 387044 diff --git a/docs/diagrams/png/16-channel-architecture.png b/docs/diagrams/png/16-channel-architecture.png index 5a97ae77..b552a473 100644 --- a/docs/diagrams/png/16-channel-architecture.png +++ b/docs/diagrams/png/16-channel-architecture.png @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:8a5957b71822656098cbe779ca619b7e56e9b11b0f1c4eac4fc880ed62b71a2a -size 358670 +oid sha256:ae4f79fb22600106f8cb0af4ba5586bb26ea5d57e27ef382fdc59b6549fdbd21 +size 415473 diff --git a/docs/docker.md b/docs/docker.md index e5a373a3..cf1b6d51 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -22,7 +22,7 @@ Console dashboard: http://localhost:8090 |---------|------|---------|-------------| | `server` | 8080 | default | Web UI + chat workstreams + LLM | | `console` | 8090 | default | Cluster dashboard | -| `channel` | — | production | Channel gateway (Discord, Slack, etc.) | +| `channel` | — | production | Channel gateway (Discord and/or Slack adapters) | | `server-1`…`server-10` | — | cluster | 10-node server fleet (PostgreSQL required) | ## Profiles @@ -108,8 +108,16 @@ The database stores workstream history, user accounts, and API tokens. When usin |----------|---------|-------------| | `TURNSTONE_DISCORD_TOKEN` | — | Discord bot token (required to enable Discord adapter) | | `TURNSTONE_DISCORD_GUILD` | `0` | Restrict to a single Discord guild (0 = all guilds) | +| `TURNSTONE_SLACK_TOKEN` | — | Slack Bot User OAuth token `xoxb-…` (required to enable Slack adapter) | +| `TURNSTONE_SLACK_APP_TOKEN` | — | Slack App-Level token `xapp-…` (required with `TURNSTONE_SLACK_TOKEN`) | +| `TURNSTONE_SLACK_CHANNELS` | — | Comma-separated Slack channel IDs to allow (empty = all) | +| `TURNSTONE_SLACK_SLASH_COMMAND` | `/turnstone` | Slash command registered in the Slack app | -The channel service runs in the `production` profile. When `TURNSTONE_DISCORD_TOKEN` is set, the Discord adapter connects to the Discord Gateway and routes messages to the server via HTTP. See [Channel Integrations](channels.md) for full setup instructions including Discord application creation and user account linking. +The channel service runs in the `production` profile. When +`TURNSTONE_DISCORD_TOKEN` or the Slack pair is set the gateway starts the +corresponding adapter; both can run in one process. See +[Channel Integrations](channels.md) for platform app setup and user +account linking. ## Scaling @@ -141,7 +149,9 @@ docker compose build docker compose build --no-cache ``` -All entry points are installed in a single image: `turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`, `turnstone-eval`. +All entry points are installed in a single image: `turnstone`, +`turnstone-server`, `turnstone-console`, `turnstone-channel`, +`turnstone-admin`, `turnstone-eval`, and `turnstone-bootstrap`. ## Cleanup diff --git a/docs/governance.md b/docs/governance.md index f8e0e121..91233930 100644 --- a/docs/governance.md +++ b/docs/governance.md @@ -62,14 +62,14 @@ etc.) since workstream templates were merged into the skills system in v0.8.0. - **Default skills**: All `is_default=true` skills auto-apply to new workstreams, concatenated in alphabetical order by name. Use name prefixes (e.g. `01-safety`, `02-style`) to control ordering. -- **Explicit selection**: `--template ` CLI flag, `template` field on +- **Explicit selection**: `--skill ` CLI flag, `skill` field on `POST /v1/api/workstreams/new`, console creation modal dropdown, scheduled task config, and channel adapter config. An explicit skill *replaces* defaults. - **Variables**: Three built-in placeholders resolved at load time: `{{model}}` (active model name), `{{ws_id}}` (workstream ID), `{{node_id}}` (server node ID). Unrecognized placeholders are kept as-is. -- **Runtime switching**: `/template ` to switch, `/template clear` to revert - to defaults, `/template` to show current. Persisted across resume. +- **Runtime switching**: `/skill ` to switch, `/skill clear` to revert + to defaults, `/skill` to show current. Persisted across resume. - **Model-driven loading**: The `skill` built-in tool lets the model discover and activate skills mid-conversation. `search` action finds skills by query (auto-approved); `load` action activates by name (requires user @@ -186,15 +186,21 @@ Full OpenAPI spec at `/openapi.json` and Swagger UI at `/docs`. ## Admin Console UI -6 new tabs added to the admin panel (11 total): +Governance-related tabs within the 18-tab admin panel: - **Roles** — CRUD roles, permission checkbox grid, user role assignment modal - **Policies** — CRUD tool policies with colored action badges (green/red/amber) -- **Skills** — CRUD skills with wide modal, textarea editor +- **Prompts** — Prompt-policy editor (heuristics for admin guardrails) +- **Skills** — CRUD skills with wide modal, textarea editor; Discover pill for + installing from skills.sh / GitHub; per-row scan badges (safe/low/med/high/critical) +- **Judge** — Intent validation configuration and verdict history - **Usage** — Summary readouts + CSS bar chart, time range + group-by selectors - **Audit** — Filterable log with relative timestamps, load-more pagination Tabs are permission-gated: hidden if the user lacks the required permission. +See [docs/console.md](console.md) for the full tab list and +[docs/settings.md](settings.md) for the Settings tab that edits live +ConfigStore values. ## SDK diff --git a/docs/mcp-registry.md b/docs/mcp-registry.md index d383fa55..f8587132 100644 --- a/docs/mcp-registry.md +++ b/docs/mcp-registry.md @@ -146,7 +146,7 @@ with TurnstoneConsole("http://localhost:8081", token="...") as client: ### TypeScript ```typescript -import { TurnstoneConsole } from "@anthropic/turnstone-sdk"; +import { TurnstoneConsole } from "@turnstone/sdk"; const client = new TurnstoneConsole({ baseUrl: "http://localhost:8081", diff --git a/docs/releasing.md b/docs/releasing.md index a4e2b131..daa3e7fc 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -1,17 +1,24 @@ # Release Process -Turnstone uses two parallel release tracks published from a single PyPI package. +Turnstone ships several parallel release tracks from a single PyPI package. ## Release Tracks | Track | Versions | Branch | Docker tags | PyPI install | |-------|----------|--------|-------------|--------------| -| **Stable** | `1.1.0`, `1.1.1` | `stable/1.1` | `:1.1.0`, `:1.1`, `:stable`, `:latest` | `pip install turnstone` | -| **Experimental** | `1.2.0a1`, `1.2.0a2` | `main` | `:1.2.0a1`, `:experimental` | `pip install turnstone --pre` | +| **Legacy 1.0** | `1.0.x` | `stable/1.0` | `:1.0.x`, `:1.0` | `pip install 'turnstone==1.0.*'` | +| **Stable 1.3** | `1.3.x` | `stable/1.3` | `:1.3.x`, `:1.3` | `pip install 'turnstone==1.3.*'` | +| **Stable 1.4** | `1.4.x` | `stable/1.4` | `:1.4.x`, `:1.4`, `:stable`, `:latest` | `pip install turnstone` | +| **Experimental** | `1.5.0aN` | `main` | `:1.5.0aN`, `:experimental` | `pip install turnstone --pre` | -- **Stable** receives bugfixes only. Production-grade. -- **Experimental** receives new features. May be rough around the edges. -- When experimental matures, it is promoted to stable. The previous stable branch stops receiving patches. +- **Stable** tracks receive bugfixes only. The most-recent stable minor + owns the `:stable` / `:latest` Docker tags and the default PyPI + install. +- **Experimental** (always on `main`) receives new features. May be + rough around the edges. +- When experimental matures, it is promoted to a new stable minor via + a `stable/X.Y` branch; older stable branches continue to receive + security fixes until explicitly retired. ## Version Scheme @@ -26,17 +33,17 @@ Turnstone uses two parallel release tracks published from a single PyPI package. ## Releasing an Experimental Version (from main) ```bash -scripts/release.sh 1.1.0a2 --push +scripts/release.sh 1.5.0a2 --push ``` -This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.1.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically. +This bumps `pyproject.toml` + `turnstone/__init__.py`, regenerates `uv.lock`, commits, tags `v1.5.0a2`, and pushes. CI runs, then publish + Docker workflows fire automatically. ## Releasing a Stable Patch (from stable/X.Y) ```bash -git checkout stable/1.0 +git checkout stable/1.4 git cherry-pick # bugfix from main -scripts/release.sh 1.0.2 --push +scripts/release.sh 1.4.1 --push ``` ## Promoting Experimental to Stable @@ -45,17 +52,19 @@ When `main` is ready for a stable release: ```bash # 1. Tag the stable release on main -scripts/release.sh 1.1.0 --push +scripts/release.sh 1.5.0 --push # 2. Create the stable maintenance branch from that tag -git branch stable/1.1 v1.1.0 -git push origin stable/1.1 +git branch stable/1.5 v1.5.0 +git push origin stable/1.5 # 3. Start the next experimental cycle on main -scripts/release.sh 1.2.0a1 --push +scripts/release.sh 1.6.0a1 --push ``` -The previous `stable/1.0` branch stops receiving patches at this point. +The previous stable branch (`stable/1.4`) continues to receive +security-only patches; older tracks (`stable/1.0`, `stable/1.3`) are +retired when they fall out of support. ## CI/CD Pipeline diff --git a/docs/sdk.md b/docs/sdk.md index 29c53784..4306b967 100644 --- a/docs/sdk.md +++ b/docs/sdk.md @@ -69,8 +69,12 @@ Both `TurnstoneServer` (sync) and `AsyncTurnstoneServer` (async) expose: |----------|--------|---------| | **Workstreams** | `list_workstreams()` | `ListWorkstreamsResponse` | | | `dashboard()` | `DashboardResponse` | -| | `create_workstream(*, name, model, auto_approve, skill)` | `CreateWorkstreamResponse` | +| | `create_workstream(*, name, model, auto_approve, skill, initial_message, attachments)` | `CreateWorkstreamResponse` | | | `close_workstream(ws_id)` | `StatusResponse` | +| **Attachments** | `upload_attachment(ws_id, file, filename, mime_type)` | `UploadAttachmentResponse` | +| | `list_attachments(ws_id)` | `ListAttachmentsResponse` | +| | `download_attachment(ws_id, attachment_id)` | `bytes` | +| | `delete_attachment(ws_id, attachment_id)` | `StatusResponse` | | **Chat** | `send(message, ws_id)` | `SendResponse` | | | `approve(*, ws_id, approved, feedback, always)` | `StatusResponse` | | | `plan_feedback(*, ws_id, feedback)` | `StatusResponse` | @@ -171,6 +175,36 @@ result.ok # True if no errors and not timed out result.timed_out # True if timeout expired ``` +### Attachments + +Upload files to a workstream and attach them to the next user turn: + +```python +# Upload separately, then send a message — attachments auto-attach +with open("screenshot.png", "rb") as f: + att = client.upload_attachment(ws.ws_id, f.read(), + filename="screenshot.png", + mime_type="image/png") +client.send("What's wrong in this screenshot?", ws.ws_id) + +# Or attach at workstream-creation time (multipart upload) +from turnstone.sdk import AttachmentUpload + +with open("notes.txt", "rb") as f: + ws = client.create_workstream( + name="triage", + initial_message="Summarize the notes", + attachments=[AttachmentUpload(data=f.read(), + filename="notes.txt", + mime_type="text/plain")], + ) +``` + +Limits: images ≤ 4 MiB (png/jpeg/gif/webp), text ≤ 512 KiB (UTF-8), +10 pending per (workstream, user). The SDK auto-generates `ws_id` on the +client so cluster-routed callers bind attachments to the owning node +before the request lands. + ### Error Handling Non-2xx responses raise `TurnstoneAPIError`: @@ -284,7 +318,7 @@ turnstone/sdk/ Python SDK (sub-package) _base.py Shared httpx async client, auth, error handling _sync.py Background event loop for sync wrappers _types.py TurnResult + TurnstoneAPIError - events.py 27 SSE event dataclasses with type registry + events.py 38 SSE event dataclasses with type registry server.py AsyncTurnstoneServer + TurnstoneServer console.py AsyncTurnstoneConsole + TurnstoneConsole diff --git a/docs/security.md b/docs/security.md index b39c4403..46311464 100644 --- a/docs/security.md +++ b/docs/security.md @@ -1,8 +1,10 @@ # Security and Authentication -Turnstone uses a layered authentication system with three token types, -hierarchical scopes, and a split architecture where the console manages -credentials while individual server nodes validate JWTs locally. +Turnstone uses a layered authentication system with two token types +(database-backed API tokens + HMAC-SHA256 JWTs), hierarchical scopes, +and a split architecture where the console manages credentials while +individual server nodes validate JWTs locally. Inter-service traffic +uses short-lived service JWTs minted by `ServiceTokenManager`. --- @@ -37,7 +39,7 @@ Claims: |-------|-------------| | `sub` | User ID | | `scopes` | Comma-separated scope list (`read,write,approve`) | -| `src` | Token source (`password`, `api_token`, `config`, `oidc`) | +| `src` | Token source (`password`, `api_token`, `oidc`, or a service origin like `console` / `cli`) | | `iss` | Issuer — always `turnstone` | | `aud` | Audience — `turnstone-server` or `turnstone-console` | | `iat` | Issued-at timestamp | diff --git a/docs/settings.md b/docs/settings.md index 6950cb88..96376db0 100644 --- a/docs/settings.md +++ b/docs/settings.md @@ -59,6 +59,22 @@ from ConfigStore. Model names and context windows are now configured per-model in the Models tab. A startup warning is logged if these keys appear in `config.toml`. +### Plan / task agent overrides + +`plan_agent` and `task_agent` sub-sessions resolve independently from the +conversation model so operators can pick a cheaper/faster model for +autonomous loops: + +| Setting | Purpose | +|---------|---------| +| `model.plan_model` | Alias used for `plan_agent` sub-sessions. Falls back to `[model].plan_model` in config.toml, then `[model].agent_model`, then the session's active model. | +| `model.task_model` | Alias used for `task_agent` sub-sessions. Same fallback chain as `plan_model`. | +| `model.plan_effort` | Reasoning effort for `plan_agent` (`none` / `minimal` / `low` / `medium` / `high` / `xhigh` / `max`). Defaults to `high`. | +| `model.task_effort` | Reasoning effort for `task_agent`. Empty string means "inherit from the session". | + +All four are live-editable from the Settings tab and take effect on the +next sub-agent invocation — no restart required. + --- ## Bootstrap vs ConfigStore @@ -79,7 +95,7 @@ initialization: | Section | Settings | |---------|----------| -| `model` | default_alias, temperature, max_tokens, reasoning_effort | +| `model` | default_alias, temperature, max_tokens, reasoning_effort, plan_model, task_model, plan_effort, task_effort | | `session` | instructions, retention_days, compact_max_tokens, auto_compact_pct | | `tools` | timeout, truncation, agent_max_turns, skip_permissions, search, search_threshold, search_max_results | | `server` | workstream_idle_timeout, max_workstreams | diff --git a/docs/tools.md b/docs/tools.md index 2212f76c..7808e663 100644 --- a/docs/tools.md +++ b/docs/tools.md @@ -169,8 +169,8 @@ Every tool defines a `primary_key`. The mapping is: | `man` | `page` | | `web_fetch` | `url` | | `web_search` | `query` | -| `task` | `prompt` | -| `plan` | `prompt` | +| `task_agent` | `prompt` | +| `plan_agent` | `prompt` | | `memory` | `name` | | `recall` | `query` | | `notify` | `message` | @@ -357,7 +357,10 @@ Search the web using a text query. ## Agent -### task +Tool names use the `_agent` suffix — bare `plan` / `task` collide with +chat-template channel names on some local models. + +### task_agent Delegate a general-purpose task to an autonomous sub-agent. @@ -371,7 +374,7 @@ Delegate a general-purpose task to an autonomous sub-agent. --- -### plan +### plan_agent Plan before implementing -- an autonomous agent explores the codebase and writes a structured plan. @@ -568,8 +571,8 @@ pre-configure skills at workstream creation. | `man` | Info | Yes | Yes | Yes | `page` | | `web_fetch` | Info | No | Yes | Yes | `url` | | `web_search` | Info | No | Yes | Yes | `query` | -| `task` | Agent | No | No | No | `prompt` | -| `plan` | Agent | No | No | No | `prompt` | +| `task_agent` | Agent | No | No | No | `prompt` | +| `plan_agent` | Agent | No | No | No | `prompt` | | `memory` | Memory | Yes | No | No | `name` | | `recall` | Memory | Yes | No | No | `query` | | `notify` | Notify | Yes | Yes | Yes | `message` |