mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-13 15:32:24 -06:00
Compare commits
23 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| bb221f4dab | |||
| 361876b17a | |||
| 5d26cd6593 | |||
| 2d4420e00d | |||
| b20548583d | |||
| f63b2915cc | |||
| bf85bbea94 | |||
| 803d8ee8f9 | |||
| 17b5961a70 | |||
| 037308f3b1 | |||
| d7a9895855 | |||
| 0ba49b8bb7 | |||
| 51b5b3ee74 | |||
| c3b0ddeba7 | |||
| a533e1c783 | |||
| d8bc78556f | |||
| 4f5854e768 | |||
| bf2dc04cb3 | |||
| bb894d073b | |||
| b3934a2d14 | |||
| 3d02cf66b4 | |||
| 7631b88792 | |||
| a07172b0c0 |
+3
-3
@@ -8,7 +8,7 @@ FROM python:3.14-slim
|
||||
LABEL org.opencontainers.image.title="turnstone" \
|
||||
org.opencontainers.image.description="Multi-node AI orchestration platform"
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.10.12 /uv /usr/local/bin/uv
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.11.1 /uv /usr/local/bin/uv
|
||||
|
||||
# System dependencies for psycopg (PostgreSQL client library)
|
||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends libpq5 \
|
||||
@@ -25,12 +25,12 @@ ENV UV_COMPILE_BYTECODE=1
|
||||
# Install dependencies first (cached layer — only re-runs when deps change)
|
||||
COPY pyproject.toml uv.lock README.md LICENSE ./
|
||||
RUN uv sync --frozen --no-install-project --no-dev \
|
||||
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic
|
||||
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic --extra ddg
|
||||
|
||||
# Install the project itself
|
||||
COPY turnstone/ turnstone/
|
||||
RUN uv sync --frozen --no-dev \
|
||||
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic
|
||||
--extra mq --extra console --extra sim --extra postgres --extra discord --extra anthropic --extra ddg
|
||||
|
||||
# Add venv to PATH so entry points are found
|
||||
ENV PATH="/app/.venv/bin:$PATH"
|
||||
|
||||
@@ -91,7 +91,7 @@ turnstone/
|
||||
_config.py Base ChannelConfig dataclass
|
||||
discord/ Discord adapter (bot, cog, views, streaming, config)
|
||||
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
|
||||
katex-0.16.40/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
katex-0.16.42/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
|
||||
ui/
|
||||
colors.py ANSI color constants with NO_COLOR support
|
||||
markdown.py Streaming terminal markdown renderer (line-buffered)
|
||||
|
||||
+1
-1
@@ -365,7 +365,7 @@ SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte
|
||||
|
||||
### Authentication
|
||||
|
||||
The proxy forwards the user's JWT to upstream server nodes — it extracts the token from the incoming request's cookie (or `Authorization` header) and adds it as a `Bearer` header on the proxied request. Since all services share the same `TURNSTONE_JWT_SECRET`, the user's JWT is valid on every node without re-authentication. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require `write` scope, preventing read-only tokens from escalating via proxy. The static `--auth-token` / `proxy_auth_token` is used as a fallback when no user JWT is present.
|
||||
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -61,6 +61,7 @@ package "Inbound Messages (Client → Bridge)" as InPkg #FFF3E0 {
|
||||
+ target_node: str = ""
|
||||
+ initial_message: str = ""
|
||||
+ skill: str = ""
|
||||
+ user_id: str = ""
|
||||
}
|
||||
|
||||
class CloseWorkstreamMessage {
|
||||
|
||||
@@ -154,7 +154,7 @@ activate Server #FFECB3
|
||||
Server -> CC : _pick_best_node() or\nget_node_detail(node_id)
|
||||
CC --> Server : node validated
|
||||
|
||||
Server -> Server : Build CreateWorkstreamMessage\n{target_node:"nodeA", name:"new-task"}
|
||||
Server -> Server : Build CreateWorkstreamMessage\n{target_node:"nodeA", name:"new-task",\nuser_id: from auth_result}
|
||||
|
||||
Server -> Redis : RPUSH turnstone:inbound:nodeA\n(directed queue)
|
||||
Server --> Browser : {status:"ok", correlation_id:"abc",\ntarget_node:"nodeA"}
|
||||
@@ -163,7 +163,8 @@ deactivate Server
|
||||
note right of Redis
|
||||
Bridge on Node-A picks up the
|
||||
message from its directed queue,
|
||||
POSTs to /v1/api/workstreams/new,
|
||||
POSTs to /v1/api/workstreams/new
|
||||
(forwarding user_id in payload),
|
||||
registers ownership, publishes
|
||||
ws_created to cluster channel.
|
||||
end note
|
||||
|
||||
@@ -176,4 +176,15 @@ note bottom of SH
|
||||
Both share JWT signing secret
|
||||
end note
|
||||
|
||||
note left of JWT
|
||||
**Console Proxy Token Minting**
|
||||
When proxying requests to server nodes:
|
||||
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
|
||||
2. Proxy mints new JWT (aud: turnstone-server)
|
||||
with real user_id, scopes, permissions
|
||||
3. src: "console-proxy" for audit traceability
|
||||
4. 5-minute expiry (fresh per request)
|
||||
5. Fallback: ServiceTokenManager if no user context
|
||||
end note
|
||||
|
||||
@enduml
|
||||
|
||||
+254
-68
@@ -2,7 +2,8 @@
|
||||
|
||||
`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
|
||||
runs test cases against the LLM, scores tool call sequences against expected
|
||||
actions, and optionally uses the model to self-optimize the developer prompt.
|
||||
actions, and optionally uses a multi-agent pipeline to optimize the developer
|
||||
prompt and tool descriptions.
|
||||
|
||||
Source: `turnstone/eval.py`
|
||||
|
||||
@@ -10,15 +11,24 @@ Source: `turnstone/eval.py`
|
||||
|
||||
## Overview
|
||||
|
||||
The system works in an iterative loop:
|
||||
The system uses UCB tree search to explore prompt variants:
|
||||
|
||||
1. Run each test case N times against the current developer prompt.
|
||||
2. Score each run by comparing the actual tool call sequence to expected actions.
|
||||
3. If not all tests pass, use the model to rewrite the prompt based on failures.
|
||||
4. Repeat until all tests pass or max iterations are reached.
|
||||
1. Maintain an **evolution tree** of prompt variants, starting from the initial prompt.
|
||||
2. Each iteration, **UCB1 selects** the most promising node to evaluate.
|
||||
3. Run each test case N times against the selected prompt.
|
||||
4. Score each run by comparing the actual tool call sequence to expected actions.
|
||||
5. If not all tests pass, run a **three-phase optimization pipeline**:
|
||||
- Phase 1: Analyst diagnoses semantic failure patterns
|
||||
- Phase 2: Tool optimizer adjusts tool descriptions (when `--optimize-tools`)
|
||||
- Phase 3: Prompt optimizer proposes a child variant (when not `--optimize-tools`)
|
||||
6. Add the child to the tree and repeat until all tests pass or max iterations reached.
|
||||
|
||||
When optimization is disabled (`--no-optimize`), only step 1 and 2 execute
|
||||
(a single iteration).
|
||||
This approach (inspired by [Learning to Self-Evolve](https://arxiv.org/abs/2603.18620))
|
||||
prevents irrecoverable collapse from bad edits — UCB naturally backtracks to
|
||||
high-scoring ancestors instead of following a linear chain.
|
||||
|
||||
When optimization is disabled (`--no-optimize`), only steps 2-4 execute
|
||||
(a single iteration evaluating the root node).
|
||||
|
||||
---
|
||||
|
||||
@@ -65,6 +75,7 @@ Test suites are JSON files with this structure:
|
||||
| `match_mode` | no | `"ordered_subset"` | How to match actual vs expected actions (see Scoring). |
|
||||
| `max_turns` | no | `10` | Maximum conversation turns before stopping. |
|
||||
| `n_runs` | no | suite default or 3 | Per-case override for number of runs. |
|
||||
| `holdout` | no | `false` | If `true`, this case is evaluated but excluded from optimizer feedback. Used to measure progress without overfitting. |
|
||||
|
||||
### Expected Action Specs
|
||||
|
||||
@@ -135,6 +146,7 @@ deterministic, non-interactive execution suitable for automated testing.
|
||||
| Stdout | Normal | Suppressed during execution |
|
||||
| Tool logging | Display only | Structured `tool_call_log` |
|
||||
| System prompt | Built-in developer prompt | Overridable via constructor |
|
||||
| Cancellation | N/A | `_cancelled` event for timeout cleanup |
|
||||
|
||||
### NullUI
|
||||
|
||||
@@ -156,20 +168,34 @@ def send_headless(
|
||||
Runs a complete multi-turn conversation:
|
||||
|
||||
1. Appends the user message.
|
||||
2. Calls the model API (non-streaming).
|
||||
3. If tool calls are returned, executes them (with stdout suppressed) and
|
||||
2. Checks `_cancelled` event — stops if set (timeout cleanup).
|
||||
3. Calls the model API (non-streaming).
|
||||
4. If tool calls are returned, executes them (with stdout suppressed) and
|
||||
logs each call to `self.tool_call_log`.
|
||||
4. Repeats up to `max_turns` or until the model responds without tool calls.
|
||||
5. Returns the tool call log: list of dicts with keys `tool`, `args`,
|
||||
5. Repeats up to `max_turns` or until the model responds without tool calls.
|
||||
6. Returns the tool call log: list of dicts with keys `tool`, `args`,
|
||||
`result` (truncated to 500 chars), and `turn`.
|
||||
|
||||
Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
|
||||
|
||||
### Timeout and Cancellation
|
||||
|
||||
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
|
||||
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
|
||||
a matching httpx read timeout. On timeout, three layers of defense prevent
|
||||
zombie connections:
|
||||
|
||||
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
|
||||
releases the server slot.
|
||||
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
|
||||
3. **`run_client.close()`**: Closes the connection pool to abort any
|
||||
in-flight request.
|
||||
|
||||
### Retry Logic
|
||||
|
||||
`send_headless()` is called inside `_run_single_test()` with retry logic:
|
||||
3 attempts with exponential backoff (sleep `2^attempt` seconds) on any
|
||||
exception. This prevents transient API errors from poisoning eval scores.
|
||||
exception. `TimeoutError` is re-raised immediately (no retry).
|
||||
|
||||
---
|
||||
|
||||
@@ -180,68 +206,175 @@ Each test case runs in isolation:
|
||||
1. A fresh temp directory is created.
|
||||
2. Setup files are written to the temp directory.
|
||||
3. The working directory is changed to the temp directory.
|
||||
4. A new `HeadlessSession` is created with the current developer prompt.
|
||||
5. `send_headless()` runs the user prompt through the conversation loop.
|
||||
6. The tool log is scored against expected actions.
|
||||
7. The temp directory is cleaned up.
|
||||
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
|
||||
5. A new `HeadlessSession` is created with the current developer prompt.
|
||||
6. `send_headless()` runs the user prompt through the conversation loop.
|
||||
7. The tool log is scored against expected actions.
|
||||
8. The temp directory is cleaned up.
|
||||
|
||||
The memory database is also isolated per test (an ephemeral SQLite database
|
||||
in the temp directory) so tests do not pollute each other or the user's
|
||||
real memory store.
|
||||
|
||||
### Parallel Execution
|
||||
|
||||
With `--parallel N` (N > 1), tests run in a `ProcessPoolExecutor` with N
|
||||
workers. Each subprocess creates its own `OpenAI` client. This is suitable
|
||||
for remote API endpoints but will overwhelm local inference servers. The
|
||||
default (`--parallel 1`) runs tests serially.
|
||||
|
||||
---
|
||||
|
||||
## Optimization Loop
|
||||
## Model Roles
|
||||
|
||||
`run_optimization()` is the main entry point for iterative prompt optimization.
|
||||
The eval pipeline uses up to five separate model roles, each independently
|
||||
configurable. All roles inherit from the test model by default, with a
|
||||
cascade chain:
|
||||
|
||||
```
|
||||
test model (--base-url, --model)
|
||||
└─ optimizer (--optimizer-*)
|
||||
├─ observer (--observer-*)
|
||||
├─ analyst (--analyst-*)
|
||||
├─ diversifier (--diversifier-*)
|
||||
└─ tool optimizer (--tool-optimizer-*)
|
||||
```
|
||||
|
||||
| Role | Purpose | When it runs |
|
||||
|------|---------|--------------|
|
||||
| **Test** | The model being evaluated | Every iteration |
|
||||
| **Analyst** | Diagnoses semantic failure patterns with tool use | When pass rate < 100% |
|
||||
| **Optimizer** | Rewrites the developer prompt | Every iteration (unless `--optimize-tools`) |
|
||||
| **Tool optimizer** | Rewrites tool descriptions | When `--optimize-tools` is set |
|
||||
| **Observer** | Tunes the optimizer's strategy | Every 3 iterations |
|
||||
| **Diversifier** | Generates prompt paraphrases | Once before the loop (when `--diversify N`) |
|
||||
|
||||
Typical setup: local model for test, Opus for analyst, Sonnet for
|
||||
optimizer/observer/diversifier.
|
||||
|
||||
---
|
||||
|
||||
## Optimization Pipeline
|
||||
|
||||
### Flow
|
||||
|
||||
```
|
||||
for iteration in 0..max_iterations:
|
||||
1. Run all test cases n_runs times with current prompt
|
||||
2. Score and aggregate results
|
||||
3. Save intermediate results to JSON
|
||||
4. If all tests pass -> stop
|
||||
5. Every 3 iterations (at iteration 2, 5, 8, ...):
|
||||
-> Observer reviews optimizer strategy
|
||||
-> Reset prompt to best-performing iteration
|
||||
6. Propose new prompt via optimizer model call
|
||||
7. If prompt unchanged -> stop
|
||||
8. Continue with new prompt
|
||||
1. UCB select → pick the most promising tree node
|
||||
2. Run all test cases n_runs times with selected node's prompt
|
||||
3. Update node score (rolling mean) and visit count
|
||||
4. Save intermediate results + tree state to JSON
|
||||
5. If all tests pass → stop
|
||||
6. Phase 1: Analyst diagnoses semantic failure patterns
|
||||
7. Phase 2 (--optimize-tools only): Tool optimizer adjusts descriptions
|
||||
8. Phase 3 (default only): Prompt optimizer proposes new prompt
|
||||
9. Every 3 iterations: Observer tunes the optimizer's strategy
|
||||
10. Add child node to tree (if prompt or tools changed)
|
||||
```
|
||||
|
||||
### Prompt Proposal (`_propose_prompt_modification`)
|
||||
### Phase 1: Analyst (`_run_analyst`)
|
||||
|
||||
Uses the model to rewrite the developer prompt based on test results:
|
||||
A multi-turn agent with `math` (Python) and `bash` tools for computing
|
||||
statistics. It receives per-case results with failure classifications and
|
||||
produces a structured diagnosis:
|
||||
|
||||
- **Input**: Current prompt, test case definitions, per-case results with
|
||||
actual vs expected tool sequences, and a history of the last 3 iterations.
|
||||
- **Optimizer system prompt** (`OPTIMIZER_SYSTEM`): Instructs the model to
|
||||
act as a text rewriter. Key guidance includes:
|
||||
- Address critical failure modes (text-only responses, write_file vs edit_file,
|
||||
unnecessary search before create, missing plan calls).
|
||||
- Preserve phrasing that drives 100% pass rate on passing tests.
|
||||
- Use direct imperative style with concrete tool call examples.
|
||||
- Stay within 130% of original prompt length.
|
||||
- **Output**: The rewritten prompt text (stripped of reasoning tags and code fences).
|
||||
- **Failure patterns**: Shared root causes across failing cases
|
||||
- **Success/failure contrast**: What distinguishes passing from failing cases
|
||||
- **Consistency signals**: Systematic (0%), flaky (1-79%), marginal (80-99%)
|
||||
- **Recommended fixes**: Priority-ordered patterns/examples to add or adjust
|
||||
|
||||
### Observer System (`_observe_and_update_optimizer`)
|
||||
The analyst is instructed to frame fixes as patterns and examples, not
|
||||
imperative rules — this feeds cleaner signal to the optimizer.
|
||||
|
||||
Every 3 iterations, a meta-level "observer" reviews the optimizer's strategy:
|
||||
In `--optimize-tools` mode, the analyst receives the current tool descriptions
|
||||
(with any overrides applied) and focuses on tool confusion and description
|
||||
issues rather than system prompt patterns.
|
||||
|
||||
- Analyzes the iteration history: score trends, regressions, prompt length changes,
|
||||
### Phase 2: Tool Optimizer (`_propose_tool_overrides`)
|
||||
|
||||
Runs when `--optimize-tools` is set. Receives the current tool descriptions,
|
||||
confusion failures (where the model picked the wrong tool), and the analyst's
|
||||
diagnosis. Returns a JSON override dict that modifies tool descriptions.
|
||||
|
||||
Overrides are validated against known tool names — only `description` and
|
||||
`parameters` changes are accepted (no tool renaming at eval time).
|
||||
|
||||
After each iteration, changed descriptions are logged as old → new diffs
|
||||
for easy visual inspection.
|
||||
|
||||
### Phase 3: Prompt Optimizer (`_propose_prompt_modification`)
|
||||
|
||||
Skipped in `--optimize-tools` mode. Receives the current prompt, test
|
||||
results with per-case pass rates and deltas from the parent node, and the
|
||||
analyst's diagnosis. Returns a rewritten prompt.
|
||||
|
||||
The optimizer is instructed to prefer patterns over rules — concrete tool
|
||||
chain examples teach better than imperative directives like "ALWAYS" or
|
||||
"NEVER." If the current prompt contains rule-heavy language, the optimizer
|
||||
is guided to replace it with examples.
|
||||
|
||||
### Two Optimization Surfaces
|
||||
|
||||
The system supports alternating between two optimization surfaces:
|
||||
|
||||
1. **System prompt optimization** (default): Freeze tool descriptions,
|
||||
optimize the developer prompt. Run until scores plateau.
|
||||
2. **Tool description optimization** (`--optimize-tools`): Freeze the system
|
||||
prompt, optimize tool descriptions only. Run until scores plateau.
|
||||
|
||||
Each surface lifts the floor for the other — tool description improvements
|
||||
may unlock system prompt gains that weren't reachable before, and vice versa.
|
||||
|
||||
### Observer (`_observe_and_update_optimizer`)
|
||||
|
||||
Every 3 iterations, a meta-level observer reviews the optimizer's strategy:
|
||||
|
||||
- Analyzes iteration history: score trends, regressions, prompt length changes,
|
||||
and diffs between iterations.
|
||||
- Summarizes the optimizer's behavioral patterns (list style, header usage, length).
|
||||
- Uses `OBSERVER_SYSTEM` to rewrite the optimizer's own system prompt.
|
||||
- Detects whether the optimizer is producing rule-heavy or pattern-based output.
|
||||
- Rewrites the optimizer's own system prompt to correct course.
|
||||
- Rejects degenerate outputs (over 200% of input length).
|
||||
- After updating the optimizer prompt, resets the developer prompt to the
|
||||
best-performing iteration so far.
|
||||
|
||||
This two-level optimization (optimizer + observer) helps the system escape
|
||||
local minima and adjust its rewriting strategy.
|
||||
### Prompt Diversification
|
||||
|
||||
### Result Persistence
|
||||
When `--diversify N` is set, the diversifier generates N paraphrased variants
|
||||
of each test case's user prompt before the optimization loop. Each run cycles
|
||||
through variants (round-robin), testing robustness across phrasings.
|
||||
|
||||
Variants can be cached back to the test suite JSON with `--save-variants`,
|
||||
and auto-loaded on subsequent runs even without `--diversify`.
|
||||
|
||||
---
|
||||
|
||||
## Evolution Tree
|
||||
|
||||
The optimization maintains a tree of prompt variants (`EvolutionNode`), where
|
||||
each node stores its prompt text, tool overrides, aggregated score, and visit
|
||||
count. The root node (ID 0) contains the initial prompt.
|
||||
|
||||
**UCB1 selection**: Each iteration picks the node with the highest Upper
|
||||
Confidence Bound score: `R_bar + C * sqrt(ln(N) / v)`, where `R_bar` is the
|
||||
node's mean score, `N` is total visits across all nodes, `v` is the node's
|
||||
visit count, and `C` is the exploration constant (`--explore-constant`,
|
||||
default sqrt(2)). Unvisited nodes are always selected first.
|
||||
|
||||
### Holdout Cases
|
||||
|
||||
Test cases with `"holdout": true` are evaluated every iteration but excluded
|
||||
from the optimizer's feedback. This prevents the optimizer from overfitting
|
||||
to specific test cases. Node scores are computed from holdout cases only
|
||||
(when present). If fewer than 2 non-holdout cases remain, holdout is disabled.
|
||||
|
||||
### Improvement-Based Feedback
|
||||
|
||||
The optimizer sees delta scores (`delta=+20%`) alongside absolute pass rates,
|
||||
showing how each case improved relative to the parent node's evaluation. This
|
||||
provides a cleaner signal than absolute scores alone — the optimizer can
|
||||
distinguish beneficial edits from harmful ones regardless of starting point.
|
||||
|
||||
---
|
||||
|
||||
## Result Persistence
|
||||
|
||||
After each iteration, results are written to the output JSON file. The
|
||||
structure is:
|
||||
@@ -251,9 +384,15 @@ structure is:
|
||||
"meta": {
|
||||
"model": "model-name",
|
||||
"base_url": "http://localhost:8000/v1",
|
||||
"optimizer_model": "claude-opus-4-6",
|
||||
"observer_model": "claude-opus-4-6",
|
||||
"started": "2025-01-01T00:00:00",
|
||||
"test_suite": "tests.json",
|
||||
"n_runs_default": 3
|
||||
"n_runs_default": 3,
|
||||
"explore_constant": 1.414,
|
||||
"holdout_ids": [],
|
||||
"diversify": 10,
|
||||
"prompt_variants": {"case_id": ["variant1", "variant2"]}
|
||||
},
|
||||
"iterations": [
|
||||
{
|
||||
@@ -261,7 +400,11 @@ structure is:
|
||||
"prompt": "the developer prompt used",
|
||||
"prompt_diff": null,
|
||||
"optimizer_system": "the optimizer system prompt",
|
||||
"analyst": "analyst diagnosis output",
|
||||
"tool_overrides": {"bash": {"description": "..."}},
|
||||
"timestamp": "2025-01-01T00:01:00",
|
||||
"tree_node_id": 0,
|
||||
"tree_child_id": 1,
|
||||
"cases": {
|
||||
"test_name": {
|
||||
"runs": [
|
||||
@@ -287,9 +430,21 @@ structure is:
|
||||
"overall_pass_rate": 0.8,
|
||||
"overall_avg_score": 0.87,
|
||||
"json_dumps": 0,
|
||||
"per_case_pass_rates": {"test_name": 1.0, ...}
|
||||
"per_case_pass_rates": {"test_name": 1.0}
|
||||
}
|
||||
}
|
||||
],
|
||||
"tree": [
|
||||
{
|
||||
"node_id": 0,
|
||||
"parent_id": null,
|
||||
"prompt": "initial prompt",
|
||||
"tool_overrides": {},
|
||||
"score": 0.85,
|
||||
"visit_count": 3,
|
||||
"children": [1, 2],
|
||||
"iteration": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
@@ -306,26 +461,57 @@ turnstone-eval tests.json # evaluate + optimize
|
||||
turnstone-eval tests.json --no-optimize # evaluate only (single iteration)
|
||||
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
|
||||
turnstone-eval tests.json --prompt custom.txt # start from a custom prompt
|
||||
turnstone-eval tests.json --optimize-tools # optimize tool descriptions only
|
||||
turnstone-eval tests.json --diversify 10 # test with prompt variants
|
||||
turnstone-eval tests.json -v # verbose per-turn logging
|
||||
```
|
||||
|
||||
### Multi-model setup (local test model, cloud optimizer)
|
||||
|
||||
```
|
||||
turnstone-eval tests.json \
|
||||
--base-url http://localhost:8000/v1 \
|
||||
--optimizer-base-url https://api.anthropic.com \
|
||||
--optimizer-model claude-sonnet-4-6 \
|
||||
--analyst-model claude-opus-4-6
|
||||
```
|
||||
|
||||
### All Options
|
||||
|
||||
| Flag | Default | Description |
|
||||
|---------------------|-------------------------------|-------------|
|
||||
| `test_file` | (positional, required) | Path to test cases JSON file. |
|
||||
| `--base-url` | `http://localhost:8000/v1` | API base URL. |
|
||||
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
|
||||
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
|
||||
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
|
||||
| `--max-iter` | 5 | Maximum optimization iterations. |
|
||||
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
|
||||
| `--temperature` | 0.7 | Sampling temperature. |
|
||||
| `--max-tokens` | 32768 | Max completion tokens. |
|
||||
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
|
||||
| `--context-window` | 131072 | Context window size. |
|
||||
| `--output` | `eval_results.json` | Output results file path. |
|
||||
| `-v`, `--verbose` | false | Show detailed per-turn logging (API calls, tool args, results). |
|
||||
| Flag | Default | Description |
|
||||
|-------------------------|----------------------------|-------------|
|
||||
| `test_file` | (positional, required) | Path to test cases JSON file. |
|
||||
| `--base-url` | `http://localhost:8000/v1` | API base URL for the test model. |
|
||||
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
|
||||
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
|
||||
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
|
||||
| `--max-iter` | 5 | Maximum optimization iterations. |
|
||||
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
|
||||
| `--temperature` | 0.7 | Sampling temperature. |
|
||||
| `--max-tokens` | 32768 | Max completion tokens. |
|
||||
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
|
||||
| `--context-window` | 131072 | Context window size. |
|
||||
| `--output` | `eval_results.json` | Output results file path. |
|
||||
| `-v`, `--verbose` | false | Show detailed per-turn logging. |
|
||||
| `--explore-constant` | 1.414 (sqrt(2)) | UCB exploration constant C. |
|
||||
| `--test-timeout` | 300 | Per-test timeout in seconds. |
|
||||
| `--suite-timeout` | 0 (unlimited) | Total suite timeout in seconds. |
|
||||
| `--no-fast-fail` | false | Disable early termination on all-zero initial runs. |
|
||||
| `--parallel` | 1 (serial) | Parallel workers (0=auto, N=use N workers). |
|
||||
| `--optimizer-model` | same as `--model` | Model for prompt optimization. |
|
||||
| `--optimizer-base-url` | same as `--base-url` | Base URL for optimizer model. |
|
||||
| `--observer-model` | same as optimizer | Model for meta-optimization (observer). |
|
||||
| `--observer-base-url` | same as optimizer | Base URL for observer model. |
|
||||
| `--analyst-model` | same as optimizer | Model for failure analysis. |
|
||||
| `--analyst-base-url` | same as optimizer | Base URL for analyst model. |
|
||||
| `--diversify` | 0 (disabled) | Generate N prompt variants per test case. |
|
||||
| `--diversifier-model` | same as optimizer | Model for prompt diversification. |
|
||||
| `--diversifier-base-url`| same as optimizer | Base URL for diversifier model. |
|
||||
| `--save-variants` | false | Save generated variants back to test suite JSON. |
|
||||
| `--optimize-tools` | false | Optimize tool descriptions only (freeze system prompt). |
|
||||
| `--tool-optimizer-model` | same as optimizer | Model for tool description optimization. |
|
||||
| `--tool-optimizer-base-url` | same as optimizer | Base URL for tool optimizer model. |
|
||||
| `--save-tools` | false | Write optimized tool descriptions back to `turnstone/tools/*.json`. |
|
||||
|
||||
### Precedence for n_runs
|
||||
|
||||
|
||||
+35
-8
@@ -457,14 +457,30 @@ without any database.
|
||||
### Proxy auth forwarding
|
||||
|
||||
When the console proxies requests to server nodes (via `/node/{id}/...`
|
||||
routes), it uses a dedicated **service proxy token** with
|
||||
`aud: turnstone-server` and `write` scope. The user's console JWT
|
||||
(which has `aud: turnstone-console`) is **not** forwarded — it would be
|
||||
rejected by the server's audience validation.
|
||||
routes), it mints a **short-lived user-scoped JWT** with
|
||||
`aud: turnstone-server` carrying the real user's `user_id`, `scopes`,
|
||||
and `permissions`. The user's console JWT (which has
|
||||
`aud: turnstone-console`) is **not** forwarded directly — it would be
|
||||
rejected by the server's audience validation. Instead, the console
|
||||
re-signs a new JWT targeted at the server audience.
|
||||
|
||||
The proxy token is managed by a `ServiceTokenManager` that auto-rotates
|
||||
1-hour JWTs, refreshing at 80% of lifetime. If `--auth-token` is
|
||||
provided, that static token is used instead.
|
||||
Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
|
||||
|
||||
- **Audit attribution** — the upstream server records the real user in
|
||||
`ctx_user_id` and audit events, not a generic service identity.
|
||||
- **Scope narrowing** — a read-only console user's proxied request
|
||||
carries only `read` scope, not the full `{read, write, approve}` set.
|
||||
The server enforces this as defense in depth.
|
||||
- **Permission forwarding** — granular RBAC permissions from the
|
||||
console JWT are carried through to the server.
|
||||
|
||||
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
|
||||
distinguish proxied requests from direct logins in audit logs.
|
||||
|
||||
When no user context is available (auth disabled, or internal requests),
|
||||
the proxy falls back to a `ServiceTokenManager` with service identity
|
||||
`console-proxy` and full scopes. If `--auth-token` is provided, that
|
||||
static token is used as a final fallback.
|
||||
|
||||
### Service-to-service authentication
|
||||
|
||||
@@ -475,7 +491,7 @@ auto-rotating JWTs when communicating with server nodes:
|
||||
|---------|----------|-------|----------|---------|
|
||||
| Bridge | `bridge` | `approve` | `turnstone-server` | Tool approval proxy, message relay |
|
||||
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
|
||||
| Console proxy | `console-proxy` | `write` | `turnstone-server` | Proxied API calls |
|
||||
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
|
||||
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
|
||||
|
||||
Service tokens use 1-hour expiry with automatic refresh via
|
||||
@@ -483,6 +499,17 @@ Service tokens use 1-hour expiry with automatic refresh via
|
||||
httpx event hooks to ensure rotated tokens are picked up on SSE
|
||||
reconnects.
|
||||
|
||||
### User identity in MQ-dispatched workstreams
|
||||
|
||||
When the console creates a workstream via MQ (the normal path), the
|
||||
authenticated user's `user_id` is embedded in the
|
||||
`CreateWorkstreamMessage`. The bridge forwards this `user_id` in the
|
||||
HTTP payload when calling the server's `POST /v1/api/workstreams/new`.
|
||||
The server accepts a `user_id` from the request body **only when the
|
||||
caller is a trusted service** — identified by `token_source` matching
|
||||
`bridge`, `console-proxy`, or `console`. Regular API callers cannot
|
||||
override `user_id`; the server always uses their JWT identity.
|
||||
|
||||
Note that the channel gateway uses a distinct JWT audience
|
||||
(`turnstone-channel`) from the server (`turnstone-server`) and console
|
||||
(`turnstone-console`). A server-scoped JWT cannot authenticate to the
|
||||
|
||||
+7
-3
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
||||
|
||||
[project]
|
||||
name = "turnstone"
|
||||
version = "0.8.5"
|
||||
version = "0.8.8"
|
||||
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
|
||||
readme = "README.md"
|
||||
license = "BUSL-1.1"
|
||||
@@ -51,7 +51,7 @@ console = ["redis>=7.2", "croniter>=3.0"]
|
||||
sim = ["redis>=7.2"]
|
||||
anthropic = ["anthropic>=0.39"]
|
||||
postgres = ["psycopg[binary]>=3.2"]
|
||||
ddg = ["duckduckgo-search>=8.0"]
|
||||
ddg = ["ddgs>=9.0"]
|
||||
discord = ["discord.py>=2.4", "redis>=7.2"]
|
||||
all = ["turnstone[mq,console,sim,anthropic,postgres,discord,ddg]"]
|
||||
|
||||
@@ -78,7 +78,7 @@ include = [
|
||||
"turnstone/console/static/*.js",
|
||||
"turnstone/shared_static/*.css",
|
||||
"turnstone/shared_static/*.js",
|
||||
"turnstone/shared_static/katex-0.16.40/**/*",
|
||||
"turnstone/shared_static/katex-0.16.42/**/*",
|
||||
"turnstone/shared_static/hljs-11.11.1/**/*",
|
||||
"turnstone/shared_static/mermaid-11.13.0/**/*",
|
||||
"turnstone/sdk/py.typed",
|
||||
@@ -165,6 +165,10 @@ ignore_missing_imports = true
|
||||
module = ["frontmatter", "frontmatter.*"]
|
||||
ignore_missing_imports = true
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = ["ddgs", "ddgs.*"]
|
||||
ignore_missing_imports = true
|
||||
|
||||
[[tool.mypy.overrides]]
|
||||
module = ["turnstone.channels.discord.*"]
|
||||
disallow_subclassing_any = false
|
||||
|
||||
+344
-48
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"description": "turnstone behavior tests — tool selection, sequencing, and multi-step reasoning",
|
||||
"description": "turnstone behavior tests \u2014 tool selection, sequencing, and multi-step reasoning",
|
||||
"defaults": {
|
||||
"n_runs": 5,
|
||||
"max_turns": 15
|
||||
@@ -8,35 +8,91 @@
|
||||
{
|
||||
"id": "read-before-edit",
|
||||
"description": "Must read_file before edit_file on the same path",
|
||||
"user_prompt": "Fix the typo in config.py — change 'recieve' to 'receive'",
|
||||
"user_prompt": "Fix the typo in config.py \u2014 change 'recieve' to 'receive'",
|
||||
"setup": {
|
||||
"files": {
|
||||
"config.py": "# Config module\ndef recieve_data(source):\n \"\"\"Recieve data from source.\"\"\"\n return source.read()\n"
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "read_file", "args": { "path": "config.py" } },
|
||||
{ "tool": "edit_file", "args": { "path": "config.py" } }
|
||||
{
|
||||
"tool": "read_file",
|
||||
"args": {
|
||||
"path": "config.py"
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "edit_file",
|
||||
"args": {
|
||||
"path": "config.py"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "ordered_subset"
|
||||
"match_mode": "ordered_subset",
|
||||
"user_prompts": [
|
||||
"Fix the typo in config.py \u2014 change 'recieve' to 'receive'",
|
||||
"In config.py, correct the misspelling of 'recieve' to 'receive'",
|
||||
"Please update config.py by replacing 'recieve' with the correct spelling 'receive'",
|
||||
"There's a typo in config.py: 'recieve' should be 'receive'. Please fix it.",
|
||||
"Could you change 'recieve' to 'receive' in config.py?",
|
||||
"Go ahead and fix 'recieve' \u2192 'receive' in config.py",
|
||||
"I need the word 'recieve' corrected to 'receive' in the file config.py",
|
||||
"config.py has a spelling error \u2014 'recieve' needs to be changed to 'receive'",
|
||||
"Kindly rectify the typographical error in config.py, replacing 'recieve' with 'receive'",
|
||||
"Hey, swap 'recieve' for 'receive' in config.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "write-file-not-bash",
|
||||
"description": "Use write_file for file creation, not bash echo/cat",
|
||||
"user_prompt": "Create a file called hello.py that prints hello world",
|
||||
"expected_actions": [
|
||||
{ "tool": "write_file", "args_pattern": { "path": "hello\\.py" } }
|
||||
{
|
||||
"tool": "write_file",
|
||||
"args_pattern": {
|
||||
"path": "hello\\.py"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Create a file called hello.py that prints hello world",
|
||||
"Make a hello.py file that outputs hello world",
|
||||
"Please write a Python file named hello.py which prints hello world",
|
||||
"I need a file called hello.py that prints hello world",
|
||||
"Could you create hello.py with code that prints hello world?",
|
||||
"Write hello.py \u2014 it should print hello world",
|
||||
"Generate a hello.py file that outputs \"hello world\"",
|
||||
"I'd like you to create a file named hello.py that prints hello world",
|
||||
"Set up a file called hello.py to print hello world",
|
||||
"Kindly produce a hello.py file whose purpose is to print hello world"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bash-for-commands",
|
||||
"description": "Use bash for running system commands",
|
||||
"user_prompt": "What Python version is installed?",
|
||||
"expected_actions": [
|
||||
{ "tool": "bash", "args_pattern": { "command": "python" } }
|
||||
{
|
||||
"tool": "bash",
|
||||
"args_pattern": {
|
||||
"command": "python"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"What Python version is installed?",
|
||||
"Check which version of Python is currently installed",
|
||||
"Can you tell me the installed Python version?",
|
||||
"python --version please",
|
||||
"I need to know what version of Python is on this system",
|
||||
"Which Python version do we have?",
|
||||
"Could you look up the Python version that's installed here?",
|
||||
"Determine the currently installed Python version",
|
||||
"What's the Python version on this machine?",
|
||||
"Please check the Python version"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "search-for-patterns",
|
||||
@@ -49,13 +105,30 @@
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "search", "args_pattern": { "query": "test_" } }
|
||||
{
|
||||
"tool": "search",
|
||||
"args_pattern": {
|
||||
"query": "test_"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Find all functions that start with 'test_' in the project",
|
||||
"List every function in the project whose name begins with 'test_'",
|
||||
"I need to locate all functions prefixed with 'test_' across the project",
|
||||
"Could you search the project for any functions starting with 'test_'?",
|
||||
"Show me all the test_ prefixed functions in this project",
|
||||
"Hunt down every function that has a 'test_' prefix in the codebase",
|
||||
"I'm looking for all functions named test_* throughout the project",
|
||||
"Search the entire project for functions whose names start with test_",
|
||||
"What functions beginning with 'test_' exist in this project?",
|
||||
"Please identify all functions with the 'test_' prefix in the project files"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "multi-file-edit",
|
||||
"description": "Read and edit multiple files — must read before editing each, and edit both",
|
||||
"description": "Read and edit multiple files \u2014 must read before editing each, and edit both",
|
||||
"user_prompt": "Change the default port from 8000 to 9000 in both server.py and config.py",
|
||||
"setup": {
|
||||
"files": {
|
||||
@@ -64,12 +137,32 @@
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "read_file" },
|
||||
{ "tool": "read_file" },
|
||||
{ "tool": "edit_file" },
|
||||
{ "tool": "edit_file" }
|
||||
{
|
||||
"tool": "read_file"
|
||||
},
|
||||
{
|
||||
"tool": "read_file"
|
||||
},
|
||||
{
|
||||
"tool": "edit_file"
|
||||
},
|
||||
{
|
||||
"tool": "edit_file"
|
||||
}
|
||||
],
|
||||
"match_mode": "ordered_subset"
|
||||
"match_mode": "ordered_subset",
|
||||
"user_prompts": [
|
||||
"Change the default port from 8000 to 9000 in both server.py and config.py",
|
||||
"Update the default port to 9000 instead of 8000 in server.py and config.py",
|
||||
"Could you modify the port number from 8000 to 9000 in both config.py and server.py?",
|
||||
"Please replace port 8000 with 9000 in server.py and config.py",
|
||||
"I need the default port switched from 8000 to 9000 in both server.py and config.py",
|
||||
"In server.py and config.py, the default port should be changed from 8000 to 9000",
|
||||
"Swap out port 8000 for 9000 in config.py and server.py",
|
||||
"Would you mind updating the default port value from 8000 to 9000 across both server.py and config.py?",
|
||||
"The default port in server.py and config.py needs to be 9000 instead of 8000 \u2014 please make that change",
|
||||
"Go ahead and change 8000 to 9000 for the default port in both server.py and config.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "search-then-edit",
|
||||
@@ -83,51 +176,147 @@
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "search", "args_pattern": { "query": "MAX_RETRIES" } },
|
||||
{ "tool": "read_file" },
|
||||
{ "tool": "edit_file", "args_pattern": { "old_string": "3" } }
|
||||
{
|
||||
"tool": "search",
|
||||
"args_pattern": {
|
||||
"query": "MAX_RETRIES"
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "read_file"
|
||||
},
|
||||
{
|
||||
"tool": "edit_file",
|
||||
"args_pattern": {
|
||||
"old_string": "3"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "ordered_subset"
|
||||
"match_mode": "ordered_subset",
|
||||
"user_prompts": [
|
||||
"Find where MAX_RETRIES is defined and change it from 3 to 5",
|
||||
"Locate the definition of MAX_RETRIES and update its value from 3 to 5",
|
||||
"Could you search for where MAX_RETRIES is defined and modify it from 3 to 5?",
|
||||
"I need MAX_RETRIES changed from 3 to 5 \u2014 find where it's defined and update it",
|
||||
"Please find the MAX_RETRIES definition and bump it from 3 to 5",
|
||||
"Hunt down MAX_RETRIES in the codebase and change its value from 3 to 5",
|
||||
"Where is MAX_RETRIES set to 3? Change it to 5.",
|
||||
"Search the code for the MAX_RETRIES definition and alter it from 3 to 5",
|
||||
"I'd like you to locate MAX_RETRIES (currently 3) and set it to 5 instead",
|
||||
"Go find MAX_RETRIES and switch it from 3 to 5"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bash-git-log",
|
||||
"description": "Use bash for git commands, not other tools",
|
||||
"user_prompt": "Show me the git log for the last 5 commits",
|
||||
"expected_actions": [
|
||||
{ "tool": "bash", "args_pattern": { "command": "git\\s+log" } }
|
||||
{
|
||||
"tool": "bash",
|
||||
"args_pattern": {
|
||||
"command": "git\\s+log"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Show me the git log for the last 5 commits",
|
||||
"Display the 5 most recent git commits",
|
||||
"Can you pull up the git log limited to the last five commits?",
|
||||
"I need to see the git log showing only the previous 5 commits",
|
||||
"git log for the 5 latest commits, please",
|
||||
"Would you mind showing me the last five entries in the git log?",
|
||||
"Print out the most recent 5 commits from the git log",
|
||||
"I'd like to review the git log \u2014 just the last 5 commits",
|
||||
"Show the recent 5 commit history using git log",
|
||||
"Could you display the git commit history for the past five commits?"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "write-then-run",
|
||||
"description": "Create a script and run it to verify it works",
|
||||
"user_prompt": "Create a Python script called fib.py that prints the first 10 Fibonacci numbers, then run it to verify",
|
||||
"expected_actions": [
|
||||
{ "tool": "write_file", "args_pattern": { "path": "fib\\.py" } },
|
||||
{ "tool": "bash", "args_pattern": { "command": "python" } }
|
||||
{
|
||||
"tool": "write_file",
|
||||
"args_pattern": {
|
||||
"path": "fib\\.py"
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "bash",
|
||||
"args_pattern": {
|
||||
"command": "python"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "ordered_subset"
|
||||
"match_mode": "ordered_subset",
|
||||
"user_prompts": [
|
||||
"Create a Python script called fib.py that prints the first 10 Fibonacci numbers, then run it to verify",
|
||||
"Write a Python file named fib.py that outputs the first 10 Fibonacci numbers, and execute it to confirm it works",
|
||||
"Please make fib.py \u2013 a Python script printing the first ten Fibonacci numbers \u2013 then run it to check the output",
|
||||
"I need a Python script fib.py that prints the first 10 Fibonacci numbers. Execute it afterwards to verify correctness.",
|
||||
"Could you create fib.py to display the first 10 Fibonacci numbers in Python, and then run it to make sure it works?",
|
||||
"Draft a script called fib.py in Python that outputs the first ten Fibonacci numbers, then execute it to validate",
|
||||
"Hey, write me a fib.py that prints the first 10 Fibonacci numbers and run it so we can see it works",
|
||||
"Generate a Python program fib.py which prints the initial 10 Fibonacci numbers, and verify by running it",
|
||||
"Kindly produce a Python script named fib.py to print the first 10 Fibonacci numbers, then execute the script to confirm its output",
|
||||
"Make a file fib.py containing Python code to print the first 10 Fibonacci numbers. Then run it to verify."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "no-bash-for-file-write",
|
||||
"description": "Should NOT use bash (echo/cat/heredoc) to create files — only write_file",
|
||||
"description": "Should NOT use bash (echo/cat/heredoc) to create files \u2014 only write_file",
|
||||
"user_prompt": "Create a new file called README.md with a title and description of this project",
|
||||
"expected_actions": [
|
||||
{ "tool": "write_file", "args_pattern": { "path": "README" } }
|
||||
{
|
||||
"tool": "write_file",
|
||||
"args_pattern": {
|
||||
"path": "README"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Create a new file called README.md with a title and description of this project",
|
||||
"Make a README.md file that includes a project title and description",
|
||||
"I need a README.md created with a title and a brief description of the project",
|
||||
"Please generate a README.md file containing the project's title and description",
|
||||
"Could you set up a README.md with a title and project description?",
|
||||
"Write a README.md that has a title and describes this project",
|
||||
"Go ahead and create README.md \u2014 it should have a title and a description of the project",
|
||||
"I'd like you to produce a new README.md file featuring a project title and description",
|
||||
"Kindly establish a README.md file incorporating both a title and a description for this project",
|
||||
"Spin up a README.md with a project title and description in it"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "plan-before-refactor",
|
||||
"description": "Use the plan tool before a large refactoring task",
|
||||
"user_prompt": "I need to refactor this codebase to separate the database layer from the API layer. Use the plan tool to think through the approach before making any changes.",
|
||||
"id": "plan-when-asked",
|
||||
"description": "Call the plan tool when the user asks to plan",
|
||||
"user_prompt": "Plan how to add user authentication to this app.",
|
||||
"setup": {
|
||||
"files": {
|
||||
"app.py": "import sqlite3\nfrom flask import Flask, jsonify\n\napp = Flask(__name__)\nDB = 'data.db'\n\ndef get_db():\n return sqlite3.connect(DB)\n\n@app.route('/users')\ndef list_users():\n db = get_db()\n users = db.execute('SELECT * FROM users').fetchall()\n db.close()\n return jsonify(users)\n\n@app.route('/users/<int:uid>')\ndef get_user(uid):\n db = get_db()\n user = db.execute('SELECT * FROM users WHERE id=?', (uid,)).fetchone()\n db.close()\n return jsonify(user)\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
|
||||
"app.py": "from flask import Flask, jsonify\n\napp = Flask(__name__)\n\n@app.route('/users')\ndef list_users():\n return jsonify([{'id': 1, 'name': 'Alice'}])\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
|
||||
}
|
||||
},
|
||||
"expected_actions": [{ "tool": "create_plan" }],
|
||||
"match_mode": "subset"
|
||||
"expected_actions": [
|
||||
{
|
||||
"tool": "plan_agent"
|
||||
}
|
||||
],
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Plan how to add user authentication to this app.",
|
||||
"Make a plan for adding pagination to the API endpoints.",
|
||||
"Plan out how to add error handling to this application.",
|
||||
"I need a plan for adding logging to this codebase.",
|
||||
"Plan the approach for adding unit tests to this app.",
|
||||
"How would you approach adding user authentication to this app? Lay out a plan.",
|
||||
"I'd like you to outline a strategy for implementing user authentication in this application.",
|
||||
"Could you come up with a plan for integrating user authentication into this app?",
|
||||
"Think through the steps needed to add user auth to this app and present a plan.",
|
||||
"Draft a plan for incorporating user authentication functionality into this application."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "edit-not-rewrite",
|
||||
@@ -139,10 +328,32 @@
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "read_file", "args": { "path": "utils.py" } },
|
||||
{ "tool": "edit_file", "args": { "path": "utils.py" } }
|
||||
{
|
||||
"tool": "read_file",
|
||||
"args": {
|
||||
"path": "utils.py"
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "edit_file",
|
||||
"args": {
|
||||
"path": "utils.py"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "ordered_subset"
|
||||
"match_mode": "ordered_subset",
|
||||
"user_prompts": [
|
||||
"Add a docstring to the process_data function in utils.py",
|
||||
"Please add a docstring to the process_data function in utils.py",
|
||||
"Could you write a docstring for process_data in utils.py?",
|
||||
"Insert a docstring into the process_data function found in utils.py",
|
||||
"I need a docstring added to process_data in utils.py",
|
||||
"Put a docstring on the process_data function in utils.py",
|
||||
"The process_data function in utils.py is missing a docstring \u2014 please add one",
|
||||
"Would you mind adding a docstring to process_data in utils.py?",
|
||||
"In utils.py, the process_data function needs a docstring",
|
||||
"Add documentation via a docstring to the process_data function within utils.py"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "bash-run-tests",
|
||||
@@ -154,45 +365,130 @@
|
||||
}
|
||||
},
|
||||
"expected_actions": [
|
||||
{ "tool": "bash", "args_pattern": { "command": "pytest|python.*test" } }
|
||||
{
|
||||
"tool": "bash",
|
||||
"args_pattern": {
|
||||
"command": "pytest|python.*test"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Run the tests",
|
||||
"Execute the test suite",
|
||||
"Please go ahead and run the tests",
|
||||
"Could you run the tests for me?",
|
||||
"I need the tests to be run",
|
||||
"Kick off the tests",
|
||||
"Let's run the tests",
|
||||
"Fire up the tests",
|
||||
"Go ahead and execute the tests",
|
||||
"I'd like you to run the tests"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "web-fetch-url",
|
||||
"description": "Use web_fetch when asked to retrieve content from a URL",
|
||||
"user_prompt": "Fetch the contents of https://example.com and summarize what's on the page",
|
||||
"expected_actions": [
|
||||
{ "tool": "web_fetch", "args_pattern": { "url": "example\\.com" } }
|
||||
{
|
||||
"tool": "web_fetch",
|
||||
"args_pattern": {
|
||||
"url": "example\\.com"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Fetch the contents of https://example.com and summarize what's on the page",
|
||||
"Go to https://example.com and give me a summary of what you find there",
|
||||
"Could you pull up https://example.com and tell me what the page is about?",
|
||||
"Retrieve the content from https://example.com, then provide a summary of it",
|
||||
"I need you to grab https://example.com and summarize its contents for me",
|
||||
"Please access https://example.com and give me an overview of the page",
|
||||
"What's on https://example.com? Fetch it and summarize for me.",
|
||||
"Download the page at https://example.com and provide a brief summary",
|
||||
"I'd like a summary of whatever is at https://example.com \u2014 please fetch it first",
|
||||
"Hit https://example.com and let me know what's there in summary form"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "man-page-lookup",
|
||||
"description": "Use man tool to look up command documentation",
|
||||
"user_prompt": "Look up the man page for tar and tell me what the --xattrs flag does",
|
||||
"expected_actions": [
|
||||
{ "tool": "man", "args_pattern": { "page": "tar" } }
|
||||
{
|
||||
"tool": "man",
|
||||
"args_pattern": {
|
||||
"page": "tar"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Look up the man page for tar and tell me what the --xattrs flag does",
|
||||
"What does the --xattrs flag do in tar? Check the man page for me.",
|
||||
"Could you pull up the man page for tar and explain the --xattrs option?",
|
||||
"I need to know what --xattrs does in tar \u2014 can you check the man page?",
|
||||
"Check tar's man page and let me know the purpose of the --xattrs flag.",
|
||||
"Please consult the tar man page and describe what the --xattrs flag is for.",
|
||||
"Hey, look at the tar man page real quick \u2014 what's --xattrs do?",
|
||||
"I'd like you to read the tar man page and summarize the --xattrs option for me.",
|
||||
"Would you mind checking the man page for tar to find out what --xattrs means?",
|
||||
"Look into the tar manual and explain the --xattrs flag to me."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "math-calculation",
|
||||
"description": "Use the math tool for precise calculations, not bash or mental math",
|
||||
"user_prompt": "What is 2^64 - 1? Use the math tool to calculate it precisely.",
|
||||
"expected_actions": [
|
||||
{ "tool": "math", "args_pattern": { "code": "2.*64" } }
|
||||
{
|
||||
"tool": "math",
|
||||
"args_pattern": {
|
||||
"code": "2.*64"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"What is 2^64 - 1? Use the math tool to calculate it precisely.",
|
||||
"Calculate 2^64 - 1 for me using the math tool, please.",
|
||||
"I need the exact value of 2^64 - 1. Please use the math tool.",
|
||||
"Could you use the math tool to compute 2^64 minus 1 precisely?",
|
||||
"Use the math tool to tell me what 2^64 - 1 equals.",
|
||||
"I'm curious: what's 2^64 - 1? Compute it with the math tool.",
|
||||
"Please precisely determine 2^64 - 1 via the math tool.",
|
||||
"Mind using the math tool to figure out 2^64 - 1 exactly?",
|
||||
"I'd like to know the precise result of 2^64 - 1 \u2014 use the math tool for this.",
|
||||
"Leverage the math tool to give me an exact answer for 2^64 - 1."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "web-search-query",
|
||||
"description": "Use web_search for general knowledge lookups, not web_fetch",
|
||||
"user_prompt": "Search the web for the current population of Tokyo",
|
||||
"expected_actions": [
|
||||
{ "tool": "web_search", "args_pattern": { "query": "Tokyo" } }
|
||||
{
|
||||
"tool": "web_search",
|
||||
"args_pattern": {
|
||||
"query": "Tokyo"
|
||||
}
|
||||
}
|
||||
],
|
||||
"match_mode": "subset"
|
||||
"match_mode": "subset",
|
||||
"user_prompts": [
|
||||
"Search the web for the current population of Tokyo",
|
||||
"What's Tokyo's current population? Look it up on the web.",
|
||||
"Could you do a web search to find out how many people currently live in Tokyo?",
|
||||
"Please search online for Tokyo's present-day population.",
|
||||
"I need you to look up the current population of Tokyo on the web.",
|
||||
"Find me Tokyo's current population via a web search.",
|
||||
"Web search: what is the current population of Tokyo?",
|
||||
"I'd like to know Tokyo's current population\u2014can you search the web for that?",
|
||||
"Look up how many people live in Tokyo right now using a web search.",
|
||||
"Do a web search for the population of Tokyo as of now."
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1237,6 +1237,52 @@ class TestJWTAudienceIssuer:
|
||||
result = validate_jwt(token, self.SECRET, audience="")
|
||||
assert result is not None
|
||||
|
||||
def test_create_jwt_expiry_seconds(self):
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.core.auth import create_jwt
|
||||
|
||||
token = create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=300)
|
||||
payload = pyjwt.decode(
|
||||
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
|
||||
)
|
||||
assert payload["exp"] - payload["iat"] == 300
|
||||
|
||||
def test_create_jwt_expiry_seconds_overrides_hours(self):
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.core.auth import create_jwt
|
||||
|
||||
token = create_jwt(
|
||||
"user1",
|
||||
frozenset({"read"}),
|
||||
"test",
|
||||
self.SECRET,
|
||||
expiry_hours=24,
|
||||
expiry_seconds=60,
|
||||
)
|
||||
payload = pyjwt.decode(
|
||||
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
|
||||
)
|
||||
# expiry_seconds takes precedence over expiry_hours
|
||||
assert payload["exp"] - payload["iat"] == 60
|
||||
|
||||
def test_create_jwt_expiry_seconds_rejects_zero(self):
|
||||
import pytest
|
||||
|
||||
from turnstone.core.auth import create_jwt
|
||||
|
||||
with pytest.raises(ValueError, match="expiry_seconds must be positive"):
|
||||
create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=0)
|
||||
|
||||
def test_create_jwt_expiry_seconds_rejects_negative(self):
|
||||
import pytest
|
||||
|
||||
from turnstone.core.auth import create_jwt
|
||||
|
||||
with pytest.raises(ValueError, match="expiry_seconds must be positive"):
|
||||
create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=-1)
|
||||
|
||||
|
||||
class TestServiceTokenManager:
|
||||
SECRET = "test-secret-that-is-at-least-32-chars"
|
||||
|
||||
@@ -1699,6 +1699,236 @@ class TestSSEProxy:
|
||||
asyncio.run(_run())
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Proxy auth header propagation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestProxyAuthHeaders:
|
||||
"""Verify _proxy_auth_headers mints user-scoped JWTs for proxy requests."""
|
||||
|
||||
SECRET = "test-secret-that-is-at-least-32-chars"
|
||||
|
||||
def _make_request(
|
||||
self, *, auth_result=None, jwt_secret="", proxy_token_mgr=None, proxy_auth_token=""
|
||||
):
|
||||
"""Build a minimal fake request for _proxy_auth_headers."""
|
||||
|
||||
class _State:
|
||||
pass
|
||||
|
||||
class _AppState:
|
||||
pass
|
||||
|
||||
class _App:
|
||||
state = _AppState()
|
||||
|
||||
class _Request:
|
||||
state = _State()
|
||||
app = _App()
|
||||
|
||||
req = _Request()
|
||||
req.state.auth_result = auth_result
|
||||
req.app.state.jwt_secret = jwt_secret
|
||||
req.app.state.proxy_token_mgr = proxy_token_mgr
|
||||
req.app.state.proxy_auth_token = proxy_auth_token
|
||||
return req
|
||||
|
||||
def test_mints_user_jwt(self):
|
||||
"""Real user auth_result → JWT with correct sub, scopes, src, aud, permissions."""
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="alice",
|
||||
scopes=frozenset({"read", "write"}),
|
||||
token_source="jwt",
|
||||
permissions=frozenset({"admin.users"}),
|
||||
)
|
||||
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
assert "Authorization" in headers
|
||||
token = headers["Authorization"].removeprefix("Bearer ")
|
||||
payload = pyjwt.decode(token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
|
||||
assert payload["sub"] == "alice"
|
||||
assert set(payload["scopes"].split(",")) == {"read", "write"}
|
||||
assert payload["src"] == "console-proxy"
|
||||
assert payload["aud"] == JWT_AUD_SERVER
|
||||
assert payload["permissions"] == "admin.users"
|
||||
|
||||
def test_narrows_scopes(self):
|
||||
"""Read-only user → JWT carries only read scope, not full {read,write,approve}."""
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="viewer",
|
||||
scopes=frozenset({"read"}),
|
||||
token_source="jwt",
|
||||
)
|
||||
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
token = headers["Authorization"].removeprefix("Bearer ")
|
||||
payload = pyjwt.decode(token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
|
||||
assert payload["scopes"] == "read"
|
||||
|
||||
def test_short_expiry(self):
|
||||
"""Minted JWT expires in 300 seconds, not hours."""
|
||||
import jwt as pyjwt
|
||||
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="alice",
|
||||
scopes=frozenset({"read"}),
|
||||
token_source="jwt",
|
||||
)
|
||||
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
token = headers["Authorization"].removeprefix("Bearer ")
|
||||
payload = pyjwt.decode(
|
||||
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
|
||||
)
|
||||
assert payload["exp"] - payload["iat"] == 300
|
||||
|
||||
def test_fallback_no_user(self):
|
||||
"""No auth_result → falls back to ServiceTokenManager."""
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import ServiceTokenManager
|
||||
|
||||
mgr = ServiceTokenManager(
|
||||
user_id="console-proxy",
|
||||
scopes=frozenset({"read", "write", "approve"}),
|
||||
source="console",
|
||||
secret=self.SECRET,
|
||||
)
|
||||
req = self._make_request(proxy_token_mgr=mgr)
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
assert "Authorization" in headers
|
||||
assert headers["Authorization"] == f"Bearer {mgr.token}"
|
||||
|
||||
def test_fallback_no_secret(self):
|
||||
"""auth_result present but empty jwt_secret → falls back to ServiceTokenManager."""
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
from turnstone.core.auth import AuthResult, ServiceTokenManager
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="alice",
|
||||
scopes=frozenset({"read"}),
|
||||
token_source="jwt",
|
||||
)
|
||||
mgr = ServiceTokenManager(
|
||||
user_id="console-proxy",
|
||||
scopes=frozenset({"read", "write", "approve"}),
|
||||
source="console",
|
||||
secret=self.SECRET,
|
||||
)
|
||||
req = self._make_request(auth_result=auth, jwt_secret="", proxy_token_mgr=mgr)
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
# Should use ServiceTokenManager, not mint a user JWT
|
||||
assert headers["Authorization"] == f"Bearer {mgr.token}"
|
||||
|
||||
def test_fallback_static_token(self):
|
||||
"""No auth_result, no ServiceTokenManager → uses static proxy_auth_token."""
|
||||
from turnstone.console.server import _proxy_auth_headers
|
||||
|
||||
req = self._make_request(proxy_auth_token="static-tok-123")
|
||||
headers = _proxy_auth_headers(req)
|
||||
|
||||
assert headers == {"Authorization": "Bearer static-tok-123"}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Server: trusted user_id forwarding on create_workstream
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCreateWorkstreamUserIdTrust:
|
||||
"""Verify that only trusted service tokens can forward user_id in create_workstream."""
|
||||
|
||||
def _extract_uid(self, body: dict, auth_result) -> str:
|
||||
"""Replicate the trust check from server.py:create_workstream."""
|
||||
auth = auth_result
|
||||
uid: str = getattr(auth, "user_id", "") or ""
|
||||
trusted_sources = {"bridge", "console"}
|
||||
if (
|
||||
body.get("user_id")
|
||||
and isinstance(body["user_id"], str)
|
||||
and auth is not None
|
||||
and auth.token_source in trusted_sources
|
||||
):
|
||||
uid = body["user_id"]
|
||||
return uid
|
||||
|
||||
def test_bridge_can_forward_user_id(self):
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="bridge",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="bridge",
|
||||
)
|
||||
uid = self._extract_uid({"user_id": "real-user-abc"}, auth)
|
||||
assert uid == "real-user-abc"
|
||||
|
||||
def test_console_service_can_forward_user_id(self):
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="console",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="console",
|
||||
)
|
||||
uid = self._extract_uid({"user_id": "real-user-abc"}, auth)
|
||||
assert uid == "real-user-abc"
|
||||
|
||||
def test_console_proxy_user_cannot_override_user_id(self):
|
||||
"""End-user tokens via console-proxy must NOT override user_id."""
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="real-user-abc",
|
||||
scopes=frozenset({"read", "write"}),
|
||||
token_source="console-proxy",
|
||||
)
|
||||
uid = self._extract_uid({"user_id": "impersonated-user"}, auth)
|
||||
# Should use JWT identity, NOT the body override
|
||||
assert uid == "real-user-abc"
|
||||
|
||||
def test_direct_user_cannot_override_user_id(self):
|
||||
"""Direct JWT login must NOT override user_id."""
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="real-user-abc",
|
||||
scopes=frozenset({"read", "write"}),
|
||||
token_source="password",
|
||||
)
|
||||
uid = self._extract_uid({"user_id": "impersonated-user"}, auth)
|
||||
assert uid == "real-user-abc"
|
||||
|
||||
def test_no_body_user_id_uses_jwt(self):
|
||||
from turnstone.core.auth import AuthResult
|
||||
|
||||
auth = AuthResult(
|
||||
user_id="bridge",
|
||||
scopes=frozenset({"approve"}),
|
||||
token_source="bridge",
|
||||
)
|
||||
uid = self._extract_uid({"name": "test-ws"}, auth)
|
||||
assert uid == "bridge"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Collector — MCP aggregation in get_overview()
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -573,7 +573,9 @@ def _fake_request(*nodes: dict[str, Any], proxy_client: Any = None) -> MagicMock
|
||||
collector.get_nodes.return_value = (list(nodes), len(nodes))
|
||||
collector.get_all_nodes.side_effect = lambda: collector.get_nodes.return_value[0]
|
||||
req = MagicMock()
|
||||
req.state.auth_result = None
|
||||
req.app.state.collector = collector
|
||||
req.app.state.jwt_secret = ""
|
||||
req.app.state.proxy_client = proxy_client or AsyncMock()
|
||||
req.app.state.proxy_token_mgr = None
|
||||
req.app.state.proxy_auth_token = "tok"
|
||||
|
||||
@@ -214,7 +214,7 @@ class TestPlanExec:
|
||||
"id": tc_id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "create_plan",
|
||||
"name": "plan_agent",
|
||||
"arguments": json.dumps({"goal": prior_prompt}),
|
||||
},
|
||||
}
|
||||
@@ -250,7 +250,7 @@ class TestPlanExec:
|
||||
m for m in messages if m["role"] == "assistant" and m.get("tool_calls")
|
||||
]
|
||||
assert len(assistant_with_tc) == 1
|
||||
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "create_plan"
|
||||
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "plan_agent"
|
||||
|
||||
# The real tool result is forwarded with its original content
|
||||
tool_msgs = [m for m in messages if m["role"] == "tool"]
|
||||
@@ -463,7 +463,7 @@ class TestPlanRefinement:
|
||||
with patch.object(session, "_refine_plan", side_effect=fake_refine):
|
||||
items = [
|
||||
{
|
||||
"func_name": "create_plan",
|
||||
"func_name": "plan_agent",
|
||||
"call_id": "c1",
|
||||
"prompt": "add auth",
|
||||
}
|
||||
@@ -576,7 +576,7 @@ class TestPlanRefinement:
|
||||
msgs = captured["messages"]
|
||||
assert msgs[0]["role"] == "system"
|
||||
assert msgs[1]["role"] == "assistant"
|
||||
assert msgs[1]["tool_calls"][0]["function"]["name"] == "create_plan"
|
||||
assert msgs[1]["tool_calls"][0]["function"]["name"] == "plan_agent"
|
||||
assert msgs[2]["role"] == "tool"
|
||||
assert msgs[2]["content"] == self.GOOD_PLAN
|
||||
assert msgs[3]["role"] == "user"
|
||||
|
||||
@@ -104,8 +104,8 @@ class TestToolsMetadata:
|
||||
"man": "page",
|
||||
"web_fetch": "url",
|
||||
"web_search": "query",
|
||||
"task": "prompt",
|
||||
"create_plan": "goal",
|
||||
"task_agent": "prompt",
|
||||
"plan_agent": "goal",
|
||||
"memory": "name",
|
||||
"recall": "query",
|
||||
"notify": "message",
|
||||
|
||||
@@ -84,7 +84,7 @@ class TestTavilyClient:
|
||||
|
||||
class TestDuckDuckGoClient:
|
||||
def test_integration_via_mock_ddgs(self):
|
||||
"""Patch the duckduckgo_search import inside DuckDuckGoClient.search."""
|
||||
"""Patch the ddgs import inside DuckDuckGoClient.search."""
|
||||
mock_ddgs = MagicMock()
|
||||
mock_ddgs.__enter__ = MagicMock(return_value=mock_ddgs)
|
||||
mock_ddgs.__exit__ = MagicMock(return_value=False)
|
||||
@@ -93,7 +93,7 @@ class TestDuckDuckGoClient:
|
||||
]
|
||||
mock_module = MagicMock()
|
||||
mock_module.DDGS.return_value = mock_ddgs
|
||||
with patch.dict("sys.modules", {"duckduckgo_search": mock_module}):
|
||||
with patch.dict("sys.modules", {"ddgs": mock_module}):
|
||||
client = DuckDuckGoClient(timeout=10)
|
||||
result = client.search("test query", max_results=3)
|
||||
mock_ddgs.text.assert_called_once_with("test query", max_results=3)
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
|
||||
|
||||
__version__ = "0.8.5"
|
||||
__version__ = "0.8.8"
|
||||
|
||||
@@ -37,7 +37,7 @@ from starlette.staticfiles import StaticFiles
|
||||
from turnstone.api.console_spec import build_console_spec
|
||||
from turnstone.api.docs import make_docs_handler, make_openapi_handler
|
||||
from turnstone.console.collector import ClusterCollector
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, AuthMiddleware
|
||||
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, AuthMiddleware, create_jwt
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncGenerator
|
||||
@@ -133,15 +133,36 @@ _CONSOLE_PROXY_STYLE = "<style>.dashboard-overlay{top:32px!important}</style>"
|
||||
|
||||
_VALID_NODE_ID = re.compile(r"^[a-zA-Z0-9._-]+$")
|
||||
|
||||
_PROXY_JWT_EXPIRY_SECONDS = 300 # 5 min — ample for any request round-trip
|
||||
|
||||
|
||||
def _proxy_auth_headers(request: Request) -> dict[str, str]:
|
||||
"""Build auth headers for proxied requests to upstream servers.
|
||||
|
||||
Uses the service proxy token (``JWT_AUD_SERVER``) so the upstream node
|
||||
accepts the request. The user's console-audience JWT is *not* forwarded
|
||||
— it would be rejected by the server's audience validation.
|
||||
Mints a short-lived JWT carrying the real user's identity and scopes
|
||||
so the upstream server records correct audit attribution and enforces
|
||||
scope narrowing. Falls back to the ServiceTokenManager when no user
|
||||
context is available.
|
||||
"""
|
||||
# Prefer the auto-rotating ServiceTokenManager when available
|
||||
auth_result = getattr(getattr(request, "state", None), "auth_result", None)
|
||||
jwt_secret: str = getattr(request.app.state, "jwt_secret", "")
|
||||
|
||||
if auth_result is not None and auth_result.user_id and jwt_secret:
|
||||
token = create_jwt(
|
||||
user_id=auth_result.user_id,
|
||||
scopes=auth_result.scopes,
|
||||
source="console-proxy",
|
||||
secret=jwt_secret,
|
||||
audience=JWT_AUD_SERVER,
|
||||
permissions=auth_result.permissions,
|
||||
expiry_seconds=_PROXY_JWT_EXPIRY_SECONDS,
|
||||
)
|
||||
return {"Authorization": f"Bearer {token}"}
|
||||
|
||||
# Fallback: service identity (no user context).
|
||||
# When auth is disabled on the console, auth_result is None, so all proxied
|
||||
# requests use the full-privilege service identity. This is safe only when
|
||||
# the upstream server also has auth disabled.
|
||||
mgr = getattr(request.app.state, "proxy_token_mgr", None)
|
||||
if mgr is not None:
|
||||
return dict(mgr.bearer_header)
|
||||
@@ -407,6 +428,9 @@ async def create_workstream(request: Request) -> JSONResponse:
|
||||
|
||||
from turnstone.mq.protocol import CreateWorkstreamMessage
|
||||
|
||||
auth = getattr(getattr(request, "state", None), "auth_result", None)
|
||||
uid: str = getattr(auth, "user_id", "") or ""
|
||||
|
||||
# General pool — push to shared queue, any bridge picks it up
|
||||
if node_id == "pool":
|
||||
msg = CreateWorkstreamMessage(
|
||||
@@ -415,6 +439,7 @@ async def create_workstream(request: Request) -> JSONResponse:
|
||||
initial_message=initial_message,
|
||||
skill=skill,
|
||||
resume_ws=resume_ws,
|
||||
user_id=uid,
|
||||
)
|
||||
broker.push_inbound(msg.to_json())
|
||||
log.debug("Pool dispatch: correlation_id=%s name=%r", msg.correlation_id, name)
|
||||
@@ -444,6 +469,7 @@ async def create_workstream(request: Request) -> JSONResponse:
|
||||
initial_message=initial_message,
|
||||
skill=skill,
|
||||
resume_ws=resume_ws,
|
||||
user_id=uid,
|
||||
)
|
||||
broker.push_inbound(msg.to_json(), node_id=node_id)
|
||||
|
||||
@@ -2748,6 +2774,16 @@ async def admin_usage(request: Request) -> JSONResponse:
|
||||
group_by=group_by,
|
||||
)
|
||||
|
||||
# Resolve user_id hex → username for display when grouped by user
|
||||
if group_by == "user" and breakdown:
|
||||
uid_to_name: dict[str, str] = {}
|
||||
for u in storage.list_users():
|
||||
uid_to_name[u["user_id"]] = u.get("username") or u["user_id"]
|
||||
for row in breakdown:
|
||||
raw_key = row.get("key", "")
|
||||
if raw_key and raw_key in uid_to_name:
|
||||
row["key"] = uid_to_name[raw_key]
|
||||
|
||||
return JSONResponse({"summary": summary, "breakdown": breakdown})
|
||||
|
||||
|
||||
@@ -2792,6 +2828,16 @@ async def admin_audit(request: Request) -> JSONResponse:
|
||||
until=until,
|
||||
)
|
||||
|
||||
# Resolve user_id hex → username for display
|
||||
if events:
|
||||
uid_to_name: dict[str, str] = {}
|
||||
for u in storage.list_users():
|
||||
uid_to_name[u["user_id"]] = u.get("username") or u["user_id"]
|
||||
for ev in events:
|
||||
raw_uid = ev.get("user_id", "")
|
||||
if raw_uid and raw_uid in uid_to_name:
|
||||
ev["username"] = uid_to_name[raw_uid]
|
||||
|
||||
return JSONResponse({"events": events, "total": total})
|
||||
|
||||
|
||||
|
||||
@@ -1838,7 +1838,9 @@ function _renderGovAudit(events, total) {
|
||||
_relativeTime(ev.timestamp) +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-auser">' +
|
||||
escapeHtml(ev.user_id ? ev.user_id.slice(0, 8) : "\u2014") +
|
||||
escapeHtml(
|
||||
ev.username || (ev.user_id ? ev.user_id.slice(0, 8) : "\u2014"),
|
||||
) +
|
||||
"</span>" +
|
||||
'<span class="admin-col admin-col-aaction"><span class="' +
|
||||
actionCls +
|
||||
|
||||
@@ -329,18 +329,22 @@ def create_jwt(
|
||||
expiry_hours: int = 24,
|
||||
audience: str = "",
|
||||
permissions: frozenset[str] = frozenset(),
|
||||
expiry_seconds: int | None = None,
|
||||
) -> str:
|
||||
"""Create a signed JWT with user identity, scopes, and permissions."""
|
||||
import jwt
|
||||
|
||||
if expiry_seconds is not None and expiry_seconds <= 0:
|
||||
raise ValueError("expiry_seconds must be positive")
|
||||
now = int(time.time())
|
||||
ttl = expiry_seconds if expiry_seconds is not None else expiry_hours * 3600
|
||||
payload: dict[str, Any] = {
|
||||
"sub": user_id,
|
||||
"scopes": ",".join(sorted(scopes)),
|
||||
"src": source,
|
||||
"iss": JWT_ISSUER,
|
||||
"iat": now,
|
||||
"exp": now + expiry_hours * 3600,
|
||||
"exp": now + ttl,
|
||||
}
|
||||
if audience:
|
||||
payload["aud"] = audience
|
||||
|
||||
+44
-24
@@ -867,17 +867,34 @@ class ChatSession:
|
||||
]
|
||||
else:
|
||||
dev_parts = [
|
||||
"You are an expert software engineer. You solve problems "
|
||||
"by reading code, making targeted edits, and running commands. "
|
||||
"Always respond with tool calls, not just text.\n\n"
|
||||
"TOOL PATTERNS:\n\n"
|
||||
"Modify existing file → read_file then edit_file:\n"
|
||||
" read_file(path='config.py') → "
|
||||
"edit_file(path='config.py')\n\n"
|
||||
"Create new file → write_file:\n"
|
||||
" write_file(path='hello.py', content='...')\n\n"
|
||||
"Modify multiple files → read_file then edit_file each:\n"
|
||||
" read_file(path='a.py') → edit_file(path='a.py') → "
|
||||
"read_file(path='b.py') → edit_file(path='b.py')\n\n"
|
||||
"Create new file → write_file (generate reasonable "
|
||||
"content even if the request is vague):\n"
|
||||
" write_file(path='hello.py', content='...')\n"
|
||||
" write_file(path='README.md', "
|
||||
"content='# Project\\nDescription.')\n\n"
|
||||
"Create a file then run it → write_file then bash:\n"
|
||||
" write_file(path='fib.py', content='...') → "
|
||||
"bash(command='python fib.py')\n\n"
|
||||
"Find something across files → search:\n"
|
||||
" search(query='test_')\n\n"
|
||||
"Plan, design, or think through an approach → create_plan:\n"
|
||||
" create_plan(goal='refactor database from API')\n\n"
|
||||
"Find and modify → search then read_file then edit_file:\n"
|
||||
" search(query='MAX_RETRIES') → "
|
||||
"read_file(path='found.py') → "
|
||||
"edit_file(path='found.py')\n\n"
|
||||
"Plan, think through, or strategize → plan_agent:\n"
|
||||
" plan_agent(goal='refactor database layer "
|
||||
"from monolith to service')\n"
|
||||
" plan_agent(goal='restructure auth module')\n\n"
|
||||
"Run a command, git, or tests → bash:\n"
|
||||
" bash(command='git log -5')\n"
|
||||
" bash(command='pytest')\n\n"
|
||||
@@ -885,8 +902,9 @@ class ChatSession:
|
||||
" web_fetch(url='https://example.com')\n\n"
|
||||
"Search the web for information → web_search:\n"
|
||||
" web_search(query='current population of Tokyo')\n\n"
|
||||
"Look up documentation → man:\n"
|
||||
" man(page='tar')",
|
||||
"Look up command flags or documentation → man:\n"
|
||||
" man(page='tar')\n"
|
||||
" man(page='grep')",
|
||||
]
|
||||
# Tool search hint (client-side mode only — native mode needs no hint)
|
||||
if self._tool_search:
|
||||
@@ -2090,8 +2108,10 @@ class ChatSession:
|
||||
}
|
||||
elif name == "notify":
|
||||
it["func_args"] = {"message": it.get("message", "")[:200]}
|
||||
elif name == "task":
|
||||
elif name == "task_agent":
|
||||
it["func_args"] = {"prompt": it.get("prompt", "")[:200]}
|
||||
elif name == "plan_agent":
|
||||
it["func_args"] = {"goal": it.get("prompt", "")[:200]}
|
||||
elif it.get("mcp_args"):
|
||||
it["func_args"] = it["mcp_args"]
|
||||
|
||||
@@ -2219,7 +2239,7 @@ class ChatSession:
|
||||
# feedback the plan agent re-runs and the revised plan is shown
|
||||
# again, up to _MAX_PLAN_REFINEMENTS rounds.
|
||||
for i, item in enumerate(items):
|
||||
if item.get("func_name") != "create_plan" or item.get("error") or item.get("denied"):
|
||||
if item.get("func_name") != "plan_agent" or item.get("error") or item.get("denied"):
|
||||
continue
|
||||
|
||||
cid, output = results[i]
|
||||
@@ -2344,8 +2364,8 @@ class ChatSession:
|
||||
"web_fetch": self._prepare_web_fetch,
|
||||
"web_search": self._prepare_web_search,
|
||||
"tool_search": self._prepare_tool_search,
|
||||
"task": self._prepare_task,
|
||||
"create_plan": self._prepare_plan,
|
||||
"task_agent": self._prepare_task,
|
||||
"plan_agent": self._prepare_plan,
|
||||
"memory": self._prepare_memory,
|
||||
"recall": self._prepare_recall,
|
||||
"notify": self._prepare_notify,
|
||||
@@ -2803,7 +2823,7 @@ class ChatSession:
|
||||
"needs_approval": False,
|
||||
"error": (
|
||||
"Error: No web search backend available. "
|
||||
"Install duckduckgo-search (`pip install duckduckgo-search`), "
|
||||
"Install the ddg extra (`pip install turnstone[ddg]`), "
|
||||
"configure a Tavily API key, or set tools.web_search_backend."
|
||||
),
|
||||
}
|
||||
@@ -2877,8 +2897,8 @@ class ChatSession:
|
||||
if not prompt:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task",
|
||||
"header": "\u2717 task: empty prompt",
|
||||
"func_name": "task_agent",
|
||||
"header": "\u2717 task_agent: empty prompt",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": "Error: empty prompt",
|
||||
@@ -2886,11 +2906,11 @@ class ChatSession:
|
||||
preview_text = prompt[:300] + ("..." if len(prompt) > 300 else "")
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "task",
|
||||
"header": "\u2699 task (autonomous agent)",
|
||||
"func_name": "task_agent",
|
||||
"header": "\u2699 task_agent (autonomous agent)",
|
||||
"preview": f" {DIM}{preview_text}{RESET}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "task",
|
||||
"approval_label": "task_agent",
|
||||
"execute": self._exec_task,
|
||||
"prompt": prompt,
|
||||
}
|
||||
@@ -2901,8 +2921,8 @@ class ChatSession:
|
||||
if not goal:
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "create_plan",
|
||||
"header": "\u2717 create_plan: empty goal",
|
||||
"func_name": "plan_agent",
|
||||
"header": "\u2717 plan_agent: empty goal",
|
||||
"preview": "",
|
||||
"needs_approval": False,
|
||||
"error": "Error: empty goal",
|
||||
@@ -2910,11 +2930,11 @@ class ChatSession:
|
||||
preview_text = goal[:300] + ("..." if len(goal) > 300 else "")
|
||||
return {
|
||||
"call_id": call_id,
|
||||
"func_name": "create_plan",
|
||||
"header": "\u2699 create_plan (planning agent)",
|
||||
"func_name": "plan_agent",
|
||||
"header": "\u2699 plan_agent (planning agent)",
|
||||
"preview": f" {DIM}{preview_text}{RESET}",
|
||||
"needs_approval": True,
|
||||
"approval_label": "create_plan",
|
||||
"approval_label": "plan_agent",
|
||||
"execute": self._exec_plan,
|
||||
"prompt": goal,
|
||||
}
|
||||
@@ -3920,7 +3940,7 @@ class ChatSession:
|
||||
tool_name = tc_dict["function"]["name"]
|
||||
|
||||
# Guard 1: block recursive agent calls.
|
||||
if tool_name in ("task", "create_plan"):
|
||||
if tool_name in ("task_agent", "plan_agent"):
|
||||
output = "Error: agents cannot spawn further agents"
|
||||
# Guard 2: tool not in this agent's API tool list.
|
||||
elif tool_name not in tool_names:
|
||||
@@ -4123,7 +4143,7 @@ class ChatSession:
|
||||
for i, msg in enumerate(self.messages):
|
||||
if msg.get("role") == "assistant" and msg.get("tool_calls"):
|
||||
for tc in msg["tool_calls"]:
|
||||
if tc.get("function", {}).get("name") == "create_plan":
|
||||
if tc.get("function", {}).get("name") == "plan_agent":
|
||||
tc_id = tc["id"]
|
||||
for j in range(i + 1, len(self.messages)):
|
||||
if (
|
||||
@@ -4217,7 +4237,7 @@ class ChatSession:
|
||||
"id": tc_id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "create_plan",
|
||||
"name": "plan_agent",
|
||||
"arguments": json.dumps({"goal": original_goal}),
|
||||
},
|
||||
}
|
||||
|
||||
@@ -19,7 +19,11 @@ def run_migrations_online() -> None:
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
render_as_batch=True,
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
@@ -28,9 +28,7 @@ def upgrade() -> None:
|
||||
sa.Column("updated", sa.Text, nullable=False),
|
||||
sa.Column("last_accessed", sa.Text, nullable=False, server_default=""),
|
||||
sa.Column("access_count", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
op.create_unique_constraint(
|
||||
"uq_smem_name_scope", "structured_memories", ["name", "scope", "scope_id"]
|
||||
sa.UniqueConstraint("name", "scope", "scope_id", name="uq_smem_name_scope"),
|
||||
)
|
||||
op.create_index("idx_smem_type", "structured_memories", ["type"])
|
||||
op.create_index("idx_smem_scope", "structured_memories", ["scope", "scope_id"])
|
||||
|
||||
@@ -27,56 +27,32 @@ depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
# Phase 1: Skills evolution columns
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("description", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("tags", sa.Text, nullable=False, server_default="[]"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("source_url", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("version", sa.Text, nullable=False, server_default="1.0.0"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("author", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("activation", sa.Text, nullable=False, server_default="named"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("token_estimate", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
# Phase 2: Security scanning + install provenance
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("allowed_tools", sa.Text, nullable=False, server_default="[]"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("scan_status", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("scan_report", sa.Text, nullable=False, server_default="{}"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("installed_at", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("installed_by", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
# Phase 1+2: Skills evolution columns + security scanning (batch for SQLite)
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.add_column(sa.Column("description", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(sa.Column("tags", sa.Text, nullable=False, server_default="[]"))
|
||||
batch_op.add_column(sa.Column("source_url", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(sa.Column("version", sa.Text, nullable=False, server_default="1.0.0"))
|
||||
batch_op.add_column(sa.Column("author", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(
|
||||
sa.Column("activation", sa.Text, nullable=False, server_default="named"),
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("token_estimate", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("allowed_tools", sa.Text, nullable=False, server_default="[]"),
|
||||
)
|
||||
batch_op.add_column(sa.Column("scan_status", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(
|
||||
sa.Column("scan_report", sa.Text, nullable=False, server_default="{}"),
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("installed_at", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("installed_by", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
|
||||
# Backfill activation from is_default
|
||||
op.execute("UPDATE prompt_templates SET activation = 'default' WHERE is_default = 1")
|
||||
@@ -101,42 +77,26 @@ def upgrade() -> None:
|
||||
)
|
||||
|
||||
# Phase 3: Session config columns (from workstream templates)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("model", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("temperature", sa.Float, nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("reasoning_effort", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("max_tokens", sa.Integer, nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("token_budget", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("agent_max_turns", sa.Integer, nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("notify_on_complete", sa.Text, nullable=False, server_default="{}"),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
|
||||
)
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.add_column(sa.Column("model", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(
|
||||
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
batch_op.add_column(sa.Column("temperature", sa.Float, nullable=True))
|
||||
batch_op.add_column(
|
||||
sa.Column("reasoning_effort", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
batch_op.add_column(sa.Column("max_tokens", sa.Integer, nullable=True))
|
||||
batch_op.add_column(
|
||||
sa.Column("token_budget", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
batch_op.add_column(sa.Column("agent_max_turns", sa.Integer, nullable=True))
|
||||
batch_op.add_column(
|
||||
sa.Column("notify_on_complete", sa.Text, nullable=False, server_default="{}"),
|
||||
)
|
||||
batch_op.add_column(
|
||||
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
|
||||
)
|
||||
|
||||
# Skill versions — version history for skills
|
||||
op.create_table(
|
||||
@@ -253,24 +213,25 @@ def upgrade() -> None:
|
||||
|
||||
# Rename workstreams table columns: ws_template_id → skill_id,
|
||||
# ws_template_version → skill_version
|
||||
op.alter_column("workstreams", "ws_template_id", new_column_name="skill_id")
|
||||
op.alter_column("workstreams", "ws_template_version", new_column_name="skill_version")
|
||||
with op.batch_alter_table("workstreams") as batch_op:
|
||||
batch_op.alter_column("ws_template_id", new_column_name="skill_id")
|
||||
batch_op.alter_column("ws_template_version", new_column_name="skill_version")
|
||||
|
||||
# Rename scheduled_tasks.template → skill
|
||||
op.alter_column("scheduled_tasks", "template", new_column_name="skill")
|
||||
# Rename scheduled_tasks.template → skill, drop ws_template
|
||||
with op.batch_alter_table("scheduled_tasks") as batch_op:
|
||||
batch_op.alter_column("template", new_column_name="skill")
|
||||
batch_op.drop_column("ws_template")
|
||||
|
||||
# Drop old tables
|
||||
op.drop_table("workstream_template_versions")
|
||||
op.drop_table("workstream_templates")
|
||||
|
||||
# Drop ws_template column from scheduled_tasks
|
||||
op.drop_column("scheduled_tasks", "ws_template")
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
# Reverse workstreams column renames
|
||||
op.alter_column("workstreams", "skill_id", new_column_name="ws_template_id")
|
||||
op.alter_column("workstreams", "skill_version", new_column_name="ws_template_version")
|
||||
with op.batch_alter_table("workstreams") as batch_op:
|
||||
batch_op.alter_column("skill_id", new_column_name="ws_template_id")
|
||||
batch_op.alter_column("skill_version", new_column_name="ws_template_version")
|
||||
|
||||
# Reverse workstream_config key renames
|
||||
op.execute("UPDATE workstream_config SET key = 'ws_template_id' WHERE key = 'applied_skill_id'")
|
||||
@@ -283,14 +244,10 @@ def downgrade() -> None:
|
||||
"WHERE key = 'applied_skill_content'"
|
||||
)
|
||||
|
||||
# Reverse scheduled_tasks column rename: skill → template
|
||||
op.alter_column("scheduled_tasks", "skill", new_column_name="template")
|
||||
|
||||
# Re-add ws_template column to scheduled_tasks
|
||||
op.add_column(
|
||||
"scheduled_tasks",
|
||||
sa.Column("ws_template", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
# Reverse scheduled_tasks column rename + re-add ws_template
|
||||
with op.batch_alter_table("scheduled_tasks") as batch_op:
|
||||
batch_op.alter_column("skill", new_column_name="template")
|
||||
batch_op.add_column(sa.Column("ws_template", sa.Text, nullable=False, server_default=""))
|
||||
|
||||
# Recreate workstream_templates (empty — destructive migration)
|
||||
op.create_table(
|
||||
@@ -333,32 +290,31 @@ def downgrade() -> None:
|
||||
op.drop_index("idx_skill_versions_skill_id", table_name="skill_versions")
|
||||
op.drop_table("skill_versions")
|
||||
|
||||
# Drop session config columns from prompt_templates
|
||||
op.drop_column("prompt_templates", "enabled")
|
||||
op.drop_column("prompt_templates", "notify_on_complete")
|
||||
op.drop_column("prompt_templates", "agent_max_turns")
|
||||
op.drop_column("prompt_templates", "token_budget")
|
||||
op.drop_column("prompt_templates", "max_tokens")
|
||||
op.drop_column("prompt_templates", "reasoning_effort")
|
||||
op.drop_column("prompt_templates", "temperature")
|
||||
op.drop_column("prompt_templates", "auto_approve")
|
||||
op.drop_column("prompt_templates", "model")
|
||||
# Drop session config + skills evolution columns from prompt_templates
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.drop_column("enabled")
|
||||
batch_op.drop_column("notify_on_complete")
|
||||
batch_op.drop_column("agent_max_turns")
|
||||
batch_op.drop_column("token_budget")
|
||||
batch_op.drop_column("max_tokens")
|
||||
batch_op.drop_column("reasoning_effort")
|
||||
batch_op.drop_column("temperature")
|
||||
batch_op.drop_column("auto_approve")
|
||||
batch_op.drop_column("model")
|
||||
batch_op.drop_column("installed_by")
|
||||
batch_op.drop_column("installed_at")
|
||||
batch_op.drop_column("scan_report")
|
||||
batch_op.drop_column("scan_status")
|
||||
batch_op.drop_column("allowed_tools")
|
||||
batch_op.drop_column("token_estimate")
|
||||
batch_op.drop_column("activation")
|
||||
batch_op.drop_column("author")
|
||||
batch_op.drop_column("version")
|
||||
batch_op.drop_column("source_url")
|
||||
batch_op.drop_column("tags")
|
||||
batch_op.drop_column("description")
|
||||
|
||||
# Drop skill resources
|
||||
op.drop_index("idx_skill_resources_skill_path", table_name="skill_resources")
|
||||
op.drop_index("idx_skill_resources_skill_id", table_name="skill_resources")
|
||||
op.drop_table("skill_resources")
|
||||
|
||||
# Drop skills evolution columns
|
||||
op.drop_column("prompt_templates", "installed_by")
|
||||
op.drop_column("prompt_templates", "installed_at")
|
||||
op.drop_column("prompt_templates", "scan_report")
|
||||
op.drop_column("prompt_templates", "scan_status")
|
||||
op.drop_column("prompt_templates", "allowed_tools")
|
||||
op.drop_column("prompt_templates", "token_estimate")
|
||||
op.drop_column("prompt_templates", "activation")
|
||||
op.drop_column("prompt_templates", "author")
|
||||
op.drop_column("prompt_templates", "version")
|
||||
op.drop_column("prompt_templates", "source_url")
|
||||
op.drop_column("prompt_templates", "tags")
|
||||
op.drop_column("prompt_templates", "description")
|
||||
|
||||
@@ -33,14 +33,13 @@ def upgrade() -> None:
|
||||
op.create_index("ix_oa_created", "output_assessments", ["created"])
|
||||
op.create_index("ix_oa_risk", "output_assessments", ["risk_level"])
|
||||
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("scan_version", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.add_column(sa.Column("scan_version", sa.Text, nullable=False, server_default=""))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("prompt_templates", "scan_version")
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.drop_column("scan_version")
|
||||
op.drop_index("ix_oa_risk", table_name="output_assessments")
|
||||
op.drop_index("ix_oa_created", table_name="output_assessments")
|
||||
op.drop_index("ix_oa_ws_id", table_name="output_assessments")
|
||||
|
||||
@@ -19,16 +19,14 @@ depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("license", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("compatibility", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.add_column(sa.Column("license", sa.Text, nullable=False, server_default=""))
|
||||
batch_op.add_column(
|
||||
sa.Column("compatibility", sa.Text, nullable=False, server_default=""),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("prompt_templates", "compatibility")
|
||||
op.drop_column("prompt_templates", "license")
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.drop_column("compatibility")
|
||||
batch_op.drop_column("license")
|
||||
|
||||
@@ -19,11 +19,10 @@ depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"prompt_templates",
|
||||
sa.Column("priority", sa.Integer, nullable=False, server_default="0"),
|
||||
)
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.add_column(sa.Column("priority", sa.Integer, nullable=False, server_default="0"))
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_column("prompt_templates", "priority")
|
||||
with op.batch_alter_table("prompt_templates") as batch_op:
|
||||
batch_op.drop_column("priority")
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
``web_search`` is an abstract capability with swappable clients:
|
||||
|
||||
* **TavilyClient** — paid, high quality, requires API key
|
||||
* **DuckDuckGoClient** — free, no API key, uses ``duckduckgo-search``
|
||||
* **DuckDuckGoClient** — free, no API key, uses ``ddgs``
|
||||
|
||||
Auto-detection (default): Tavily if key present, else DDG if installed,
|
||||
else ``None`` (tool removed from tool list).
|
||||
@@ -63,7 +63,7 @@ class DuckDuckGoClient:
|
||||
self._timeout = timeout
|
||||
|
||||
def search(self, query: str, max_results: int = 5, **kwargs: Any) -> str:
|
||||
from duckduckgo_search import DDGS # type: ignore[import-not-found]
|
||||
from ddgs import DDGS
|
||||
|
||||
with DDGS(timeout=int(self._timeout)) as ddgs:
|
||||
raw = list(ddgs.text(query, max_results=max_results))
|
||||
@@ -130,9 +130,9 @@ def _format_ddg(results: list[dict[str, Any]], query: str) -> str:
|
||||
|
||||
|
||||
def _ddg_available() -> bool:
|
||||
"""Check if duckduckgo-search is installed."""
|
||||
"""Check if the ddgs package is installed."""
|
||||
try:
|
||||
import duckduckgo_search # noqa: F401
|
||||
import ddgs # noqa: F401
|
||||
|
||||
return True
|
||||
except ImportError:
|
||||
|
||||
+1530
-193
File diff suppressed because it is too large
Load Diff
@@ -423,6 +423,7 @@ class Bridge:
|
||||
model=model,
|
||||
resume_ws=resume_ws,
|
||||
skill=skill,
|
||||
user_id=user_id,
|
||||
)
|
||||
# Send initial_message only when no workstream was actually resumed.
|
||||
# Use the server's `resumed` response (not just the intent) so that
|
||||
@@ -491,6 +492,7 @@ class Bridge:
|
||||
model: str = "",
|
||||
resume_ws: str = "",
|
||||
skill: str = "",
|
||||
user_id: str = "",
|
||||
) -> tuple[str, bool]:
|
||||
"""Create a workstream on the server. Returns (ws_id, resumed)."""
|
||||
try:
|
||||
@@ -501,6 +503,8 @@ class Bridge:
|
||||
payload["resume_ws"] = resume_ws
|
||||
if skill:
|
||||
payload["skill"] = skill
|
||||
if user_id:
|
||||
payload["user_id"] = user_id
|
||||
resp = self._http.post(
|
||||
"/v1/api/workstreams/new",
|
||||
json=payload,
|
||||
|
||||
@@ -1280,6 +1280,18 @@ async def create_workstream(request: Request) -> JSONResponse:
|
||||
skip: bool = request.app.state.skip_permissions
|
||||
auth = getattr(getattr(request, "state", None), "auth_result", None)
|
||||
uid: str = getattr(auth, "user_id", "") or ""
|
||||
# Trusted services (bridge, console) may forward the real user_id in the
|
||||
# request body when creating workstreams on behalf of a user. Only service
|
||||
# identities are trusted — end-user tokens (including console-proxy tokens
|
||||
# that carry the real user's identity) must not override user_id.
|
||||
trusted_sources = {"bridge", "console"}
|
||||
if (
|
||||
body.get("user_id")
|
||||
and isinstance(body["user_id"], str)
|
||||
and auth is not None
|
||||
and auth.token_source in trusted_sources
|
||||
):
|
||||
uid = body["user_id"]
|
||||
body_skill = body.get("skill", "")
|
||||
resume_ws_id = body.get("resume_ws", "")
|
||||
# Resolve skill — applies content + session config (model, temperature, etc.)
|
||||
|
||||
File diff suppressed because one or more lines are too long
+1
-1
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "bash",
|
||||
"description": "Execute a bash command and return stdout + stderr. Use this tool freely for any task: checking time, reading files, running programs, system info, etc.",
|
||||
"description": "Execute a bash command and return stdout + stderr. Use for running programs, git, tests, system commands, installing packages, etc. Environment questions ('What Python version?', 'Is X installed?') are tool-use tasks — e.g. bash(command='python --version'). For file creation use write_file instead; for man pages use man instead.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
{
|
||||
"name": "create_plan",
|
||||
"description": "Create a structured plan before taking action. An autonomous agent explores the available context, identifies what needs to change, and writes a step-by-step plan. Call this tool when the user asks to plan, design, or think through an approach, or when a task is complex, touches multiple areas, or has unclear scope.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"goal": {
|
||||
"type": "string",
|
||||
"description": "The goal and scope of the plan, including any constraints."
|
||||
}
|
||||
},
|
||||
"required": ["goal"]
|
||||
},
|
||||
"primary_key": "goal"
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "edit_file",
|
||||
"description": "Replace an exact string in a file with new content. Fails if old_string is not found or matches multiple locations (use near_line to disambiguate). Requires read_file on the same path first.",
|
||||
"description": "Replace an exact string in a file with new content. Fails if old_string is not found or matches multiple locations (use near_line to disambiguate). Requires read_file on the same path first — it will fail without this. Use for any modification to existing files: changing values, renaming, inserting code, adding docstrings. Prefer this over write_file for partial modifications. For multi-file edits when filenames are known, go directly to read_file + edit_file for each file — no need to search first. For generated content (docstrings, type hints), call edit_file with your best-effort content inline — e.g. edit_file(old_string='def process(...):', new_string='def process(...):\\n \"\"\"Process input data.\"\"\"'). When asked to change something in a file, follow through with edit_file after reading — reading alone is not enough.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "man",
|
||||
"description": "Read a man page. Use this instead of bash('man ...') or web_search. Returns the full formatted manual entry.",
|
||||
"description": "Read a man page. Questions about command flags, options, or usage are tool-use tasks — use man, not memory. Use this to look up, check, or explain a command's flags, options, or usage — e.g. 'What does --xattrs do in tar?' → man(page='tar'). Use this instead of bash('man ...') or web_search for command documentation.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "math",
|
||||
"description": "Execute Python code for math/computation. Code MUST use print() to produce output. Available: sympy, numpy, scipy (with scipy.special, scipy.optimize, scipy.integrate, scipy.linalg), math, fractions, itertools, functools, collections, decimal, operator, random, re, string. Common sympy names (symbols, solve, simplify, expand, factor, sqrt, Rational, Matrix, integrate, diff, etc.) are pre-imported. Example: x = symbols('x'); print(solve(x**2 - 4, x))",
|
||||
"description": "Execute Python code for math/computation. Use this for any arithmetic, algebra, or numerical task — e.g. math(code='print(2**64 - 1)'). Code must use print() to produce output. Available: sympy, numpy, scipy (with scipy.special, scipy.optimize, scipy.integrate, scipy.linalg), math, fractions, itertools, functools, collections, decimal, operator, random, re, string. Common sympy names (symbols, solve, simplify, expand, factor, sqrt, Rational, Matrix, integrate, diff, etc.) are pre-imported. Example: x = symbols('x'); print(solve(x**2 - 4, x))",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user