Compare commits

...

19 Commits

Author SHA1 Message Date
Patrick Buckley bb221f4dab chore: bump v0.8.8, update vendored katex 0.16.40 → 0.16.42 2026-03-25 12:00:26 -07:00
Patrick Buckley 361876b17a feat: eval pipeline improvements + tool description optimization (#174)
Eval pipeline:
- --optimize-tools mode freezes system prompt, optimizes tool descriptions only
- Analyst sees available tool list (prevents hallucinated "tool not in schema")
- Analyst sees current tool descriptions in --optimize-tools mode
- Three-layer timeout defense: httpx timeout + _cancelled event + client.close()
- Filter MCP-only tools (read_resource, use_prompt) from headless eval
- Pattern-over-rules framing in optimizer, analyst, and observer prompts
- Tool description diffs logged after each iteration
- Tool optimizer failure retries instead of stopping the loop

Tool renames (avoid chat template channel collision on local models):
- create_plan -> plan_agent
- task -> task_agent

Tool descriptions (from eval-driven optimization, 79% -> 98%):
- bash: environment question examples, disambiguation from write_file/man
- edit_file: multi-file workflow, prerequisite clarification, docstring example
- man: "questions about flags are tool-use tasks" prefix
- math: simple example up front
- plan_agent: "delegate to sub-agent" framing, negative boundary for direct edits
- read_file: multi-file workflow hint
- search: trigger phrases, prerequisite clarification, disambiguation
- write_file: immediate action framing, placeholder example

System prompt: enriched tool patterns from eval results, added identity opener

Test suite: plan-before-refactor -> plan-when-asked (simplified)

Docs: comprehensive eval.md rewrite covering all current features
2026-03-25 11:58:10 -07:00
renovate[bot] 5d26cd6593 chore(deps): update dependency katex to v0.16.42 (#175)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-25 11:57:59 -07:00
renovate[bot] 2d4420e00d chore(deps): lock file maintenance (#177)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-25 11:57:47 -07:00
renovate[bot] b20548583d chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.1 (#176)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-25 11:57:44 -07:00
Patrick Buckley f63b2915cc review: address copilot feedback on user_id trust check
Remove console-proxy from trusted_sources — end-user tokens via the
console proxy already carry the real user_id in the JWT, so they must
not be able to override it via the request body (impersonation risk).
Only bridge and console service identities are trusted to forward
user_id on behalf of users. Add 5 tests covering the trust boundary.
2026-03-24 03:13:23 -07:00
Patrick Buckley bf85bbea94 fix: resolve user_id to username in usage and audit displays
Usage tab grouped by user showed raw hex user_id instead of username.
Audit tab showed truncated hex. Now both API endpoints resolve user_id
to username via list_users() lookup before returning the response.
2026-03-24 03:13:23 -07:00
Patrick Buckley 803d8ee8f9 docs: document user_id propagation through MQ path
Update security.md with trusted service user_id forwarding. Update
MQ protocol diagram to include user_id field on CreateWorkstreamMessage.
Update console data flow diagram to show user_id in message and bridge
forwarding.
2026-03-24 03:13:23 -07:00
Patrick Buckley 17b5961a70 fix: propagate user_id through MQ workstream creation path
Console create_workstream was constructing CreateWorkstreamMessage
without setting user_id, so workstreams created via console→MQ→bridge
→server had empty user_id in usage events. Now the console extracts
user_id from auth_result and passes it through the MQ message. The
bridge forwards it in the HTTP payload, and the server accepts it
from trusted service callers (bridge, console-proxy).
2026-03-24 03:13:23 -07:00
Patrick Buckley 037308f3b1 fix: propagate user identity through console proxy
Console proxy previously used a fixed service identity (console-proxy)
with full {read,write,approve} scopes for all proxied requests, losing
the real user's identity at the proxy boundary. Now mints per-request
short-lived JWTs carrying the authenticated user's actual user_id,
scopes, and permissions so upstream servers record correct audit
attribution and enforce scope narrowing as defense in depth.
2026-03-24 03:13:23 -07:00
Patrick Buckley d7a9895855 feat: tool description optimization + OOM and logging fixes (#172)
* feat: tool description optimization + OOM and logging fixes

Tool description optimization (three-phase pipeline):
- Tool optimizer (phase 2) modifies tool descriptions to resolve
  confusion, gated by --optimize-tools and wrong_tool detection
- _apply_tool_overrides deep-copies modified tools, never mutates TOOLS
- _propose_tool_overrides validates JSON, deep-merges parameter overrides
- tool_overrides on EvolutionNode, plumbed through full eval pipeline
- --save-tools writes best overrides back to turnstone/tools/*.json
- Prompt optimizer informed when tool descriptions have been modified
- TSV tool_changes column

OOM fix (session lifecycle):
- HeadlessSession created inside retry loop, not outside — timed-out
  orphan threads no longer pin old sessions in memory
- Session ref cleared immediately after extracting results
- Previous behavior leaked unbounded memory per timeout (~515GB OOM)

Logging fix:
- Removed _suppress_stdout entirely — redirected fd 1 process-wide,
  causing main thread print() to vanish during slow API calls
- NullUI already discards session output; tools return strings

New CLI: --optimize-tools, --tool-optimizer-model/base-url, --save-tools

* review: address copilot feedback on tool optimization
2026-03-23 22:47:06 -07:00
Patrick Buckley 0ba49b8bb7 review: address copilot feedback on eval pipeline
- Record prompt_variant in parallel execution path (was serial-only)
- Handle "Subprocess error:" and "Skipped (fast-fail)" in failure
  classifier instead of mis-bucketing as missing_tool
- Build case_id->case_def dict once in _build_failure_analysis,
  _run_analyst, _propose_prompt_modification (was O(n²) linear scan)
- Enforce original prompt at slot 0 of cached user_prompts variants
- Use wall clock time for TSV elapsed_s instead of sum of run times
2026-03-23 19:44:32 -07:00
Patrick Buckley 51b5b3ee74 feat: multi-agent eval pipeline with tree search, analyst, diversifier
UCB tree search for prompt optimization (arXiv:2603.18620):
- EvolutionNode dataclass, UCB1 selection, rolling mean scores
- Replaces fragile linear chain with backtracking via tree
- Holdout set separation prevents optimizer overfitting
- Improvement-based delta feedback to optimizer

Multi-agent optimization pipeline:
- Analyst agent (phase 1): multi-turn with math/bash tools, identifies
  semantic failure patterns, computes statistics across test results
- Optimizer (phase 2): uses analyst diagnosis to edit developer prompt
- Observer: tunes optimizer strategy every 3 iterations
- Diversifier: generates paraphrased prompt variants for phrasing
  robustness, with dedup, delta generation, and JSON caching

Failure classification:
- 8 failure mode buckets (no_tool_call, wrong_tool, missing_tool,
  wrong_args, extra_tools, timeout, error, json_dump)
- Consistency signals (systematic, flaky, marginal)
- Rule-based pre-analysis feeds into analyst as structured input

Logging and observability:
- Config summary at startup (models, case count, runs)
- Per-case diversifier progress with dedup stats
- UCB selection reasoning, node score updates, tree growth
- Extended TSV: node_score, elapsed_s, prompt_len, iter_tokens,
  cumul_tokens columns plus 4-decimal precision

Infrastructure:
- Thread-safe fd-level stdout suppression (os.dup2)
- Prompt variants plumbed through parallel execution path
- Cached variants auto-detected from tests.json user_prompts field

New CLI flags: --explore-constant, --analyst-model/base-url,
--diversifier-model/base-url, --diversify N, --save-variants
2026-03-23 19:44:32 -07:00
Patrick Buckley c3b0ddeba7 fix: add ddgs to mypy ignore_missing_imports for CI compat 2026-03-23 19:11:50 -07:00
Patrick Buckley a533e1c783 fix: use fd-level stdout redirect in eval to avoid thread race
_suppress_stdout() was setting sys.stdout = StringIO() which is
process-global. When send_headless runs in a ThreadPoolExecutor and
blocks on an API call inside the suppression context, the main
thread's print() calls silently go to the StringIO and vanish.

Switch to os.dup2 fd-level redirect which is thread-safe.
2026-03-23 18:15:45 -07:00
Patrick Buckley d8bc78556f bump: v0.8.7 2026-03-23 17:27:23 -07:00
Patrick Buckley 4f5854e768 fix: remove stale type: ignore on ddgs import 2026-03-23 17:10:27 -07:00
Patrick Buckley bf2dc04cb3 feat: UCB tree search for eval prompt optimization
Replace linear optimization chain with UCB1 evolution tree. Each
iteration selects the most promising node to extend, preventing
irrecoverable collapse from bad edits. Also adds improvement-based
delta feedback to the optimizer and optional holdout set separation.

Inspired by "Learning to Self-Evolve" (arXiv:2603.18620).
2026-03-23 17:09:22 -07:00
Patrick Buckley bb894d073b fix: SQLite migrations use batch_alter_table for compat
Migrations 014, 021-024 used bare op.add_column/alter_column/drop_column
which fails on SQLite (no ALTER of constraints). Switch to
batch_alter_table and enable render_as_batch in env.py. Migration 014
also moved UniqueConstraint inline into create_table.
2026-03-23 17:09:13 -07:00
106 changed files with 2725 additions and 542 deletions
+1 -1
View File
@@ -8,7 +8,7 @@ FROM python:3.14-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
COPY --from=ghcr.io/astral-sh/uv:0.10.12 /uv /usr/local/bin/uv
COPY --from=ghcr.io/astral-sh/uv:0.11.1 /uv /usr/local/bin/uv
# System dependencies for psycopg (PostgreSQL client library)
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends libpq5 \
+1 -1
View File
@@ -91,7 +91,7 @@ turnstone/
_config.py Base ChannelConfig dataclass
discord/ Discord adapter (bot, cog, views, streaming, config)
shared_static/ Shared design system (base.css, auth.js, theme.js, toast.js, utils.js, kb.js)
katex-0.16.40/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
katex-0.16.42/ Vendored KaTeX math rendering library (MIT, woff2 fonts)
ui/
colors.py ANSI color constants with NO_COLOR support
markdown.py Streaming terminal markdown renderer (line-buffered)
+1 -1
View File
@@ -365,7 +365,7 @@ SSE streams (`/v1/api/events`, `/v1/api/events/global`) are proxied as raw byte
### Authentication
The proxy forwards the user's JWT to upstream server nodes — it extracts the token from the incoming request's cookie (or `Authorization` header) and adds it as a `Bearer` header on the proxied request. Since all services share the same `TURNSTONE_JWT_SECRET`, the user's JWT is valid on every node without re-authentication. The console's own auth middleware also checks proxy routes — `POST` requests to proxy write endpoints (`/v1/api/send`, `/v1/api/approve`, etc.) require `write` scope, preventing read-only tokens from escalating via proxy. The static `--auth-token` / `proxy_auth_token` is used as a fallback when no user JWT is present.
The proxy mints a short-lived (5-minute) JWT per request carrying the real user's `user_id`, `scopes`, and `permissions` with `aud: turnstone-server`. The user's console JWT (`aud: turnstone-console`) cannot be forwarded directly — it would be rejected by the server's audience validation — so the console re-signs a new server-audience JWT from the validated `AuthResult`. This preserves audit attribution (the upstream server sees the real user, not a service identity) and enforces scope narrowing as defense in depth (a read-only console user's proxied request carries only `read` scope). The JWT `src` claim is set to `"console-proxy"` for audit traceability. When no user context is available (auth disabled), the proxy falls back to a `ServiceTokenManager` with service identity `console-proxy`. The static `--auth-token` / `proxy_auth_token` is used as a final fallback.
---
+1
View File
@@ -61,6 +61,7 @@ package "Inbound Messages (Client → Bridge)" as InPkg #FFF3E0 {
+ target_node: str = ""
+ initial_message: str = ""
+ skill: str = ""
+ user_id: str = ""
}
class CloseWorkstreamMessage {
+3 -2
View File
@@ -154,7 +154,7 @@ activate Server #FFECB3
Server -> CC : _pick_best_node() or\nget_node_detail(node_id)
CC --> Server : node validated
Server -> Server : Build CreateWorkstreamMessage\n{target_node:"nodeA", name:"new-task"}
Server -> Server : Build CreateWorkstreamMessage\n{target_node:"nodeA", name:"new-task",\nuser_id: from auth_result}
Server -> Redis : RPUSH turnstone:inbound:nodeA\n(directed queue)
Server --> Browser : {status:"ok", correlation_id:"abc",\ntarget_node:"nodeA"}
@@ -163,7 +163,8 @@ deactivate Server
note right of Redis
Bridge on Node-A picks up the
message from its directed queue,
POSTs to /v1/api/workstreams/new,
POSTs to /v1/api/workstreams/new
(forwarding user_id in payload),
registers ownership, publishes
ws_created to cluster channel.
end note
+11
View File
@@ -176,4 +176,15 @@ note bottom of SH
Both share JWT signing secret
end note
note left of JWT
**Console Proxy Token Minting**
When proxying requests to server nodes:
1. Console AuthMiddleware validates user JWT (aud: turnstone-console)
2. Proxy mints new JWT (aud: turnstone-server)
with real user_id, scopes, permissions
3. src: "console-proxy" for audit traceability
4. 5-minute expiry (fresh per request)
5. Fallback: ServiceTokenManager if no user context
end note
@enduml
+254 -68
View File
@@ -2,7 +2,8 @@
`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
runs test cases against the LLM, scores tool call sequences against expected
actions, and optionally uses the model to self-optimize the developer prompt.
actions, and optionally uses a multi-agent pipeline to optimize the developer
prompt and tool descriptions.
Source: `turnstone/eval.py`
@@ -10,15 +11,24 @@ Source: `turnstone/eval.py`
## Overview
The system works in an iterative loop:
The system uses UCB tree search to explore prompt variants:
1. Run each test case N times against the current developer prompt.
2. Score each run by comparing the actual tool call sequence to expected actions.
3. If not all tests pass, use the model to rewrite the prompt based on failures.
4. Repeat until all tests pass or max iterations are reached.
1. Maintain an **evolution tree** of prompt variants, starting from the initial prompt.
2. Each iteration, **UCB1 selects** the most promising node to evaluate.
3. Run each test case N times against the selected prompt.
4. Score each run by comparing the actual tool call sequence to expected actions.
5. If not all tests pass, run a **three-phase optimization pipeline**:
- Phase 1: Analyst diagnoses semantic failure patterns
- Phase 2: Tool optimizer adjusts tool descriptions (when `--optimize-tools`)
- Phase 3: Prompt optimizer proposes a child variant (when not `--optimize-tools`)
6. Add the child to the tree and repeat until all tests pass or max iterations reached.
When optimization is disabled (`--no-optimize`), only step 1 and 2 execute
(a single iteration).
This approach (inspired by [Learning to Self-Evolve](https://arxiv.org/abs/2603.18620))
prevents irrecoverable collapse from bad edits — UCB naturally backtracks to
high-scoring ancestors instead of following a linear chain.
When optimization is disabled (`--no-optimize`), only steps 2-4 execute
(a single iteration evaluating the root node).
---
@@ -65,6 +75,7 @@ Test suites are JSON files with this structure:
| `match_mode` | no | `"ordered_subset"` | How to match actual vs expected actions (see Scoring). |
| `max_turns` | no | `10` | Maximum conversation turns before stopping. |
| `n_runs` | no | suite default or 3 | Per-case override for number of runs. |
| `holdout` | no | `false` | If `true`, this case is evaluated but excluded from optimizer feedback. Used to measure progress without overfitting. |
### Expected Action Specs
@@ -135,6 +146,7 @@ deterministic, non-interactive execution suitable for automated testing.
| Stdout | Normal | Suppressed during execution |
| Tool logging | Display only | Structured `tool_call_log` |
| System prompt | Built-in developer prompt | Overridable via constructor |
| Cancellation | N/A | `_cancelled` event for timeout cleanup |
### NullUI
@@ -156,20 +168,34 @@ def send_headless(
Runs a complete multi-turn conversation:
1. Appends the user message.
2. Calls the model API (non-streaming).
3. If tool calls are returned, executes them (with stdout suppressed) and
2. Checks `_cancelled` event — stops if set (timeout cleanup).
3. Calls the model API (non-streaming).
4. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
4. Repeats up to `max_turns` or until the model responds without tool calls.
5. Returns the tool call log: list of dicts with keys `tool`, `args`,
5. Repeats up to `max_turns` or until the model responds without tool calls.
6. Returns the tool call log: list of dicts with keys `tool`, `args`,
`result` (truncated to 500 chars), and `turn`.
Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
### Timeout and Cancellation
Each test runs in a `ThreadPoolExecutor(max_workers=1)` with a per-test
timeout (`--test-timeout`). Each attempt gets its own `OpenAI` client with
a matching httpx read timeout. On timeout, three layers of defense prevent
zombie connections:
1. **httpx timeout**: Per-request read timeout aborts the HTTP call and
releases the server slot.
2. **`_cancelled` event**: Prevents the orphan thread from starting new turns.
3. **`run_client.close()`**: Closes the connection pool to abort any
in-flight request.
### Retry Logic
`send_headless()` is called inside `_run_single_test()` with retry logic:
3 attempts with exponential backoff (sleep `2^attempt` seconds) on any
exception. This prevents transient API errors from poisoning eval scores.
exception. `TimeoutError` is re-raised immediately (no retry).
---
@@ -180,68 +206,175 @@ Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A new `HeadlessSession` is created with the current developer prompt.
5. `send_headless()` runs the user prompt through the conversation loop.
6. The tool log is scored against expected actions.
7. The temp directory is cleaned up.
4. A per-attempt `OpenAI` client is created with httpx timeout matching `--test-timeout`.
5. A new `HeadlessSession` is created with the current developer prompt.
6. `send_headless()` runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
8. The temp directory is cleaned up.
The memory database is also isolated per test (an ephemeral SQLite database
in the temp directory) so tests do not pollute each other or the user's
real memory store.
### Parallel Execution
With `--parallel N` (N > 1), tests run in a `ProcessPoolExecutor` with N
workers. Each subprocess creates its own `OpenAI` client. This is suitable
for remote API endpoints but will overwhelm local inference servers. The
default (`--parallel 1`) runs tests serially.
---
## Optimization Loop
## Model Roles
`run_optimization()` is the main entry point for iterative prompt optimization.
The eval pipeline uses up to five separate model roles, each independently
configurable. All roles inherit from the test model by default, with a
cascade chain:
```
test model (--base-url, --model)
└─ optimizer (--optimizer-*)
├─ observer (--observer-*)
├─ analyst (--analyst-*)
├─ diversifier (--diversifier-*)
└─ tool optimizer (--tool-optimizer-*)
```
| Role | Purpose | When it runs |
|------|---------|--------------|
| **Test** | The model being evaluated | Every iteration |
| **Analyst** | Diagnoses semantic failure patterns with tool use | When pass rate < 100% |
| **Optimizer** | Rewrites the developer prompt | Every iteration (unless `--optimize-tools`) |
| **Tool optimizer** | Rewrites tool descriptions | When `--optimize-tools` is set |
| **Observer** | Tunes the optimizer's strategy | Every 3 iterations |
| **Diversifier** | Generates prompt paraphrases | Once before the loop (when `--diversify N`) |
Typical setup: local model for test, Opus for analyst, Sonnet for
optimizer/observer/diversifier.
---
## Optimization Pipeline
### Flow
```
for iteration in 0..max_iterations:
1. Run all test cases n_runs times with current prompt
2. Score and aggregate results
3. Save intermediate results to JSON
4. If all tests pass -> stop
5. Every 3 iterations (at iteration 2, 5, 8, ...):
-> Observer reviews optimizer strategy
-> Reset prompt to best-performing iteration
6. Propose new prompt via optimizer model call
7. If prompt unchanged -> stop
8. Continue with new prompt
1. UCB select → pick the most promising tree node
2. Run all test cases n_runs times with selected node's prompt
3. Update node score (rolling mean) and visit count
4. Save intermediate results + tree state to JSON
5. If all tests pass → stop
6. Phase 1: Analyst diagnoses semantic failure patterns
7. Phase 2 (--optimize-tools only): Tool optimizer adjusts descriptions
8. Phase 3 (default only): Prompt optimizer proposes new prompt
9. Every 3 iterations: Observer tunes the optimizer's strategy
10. Add child node to tree (if prompt or tools changed)
```
### Prompt Proposal (`_propose_prompt_modification`)
### Phase 1: Analyst (`_run_analyst`)
Uses the model to rewrite the developer prompt based on test results:
A multi-turn agent with `math` (Python) and `bash` tools for computing
statistics. It receives per-case results with failure classifications and
produces a structured diagnosis:
- **Input**: Current prompt, test case definitions, per-case results with
actual vs expected tool sequences, and a history of the last 3 iterations.
- **Optimizer system prompt** (`OPTIMIZER_SYSTEM`): Instructs the model to
act as a text rewriter. Key guidance includes:
- Address critical failure modes (text-only responses, write_file vs edit_file,
unnecessary search before create, missing plan calls).
- Preserve phrasing that drives 100% pass rate on passing tests.
- Use direct imperative style with concrete tool call examples.
- Stay within 130% of original prompt length.
- **Output**: The rewritten prompt text (stripped of reasoning tags and code fences).
- **Failure patterns**: Shared root causes across failing cases
- **Success/failure contrast**: What distinguishes passing from failing cases
- **Consistency signals**: Systematic (0%), flaky (1-79%), marginal (80-99%)
- **Recommended fixes**: Priority-ordered patterns/examples to add or adjust
### Observer System (`_observe_and_update_optimizer`)
The analyst is instructed to frame fixes as patterns and examples, not
imperative rules — this feeds cleaner signal to the optimizer.
Every 3 iterations, a meta-level "observer" reviews the optimizer's strategy:
In `--optimize-tools` mode, the analyst receives the current tool descriptions
(with any overrides applied) and focuses on tool confusion and description
issues rather than system prompt patterns.
- Analyzes the iteration history: score trends, regressions, prompt length changes,
### Phase 2: Tool Optimizer (`_propose_tool_overrides`)
Runs when `--optimize-tools` is set. Receives the current tool descriptions,
confusion failures (where the model picked the wrong tool), and the analyst's
diagnosis. Returns a JSON override dict that modifies tool descriptions.
Overrides are validated against known tool names — only `description` and
`parameters` changes are accepted (no tool renaming at eval time).
After each iteration, changed descriptions are logged as old → new diffs
for easy visual inspection.
### Phase 3: Prompt Optimizer (`_propose_prompt_modification`)
Skipped in `--optimize-tools` mode. Receives the current prompt, test
results with per-case pass rates and deltas from the parent node, and the
analyst's diagnosis. Returns a rewritten prompt.
The optimizer is instructed to prefer patterns over rules — concrete tool
chain examples teach better than imperative directives like "ALWAYS" or
"NEVER." If the current prompt contains rule-heavy language, the optimizer
is guided to replace it with examples.
### Two Optimization Surfaces
The system supports alternating between two optimization surfaces:
1. **System prompt optimization** (default): Freeze tool descriptions,
optimize the developer prompt. Run until scores plateau.
2. **Tool description optimization** (`--optimize-tools`): Freeze the system
prompt, optimize tool descriptions only. Run until scores plateau.
Each surface lifts the floor for the other — tool description improvements
may unlock system prompt gains that weren't reachable before, and vice versa.
### Observer (`_observe_and_update_optimizer`)
Every 3 iterations, a meta-level observer reviews the optimizer's strategy:
- Analyzes iteration history: score trends, regressions, prompt length changes,
and diffs between iterations.
- Summarizes the optimizer's behavioral patterns (list style, header usage, length).
- Uses `OBSERVER_SYSTEM` to rewrite the optimizer's own system prompt.
- Detects whether the optimizer is producing rule-heavy or pattern-based output.
- Rewrites the optimizer's own system prompt to correct course.
- Rejects degenerate outputs (over 200% of input length).
- After updating the optimizer prompt, resets the developer prompt to the
best-performing iteration so far.
This two-level optimization (optimizer + observer) helps the system escape
local minima and adjust its rewriting strategy.
### Prompt Diversification
### Result Persistence
When `--diversify N` is set, the diversifier generates N paraphrased variants
of each test case's user prompt before the optimization loop. Each run cycles
through variants (round-robin), testing robustness across phrasings.
Variants can be cached back to the test suite JSON with `--save-variants`,
and auto-loaded on subsequent runs even without `--diversify`.
---
## Evolution Tree
The optimization maintains a tree of prompt variants (`EvolutionNode`), where
each node stores its prompt text, tool overrides, aggregated score, and visit
count. The root node (ID 0) contains the initial prompt.
**UCB1 selection**: Each iteration picks the node with the highest Upper
Confidence Bound score: `R_bar + C * sqrt(ln(N) / v)`, where `R_bar` is the
node's mean score, `N` is total visits across all nodes, `v` is the node's
visit count, and `C` is the exploration constant (`--explore-constant`,
default sqrt(2)). Unvisited nodes are always selected first.
### Holdout Cases
Test cases with `"holdout": true` are evaluated every iteration but excluded
from the optimizer's feedback. This prevents the optimizer from overfitting
to specific test cases. Node scores are computed from holdout cases only
(when present). If fewer than 2 non-holdout cases remain, holdout is disabled.
### Improvement-Based Feedback
The optimizer sees delta scores (`delta=+20%`) alongside absolute pass rates,
showing how each case improved relative to the parent node's evaluation. This
provides a cleaner signal than absolute scores alone — the optimizer can
distinguish beneficial edits from harmful ones regardless of starting point.
---
## Result Persistence
After each iteration, results are written to the output JSON file. The
structure is:
@@ -251,9 +384,15 @@ structure is:
"meta": {
"model": "model-name",
"base_url": "http://localhost:8000/v1",
"optimizer_model": "claude-opus-4-6",
"observer_model": "claude-opus-4-6",
"started": "2025-01-01T00:00:00",
"test_suite": "tests.json",
"n_runs_default": 3
"n_runs_default": 3,
"explore_constant": 1.414,
"holdout_ids": [],
"diversify": 10,
"prompt_variants": {"case_id": ["variant1", "variant2"]}
},
"iterations": [
{
@@ -261,7 +400,11 @@ structure is:
"prompt": "the developer prompt used",
"prompt_diff": null,
"optimizer_system": "the optimizer system prompt",
"analyst": "analyst diagnosis output",
"tool_overrides": {"bash": {"description": "..."}},
"timestamp": "2025-01-01T00:01:00",
"tree_node_id": 0,
"tree_child_id": 1,
"cases": {
"test_name": {
"runs": [
@@ -287,9 +430,21 @@ structure is:
"overall_pass_rate": 0.8,
"overall_avg_score": 0.87,
"json_dumps": 0,
"per_case_pass_rates": {"test_name": 1.0, ...}
"per_case_pass_rates": {"test_name": 1.0}
}
}
],
"tree": [
{
"node_id": 0,
"parent_id": null,
"prompt": "initial prompt",
"tool_overrides": {},
"score": 0.85,
"visit_count": 3,
"children": [1, 2],
"iteration": 0
}
]
}
```
@@ -306,26 +461,57 @@ turnstone-eval tests.json # evaluate + optimize
turnstone-eval tests.json --no-optimize # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt # start from a custom prompt
turnstone-eval tests.json --optimize-tools # optimize tool descriptions only
turnstone-eval tests.json --diversify 10 # test with prompt variants
turnstone-eval tests.json -v # verbose per-turn logging
```
### Multi-model setup (local test model, cloud optimizer)
```
turnstone-eval tests.json \
--base-url http://localhost:8000/v1 \
--optimizer-base-url https://api.anthropic.com \
--optimizer-model claude-sonnet-4-6 \
--analyst-model claude-opus-4-6
```
### All Options
| Flag | Default | Description |
|---------------------|-------------------------------|-------------|
| `test_file` | (positional, required) | Path to test cases JSON file. |
| `--base-url` | `http://localhost:8000/v1` | API base URL. |
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
| `--temperature` | 0.7 | Sampling temperature. |
| `--max-tokens` | 32768 | Max completion tokens. |
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
| `--context-window` | 131072 | Context window size. |
| `--output` | `eval_results.json` | Output results file path. |
| `-v`, `--verbose` | false | Show detailed per-turn logging (API calls, tool args, results). |
| Flag | Default | Description |
|-------------------------|----------------------------|-------------|
| `test_file` | (positional, required) | Path to test cases JSON file. |
| `--base-url` | `http://localhost:8000/v1` | API base URL for the test model. |
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
| `--temperature` | 0.7 | Sampling temperature. |
| `--max-tokens` | 32768 | Max completion tokens. |
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
| `--context-window` | 131072 | Context window size. |
| `--output` | `eval_results.json` | Output results file path. |
| `-v`, `--verbose` | false | Show detailed per-turn logging. |
| `--explore-constant` | 1.414 (sqrt(2)) | UCB exploration constant C. |
| `--test-timeout` | 300 | Per-test timeout in seconds. |
| `--suite-timeout` | 0 (unlimited) | Total suite timeout in seconds. |
| `--no-fast-fail` | false | Disable early termination on all-zero initial runs. |
| `--parallel` | 1 (serial) | Parallel workers (0=auto, N=use N workers). |
| `--optimizer-model` | same as `--model` | Model for prompt optimization. |
| `--optimizer-base-url` | same as `--base-url` | Base URL for optimizer model. |
| `--observer-model` | same as optimizer | Model for meta-optimization (observer). |
| `--observer-base-url` | same as optimizer | Base URL for observer model. |
| `--analyst-model` | same as optimizer | Model for failure analysis. |
| `--analyst-base-url` | same as optimizer | Base URL for analyst model. |
| `--diversify` | 0 (disabled) | Generate N prompt variants per test case. |
| `--diversifier-model` | same as optimizer | Model for prompt diversification. |
| `--diversifier-base-url`| same as optimizer | Base URL for diversifier model. |
| `--save-variants` | false | Save generated variants back to test suite JSON. |
| `--optimize-tools` | false | Optimize tool descriptions only (freeze system prompt). |
| `--tool-optimizer-model` | same as optimizer | Model for tool description optimization. |
| `--tool-optimizer-base-url` | same as optimizer | Base URL for tool optimizer model. |
| `--save-tools` | false | Write optimized tool descriptions back to `turnstone/tools/*.json`. |
### Precedence for n_runs
+35 -8
View File
@@ -457,14 +457,30 @@ without any database.
### Proxy auth forwarding
When the console proxies requests to server nodes (via `/node/{id}/...`
routes), it uses a dedicated **service proxy token** with
`aud: turnstone-server` and `write` scope. The user's console JWT
(which has `aud: turnstone-console`) is **not** forwarded — it would be
rejected by the server's audience validation.
routes), it mints a **short-lived user-scoped JWT** with
`aud: turnstone-server` carrying the real user's `user_id`, `scopes`,
and `permissions`. The user's console JWT (which has
`aud: turnstone-console`) is **not** forwarded directly — it would be
rejected by the server's audience validation. Instead, the console
re-signs a new JWT targeted at the server audience.
The proxy token is managed by a `ServiceTokenManager` that auto-rotates
1-hour JWTs, refreshing at 80% of lifetime. If `--auth-token` is
provided, that static token is used instead.
Each proxied request gets a fresh JWT (5-minute expiry). This ensures:
- **Audit attribution** — the upstream server records the real user in
`ctx_user_id` and audit events, not a generic service identity.
- **Scope narrowing** — a read-only console user's proxied request
carries only `read` scope, not the full `{read, write, approve}` set.
The server enforces this as defense in depth.
- **Permission forwarding** — granular RBAC permissions from the
console JWT are carried through to the server.
The JWT `src` claim is set to `"console-proxy"`, allowing servers to
distinguish proxied requests from direct logins in audit logs.
When no user context is available (auth disabled, or internal requests),
the proxy falls back to a `ServiceTokenManager` with service identity
`console-proxy` and full scopes. If `--auth-token` is provided, that
static token is used as a final fallback.
### Service-to-service authentication
@@ -475,7 +491,7 @@ auto-rotating JWTs when communicating with server nodes:
|---------|----------|-------|----------|---------|
| Bridge | `bridge` | `approve` | `turnstone-server` | Tool approval proxy, message relay |
| Console collector | `console-collector` | `read` | `turnstone-server` | Node health polling |
| Console proxy | `console-proxy` | `write` | `turnstone-server` | Proxied API calls |
| Console proxy (fallback) | `console-proxy` | `approve` | `turnstone-server` | Proxied API calls when no user context |
| Channel notify | `system` | `write` | `turnstone-channel` | Notification delivery to channel gateway |
Service tokens use 1-hour expiry with automatic refresh via
@@ -483,6 +499,17 @@ Service tokens use 1-hour expiry with automatic refresh via
httpx event hooks to ensure rotated tokens are picked up on SSE
reconnects.
### User identity in MQ-dispatched workstreams
When the console creates a workstream via MQ (the normal path), the
authenticated user's `user_id` is embedded in the
`CreateWorkstreamMessage`. The bridge forwards this `user_id` in the
HTTP payload when calling the server's `POST /v1/api/workstreams/new`.
The server accepts a `user_id` from the request body **only when the
caller is a trusted service** — identified by `token_source` matching
`bridge`, `console-proxy`, or `console`. Regular API callers cannot
override `user_id`; the server always uses their JWT identity.
Note that the channel gateway uses a distinct JWT audience
(`turnstone-channel`) from the server (`turnstone-server`) and console
(`turnstone-console`). A server-scoped JWT cannot authenticate to the
+6 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "turnstone"
version = "0.8.6"
version = "0.8.8"
description = "Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."
readme = "README.md"
license = "BUSL-1.1"
@@ -78,7 +78,7 @@ include = [
"turnstone/console/static/*.js",
"turnstone/shared_static/*.css",
"turnstone/shared_static/*.js",
"turnstone/shared_static/katex-0.16.40/**/*",
"turnstone/shared_static/katex-0.16.42/**/*",
"turnstone/shared_static/hljs-11.11.1/**/*",
"turnstone/shared_static/mermaid-11.13.0/**/*",
"turnstone/sdk/py.typed",
@@ -165,6 +165,10 @@ ignore_missing_imports = true
module = ["frontmatter", "frontmatter.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["ddgs", "ddgs.*"]
ignore_missing_imports = true
[[tool.mypy.overrides]]
module = ["turnstone.channels.discord.*"]
disallow_subclassing_any = false
+344 -48
View File
@@ -1,5 +1,5 @@
{
"description": "turnstone behavior tests tool selection, sequencing, and multi-step reasoning",
"description": "turnstone behavior tests \u2014 tool selection, sequencing, and multi-step reasoning",
"defaults": {
"n_runs": 5,
"max_turns": 15
@@ -8,35 +8,91 @@
{
"id": "read-before-edit",
"description": "Must read_file before edit_file on the same path",
"user_prompt": "Fix the typo in config.py change 'recieve' to 'receive'",
"user_prompt": "Fix the typo in config.py \u2014 change 'recieve' to 'receive'",
"setup": {
"files": {
"config.py": "# Config module\ndef recieve_data(source):\n \"\"\"Recieve data from source.\"\"\"\n return source.read()\n"
}
},
"expected_actions": [
{ "tool": "read_file", "args": { "path": "config.py" } },
{ "tool": "edit_file", "args": { "path": "config.py" } }
{
"tool": "read_file",
"args": {
"path": "config.py"
}
},
{
"tool": "edit_file",
"args": {
"path": "config.py"
}
}
],
"match_mode": "ordered_subset"
"match_mode": "ordered_subset",
"user_prompts": [
"Fix the typo in config.py \u2014 change 'recieve' to 'receive'",
"In config.py, correct the misspelling of 'recieve' to 'receive'",
"Please update config.py by replacing 'recieve' with the correct spelling 'receive'",
"There's a typo in config.py: 'recieve' should be 'receive'. Please fix it.",
"Could you change 'recieve' to 'receive' in config.py?",
"Go ahead and fix 'recieve' \u2192 'receive' in config.py",
"I need the word 'recieve' corrected to 'receive' in the file config.py",
"config.py has a spelling error \u2014 'recieve' needs to be changed to 'receive'",
"Kindly rectify the typographical error in config.py, replacing 'recieve' with 'receive'",
"Hey, swap 'recieve' for 'receive' in config.py"
]
},
{
"id": "write-file-not-bash",
"description": "Use write_file for file creation, not bash echo/cat",
"user_prompt": "Create a file called hello.py that prints hello world",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "hello\\.py" } }
{
"tool": "write_file",
"args_pattern": {
"path": "hello\\.py"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Create a file called hello.py that prints hello world",
"Make a hello.py file that outputs hello world",
"Please write a Python file named hello.py which prints hello world",
"I need a file called hello.py that prints hello world",
"Could you create hello.py with code that prints hello world?",
"Write hello.py \u2014 it should print hello world",
"Generate a hello.py file that outputs \"hello world\"",
"I'd like you to create a file named hello.py that prints hello world",
"Set up a file called hello.py to print hello world",
"Kindly produce a hello.py file whose purpose is to print hello world"
]
},
{
"id": "bash-for-commands",
"description": "Use bash for running system commands",
"user_prompt": "What Python version is installed?",
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "python" } }
{
"tool": "bash",
"args_pattern": {
"command": "python"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"What Python version is installed?",
"Check which version of Python is currently installed",
"Can you tell me the installed Python version?",
"python --version please",
"I need to know what version of Python is on this system",
"Which Python version do we have?",
"Could you look up the Python version that's installed here?",
"Determine the currently installed Python version",
"What's the Python version on this machine?",
"Please check the Python version"
]
},
{
"id": "search-for-patterns",
@@ -49,13 +105,30 @@
}
},
"expected_actions": [
{ "tool": "search", "args_pattern": { "query": "test_" } }
{
"tool": "search",
"args_pattern": {
"query": "test_"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Find all functions that start with 'test_' in the project",
"List every function in the project whose name begins with 'test_'",
"I need to locate all functions prefixed with 'test_' across the project",
"Could you search the project for any functions starting with 'test_'?",
"Show me all the test_ prefixed functions in this project",
"Hunt down every function that has a 'test_' prefix in the codebase",
"I'm looking for all functions named test_* throughout the project",
"Search the entire project for functions whose names start with test_",
"What functions beginning with 'test_' exist in this project?",
"Please identify all functions with the 'test_' prefix in the project files"
]
},
{
"id": "multi-file-edit",
"description": "Read and edit multiple files must read before editing each, and edit both",
"description": "Read and edit multiple files \u2014 must read before editing each, and edit both",
"user_prompt": "Change the default port from 8000 to 9000 in both server.py and config.py",
"setup": {
"files": {
@@ -64,12 +137,32 @@
}
},
"expected_actions": [
{ "tool": "read_file" },
{ "tool": "read_file" },
{ "tool": "edit_file" },
{ "tool": "edit_file" }
{
"tool": "read_file"
},
{
"tool": "read_file"
},
{
"tool": "edit_file"
},
{
"tool": "edit_file"
}
],
"match_mode": "ordered_subset"
"match_mode": "ordered_subset",
"user_prompts": [
"Change the default port from 8000 to 9000 in both server.py and config.py",
"Update the default port to 9000 instead of 8000 in server.py and config.py",
"Could you modify the port number from 8000 to 9000 in both config.py and server.py?",
"Please replace port 8000 with 9000 in server.py and config.py",
"I need the default port switched from 8000 to 9000 in both server.py and config.py",
"In server.py and config.py, the default port should be changed from 8000 to 9000",
"Swap out port 8000 for 9000 in config.py and server.py",
"Would you mind updating the default port value from 8000 to 9000 across both server.py and config.py?",
"The default port in server.py and config.py needs to be 9000 instead of 8000 \u2014 please make that change",
"Go ahead and change 8000 to 9000 for the default port in both server.py and config.py"
]
},
{
"id": "search-then-edit",
@@ -83,51 +176,147 @@
}
},
"expected_actions": [
{ "tool": "search", "args_pattern": { "query": "MAX_RETRIES" } },
{ "tool": "read_file" },
{ "tool": "edit_file", "args_pattern": { "old_string": "3" } }
{
"tool": "search",
"args_pattern": {
"query": "MAX_RETRIES"
}
},
{
"tool": "read_file"
},
{
"tool": "edit_file",
"args_pattern": {
"old_string": "3"
}
}
],
"match_mode": "ordered_subset"
"match_mode": "ordered_subset",
"user_prompts": [
"Find where MAX_RETRIES is defined and change it from 3 to 5",
"Locate the definition of MAX_RETRIES and update its value from 3 to 5",
"Could you search for where MAX_RETRIES is defined and modify it from 3 to 5?",
"I need MAX_RETRIES changed from 3 to 5 \u2014 find where it's defined and update it",
"Please find the MAX_RETRIES definition and bump it from 3 to 5",
"Hunt down MAX_RETRIES in the codebase and change its value from 3 to 5",
"Where is MAX_RETRIES set to 3? Change it to 5.",
"Search the code for the MAX_RETRIES definition and alter it from 3 to 5",
"I'd like you to locate MAX_RETRIES (currently 3) and set it to 5 instead",
"Go find MAX_RETRIES and switch it from 3 to 5"
]
},
{
"id": "bash-git-log",
"description": "Use bash for git commands, not other tools",
"user_prompt": "Show me the git log for the last 5 commits",
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "git\\s+log" } }
{
"tool": "bash",
"args_pattern": {
"command": "git\\s+log"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Show me the git log for the last 5 commits",
"Display the 5 most recent git commits",
"Can you pull up the git log limited to the last five commits?",
"I need to see the git log showing only the previous 5 commits",
"git log for the 5 latest commits, please",
"Would you mind showing me the last five entries in the git log?",
"Print out the most recent 5 commits from the git log",
"I'd like to review the git log \u2014 just the last 5 commits",
"Show the recent 5 commit history using git log",
"Could you display the git commit history for the past five commits?"
]
},
{
"id": "write-then-run",
"description": "Create a script and run it to verify it works",
"user_prompt": "Create a Python script called fib.py that prints the first 10 Fibonacci numbers, then run it to verify",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "fib\\.py" } },
{ "tool": "bash", "args_pattern": { "command": "python" } }
{
"tool": "write_file",
"args_pattern": {
"path": "fib\\.py"
}
},
{
"tool": "bash",
"args_pattern": {
"command": "python"
}
}
],
"match_mode": "ordered_subset"
"match_mode": "ordered_subset",
"user_prompts": [
"Create a Python script called fib.py that prints the first 10 Fibonacci numbers, then run it to verify",
"Write a Python file named fib.py that outputs the first 10 Fibonacci numbers, and execute it to confirm it works",
"Please make fib.py \u2013 a Python script printing the first ten Fibonacci numbers \u2013 then run it to check the output",
"I need a Python script fib.py that prints the first 10 Fibonacci numbers. Execute it afterwards to verify correctness.",
"Could you create fib.py to display the first 10 Fibonacci numbers in Python, and then run it to make sure it works?",
"Draft a script called fib.py in Python that outputs the first ten Fibonacci numbers, then execute it to validate",
"Hey, write me a fib.py that prints the first 10 Fibonacci numbers and run it so we can see it works",
"Generate a Python program fib.py which prints the initial 10 Fibonacci numbers, and verify by running it",
"Kindly produce a Python script named fib.py to print the first 10 Fibonacci numbers, then execute the script to confirm its output",
"Make a file fib.py containing Python code to print the first 10 Fibonacci numbers. Then run it to verify."
]
},
{
"id": "no-bash-for-file-write",
"description": "Should NOT use bash (echo/cat/heredoc) to create files only write_file",
"description": "Should NOT use bash (echo/cat/heredoc) to create files \u2014 only write_file",
"user_prompt": "Create a new file called README.md with a title and description of this project",
"expected_actions": [
{ "tool": "write_file", "args_pattern": { "path": "README" } }
{
"tool": "write_file",
"args_pattern": {
"path": "README"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Create a new file called README.md with a title and description of this project",
"Make a README.md file that includes a project title and description",
"I need a README.md created with a title and a brief description of the project",
"Please generate a README.md file containing the project's title and description",
"Could you set up a README.md with a title and project description?",
"Write a README.md that has a title and describes this project",
"Go ahead and create README.md \u2014 it should have a title and a description of the project",
"I'd like you to produce a new README.md file featuring a project title and description",
"Kindly establish a README.md file incorporating both a title and a description for this project",
"Spin up a README.md with a project title and description in it"
]
},
{
"id": "plan-before-refactor",
"description": "Use the plan tool before a large refactoring task",
"user_prompt": "I need to refactor this codebase to separate the database layer from the API layer. Use the plan tool to think through the approach before making any changes.",
"id": "plan-when-asked",
"description": "Call the plan tool when the user asks to plan",
"user_prompt": "Plan how to add user authentication to this app.",
"setup": {
"files": {
"app.py": "import sqlite3\nfrom flask import Flask, jsonify\n\napp = Flask(__name__)\nDB = 'data.db'\n\ndef get_db():\n return sqlite3.connect(DB)\n\n@app.route('/users')\ndef list_users():\n db = get_db()\n users = db.execute('SELECT * FROM users').fetchall()\n db.close()\n return jsonify(users)\n\n@app.route('/users/<int:uid>')\ndef get_user(uid):\n db = get_db()\n user = db.execute('SELECT * FROM users WHERE id=?', (uid,)).fetchone()\n db.close()\n return jsonify(user)\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
"app.py": "from flask import Flask, jsonify\n\napp = Flask(__name__)\n\n@app.route('/users')\ndef list_users():\n return jsonify([{'id': 1, 'name': 'Alice'}])\n\nif __name__ == '__main__':\n app.run(port=8000)\n"
}
},
"expected_actions": [{ "tool": "create_plan" }],
"match_mode": "subset"
"expected_actions": [
{
"tool": "plan_agent"
}
],
"match_mode": "subset",
"user_prompts": [
"Plan how to add user authentication to this app.",
"Make a plan for adding pagination to the API endpoints.",
"Plan out how to add error handling to this application.",
"I need a plan for adding logging to this codebase.",
"Plan the approach for adding unit tests to this app.",
"How would you approach adding user authentication to this app? Lay out a plan.",
"I'd like you to outline a strategy for implementing user authentication in this application.",
"Could you come up with a plan for integrating user authentication into this app?",
"Think through the steps needed to add user auth to this app and present a plan.",
"Draft a plan for incorporating user authentication functionality into this application."
]
},
{
"id": "edit-not-rewrite",
@@ -139,10 +328,32 @@
}
},
"expected_actions": [
{ "tool": "read_file", "args": { "path": "utils.py" } },
{ "tool": "edit_file", "args": { "path": "utils.py" } }
{
"tool": "read_file",
"args": {
"path": "utils.py"
}
},
{
"tool": "edit_file",
"args": {
"path": "utils.py"
}
}
],
"match_mode": "ordered_subset"
"match_mode": "ordered_subset",
"user_prompts": [
"Add a docstring to the process_data function in utils.py",
"Please add a docstring to the process_data function in utils.py",
"Could you write a docstring for process_data in utils.py?",
"Insert a docstring into the process_data function found in utils.py",
"I need a docstring added to process_data in utils.py",
"Put a docstring on the process_data function in utils.py",
"The process_data function in utils.py is missing a docstring \u2014 please add one",
"Would you mind adding a docstring to process_data in utils.py?",
"In utils.py, the process_data function needs a docstring",
"Add documentation via a docstring to the process_data function within utils.py"
]
},
{
"id": "bash-run-tests",
@@ -154,45 +365,130 @@
}
},
"expected_actions": [
{ "tool": "bash", "args_pattern": { "command": "pytest|python.*test" } }
{
"tool": "bash",
"args_pattern": {
"command": "pytest|python.*test"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Run the tests",
"Execute the test suite",
"Please go ahead and run the tests",
"Could you run the tests for me?",
"I need the tests to be run",
"Kick off the tests",
"Let's run the tests",
"Fire up the tests",
"Go ahead and execute the tests",
"I'd like you to run the tests"
]
},
{
"id": "web-fetch-url",
"description": "Use web_fetch when asked to retrieve content from a URL",
"user_prompt": "Fetch the contents of https://example.com and summarize what's on the page",
"expected_actions": [
{ "tool": "web_fetch", "args_pattern": { "url": "example\\.com" } }
{
"tool": "web_fetch",
"args_pattern": {
"url": "example\\.com"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Fetch the contents of https://example.com and summarize what's on the page",
"Go to https://example.com and give me a summary of what you find there",
"Could you pull up https://example.com and tell me what the page is about?",
"Retrieve the content from https://example.com, then provide a summary of it",
"I need you to grab https://example.com and summarize its contents for me",
"Please access https://example.com and give me an overview of the page",
"What's on https://example.com? Fetch it and summarize for me.",
"Download the page at https://example.com and provide a brief summary",
"I'd like a summary of whatever is at https://example.com \u2014 please fetch it first",
"Hit https://example.com and let me know what's there in summary form"
]
},
{
"id": "man-page-lookup",
"description": "Use man tool to look up command documentation",
"user_prompt": "Look up the man page for tar and tell me what the --xattrs flag does",
"expected_actions": [
{ "tool": "man", "args_pattern": { "page": "tar" } }
{
"tool": "man",
"args_pattern": {
"page": "tar"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Look up the man page for tar and tell me what the --xattrs flag does",
"What does the --xattrs flag do in tar? Check the man page for me.",
"Could you pull up the man page for tar and explain the --xattrs option?",
"I need to know what --xattrs does in tar \u2014 can you check the man page?",
"Check tar's man page and let me know the purpose of the --xattrs flag.",
"Please consult the tar man page and describe what the --xattrs flag is for.",
"Hey, look at the tar man page real quick \u2014 what's --xattrs do?",
"I'd like you to read the tar man page and summarize the --xattrs option for me.",
"Would you mind checking the man page for tar to find out what --xattrs means?",
"Look into the tar manual and explain the --xattrs flag to me."
]
},
{
"id": "math-calculation",
"description": "Use the math tool for precise calculations, not bash or mental math",
"user_prompt": "What is 2^64 - 1? Use the math tool to calculate it precisely.",
"expected_actions": [
{ "tool": "math", "args_pattern": { "code": "2.*64" } }
{
"tool": "math",
"args_pattern": {
"code": "2.*64"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"What is 2^64 - 1? Use the math tool to calculate it precisely.",
"Calculate 2^64 - 1 for me using the math tool, please.",
"I need the exact value of 2^64 - 1. Please use the math tool.",
"Could you use the math tool to compute 2^64 minus 1 precisely?",
"Use the math tool to tell me what 2^64 - 1 equals.",
"I'm curious: what's 2^64 - 1? Compute it with the math tool.",
"Please precisely determine 2^64 - 1 via the math tool.",
"Mind using the math tool to figure out 2^64 - 1 exactly?",
"I'd like to know the precise result of 2^64 - 1 \u2014 use the math tool for this.",
"Leverage the math tool to give me an exact answer for 2^64 - 1."
]
},
{
"id": "web-search-query",
"description": "Use web_search for general knowledge lookups, not web_fetch",
"user_prompt": "Search the web for the current population of Tokyo",
"expected_actions": [
{ "tool": "web_search", "args_pattern": { "query": "Tokyo" } }
{
"tool": "web_search",
"args_pattern": {
"query": "Tokyo"
}
}
],
"match_mode": "subset"
"match_mode": "subset",
"user_prompts": [
"Search the web for the current population of Tokyo",
"What's Tokyo's current population? Look it up on the web.",
"Could you do a web search to find out how many people currently live in Tokyo?",
"Please search online for Tokyo's present-day population.",
"I need you to look up the current population of Tokyo on the web.",
"Find me Tokyo's current population via a web search.",
"Web search: what is the current population of Tokyo?",
"I'd like to know Tokyo's current population\u2014can you search the web for that?",
"Look up how many people live in Tokyo right now using a web search.",
"Do a web search for the population of Tokyo as of now."
]
}
]
}
+46
View File
@@ -1237,6 +1237,52 @@ class TestJWTAudienceIssuer:
result = validate_jwt(token, self.SECRET, audience="")
assert result is not None
def test_create_jwt_expiry_seconds(self):
import jwt as pyjwt
from turnstone.core.auth import create_jwt
token = create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=300)
payload = pyjwt.decode(
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
)
assert payload["exp"] - payload["iat"] == 300
def test_create_jwt_expiry_seconds_overrides_hours(self):
import jwt as pyjwt
from turnstone.core.auth import create_jwt
token = create_jwt(
"user1",
frozenset({"read"}),
"test",
self.SECRET,
expiry_hours=24,
expiry_seconds=60,
)
payload = pyjwt.decode(
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
)
# expiry_seconds takes precedence over expiry_hours
assert payload["exp"] - payload["iat"] == 60
def test_create_jwt_expiry_seconds_rejects_zero(self):
import pytest
from turnstone.core.auth import create_jwt
with pytest.raises(ValueError, match="expiry_seconds must be positive"):
create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=0)
def test_create_jwt_expiry_seconds_rejects_negative(self):
import pytest
from turnstone.core.auth import create_jwt
with pytest.raises(ValueError, match="expiry_seconds must be positive"):
create_jwt("user1", frozenset({"read"}), "test", self.SECRET, expiry_seconds=-1)
class TestServiceTokenManager:
SECRET = "test-secret-that-is-at-least-32-chars"
+230
View File
@@ -1699,6 +1699,236 @@ class TestSSEProxy:
asyncio.run(_run())
# ---------------------------------------------------------------------------
# Proxy auth header propagation
# ---------------------------------------------------------------------------
class TestProxyAuthHeaders:
"""Verify _proxy_auth_headers mints user-scoped JWTs for proxy requests."""
SECRET = "test-secret-that-is-at-least-32-chars"
def _make_request(
self, *, auth_result=None, jwt_secret="", proxy_token_mgr=None, proxy_auth_token=""
):
"""Build a minimal fake request for _proxy_auth_headers."""
class _State:
pass
class _AppState:
pass
class _App:
state = _AppState()
class _Request:
state = _State()
app = _App()
req = _Request()
req.state.auth_result = auth_result
req.app.state.jwt_secret = jwt_secret
req.app.state.proxy_token_mgr = proxy_token_mgr
req.app.state.proxy_auth_token = proxy_auth_token
return req
def test_mints_user_jwt(self):
"""Real user auth_result → JWT with correct sub, scopes, src, aud, permissions."""
import jwt as pyjwt
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
auth = AuthResult(
user_id="alice",
scopes=frozenset({"read", "write"}),
token_source="jwt",
permissions=frozenset({"admin.users"}),
)
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
headers = _proxy_auth_headers(req)
assert "Authorization" in headers
token = headers["Authorization"].removeprefix("Bearer ")
payload = pyjwt.decode(token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
assert payload["sub"] == "alice"
assert set(payload["scopes"].split(",")) == {"read", "write"}
assert payload["src"] == "console-proxy"
assert payload["aud"] == JWT_AUD_SERVER
assert payload["permissions"] == "admin.users"
def test_narrows_scopes(self):
"""Read-only user → JWT carries only read scope, not full {read,write,approve}."""
import jwt as pyjwt
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import JWT_AUD_SERVER, AuthResult
auth = AuthResult(
user_id="viewer",
scopes=frozenset({"read"}),
token_source="jwt",
)
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
headers = _proxy_auth_headers(req)
token = headers["Authorization"].removeprefix("Bearer ")
payload = pyjwt.decode(token, self.SECRET, algorithms=["HS256"], audience=JWT_AUD_SERVER)
assert payload["scopes"] == "read"
def test_short_expiry(self):
"""Minted JWT expires in 300 seconds, not hours."""
import jwt as pyjwt
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="alice",
scopes=frozenset({"read"}),
token_source="jwt",
)
req = self._make_request(auth_result=auth, jwt_secret=self.SECRET)
headers = _proxy_auth_headers(req)
token = headers["Authorization"].removeprefix("Bearer ")
payload = pyjwt.decode(
token, self.SECRET, algorithms=["HS256"], options={"verify_aud": False}
)
assert payload["exp"] - payload["iat"] == 300
def test_fallback_no_user(self):
"""No auth_result → falls back to ServiceTokenManager."""
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import ServiceTokenManager
mgr = ServiceTokenManager(
user_id="console-proxy",
scopes=frozenset({"read", "write", "approve"}),
source="console",
secret=self.SECRET,
)
req = self._make_request(proxy_token_mgr=mgr)
headers = _proxy_auth_headers(req)
assert "Authorization" in headers
assert headers["Authorization"] == f"Bearer {mgr.token}"
def test_fallback_no_secret(self):
"""auth_result present but empty jwt_secret → falls back to ServiceTokenManager."""
from turnstone.console.server import _proxy_auth_headers
from turnstone.core.auth import AuthResult, ServiceTokenManager
auth = AuthResult(
user_id="alice",
scopes=frozenset({"read"}),
token_source="jwt",
)
mgr = ServiceTokenManager(
user_id="console-proxy",
scopes=frozenset({"read", "write", "approve"}),
source="console",
secret=self.SECRET,
)
req = self._make_request(auth_result=auth, jwt_secret="", proxy_token_mgr=mgr)
headers = _proxy_auth_headers(req)
# Should use ServiceTokenManager, not mint a user JWT
assert headers["Authorization"] == f"Bearer {mgr.token}"
def test_fallback_static_token(self):
"""No auth_result, no ServiceTokenManager → uses static proxy_auth_token."""
from turnstone.console.server import _proxy_auth_headers
req = self._make_request(proxy_auth_token="static-tok-123")
headers = _proxy_auth_headers(req)
assert headers == {"Authorization": "Bearer static-tok-123"}
# ---------------------------------------------------------------------------
# Server: trusted user_id forwarding on create_workstream
# ---------------------------------------------------------------------------
class TestCreateWorkstreamUserIdTrust:
"""Verify that only trusted service tokens can forward user_id in create_workstream."""
def _extract_uid(self, body: dict, auth_result) -> str:
"""Replicate the trust check from server.py:create_workstream."""
auth = auth_result
uid: str = getattr(auth, "user_id", "") or ""
trusted_sources = {"bridge", "console"}
if (
body.get("user_id")
and isinstance(body["user_id"], str)
and auth is not None
and auth.token_source in trusted_sources
):
uid = body["user_id"]
return uid
def test_bridge_can_forward_user_id(self):
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="bridge",
scopes=frozenset({"approve"}),
token_source="bridge",
)
uid = self._extract_uid({"user_id": "real-user-abc"}, auth)
assert uid == "real-user-abc"
def test_console_service_can_forward_user_id(self):
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="console",
scopes=frozenset({"approve"}),
token_source="console",
)
uid = self._extract_uid({"user_id": "real-user-abc"}, auth)
assert uid == "real-user-abc"
def test_console_proxy_user_cannot_override_user_id(self):
"""End-user tokens via console-proxy must NOT override user_id."""
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="real-user-abc",
scopes=frozenset({"read", "write"}),
token_source="console-proxy",
)
uid = self._extract_uid({"user_id": "impersonated-user"}, auth)
# Should use JWT identity, NOT the body override
assert uid == "real-user-abc"
def test_direct_user_cannot_override_user_id(self):
"""Direct JWT login must NOT override user_id."""
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="real-user-abc",
scopes=frozenset({"read", "write"}),
token_source="password",
)
uid = self._extract_uid({"user_id": "impersonated-user"}, auth)
assert uid == "real-user-abc"
def test_no_body_user_id_uses_jwt(self):
from turnstone.core.auth import AuthResult
auth = AuthResult(
user_id="bridge",
scopes=frozenset({"approve"}),
token_source="bridge",
)
uid = self._extract_uid({"name": "test-ws"}, auth)
assert uid == "bridge"
# ---------------------------------------------------------------------------
# Collector — MCP aggregation in get_overview()
# ---------------------------------------------------------------------------
+2
View File
@@ -573,7 +573,9 @@ def _fake_request(*nodes: dict[str, Any], proxy_client: Any = None) -> MagicMock
collector.get_nodes.return_value = (list(nodes), len(nodes))
collector.get_all_nodes.side_effect = lambda: collector.get_nodes.return_value[0]
req = MagicMock()
req.state.auth_result = None
req.app.state.collector = collector
req.app.state.jwt_secret = ""
req.app.state.proxy_client = proxy_client or AsyncMock()
req.app.state.proxy_token_mgr = None
req.app.state.proxy_auth_token = "tok"
+4 -4
View File
@@ -214,7 +214,7 @@ class TestPlanExec:
"id": tc_id,
"type": "function",
"function": {
"name": "create_plan",
"name": "plan_agent",
"arguments": json.dumps({"goal": prior_prompt}),
},
}
@@ -250,7 +250,7 @@ class TestPlanExec:
m for m in messages if m["role"] == "assistant" and m.get("tool_calls")
]
assert len(assistant_with_tc) == 1
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "create_plan"
assert assistant_with_tc[0]["tool_calls"][0]["function"]["name"] == "plan_agent"
# The real tool result is forwarded with its original content
tool_msgs = [m for m in messages if m["role"] == "tool"]
@@ -463,7 +463,7 @@ class TestPlanRefinement:
with patch.object(session, "_refine_plan", side_effect=fake_refine):
items = [
{
"func_name": "create_plan",
"func_name": "plan_agent",
"call_id": "c1",
"prompt": "add auth",
}
@@ -576,7 +576,7 @@ class TestPlanRefinement:
msgs = captured["messages"]
assert msgs[0]["role"] == "system"
assert msgs[1]["role"] == "assistant"
assert msgs[1]["tool_calls"][0]["function"]["name"] == "create_plan"
assert msgs[1]["tool_calls"][0]["function"]["name"] == "plan_agent"
assert msgs[2]["role"] == "tool"
assert msgs[2]["content"] == self.GOOD_PLAN
assert msgs[3]["role"] == "user"
+2 -2
View File
@@ -104,8 +104,8 @@ class TestToolsMetadata:
"man": "page",
"web_fetch": "url",
"web_search": "query",
"task": "prompt",
"create_plan": "goal",
"task_agent": "prompt",
"plan_agent": "goal",
"memory": "name",
"recall": "query",
"notify": "message",
+1 -1
View File
@@ -1,3 +1,3 @@
"""turnstone - Multi-node AI orchestration platform with tool use, agent routing, and cluster simulation."""
__version__ = "0.8.6"
__version__ = "0.8.8"
+51 -5
View File
@@ -37,7 +37,7 @@ from starlette.staticfiles import StaticFiles
from turnstone.api.console_spec import build_console_spec
from turnstone.api.docs import make_docs_handler, make_openapi_handler
from turnstone.console.collector import ClusterCollector
from turnstone.core.auth import JWT_AUD_CONSOLE, AuthMiddleware
from turnstone.core.auth import JWT_AUD_CONSOLE, JWT_AUD_SERVER, AuthMiddleware, create_jwt
if TYPE_CHECKING:
from collections.abc import AsyncGenerator
@@ -133,15 +133,36 @@ _CONSOLE_PROXY_STYLE = "<style>.dashboard-overlay{top:32px!important}</style>"
_VALID_NODE_ID = re.compile(r"^[a-zA-Z0-9._-]+$")
_PROXY_JWT_EXPIRY_SECONDS = 300 # 5 min — ample for any request round-trip
def _proxy_auth_headers(request: Request) -> dict[str, str]:
"""Build auth headers for proxied requests to upstream servers.
Uses the service proxy token (``JWT_AUD_SERVER``) so the upstream node
accepts the request. The user's console-audience JWT is *not* forwarded
it would be rejected by the server's audience validation.
Mints a short-lived JWT carrying the real user's identity and scopes
so the upstream server records correct audit attribution and enforces
scope narrowing. Falls back to the ServiceTokenManager when no user
context is available.
"""
# Prefer the auto-rotating ServiceTokenManager when available
auth_result = getattr(getattr(request, "state", None), "auth_result", None)
jwt_secret: str = getattr(request.app.state, "jwt_secret", "")
if auth_result is not None and auth_result.user_id and jwt_secret:
token = create_jwt(
user_id=auth_result.user_id,
scopes=auth_result.scopes,
source="console-proxy",
secret=jwt_secret,
audience=JWT_AUD_SERVER,
permissions=auth_result.permissions,
expiry_seconds=_PROXY_JWT_EXPIRY_SECONDS,
)
return {"Authorization": f"Bearer {token}"}
# Fallback: service identity (no user context).
# When auth is disabled on the console, auth_result is None, so all proxied
# requests use the full-privilege service identity. This is safe only when
# the upstream server also has auth disabled.
mgr = getattr(request.app.state, "proxy_token_mgr", None)
if mgr is not None:
return dict(mgr.bearer_header)
@@ -407,6 +428,9 @@ async def create_workstream(request: Request) -> JSONResponse:
from turnstone.mq.protocol import CreateWorkstreamMessage
auth = getattr(getattr(request, "state", None), "auth_result", None)
uid: str = getattr(auth, "user_id", "") or ""
# General pool — push to shared queue, any bridge picks it up
if node_id == "pool":
msg = CreateWorkstreamMessage(
@@ -415,6 +439,7 @@ async def create_workstream(request: Request) -> JSONResponse:
initial_message=initial_message,
skill=skill,
resume_ws=resume_ws,
user_id=uid,
)
broker.push_inbound(msg.to_json())
log.debug("Pool dispatch: correlation_id=%s name=%r", msg.correlation_id, name)
@@ -444,6 +469,7 @@ async def create_workstream(request: Request) -> JSONResponse:
initial_message=initial_message,
skill=skill,
resume_ws=resume_ws,
user_id=uid,
)
broker.push_inbound(msg.to_json(), node_id=node_id)
@@ -2748,6 +2774,16 @@ async def admin_usage(request: Request) -> JSONResponse:
group_by=group_by,
)
# Resolve user_id hex → username for display when grouped by user
if group_by == "user" and breakdown:
uid_to_name: dict[str, str] = {}
for u in storage.list_users():
uid_to_name[u["user_id"]] = u.get("username") or u["user_id"]
for row in breakdown:
raw_key = row.get("key", "")
if raw_key and raw_key in uid_to_name:
row["key"] = uid_to_name[raw_key]
return JSONResponse({"summary": summary, "breakdown": breakdown})
@@ -2792,6 +2828,16 @@ async def admin_audit(request: Request) -> JSONResponse:
until=until,
)
# Resolve user_id hex → username for display
if events:
uid_to_name: dict[str, str] = {}
for u in storage.list_users():
uid_to_name[u["user_id"]] = u.get("username") or u["user_id"]
for ev in events:
raw_uid = ev.get("user_id", "")
if raw_uid and raw_uid in uid_to_name:
ev["username"] = uid_to_name[raw_uid]
return JSONResponse({"events": events, "total": total})
+3 -1
View File
@@ -1838,7 +1838,9 @@ function _renderGovAudit(events, total) {
_relativeTime(ev.timestamp) +
"</span>" +
'<span class="admin-col admin-col-auser">' +
escapeHtml(ev.user_id ? ev.user_id.slice(0, 8) : "\u2014") +
escapeHtml(
ev.username || (ev.user_id ? ev.user_id.slice(0, 8) : "\u2014"),
) +
"</span>" +
'<span class="admin-col admin-col-aaction"><span class="' +
actionCls +
+5 -1
View File
@@ -329,18 +329,22 @@ def create_jwt(
expiry_hours: int = 24,
audience: str = "",
permissions: frozenset[str] = frozenset(),
expiry_seconds: int | None = None,
) -> str:
"""Create a signed JWT with user identity, scopes, and permissions."""
import jwt
if expiry_seconds is not None and expiry_seconds <= 0:
raise ValueError("expiry_seconds must be positive")
now = int(time.time())
ttl = expiry_seconds if expiry_seconds is not None else expiry_hours * 3600
payload: dict[str, Any] = {
"sub": user_id,
"scopes": ",".join(sorted(scopes)),
"src": source,
"iss": JWT_ISSUER,
"iat": now,
"exp": now + expiry_hours * 3600,
"exp": now + ttl,
}
if audience:
payload["aud"] = audience
+43 -23
View File
@@ -867,17 +867,34 @@ class ChatSession:
]
else:
dev_parts = [
"You are an expert software engineer. You solve problems "
"by reading code, making targeted edits, and running commands. "
"Always respond with tool calls, not just text.\n\n"
"TOOL PATTERNS:\n\n"
"Modify existing file → read_file then edit_file:\n"
" read_file(path='config.py') → "
"edit_file(path='config.py')\n\n"
"Create new file → write_file:\n"
" write_file(path='hello.py', content='...')\n\n"
"Modify multiple filesread_file then edit_file each:\n"
" read_file(path='a.py') → edit_file(path='a.py')"
"read_file(path='b.py') → edit_file(path='b.py')\n\n"
"Create new file → write_file (generate reasonable "
"content even if the request is vague):\n"
" write_file(path='hello.py', content='...')\n"
" write_file(path='README.md', "
"content='# Project\\nDescription.')\n\n"
"Create a file then run it → write_file then bash:\n"
" write_file(path='fib.py', content='...') → "
"bash(command='python fib.py')\n\n"
"Find something across files → search:\n"
" search(query='test_')\n\n"
"Plan, design, or think through an approach → create_plan:\n"
" create_plan(goal='refactor database from API')\n\n"
"Find and modify → search then read_file then edit_file:\n"
" search(query='MAX_RETRIES') → "
"read_file(path='found.py') → "
"edit_file(path='found.py')\n\n"
"Plan, think through, or strategize → plan_agent:\n"
" plan_agent(goal='refactor database layer "
"from monolith to service')\n"
" plan_agent(goal='restructure auth module')\n\n"
"Run a command, git, or tests → bash:\n"
" bash(command='git log -5')\n"
" bash(command='pytest')\n\n"
@@ -885,8 +902,9 @@ class ChatSession:
" web_fetch(url='https://example.com')\n\n"
"Search the web for information → web_search:\n"
" web_search(query='current population of Tokyo')\n\n"
"Look up documentation → man:\n"
" man(page='tar')",
"Look up command flags or documentation → man:\n"
" man(page='tar')\n"
" man(page='grep')",
]
# Tool search hint (client-side mode only — native mode needs no hint)
if self._tool_search:
@@ -2090,8 +2108,10 @@ class ChatSession:
}
elif name == "notify":
it["func_args"] = {"message": it.get("message", "")[:200]}
elif name == "task":
elif name == "task_agent":
it["func_args"] = {"prompt": it.get("prompt", "")[:200]}
elif name == "plan_agent":
it["func_args"] = {"goal": it.get("prompt", "")[:200]}
elif it.get("mcp_args"):
it["func_args"] = it["mcp_args"]
@@ -2219,7 +2239,7 @@ class ChatSession:
# feedback the plan agent re-runs and the revised plan is shown
# again, up to _MAX_PLAN_REFINEMENTS rounds.
for i, item in enumerate(items):
if item.get("func_name") != "create_plan" or item.get("error") or item.get("denied"):
if item.get("func_name") != "plan_agent" or item.get("error") or item.get("denied"):
continue
cid, output = results[i]
@@ -2344,8 +2364,8 @@ class ChatSession:
"web_fetch": self._prepare_web_fetch,
"web_search": self._prepare_web_search,
"tool_search": self._prepare_tool_search,
"task": self._prepare_task,
"create_plan": self._prepare_plan,
"task_agent": self._prepare_task,
"plan_agent": self._prepare_plan,
"memory": self._prepare_memory,
"recall": self._prepare_recall,
"notify": self._prepare_notify,
@@ -2877,8 +2897,8 @@ class ChatSession:
if not prompt:
return {
"call_id": call_id,
"func_name": "task",
"header": "\u2717 task: empty prompt",
"func_name": "task_agent",
"header": "\u2717 task_agent: empty prompt",
"preview": "",
"needs_approval": False,
"error": "Error: empty prompt",
@@ -2886,11 +2906,11 @@ class ChatSession:
preview_text = prompt[:300] + ("..." if len(prompt) > 300 else "")
return {
"call_id": call_id,
"func_name": "task",
"header": "\u2699 task (autonomous agent)",
"func_name": "task_agent",
"header": "\u2699 task_agent (autonomous agent)",
"preview": f" {DIM}{preview_text}{RESET}",
"needs_approval": True,
"approval_label": "task",
"approval_label": "task_agent",
"execute": self._exec_task,
"prompt": prompt,
}
@@ -2901,8 +2921,8 @@ class ChatSession:
if not goal:
return {
"call_id": call_id,
"func_name": "create_plan",
"header": "\u2717 create_plan: empty goal",
"func_name": "plan_agent",
"header": "\u2717 plan_agent: empty goal",
"preview": "",
"needs_approval": False,
"error": "Error: empty goal",
@@ -2910,11 +2930,11 @@ class ChatSession:
preview_text = goal[:300] + ("..." if len(goal) > 300 else "")
return {
"call_id": call_id,
"func_name": "create_plan",
"header": "\u2699 create_plan (planning agent)",
"func_name": "plan_agent",
"header": "\u2699 plan_agent (planning agent)",
"preview": f" {DIM}{preview_text}{RESET}",
"needs_approval": True,
"approval_label": "create_plan",
"approval_label": "plan_agent",
"execute": self._exec_plan,
"prompt": goal,
}
@@ -3920,7 +3940,7 @@ class ChatSession:
tool_name = tc_dict["function"]["name"]
# Guard 1: block recursive agent calls.
if tool_name in ("task", "create_plan"):
if tool_name in ("task_agent", "plan_agent"):
output = "Error: agents cannot spawn further agents"
# Guard 2: tool not in this agent's API tool list.
elif tool_name not in tool_names:
@@ -4123,7 +4143,7 @@ class ChatSession:
for i, msg in enumerate(self.messages):
if msg.get("role") == "assistant" and msg.get("tool_calls"):
for tc in msg["tool_calls"]:
if tc.get("function", {}).get("name") == "create_plan":
if tc.get("function", {}).get("name") == "plan_agent":
tc_id = tc["id"]
for j in range(i + 1, len(self.messages)):
if (
@@ -4217,7 +4237,7 @@ class ChatSession:
"id": tc_id,
"type": "function",
"function": {
"name": "create_plan",
"name": "plan_agent",
"arguments": json.dumps({"goal": original_goal}),
},
}
+5 -1
View File
@@ -19,7 +19,11 @@ def run_migrations_online() -> None:
)
with connectable.connect() as connection:
context.configure(connection=connection, target_metadata=target_metadata)
context.configure(
connection=connection,
target_metadata=target_metadata,
render_as_batch=True,
)
with context.begin_transaction():
context.run_migrations()
@@ -28,9 +28,7 @@ def upgrade() -> None:
sa.Column("updated", sa.Text, nullable=False),
sa.Column("last_accessed", sa.Text, nullable=False, server_default=""),
sa.Column("access_count", sa.Integer, nullable=False, server_default="0"),
)
op.create_unique_constraint(
"uq_smem_name_scope", "structured_memories", ["name", "scope", "scope_id"]
sa.UniqueConstraint("name", "scope", "scope_id", name="uq_smem_name_scope"),
)
op.create_index("idx_smem_type", "structured_memories", ["type"])
op.create_index("idx_smem_scope", "structured_memories", ["scope", "scope_id"])
@@ -27,56 +27,32 @@ depends_on = None
def upgrade() -> None:
# Phase 1: Skills evolution columns
op.add_column(
"prompt_templates",
sa.Column("description", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("tags", sa.Text, nullable=False, server_default="[]"),
)
op.add_column(
"prompt_templates",
sa.Column("source_url", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("version", sa.Text, nullable=False, server_default="1.0.0"),
)
op.add_column(
"prompt_templates",
sa.Column("author", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("activation", sa.Text, nullable=False, server_default="named"),
)
op.add_column(
"prompt_templates",
sa.Column("token_estimate", sa.Integer, nullable=False, server_default="0"),
)
# Phase 2: Security scanning + install provenance
op.add_column(
"prompt_templates",
sa.Column("allowed_tools", sa.Text, nullable=False, server_default="[]"),
)
op.add_column(
"prompt_templates",
sa.Column("scan_status", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("scan_report", sa.Text, nullable=False, server_default="{}"),
)
op.add_column(
"prompt_templates",
sa.Column("installed_at", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("installed_by", sa.Text, nullable=False, server_default=""),
)
# Phase 1+2: Skills evolution columns + security scanning (batch for SQLite)
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("description", sa.Text, nullable=False, server_default=""))
batch_op.add_column(sa.Column("tags", sa.Text, nullable=False, server_default="[]"))
batch_op.add_column(sa.Column("source_url", sa.Text, nullable=False, server_default=""))
batch_op.add_column(sa.Column("version", sa.Text, nullable=False, server_default="1.0.0"))
batch_op.add_column(sa.Column("author", sa.Text, nullable=False, server_default=""))
batch_op.add_column(
sa.Column("activation", sa.Text, nullable=False, server_default="named"),
)
batch_op.add_column(
sa.Column("token_estimate", sa.Integer, nullable=False, server_default="0"),
)
batch_op.add_column(
sa.Column("allowed_tools", sa.Text, nullable=False, server_default="[]"),
)
batch_op.add_column(sa.Column("scan_status", sa.Text, nullable=False, server_default=""))
batch_op.add_column(
sa.Column("scan_report", sa.Text, nullable=False, server_default="{}"),
)
batch_op.add_column(
sa.Column("installed_at", sa.Text, nullable=False, server_default=""),
)
batch_op.add_column(
sa.Column("installed_by", sa.Text, nullable=False, server_default=""),
)
# Backfill activation from is_default
op.execute("UPDATE prompt_templates SET activation = 'default' WHERE is_default = 1")
@@ -101,42 +77,26 @@ def upgrade() -> None:
)
# Phase 3: Session config columns (from workstream templates)
op.add_column(
"prompt_templates",
sa.Column("model", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
)
op.add_column(
"prompt_templates",
sa.Column("temperature", sa.Float, nullable=True),
)
op.add_column(
"prompt_templates",
sa.Column("reasoning_effort", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("max_tokens", sa.Integer, nullable=True),
)
op.add_column(
"prompt_templates",
sa.Column("token_budget", sa.Integer, nullable=False, server_default="0"),
)
op.add_column(
"prompt_templates",
sa.Column("agent_max_turns", sa.Integer, nullable=True),
)
op.add_column(
"prompt_templates",
sa.Column("notify_on_complete", sa.Text, nullable=False, server_default="{}"),
)
op.add_column(
"prompt_templates",
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
)
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("model", sa.Text, nullable=False, server_default=""))
batch_op.add_column(
sa.Column("auto_approve", sa.Integer, nullable=False, server_default="0"),
)
batch_op.add_column(sa.Column("temperature", sa.Float, nullable=True))
batch_op.add_column(
sa.Column("reasoning_effort", sa.Text, nullable=False, server_default=""),
)
batch_op.add_column(sa.Column("max_tokens", sa.Integer, nullable=True))
batch_op.add_column(
sa.Column("token_budget", sa.Integer, nullable=False, server_default="0"),
)
batch_op.add_column(sa.Column("agent_max_turns", sa.Integer, nullable=True))
batch_op.add_column(
sa.Column("notify_on_complete", sa.Text, nullable=False, server_default="{}"),
)
batch_op.add_column(
sa.Column("enabled", sa.Integer, nullable=False, server_default="1"),
)
# Skill versions — version history for skills
op.create_table(
@@ -253,24 +213,25 @@ def upgrade() -> None:
# Rename workstreams table columns: ws_template_id → skill_id,
# ws_template_version → skill_version
op.alter_column("workstreams", "ws_template_id", new_column_name="skill_id")
op.alter_column("workstreams", "ws_template_version", new_column_name="skill_version")
with op.batch_alter_table("workstreams") as batch_op:
batch_op.alter_column("ws_template_id", new_column_name="skill_id")
batch_op.alter_column("ws_template_version", new_column_name="skill_version")
# Rename scheduled_tasks.template → skill
op.alter_column("scheduled_tasks", "template", new_column_name="skill")
# Rename scheduled_tasks.template → skill, drop ws_template
with op.batch_alter_table("scheduled_tasks") as batch_op:
batch_op.alter_column("template", new_column_name="skill")
batch_op.drop_column("ws_template")
# Drop old tables
op.drop_table("workstream_template_versions")
op.drop_table("workstream_templates")
# Drop ws_template column from scheduled_tasks
op.drop_column("scheduled_tasks", "ws_template")
def downgrade() -> None:
# Reverse workstreams column renames
op.alter_column("workstreams", "skill_id", new_column_name="ws_template_id")
op.alter_column("workstreams", "skill_version", new_column_name="ws_template_version")
with op.batch_alter_table("workstreams") as batch_op:
batch_op.alter_column("skill_id", new_column_name="ws_template_id")
batch_op.alter_column("skill_version", new_column_name="ws_template_version")
# Reverse workstream_config key renames
op.execute("UPDATE workstream_config SET key = 'ws_template_id' WHERE key = 'applied_skill_id'")
@@ -283,14 +244,10 @@ def downgrade() -> None:
"WHERE key = 'applied_skill_content'"
)
# Reverse scheduled_tasks column rename: skill → template
op.alter_column("scheduled_tasks", "skill", new_column_name="template")
# Re-add ws_template column to scheduled_tasks
op.add_column(
"scheduled_tasks",
sa.Column("ws_template", sa.Text, nullable=False, server_default=""),
)
# Reverse scheduled_tasks column rename + re-add ws_template
with op.batch_alter_table("scheduled_tasks") as batch_op:
batch_op.alter_column("skill", new_column_name="template")
batch_op.add_column(sa.Column("ws_template", sa.Text, nullable=False, server_default=""))
# Recreate workstream_templates (empty — destructive migration)
op.create_table(
@@ -333,32 +290,31 @@ def downgrade() -> None:
op.drop_index("idx_skill_versions_skill_id", table_name="skill_versions")
op.drop_table("skill_versions")
# Drop session config columns from prompt_templates
op.drop_column("prompt_templates", "enabled")
op.drop_column("prompt_templates", "notify_on_complete")
op.drop_column("prompt_templates", "agent_max_turns")
op.drop_column("prompt_templates", "token_budget")
op.drop_column("prompt_templates", "max_tokens")
op.drop_column("prompt_templates", "reasoning_effort")
op.drop_column("prompt_templates", "temperature")
op.drop_column("prompt_templates", "auto_approve")
op.drop_column("prompt_templates", "model")
# Drop session config + skills evolution columns from prompt_templates
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.drop_column("enabled")
batch_op.drop_column("notify_on_complete")
batch_op.drop_column("agent_max_turns")
batch_op.drop_column("token_budget")
batch_op.drop_column("max_tokens")
batch_op.drop_column("reasoning_effort")
batch_op.drop_column("temperature")
batch_op.drop_column("auto_approve")
batch_op.drop_column("model")
batch_op.drop_column("installed_by")
batch_op.drop_column("installed_at")
batch_op.drop_column("scan_report")
batch_op.drop_column("scan_status")
batch_op.drop_column("allowed_tools")
batch_op.drop_column("token_estimate")
batch_op.drop_column("activation")
batch_op.drop_column("author")
batch_op.drop_column("version")
batch_op.drop_column("source_url")
batch_op.drop_column("tags")
batch_op.drop_column("description")
# Drop skill resources
op.drop_index("idx_skill_resources_skill_path", table_name="skill_resources")
op.drop_index("idx_skill_resources_skill_id", table_name="skill_resources")
op.drop_table("skill_resources")
# Drop skills evolution columns
op.drop_column("prompt_templates", "installed_by")
op.drop_column("prompt_templates", "installed_at")
op.drop_column("prompt_templates", "scan_report")
op.drop_column("prompt_templates", "scan_status")
op.drop_column("prompt_templates", "allowed_tools")
op.drop_column("prompt_templates", "token_estimate")
op.drop_column("prompt_templates", "activation")
op.drop_column("prompt_templates", "author")
op.drop_column("prompt_templates", "version")
op.drop_column("prompt_templates", "source_url")
op.drop_column("prompt_templates", "tags")
op.drop_column("prompt_templates", "description")
@@ -33,14 +33,13 @@ def upgrade() -> None:
op.create_index("ix_oa_created", "output_assessments", ["created"])
op.create_index("ix_oa_risk", "output_assessments", ["risk_level"])
op.add_column(
"prompt_templates",
sa.Column("scan_version", sa.Text, nullable=False, server_default=""),
)
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("scan_version", sa.Text, nullable=False, server_default=""))
def downgrade() -> None:
op.drop_column("prompt_templates", "scan_version")
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.drop_column("scan_version")
op.drop_index("ix_oa_risk", table_name="output_assessments")
op.drop_index("ix_oa_created", table_name="output_assessments")
op.drop_index("ix_oa_ws_id", table_name="output_assessments")
@@ -19,16 +19,14 @@ depends_on = None
def upgrade() -> None:
op.add_column(
"prompt_templates",
sa.Column("license", sa.Text, nullable=False, server_default=""),
)
op.add_column(
"prompt_templates",
sa.Column("compatibility", sa.Text, nullable=False, server_default=""),
)
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("license", sa.Text, nullable=False, server_default=""))
batch_op.add_column(
sa.Column("compatibility", sa.Text, nullable=False, server_default=""),
)
def downgrade() -> None:
op.drop_column("prompt_templates", "compatibility")
op.drop_column("prompt_templates", "license")
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.drop_column("compatibility")
batch_op.drop_column("license")
@@ -19,11 +19,10 @@ depends_on = None
def upgrade() -> None:
op.add_column(
"prompt_templates",
sa.Column("priority", sa.Integer, nullable=False, server_default="0"),
)
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.add_column(sa.Column("priority", sa.Integer, nullable=False, server_default="0"))
def downgrade() -> None:
op.drop_column("prompt_templates", "priority")
with op.batch_alter_table("prompt_templates") as batch_op:
batch_op.drop_column("priority")
+1 -1
View File
@@ -63,7 +63,7 @@ class DuckDuckGoClient:
self._timeout = timeout
def search(self, query: str, max_results: int = 5, **kwargs: Any) -> str:
from ddgs import DDGS # type: ignore[import-not-found]
from ddgs import DDGS
with DDGS(timeout=int(self._timeout)) as ddgs:
raw = list(ddgs.text(query, max_results=max_results))
+1530 -193
View File
File diff suppressed because it is too large Load Diff
+4
View File
@@ -423,6 +423,7 @@ class Bridge:
model=model,
resume_ws=resume_ws,
skill=skill,
user_id=user_id,
)
# Send initial_message only when no workstream was actually resumed.
# Use the server's `resumed` response (not just the intent) so that
@@ -491,6 +492,7 @@ class Bridge:
model: str = "",
resume_ws: str = "",
skill: str = "",
user_id: str = "",
) -> tuple[str, bool]:
"""Create a workstream on the server. Returns (ws_id, resumed)."""
try:
@@ -501,6 +503,8 @@ class Bridge:
payload["resume_ws"] = resume_ws
if skill:
payload["skill"] = skill
if user_id:
payload["user_id"] = user_id
resp = self._http.post(
"/v1/api/workstreams/new",
json=payload,
+12
View File
@@ -1280,6 +1280,18 @@ async def create_workstream(request: Request) -> JSONResponse:
skip: bool = request.app.state.skip_permissions
auth = getattr(getattr(request, "state", None), "auth_result", None)
uid: str = getattr(auth, "user_id", "") or ""
# Trusted services (bridge, console) may forward the real user_id in the
# request body when creating workstreams on behalf of a user. Only service
# identities are trusted — end-user tokens (including console-proxy tokens
# that carry the real user's identity) must not override user_id.
trusted_sources = {"bridge", "console"}
if (
body.get("user_id")
and isinstance(body["user_id"], str)
and auth is not None
and auth.token_source in trusted_sources
):
uid = body["user_id"]
body_skill = body.get("skill", "")
resume_ws_id = body.get("resume_ws", "")
# Resolve skill — applies content + session config (model, temperature, etc.)
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "bash",
"description": "Execute a bash command and return stdout + stderr. Use this tool freely for any task: checking time, reading files, running programs, system info, etc.",
"description": "Execute a bash command and return stdout + stderr. Use for running programs, git, tests, system commands, installing packages, etc. Environment questions ('What Python version?', 'Is X installed?') are tool-use tasks — e.g. bash(command='python --version'). For file creation use write_file instead; for man pages use man instead.",
"parameters": {
"type": "object",
"properties": {
-15
View File
@@ -1,15 +0,0 @@
{
"name": "create_plan",
"description": "Create a structured plan before taking action. An autonomous agent explores the available context, identifies what needs to change, and writes a step-by-step plan. Call this tool when the user asks to plan, design, or think through an approach, or when a task is complex, touches multiple areas, or has unclear scope.",
"parameters": {
"type": "object",
"properties": {
"goal": {
"type": "string",
"description": "The goal and scope of the plan, including any constraints."
}
},
"required": ["goal"]
},
"primary_key": "goal"
}
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "edit_file",
"description": "Replace an exact string in a file with new content. Fails if old_string is not found or matches multiple locations (use near_line to disambiguate). Requires read_file on the same path first.",
"description": "Replace an exact string in a file with new content. Fails if old_string is not found or matches multiple locations (use near_line to disambiguate). Requires read_file on the same path first — it will fail without this. Use for any modification to existing files: changing values, renaming, inserting code, adding docstrings. Prefer this over write_file for partial modifications. For multi-file edits when filenames are known, go directly to read_file + edit_file for each file — no need to search first. For generated content (docstrings, type hints), call edit_file with your best-effort content inline — e.g. edit_file(old_string='def process(...):', new_string='def process(...):\\n \"\"\"Process input data.\"\"\"'). When asked to change something in a file, follow through with edit_file after reading — reading alone is not enough.",
"parameters": {
"type": "object",
"properties": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "man",
"description": "Read a man page. Use this instead of bash('man ...') or web_search. Returns the full formatted manual entry.",
"description": "Read a man page. Questions about command flags, options, or usage are tool-use tasks — use man, not memory. Use this to look up, check, or explain a command's flags, options, or usage — e.g. 'What does --xattrs do in tar?' → man(page='tar'). Use this instead of bash('man ...') or web_search for command documentation.",
"parameters": {
"type": "object",
"properties": {
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "math",
"description": "Execute Python code for math/computation. Code MUST use print() to produce output. Available: sympy, numpy, scipy (with scipy.special, scipy.optimize, scipy.integrate, scipy.linalg), math, fractions, itertools, functools, collections, decimal, operator, random, re, string. Common sympy names (symbols, solve, simplify, expand, factor, sqrt, Rational, Matrix, integrate, diff, etc.) are pre-imported. Example: x = symbols('x'); print(solve(x**2 - 4, x))",
"description": "Execute Python code for math/computation. Use this for any arithmetic, algebra, or numerical task — e.g. math(code='print(2**64 - 1)'). Code must use print() to produce output. Available: sympy, numpy, scipy (with scipy.special, scipy.optimize, scipy.integrate, scipy.linalg), math, fractions, itertools, functools, collections, decimal, operator, random, re, string. Common sympy names (symbols, solve, simplify, expand, factor, sqrt, Rational, Matrix, integrate, diff, etc.) are pre-imported. Example: x = symbols('x'); print(solve(x**2 - 4, x))",
"parameters": {
"type": "object",
"properties": {
+15
View File
@@ -0,0 +1,15 @@
{
"name": "plan_agent",
"description": "Delegate planning to a sub-agent. The agent autonomously explores the codebase, gathers context, and writes a step-by-step plan — just pass the goal directly. Use when asked to plan, design, think through, or strategize. Not for direct code changes like 'add a docstring' or 'fix a bug' — use read_file+edit_file for those.",
"parameters": {
"type": "object",
"properties": {
"goal": {
"type": "string",
"description": "The goal and scope of the plan, including any constraints."
}
},
"required": ["goal"]
},
"primary_key": "goal"
}

Some files were not shown because too many files have changed in this diff Show More