ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity
Files
0d6252dd7d5cde7f9c36c727dc8032b70202a853
turnstone/docs/eval.md
T

338 lines
12 KiB
Markdown
Raw Blame History

# Evaluation and Prompt Optimization (turnstone-eval)
`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
runs test cases against the LLM, scores tool call sequences against expected
actions, and optionally uses the model to self-optimize the developer prompt.
Source: `turnstone/eval.py`
---
## Overview
The system works in an iterative loop:
1. Run each test case N times against the current developer prompt.
2. Score each run by comparing the actual tool call sequence to expected actions.
3. If not all tests pass, use the model to rewrite the prompt based on failures.
4. Repeat until all tests pass or max iterations are reached.
When optimization is disabled (`--no-optimize`), only step 1 and 2 execute
(a single iteration).
---
## Test Case Format
Test suites are JSON files with this structure:
```json
{
"defaults": {
"n_runs": 3
},
"cases": [
{
"id": "test_name",
"user_prompt": "the prompt to send to the model",
"setup": {
"files": {
"filename.py": "file content here",
"src/utils.py": "another file"
}
},
"expected_actions": [
{"tool": "read_file", "args": {"path": "filename.py"}},
{"tool": "bash", "args_pattern": {"command": "python.*test"}},
{"tool": "edit_file"}
],
"match_mode": "ordered_subset",
"max_turns": 10,
"n_runs": 5
}
]
}
```
### Fields
| Field | Required | Default | Description |
|--------------------|----------|--------------------|-------------|
| `id` | yes | -- | Unique test case identifier. |
| `user_prompt` | yes | -- | The message sent to the model. |
| `setup.files` | no | `{}` | Files to create in the temp directory before running. Keys are relative paths, values are file content. |
| `expected_actions` | no | `[]` | List of expected tool calls to match against. |
| `match_mode` | no | `"ordered_subset"` | How to match actual vs expected actions (see Scoring). |
| `max_turns` | no | `10` | Maximum conversation turns before stopping. |
| `n_runs` | no | suite default or 3 | Per-case override for number of runs. |
### Expected Action Specs
Each entry in `expected_actions` can contain:
- `tool` (required): The tool name to match (e.g. `"read_file"`, `"bash"`).
- `args`: Exact key-value matching. Each key in `args` must exist in the actual call with the same string value.
- `args_pattern`: Regex key-value matching. Each key's value is a regex pattern tested against the actual argument value.
- If neither `args` nor `args_pattern` is specified, only the tool name is matched.
---
## Scoring
Scoring is handled by `score_run()`, which compares a run's tool call log
against the expected actions.
### Match Modes
| Mode | Description |
|-------------------|-------------|
| `exact` | Tool calls must match expected actions in exact order and exact count. Extra or missing calls cause failure. |
| `ordered_subset` | Expected actions must appear in order within the actual tool log, but extra calls between them are allowed. This is the default. |
| `subset` | Expected actions must all appear somewhere in the tool log, in any order. Each actual call can only match one expected action. |
| `contains_any` | Passes if at least one expected action appears anywhere in the tool log. |
### Action Matching (`_match_action`)
A single actual tool call matches an expected action when:
1. The tool names are equal.
2. If `args` is specified: every key in `args` must exist in the actual call's
arguments with the same string value (partial key matching -- extra actual
args are ignored).
3. If `args_pattern` is specified: every key's regex pattern must match the
corresponding actual argument value via `re.search()`.
4. If the actual args contain only `_raw` (unparseable JSON fallback), the
action matches only when no `args` or `args_pattern` is expected.
### Score Calculation
- **Score** = number of matched expected actions / total expected actions.
- **Pass** = score equals 1.0 (all expected actions matched).
- The return dict includes: `pass`, `score`, `matched` (indices), `unmatched`
(indices), `extra_tools`, and `detail` (human-readable summary).
### JSON Dump Detection
When a run fails and the model's final text content contains JSON that looks
like a tool call (keys like `"tool"`, `"command"`, `"path"`), the run is
flagged with `json_dump: true`. This indicates the model tried to call a tool
but emitted JSON as text instead of using the function-calling interface.
---
## HeadlessSession
`HeadlessSession` extends `ChatSession` for headless evaluation. It provides
deterministic, non-interactive execution suitable for automated testing.
### Differences from ChatSession
| Aspect | ChatSession | HeadlessSession |
|-----------------|-------------------------|----------------------------|
| Streaming | Streaming API | Non-streaming (`stream=False`) |
| Tool approval | User confirmation | `auto_approve = True` |
| UI | Terminal/Web UI | `NullUI` (discards output) |
| Stdout | Normal | Suppressed during execution |
| Tool logging | Display only | Structured `tool_call_log` |
| System prompt | Built-in developer prompt | Overridable via constructor |
### NullUI
A minimal UI adapter that satisfies the `SessionUI` protocol by discarding
all output. `approve_tools()` always returns `(True, None)`.
### send_headless()
```python
def send_headless(
self,
user_input: str,
max_turns: int = 10,
verbose: bool = False,
log_prefix: str = "",
) -> list[dict]:
```
Runs a complete multi-turn conversation:
1. Appends the user message.
2. Calls the model API (non-streaming).
3. If tool calls are returned, executes them (with stdout suppressed) and
logs each call to `self.tool_call_log`.
4. Repeats up to `max_turns` or until the model responds without tool calls.
5. Returns the tool call log: list of dicts with keys `tool`, `args`,
`result` (truncated to 500 chars), and `turn`.
Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.
### Retry Logic
`send_headless()` is called inside `_run_single_test()` with retry logic:
3 attempts with exponential backoff (sleep `2^attempt` seconds) on any
exception. This prevents transient API errors from poisoning eval scores.
---
## Test Execution
Each test case runs in isolation:
1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A new `HeadlessSession` is created with the current developer prompt.
5. `send_headless()` runs the user prompt through the conversation loop.
6. The tool log is scored against expected actions.
7. The temp directory is cleaned up.
The memory database is also isolated per test (an ephemeral SQLite database
in the temp directory) so tests do not pollute each other or the user's
real memory store.
---
## Optimization Loop
`run_optimization()` is the main entry point for iterative prompt optimization.
### Flow
```
for iteration in 0..max_iterations:
1. Run all test cases n_runs times with current prompt
2. Score and aggregate results
3. Save intermediate results to JSON
4. If all tests pass -> stop
5. Every 3 iterations (at iteration 2, 5, 8, ...):
-> Observer reviews optimizer strategy
-> Reset prompt to best-performing iteration
6. Propose new prompt via optimizer model call
7. If prompt unchanged -> stop
8. Continue with new prompt
```
### Prompt Proposal (`_propose_prompt_modification`)
Uses the model to rewrite the developer prompt based on test results:
- **Input**: Current prompt, test case definitions, per-case results with
actual vs expected tool sequences, and a history of the last 3 iterations.
- **Optimizer system prompt** (`OPTIMIZER_SYSTEM`): Instructs the model to
act as a text rewriter. Key guidance includes:
- Address critical failure modes (text-only responses, write_file vs edit_file,
unnecessary search before create, missing plan calls).
- Preserve phrasing that drives 100% pass rate on passing tests.
- Use direct imperative style with concrete tool call examples.
- Stay within 130% of original prompt length.
- **Output**: The rewritten prompt text (stripped of reasoning tags and code fences).
### Observer System (`_observe_and_update_optimizer`)
Every 3 iterations, a meta-level "observer" reviews the optimizer's strategy:
- Analyzes the iteration history: score trends, regressions, prompt length changes,
and diffs between iterations.
- Summarizes the optimizer's behavioral patterns (list style, header usage, length).
- Uses `OBSERVER_SYSTEM` to rewrite the optimizer's own system prompt.
- Rejects degenerate outputs (over 200% of input length).
- After updating the optimizer prompt, resets the developer prompt to the
best-performing iteration so far.
This two-level optimization (optimizer + observer) helps the system escape
local minima and adjust its rewriting strategy.
### Result Persistence
After each iteration, results are written to the output JSON file. The
structure is:
```json
{
"meta": {
"model": "model-name",
"base_url": "http://localhost:8000/v1",
"started": "2025-01-01T00:00:00",
"test_suite": "tests.json",
"n_runs_default": 3
},
"iterations": [
{
"iteration": 0,
"prompt": "the developer prompt used",
"prompt_diff": null,
"optimizer_system": "the optimizer system prompt",
"timestamp": "2025-01-01T00:01:00",
"cases": {
"test_name": {
"runs": [
{
"pass": true,
"score": 1.0,
"matched": [0, 1],
"unmatched": [],
"extra_tools": [],
"detail": "Ordered subset: 2/2",
"tool_sequence": ["read_file", "edit_file"],
"tool_args": [{"read_file": {"path": "f.py"}}, ...],
"elapsed": 3.2
}
],
"pass_rate": 1.0,
"avg_score": 1.0
}
},
"aggregate": {
"total_cases": 5,
"total_runs": 15,
"overall_pass_rate": 0.8,
"overall_avg_score": 0.87,
"json_dumps": 0,
"per_case_pass_rates": {"test_name": 1.0, ...}
}
}
]
}
```
---
## CLI Usage
The entry point is `turnstone-eval` (installed as a console script) or
`python -m turnstone.eval`.
```
turnstone-eval tests.json # evaluate + optimize
turnstone-eval tests.json --no-optimize # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt # start from a custom prompt
turnstone-eval tests.json -v # verbose per-turn logging
```
### All Options
| Flag | Default | Description |
|---------------------|-------------------------------|-------------|
| `test_file` | (positional, required) | Path to test cases JSON file. |
| `--base-url` | `http://localhost:8000/v1` | API base URL. |
| `--model` | auto-detect | Model name. Auto-detected from the API if not specified. |
| `--prompt` | turnstone built-in prompt | Path to initial prompt text file. |
| `--n-runs` | from tests.json or 3 | Number of runs per test case. |
| `--max-iter` | 5 | Maximum optimization iterations. |
| `--no-optimize` | false | Run evaluation only (sets max-iter to 1). |
| `--temperature` | 0.7 | Sampling temperature. |
| `--max-tokens` | 32768 | Max completion tokens. |
| `--reasoning-effort` | `medium` | Reasoning effort: `low`, `medium`, or `high`. |
| `--context-window` | 131072 | Context window size. |
| `--output` | `eval_results.json` | Output results file path. |
| `-v`, `--verbose` | false | Show detailed per-turn logging (API calls, tool args, results). |
### Precedence for n_runs
The number of runs per test case is resolved in this order:
1. Per-case `n_runs` field in the test case definition.
2. CLI `--n-runs` argument (if provided).
3. Suite-level `defaults.n_runs` in the test JSON file.
4. Code default: 3.
Reference in New Issue View Git Blame Copy Permalink