turnstone/docs/eval.md

Evaluation and Prompt Optimization (turnstone-eval)

turnstone-eval is the evaluation and prompt optimization system for turnstone. It runs test cases against the LLM, scores tool call sequences against expected actions, and optionally uses a multi-agent pipeline to optimize the developer prompt and tool descriptions.

Source: turnstone/eval.py

---

Overview

The system uses UCB tree search to explore prompt variants:

1. Maintain an evolution tree of prompt variants, starting from the initial prompt.
2. Each iteration, UCB1 selects the most promising node to evaluate.
3. Run each test case N times against the selected prompt.
4. Score each run by comparing the actual tool call sequence to expected actions.
5. If not all tests pass, run a three-phase optimization pipeline:
   • Phase 1: Analyst diagnoses semantic failure patterns
   • Phase 2: Tool optimizer adjusts tool descriptions (when --optimize-tools)
   • Phase 3: Prompt optimizer proposes a child variant (when not --optimize-tools)
6. Add the child to the tree and repeat until all tests pass or max iterations reached.

This approach (inspired by Learning to Self-Evolve) prevents irrecoverable collapse from bad edits — UCB naturally backtracks to high-scoring ancestors instead of following a linear chain.

When optimization is disabled (--no-optimize), only steps 2-4 execute (a single iteration evaluating the root node).

---

Test Case Format

Test suites are JSON files with this structure:

{
  "defaults": {
    "n_runs": 3
  },
  "cases": [
    {
      "id": "test_name",
      "user_prompt": "the prompt to send to the model",
      "setup": {
        "files": {
          "filename.py": "file content here",
          "src/utils.py": "another file"
        }
      },
      "expected_actions": [
        {"tool": "read_file", "args": {"path": "filename.py"}},
        {"tool": "bash", "args_pattern": {"command": "python.*test"}},
        {"tool": "edit_file"}
      ],
      "match_mode": "ordered_subset",
      "max_turns": 10,
      "n_runs": 5
    }
  ]
}

Fields

Field	Required	Default	Description
id	yes	--	Unique test case identifier.
user_prompt	yes	--	The message sent to the model.
setup.files	no	{}	Files to create in the temp directory before running. Keys are relative paths, values are file content.
expected_actions	no	[]	List of expected tool calls to match against.
match_mode	no	"ordered_subset"	How to match actual vs expected actions (see Scoring).
max_turns	no	10	Maximum conversation turns before stopping.
n_runs	no	suite default or 3	Per-case override for number of runs.
holdout	no	false	If true, this case is evaluated but excluded from optimizer feedback. Used to measure progress without overfitting.

Expected Action Specs

Each entry in expected_actions can contain:

• tool (required): The tool name to match (e.g. "read_file", "bash").
• args: Exact key-value matching. Each key in args must exist in the actual call with the same string value.
• args_pattern: Regex key-value matching. Each key's value is a regex pattern tested against the actual argument value.
• If neither args nor args_pattern is specified, only the tool name is matched.

---

Scoring

Scoring is handled by score_run(), which compares a run's tool call log against the expected actions.

Match Modes

Mode	Description
exact	Tool calls must match expected actions in exact order and exact count. Extra or missing calls cause failure.
ordered_subset	Expected actions must appear in order within the actual tool log, but extra calls between them are allowed. This is the default.
subset	Expected actions must all appear somewhere in the tool log, in any order. Each actual call can only match one expected action.
contains_any	Passes if at least one expected action appears anywhere in the tool log.

Action Matching (_match_action)

A single actual tool call matches an expected action when:

1. The tool names are equal.
2. If args is specified: every key in args must exist in the actual call's arguments with the same string value (partial key matching -- extra actual args are ignored).
3. If args_pattern is specified: every key's regex pattern must match the corresponding actual argument value via re.search().
4. If the actual args contain only _raw (unparseable JSON fallback), the action matches only when no args or args_pattern is expected.

Score Calculation

• Score = number of matched expected actions / total expected actions.
• Pass = score equals 1.0 (all expected actions matched).
• The return dict includes: pass, score, matched (indices), unmatched (indices), extra_tools, and detail (human-readable summary).

JSON Dump Detection

When a run fails and the model's final text content contains JSON that looks like a tool call (keys like "tool", "command", "path"), the run is flagged with json_dump: true. This indicates the model tried to call a tool but emitted JSON as text instead of using the function-calling interface.

---

HeadlessSession

HeadlessSession extends ChatSession for headless evaluation. It provides deterministic, non-interactive execution suitable for automated testing.

Differences from ChatSession

Aspect	ChatSession	HeadlessSession
Streaming	Streaming API	Non-streaming (stream=False)
Tool approval	User confirmation	auto_approve = True
UI	Terminal/Web UI	NullUI (discards output)
Stdout	Normal	Suppressed during execution
Tool logging	Display only	Structured tool_call_log
System prompt	Built-in developer prompt	Overridable via constructor
Cancellation	N/A	_cancelled event for timeout cleanup

NullUI

A minimal UI adapter that satisfies the SessionUI protocol by discarding all output. approve_tools() always returns (True, None).

send_headless()

def send_headless(
    self,
    user_input: str,
    max_turns: int = 10,
    verbose: bool = False,
    log_prefix: str = "",
) -> list[dict]:

Runs a complete multi-turn conversation:

1. Appends the user message.
2. Checks _cancelled event — stops if set (timeout cleanup).
3. Calls the model API (non-streaming).
4. If tool calls are returned, executes them (with stdout suppressed) and logs each call to self.tool_call_log.
5. Repeats up to max_turns or until the model responds without tool calls.
6. Returns the tool call log: list of dicts with keys tool, args, result (truncated to 500 chars), and turn.

Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.

Timeout and Cancellation

Each test runs in a ThreadPoolExecutor(max_workers=1) with a per-test timeout (--test-timeout). Each attempt gets its own OpenAI client with a matching httpx read timeout. On timeout, three layers of defense prevent zombie connections:

1. httpx timeout: Per-request read timeout aborts the HTTP call and releases the server slot.
2. _cancelled event: Prevents the orphan thread from starting new turns.
3. run_client.close(): Closes the connection pool to abort any in-flight request.

Retry Logic

send_headless() is called inside _run_single_test() with retry logic: 3 attempts with exponential backoff (sleep 2^attempt seconds) on any exception. TimeoutError is re-raised immediately (no retry).

---

Test Execution

Each test case runs in isolation:

1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A per-attempt OpenAI client is created with httpx timeout matching --test-timeout.
5. A new HeadlessSession is created with the current developer prompt.
6. send_headless() runs the user prompt through the conversation loop.
7. The tool log is scored against expected actions.
8. The temp directory is cleaned up.

The memory database is also isolated per test (an ephemeral SQLite database in the temp directory) so tests do not pollute each other or the user's real memory store.

Parallel Execution

With --parallel N (N > 1), tests run in a ProcessPoolExecutor with N workers. Each subprocess creates its own OpenAI client. This is suitable for remote API endpoints but will overwhelm local inference servers. The default (--parallel 1) runs tests serially.

---

Model Roles

The eval pipeline uses up to five separate model roles, each independently configurable. All roles inherit from the test model by default, with a cascade chain:

test model (--base-url, --model)
  └─ optimizer (--optimizer-*)
       ├─ observer (--observer-*)
       ├─ analyst (--analyst-*)
       ├─ diversifier (--diversifier-*)
       └─ tool optimizer (--tool-optimizer-*)

Role	Purpose	When it runs
Test	The model being evaluated	Every iteration
Analyst	Diagnoses semantic failure patterns with tool use	When pass rate < 100%
Optimizer	Rewrites the developer prompt	Every iteration (unless --optimize-tools)
Tool optimizer	Rewrites tool descriptions	When --optimize-tools is set
Observer	Tunes the optimizer's strategy	Every 3 iterations
Diversifier	Generates prompt paraphrases	Once before the loop (when --diversify N)

Typical setup: local model for test, Opus for analyst, Sonnet for optimizer/observer/diversifier.

---

Optimization Pipeline

Flow

for iteration in 0..max_iterations:
    1. UCB select → pick the most promising tree node
    2. Run all test cases n_runs times with selected node's prompt
    3. Update node score (rolling mean) and visit count
    4. Save intermediate results + tree state to JSON
    5. If all tests pass → stop
    6. Phase 1: Analyst diagnoses semantic failure patterns
    7. Phase 2 (--optimize-tools only): Tool optimizer adjusts descriptions
    8. Phase 3 (default only): Prompt optimizer proposes new prompt
    9. Every 3 iterations: Observer tunes the optimizer's strategy
   10. Add child node to tree (if prompt or tools changed)

Phase 1: Analyst (_run_analyst)

A multi-turn agent with a bash tool for computing statistics. It receives per-case results with failure classifications and produces a structured diagnosis:

• Failure patterns: Shared root causes across failing cases
• Success/failure contrast: What distinguishes passing from failing cases
• Consistency signals: Systematic (0%), flaky (1-79%), marginal (80-99%)
• Recommended fixes: Priority-ordered patterns/examples to add or adjust

The analyst is instructed to frame fixes as patterns and examples, not imperative rules — this feeds cleaner signal to the optimizer.

In --optimize-tools mode, the analyst receives the current tool descriptions (with any overrides applied) and focuses on tool confusion and description issues rather than system prompt patterns.

Phase 2: Tool Optimizer (_propose_tool_overrides)

Runs when --optimize-tools is set. Receives the current tool descriptions, confusion failures (where the model picked the wrong tool), and the analyst's diagnosis. Returns a JSON override dict that modifies tool descriptions.

Overrides are validated against known tool names — only description and parameters changes are accepted (no tool renaming at eval time).

After each iteration, changed descriptions are logged as old → new diffs for easy visual inspection.

Phase 3: Prompt Optimizer (_propose_prompt_modification)

Skipped in --optimize-tools mode. Receives the current prompt, test results with per-case pass rates and deltas from the parent node, and the analyst's diagnosis. Returns a rewritten prompt.

The optimizer is instructed to prefer patterns over rules — concrete tool chain examples teach better than imperative directives like "ALWAYS" or "NEVER." If the current prompt contains rule-heavy language, the optimizer is guided to replace it with examples.

Two Optimization Surfaces

The system supports alternating between two optimization surfaces:

1. System prompt optimization (default): Freeze tool descriptions, optimize the developer prompt. Run until scores plateau.
2. Tool description optimization (--optimize-tools): Freeze the system prompt, optimize tool descriptions only. Run until scores plateau.

Each surface lifts the floor for the other — tool description improvements may unlock system prompt gains that weren't reachable before, and vice versa.

Observer (_observe_and_update_optimizer)

Every 3 iterations, a meta-level observer reviews the optimizer's strategy:

• Analyzes iteration history: score trends, regressions, prompt length changes, and diffs between iterations.
• Detects whether the optimizer is producing rule-heavy or pattern-based output.
• Rewrites the optimizer's own system prompt to correct course.
• Rejects degenerate outputs (over 200% of input length).

Prompt Diversification

When --diversify N is set, the diversifier generates N paraphrased variants of each test case's user prompt before the optimization loop. Each run cycles through variants (round-robin), testing robustness across phrasings.

Variants can be cached back to the test suite JSON with --save-variants, and auto-loaded on subsequent runs even without --diversify.

---

Evolution Tree

The optimization maintains a tree of prompt variants (EvolutionNode), where each node stores its prompt text, tool overrides, aggregated score, and visit count. The root node (ID 0) contains the initial prompt.

UCB1 selection: Each iteration picks the node with the highest Upper Confidence Bound score: R_bar + C * sqrt(ln(N) / v), where R_bar is the node's mean score, N is total visits across all nodes, v is the node's visit count, and C is the exploration constant (--explore-constant, default sqrt(2)). Unvisited nodes are always selected first.

Holdout Cases

Test cases with "holdout": true are evaluated every iteration but excluded from the optimizer's feedback. This prevents the optimizer from overfitting to specific test cases. Node scores are computed from holdout cases only (when present). If fewer than 2 non-holdout cases remain, holdout is disabled.

Improvement-Based Feedback

The optimizer sees delta scores (delta=+20%) alongside absolute pass rates, showing how each case improved relative to the parent node's evaluation. This provides a cleaner signal than absolute scores alone — the optimizer can distinguish beneficial edits from harmful ones regardless of starting point.

---

Result Persistence

After each iteration, results are written to the output JSON file. The structure is:

{
  "meta": {
    "model": "model-name",
    "base_url": "http://localhost:8000/v1",
    "optimizer_model": "claude-opus-4-6",
    "observer_model": "claude-opus-4-6",
    "started": "2025-01-01T00:00:00",
    "test_suite": "tests.json",
    "n_runs_default": 3,
    "explore_constant": 1.414,
    "holdout_ids": [],
    "diversify": 10,
    "prompt_variants": {"case_id": ["variant1", "variant2"]}
  },
  "iterations": [
    {
      "iteration": 0,
      "prompt": "the developer prompt used",
      "prompt_diff": null,
      "optimizer_system": "the optimizer system prompt",
      "analyst": "analyst diagnosis output",
      "tool_overrides": {"bash": {"description": "..."}},
      "timestamp": "2025-01-01T00:01:00",
      "tree_node_id": 0,
      "tree_child_id": 1,
      "cases": {
        "test_name": {
          "runs": [
            {
              "pass": true,
              "score": 1.0,
              "matched": [0, 1],
              "unmatched": [],
              "extra_tools": [],
              "detail": "Ordered subset: 2/2",
              "tool_sequence": ["read_file", "edit_file"],
              "tool_args": [{"read_file": {"path": "f.py"}}, ...],
              "elapsed": 3.2
            }
          ],
          "pass_rate": 1.0,
          "avg_score": 1.0
        }
      },
      "aggregate": {
        "total_cases": 5,
        "total_runs": 15,
        "overall_pass_rate": 0.8,
        "overall_avg_score": 0.87,
        "json_dumps": 0,
        "per_case_pass_rates": {"test_name": 1.0}
      }
    }
  ],
  "tree": [
    {
      "node_id": 0,
      "parent_id": null,
      "prompt": "initial prompt",
      "tool_overrides": {},
      "score": 0.85,
      "visit_count": 3,
      "children": [1, 2],
      "iteration": 0
    }
  ]
}

---

CLI Usage

The entry point is turnstone-eval (installed as a console script) or python -m turnstone.eval.

turnstone-eval tests.json                          # evaluate + optimize
turnstone-eval tests.json --no-optimize            # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt      # start from a custom prompt
turnstone-eval tests.json --optimize-tools         # optimize tool descriptions only
turnstone-eval tests.json --diversify 10           # test with prompt variants
turnstone-eval tests.json -v                       # verbose per-turn logging

Multi-model setup (local test model, cloud optimizer)

turnstone-eval tests.json \
  --base-url http://localhost:8000/v1 \
  --optimizer-base-url https://api.anthropic.com \
  --optimizer-model claude-sonnet-4-6 \
  --analyst-model claude-opus-4-6

All Options

Flag	Default	Description
test_file	(positional, required)	Path to test cases JSON file.
--base-url	http://localhost:8000/v1	API base URL for the test model.
--model	auto-detect	Model name. Auto-detected from the API if not specified.
--prompt	turnstone built-in prompt	Path to initial prompt text file.
--n-runs	from tests.json or 3	Number of runs per test case.
--max-iter	5	Maximum optimization iterations.
--no-optimize	false	Run evaluation only (sets max-iter to 1).
--temperature	0.7	Sampling temperature.
--max-tokens	32768	Max completion tokens.
--reasoning-effort	medium	Reasoning effort: low, medium, or high.
--context-window	131072	Context window size.
--output	eval_results.json	Output results file path.
-v, --verbose	false	Show detailed per-turn logging.
--explore-constant	1.414 (sqrt(2))	UCB exploration constant C.
--test-timeout	300	Per-test timeout in seconds.
--suite-timeout	0 (unlimited)	Total suite timeout in seconds.
--no-fast-fail	false	Disable early termination on all-zero initial runs.
--parallel	1 (serial)	Parallel workers (0=auto, N=use N workers).
--optimizer-model	same as --model	Model for prompt optimization.
--optimizer-base-url	same as --base-url	Base URL for optimizer model.
--observer-model	same as optimizer	Model for meta-optimization (observer).
--observer-base-url	same as optimizer	Base URL for observer model.
--analyst-model	same as optimizer	Model for failure analysis.
--analyst-base-url	same as optimizer	Base URL for analyst model.
--diversify	0 (disabled)	Generate N prompt variants per test case.
--diversifier-model	same as optimizer	Model for prompt diversification.
--diversifier-base-url	same as optimizer	Base URL for diversifier model.
--save-variants	false	Save generated variants back to test suite JSON.
--optimize-tools	false	Optimize tool descriptions only (freeze system prompt).
--tool-optimizer-model	same as optimizer	Model for tool description optimization.
--tool-optimizer-base-url	same as optimizer	Base URL for tool optimizer model.
--save-tools	false	Write optimized tool descriptions back to turnstone/tools/*.json.

Precedence for n_runs

The number of runs per test case is resolved in this order:

1. Per-case n_runs field in the test case definition.
2. CLI --n-runs argument (if provided).
3. Suite-level defaults.n_runs in the test JSON file.
4. Code default: 3.