turnstone/docs/eval.md

Evaluation and Prompt Optimization (turnstone-eval)

turnstone-eval is the evaluation and prompt optimization system for turnstone. It runs test cases against the LLM, scores tool call sequences against expected actions, and optionally uses the model to self-optimize the developer prompt.

Source: turnstone/eval.py

---

Overview

The system uses UCB tree search to explore prompt variants:

1. Maintain an evolution tree of prompt variants, starting from the initial prompt.
2. Each iteration, UCB1 selects the most promising node to evaluate.
3. Run each test case N times against the selected prompt.
4. Score each run by comparing the actual tool call sequence to expected actions.
5. If not all tests pass, use the optimizer model to propose a child variant.
6. Add the child to the tree and repeat until all tests pass or max iterations reached.

This approach (inspired by Learning to Self-Evolve) prevents irrecoverable collapse from bad edits — UCB naturally backtracks to high-scoring ancestors instead of following a linear chain.

When optimization is disabled (--no-optimize), only steps 2-4 execute (a single iteration evaluating the root node).

---

Test Case Format

Test suites are JSON files with this structure:

{
  "defaults": {
    "n_runs": 3
  },
  "cases": [
    {
      "id": "test_name",
      "user_prompt": "the prompt to send to the model",
      "setup": {
        "files": {
          "filename.py": "file content here",
          "src/utils.py": "another file"
        }
      },
      "expected_actions": [
        {"tool": "read_file", "args": {"path": "filename.py"}},
        {"tool": "bash", "args_pattern": {"command": "python.*test"}},
        {"tool": "edit_file"}
      ],
      "match_mode": "ordered_subset",
      "max_turns": 10,
      "n_runs": 5
    }
  ]
}

Fields

Field	Required	Default	Description
id	yes	--	Unique test case identifier.
user_prompt	yes	--	The message sent to the model.
setup.files	no	{}	Files to create in the temp directory before running. Keys are relative paths, values are file content.
expected_actions	no	[]	List of expected tool calls to match against.
match_mode	no	"ordered_subset"	How to match actual vs expected actions (see Scoring).
max_turns	no	10	Maximum conversation turns before stopping.
n_runs	no	suite default or 3	Per-case override for number of runs.
holdout	no	false	If true, this case is evaluated but excluded from optimizer feedback. Used to measure progress without overfitting.

Expected Action Specs

Each entry in expected_actions can contain:

• tool (required): The tool name to match (e.g. "read_file", "bash").
• args: Exact key-value matching. Each key in args must exist in the actual call with the same string value.
• args_pattern: Regex key-value matching. Each key's value is a regex pattern tested against the actual argument value.
• If neither args nor args_pattern is specified, only the tool name is matched.

---

Scoring

Scoring is handled by score_run(), which compares a run's tool call log against the expected actions.

Match Modes

Mode	Description
exact	Tool calls must match expected actions in exact order and exact count. Extra or missing calls cause failure.
ordered_subset	Expected actions must appear in order within the actual tool log, but extra calls between them are allowed. This is the default.
subset	Expected actions must all appear somewhere in the tool log, in any order. Each actual call can only match one expected action.
contains_any	Passes if at least one expected action appears anywhere in the tool log.

Action Matching (_match_action)

A single actual tool call matches an expected action when:

1. The tool names are equal.
2. If args is specified: every key in args must exist in the actual call's arguments with the same string value (partial key matching -- extra actual args are ignored).
3. If args_pattern is specified: every key's regex pattern must match the corresponding actual argument value via re.search().
4. If the actual args contain only _raw (unparseable JSON fallback), the action matches only when no args or args_pattern is expected.

Score Calculation

• Score = number of matched expected actions / total expected actions.
• Pass = score equals 1.0 (all expected actions matched).
• The return dict includes: pass, score, matched (indices), unmatched (indices), extra_tools, and detail (human-readable summary).

JSON Dump Detection

When a run fails and the model's final text content contains JSON that looks like a tool call (keys like "tool", "command", "path"), the run is flagged with json_dump: true. This indicates the model tried to call a tool but emitted JSON as text instead of using the function-calling interface.

---

HeadlessSession

HeadlessSession extends ChatSession for headless evaluation. It provides deterministic, non-interactive execution suitable for automated testing.

Differences from ChatSession

Aspect	ChatSession	HeadlessSession
Streaming	Streaming API	Non-streaming (stream=False)
Tool approval	User confirmation	auto_approve = True
UI	Terminal/Web UI	NullUI (discards output)
Stdout	Normal	Suppressed during execution
Tool logging	Display only	Structured tool_call_log
System prompt	Built-in developer prompt	Overridable via constructor

NullUI

A minimal UI adapter that satisfies the SessionUI protocol by discarding all output. approve_tools() always returns (True, None).

send_headless()

def send_headless(
    self,
    user_input: str,
    max_turns: int = 10,
    verbose: bool = False,
    log_prefix: str = "",
) -> list[dict]:

Runs a complete multi-turn conversation:

1. Appends the user message.
2. Calls the model API (non-streaming).
3. If tool calls are returned, executes them (with stdout suppressed) and logs each call to self.tool_call_log.
4. Repeats up to max_turns or until the model responds without tool calls.
5. Returns the tool call log: list of dicts with keys tool, args, result (truncated to 500 chars), and turn.

Parallel tool calls are capped at 10 per turn to prevent degenerate repetition.

Retry Logic

send_headless() is called inside _run_single_test() with retry logic: 3 attempts with exponential backoff (sleep 2^attempt seconds) on any exception. This prevents transient API errors from poisoning eval scores.

---

Test Execution

Each test case runs in isolation:

1. A fresh temp directory is created.
2. Setup files are written to the temp directory.
3. The working directory is changed to the temp directory.
4. A new HeadlessSession is created with the current developer prompt.
5. send_headless() runs the user prompt through the conversation loop.
6. The tool log is scored against expected actions.
7. The temp directory is cleaned up.

The memory database is also isolated per test (an ephemeral SQLite database in the temp directory) so tests do not pollute each other or the user's real memory store.

---

Optimization Loop

run_optimization() is the main entry point for iterative prompt optimization. It uses UCB tree search to explore prompt variants, with an observer that tunes the optimizer's strategy.

Evolution Tree

The optimization maintains a tree of prompt variants (EvolutionNode), where each node stores its prompt text, aggregated score, and visit count. The root node (ID 0) contains the initial prompt.

UCB1 selection: Each iteration picks the node with the highest Upper Confidence Bound score: R_bar + C * sqrt(ln(N) / v), where R_bar is the node's mean score, N is total visits across all nodes, v is the node's visit count, and C is the exploration constant (--explore-constant, default sqrt(2)). Unvisited nodes are always selected first.

Flow

for iteration in 0..max_iterations:
    1. UCB select -> pick the most promising tree node
    2. Run all test cases n_runs times with selected node's prompt
    3. Update node score (rolling mean) and visit count
    4. Save intermediate results + tree state to JSON
    5. If all tests pass -> stop
    6. Every 3 iterations (at iteration 2, 5, 8, ...):
       -> Observer reviews optimizer strategy
    7. Optimizer proposes new prompt (child of selected node)
       with improvement-based feedback (delta from parent node)
    8. Add child node to tree

Holdout Cases

Test cases with "holdout": true are evaluated every iteration but excluded from the optimizer's feedback. This prevents the optimizer from overfitting to specific test cases. Node scores are computed from holdout cases only (when present). If fewer than 2 non-holdout cases remain, holdout is disabled.

Improvement-Based Feedback

The optimizer sees delta scores (delta=+20%) alongside absolute pass rates, showing how each case improved relative to the parent node's evaluation. This provides a cleaner signal than absolute scores alone — the optimizer can distinguish beneficial edits from harmful ones regardless of starting point.

Prompt Proposal (_propose_prompt_modification)

Uses the model to rewrite the developer prompt based on test results:

• Input: Current prompt, test case definitions, per-case results with actual vs expected tool sequences, and a history of the last 3 iterations.
• Optimizer system prompt (OPTIMIZER_SYSTEM): Instructs the model to act as a text rewriter. Key guidance includes:
  • Address critical failure modes (text-only responses, write_file vs edit_file, unnecessary search before create, missing plan calls).
  • Preserve phrasing that drives 100% pass rate on passing tests.
  • Use direct imperative style with concrete tool call examples.
  • Stay within 130% of original prompt length.
• Output: The rewritten prompt text (stripped of reasoning tags and code fences).

Observer System (_observe_and_update_optimizer)

Every 3 iterations, a meta-level "observer" reviews the optimizer's strategy:

• Analyzes the iteration history: score trends, regressions, prompt length changes, and diffs between iterations.
• Summarizes the optimizer's behavioral patterns (list style, header usage, length).
• Uses OBSERVER_SYSTEM to rewrite the optimizer's own system prompt.
• Rejects degenerate outputs (over 200% of input length). This two-level optimization (optimizer + observer) helps the system escape local minima and adjust its rewriting strategy. With tree search, the observer no longer needs to reset the developer prompt — UCB naturally gravitates toward high-scoring nodes.

Result Persistence

After each iteration, results are written to the output JSON file. The structure is:

{
  "meta": {
    "model": "model-name",
    "base_url": "http://localhost:8000/v1",
    "started": "2025-01-01T00:00:00",
    "test_suite": "tests.json",
    "n_runs_default": 3,
    "explore_constant": 1.414,
    "holdout_ids": []
  },
  "iterations": [
    {
      "iteration": 0,
      "prompt": "the developer prompt used",
      "prompt_diff": null,
      "optimizer_system": "the optimizer system prompt",
      "timestamp": "2025-01-01T00:01:00",
      "tree_node_id": 0,
      "tree_child_id": 1,
      "cases": {
        "test_name": {
          "runs": [
            {
              "pass": true,
              "score": 1.0,
              "matched": [0, 1],
              "unmatched": [],
              "extra_tools": [],
              "detail": "Ordered subset: 2/2",
              "tool_sequence": ["read_file", "edit_file"],
              "tool_args": [{"read_file": {"path": "f.py"}}, ...],
              "elapsed": 3.2
            }
          ],
          "pass_rate": 1.0,
          "avg_score": 1.0
        }
      },
      "aggregate": {
        "total_cases": 5,
        "total_runs": 15,
        "overall_pass_rate": 0.8,
        "overall_avg_score": 0.87,
        "json_dumps": 0,
        "per_case_pass_rates": {"test_name": 1.0, ...}
      }
    }
  ],
  "tree": [
    {
      "node_id": 0,
      "parent_id": null,
      "prompt": "initial prompt",
      "score": 0.85,
      "visit_count": 3,
      "children": [1, 2],
      "iteration": 0
    }
  ]
}

---

CLI Usage

The entry point is turnstone-eval (installed as a console script) or python -m turnstone.eval.

turnstone-eval tests.json                          # evaluate + optimize
turnstone-eval tests.json --no-optimize            # evaluate only (single iteration)
turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
turnstone-eval tests.json --prompt custom.txt      # start from a custom prompt
turnstone-eval tests.json -v                       # verbose per-turn logging

All Options

Flag	Default	Description
test_file	(positional, required)	Path to test cases JSON file.
--base-url	http://localhost:8000/v1	API base URL.
--model	auto-detect	Model name. Auto-detected from the API if not specified.
--prompt	turnstone built-in prompt	Path to initial prompt text file.
--n-runs	from tests.json or 3	Number of runs per test case.
--max-iter	5	Maximum optimization iterations.
--no-optimize	false	Run evaluation only (sets max-iter to 1).
--temperature	0.7	Sampling temperature.
--max-tokens	32768	Max completion tokens.
--reasoning-effort	medium	Reasoning effort: low, medium, or high.
--context-window	131072	Context window size.
--output	eval_results.json	Output results file path.
-v, --verbose	false	Show detailed per-turn logging (API calls, tool args, results).
--explore-constant	1.414 (sqrt(2))	UCB exploration constant C. Lower values favor exploitation (extend best nodes), higher values favor exploration (try more branches).

Precedence for n_runs

The number of runs per test case is resolved in this order:

1. Per-case n_runs field in the test case definition.
2. CLI --n-runs argument (if provided).
3. Suite-level defaults.n_runs in the test JSON file.
4. Code default: 3.