refactor(eval): split measurement core from prompt optimizer

turnstone-eval was misnamed: it was a prompt optimizer, not a measurement harness. Split the 3252-line turnstone/eval.py into a strictly one-way dependency (optimizer -> eval-core; core never imports the optimizer): - turnstone/eval/core.py measurement substrate — everything up to and including _run_iteration: provider detection, NullUI, HeadlessSession, the test runner, score_run, aggregation, and neutral reporting. - turnstone/eval/cli.py new measure-only `turnstone-eval` — the old --no-optimize path promoted to the whole job (one _run_iteration call, then print the summary table). - turnstone/optimizer.py the UCB self-modify loop and its multi-agent pipeline (analyst/optimizer/observer/diversifier/tool optimizer), now `turnstone-optimizer`; imports from eval.core only. - turnstone/eval/__init__.py re-exports the core public API for back-compat (score_run, _match_action, _run_iteration, HeadlessSession). _apply_tool_overrides lives in core (HeadlessSession needs it) rather than alongside the other tree helpers, so the dependency stays one-way. Breaking change: `turnstone-eval` now measures; use `turnstone-optimizer` to optimize. Both code paths are behaviour-preserving — the moved function bodies are byte-identical.
2026-08-12 23:12:23 -06:00 · 2026-07-03 15:54:55 -07:00
parent cf05ffee7d
commit 7053439e84
9 changed files with 1670 additions and 1305 deletions
@@ -124,7 +124,8 @@ Built-in tools for shell, files, search, web, memory, notifications, and autonom
 | `turnstone-console` | Cluster dashboard + routing proxy + admin panel |
 | `turnstone-channel` | Channel gateway (Discord and Slack adapters) |
 | `turnstone-admin` | User/token management CLI |
-| `turnstone-eval` | Eval harness for prompt/tool optimization |
+| `turnstone-eval` | Headless measurement — scores tool-use against expected actions |
+| `turnstone-optimizer` | Prompt/tool optimizer (UCB self-modify loop over the eval substrate) |
 | `turnstone-doctor` | LLM-backed cluster diagnostics |

 ### Diagrams
@@ -19,7 +19,8 @@ plugs in.
 | `turnstone` | `turnstone.cli` | `TerminalUI` | Interactive terminal REPL |
 | `turnstone-server` | `turnstone.server` | `WebUI` | Browser-based chat (HTTP + SSE) |
 | `turnstone-console` | `turnstone.console.server` | ClusterCollector | Cluster dashboard (aggregates all nodes) |
-| `turnstone-eval` | `turnstone.eval` | `NullUI` | Headless evaluation and prompt optimization |
+| `turnstone-eval` | `turnstone.eval.cli` | `NullUI` | Headless measurement (scores tool-use against expected actions) |
+| `turnstone-optimizer` | `turnstone.optimizer` | `NullUI` | Prompt/tool optimization (UCB self-modify loop over the eval substrate) |
 | `turnstone-channel` | `turnstone.channels.cli` | ChannelAdapter | Channel gateway (Discord, Slack, etc.) |
 | `turnstone-admin` | `turnstone.admin` | — | Offline user and API token management |
 | `turnstone-doctor` | `turnstone.doctor` | — | LLM-backed cluster diagnostics |
@@ -267,7 +268,7 @@ the per-workstream events stream in
 |-------|--------|-------|
 | `TerminalUI` | `turnstone.cli` | ANSI colors, `MarkdownRenderer`, `Spinner`, readline-based `input()` for approval |
 | `WebUI` | `turnstone.server` | SSE event queue per workstream + global broadcast, `threading.Event` for blocking on approval. `on_state_change` sends to both per-workstream and global SSE (the browser UI uses per-workstream `state_change` events to manage busy/idle transitions; `stream_end` only finalizes markdown rendering). |
-| `NullUI` | `turnstone.eval` | Discards all output; `approve_tools` always returns `(True, None)` |
+| `NullUI` | `turnstone.eval.core` | Discards all output; `approve_tools` always returns `(True, None)` |

 ### WorkstreamTerminalUI

@@ -260,7 +260,7 @@ interface, or anyone who can reach it can search through your instance.

 Both stacks install all entry points into a single image (`turnstone`,
 `turnstone-server`, `turnstone-console`, `turnstone-channel`, `turnstone-admin`,
-`turnstone-eval`, `turnstone-doctor`):
+`turnstone-eval`, `turnstone-optimizer`, `turnstone-doctor`):

 ```bash
 docker compose build            # build the dev image
@@ -1,11 +1,19 @@
-# Evaluation and Prompt Optimization (turnstone-eval)
+# Evaluation and Prompt Optimization (turnstone-eval, turnstone-optimizer)

-`turnstone-eval` is the evaluation and prompt optimization system for turnstone. It
-runs test cases against the LLM, scores tool call sequences against expected
-actions, and optionally uses a multi-agent pipeline to optimize the developer
-prompt and tool descriptions.
+Evaluation for turnstone is split into two commands:

-Source: `turnstone/eval.py`
+- **`turnstone-eval`** — the measurement substrate. Runs test cases against the LLM
+  and scores tool call sequences against expected actions. A single measurement pass,
+  no self-modification.
+- **`turnstone-optimizer`** — the prompt/tool optimizer. Loops over the measurement
+  substrate, using a multi-agent pipeline (analyst, optimizer, observer, diversifier,
+  tool optimizer) to edit the developer prompt and tool descriptions so more tests pass.
+
+The dependency is strictly one-way: the optimizer consumes the eval substrate; the
+substrate never depends on the optimizer.
+
+Source: `turnstone/eval/core.py` (measurement substrate), `turnstone/eval/cli.py`
+(the `turnstone-eval` CLI), `turnstone/optimizer.py` (the `turnstone-optimizer` CLI).

 ---

@@ -27,8 +35,8 @@ This approach (inspired by [Learning to Self-Evolve](https://arxiv.org/abs/2603.
 prevents irrecoverable collapse from bad edits — UCB naturally backtracks to
 high-scoring ancestors instead of following a linear chain.

-When optimization is disabled (`--no-optimize`), only steps 2-4 execute
-(a single iteration evaluating the root node).
+The `turnstone-eval` command (or `turnstone-optimizer --no-optimize`) executes only
+steps 2-4: a single measurement pass over the root prompt, no optimization.

 ---

@@ -452,30 +460,46 @@ structure is:

 ## CLI Usage

-The entry point is `turnstone-eval` (installed as a console script) or
-`python -m turnstone.eval`.
+Two console scripts (installed as entry points), or the equivalent `python -m`
+invocations:
+
+- `turnstone-eval` / `python -m turnstone.eval.cli` — measure only.
+- `turnstone-optimizer` / `python -m turnstone.optimizer` — optimize.
+
+### Measure (`turnstone-eval`)

 ```
-turnstone-eval tests.json                          # evaluate + optimize
-turnstone-eval tests.json --no-optimize            # evaluate only (single iteration)
-turnstone-eval tests.json --n-runs 5 --max-iter 10 # more thorough evaluation
-turnstone-eval tests.json --prompt custom.txt      # start from a custom prompt
-turnstone-eval tests.json --optimize-tools         # optimize tool descriptions only
-turnstone-eval tests.json --diversify 10           # test with prompt variants
-turnstone-eval tests.json -v                       # verbose per-turn logging
+turnstone-eval tests.json                     # one measurement pass, print scores
+turnstone-eval tests.json --prompt custom.txt # measure a custom prompt
+turnstone-eval tests.json --n-runs 5          # more runs per case
+turnstone-eval tests.json --parallel 4        # run cases across 4 workers
+turnstone-eval tests.json -v                  # verbose per-turn logging
 ```

-### Multi-model setup (local test model, cloud optimizer)
+### Optimize (`turnstone-optimizer`)

 ```
-turnstone-eval tests.json \
+turnstone-optimizer tests.json                          # evaluate + optimize
+turnstone-optimizer tests.json --no-optimize            # single pass, no optimization
+turnstone-optimizer tests.json --n-runs 5 --max-iter 10 # more thorough optimization
+turnstone-optimizer tests.json --prompt custom.txt      # start from a custom prompt
+turnstone-optimizer tests.json --optimize-tools         # optimize tool descriptions only
+turnstone-optimizer tests.json --diversify 10           # test with prompt variants
+```
+
+#### Multi-model setup (local test model, cloud optimizer)
+
+```
+turnstone-optimizer tests.json \
  --base-url http://localhost:8000/v1 \
  --optimizer-base-url https://api.anthropic.com \
  --optimizer-model claude-sonnet-4-6 \
  --analyst-model claude-opus-4-6
 ```

-### All Options
+### Measurement Options
+
+Accepted by **both** commands.

 | Flag                    | Default                    | Description |
 |-------------------------|----------------------------|-------------|
@@ -484,19 +508,26 @@ turnstone-eval tests.json \
 | `--model`               | auto-detect                | Model name. Auto-detected from the API if not specified. |
 | `--prompt`              | turnstone built-in prompt  | Path to initial prompt text file. |
 | `--n-runs`              | from tests.json or 3       | Number of runs per test case. |
-| `--max-iter`            | 5                          | Maximum optimization iterations. |
-| `--no-optimize`         | false                      | Run evaluation only (sets max-iter to 1). |
 | `--temperature`         | 0.7                        | Sampling temperature. |
 | `--max-tokens`          | 32768                      | Max completion tokens. |
 | `--reasoning-effort`    | `medium`                   | Reasoning effort: `low`, `medium`, or `high`. |
 | `--context-window`      | 131072                     | Context window size. |
 | `--output`              | `eval_results.json`        | Output results file path. |
 | `-v`, `--verbose`       | false                      | Show detailed per-turn logging. |
-| `--explore-constant`    | 1.414 (sqrt(2))            | UCB exploration constant C. |
 | `--test-timeout`        | 300                        | Per-test timeout in seconds. |
-| `--suite-timeout`       | 0 (unlimited)              | Total suite timeout in seconds. |
 | `--no-fast-fail`        | false                      | Disable early termination on all-zero initial runs. |
 | `--parallel`            | 1 (serial)                 | Parallel workers (0=auto, N=use N workers). |
+
+### Optimizer Options
+
+Accepted by **`turnstone-optimizer`** only.
+
+| Flag                    | Default                    | Description |
+|-------------------------|----------------------------|-------------|
+| `--max-iter`            | 5                          | Maximum optimization iterations. |
+| `--no-optimize`         | false                      | Run a single measurement pass (sets max-iter to 1). |
+| `--explore-constant`    | 1.414 (sqrt(2))            | UCB exploration constant C. |
+| `--suite-timeout`       | 0 (unlimited)              | Total suite timeout in seconds. |
 | `--optimizer-model`     | same as `--model`          | Model for prompt optimization. |
 | `--optimizer-base-url`  | same as `--base-url`       | Base URL for optimizer model. |
 | `--observer-model`      | same as optimizer           | Model for meta-optimization (observer). |
@@ -64,7 +64,8 @@ all = ["turnstone[discord,slack]"]

 [project.scripts]
 turnstone = "turnstone.cli:main"
-turnstone-eval = "turnstone.eval:main"
+turnstone-eval = "turnstone.eval.cli:main"
+turnstone-optimizer = "turnstone.optimizer:main"
 turnstone-server = "turnstone.server:main"
 turnstone-console = "turnstone.console.server:main"
 turnstone-admin = "turnstone.admin:main"
@@ -0,0 +1,27 @@
+"""turnstone.eval — headless measurement substrate and its measure-only CLI.
+
+The measurement substrate lives in :mod:`turnstone.eval.core`; the measure-only
+``turnstone-eval`` command lives in :mod:`turnstone.eval.cli`. The prompt
+optimizer that consumes this substrate lives in :mod:`turnstone.optimizer`.
+
+The core public API is re-exported here for back-compatibility with existing
+importers (e.g. ``from turnstone.eval import score_run, _match_action``).
+"""
+
+from turnstone.eval.core import (
+    HeadlessSession,
+    NullUI,
+    _match_action,
+    _run_iteration,
+    _run_single_test,
+    score_run,
+)
+
+__all__ = [
+    "HeadlessSession",
+    "NullUI",
+    "_match_action",
+    "_run_iteration",
+    "_run_single_test",
+    "score_run",
+]
@@ -0,0 +1,253 @@
+#!/usr/bin/env python3
+"""cli.py — measure-only ``turnstone-eval`` entry point.
+
+Runs a test suite once against a model and prints a scored summary. This is the
+old ``--no-optimize`` behaviour promoted to the whole job: it calls
+:func:`turnstone.eval.core._run_iteration` exactly once and reports the result.
+Prompt optimization lives in :mod:`turnstone.optimizer` (``turnstone-optimizer``).
+
+Usage:
+    turnstone-eval tests.json
+    turnstone-eval tests.json --prompt prompt.txt --n-runs 5 --parallel 4
+"""
+
+import argparse
+import json
+import os
+import re
+import textwrap
+from datetime import datetime
+from typing import Any
+
+from openai import OpenAI
+
+from turnstone.core.session import ChatSession
+from turnstone.eval.core import (
+    NullUI,
+    _append_summary_tsv,
+    _print_summary_table,
+    _run_iteration,
+)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Headless measurement for turnstone (scores tool use against expected actions)",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=textwrap.dedent("""\
+            Examples:
+              # Measure with the built-in developer prompt
+              turnstone-eval tests.json
+
+              # Measure a custom prompt with more runs, in parallel
+              turnstone-eval tests.json --prompt prompt.txt --n-runs 5 --parallel 4
+
+            Prompt optimization now lives in turnstone-optimizer.
+        """),
+    )
+    parser.add_argument(
+        "test_file",
+        help="Path to test cases JSON file",
+    )
+    parser.add_argument(
+        "--base-url",
+        default="http://localhost:8000/v1",
+        help="API base URL (default: http://localhost:8000/v1)",
+    )
+    parser.add_argument(
+        "--model",
+        default=None,
+        help="Model name (default: auto-detect)",
+    )
+    parser.add_argument(
+        "--prompt",
+        default=None,
+        help="Path to initial prompt text file (default: use turnstone's built-in)",
+    )
+    parser.add_argument(
+        "--n-runs",
+        type=int,
+        default=None,
+        help="Number of runs per test case (default: from tests.json or 3)",
+    )
+    parser.add_argument(
+        "--temperature",
+        type=float,
+        default=0.7,
+        help="Sampling temperature (default: 0.7)",
+    )
+    parser.add_argument(
+        "--max-tokens",
+        type=int,
+        default=32768,
+        help="Max completion tokens (default: 32768)",
+    )
+    parser.add_argument(
+        "--reasoning-effort",
+        default="medium",
+        choices=["low", "medium", "high"],
+        help="Reasoning effort (default: medium)",
+    )
+    parser.add_argument(
+        "--context-window",
+        type=int,
+        default=131072,
+        help="Context window size (default: 131072)",
+    )
+    parser.add_argument(
+        "--output",
+        default="eval_results.json",
+        help="Output results file (default: eval_results.json)",
+    )
+    parser.add_argument(
+        "--test-timeout",
+        type=int,
+        default=300,
+        help="Per-test timeout in seconds (default: 300)",
+    )
+    parser.add_argument(
+        "--no-fast-fail",
+        action="store_true",
+        help="Disable early termination when all initial runs score 0.0",
+    )
+    parser.add_argument(
+        "--parallel",
+        type=int,
+        default=1,
+        help="Parallel workers (default: 1=serial, 0=auto)",
+    )
+    parser.add_argument(
+        "-v",
+        "--verbose",
+        action="store_true",
+        help="Show detailed per-turn logging (API calls, tool args, results)",
+    )
+    from turnstone.core.config import add_config_arg, apply_config
+
+    add_config_arg(parser)
+    apply_config(parser, ["api", "model"])
+    args = parser.parse_args()
+
+    api_key = os.environ.get("OPENAI_API_KEY", "dummy")
+    client = OpenAI(
+        base_url=args.base_url,
+        api_key=api_key,
+    )
+
+    model = args.model
+    if not model:
+        from turnstone.core.model_registry import detect_model
+
+        detected, _ = detect_model(client)
+        assert detected is not None  # fatal=True guarantees non-None or SystemExit
+        model = detected
+
+    # Resolve the initial prompt: --prompt file, else turnstone's built-in.
+    initial_prompt: str | None = None
+    if args.prompt:
+        with open(args.prompt) as f:
+            initial_prompt = f.read()
+    if initial_prompt is None:
+        # Extract the default developer message from a temporary ChatSession
+        tmp = ChatSession(
+            client=client,
+            model=model,
+            ui=NullUI(),
+            instructions=None,
+            temperature=args.temperature,
+            max_tokens=args.max_tokens,
+            tool_timeout=30,
+            reasoning_effort=args.reasoning_effort,
+            context_window=args.context_window,
+        )
+        initial_prompt = next(
+            (m["content"] for m in tmp.system_messages if m["role"] in ("developer", "system")),
+            None,
+        )
+        if initial_prompt is None:
+            raise SystemExit("No developer prompt found. Provide one with --prompt <file>")
+        # Strip memory reminder — it's a runtime artifact, not part of the prompt
+        initial_prompt = re.sub(
+            r"\n*REMINDER: You currently have \d+ memories stored\..*$",
+            "",
+            initial_prompt,
+        ).strip()
+
+    # Load test cases
+    with open(args.test_file) as f:
+        suite: dict[str, Any] = json.load(f)
+
+    cases: list[dict[str, Any]] = suite["cases"]
+    for i, case in enumerate(cases):
+        if "id" not in case:
+            raise SystemExit(f"Test case {i} missing required 'id' field")
+        if "user_prompt" not in case:
+            raise SystemExit(f"Test case '{case.get('id', i)}' missing 'user_prompt'")
+    defaults = suite.get("defaults", {})
+    # Precedence: CLI arg (non-None) > tests.json defaults > code default (3)
+    resolved_n_runs: int = (
+        args.n_runs if args.n_runs is not None else int(defaults.get("n_runs", 3))
+    )
+
+    # Auto-load cached prompt variants if present (parity with the optimizer path).
+    prompt_variants: dict[str, list[str]] | None = None
+    cached_variants = {
+        c["id"]: c["user_prompts"]
+        for c in cases
+        if isinstance(c.get("user_prompts"), list) and len(c["user_prompts"]) > 1
+    }
+    if cached_variants:
+        prompt_variants = cached_variants
+        total_v = sum(len(v) for v in cached_variants.values())
+        print(
+            f"\n  Using cached variants for {len(cached_variants)} cases ({total_v} total prompts)"
+        )
+
+    parallel = args.parallel if args.parallel != 0 else (os.cpu_count() or 4)
+
+    iter_result = _run_iteration(
+        client=client,
+        model=model,
+        system_prompt=initial_prompt,
+        cases=cases,
+        n_runs=resolved_n_runs,
+        temperature=args.temperature,
+        max_tokens=args.max_tokens,
+        reasoning_effort=args.reasoning_effort,
+        context_window=args.context_window,
+        verbose=args.verbose,
+        test_timeout=args.test_timeout,
+        fast_fail=not args.no_fast_fail,
+        parallel=parallel,
+        base_url=args.base_url,
+        api_key=api_key,
+        prompt_variants=prompt_variants,
+    )
+    iter_result["iteration"] = 0
+    iter_result["prompt"] = initial_prompt
+    iter_result["timestamp"] = datetime.now().isoformat()
+
+    _print_summary_table(iter_result)
+
+    results = {
+        "meta": {
+            "model": model,
+            "base_url": args.base_url,
+            "test_suite": args.test_file,
+            "n_runs_default": resolved_n_runs,
+            "started": iter_result["timestamp"],
+        },
+        "iterations": [iter_result],
+    }
+    with open(args.output, "w") as f:
+        json.dump(results, f, indent=2)
+        f.write("\n")
+
+    tsv_path = os.path.splitext(args.output)[0] + ".tsv"
+    _append_summary_tsv(tsv_path, iter_result, [c["id"] for c in cases])
+
+    print(f"Results written to {args.output}")
+
+
+if __name__ == "__main__":
+    main()