- Docstrings/help said the treatment skill 'composes into the system
message'. This harness runs on both checkouts (system on main, a context
turn on the placement-refactor branch), so the wording now describes the
natural set_skill composition path without asserting a placement.
- Validate each skill-bearing case's 'skill' shape up front (driver +
CLI) so a malformed dataset fails with a clear error, not a mid-run
KeyError. Pinned by test_rejects_malformed_skill.
Add a two-arm skill-adherence mode to the eval measurement substrate that
measures whether a NAMED skill changes tool-use behaviour, so skill-in-system
(main) can be compared against skill-in-context.
- _run_single_test gains skill/skill_mode: skill_mode builds HeadlessSession
under natural composition (no system_prompt_override) and, for the treatment
arm, seeds the skill into the temp DB and activates it via the real
set_skill path so the skill body folds into the system message under test.
skill_mode defaults False, so the optimizer/measure paths are unchanged.
- Thread skill/skill_mode through _run_and_score_subprocess, _run_iteration
and _run_iteration_parallel (serial + parallel).
- run_skill_adherence: per case, run treatment (skill) vs control (no skill)
n_runs each, score against expected_actions, report per-case lift =
pass_rate(treatment) - pass_rate(control) and the mean lift. The control
isolates the skill's causal effect.
- turnstone-eval --skill-adherence <dataset>: loads a skill-scenario dataset
and prints a treatment/control/lift table.
- eval_skill_adherence.json: authored search-first / test-after-edit /
changelog-update scenarios, chosen so the base model does not do the action
by default.
- tests: plumbing proof (skill folds into system_messages for treatment,
absent for control) + lift-math aggregation.
Patrick Buckley