361876b17a
feat: eval pipeline improvements + tool description optimization ( #174 )
...
Eval pipeline:
- --optimize-tools mode freezes system prompt, optimizes tool descriptions only
- Analyst sees available tool list (prevents hallucinated "tool not in schema")
- Analyst sees current tool descriptions in --optimize-tools mode
- Three-layer timeout defense: httpx timeout + _cancelled event + client.close()
- Filter MCP-only tools (read_resource, use_prompt) from headless eval
- Pattern-over-rules framing in optimizer, analyst, and observer prompts
- Tool description diffs logged after each iteration
- Tool optimizer failure retries instead of stopping the loop
Tool renames (avoid chat template channel collision on local models):
- create_plan -> plan_agent
- task -> task_agent
Tool descriptions (from eval-driven optimization, 79% -> 98%):
- bash: environment question examples, disambiguation from write_file/man
- edit_file: multi-file workflow, prerequisite clarification, docstring example
- man: "questions about flags are tool-use tasks" prefix
- math: simple example up front
- plan_agent: "delegate to sub-agent" framing, negative boundary for direct edits
- read_file: multi-file workflow hint
- search: trigger phrases, prerequisite clarification, disambiguation
- write_file: immediate action framing, placeholder example
System prompt: enriched tool patterns from eval results, added identity opener
Test suite: plan-before-refactor -> plan-when-asked (simplified)
Docs: comprehensive eval.md rewrite covering all current features
2026-03-25 11:58:10 -07:00
Patrick Buckley