Patrick Buckley 361876b17a feat: eval pipeline improvements + tool description optimization (#174) ...

Eval pipeline:
- --optimize-tools mode freezes system prompt, optimizes tool descriptions only
- Analyst sees available tool list (prevents hallucinated "tool not in schema")
- Analyst sees current tool descriptions in --optimize-tools mode
- Three-layer timeout defense: httpx timeout + _cancelled event + client.close()
- Filter MCP-only tools (read_resource, use_prompt) from headless eval
- Pattern-over-rules framing in optimizer, analyst, and observer prompts
- Tool description diffs logged after each iteration
- Tool optimizer failure retries instead of stopping the loop

Tool renames (avoid chat template channel collision on local models):
- create_plan -> plan_agent
- task -> task_agent

Tool descriptions (from eval-driven optimization, 79% -> 98%):
- bash: environment question examples, disambiguation from write_file/man
- edit_file: multi-file workflow, prerequisite clarification, docstring example
- man: "questions about flags are tool-use tasks" prefix
- math: simple example up front
- plan_agent: "delegate to sub-agent" framing, negative boundary for direct edits
- read_file: multi-file workflow hint
- search: trigger phrases, prerequisite clarification, disambiguation
- write_file: immediate action framing, placeholder example

System prompt: enriched tool patterns from eval results, added identity opener

Test suite: plan-before-refactor -> plan-when-asked (simplified)

Docs: comprehensive eval.md rewrite covering all current features

2026-03-25 11:58:10 -07:00

..

__init__.py

Initial commit — turnstone multi-node AI orchestration platform.

2026-03-02 00:33:37 -08:00

conftest.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_api_versioning.py

fix: wire resume_ws through console + expose max_ws in heartbeat (#124)

2026-03-18 14:24:10 -07:00

test_async_broker.py

Add channel integrations with Discord adapter and atomic session resu… (#24)

2026-03-04 13:02:58 -08:00

test_audit.py

feat: governance — RBAC, tool policies, prompt templates, usage track… (#39)

2026-03-10 19:32:37 -07:00

test_auth_identity.py

chore: remove dead code and fix noisy JWT test warnings

2026-03-17 17:02:25 -07:00

test_auth.py

fix: propagate user identity through console proxy

2026-03-24 03:13:23 -07:00

test_bm25.py

feat: structured memory system — typed/scoped memories with BM25 rele… (#53)

2026-03-13 21:21:09 -07:00

test_bootstrap.py

feat: bootstrap wizard — LLM-guided interactive setup for deployments

2026-03-11 01:58:39 -07:00

test_bridge_events.py

refactor: move bridge content buffer to server-side single source of truth

2026-03-15 15:04:22 -07:00

test_bridge_stress.py

fix: bridge approval & plan review TOCTOU races (#158, #159) (#167)

2026-03-23 14:04:12 -07:00

test_cancel.py

feat: output guard — evaluate tool results before they enter context (#109)

2026-03-16 16:22:10 -07:00

test_channel_discord.py

feat: Discord content catch-up + bidirectional notification replies (… (#64)

2026-03-14 23:58:27 -07:00

test_channel_protocol.py

feat: intent validation v1 — advisory LLM judge for tool approvals (#50) (#50)

2026-03-13 04:12:46 -07:00

test_channel_routing.py

Add channel integrations with Discord adapter and atomic session resu… (#24)

2026-03-04 13:02:58 -08:00

test_channel_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_config_store.py

feat: database-backed settings (ConfigStore) with admin API (#59)

2026-03-14 11:42:18 -07:00

test_config.py

feat: --config flag and $TURNSTONE_CONFIG env var (#160)

2026-03-23 12:26:41 -07:00

test_console.py

review: address copilot feedback on user_id trust check

2026-03-24 03:13:23 -07:00

test_db.py

feat: structured memory system — typed/scoped memories with BM25 rele… (#53)

2026-03-13 21:21:09 -07:00

test_edit.py

Initial commit — turnstone multi-node AI orchestration platform.

2026-03-02 00:33:37 -08:00

test_env_scrub.py

review: fix mypy, tighten env scrub, bridge storage safety

2026-03-23 14:55:14 -07:00

test_fts5.py

Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)

2026-03-03 22:57:34 -08:00

test_governance_endpoints.py

feat: unified skills system — merge prompt templates + workstream tem… (#106)

2026-03-16 14:47:06 -07:00

test_governance_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_healthcheck.py

fix: health monitor probe loop transitions OPEN → HALF_OPEN autonomously (#152)

2026-03-21 16:35:47 -07:00

test_html.py

Initial commit — turnstone multi-node AI orchestration platform.

2026-03-02 00:33:37 -08:00

test_judge_heuristic.py

feat: intent validation v1 — advisory LLM judge for tool approvals (#50) (#50)

2026-03-13 04:12:46 -07:00

test_judge_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_judge.py

chore: remove dead code and fix noisy JWT test warnings

2026-03-17 17:02:25 -07:00

test_load_skill.py

feat: Agent Skills standard compliance + frontend spec fields

2026-03-17 16:09:06 -07:00

test_logging.py

Add structured logging with structlog and context propagation

2026-03-04 06:47:30 -08:00

test_markdown.py

Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)

2026-03-02 16:55:12 -08:00

test_mcp_admin_api.py

review: address copilot feedback on user_id trust check

2026-03-24 03:13:23 -07:00

test_mcp_client.py

fix: MCP resource template URI expansion via prefix matching (#46)

2026-03-12 15:46:00 -07:00

test_mcp_hot_reload.py

fix: surface MCP server errors in admin UI instead of silent logging (#100)

2026-03-15 17:12:13 -07:00

test_mcp_integration.py

fix: MCP resource template URI expansion via prefix matching (#46)

2026-03-12 15:46:00 -07:00

test_mcp_prompt_sync.py

feat: wire prompt templates into session startup with full creation-p… (#47)

2026-03-12 16:57:31 -07:00

test_mcp_registry_api.py

fix: align ConfigStore implementation with spec (#153)

2026-03-21 16:12:50 -07:00

test_mcp_registry.py

fix: validate URL scheme after MCP registry template substitution (#133)

2026-03-20 20:01:26 -07:00

test_mcp_server_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_memory_api.py

fix: validate scope_id requires scope in memory API (#80)

2026-03-15 14:28:03 -07:00

test_memory_relevance.py

feat: structured memory system — typed/scoped memories with BM25 rele… (#53)

2026-03-13 21:21:09 -07:00

test_metacognition.py

feat: metacognitive nudge on tool error — search memories for guidance

2026-03-17 02:06:10 -07:00

test_model_registry.py

fix: server startup stampede — timeout model detection, non-fatal PG … (#132)

2026-03-20 20:01:21 -07:00

test_notify_http.py

Add channel notification tool with security hardening (#27)

2026-03-05 17:24:00 -08:00

test_notify_tool.py

Normalize session_id into ws_id as sole persistent identity (#29)

2026-03-07 12:49:07 -08:00

test_oidc_handlers.py

chore: remove dead code and fix noisy JWT test warnings

2026-03-17 17:02:25 -07:00

test_oidc_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_oidc.py

fix: validate OIDC issuer URLs against SSRF before discovery fetch (#140)

2026-03-21 00:14:46 -07:00

test_openapi.py

Normalize session_id into ws_id as sole persistent identity (#29)

2026-03-07 12:49:07 -08:00

test_output_assessment_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_output_guard.py

fix: output guard detects single secret-bearing env lines (#115)

2026-03-17 02:19:22 -07:00

test_prompt_templates_runtime.py

feat: output guard — evaluate tool results before they enter context (#109)

2026-03-16 16:22:10 -07:00

test_protocol.py

feat: unified skills system — merge prompt templates + workstream tem… (#106)

2026-03-16 14:47:06 -07:00

test_providers.py

feat: enable prompt caching for Anthropic and OpenAI providers (#104)

2026-03-16 01:59:55 -07:00

test_ratelimit.py

Fix circuit breaker, rate limiter, and Anthropic web search correctne… (#15)

2026-03-03 18:28:39 -08:00

test_reconstruct_messages.py

fix: simplify conversation storage — atomic assistant rows with tool_… (#51)

2026-03-13 14:26:05 -07:00

test_resume_flow.py

Normalize session_id into ws_id as sole persistent identity (#29)

2026-03-07 12:49:07 -08:00

test_safety.py

Initial commit — turnstone multi-node AI orchestration platform.

2026-03-02 00:33:37 -08:00

test_sandbox.py

Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)

2026-03-02 16:55:12 -08:00

test_schedule_api.py

feat: governance — RBAC, tool policies, prompt templates, usage track… (#39)

2026-03-10 19:32:37 -07:00

test_scheduled_tasks_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_scheduler.py

Add scheduled task system with cron/at scheduling (#26)

2026-03-05 16:05:00 -08:00

test_scoring.py

Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)

2026-03-02 16:55:12 -08:00

test_sdk_console.py

Add scheduled task docs and SDK client methods

2026-03-05 18:11:36 -08:00

test_sdk_events.py

fix: approval timeout UI state and content flush before tool calls (#42)

2026-03-12 11:43:06 -07:00

test_sdk_governance.py

test: add governance SDK integration tests against real Starlette app (#142)

2026-03-21 00:31:53 -07:00

test_sdk_server.py

Normalize session_id into ws_id as sole persistent identity (#29)

2026-03-07 12:49:07 -08:00

test_sdk_sse.py

Add Python and TypeScript client SDKs for server and console APIs (#19)

2026-03-03 21:22:24 -08:00

test_sdk_sync.py

Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)

2026-03-04 06:04:59 -08:00

test_server_live.py

fix: wire resume_ws through console + expose max_ws in heartbeat (#124)

2026-03-18 14:24:10 -07:00

test_services_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_session.py

feat: eval pipeline improvements + tool description optimization (#174)

2026-03-25 11:58:10 -07:00

test_sessions.py

fix: isolate parallel tool exceptions + gate web_search without backend

2026-03-17 16:49:11 -07:00

test_settings_api.py

fix: align ConfigStore implementation with spec (#153)

2026-03-21 16:12:50 -07:00

test_settings_registry.py

feat: database-backed settings (ConfigStore) with admin API (#59)

2026-03-14 11:42:18 -07:00

test_sim.py

chore: remove dead code and fix noisy JWT test warnings

2026-03-17 17:02:25 -07:00

test_skill_discovery_api.py

fix: normalize install response + review fixes

2026-03-17 01:26:44 -07:00

test_skill_parser.py

fix: PR #118 round 2 — null-safe parser, docs, consistency

2026-03-17 16:09:06 -07:00

test_skill_resources_api.py

feat: skill resources — API, admin UI, runtime injection, and SDK

2026-03-17 01:26:44 -07:00

test_skill_resources_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_skill_scanner.py

feat: skill scanner — evaluate SKILL.md content at install time

2026-03-16 15:45:21 -07:00

test_skill_sources.py

fix: update test_skill_sources for 5-tuple _parse_github_url

2026-03-17 01:26:44 -07:00

test_skills.py

feat: add priority column for skill ordering control (#144)

2026-03-21 00:53:51 -07:00

test_storage_registry.py

Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)

2026-03-03 22:57:34 -08:00

test_storage_sqlite.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_structured_memory_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_structured_memory.py

feat: structured memory system — typed/scoped memories with BM25 rele… (#53)

2026-03-13 21:21:09 -07:00

test_tool_policies_enforcement.py

review: fix mypy, tighten env scrub, bridge storage safety

2026-03-23 14:55:14 -07:00

test_tool_policy.py

feat: MCP resource and prompt discovery with read_resource tool (#44)

2026-03-12 14:49:58 -07:00

test_tool_search.py

chore: remove dead code and fix noisy JWT test warnings

2026-03-17 17:02:25 -07:00

test_tools_schema.py

feat: eval pipeline improvements + tool description optimization (#174)

2026-03-25 11:58:10 -07:00

test_user_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_watch_storage.py

feat: add PostgreSQL CI integration tests (#156)

2026-03-23 11:11:40 -07:00

test_watch.py

feat: watch tool — periodic command polling within workstreams (#36)

2026-03-10 08:18:28 -07:00

test_web_search.py

review: update stale duckduckgo-search references to ddgs

2026-03-23 16:09:03 -07:00

test_webui_content.py

refactor: move bridge content buffer to server-side single source of truth

2026-03-15 15:04:22 -07:00

test_workstream.py

feat: output guard — evaluate tool results before they enter context (#109)

2026-03-16 16:22:10 -07:00