Commit Graph

3 Commits

Author SHA1 Message Date
Patrick Buckley de64535221 Feat/eval improvements (#37)
* ci: add GitHub Release creation on tag push

* refactor: rename plan tool to create_plan

Rename plan → create_plan to resolve cross-provider tool selection
failures. Models consistently treated "plan" as a reasoning concept
rather than a callable tool. The new name is an unambiguous verb+noun
action. Also rename the parameter from prompt → goal for clarity,
add web_search to the default system prompt tool patterns

* feat: eval harness improvements inspired by autoresearch patterns

Major enhancements to turnstone-eval:

- Per-test timeout (--test-timeout, default 300s) and suite timeout
  (--suite-timeout) prevent stuck runs from blocking the suite
- Fast-fail skips remaining runs after ceil(n/2) consecutive zeros
- Summary table with colored PASS/WEAK/FAIL and append-only TSV output
- Progress reporting with running pass rate, token count, and ETA
- Parallel test execution via ProcessPoolExecutor (--parallel N)
- Per-role model assignment: test/optimizer/observer can use different
  models and providers (--optimizer-model, --observer-model, etc.)
  with auto-detection from base URL
- Improved optimizer and observer system prompts with structured
  failure-mode diagnosis, keep/discard rules, and trend analysis
- Fixed token counting (prompt tokens use last-turn value, not sum)
- Added math-calculation and web-search-query test cases
- Fixed multi-file-edit test (both files now contain the target string)

* fix: address Copilot review feedback

- Revert prompt token counting to sum (reflects billed usage)
- Add tool_args to fast-fail skipped run dicts for schema consistency
- Align approval_label with func_name ("create_plan")
- Add timeout to future.result() in parallel path (test_timeout + 30s)
- Document thread-leak trade-off on serial timeout path
2026-03-10 13:06:39 -07:00
Patrick Buckley 9be155b97a Quality overhaul: code tooling, CI/CD, architecture diagrams, UI rede… (#1)
* Quality overhaul: code tooling, CI/CD, architecture diagrams, UI redesign, and legacy cleanup

- Add ruff (lint+format) and mypy (strict) with zero errors across 37 source files
- Add GitHub Actions CI (lint, typecheck, test matrix 3.11/3.12/3.13) and PyPI publish workflow
- Create 12 PlantUML architecture diagrams with PNG renders covering all subsystems
- Refresh README and docs with badges, diagram links, and current descriptions
- Refactor test_server_live.py with mock streaming helpers for deterministic CI testing
- Update dependencies to current versions (openai>=2.24, httpx>=0.28, redis>=7.2)

Console dashboard:
- Move state indicators from top cards to fixed bottom status bar with cluster metrics
- Replace flat 50-node list with hostname-prefix grouped nodes (expand/collapse, up to 1000)
- Apply "Instrument Panel" visual redesign: IBM Plex Mono + Outfit fonts, warm amber accent,
  LED glow state indicators, deep charcoal surfaces, WCAG AA contrast compliance
- Add render cache, stale indicator, active filter highlight, loading states

Server web UI:
- Apply matching Instrument Panel aesthetic for visual consistency with console
- Fix branding (pcode → turnstone), extract inline styles to CSS classes
- Rename pcode localStorage keys and history state to turnstone

Legacy cleanup:
- Remove persona-model-specific --persona flag and /persona slash command
- Remove model_identity from chat_template_kwargs (vLLM-specific mechanism)
- Refactor plan agent to use standard developer message instead of model_identity
- Remove dead code (unused date/has_tools variables, noqa suppressions)

* Fix CI typecheck: add mypy overrides for optional sympy/numpy imports

The math sandbox optionally imports sympy and numpy at runtime (try/except
ImportError). In CI these packages are not installed, so mypy raises
import-not-found rather than import-untyped. Add mypy overrides to
ignore missing imports for these optional dependencies.

* Fix Copilot review findings: ARIA role, status bar cache, and pulse opacity

- Change #node-table from role="tree" to role="list" and group elements
  from role="treeitem" to role="listitem" (proper ARIA semantics)
- Include currentView and currentFilter.state in renderStatusBar cache key
  so active pill highlight updates when switching views
- Align pulse animation to 0.35 opacity (already applied in CSS)
2026-03-02 16:55:12 -08:00
Patrick Buckley 0d6252dd7d Initial commit — turnstone multi-node AI orchestration platform. 2026-03-02 00:33:37 -08:00