ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity
1,818 Commits 12 Branches 135 Tags
main
Commit Graph

6 Commits

Author SHA1 Message Date
Patrick Buckley dbf389783e refactor(personas): file-backed built-in prompts, explicit source column 
Built-in persona base prompts move from inline DB text / base.md into
prompts/personas/<slug>.md — code-owned, PR-reviewable, drift-proof.
base.md / base_coordinator.md become personas/engineer.md / orchestrator.md.

Prompt source is now explicit in storage instead of inferred in app logic:
a new base_prompt_file column plus CHECK (base_prompt IS NOT NULL OR
base_prompt_file IS NOT NULL) — two nullable columns, never both empty.
Resolution is a coalesce (base_prompt else load(base_prompt_file)), frozen
into the workstream stamp at creation. base_prompt_file marks a persona as
built-in (code-only, un-archivable); an operator override on a built-in is
allowed and wins over the file. "Inherit the kind default" is a
workstream-creation act (is_default), not a persona-row state.

Migration 063:
- seeds reference their file (base_prompt NULL); no runtime file reads —
  the backfill's frozen prompt text is inlined as a point-in-time snapshot
  so migration history stays self-contained and reproducible.
- every existing workstream is stamped by kind (creative -> writer, else
  the kind default), set-based (INSERT..SELECT via temp tables) with the
  persona column added after the bulk writes to shorten its lock window.

Storage guards (both backends): operators must supply base_prompt;
built-ins can't be archived or have base_prompt_file set via the API;
clearing an operator persona's only source is rejected.

Follow-ups reviewed alongside (#756): soft-set visibility docstring scoped
to per-process; _apply_persona_snapshot / _current_persona_snapshot own the
stamp round-trip; spawn approval-header args (skill/name/target_node)
flattened+capped like persona; server-side tool injection generalized to
replace-only (client-def gated, incl. the xAI include forwarding). Seed
copy revised (researcher soft; de-costumed prose; engineer de-biased).
New test_schema_parity asserts create_all matches the alembic head.

Closes #683 groundwork; ruff + strict mypy clean, full suite green.
 2026-07-03 00:29:26 -07:00
Patrick Buckley 2329cb8ad5 feat(webui): persona picker, Service Hatch shelf, labels, row-emitter sweep, kind-id reclamation 
Creation surfaces: the console launcher, the server webui new-workstream
dialog, and the dashboard composer all gain a Persona select fed by the
shared personas.js data layer (module cache + fingerprint + never-reject
fetch + window.TurnstonePersonas bridge, cloned from projects.js), kind-
filtered with the kind default preselected so a zero-touch launch is
byte-identical to today.

Authoring: a Personas tab in the console Manage surface (Governance
group, persona.read-gated) with a Service Hatch shelf exposing exactly
the four levers — base prompt, tool-visibility checklist (kind inventory
+ free-text row for MCP/dynamic names; tool_search membership decides
soft vs hard), MCP and memory toggles — plus kinds, default flip, and
archive.  No delete action anywhere (archive-only lifecycle).

The workstream wears it: SavedColumns.persona() on both saved tables,
hover/aria labels on the rail rows (raw slug fallback keeps archived
personas labelling their workstreams), and the full row-emitter sweep —
storage projections (list_workstreams tail, get_workstreams_batch,
list_workstreams_with_history) on both backends, the server dict
builders and ws_created events, both _coordinator_rows lanes, the
collector delta + pseudo-node paths, and the console cluster-create
proxy that rebuilds its body.

Naming reclamation: the launcher kind-toggle ids/classes that squatted
on 'persona' (launcher-personas, persona-coordinator/-interactive,
.persona-btn/.persona-led/.persona-tag) are renamed to kind-* before
'persona' becomes user-facing vocabulary, along with the prompts-module
and test wording that used persona to mean kind.
 2026-07-03 00:29:26 -07:00
Patrick Buckley 64d5205dd6 perf(session): split metacognitive nudges out of the system message 
The system-message developer block was rebuilt every turn with two
unstable inputs: minute-precision current_datetime in the middle of
the composed prefix, and _pending_nudge entries appended-then-cleared
at the bottom. Both invalidated prompt-cache reuse on Anthropic /
OpenAI for the entire prefix, every turn.

current_datetime now rounds to the top of the hour. Nudges no longer
ride on the system message at all — they drain through two channels:

- tool_error and repeat ride the existing tool-result <system-reminder>
  envelope via a new MetacognitiveAdvisory ToolAdvisory subtype, drained
  in _collect_advisories alongside GuardAdvisory and UserInterjection.
- correction, denial, resume, start, completion splice as
  <system-reminder> blocks at the trailing edge of the next user
  message via a new _splice_pending_user_advisories helper.

User content passes through escape_wrapper_tags before concatenation
so a user typing literal <system-reminder> tags cannot fabricate an
envelope; the same escape now runs on advisory.render() output inside
wrap_tool_result for defense-in-depth across all advisory types.

Cancel handlers (GenerationCancelled / KeyboardInterrupt / bare
Exception) now clear _pending_tool_advisories alongside the existing
_flush_queued_messages so a queued nudge from an aborted batch
cannot leak into the next generation.

Visibility ping ([metacognition: nudge injected — ...]) preserved at
both new attach points via a single _emit_nudge_ping helper.

Also adds a "Session kind" line (interactive | coordinator) to the
composed Session Context so the model can see which manager hosts
its session.

Tests: 4847 passing (+7 new in TestMetacognitiveBuffers and
test_tool_advisory). ruff + mypy clean.
 2026-04-28 19:51:00 -07:00
Patrick Buckley dea2729292 refactor(coordinator): rename task_list → tasks, doc/prompt sweep (#437) 
Four themes from a coordinator-feature shakedown:

1. Correctness fixes (return shapes / examples / behavior)

   - tools_coordinator.md: drop fake skill names from spawn examples;
     fix wrong kwarg ``node_id=`` → ``target_node=``.
   - wait_for_workstream.json: document ``message`` + ``truncated``
     per-ws fields (always enriched in the client; the JSON shape
     lagged the docstring).
   - cancel_workstream.json: document the conditional ``dropped``
     payload — ``was_running`` always present when ``dropped`` is,
     ``pending_approval`` and ``queued_messages`` conditional sub-shapes.
   - spawn_workstream.json: document full return shape including
     ``routing_strategy ∈ {rendezvous, target_node, resume}`` and
     ``status``.
   - close_all_children.json: clarify ``skipped`` covers BOTH
     hard-deleted children AND already-closed-and-evicted children
     (wire shape doesn't distinguish); drop incorrect "echoed back
     in response" claim — server returns ``{status, closed, failed,
     skipped}``, never echoes ``reason``.
   - console/server.py: comment in ``_fanout_on_children`` clarifying
     that the 400 "No session" branch fires for cancel-cascade
     callers and is unreachable from close_all_children (close
     handler 404s instead).
   - coordinator_client._utc_now_iso(): switch to bare ISO format
     matching the rest of the storage row format used in the codebase.

2. Tightened the 11 longest tool descriptions (~23% cut on the
   coord set). Removed ALL-CAPS emphasis, normalised em-dashes,
   dropped informal phrasing. No new claims.

3. Removed static approval annotations from descriptions.
   Approval is governed at runtime by the unified ``approve_tools``
   body and admin-defined ``tool_policies`` (#436); static
   "Auto-approved" / "Approval required" / per-action approval
   tags become a stale signal. Field names (``pending_approval``)
   and operational verb behaviour ("cancel unblocks pending
   approvals") stay.

4. Renamed ``task_list`` coord tool → ``tasks``. The previous name
   compounded the bare word ``task`` (which collides with chat-template
   channels on local models — same reason ``task_agent`` carries
   the suffix); the plural form sidesteps the collision and reads
   more accurately, since the tool acts on the whole list rather
   than a single task. Sweep covers tool JSON, Python methods (5
   client methods + 2 session methods + 1 helper + 1 constant),
   audit event name (``task_list.update`` → ``tasks.update``), log
   tag (``task_list.corrupt_envelope`` → ``tasks.corrupt_envelope``),
   frontend SSE event matcher, prompts, docs, and tests. CHANGELOG
   entry added.

Plus: dropped the ENV block (Output Environment / Available
rendering / Formatting principles) from coordinator system
prompts. Coordinators orchestrate rather than render rich output
to the user, so the rendering capability matrix is not actionable
for them. Coord prompt drops ~29% (6309 → 4493 chars).

SDK regeneration via ``generate-types.py`` updates both
``openapi-console.json`` (the rename's downstream change) and
``openapi-server.json`` (PR #436 drift — its merge added
``pending_approval_detail`` + ``recent_auto_approvals`` fields to
the Python schemas but didn't regenerate the JSON artifact).

## Behavior changes (operator-visible)

- Audit event name: ``task_list.update`` → ``tasks.update``.
  Audit dashboards / SIEM filters / log greps that pinned the old
  prefix should update.
- SSE ``tool_result`` events now ship ``name="tasks"`` for the
  scratchpad tool. The bundled coord-tree UI is updated atomically;
  external consumers reading SSE events by tool name need to update.
- Existing task envelopes in production storage have ``+00:00``
  timestamps from the old ``_utc_now_iso``. New writes are bare;
  old rows are not backfilled. Within an envelope you may briefly
  see mixed formats until each row is re-touched. No code path
  string-compares timestamps within an envelope, so this is
  cosmetic.

## Validation

- ``ruff check`` + ``ruff format --check`` clean
- ``mypy turnstone/`` clean (175 source files)
- ``pytest -m "not live"`` — 4679 passed, 3 deselected
 2026-04-27 22:51:31 -07:00
Patrick Buckley d3f6514e11 feat(ui): phase 4 — chat-UX unification + coordinator-first console landing (#373) 
* feat(ui): phase 4 — chat-UX unification + coordinator-first console landing

Phase 4 unifies the three turnstone UIs (server-node chat, console
dashboard, coordinator page) around a shared design-system layer,
promotes coordinator sessions to first-class citizens on the console
landing, and folds the chat-view itself onto a shared vocabulary so
the two chat pages no longer reinvent messages / approvals / composer /
header / sidebar chrome from scratch.

## Shared static consolidation

- turnstone/shared_static/renderer.js — consolidates the two copies
  (ui/static/ + console/static/coordinator/) into one.  Adds
  streamingRender / streamingRenderFinalize helpers with
  requestAnimationFrame coalescing + per-element buffer cache so both
  chat views re-render the streamed markdown smoothly without the
  prior "plain-text → final pop" on the coordinator page and without
  thrashing renderMarkdown + DOM replacement faster than the paint
  cycle.  renderMarkdown stays the trust boundary for innerHTML
  assignment (escapeHtml internal); postRenderMarkdown (syntax
  highlighting, mermaid, KaTeX) is deferred to finalize.
- turnstone/shared_static/ui-base.css — flat form-control + button +
  state-glyph + pill + panel vocabulary on top of base.css.  Sizes in
  px to match the 11/12/13px scale used elsewhere.  Namespace rubric
  documented inline (.ui-* shared controls; .dash-* dashboard legacy;
  .ts-* chat vocabulary; page-local stays unprefixed).
- turnstone/shared_static/chat.css (new) — chat-view component
  vocabulary: .ts-msg (user / assistant / reasoning / tool / error /
  info), .ts-msg-actions floating toolbar, .ts-approval (inline +
  batch layout hooks sharing a visual language), .ts-verdict-badge,
  .ts-composer shell, .ts-header shell, .ts-sidebar shell.  Mobile +
  reduced-motion covered.

## Interactive server UI migration

- turnstone/ui/static/app.js — dual-class adoption of .ts-msg + .ts-
  approval + .ts-composer + .ts-msg-actions alongside existing class
  names so feature-specific rules (.msg-user-text, .msg-queued,
  .msg-editing, .msg-action-btn toolbar, attachment chips, verdict
  details, media embeds, plan inline) keep working while the shared
  chat.css baseline takes over padding / border / typography.
- Left-aligned user messages: .msg-user loses align-self: flex-end
  and .msg-assistant loses align-self: flex-start.  Both roles now
  render as single-column blocks distinguished by left-border colour
  (amber for user, neutral for assistant, dashed for reasoning,
  mono + code-bg for tool, red for error) per the locked design.
- style.css trimmed: .msg / .msg-info / .msg-error baseline rules
  dropped (chat.css provides); all other feature rules intact.
- index.html links /shared/chat.css and tags the header with
  .ts-header + .ts-header-title.

## Coordinator page migration

- coordinator.js appendMsg drops the visible .role-label <div> per
  the hybrid no-labels design, preserving the role text on
  data-ts-role + aria-label so screen readers and SSE dedup-by-call-
  id continue to see meaningful labels.  Adds .ts-msg + .ts-msg--*
  variants + .ts-msg-body onto the existing .coord-msg / .coord-body
  elements.
- coordinator/index.html adopts .ts-header, .ts-header-title,
  .ts-header-spacer, .ts-header-status on the header; .ts-approval +
  .ts-approval--batch on the pinned bar with .ts-approval-btn
  variants on the buttons; .ts-composer + .ts-composer-input +
  .ts-composer-send on the composer; .ts-sidebar + .ts-sidebar-
  section + .ts-sidebar-section-heading on the children + tasks
  sidebar.  Inline <style> pared from ~300 to ~130 lines — only
  genuinely coordinator-specific layout (flex wiring, sidebar list
  rows, mobile accordion breakpoint) remains.
- Nits fixed along the way: .ch-row .glyph-thinking recoloured cyan
  to match the shared .ui-glyph vocabulary; .task-row .status-done
  lost its 0.7 opacity (colour already signals done; opacity
  reduced contrast for no gain).

## Console landing + admin panel redesign (phase 4 scope-expansion)

Replaces the node-list-first console landing with a coordinator-first
layout on a new #view-home pane:

- #coord-composer-panel — persistent "Start a new coordinator task"
  composer (textarea + optional name + skill dropdown + submit).
  Permission-gated on admin.coordinator (same rule the +coordinator
  header button uses).  Pre-probes GET /v1/api/coordinator on init
  and after login (bug-1 fix) so a 503 (no coordinator.model_alias
  resolvable) surfaces as a remediation banner linking to Admin →
  Models instead of failing on submit.  Probe gates on r.ok instead
  of r.status !== 503 (bug-2 fix) so auth/permission errors don't
  incorrectly flip the banner to ready.  Composer and modal share a
  _createCoordinator helper (q-1 fix) — POST + redirect + error-
  handling tail is not forked.
- #active-coordinators — SSE-driven list of kind=="coordinator"
  workstreams, rendered through the shared _renderWsRow helper so
  state glyphs + child-count badges match the existing tree view.
- #cluster-summary-compact — one-line aggregate.  Clicking expands
  into the legacy #view-overview via showOverview() so deep-link
  callers of ?view=overview / ?view=node / ?view=filtered keep
  working unchanged.

View switching consolidated into a _setLandingView helper so every
show* / drillDown* function toggles the four landing panes through
one call path.  Default currentView flipped from "overview" to
"home"; popstate + init history.replaceState land on {view: "home"}.

patchClusterState preserves kind / parent_ws_id / user_id on
ws_created events (phase 3 invariant) so the active-coordinators
list picks up new coordinators immediately without a snapshot
refetch.

Header H1 is now a home link so operators have a single-click path
back to the coordinator landing from any drill-down / admin view.

## Design polish (review pipeline fixes)

- .home-panel-title dropped from 13px/accent to 11px/fg-dim so it
  sits in the same heading tier as .ui-section-heading / .dash-
  header-title / .home-section-title instead of outweighing them
  (dsn-3).
- .home-composer-banner recoloured from amber-on-amber-glow to
  bg-surface + 1px yellow border + fg-bright text + accent link
  with thicker underline (dsn-1).
- .ui-pill--done dropped the 0.75 opacity — colour signals done,
  opacity reduced contrast for no gain (dsn-6).
- .ui-heading fleshed out with --sm/--md/--lg tiers so the utility
  actually conveys size (dsn-12).
- ui-base.css size scale moved from rem to px matching 11/12/13px
  (dsn-2).

* fixup: address Copilot feedback on PR #373

- admin.js: drop the stale `#view-overview` display:none mutation in
  showAdmin.  #view-overview is now nested inside #view-home and
  toggled via the `hidden` attribute; inline display:none here would
  stick after returning to home and suppress the cluster-details
  expand.
- app.js: reword the _renderHomeView token-bucket fingerprint comment
  to match the actual `Math.floor(tokens / 100)` bucketing — the
  prior comment said "thousands / sub-thousand drift".
 2026-04-17 18:53:14 -07:00
Patrick Buckley 3ce66960bc feat: modular system message composition with admin prompt policies (… (#267) 
* feat: modular system message composition with admin prompt policies (#267)

Replace the monolithic persona+tools block in _init_system_messages()
with a modular composition harness (turnstone/prompts/). System messages
are now assembled from five typed layers: BASE (persona), ENV (client
surface — web/cli/chat), CONTEXT (datetime, timezone, username), TOOLS
(usage patterns), and POLICIES (behavioral rules with tool gating).

Prompt policies are admin-managed via a new Prompts tab in the Governance
group (CRUD with modal forms, tool gating, priority ordering, enable/disable).
DB policies override file-based defaults by name; file-based policies serve
as deployment defaults. Migration 031 adds the prompt_policies table.

ClientType is threaded end-to-end from channel adapters through the SDK,
HTTP API, WorkstreamManager, and session factory to ChatSession. Discord
sessions now receive chat-optimized formatting (no tables, no Mermaid,
concise output) instead of the web UI's rich markdown instructions.

* fix: address CI failures and Copilot review feedback

- Add client_type param to CLI session_factory (mypy protocol match)
- Add prompt policy CRUD to PostgreSQL backend (test-postgres CI)
- Fix ClientType resolution: compare against enum values, not members
- Fix null client_type coercion (body.get returns None, not "")
- Use local time with astimezone() instead of UTC with local tz name
- Sanitize tool_gate in update endpoint (coerce null to empty string)
 2026-03-31 15:25:55 -07:00