mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
723cad24bb
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting Replace flat key-value memories table with structured_memories (migration 014). Four memory types (user/project/feedback/reference), three scopes (global/workstream/user). Consolidate remember/recall/forget into two tools: memory (action-based: save/search/delete/list) and recall (conversation history only). BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5 memories for system message injection based on conversation context. Metacognitive prompting injects ephemeral nudges after corrections, tool denials, workstream resume, and completion signals. Scope isolation enforced: system message injection and nudge counts filtered to visible memories only (global + current workstream + authenticated user). User scope requires authentication. Content capped at 32KB. ILIKE/LIKE metacharacters escaped in both backends. 113 new tests (2053 total). * fix: CI failure + copilot review feedback - Fix time.monotonic() cooldown: use None sentinel instead of 0.0 default (monotonic clock starts at boot, not epoch — fresh CI runners have uptime < 300s so cooldown check always triggered) - Catch sa.exc.IntegrityError specifically in upsert instead of broad Exception (copilot review) - Preserve existing description/type on upsert when caller doesn't explicitly set them (copilot review) - Add last_accessed + access_count columns to schema/migration for future LRU/LFU eviction support
143 lines
5.5 KiB
Plaintext
143 lines
5.5 KiB
Plaintext
@startuml
|
||
!theme plain
|
||
title Turnstone — Tool Execution Pipeline (Three Phases)
|
||
|
||
start
|
||
|
||
partition "Phase 1: Prepare" #E8F5E9 {
|
||
:Receive tool_calls list from LLM response;
|
||
|
||
while (more tool_calls?) is (yes)
|
||
:Extract call_id, func_name, raw_args;
|
||
|
||
if (json.loads(raw_args) succeeds?) then (yes)
|
||
:parsed_args = JSON dict;
|
||
else (no)
|
||
:Fallback 1: regex extraction;
|
||
if (regex found keys?) then (yes)
|
||
:parsed_args = extracted dict;
|
||
else (no)
|
||
:Fallback 2: bare string →\nPRIMARY_KEY_MAP[func_name];
|
||
endif
|
||
endif
|
||
|
||
:Dispatch to _prepare_{func_name}();
|
||
|
||
note right
|
||
**Dispatch table (17 tools):**
|
||
┌───────────────┬──────────────────┐
|
||
│ Tool │ Needs Approval? │
|
||
├───────────────┼──────────────────┤
|
||
│ bash │ ✓ Yes │
|
||
│ read_file │ ✗ Auto-approve │
|
||
│ write_file │ ✓ Yes │
|
||
│ edit_file │ ✓ Yes │
|
||
│ search │ ✗ Auto-approve │
|
||
│ math │ ✗ Auto-approve │
|
||
│ man │ ✗ Auto-approve │
|
||
│ web_fetch │ ✗ Auto-approve │
|
||
│ web_search │ ✗ Auto-approve │
|
||
│ tool_search │ ✗ Auto-approve │
|
||
│ task │ ✓ Yes │
|
||
│ plan │ ✓ Yes │
|
||
│ memory │ ✗ Auto-approve │
|
||
│ recall │ ✗ Auto-approve │
|
||
│ notify │ ✗ Auto-approve │
|
||
│ read_resource │ ✓ Yes │
|
||
│ use_prompt │ ✓ Yes │
|
||
├───────────────┼──────────────────┤
|
||
│ mcp__* │ ✓ Yes (external) │
|
||
└───────────────┴──────────────────┘
|
||
end note
|
||
|
||
:Build item dict:
|
||
{call_id, func_name, header,
|
||
preview, needs_approval,
|
||
approval_label, execute: Callable};
|
||
endwhile (no)
|
||
}
|
||
|
||
partition "Phase 2: Approve" #FFF3E0 {
|
||
if (any items need approval?) then (yes)
|
||
:_emit_state("attention");
|
||
:ui.approve_tools(items);
|
||
|
||
note right
|
||
**auto_approve check is handled
|
||
internally by ui.approve_tools()**
|
||
|
||
**TerminalUI**: Print headers/previews,
|
||
prompt [y/n/a, optional message]
|
||
If user chose "always":
|
||
Set ui.auto_approve = True
|
||
(auto-approve all future tools in this session)
|
||
**WebUI**: Enqueue approve_request,
|
||
block on _approval_event.wait()
|
||
**NullUI**: Return (True, None)
|
||
end note
|
||
|
||
if (user approved?) then (yes)
|
||
:_emit_state("running");
|
||
else (denied)
|
||
:Mark all pending items as denied;
|
||
:denial_msg = "Denied by user";
|
||
:_emit_state("running");
|
||
endif
|
||
else (all auto-approved)
|
||
:ui enqueues tool_info event\n(no blocking);
|
||
endif
|
||
}
|
||
|
||
partition "Phase 3: Execute" #E3F2FD {
|
||
:_check_cancelled();
|
||
note right: Cancellation checkpoint:\nraises GenerationCancelled if\ncancel event is set
|
||
if (single tool call?) then (yes)
|
||
:Execute sequentially:\nrun_one(items[0]);
|
||
else (multiple)
|
||
:Execute in parallel:\nThreadPoolExecutor(max_workers=4)\npool.map(run_one, items);
|
||
endif
|
||
|
||
note right
|
||
**run_one(item):**
|
||
if item.error → return error string
|
||
if item.denied → return denial message
|
||
else → item["execute"](item)
|
||
├─ _exec_bash: subprocess.run(["bash", script.sh])
|
||
├─ _exec_read_file: open().readlines() or _exec_read_image (base64)
|
||
├─ _exec_write_file: makedirs + write
|
||
├─ _exec_edit_file: find_occurrences + replace
|
||
├─ _exec_search: grep subprocess
|
||
├─ _exec_math: sandboxed subprocess
|
||
├─ _exec_man: man/info subprocess
|
||
├─ _exec_web_fetch: httpx.get + LLM summary
|
||
├─ _exec_web_search: Tavily API POST (fallback for local models)
|
||
├─ _exec_tool_search: BM25 search + expand_visible()
|
||
├─ _exec_task: _run_agent(TASK_AGENT_TOOLS)
|
||
├─ _exec_plan: _run_agent(AGENT_TOOLS, read-only)
|
||
├─ _exec_notify: HTTP POST to channel gateway
|
||
├─ _exec_memory: structured memory save/search/delete/list
|
||
├─ _exec_recall: conversation history FTS5 search
|
||
├─ _exec_read_resource: MCPClientManager.read_resource_sync()
|
||
├─ _exec_use_prompt: MCPClientManager.get_prompt_sync()
|
||
└─ _exec_mcp_tool: MCPClientManager.call_tool_sync()
|
||
end note
|
||
|
||
:Collect results: [(call_id, output), ...];
|
||
|
||
:_truncate_output() on each result\n(max context_window × chars_per_token × 0.5 chars\ndefault: ~context_window × 2 chars);
|
||
|
||
:bash: ui.on_tool_output_chunk(call_id, line) per stdout line;
|
||
:ui.on_tool_result(call_id, name, output) for each;
|
||
|
||
if (plan tool was executed?) then (yes)
|
||
:ui.on_plan_review(output);
|
||
:Block for user review/feedback;
|
||
endif
|
||
}
|
||
|
||
:Return (results, user_feedback);
|
||
|
||
stop
|
||
|
||
@enduml
|