Commit Graph

5 Commits

Author SHA1 Message Date
Patrick Buckley 723cad24bb feat: structured memory system — typed/scoped memories with BM25 rele… (#53)
* feat: structured memory system — typed/scoped memories with BM25 relevance and metacognitive prompting

Replace flat key-value memories table with structured_memories (migration 014).
Four memory types (user/project/feedback/reference), three scopes
(global/workstream/user). Consolidate remember/recall/forget into two tools:
memory (action-based: save/search/delete/list) and recall (conversation
history only).

BM25 relevance scoring (extracted to turnstone/core/bm25.py) selects top-5
memories for system message injection based on conversation context.
Metacognitive prompting injects ephemeral nudges after corrections, tool
denials, workstream resume, and completion signals.

Scope isolation enforced: system message injection and nudge counts filtered
to visible memories only (global + current workstream + authenticated user).
User scope requires authentication. Content capped at 32KB. ILIKE/LIKE
metacharacters escaped in both backends.

113 new tests (2053 total).

* fix: CI failure + copilot review feedback

- Fix time.monotonic() cooldown: use None sentinel instead of 0.0
  default (monotonic clock starts at boot, not epoch — fresh CI
  runners have uptime < 300s so cooldown check always triggered)
- Catch sa.exc.IntegrityError specifically in upsert instead of
  broad Exception (copilot review)
- Preserve existing description/type on upsert when caller doesn't
  explicitly set them (copilot review)
- Add last_accessed + access_count columns to schema/migration for
  future LRU/LFU eviction support
2026-03-13 21:21:09 -07:00
Patrick Buckley 1295919613 fix: simplify conversation storage — atomic assistant rows with tool_… (#51)
* fix: simplify conversation storage — atomic assistant rows with tool_calls JSON

Replace the denormalized storage model (separate rows for assistant
content, tool_call, tool_result) with atomic assistant rows carrying
tool_calls as a JSON column. Eliminates the 100-line heuristic
reconstruct_messages function and its cross-turn merge bug.

Schema: add tool_calls TEXT column to conversations (migration 013).
Migration backfills existing data — merges tool_call rows into their
parent assistant row as JSON, renames tool_result to tool, deletes
consumed tool_call rows.

Session save path: assistant content + tool_calls saved in one
save_message call before tool execution (crash resilient). Tool
results saved as role="tool".

Extract shared storage utilities to _utils.py: row_to_dict, mutable
field frozensets, reconstruct_messages. Both backends import from
_utils — PostgreSQL no longer depends on _sqlite.py.

Includes denied/blocked tool call badge fix on resume: _build_history
detects denied results and propagates flag to parent assistant entry.
Frontend uses flag for correct badge-denied rendering. Denied tools
visually muted. role="status" on badges for accessibility.

Net -45 lines. 8 new tests for reconstruction, all 1914 tests pass.

* fix: migration 013 uses parameterized deletes and ordered downgrade

- DELETE of consumed tool_call rows now uses parameterized batches
  (chunks of 500) instead of string interpolation
- Downgrade rebuilds via temp table to preserve chronological id
  ordering when re-inserting tool_call rows
2026-03-13 14:26:05 -07:00
Patrick Buckley fb190f8977 Normalize session_id into ws_id as sole persistent identity (#29)
* Normalize session_id into ws_id as sole persistent identity

Eliminate the separate session_id concept. The workstream ID (ws_id) is
now the single identity used for both real-time routing and conversation
persistence, removing a layer of indirection that was 1:1 in practice
and buggy on resume (stale pointers, orphaned rows).

Schema changes (migration 006):
- Drop sessions table; add alias/title columns to workstreams
- Rename conversations.session_id → ws_id
- Rename session_config table → workstream_config (ws_id column)
- Data migration remaps existing conversations to ws_id

Storage/API renames:
- register_session → register_workstream (already existed, merged)
- save_message/load_messages now keyed by ws_id
- resolve_session → resolve_workstream
- ChatSession.session_id property → ws_id
- ChatSession.resume_session() → resume()
- resume_session field → resume_ws
- SessionResumedEvent → WorkstreamResumedEvent
- /api/sessions → /api/workstreams/saved
- /sessions slash command → /workstreams
- --session-retention-days → --retention-days

Channel eviction recovery simplified: reuses old ws_id directly
instead of get_session_id_by_ws() reverse lookup.

* Fix Copilot review feedback: stale session wording in docs, regenerate OpenAPI spec

- docs/channels.md: "resumes the session" → "resumes the workstream",
  "Session resumed:" → "Resumed:", "old session was pruned" → "old
  workstream was pruned"
- docs/api-reference.md: "Each session object" → "Each saved workstream
  object", field descriptions updated, removed stale node_id field
- sdk/typescript/openapi-server.json: fully regenerated from Python
  models — removes all stale session_id properties from WorkstreamInfo,
  DashboardWorkstream, CreateWorkstreamResponse schemas
2026-03-07 12:49:07 -08:00
Patrick Buckley a20a058c59 Add cluster-scale schema, fix console proxy UX, harden SDK sync runner (#22)
* Add cluster-scale schema, fix console proxy UX, harden SDK sync runner

Schema redesign for multi-node deployments:
- New `workstreams` table with node_id, state, lifecycle tracking
- Add node_id + ws_id columns to sessions table with indexes
- Full UUID (32 hex) for session_id and ws_id (was truncated 12/8)
- Server generates and owns node_id, bridge retrieves via /health
- Bridge retries with exponential backoff, fatal on auth errors
- WorkstreamManager persists workstreams and state changes to storage
- /health endpoint exposes node_id for bridge discovery

Console proxy UX fixes:
- Remove duplicate turnstone branding from proxy banner
- Same-tab navigation for Open Node UI and workstream deep links

SDK _SyncRunner fix:
- Sentinel pattern for StopAsyncIteration across thread boundary

Remove misplaced PNGs from docs/diagrams/ (correct copies in png/ subdir).

* Address PR #22 review feedback

- Fix CLI session_factory signature (ws_id param) — CI typecheck failure
- First-phase eviction in create() now calls _cleanup_ui + record_eviction
- close() persists "closed" state to storage via update_workstream_state
- Fix noqa comment in test to pragma: no cover
2026-03-04 06:04:59 -08:00
Patrick Buckley 2b58c127b1 Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
2026-03-03 22:57:34 -08:00