ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity
Files
stable/1.5
turnstone/tests/test_structured_memory_storage.py
T
Patrick Buckley 2fd0c29a92 fix(memory): query-aware candidate selection + OR-of-terms search (#468) 
* fix(memory): query-aware candidate selection + OR-of-terms search

The system-message memory composition path used a recency-ordered
candidate set (`_list_visible_memories(limit=fetch_limit)`).  On
deployments with more than `fetch_limit` (default 50) visible
memories, BM25 only ever ranked the 50 most-recently-touched memories
— a relevant memory written months ago was silently invisible
regardless of how well it matched the recent context.  Multi-word
search at the SQL layer used AND-of-terms, killing recall on any
multi-word query without an exact field overlap.

## Functional changes

- `_init_system_messages` (`turnstone/core/session.py`): extract
  recent context first, then `_search_visible_memories(context)` to
  pull query-aware candidates.  Search hits below `fetch_limit` union
  with the recency list (deduped by memory_id) so the BM25 candidate
  pool is always a SUPERSET of the prior recency-only pool — even on
  noisy queries where the cap fills with stopwords, the recency-50
  the original bug surfaced still reaches BM25.  Empty context skips
  search entirely.  Candidate-selection logic extracted into
  `_select_memory_candidates`.

- `search_structured_memories` (PostgreSQL + SQLite): per-term
  clauses join with OR instead of AND.  A row matches if ANY term
  matches ANY of name/description/content.  Downstream BM25 narrows
  back down by relevance.

## Perf hardening

- Collapse the 1-3 fanned scope queries into a single SQL.  New
  backend methods `list_visible_structured_memories` /
  `search_visible_structured_memories` union the visibility scopes
  into one WHERE OR-group, so a composition rebuild now hits the DB
  at most twice (search + recency) instead of up to six times.

- Cap and normalize search terms.  Composition can hand a multi-KB
  pasted message to ILIKE-based search; without a cap, every distinct
  token would emit one unindexable predicate per scope-fanned query.
  `normalize_search_terms` (`storage/_utils.py`) de-dupes
  case-insensitively, drops <2-char tokens, and hard-caps at 16.

- Per-turn search cache.  `_init_system_messages` fires from many
  call sites within one turn (state transitions, MCP refresh, tool
  results) and the recent-context query is identical across them.
  Session-instance cache keyed by (query, mem_type, limit) absorbs
  the duplicates; invalidated in `_append_user_turn` and after
  memory save/delete tool actions.

- Stable secondary sort by `memory_id`.  `updated` is second-precision
  and `touch_structured_memories` can land a batch on identical
  timestamps; without a tie-breaker SQL returns rows in
  implementation-defined order, BM25 input shuffles, and the
  LLM-side prompt cache misses across calls.  All four backend ORDER
  BYs now break ties on `memory_id ASC`.

## Quality cleanups

- Coalesce `memory.search.term_count` + `memory.search.zero_results`
  into a single `memory.search` log carrying both `term_count` and
  `result_count`.
- New `memory.composition` log: source / candidates / injected.
- Promote a shared `make_chat_session` factory to `tests/_helpers.py`.
- Rename SQL builder local `extra` -> `scope_filters` for clarity.
- Add docstrings on `search_structured_memories` so the AND->OR flip
  survives future readers.

## Tests

Adds 20 tests across `tests/test_structured_memory.py`,
`tests/test_structured_memory_storage.py`, and
`tests/test_memory_relevance.py`: recency-ceiling regression,
empty-query fallback, sparse-match union, recency-preserved-when-
search-returns-noise (locks in the pool-superset invariant),
OR-of-terms on both backends, scope filtering preserved,
search-facade multi-word behavior, term-cap normalization, the new
visible-scope helpers (list + search + empty-scopes guard),
coord-scope composition isolation, end-to-end
`memory(action='search')` tool execution, per-turn cache hit +
invalidation, and stable ordering under tied `updated` timestamps.

Memory test sweep: 102/102.  Broader regression
(session, storage, coordinator, load_skill): 411/411.

* fix(memory): address Copilot review on PR #468

Three follow-ups from Copilot's inline review:

1. SUPERSET invariant violation (Copilot, session.py:5510).
   `(search_hits + extra)[:fetch_limit]` capped the union back down to
   fetch_limit, evicting the recency tail when search added distinct
   hits.  Recency tail is exactly where ancient-but-recently-touched
   memories live — the recall this PR is supposed to improve — so
   tail eviction recreated the bug for the narrow case where a query
   term fell off the 16-cap and the matching memory sat in
   recency[40-49].  Drop the cap; both halves are already SQL-capped
   at fetch_limit, so the union is at most 2 × fetch_limit (~100 with
   defaults).  BM25 over 100 candidates in pure Python is sub-ms;
   irrelevant recency fillers get score=0 and don't pollute ranking.
   Updates the docstring to actually be honest about the invariant.
   Adds `test_recency_tail_preserved_when_search_adds_distinct_hits`
   that locks the behavior in: 5 search hits + 10 recency = 15-item
   pool, every recency item present, source="union".

2. Unbounded `query.split()` in normalize_search_terms (Copilot,
   _utils.py:74).  `str.split()` allocates the full token list before
   the cap-after-16 break, so a 100KB pasted query did MB of throwaway
   work even though only 16 tokens entered SQL.  Switch to
   `re.finditer(r'\S+', query)` — streaming iterator, stops scanning
   at the first 16 normalized terms regardless of input size.

3. Misleading + unbounded log term_count (Copilot, session.py:8571).
   `len(item["query"].split())` had two problems: same unbounded
   split as #2, and the value reported the raw input token count
   rather than the normalized term count that actually hit the SQL
   WHERE clause — misleading metric for an operator trying to
   understand storage-side behavior.  Switch to
   `len(normalize_search_terms(item["query"]))` — accurate count, and
   bounded for free via #2.

Refuted: github-code-quality flagged `...` bodies in the new Protocol
methods as "statement has no effect."  False positive — `...` is the
canonical Protocol body convention, used 213 other times in the same
file.

Memory test sweep: 103/103.  Broader regression: 411/411.
2026-05-03 13:40:30 -07:00

274 lines
13 KiB
Python
Raw Permalink Blame History

"""Tests for structured memory storage backend operations."""
class TestCreateAndGet:
def test_create_and_get_by_id(self, backend):
backend.create_structured_memory("m1", "test_key", "desc", "project", "global", "", "data")
mem = backend.get_structured_memory("m1")
assert mem is not None
assert mem["name"] == "test_key"
assert mem["content"] == "data"
assert mem["type"] == "project"
def test_get_nonexistent(self, backend):
assert backend.get_structured_memory("nope") is None
def test_get_by_name(self, backend):
backend.create_structured_memory("m1", "mykey", "d", "project", "global", "", "val")
mem = backend.get_structured_memory_by_name("mykey", "global", "")
assert mem is not None
assert mem["memory_id"] == "m1"
def test_get_by_name_scoped(self, backend):
backend.create_structured_memory("m1", "key", "d", "project", "global", "", "g")
backend.create_structured_memory("m2", "key", "d", "project", "workstream", "ws1", "w")
g = backend.get_structured_memory_by_name("key", "global", "")
w = backend.get_structured_memory_by_name("key", "workstream", "ws1")
assert g["content"] == "g"
assert w["content"] == "w"
class TestUpdate:
def test_update_content(self, backend):
backend.create_structured_memory("m1", "k", "d", "project", "global", "", "old")
assert backend.update_structured_memory("m1", content="new")
mem = backend.get_structured_memory("m1")
assert mem["content"] == "new"
def test_update_nonexistent(self, backend):
assert not backend.update_structured_memory("nope", content="x")
def test_update_no_fields(self, backend):
backend.create_structured_memory("m1", "k", "d", "project", "global", "", "data")
assert not backend.update_structured_memory("m1", bogus="val")
def test_update_bumps_timestamp(self, backend):
backend.create_structured_memory("m1", "k", "d", "project", "global", "", "data")
old = backend.get_structured_memory("m1")["updated"]
import time
time.sleep(0.01)
backend.update_structured_memory("m1", content="new")
new = backend.get_structured_memory("m1")["updated"]
assert new >= old
class TestDelete:
def test_delete_existing(self, backend):
backend.create_structured_memory("m1", "k", "d", "project", "global", "", "data")
assert backend.delete_structured_memory("k", "global", "")
assert backend.get_structured_memory("m1") is None
def test_delete_nonexistent(self, backend):
assert not backend.delete_structured_memory("nope", "global", "")
def test_delete_scoped(self, backend):
backend.create_structured_memory("m1", "k", "d", "project", "workstream", "ws1", "data")
assert not backend.delete_structured_memory("k", "global", "")
assert backend.delete_structured_memory("k", "workstream", "ws1")
class TestList:
def test_list_all(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "user", "global", "", "2")
mems = backend.list_structured_memories()
assert len(mems) == 2
def test_list_by_type(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "user", "global", "", "2")
mems = backend.list_structured_memories(mem_type="user")
assert len(mems) == 1
assert mems[0]["name"] == "b"
def test_list_by_scope(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "project", "workstream", "ws1", "2")
mems = backend.list_structured_memories(scope="workstream")
assert len(mems) == 1
def test_list_respects_limit(self, backend):
for i in range(10):
backend.create_structured_memory(f"m{i}", f"k{i}", "", "project", "global", "", f"{i}")
mems = backend.list_structured_memories(limit=3)
assert len(mems) == 3
class TestSearch:
def test_search_by_name(self, backend):
backend.create_structured_memory("m1", "database_config", "", "project", "global", "", "pg")
backend.create_structured_memory("m2", "api_key", "", "project", "global", "", "secret")
results = backend.search_structured_memories("database")
assert len(results) == 1
assert results[0]["name"] == "database_config"
def test_search_by_content(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "postgresql host")
results = backend.search_structured_memories("postgresql")
assert len(results) == 1
def test_search_empty_lists_all(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "project", "global", "", "2")
results = backend.search_structured_memories("")
assert len(results) == 2
class TestCount:
def test_count_all(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "project", "global", "", "2")
assert backend.count_structured_memories() == 2
def test_count_by_scope(self, backend):
backend.create_structured_memory("m1", "a", "", "project", "global", "", "1")
backend.create_structured_memory("m2", "b", "", "project", "workstream", "ws1", "2")
assert backend.count_structured_memories(scope="global") == 1
assert backend.count_structured_memories(scope="workstream") == 1
class TestSearchOrOfTerms:
"""Verify that multi-word search uses OR-of-terms (any term matches → row included)."""
def test_single_matching_term_in_multi_word_query(self, backend):
"""Memory with content 'apple' found when query is 'apple banana cherry'."""
backend.create_structured_memory("m1", "apple_mem", "", "project", "global", "", "apple")
backend.create_structured_memory("m2", "other_mem", "", "project", "global", "", "grape")
results = backend.search_structured_memories("apple banana cherry")
names = {r["name"] for r in results}
assert "apple_mem" in names # matches "apple" — OR-of-terms keeps it
assert "other_mem" not in names # "grape" matches nothing in the query
def test_partial_overlap_across_memories(self, backend):
"""Each memory matches one of three terms; all three are returned."""
backend.create_structured_memory("m1", "alpha_doc", "", "project", "global", "", "alpha")
backend.create_structured_memory("m2", "beta_doc", "", "project", "global", "", "beta")
backend.create_structured_memory("m3", "gamma_doc", "", "project", "global", "", "gamma")
backend.create_structured_memory("m4", "unrelated", "", "project", "global", "", "delta")
results = backend.search_structured_memories("alpha beta gamma")
names = {r["name"] for r in results}
assert "alpha_doc" in names
assert "beta_doc" in names
assert "gamma_doc" in names
assert "unrelated" not in names # "delta" doesn't appear in the query
def test_scope_filter_preserved(self, backend):
"""OR-of-terms search still respects scope / scope_id filters."""
backend.create_structured_memory(
"m1", "ws1_note", "", "project", "workstream", "ws1", "info"
)
backend.create_structured_memory(
"m2", "ws2_note", "", "project", "workstream", "ws2", "info"
)
backend.create_structured_memory("m3", "global_note", "", "project", "global", "", "info")
results = backend.search_structured_memories("info", scope="workstream", scope_id="ws1")
names = {r["name"] for r in results}
assert "ws1_note" in names
assert "ws2_note" not in names
assert "global_note" not in names
def test_term_cap_normalizes_unbounded_query(self, backend):
"""A multi-KB query collapses to <= MAX terms (de-dupe + length filter)."""
backend.create_structured_memory("m1", "alpha_doc", "", "project", "global", "", "alpha")
backend.create_structured_memory(
"m2", "other_doc", "", "project", "global", "", "irrelevant"
)
# Build a noisy query: same word repeated, plus 1-char tokens that
# the normalizer drops, plus the actual signal "alpha".
noisy = " ".join(["x"] * 100 + ["alpha"] * 50)
results = backend.search_structured_memories(noisy)
names = {r["name"] for r in results}
assert "alpha_doc" in names
class TestVisibleStructuredMemories:
"""Single-query union helpers used by the composition path."""
def test_list_visible_unions_global_workstream_user(self, backend):
backend.create_structured_memory("m1", "g_note", "", "project", "global", "", "g")
backend.create_structured_memory("m2", "ws_note", "", "project", "workstream", "ws1", "w")
backend.create_structured_memory("m3", "u_note", "", "project", "user", "u1", "u")
backend.create_structured_memory("m4", "other_ws", "", "project", "workstream", "ws2", "x")
scopes = [("global", ""), ("workstream", "ws1"), ("user", "u1")]
rows = backend.list_visible_structured_memories(scopes)
names = {r["name"] for r in rows}
assert names == {"g_note", "ws_note", "u_note"} # ws2 excluded
def test_search_visible_unions_scopes_and_terms(self, backend):
backend.create_structured_memory("m1", "g_alpha", "", "project", "global", "", "alpha")
backend.create_structured_memory(
"m2", "ws_beta", "", "project", "workstream", "ws1", "beta"
)
backend.create_structured_memory(
"m3", "ws_other", "", "project", "workstream", "ws2", "alpha"
)
scopes = [("global", ""), ("workstream", "ws1")]
rows = backend.search_visible_structured_memories("alpha beta", scopes)
names = {r["name"] for r in rows}
assert "g_alpha" in names # global, matches "alpha"
assert "ws_beta" in names # ws1, matches "beta"
assert "ws_other" not in names # ws2 -> outside visibility
def test_visible_helpers_handle_empty_scopes(self, backend):
backend.create_structured_memory("m1", "anything", "", "project", "global", "", "x")
assert backend.list_visible_structured_memories([]) == []
assert backend.search_visible_structured_memories("x", []) == []
class TestStableOrderingOnTimestampTies:
"""When two memories share an `updated` timestamp, secondary sort on
memory_id keeps the order deterministic across calls.
`updated` is second-precision, and touch_structured_memories() can bump
a batch to identical timestamps — without a tie-breaker BM25 input
order shuffles run-to-run, busting the LLM-side prompt cache.
"""
def _seed_with_shared_timestamp(self, backend):
# Create three memories then force their `updated` columns equal —
# mirrors the real-world case where a touch_structured_memories
# batch lands them in the same second.
for mid in ("zebra_id", "apple_id", "mango_id"):
backend.create_structured_memory(
mid, f"name_{mid}", "", "project", "global", "", "shared content"
)
import sqlalchemy as sa
with backend._conn() as conn:
conn.execute(sa.text("UPDATE structured_memories SET updated = '2024-01-01T00:00:00'"))
conn.commit()
def test_list_stable_order_under_tied_updated(self, backend):
self._seed_with_shared_timestamp(backend)
first = [r["memory_id"] for r in backend.list_structured_memories()]
second = [r["memory_id"] for r in backend.list_structured_memories()]
# Deterministic across calls AND sorted by memory_id ASC for ties
assert first == second
assert first == ["apple_id", "mango_id", "zebra_id"]
def test_search_stable_order_under_tied_updated(self, backend):
self._seed_with_shared_timestamp(backend)
first = [r["memory_id"] for r in backend.search_structured_memories("shared")]
second = [r["memory_id"] for r in backend.search_structured_memories("shared")]
assert first == second
assert first == ["apple_id", "mango_id", "zebra_id"]
def test_visible_search_stable_order_under_tied_updated(self, backend):
self._seed_with_shared_timestamp(backend)
scopes = [("global", "")]
first = [
r["memory_id"] for r in backend.search_visible_structured_memories("shared", scopes)
]
second = [
r["memory_id"] for r in backend.search_visible_structured_memories("shared", scopes)
]
assert first == second
assert first == ["apple_id", "mango_id", "zebra_id"]
Reference in New Issue View Git Blame Copy Permalink