mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-27 06:14:48 -06:00
fix(watch): harden nudge/wake delivery across eviction, cancel, and identity rebinds
Wake path: - Denial metacog nudge moves to the tool channel so it drains with the denied tool batch instead of the next user-message seam. - wake_workstream_if_pending: shared wake gate for watch fires on already-idle workstreams (no IDLE transition for the watcher to observe), wired as wake_fn at every set_watch_runner site via the shared _watch_fire_wake_fn helper (closes over the Workstream OBJECT — after eviction+restore an id-keyed manager lookup would miss). - session_worker exit backstop re-runs the wake gate the moment worker ownership clears: IDLE fans out on the worker thread, so transition-time wakes always landed on the reuse path and no-op'd (the coordinator idle_children strand). - deliver_wake_nudge_from_queue contains GenerationCancelled — it is the wake worker's run() closure and only Exception is caught downstream. Watch delivery: - Terminal fires that cannot reach their workstream are HELD and redelivered on min(interval, 60s) without re-running the command, bounded by MAX_DELIVERY_ATTEMPTS per cycle and the watch's own max_polls across cycles; the poll charge commits durably at hold time so restarts stay budget-bounded. - Restore admission control: per-ws dedup + MAX_CONCURRENT_RESTORES cap, presence-only re-check under the lock, detection-only stall alerts (reclaiming a wedged admission would trade capped degradation for total poll-pool collapse). - Permanent-vs-transient restore taxonomy: corrupt persona stamp and genuinely-missing history (confirmed by a raising storage probe — the resume loader swallows read blips into []) deactivate the watch immediately; everything else holds and retries. - Cancel-race defense: delivery paths re-check is_watch_active before stashing/dispatching, cancel paths write the row BEFORE forget_terminal_dispatched, the HTTP cancel endpoint clears runner state, and a per-tick sweep bounds the residual stash-after-clear interleaving to one check_interval. - Abandon/exhaustion commits are write-then-clear so storage that can read but not write retries the row write instead of re-running the command every cycle; the fresh-fire unrestorable path stashes before its deactivation write for the same reason. Registry follows identity: - The dispatch registry is keyed by _ws_id at registration time; every rebind now moves it: non-fork resume() and /new go through _follow_watch_registration (new key live before the old is removed, never stealing a registration another live session holds), removals are owner-checked so tearing down a watch-restore shell or a resumed-away session cannot unregister a live pane, the restore shell yields to a registration that appears mid-restore, CLI --resume registers after the successful resume, and both the open path and the detail-GET lazy rehydrate wire the registration. Teardown gating and backpressure honesty: - cleanup_session_ui marks ws._closed FIRST under ws._lock — every teardown path (close, close_idle, evict, delete, discard) funnels through it — and session_worker.send re-checks under the same lock, so a wake can never spawn a worker on a torn-down workstream. - Create responses carry initial_message_status when the initial message could not be delivered (queue_full / refused_closed) instead of reading as success; staged attachments survive for the retry; /send surfaces a closed workstream as 404 rather than queue_full. Docs/spec: OpenAPI artifacts regenerated; api-reference documents the new create-response field; TS SDK type extended. Tests: ~30 new pins (cancel races, budget durability across restarts, owner-checked registry moves, teardown gating, stall alerts, backpressure surfaces, wait_until final re-check); wide subsystem sweep green (2353 passed).
This commit is contained in:
@@ -5086,6 +5086,55 @@ class TestMetacognitiveBuffers:
|
||||
assert sys_turn["_source"] == "tool_error"
|
||||
assert sys_turn["content"] == "you hit an error; check memory"
|
||||
|
||||
def test_denial_nudge_queues_on_tool_channel(self, tmp_db):
|
||||
"""A denial responds to the tool batch the user just rejected — the
|
||||
producer must queue it on the TOOL channel so it drains through
|
||||
``_collect_advisories`` alongside the denied results (the same seam
|
||||
tool_error / repeat use), not sit on the user channel until the next
|
||||
user-message seam — by which point the model has already reacted to
|
||||
the denial without the nudge.
|
||||
|
||||
Drives the REAL ``_execute_tools`` two-phase gate with real
|
||||
``_nudges_enabled`` / ``should_nudge`` gating; only the prepare
|
||||
step and the UI approval are stubbed."""
|
||||
from turnstone.core.metacognition import format_nudge
|
||||
|
||||
session = _make_session()
|
||||
# ``should_nudge`` skips the very first message — give the session
|
||||
# the natural pre-batch shape (user turn + assistant tool-call turn).
|
||||
session.messages.append(turn_from_dict({"role": "user", "content": "do the thing"}))
|
||||
session.messages.append(turn_from_dict({"role": "assistant", "content": "calling"}))
|
||||
|
||||
item = {
|
||||
"call_id": "call_1",
|
||||
"func_name": "notify",
|
||||
"needs_approval": True,
|
||||
# Must NOT run — a denied tool never executes.
|
||||
"execute": lambda p: (p["call_id"], "EXECUTED — must not happen"),
|
||||
}
|
||||
with (
|
||||
patch.object(session, "_safe_prepare_tool", return_value=item),
|
||||
patch.object(session.ui, "approve_tools", return_value=(False, "use /tmp instead")),
|
||||
patch.object(session, "_visible_memory_count", return_value=0),
|
||||
):
|
||||
tool_calls = [
|
||||
{
|
||||
"id": "call_1",
|
||||
"type": "function",
|
||||
"function": {"name": "notify", "arguments": "{}"},
|
||||
}
|
||||
]
|
||||
results, feedback = session._execute_tools(tool_calls)
|
||||
|
||||
# The denied item surfaced the operator's feedback as its result…
|
||||
assert results == [("call_1", "Denied by user: use /tmp instead")]
|
||||
assert feedback is None
|
||||
# …and the denial nudge is queued on the TOOL channel, so the same
|
||||
# batch's ``_collect_advisories`` drain delivers it; nothing defers
|
||||
# to the next user turn.
|
||||
assert session._nudge_queue.pending(channel="tool") == [("denial", format_nudge("denial"))]
|
||||
assert session._nudge_queue.pending(channel="user") == []
|
||||
|
||||
def test_queued_message_appends_system_turn_after_tool_batch(self, tmp_db):
|
||||
"""A queued message arriving during a tool batch becomes a
|
||||
first-class ``{"role": "system", "_source": "user_interjection"}``
|
||||
|
||||
Reference in New Issue
Block a user