ewilliams28/openclaw
1
0
Fork 0
mirror of https://github.com/openclaw/openclaw.git synced 2026-08-12 21:53:00 -06:00
Code Issues Packages Projects Releases Wiki Activity

refactor: flip sessions and transcripts to sqlite storage (#98236)

Browse Source 
* refactor(sessions): migrate runtime storage to sqlite

* test(sessions): fix sqlite CI regressions

* test(sessions): align remaining sqlite fixtures

* fix(codex): require sqlite trajectory recorder

* test(sessions): align orphan recovery sqlite fixture

* test(sessions): align sqlite rebase fixtures

* fix(sessions): finish current-main integration of the sqlite flip

Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.

* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs

Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.

* feat(sessions): keep archived transcripts by default with zstd cold storage

Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.

* fix(state): converge agent DB migration lineages and bound database growth

Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.

Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.

* fix(codex): resolve the store path for thread-history import via the SDK

The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.

* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors

A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.

* fix(sessions): complete sqlite conflict translations

* test(sqlite): align hardening checks with maintenance

* test(sessions): inspect compressed transcript archives

* fix(tests): await session seeds and drop unused helpers flagged by CI lint

The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.

* test(sessions): type legacy proof events directly

* test(sessions): align hardening contracts

* perf(sessions): read usage transcript sizes from SQL aggregates

Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.

* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir

Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.

* test(agents): seed harness admission through sqlite

* fix(sqlite): close agent db on pragma setup failure

* fix(doctor): compact and retrofit incremental auto-vacuum after session import

The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.

---------

Co-authored-by: Peter Steinberger <steipete@gmail.com>
This commit is contained in:
Josh Lehman
2026-07-11 14:50:37 -07:00
committed by GitHub
parent ca09fcaff1
commit 0a8e3604ba
612 changed files with 49731 additions and 20338 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/ci.yml
+27
View File
@@ -1580,6 +1580,12 @@ jobs:
- check_name: check-session-transcript-reader-boundary
group: session-transcript-reader-boundary
runner: blacksmith-4vcpu-ubuntu-2404
- check_name: check-sqlite-session-schema-baseline
group: sqlite-session-schema-baseline
runner: blacksmith-4vcpu-ubuntu-2404
- check_name: check-sqlite-session-flip-proof
group: sqlite-session-flip-proof
runner: blacksmith-8vcpu-ubuntu-2404
- check_name: check-additional-extension-channels
group: extension-channels
runner: blacksmith-8vcpu-ubuntu-2404
@@ -1695,6 +1701,13 @@ jobs:
else
run_check "lint:tmp:session-accessor-boundary" pnpm run lint:tmp:session-accessor-boundary
fi
if [ ! -f scripts/check-sqlite-transaction-boundary.mjs ]; then
echo "[skip] SQLite transaction boundary check is not present in this checkout"
elif ! node -e 'const pkg = require("./package.json"); process.exit(pkg.scripts?.["lint:tmp:sqlite-transaction-boundary"] ? 0 : 1);'; then
echo "[skip] SQLite transaction boundary script is not present in package.json"
else
run_check "lint:tmp:sqlite-transaction-boundary" pnpm run lint:tmp:sqlite-transaction-boundary
fi
;;
session-transcript-reader-boundary)
if [ ! -f scripts/check-session-transcript-reader-boundary.mjs ]; then
@@ -1705,6 +1718,20 @@ jobs:
run_check "lint:tmp:session-transcript-reader-boundary" pnpm run lint:tmp:session-transcript-reader-boundary
fi
;;
sqlite-session-schema-baseline)
if ! node -e 'const pkg = require("./package.json"); process.exit(pkg.scripts?.["sqlite:sessions-schema:check"] ? 0 : 1);'; then
echo "[skip] SQLite sessions/transcripts schema baseline script is not present in package.json"
else
run_check "sqlite:sessions-schema:check" pnpm run sqlite:sessions-schema:check
fi
;;
sqlite-session-flip-proof)
if [ ! -f test/scripts/sqlite-sessions-transcripts-flip-proof.e2e.test.ts ]; then
echo "[skip] SQLite sessions/transcripts flip proof is not present in this checkout"
else
run_check "sqlite sessions/transcripts flip proof" node scripts/run-vitest.mjs run --config test/vitest/vitest.e2e.config.ts test/scripts/sqlite-sessions-transcripts-flip-proof.e2e.test.ts
fi
;;
extension-channels)
run_check "lint:extensions:channels" pnpm run lint:extensions:channels
;;
.github/workflows/workflow-sanity.yml
+3
View File
@@ -212,5 +212,8 @@ jobs:
- name: Check plugin SDK API baseline drift
run: pnpm plugin-sdk:api:check
- name: Check SQLite sessions/transcripts schema baseline drift
run: pnpm sqlite:sessions-schema:check
- name: Check plugin SDK surface budget
run: pnpm plugin-sdk:surface:check
AGENTS.md
+1
View File
@@ -76,6 +76,7 @@ Skills own workflows; root owns hard policy and routing.
- State/storage migrations are database-first. Runtime reads/writes the canonical store only. Old file stores, sidecars, aliases, and fallback readers belong in `openclaw doctor --fix` migration code only, never steady-state runtime.
- Storage default: SQLite only. Do not add JSON/JSONL/TXT/sidecar files for OpenClaw-owned runtime state, caches, queues, registries, indexes, cursors, checkpoints, or plugin scratch data.
- SQLite runtime access uses Kysely helpers, not raw SQL statement strings, except schema DDL, migrations, low-level DB bootstrap, or narrowly justified SQLite primitives.
- SQLite write transactions are synchronous commit sections only. Finish async planning, filesystem access, plugin hooks, and predicates before `BEGIN`; then reread and validate authoritative rows before writing. Never return a Promise or execute `await` from a transaction callback.
- Use the shared state DB (`state/openclaw.sqlite`) for global runtime state and plugin KV data. Use the per-agent DB (`agents/<agentId>/agent/openclaw-agent.sqlite`) for agent-scoped state/cache. Use a dedicated SQLite DB only when schema, volume, or lifecycle clearly does not fit those stores.
- Legacy state/cache files are migration debt. When touching code that reads/writes them, prefer moving the data into SQLite or calling out the refactor follow-up; do not add parallel file paths.
- File storage must be a named product artifact: import/export, user attachment, log, backup, or external tool contract. If it is app state or cache, it belongs in SQLite.
docs/.generated/README.md
+6 -2
View File
@@ -1,17 +1,19 @@
# Generated Docs Artifacts
SHA-256 hash files are the tracked drift-detection artifacts. The full JSON
baselines are generated locally (gitignored) for inspection only.
SHA-256 hash files are the tracked drift-detection artifacts. Full baselines are
generated locally for inspection only.
**Tracked (committed to git):**
- `config-baseline.sha256` — hashes of config baseline JSON artifacts.
- `plugin-sdk-api-baseline.sha256` — hashes of Plugin SDK API baseline artifacts.
- `sqlite-session-transcript-schema-baseline.sha256` — hash of the sessions/transcripts SQLite schema baseline.
**Local only (gitignored):**
- `config-baseline.json`, `config-baseline.core.json`, `config-baseline.channel.json`, `config-baseline.plugin.json`
- `plugin-sdk-api-baseline.json`, `plugin-sdk-api-baseline.jsonl`
- `.artifacts/sqlite-session-transcript-schema-baseline.sql`
Do not edit any of these files by hand.
@@ -19,3 +21,5 @@ Do not edit any of these files by hand.
- Validate config baseline: `pnpm config:docs:check`
- Regenerate Plugin SDK API baseline: `pnpm plugin-sdk:api:gen`
- Validate Plugin SDK API baseline: `pnpm plugin-sdk:api:check`
- Regenerate SQLite sessions/transcripts schema baseline: `pnpm sqlite:sessions-schema:gen`
- Validate SQLite sessions/transcripts schema baseline: `pnpm sqlite:sessions-schema:check`
docs/.generated/plugin-sdk-api-baseline.sha256
+2 -2
View File
@@ -1,2 +1,2 @@
6f7e0f6f2e6d5b107500b62f9556abdc5e450eb80a7a8586c1baf7d82894ed2e plugin-sdk-api-baseline.json
a3d7d4f1a3aff973cb59bac966f884a812cbc633f90a8d9cd793209192916e5c plugin-sdk-api-baseline.jsonl
07c747212af5b8a228a7ff23101a2fa4096f148600d1d2e0fc83ac8f151cb58b plugin-sdk-api-baseline.json
85868eedeb4a99e714468f7f8c81d070227f8ed7e247939a4ac0d458ad04426d plugin-sdk-api-baseline.jsonl
docs/.generated/sqlite-session-transcript-schema-baseline.sha256
+1
View File
@@ -0,0 +1 @@
4bb5f50a60c3664656d56f2be8d9884d48287f084c03959120b2825991b23f21 sqlite-session-transcript-schema-baseline.sql
docs/.i18n/glossary.zh-CN.json
+8
View File
@@ -1458,5 +1458,13 @@
{
"source": "Crabbox plugin",
"target": "Crabbox 插件"
},
{
"source": "Path 3 SQLite session artifact family",
"target": "Path 3 SQLite 会话工件族"
},
{
"source": "Path 3 live SQLite E2E harness",
"target": "Path 3 live SQLite E2E harness"
}
]
docs/channels/channel-routing.md
+13 -5
View File
@@ -134,16 +134,24 @@ Example:
## Session storage
Session stores live under the state directory (default `~/.openclaw`):
Runtime session rows live in each agent's SQLite database under the state
directory (default `~/.openclaw`):
- `~/.openclaw/agents/<agentId>/sessions/sessions.json`
- JSONL transcripts live alongside the store
- `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
You can override the store path via `session.store` and `{agentId}` templating.
Older installs may have legacy transcript JSONL files and a `sessions.json` row
store under `~/.openclaw/agents/<agentId>/sessions/`. Gateway startup and
`openclaw doctor --fix` import hot legacy rows/history into SQLite
automatically. Use `openclaw doctor --session-sqlite inspect
--session-sqlite-all-agents` and the
[Doctor](/cli/doctor#session-sqlite-migration) validation sequence when you need
explicit migration evidence.
You can still select a legacy store path via `session.store` and `{agentId}`
templating for migration and offline-maintenance workflows.
Gateway and ACP session discovery also scans disk-backed agent stores under the
default `agents/` root and under templated `session.store` roots. Discovered
stores must stay inside that resolved agent root and use a regular
stores must stay inside that resolved agent root and use a regular legacy
`sessions.json` file. Symlinks and out-of-root paths are ignored.
## WebChat behavior
docs/channels/group-messages.md
+1 -1
View File
@@ -87,7 +87,7 @@ Only owner numbers (from `channels.whatsapp.allowFrom`, or the bot's own E.164 w
- Heartbeats run in the agent's main session; group sessions never get heartbeat runs.
- Echo suppression remembers the combined prompt (history + current message) per session so the bot's own delivered messages do not retrigger it; an identical repeated batch can be skipped as an echo.
- Session store entries appear as `agent:<agentId>:whatsapp:group:<jid>` in the session store (`~/.openclaw/agents/<agentId>/sessions/sessions.json` by default); a missing entry just means the group has not triggered a run yet.
- Session store entries appear as `agent:<agentId>:whatsapp:group:<jid>` in the per-agent SQLite session store; a missing entry just means the group has not triggered a run yet.
- Typing indicators follow `session.typingMode` / `agents.defaults.typingMode`. When visible replies are opted into message-tool-only mode, typing starts immediately by default so group members can see the agent working even if no automatic final reply is posted. Explicit typing-mode config still wins.
## Related
docs/ci.md
+2 -2
View File
@@ -26,7 +26,7 @@ dispatch.
## Pipeline overview
| Job | Purpose | When it runs |
| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------- |
| `preflight` | Detect docs-only changes, changed scopes, changed extensions, and build the CI manifest | Always on non-draft pushes and PRs |
| `runner-admission` | Hosted 90-second debounce for canonical `main` pushes before Blacksmith work is registered | Every CI run; sleep only on canonical `main` pushes |
| `security-fast` | Private key detection, changed-workflow audit via `zizmor`, and production lockfile audit | Always on non-draft pushes and PRs |
@@ -38,7 +38,7 @@ dispatch.
| `checks-fast-contracts-channels-*` | Two weighted channel contract shards | Node-relevant changes |
| `checks-node-*` | Core Node test shards, excluding channel, bundled, contract, and extension lanes | Node-relevant changes |
| `check-*` | Sharded main local gate equivalent: guards, shrinkwrap, bundled-channel config metadata, prod types, lint, dependencies, test types | Node-relevant changes |
| `check-additional-*` | Boundary check stripes (including prompt snapshot drift), session accessor/transcript reader boundaries, extension lint groups, package boundary compile/canary, and runtime topology architecture | Node-relevant changes |
| `check-additional-*` | Boundary check stripes (including prompt snapshot drift), session accessor/transcript reader/SQLite transaction boundaries, extension lint groups, package boundary compile/canary, and runtime topology architecture | Node-relevant changes |
| `checks-node-compat-node22` | Node 22 compatibility build and smoke lane | Manual CI dispatch for releases |
| `check-docs` | Docs formatting, lint, and broken-link checks | Docs changed (PRs and manual dispatch) |
| `native-i18n` | Native app, Android, and Apple i18n inventory checks | Native i18n-relevant changes |
docs/cli/doctor.md
+125 -3
View File
@@ -23,6 +23,15 @@ Related:
| Repair | `openclaw doctor --fix` | Applies supported repairs, prompting unless non-interactive repair is safe. |
| Lint | `openclaw doctor --lint` | Read-only structured findings for CI, preflight, and review gates. |
Doctor has four postures:
| Posture | Command | Behavior |
| ------------------------ | ----------------------------------------- | ------------------------------------------------------------------------------- |
| Inspect | `openclaw doctor` | Human-oriented checks and guided prompts. |
| Repair | `openclaw doctor --fix` | Applies supported repairs, using prompts unless non-interactive repair is safe. |
| Lint | `openclaw doctor --lint` | Read-only structured findings for CI, preflight, and review gates. |
| Session SQLite migration | `openclaw doctor --session-sqlite <mode>` | Inspects, imports, validates, compacts, recovers, or restores session state. |
Prefer `--lint` when automation needs a stable result. Prefer `--fix` when a human operator wants doctor to edit config or state.
## Examples
@@ -40,6 +49,13 @@ openclaw doctor --fix --non-interactive
openclaw doctor --generate-gateway-token
openclaw doctor --post-upgrade
openclaw doctor --post-upgrade --json
openclaw doctor --session-sqlite inspect --session-sqlite-all-agents
openclaw doctor --session-sqlite dry-run --session-sqlite-agent main --json
openclaw doctor --session-sqlite import --session-sqlite-all-agents
openclaw doctor --session-sqlite validate --session-sqlite-all-agents --json
openclaw doctor --session-sqlite compact --session-sqlite-all-agents
openclaw doctor --session-sqlite recover --github-issue
openclaw doctor --session-sqlite restore --session-sqlite-all-agents
```
For channel-specific permissions, use the channel probes instead of `doctor`:
@@ -54,7 +70,7 @@ openclaw channels status --probe
## Options
| Option | Effect |
| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--no-workspace-suggestions` | Disable workspace memory/search suggestions. |
| `--yes` | Accept defaults without prompting. |
| `--repair` / `--fix` | Apply recommended non-service repairs without prompting (`--fix` is an alias). Gateway service installs/rewrites still require interactive confirmation or explicit `gateway` commands. |
@@ -65,13 +81,18 @@ openclaw channels status --probe
| `--deep` | Scan system services for extra gateway installs; report recent Gateway supervisor restart handoffs. |
| `--lint` | Run modernized health checks in read-only mode and emit diagnostic findings. |
| `--post-upgrade` | Run post-upgrade plugin compatibility probes; findings go to stdout; exit code 1 if any error-level finding is present. |
| `--json` | With `--lint`: JSON findings. With `--post-upgrade`: machine-readable envelope `{ probesRun, findings }`. |
| `--session-sqlite <mode>` | Run the targeted session SQLite migration mode: `inspect`, `dry-run`, `import`, `validate`, `compact`, `recover`, or `restore`. |
| `--session-sqlite-store <path>` | With `--session-sqlite`: select one legacy `sessions.json` store path. |
| `--session-sqlite-agent <id>` | With `--session-sqlite`: select one configured agent. |
| `--session-sqlite-all-agents` | With `--session-sqlite`: select configured and discovered agent stores. |
| `--github-issue` | With `--session-sqlite recover`: prepare a sanitized openclaw/openclaw issue report; doctor creates it with `gh` after `--yes` or interactive confirmation. |
| `--json` | With `--lint`: JSON findings. With `--post-upgrade`: machine-readable envelope `{ probesRun, findings }`. With `--session-sqlite`: the migration report as JSON. |
| `--severity-min <level>` | With `--lint`: drop findings below `info`, `warning`, or `error`. |
| `--all` | With `--lint`: run all registered checks, including opt-in checks excluded from the default set. |
| `--skip <id>` | With `--lint`: skip a check id. Repeatable. |
| `--only <id>` | With `--lint`: run only the given check id(s). Repeatable. |
`--json`, `--severity-min`, `--all`, `--only`, and `--skip` are only accepted together with `--lint`.
`--severity-min`, `--all`, `--only`, and `--skip` are only accepted together with `--lint`; `--json` is accepted with `--lint`, `--post-upgrade`, and `--session-sqlite`.
## Lint mode
@@ -178,6 +199,107 @@ finish safely, startup exits and tells you to run the same image once with
`openclaw doctor --fix` against the same mounted state/config before restarting
the container normally.
## Session SQLite migration
OpenClaw imports legacy session rows and transcript history into each agent's
SQLite database automatically during gateway startup and during
`openclaw doctor --fix`. `openclaw doctor --session-sqlite <mode>` is the
targeted inspection and validation tool for that migration. Current runtime
session rows live in
`~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`. Legacy
`sessions.json` files are migration sources. Hot transcript JSONL files are
imported and archived out of the active sessions directory after successful
import; archive-tier JSONL files remain support artifacts, not runtime
fallbacks.
Modes:
| Mode | Behavior |
| ---------- | ---------------------------------------------------------------------------------------------------------------------- |
| `inspect` | Read legacy and SQLite counts, plus unreferenced JSONL files, without importing. |
| `dry-run` | Parse legacy entries and transcript JSONL files, count importable rows, and report issues without writing SQLite rows. |
| `import` | Import legacy entries and transcript events into SQLite for the selected targets. |
| `validate` | Compare the selected legacy sources against SQLite rows and transcript event counts. |
| `compact` | Checkpoint and VACUUM selected agent SQLite databases to reclaim free pages after large deletes or archive cleanup. |
| `recover` | Restore the latest failed migration run, validate its targets, and prepare a sanitized GitHub issue report. |
| `restore` | Restore archived transcript artifacts from recorded migration manifests without deleting SQLite data. |
Selectors:
- Default: the configured default agent store, when that legacy store file exists.
- `--session-sqlite-agent <id>`: one configured agent.
- `--session-sqlite-all-agents`: configured agent stores plus discovered agent stores.
- `--session-sqlite-store <path>`: one explicit legacy `sessions.json` path.
Manual inspection sequence:
```bash
openclaw doctor --session-sqlite inspect --session-sqlite-all-agents
openclaw doctor --session-sqlite dry-run --session-sqlite-all-agents --json
openclaw doctor --session-sqlite import --session-sqlite-all-agents
openclaw doctor --session-sqlite validate --session-sqlite-all-agents --json
openclaw doctor --session-sqlite compact --session-sqlite-all-agents
openclaw doctor --session-sqlite recover --github-issue
```
Back up the OpenClaw state directory before running `import` on an install with
important history. `validate` exits non-zero when a selected legacy entry is
missing from SQLite, a session id differs, or a transcript event count differs.
When using `--session-sqlite-store <path>`, check that the report contains the
expected target count; a nonexistent explicit store path selects no targets.
SQLite deletes reclaim pages inside the database first; they do not necessarily
shrink the database file immediately. After deleting or archiving large
transcripts, run `openclaw doctor --session-sqlite compact --session-sqlite-all-agents`
to checkpoint WAL files, run `VACUUM`, and report before/after database and WAL
sizes. This is explicit doctor maintenance so normal Gateway writes do not pause
for background vacuum work.
Each import writes a manifest under
`~/.openclaw/session-sqlite-migration-runs/` before moving transcript artifacts
into the archive. If startup reports a failed session SQLite migration after
artifacts moved, run recovery:
```bash
openclaw doctor --session-sqlite recover --github-issue
```
Recovery selects the latest failed migration manifest, restores only the
manifest's archived artifacts, validates the affected targets, refreshes the
sanitized `.failure.md` and `.failure.json` reports, and prepares a GitHub issue
body that avoids transcript contents, raw environment, secrets, and unbounded
config. When no failed migration manifest exists but a selected agent SQLite
database is corrupt or not a database, recovery preserves the DB, WAL, and SHM
files by renaming them with a `.corrupt-<timestamp>` suffix so the next startup
can create a fresh database. With `--github-issue --yes`, doctor uses the GitHub
CLI to create the issue in `openclaw/openclaw`; without confirmation it writes
the local support report and prints a prefilled issue URL.
`restore` remains the lower-level undo operation. It uses manifest
`sourcePath -> archivePath` records, moves archived artifacts back only when the
original path is missing, reports conflicts when both paths exist, and leaves
the SQLite database in place.
### Downgrading After Session SQLite Migration
Before starting an older file-backed OpenClaw version, restore the archived
legacy transcript artifacts:
```bash
openclaw doctor --session-sqlite restore --session-sqlite-all-agents
```
Older versions read `sessions.json` entries and the `sessionFile` paths recorded
in those entries. After the SQLite migration, successful imports move hot JSONL
transcripts into `session-sqlite-import-archive/`, so the older runtime cannot
see that history until restore moves those manifest-recorded artifacts back to
their original paths.
Restore does not delete SQLite data. Sessions created after the SQLite flip
exist only in SQLite and will not appear to the older runtime. If you later
upgrade again, run the normal migration validation sequence above so OpenClaw can
compare restored legacy artifacts with the SQLite rows before importing.
## Notes
- In Nix mode (`OPENCLAW_NIX_MODE=1`), read-only doctor checks still work, but `doctor --fix`, `doctor --repair`, `doctor --yes`, and `doctor --generate-gateway-token` are disabled because `openclaw.json` is immutable. Edit the Nix source for this install instead; for nix-openclaw, use the agent-first [Quick Start](https://github.com/openclaw/nix-openclaw#quick-start).
docs/cli/security.md
+1 -1
View File
@@ -124,7 +124,7 @@ Applies safe, deterministic remediations:
- flips common `groupPolicy="open"` to `groupPolicy="allowlist"` (including account variants in supported channels)
- when WhatsApp group policy flips to `allowlist`, seeds `groupAllowFrom` from the stored `allowFrom` file when that list exists and config does not already define `allowFrom`
- sets `logging.redactSensitive` from `"off"` to `"tools"`
- tightens permissions for state/config and common sensitive files (`credentials/*.json`, `auth-profiles.json`, `sessions.json`, session `*.jsonl`)
- tightens permissions for state/config and common sensitive files (`credentials/*.json`, `auth-profiles.json`, `openclaw-agent.sqlite`, and legacy session artifacts)
- also tightens config include files referenced from `openclaw.json`
- uses `chmod` on POSIX hosts and `icacls` resets on Windows
docs/cli/sessions.md
+19 -17
View File
@@ -51,10 +51,10 @@ Control UI uses that mode by default so deleted or disk-only agent stores do
not reappear in the Sessions view.
`--all-agents` reads configured agent stores. Gateway and ACP session
discovery are broader: they also include disk-only stores found under the
default `agents/` root or a templated `session.store` root. Those discovered
stores must resolve to regular `sessions.json` files inside the agent root;
symlinks and out-of-root paths are skipped.
discovery are broader: they also include SQLite stores resolved from
configured agent roots or a templated `session.store` root. Legacy selector
paths must resolve inside the agent root; symlinks and out-of-root paths are
skipped.
`openclaw sessions --all-agents --json`:
@@ -88,12 +88,12 @@ openclaw sessions --agent work tail --follow
openclaw sessions --all-agents tail --follow
```
`openclaw sessions tail` renders recent trajectory JSONL events as compact
`openclaw sessions tail` renders recent runtime trajectory events as compact
progress lines. Without `--session-key`, it tails running sessions first, then
the latest stored session. `--tail <count>` controls how many existing events
print before follow mode; default `80`, and `0` starts at the current end.
`--follow` keeps watching the selected trajectory files, including relocated
files referenced by `<session>.trajectory-path.json`.
`--follow` keeps watching the selected SQLite-backed session or an explicit
legacy trajectory file.
The progress view is intentionally conservative: prompt text, tool arguments,
and tool result bodies are not printed. Tool calls show the tool name with
@@ -129,12 +129,13 @@ openclaw sessions cleanup --json
([Configuration reference](/gateway/config-agents#session)):
- Scope note: `openclaw sessions cleanup` maintains session stores,
transcripts, and trajectory sidecars. It does not prune cron run history,
which is managed by `cron.runLog.keepLines`
transcripts, trajectory rows, and legacy trajectory sidecars. It does not
prune cron run history, which is managed by `cron.runLog.keepLines`
([Cron configuration](/automation/cron-jobs#configuration)).
- Cleanup also prunes unreferenced primary transcripts, compaction
checkpoints, and trajectory sidecars older than `session.maintenance.pruneAfter`;
files still referenced by `sessions.json` are preserved.
- Cleanup also prunes unreferenced legacy/archive transcript artifacts,
compaction checkpoints, and trajectory sidecars older than
`session.maintenance.pruneAfter`; artifacts still referenced by SQLite
session rows are preserved.
- Cleanup reports short-lived Gateway model-run probe cleanup separately as
`modelRunPruned`. This only matches strict explicit keys shaped like
`agent:*:explicit:model-run-<uuid>`. Retention is a fixed `24h` and is
@@ -145,20 +146,21 @@ openclaw sessions cleanup --json
Flags:
| Flag | Description |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--dry-run` | Preview how many entries would be pruned/capped without writing. In text mode, prints a per-session action table (`Action`, `Key`, `Age`, `Model`, `Flags`) plus a summary grouped by session label. |
| `--enforce` | Apply maintenance even when `session.maintenance.mode` is `warn`. |
| `--fix-missing` | Remove entries whose transcript files are missing or header-only/empty, even if they would not normally age/count out yet. |
| `--fix-dm-scope` | When `session.dmScope` is `main`, retire stale peer-keyed direct-DM rows left behind by earlier `per-peer`, `per-channel-peer`, or `per-account-channel-peer` routing. Use `--dry-run` first; applying removes those rows from `sessions.json` and preserves their transcripts as deleted archives. |
| `--fix-missing` | Remove legacy entries whose archived transcript artifacts are missing or header-only/empty, even if they would not normally age/count out yet. |
| `--fix-dm-scope` | When `session.dmScope` is `main`, retire stale peer-keyed direct-DM rows left behind by earlier `per-peer`, `per-channel-peer`, or `per-account-channel-peer` routing. Use `--dry-run` first; applying removes those rows from SQLite and preserves their legacy transcript artifacts as deleted archives. |
| `--active-key <key>` | Protect a specific active key from disk-budget eviction. Durable external conversation pointers, such as group sessions and thread-scoped chat sessions, are also kept by age/count/disk-budget maintenance. |
| `--agent <id>` | Run cleanup for one configured agent store. |
| `--all-agents` | Run cleanup for all configured agent stores. |
| `--store <path>` | Run against a specific `sessions.json` file. |
| `--store <path>` | Run against a specific legacy store selector path. |
| `--json` | Print a JSON summary. With `--all-agents`, output includes one summary per store. |
When a Gateway is reachable, non-dry-run cleanup for configured agent stores is
sent through the Gateway so it shares the same session-store writer as runtime
traffic. Use `--store <path>` for explicit offline repair of a store file.
traffic. Use `--store <path>` for explicit offline repair of a legacy store
selector.
`openclaw sessions cleanup --all-agents --dry-run --json`:
docs/cli/update.md
+3
View File
@@ -63,6 +63,9 @@ In Nix mode (`OPENCLAW_NIX_MODE=1`), mutating `openclaw update` runs are disable
<Warning>
Downgrades require confirmation because older versions can break configuration.
If the install has already migrated sessions to SQLite, restore archived legacy
transcript artifacts before starting an older file-backed version. See
[Doctor: Downgrading after session SQLite migration](/cli/doctor#downgrading-after-session-sqlite-migration).
</Warning>
## `update status`
docs/concepts/agent-loop.md
+1 -1
View File
@@ -36,7 +36,7 @@ Session write locks are non-reentrant by default. A helper that intentionally ne
- Workspace is resolved and created; sandboxed runs may redirect to a sandbox workspace root.
- Skills are loaded (or reused from a snapshot) and injected into env and prompt.
- Bootstrap/context files are resolved and injected into the system prompt.
- A session write lock is acquired and `SessionManager` is opened and prepared before streaming starts. Any later transcript rewrite, compaction, or truncation path must take the same lock before opening or mutating the transcript file.
- A session write lock is acquired and the session transcript target is prepared before streaming starts. Any later transcript rewrite, compaction, or truncation path must take the same lock before mutating the SQLite transcript rows.
## Prompt assembly
docs/concepts/agent-workspace.md
+5 -2
View File
@@ -112,9 +112,10 @@ These live under `~/.openclaw/` and should NOT be committed to the workspace rep
- `~/.openclaw/openclaw.json` (config)
- `~/.openclaw/agents/<agentId>/agent/auth-profiles.json` (model auth profiles: OAuth + API keys)
- `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite` (session rows, transcripts, and per-agent runtime state)
- `~/.openclaw/agents/<agentId>/agent/codex-home/` (per-agent Codex runtime account, config, skills, plugins, and native thread state)
- `~/.openclaw/credentials/` (channel/provider state plus legacy OAuth import data)
- `~/.openclaw/agents/<agentId>/sessions/` (session transcripts + metadata)
- `~/.openclaw/agents/<agentId>/sessions/` (legacy migration sources and archive/support artifacts)
- `~/.openclaw/skills/` (managed skills)
If you need to migrate sessions or config, copy them separately and keep them out of version control.
@@ -217,7 +218,9 @@ Suggested `.gitignore` starter:
Run `openclaw setup --workspace <path>` to seed any missing files.
</Step>
<Step title="Copy sessions (optional)">
If you need sessions, copy `~/.openclaw/agents/<agentId>/sessions/` from the old machine separately.
If you need sessions, copy `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
from the old machine separately. Copy `~/.openclaw/agents/<agentId>/sessions/`
only when you also need legacy migration inputs or archive/support artifacts.
</Step>
</Steps>
docs/concepts/agent.md
+7 -3
View File
@@ -87,11 +87,15 @@ runtime surface.
## Sessions
Session transcripts are stored as JSONL at:
Session rows are stored in the per-agent SQLite database:
- `~/.openclaw/agents/<agentId>/sessions/<SessionId>.jsonl`
- `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
The session ID is stable and chosen by OpenClaw. OpenClaw does not read session folders from other tools.
Transcript JSONL files can still live under
`~/.openclaw/agents/<agentId>/sessions/` as legacy migration inputs, deleted or
reset archives, imports, exports, and support artifacts. Active agent history is
stored in SQLite with the session rows. The session ID is stable and chosen by
OpenClaw. OpenClaw does not read session folders from other tools.
## Steering while streaming
docs/concepts/compaction.md
+8 -2
View File
@@ -104,10 +104,16 @@ Compaction summarization preserves opaque identifiers by default (`identifierPol
### Active transcript byte guard
When `agents.defaults.compaction.maxActiveTranscriptBytes` is set, OpenClaw triggers normal local compaction before a run if the active JSONL reaches that size. This is useful for long-running sessions where provider-side context management may keep model context healthy while the local transcript keeps growing. It does not split raw JSONL bytes; it asks the normal compaction pipeline to create a semantic summary.
When `agents.defaults.compaction.maxActiveTranscriptBytes` is set, OpenClaw
triggers normal local compaction before a run if transcript history reaches
that size. This is useful for long-running sessions where provider-side context
management may keep model context healthy while persisted transcript history
keeps growing. It does not split raw bytes; it asks the normal compaction
pipeline to create a semantic summary.
<Warning>
The byte guard requires `truncateAfterCompaction: true`. Without transcript rotation, the active file would not shrink and the guard remains inactive.
The byte guard applies to the active SQLite transcript history. Legacy JSONL
checkpoint artifacts are not the active compaction target.
</Warning>
### Successor transcripts
docs/concepts/context-engine.md
+4 -6
View File
@@ -226,12 +226,10 @@ Required members:
Optional projection lifecycle for hosts with persistent backend threads (for example Codex app-server). `mode: "thread_bootstrap"` with a stable `epoch` asks the host to inject the assembled context once per epoch and reuse the backend thread until the epoch changes, instead of re-projecting every turn. Omit this field for normal per-turn projection.
</ParamField>
`compact` returns a `CompactResult`. When compaction rotates the active
transcript, `result.sessionTarget` (a typed `ContextEngineSessionTarget`
carrying the storage mode, session identity, and transcript artifact path)
identifies the successor session that the next retry or turn must use;
`result.sessionId` mirrors the successor id. `result.sessionFile` is
deprecated - report successors through `sessionTarget` instead.
`compact` returns a `CompactResult`. When compaction changes the active session
identity, `result.sessionTarget` (a typed `ContextEngineSessionTarget` carrying
the session identity and store scope) identifies the successor session that the
next retry or turn must use; `result.sessionId` mirrors the successor id.
Optional members:
docs/concepts/multi-agent.md
+5 -4
View File
@@ -6,7 +6,7 @@ read_when: "You want multiple agents with separate workspaces, auth, and session
status: active
---
Run multiple agents in one Gateway process, each with its own workspace, state directory (`agentDir`), and session store, plus multiple channel accounts (e.g. two WhatsApp numbers). Inbound messages route to the right agent through **bindings**.
Run multiple _isolated_ agents in one Gateway process, each with its own workspace, state directory (`agentDir`), and SQLite-backed session history, plus multiple channel accounts (e.g. two WhatsApp numbers). Inbound messages route to the right agent through **bindings**.
An **agent** is the full per-persona scope: workspace files, auth profiles, model registry, and session store. A **binding** maps a channel account (a Slack workspace, a WhatsApp number, etc.) to one of those agents.
@@ -16,7 +16,7 @@ Each agent has its own:
- **Workspace**: files, `AGENTS.md`/`SOUL.md`/`USER.md`, local notes, persona rules.
- **State directory** (`agentDir`): auth profiles, model registry, per-agent config.
- **Session store**: chat history and routing state under `~/.openclaw/agents/<agentId>/sessions`.
- **Session store**: chat history and routing state in `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`.
Auth profiles are per-agent, read from:
@@ -46,13 +46,14 @@ when personas must not share compiled wiki knowledge.
## Paths
| What | Default | Override |
| ------------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| -------------------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Config | `~/.openclaw/openclaw.json` | `OPENCLAW_CONFIG_PATH` |
| State dir | `~/.openclaw` | `OPENCLAW_STATE_DIR` |
| Default agent's workspace | `~/.openclaw/workspace` (or `workspace-<profile>` when `OPENCLAW_PROFILE` is set) | `agents.list[].workspace`, then `agents.defaults.workspace`, or `OPENCLAW_WORKSPACE_DIR` |
| Other agents' workspace | `<stateDir>/workspace-<agentId>` (or `<agents.defaults.workspace>/<agentId>` when set) | `agents.list[].workspace` |
| Agent dir | `~/.openclaw/agents/<agentId>/agent` | `agents.list[].agentDir` |
| Sessions | `~/.openclaw/agents/<agentId>/sessions` | — |
| Sessions and transcripts | `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite` | — |
| Legacy/archive session artifacts | `~/.openclaw/agents/<agentId>/sessions` | — |
### Single-agent mode (default)
docs/concepts/session-tool.md
+1 -1
View File
@@ -56,7 +56,7 @@ The returned view is intentionally bounded and safety-filtered:
Both tools accept either a **session key** (like `"main"`) or a **session ID** from a previous list call.
If you need the exact byte-for-byte transcript, inspect the transcript file on disk instead of treating `sessions_history` as a raw dump.
If you need the exact raw transcript, inspect the scoped SQLite transcript rows instead of treating `sessions_history` as an unfiltered dump.
## Sending cross-session messages
docs/concepts/session.md
+14 -7
View File
@@ -113,20 +113,27 @@ an idle-mode default when no `session.reset`/`resetByType` block is set.
## Where state lives
- **Store:** `~/.openclaw/agents/<agentId>/sessions/sessions.json`
- **Transcripts:** `~/.openclaw/agents/<agentId>/sessions/<sessionId>.jsonl`
- **Runtime session rows:** `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
- **Archived transcript files:** `~/.openclaw/agents/<agentId>/sessions/`
- **Legacy row migration source:** `~/.openclaw/agents/<agentId>/sessions/sessions.json`
`sessions.json` keeps separate lifecycle timestamps:
The session rows in the per-agent SQLite database keep separate lifecycle
timestamps:
- `sessionStartedAt`: when the current `sessionId` began; daily reset uses this.
- `lastInteractionAt`: last user/channel interaction that extends idle lifetime.
- `updatedAt`: last store-row mutation; useful for listing and pruning, but not
authoritative for daily/idle reset freshness.
Older rows without `sessionStartedAt` are resolved from the transcript JSONL
session header when available. If an older row also lacks `lastInteractionAt`,
idle freshness falls back to that session start time, not to later bookkeeping
writes.
During migration from older installs, gateway startup and `openclaw doctor
--fix` import legacy `sessions.json` rows and hot transcript JSONL history into
SQLite automatically. Rows without `sessionStartedAt` are resolved from the
legacy transcript JSONL session header when available. If an older row also
lacks `lastInteractionAt`, idle freshness falls back to that session start time,
not to later bookkeeping writes. Use `openclaw doctor --session-sqlite inspect
--session-sqlite-all-agents` and the [Doctor migration
sequence](/cli/doctor#session-sqlite-migration) when you want explicit
inspection or validation evidence.
## Session maintenance
docs/docs_map.md
+29 -3
View File
@@ -1491,6 +1491,8 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Structured health checks
- H2: Check selection
- H2: Post-upgrade mode
- H2: Session SQLite migration
- H3: Downgrading After Session SQLite Migration
- H2: Notes
- H2: macOS: launchctl env overrides
- H2: Related
@@ -4931,6 +4933,16 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Milestones
- H2: Open questions
## plan/path3-sqlite-session-artifact-family.md
- Route: /plan/path3-sqlite-session-artifact-family
- Headings:
- H1: Path 3 SQLite Session Artifact Family
- H2: Authoritative family
- H2: Non-family artifacts after the flip
- H2: Patch points
- H2: Focused tests
## plan/ui-channels.md
- Route: /plan/ui-channels
@@ -8442,6 +8454,19 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Design contract
- H2: Subpath exports
## reference/path3-live-sqlite-e2e-harness.md
- Route: /reference/path3-live-sqlite-e2e-harness
- Headings:
- H2: Command shape
- H2: Isolated built-CLI proof
- H2: Preflight
- H2: Agent-driven scenario
- H2: Per-step assertions
- H2: Evidence artifact
- H2: Safety rules
- H2: Passing result
## reference/prompt-caching.md
- Route: /reference/prompt-caching
@@ -8531,11 +8556,12 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Two persistence layers
- H2: On-disk locations
- H2: Store maintenance and disk controls
- H3: Downgrading After The SQLite Flip
- H2: Cron sessions and run logs
- H2: Session keys (sessionKey)
- H2: Session ids (sessionId)
- H2: Session store schema (sessions.json)
- H2: Transcript structure (.jsonl)
- H2: Session store schema
- H2: Transcript event structure
- H2: Context windows vs tracked tokens
- H2: Compaction: what it is
- H3: Chunk boundaries and tool pairing
@@ -9935,7 +9961,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
- H2: Access
- H2: What gets recorded
- H2: Bundle files
- H2: Capture location
- H2: Capture storage
- H2: Disable capture
- H2: Tune flush timeout
- H2: Privacy and limits
docs/gateway/config-agents.md
+4 -4
View File
@@ -661,10 +661,10 @@ Periodic heartbeat runs.
- `postIndexSync`: post-compaction session-memory reindex mode. Default: `"async"`. Use `"await"` for strongest freshness, `"async"` for lower compaction latency, or `"off"` only when session-memory sync is handled elsewhere.
- `postCompactionSections`: optional AGENTS.md H2/H3 section names to re-inject after compaction. Reinjection is disabled when unset or set to `[]`. Explicitly setting `["Session Startup", "Red Lines"]` enables that pair and preserves the legacy `Every Session`/`Safety` fallback. Enable this only when the extra context is worth the risk of duplicating project guidance already captured in the compaction summary.
- `model`: optional `provider/model-id` or bare alias from `agents.defaults.models` for compaction summarization only. Bare aliases resolve before dispatch; configured literal model IDs retain precedence on collisions. Use this when the main session should keep one model but compaction summaries should run on another; when unset, compaction uses the session's primary model.
- `truncateAfterCompaction`: rotates the active session JSONL after compaction so future turns load only the summary and unsummarized tail, while the previous full transcript remains archived. Prevents unbounded active transcript growth in long-running sessions. Default: `false`.
- `maxActiveTranscriptBytes`: optional byte threshold (`number` or strings like `"20mb"`) that triggers normal local compaction before a run when the active JSONL grows past the threshold. Requires `truncateAfterCompaction` so successful compaction can rotate to a smaller successor transcript. Disabled when unset or `0`.
- `truncateAfterCompaction`: rotates the active session transcript after compaction so future turns load only the summary and unsummarized tail, while the previous full transcript remains archived. Prevents unbounded active transcript growth in long-running sessions. Default: `false`.
- `maxActiveTranscriptBytes`: optional byte threshold (`number` or strings like `"20mb"`) that triggers normal local compaction before a run when transcript history grows past the threshold. Requires `truncateAfterCompaction` so successful compaction can rotate to a smaller successor transcript. Disabled when unset or `0`.
- `notifyUser`: when `true`, sends brief context-maintenance notices to the user: when compaction starts and completes (for example, "Compacting context..." and "Compaction complete"), and when a pre-compaction memory flush is exhausted so the reply continues in a degraded state (for example, "Memory maintenance temporarily failed; continuing your reply."). Disabled by default to keep these notices silent.
- `memoryFlush`: silent agentic turn before auto-compaction to store durable memories. Set `model` to an exact provider/model such as `ollama/qwen3:8b` when this housekeeping turn should stay on a local model; the override does not inherit the active session fallback chain. `forceFlushTranscriptBytes` forces the flush when transcript file size reaches the threshold even if token counters are stale. Skipped when workspace is read-only.
- `memoryFlush`: silent agentic turn before auto-compaction to store durable memories. Set `model` to an exact provider/model such as `ollama/qwen3:8b` when this housekeeping turn should stay on a local model; the override does not inherit the active session fallback chain. `forceFlushTranscriptBytes` forces the flush when transcript size reaches the threshold even if token counters are stale. Skipped when workspace is read-only.
### `agents.defaults.runRetries`
@@ -1325,7 +1325,7 @@ See [Multi-Agent Sandbox & Tools](/tools/multi-agent-sandbox-tools) for preceden
- **`maintenance`**: session-store cleanup + retention controls.
- `mode`: `enforce` applies cleanup and is the default; `warn` emits warnings only.
- `pruneAfter`: age cutoff for stale entries (default `30d`).
- `maxEntries`: maximum number of entries in `sessions.json` (default `500`). Runtime writes batch cleanup with a small high-water buffer for production-sized caps; `openclaw sessions cleanup --enforce` applies the cap immediately.
- `maxEntries`: maximum number of SQLite session entries (default `500`). Runtime writes batch cleanup with a small high-water buffer for production-sized caps; `openclaw sessions cleanup --enforce` applies the cap immediately.
- Short-lived gateway model-run probe sessions use fixed `24h` retention, but cleanup is pressure-gated: it only removes stale strict model-run probe rows when session-entry maintenance/cap pressure is reached. Only strict explicit probe keys matching `agent:*:explicit:model-run-<uuid>` are eligible; normal direct, group, thread, cron, hook, heartbeat, ACP, and sub-agent sessions do not inherit this 24h retention. When model-run cleanup runs, it runs before the broader `pruneAfter` stale-entry cleanup and `maxEntries` cap.
- `rotateBytes`: deprecated and ignored; `openclaw doctor --fix` removes it from older configs.
- `resetArchiveRetention`: retention for `*.reset.<timestamp>` transcript archives. Defaults to `pruneAfter`; set `false` to disable.
docs/gateway/configuration-reference.md
+1 -1
View File
@@ -1504,7 +1504,7 @@ Current builds no longer include the TCP bridge. Nodes connect over the Gateway
}
```
- `sessionRetention`: how long to keep completed isolated cron run sessions before pruning from `sessions.json`. Also controls cleanup of archived deleted cron transcripts. Default: `24h`; set `false` to disable.
- `sessionRetention`: how long to keep completed isolated cron run sessions before pruning SQLite session rows. Also controls cleanup of archived deleted cron transcripts. Default: `24h`; set `false` to disable.
- `runLog.maxBytes`: accepted for compatibility with older file-backed cron run logs. Default: `2_000_000` bytes.
- `runLog.keepLines`: newest SQLite run-history rows retained per job. Default: `2000`.
- `webhookToken`: bearer token used for cron webhook POST delivery (`delivery.mode = "webhook"`), if omitted no auth header is sent.
docs/gateway/configuration.md
+1 -1
View File
@@ -430,7 +430,7 @@ candidate contains a redacted secret placeholder such as `***` or `[redacted]`.
}
```
- `sessionRetention`: prune completed isolated run sessions from `sessions.json` (default `24h`; set `false` to disable).
- `sessionRetention`: prune completed isolated run sessions from SQLite session rows (default `24h`; set `false` to disable).
- `runLog`: prune retained cron run-history rows per job. History is stored in SQLite; `maxBytes` (default `2_000_000`) is retained for compatibility with older file-backed run logs, `keepLines` defaults to `2000`.
- See [Cron jobs](/automation/cron-jobs) for feature overview and CLI examples.
docs/gateway/health.md
+1 -1
View File
@@ -29,7 +29,7 @@ health commands above for live connectivity checks.
## Deep diagnostics
- Creds on disk: `ls -l ~/.openclaw/credentials/whatsapp/<accountId>/creds.json` (mtime should be recent).
- Session store: `ls -l ~/.openclaw/agents/<agentId>/sessions/sessions.json` (path can be overridden in config). Count and recent recipients are surfaced via `status`.
- Session store: `ls -l ~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`. Count and recent recipients are surfaced via `status`.
- Relink flow: `openclaw channels logout && openclaw channels login --verbose` when status codes 409-515 or `loggedOut` appear in logs. The QR login flow auto-restarts once for status 515 after pairing.
- Diagnostics are enabled by default (`diagnostics.enabled: false` disables them). Memory events record RSS/heap byte counts and threshold/growth pressure; critical memory pressure logs through the gateway logger and, when `diagnostics.memoryPressureSnapshot: true` is set, also writes a pre-OOM stability bundle (V8 heap stats, Linux cgroup counters when available, active resource counts, largest session/transcript files by redacted relative path). Liveness warnings record event-loop delay/utilization, CPU-core ratio, and active/waiting/queued session counts when the process is running but saturated. Oversized-payload events record what was rejected/truncated/chunked plus sizes and limits, never message text, attachment contents, webhook bodies, raw request/response bodies, tokens, cookies, or secret values.
- The same heartbeat drives the bounded stability recorder: `openclaw gateway stability` (or the `diagnostics.stability` Gateway RPC). Fatal Gateway exits, shutdown timeouts, restart startup failures, and (when `diagnostics.memoryPressureSnapshot: true`) critical memory pressure persist the latest snapshot under `~/.openclaw/logs/stability/`. Inspect the newest bundle with `openclaw gateway stability --bundle latest`.
docs/gateway/security/index.md
+3 -2
View File
@@ -717,7 +717,7 @@ The Control UI needs a secure context (HTTPS or localhost) to generate device id
Assume anything under `~/.openclaw/` (or `$OPENCLAW_STATE_DIR/`) may contain secrets or private data:
| Path | Contents |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `openclaw.json` | Config may include tokens (gateway, remote gateway), provider settings, and allowlists. |
| `credentials/**` | Channel credentials (for example WhatsApp creds), pairing allowlists, legacy OAuth imports. |
| `agents/<agentId>/agent/auth-profiles.json` | API keys, token profiles, OAuth tokens, optional `keyRef`/`tokenRef`. |
@@ -725,7 +725,8 @@ Assume anything under `~/.openclaw/` (or `$OPENCLAW_STATE_DIR/`) may contain sec
| `$CODEX_HOME/**` or `~/.codex/**` | Native Codex runtime state. The ordinary harness accesses it only with explicit `plugins.entries.codex.config.appServer.homeScope: "user"`. The separate supervision connection accesses it when its resolved home scope is `"user"`, which is the default for stdio or Unix when unset. Contains the native Codex account, config, plugins, and thread store. Supervision lists source metadata and keeps a continued Chat's canonical native branch and later turns on that connection; branching copies bounded persisted user and assistant history into an authenticated, model-locked OpenClaw Chat. Enable only for an owner-controlled Gateway. See [Codex harness](/plugins/codex-harness#share-threads-with-codex-desktop-and-cli) and [Codex supervision](/plugins/codex-supervision). |
| `secrets.json` (optional) | File-backed secret payload used by `file` SecretRef providers (`secrets.providers`). |
| `agents/<agentId>/agent/auth.json` | Legacy compatibility file; static `api_key` entries are scrubbed when discovered. |
| `agents/<agentId>/sessions/**` | Session transcripts (`*.jsonl`) + routing metadata (`sessions.json`) that can contain private messages and tool output. |
| `agents/<agentId>/agent/openclaw-agent.sqlite` | Per-agent runtime state, including session rows and transcripts that can contain private messages and tool output. |
| `agents/<agentId>/sessions/**` | Legacy session migration sources and archives that can contain private messages and tool output. |
| bundled plugin packages | Installed plugins (plus their `node_modules/`). |
| `sandboxes/**` | Tool sandbox workspaces; can accumulate copies of files read/written inside the sandbox. |
docs/help/faq-first-run.md
+1 -1
View File
@@ -225,7 +225,7 @@ and troubleshooting see the main [FAQ](/help/faq).
**Important:** if you only commit/push your workspace to GitHub, you back up
**memory + bootstrap files**, but not session history or auth. Those live under
`~/.openclaw/` (for example `~/.openclaw/agents/<agentId>/sessions/`).
`~/.openclaw/` (for example `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`).
Related: [Migrating](/install/migrating), [Where things live on disk](/help/faq#where-things-live-on-disk),
[Agent workspace](/concepts/agent-workspace), [Doctor](/gateway/doctor),
docs/help/faq.md
+5 -5
View File
@@ -462,9 +462,9 @@ First-run Q&A - install, onboard, auth routes, subscriptions, initial failures -
| `$OPENCLAW_STATE_DIR/secrets.json` | Optional file-backed secret payload for `file` SecretRef providers |
| `$OPENCLAW_STATE_DIR/agents/<agentId>/agent/auth.json` | Legacy compatibility file (static `api_key` entries scrubbed) |
| `$OPENCLAW_STATE_DIR/credentials/` | Provider state (for example `whatsapp/<accountId>/creds.json`) |
| `$OPENCLAW_STATE_DIR/agents/` | Per-agent state (agentDir + sessions) |
| `$OPENCLAW_STATE_DIR/agents/<agentId>/sessions/` | Conversation history and state (per agent) |
| `$OPENCLAW_STATE_DIR/agents/<agentId>/sessions/sessions.json` | Session metadata (per agent) |
| `$OPENCLAW_STATE_DIR/agents/` | Per-agent state (agentDir + legacy/archive session artifacts) |
| `$OPENCLAW_STATE_DIR/agents/<agentId>/agent/openclaw-agent.sqlite` | Per-agent SQLite state, including session rows and transcripts |
| `$OPENCLAW_STATE_DIR/agents/<agentId>/sessions/` | Legacy session migration sources and archive/support artifacts |
Legacy single-agent path `~/.openclaw/agent/*` is migrated by `openclaw doctor`.
@@ -1111,11 +1111,11 @@ First-run Q&A - install, onboard, auth routes, subscriptions, initial failures -
<Accordion title="How many workspaces and agents can I create?">
No hard limits - dozens or even hundreds are fine, but watch:
- **Disk growth**: sessions and transcripts live under `~/.openclaw/agents/<agentId>/sessions/`.
- **Disk growth**: active sessions and transcripts live in the per-agent SQLite database; legacy/archive artifacts can still accumulate under `~/.openclaw/agents/<agentId>/sessions/`.
- **Token cost**: more agents means more concurrent model usage.
- **Ops overhead**: per-agent auth profiles, workspaces, and channel routing.
Keep one **active** workspace per agent (`agents.defaults.workspace`), prune old sessions if disk grows, and use `openclaw doctor` to spot stray workspaces and profile mismatches.
Keep one **active** workspace per agent (`agents.defaults.workspace`), prune old sessions with `openclaw sessions cleanup` if disk grows (do not edit active SQLite state by hand), and use `openclaw doctor` to spot stray workspaces and profile mismatches.
</Accordion>
docs/install/docker.md
+1 -1
View File
@@ -409,7 +409,7 @@ Installed downloadable plugins store package state under the mounted OpenClaw ho
For full VM persistence details, see [Docker VM Runtime - What persists where](/install/docker-vm-runtime#what-persists-where).
**Disk growth hotspots:** `media/`, session JSONL files, the shared SQLite state database, installed plugin package roots, and rolling file logs under `/tmp/openclaw/`.
**Disk growth hotspots:** `media/`, per-agent SQLite databases, legacy session JSONL transcripts, the shared SQLite state database, installed plugin package roots, and rolling file logs under `/tmp/openclaw/`.
### Shell helpers (optional)
docs/openclaw-agent-runtime.md
+2 -2
View File
@@ -55,13 +55,13 @@ State lives in the OpenClaw state directory: `~/.openclaw` by default, or
| `state/openclaw.sqlite` | Shared runtime state database |
| `agents/<agentId>/agent/openclaw-agent.sqlite` | Per-agent model auth profiles (API keys + OAuth) and runtime state |
| `credentials/` | Provider/channel credentials outside the auth profile store |
| `agents/<agentId>/sessions/` | Session transcripts plus the `sessions.json` index |
| `agents/<agentId>/sessions/` | Transcript history and legacy session migration sources |
| `sessions/` | Legacy single-agent session store (old installs only) |
| `workspace/` | Default agent workspace (extra agents use `workspace-<agentId>`) |
Delete those paths for a full reset. Narrower resets:
- Sessions only: delete `agents/<agentId>/sessions/` for that agent.
- Sessions only: do not delete `agents/<agentId>/agent/openclaw-agent.sqlite`; session rows live there alongside other per-agent state. Use `/new` or `/reset` to start a fresh session for one chat, and `openclaw sessions cleanup` for session maintenance.
- Keep auth: leave `agents/<agentId>/agent/openclaw-agent.sqlite` and `credentials/` in place.
Legacy `auth-profiles.json` files are no longer read at runtime;
docs/plan/path3-sqlite-session-artifact-family.md
+105
View File
@@ -0,0 +1,105 @@
---
summary: "Path 3 plan for archiving all SQLite transcript artifacts that belong to a session"
read_when:
- You are implementing clawdbot-d63.2 / clawdbot-04b
- You are touching SQLite session retention, reset, delete, or agent-deletion archival
- You need to distinguish SQLite-era artifact families from legacy JSONL sidecars
title: "Path 3 SQLite session artifact family"
---
# Path 3 SQLite Session Artifact Family
This note scopes `clawdbot-d63.2` while `clawdbot-d63.1` owns the overlapping
reset/delete archive helper in `src/config/sessions/session-accessor.sqlite.ts`.
The implementation file was dirty during this pass, so this artifact records
the exact contract and patch points without racing the sibling worker.
## Authoritative family
After the SQLite flip, active session transcripts are SQLite rows. A session's
archive family is:
- The `transcript_events`, `transcript_event_identities`, and `sessions` rows
for the entry's current `sessionId`.
- The same SQLite transcript row set for every `sessionId` referenced by
`entry.compactionCheckpoints[*].preCompaction.sessionId`.
- The same SQLite transcript row set for every `sessionId` referenced by
`entry.compactionCheckpoints[*].postCompaction.sessionId`.
- The same SQLite transcript row set for every `sessionId` in
`entry.usageFamilySessionIds`.
Archive only rows that are no longer referenced by any remaining
`session_entries` row or by any remaining entry's compaction or usage-family
metadata. This preserves checkpoint branch/restore and usage rollup state until
the final live reference is gone.
## Non-family artifacts after the flip
Generated topic transcript file variants and trajectory sidecars are not active
SQLite runtime state. They are legacy file artifacts:
- Topic variants such as `<sessionId>-topic-<thread>.jsonl` only exist for the
file-backed transcript format. SQLite uses the canonical session id plus
`session_routes`/entry delivery metadata instead of per-topic JSONL files.
- Trajectory sidecars such as `.trajectory.jsonl` and `.trajectory-path.json`
are named from real JSONL `sessionFile` paths. SQLite `sessionFile` values are
`sqlite:<agentId>:<sessionId>:<storePath>` markers and do not name sidecar
files.
- Archive-tier readers must keep reading legacy archived JSONL files, but
runtime retention must not scan active sessions directories or reopen JSONL
transcript files for SQLite sessions.
Doctor import remains the migration owner for legacy primary JSONL files and
their adjacent trajectory sidecars. Runtime SQLite retention should not add a
second importer or file fallback.
## Patch points
Extend the SQLite archive helper introduced by `clawdbot-d63.1` rather than
adding a parallel path.
1. Add a local collector near `deleteSqliteSessionStateIfUnreferenced`:
- `collectSqliteSessionArtifactFamily(entry: SessionEntry): Set<string>`
- Include `entry.sessionId`, checkpoint pre/post session ids, and
`usageFamilySessionIds`.
- Filter empty strings and dedupe deterministically.
2. Add a reference collector for the post-removal store:
- `readReferencedSqliteSessionArtifactFamilyIds(database): Set<string>`
- Iterate current `session_entries`, parse each `entry_json`, and collect
the same family ids from every surviving entry.
3. Change the reset/delete/maintenance callers that currently archive one
removed `sessionId` to pass the removed entry's full family.
4. For each family id, archive the SQLite transcript rows with the caller's
reason (`reset` or `deleted`), then delete the `sessions` row only when the
family id is absent from the post-removal reference set.
5. Keep transcript event deletion centralized through the existing SQLite
session-row cleanup path. Do not add active JSONL reads.
## Focused tests
Add SQLite-only tests to `src/config/sessions/session-accessor.conformance.test.ts`
or the sibling lifecycle test after `clawdbot-d63.1` commits:
- Deleting an entry with a pre-compaction transcript archives both the current
session and the pre-compaction session, then removes both SQLite row sets.
- Deleting one of two entries that share a compaction pre-session archives
nothing for the shared pre-session until the final referencing entry is
removed.
- Deleting an entry with `usageFamilySessionIds` archives predecessor SQLite
transcript rows when no other entry references that usage family.
- A topic-shaped session key with a SQLite marker does not cause any generated
topic JSONL read or sidecar lookup.
The focused proof should use:
```bash
node scripts/run-vitest.mjs src/config/sessions/session-accessor.conformance.test.ts
```
If the final tests live in `store.session-lifecycle-mutation.test.ts`, run that
file explicitly with the same wrapper. Broad `pnpm` gates should stay on
Crabbox/Testbox for this Codex worktree.
docs/plugins/sdk-migration.md
+29 -1
View File
@@ -459,7 +459,7 @@ SDK.
| `plugin-sdk/reply-history` | Reply-history helpers | `createChannelHistoryWindow`; deprecated map-helper compatibility exports such as `buildPendingHistoryContextFromMap`, `recordPendingHistoryEntry`, and `clearHistoryEntriesIfEnabled` |
| `plugin-sdk/reply-reference` | Reply reference planning | `createReplyReferencePlanner` |
| `plugin-sdk/reply-chunking` | Reply chunk helpers | Text/markdown chunking helpers |
| `plugin-sdk/session-store-runtime` | Session store helpers | Store path + updated-at helpers |
| `plugin-sdk/session-store-runtime` | Session store helpers | Scoped session row helpers, store path helpers, and updated-at reads |
| `plugin-sdk/state-paths` | State path helpers | State and OAuth dir helpers |
| `plugin-sdk/routing` | Routing/session-key helpers | `resolveAgentRoute`, `buildAgentSessionKey`, `resolveDefaultAgentBoundAccountId`, session-key normalization helpers |
| `plugin-sdk/status-helpers` | Channel status helpers | Channel/account status summary builders, runtime-state defaults, issue metadata helpers |
@@ -803,6 +803,34 @@ major release. Every entry maps the old API to its canonical replacement.
</Accordion>
<Accordion title="Removed session and transcript file APIs">
The SQLite session/transcript flip removes plugin-facing APIs that exposed
active `sessions.json` stores, JSONL transcript paths, or lists of session
files. Runtime plugins should use session identity and SDK runtime helpers
instead of resolving or mutating active files.
| Removed surface | Replacement |
| ---------------- | ----------- |
| `loadSessionStore(...)`, `saveSessionStore(...)`, `updateSessionStore(...)` | Gateway-owned session runtime APIs; plugin code should request session state through documented runtime/context helpers instead of reading the active store file. |
| `resolveSessionFilePath(...)`, `resolveSessionTranscriptPathInDir(...)`, `resolveAndPersistSessionFile(...)` | Session identity (`sessionKey`, `sessionId`, and SDK runtime target helpers) plus Gateway methods that operate on the current session. |
| `readLatestAssistantTextFromSessionTranscript(...)` | Identity-backed transcript readers exposed by the current runtime context, or Gateway history/session methods when the plugin is outside the transcript owner path. |
| `SessionTranscriptUpdate.sessionFile` | `SessionTranscriptUpdate.target` with `agentId`, `sessionKey`, and `sessionId`. |
| Memory sync inputs such as `sessionFiles` | Identity-backed transcript/session sources provided by the host; do not crawl active JSONL files for live sessions. |
| Runtime options named `transcriptPath` or `sessionFile` for active sessions | `sessionTarget`/runtime target objects that carry storage-neutral session identity. |
Legacy JSONL transcript files remain valid as import, archive, export, and
support artifacts. They are no longer the steady-state runtime contract for
active sessions.
`openclaw plugins inspect --all --runtime` reports non-bundled plugins whose
load errors or diagnostics still reference these removed file APIs. The
`@openclaw/plugin-inspector` advisory sweep must use version `0.3.17` or
newer so external package scans also flag whole-store session helpers,
session file-path helpers, legacy transcript file targets, and low-level
transcript helpers before release.
</Accordion>
<Accordion title="runtime.tasks.flow -> runtime.tasks.managedFlows">
**Old**: `runtime.tasks.flow` (singular) returned a live task-flow
accessor.
docs/plugins/sdk-runtime.md
+8 -2
View File
@@ -190,11 +190,17 @@ two-party event loops that do not go through the shared inbound reply runner.
Use `runWithWorkAdmission(...)` when a plugin starts work on a persisted session. The callback rejects archived or concurrently replaced sessions, keeps archive/reset/delete mutations coordinated through completion, and receives an `AbortSignal` that must be forwarded to the agent run. A harness may explicitly name trusted execution delegates through its experimental `delegatedExecutionPluginIds` registration field. Delegates can admit and run only an exact existing model-locked session; all session mutations remain restricted to the harness owner. See [Agent harness plugins](/plugins/sdk-agent-harness#delegated-execution).
For transcript reads and writes, import `openclaw/plugin-sdk/session-transcript-runtime` and use `resolveSessionTranscriptIdentity(...)`, `resolveSessionTranscriptTarget(...)`, `readSessionTranscriptEvents(...)`, `appendSessionTranscriptMessageByIdentity(...)`, `publishSessionTranscriptUpdateByIdentity(...)`, or `withSessionTranscriptWriteLock(...)` with `{ agentId, sessionKey, sessionId }`. These APIs let plugins identify a transcript, read its events, append messages, publish updates, and run related operations under the same transcript write lock. Passing `sessionFile`, using `resolveSessionTranscriptLegacyFileTarget(...)`, or importing low-level `appendSessionTranscriptMessage(...)` / `emitSessionTranscriptUpdate(...)` from `openclaw/plugin-sdk/agent-harness-runtime` is deprecated; those paths exist only for legacy code that already receives an active transcript artifact.
Maintenance and repair plugins may use `deleteSessionEntry(...)` for one scoped session entry, `cleanupSessionLifecycleArtifacts(...)` for lifecycle-owned scratch sessions, and `resolveSessionStoreBackupPaths(...)` before mutating a store. These helpers are narrow repair/lifecycle surfaces, not a general store deletion API.
`resolveStorePath(...)` and `updateSessionStoreEntry(...)` round out the session helpers: `resolveStorePath` resolves the session store path for a given scope, and `updateSessionStoreEntry({ storePath, sessionKey, update })` patches one entry directly by store path when the caller already knows it.
`loadSessionStore(...)`, `saveSessionStore(...)`, `updateSessionStore(...)`, and `resolveSessionFilePath(...)` are deprecated compatibility helpers for plugins that still intentionally depend on the legacy whole-store or transcript-file shape. New plugin code must not use those helpers, and existing callers should migrate to entry helpers and transcript identity helpers.
`loadTranscriptEventsSync(...)` is available for synchronous doctor and repair paths that cannot use the async transcript runtime. It returns raw `SessionStoreTranscriptEvent` records. Normal plugin runtime code should prefer `openclaw/plugin-sdk/session-transcript-runtime`.
`formatSqliteSessionFileMarker(...)`, `parseSqliteSessionFileMarker(...)`, and `sqliteSessionFileMarkerMatchesSession(...)` are transitional helpers for code that still receives a legacy field named `sessionFile`. A parsed SQLite marker identifies a live SQLite transcript target; it is not a filesystem path. New APIs should carry typed session identity instead of marker strings.
For transcript reads and writes, import `openclaw/plugin-sdk/session-transcript-runtime` and use `resolveSessionTranscriptIdentity(...)`, `resolveSessionTranscriptTarget(...)`, `readSessionTranscriptEvents(...)`, `readVisibleSessionTranscriptMessageEntries(...)`, `appendSessionTranscriptMessageByIdentity(...)`, `publishSessionTranscriptUpdateByIdentity(...)`, or `withSessionTranscriptWriteLock(...)` with `{ agentId, sessionKey, sessionId }`. These APIs let plugins identify a transcript, read raw events or visible branch-safe message entries, append messages, publish updates, and run related operations under the same transcript write lock without depending on active transcript file paths. `readVisibleSessionTranscriptMessageEntries(...)` returns ordered read metadata; its `seq` field is not a resumable cursor.
The legacy whole-store and active transcript file helpers are no longer exported from the plugin SDK. Use the scoped entry helpers for session metadata and the transcript identity helpers for active transcript operations. Archive/support workflows that need file artifacts should use their dedicated archive surfaces instead of active session runtime APIs.
</Accordion>
<Accordion title="api.runtime.agent.defaults">
docs/plugins/sdk-subpaths.md
+2 -2
View File
@@ -272,8 +272,8 @@ usage endpoint failed or returned no usable usage data.
| `plugin-sdk/reply-history` | Shared short-window reply-history helpers. New message-turn code should use `createChannelHistoryWindow`; lower-level map helpers remain deprecated compatibility exports only |
| `plugin-sdk/reply-reference` | `createReplyReferencePlanner` |
| `plugin-sdk/reply-chunking` | Narrow text/markdown chunking helpers |
| `plugin-sdk/session-store-runtime` | Session workflow helpers (`getSessionEntry`, `listSessionEntries`, `patchSessionEntry`, `upsertSessionEntry`), bounded recent user/assistant transcript text reads by session identity, legacy session store path/session-key helpers, updated-at reads, and transition-only whole-store/file-path compatibility helpers, without broad config writes/maintenance imports |
| `plugin-sdk/session-transcript-runtime` | Transcript identity, scoped target/read/write helpers, update publishing, write locks, and transcript memory hit keys |
| `plugin-sdk/session-store-runtime` | Session workflow helpers (`getSessionEntry`, `listSessionEntries`, `patchSessionEntry`, `upsertSessionEntry`), repair/lifecycle helpers (`deleteSessionEntry`, `cleanupSessionLifecycleArtifacts`, `resolveSessionStoreBackupPaths`), marker helpers for transitional `sessionFile` values, bounded recent user/assistant transcript text reads by session identity, session store path/session-key helpers, and updated-at reads, without broad config writes/maintenance imports |
| `plugin-sdk/session-transcript-runtime` | Transcript identity, scoped target/read/write helpers, visible message-entry projection, update publishing, write locks, and transcript memory hit keys |
| `plugin-sdk/sqlite-runtime` | Focused SQLite agent-schema, path, and transaction helpers for first-party runtime, without database lifecycle controls |
| `plugin-sdk/cron-store-runtime` | Cron store path/load/save helpers |
| `plugin-sdk/state-paths` | State/OAuth dir path helpers |
docs/refactor/database-first.md
+9 -7
View File
@@ -264,7 +264,8 @@ file world:
status payload field. Runtime and bridge tests no longer contain the
`storePath` contract name; doctor/migration code owns that legacy vocabulary.
- Session writes no longer pass through the old in-process `store-writer.ts`
queue. SQLite patch writes use conflict detection and bounded retry instead.
queue. SQLite patch writes prepare outside the transaction, then use a short
synchronous validate/apply transaction with explicit conflict detection.
- Legacy path discovery still has valid migration uses, but runtime code should
stop treating `sessions.json` and transcript JSONL files as possible write
targets.
@@ -582,10 +583,10 @@ Completed consolidation/deletion highlights:
- Channel session runtime types now expose `{agentId, sessionKey}` for
updated-at reads, inbound metadata, and last-route updates. The old
`saveSessionStore(storePath, store)` compatibility type is gone.
- Plugin runtime, extension API, and `config/sessions` barrel surfaces now steer
plugin code to SQLite-backed session row helpers. Root library compatibility
exports (`loadSessionStore`, `saveSessionStore`, `resolveStorePath`) remain as
deprecated shims for existing consumers. The old
- Plugin runtime, extension API, and plugin SDK session surfaces now expose
SQLite-backed session row helpers instead of active-session whole-store/file
compatibility helpers. Root library compatibility exports remain available
only outside the plugin SDK for legacy internal and migration callers. The old
`resolveLegacySessionStorePath` helper is gone; legacy `sessions.json` path
construction is now local to migration and test fixtures.
- `src/config/sessions/session-entries.sqlite.ts` now stores canonical session
@@ -2114,8 +2115,9 @@ restore` validates before extraction, uses the verifier's normalized
- One connection per thread/process is fine; do not share handles across
workers.
- Use WAL, `foreign_keys=ON`, a 30s busy timeout, and short `BEGIN IMMEDIATE`
write transactions.
- Use WAL, `foreign_keys=ON`, a 5s busy timeout, and short `BEGIN IMMEDIATE`
write transactions. Do not layer synchronous lock retries above SQLite's
single busy wait.
- Keep write transaction helpers synchronous unless/until an async transaction
API adds explicit mutex/backpressure semantics.
- Keep parent delivery writes small and transactional.
docs/reference/path3-live-sqlite-e2e-harness.md
+168
View File
@@ -0,0 +1,168 @@
---
summary: "Design for live Gateway proof of the Path 3 SQLite session/transcript flip"
read_when:
- You are proving the Path 3 SQLite storage flip against a live Gateway
- You need to distinguish expected legacy JSONL drift from runtime failures
- You are building or reviewing the agent-driven live SQLite E2E harness
title: "Path 3 live SQLite E2E harness"
---
The Path 3 live SQLite E2E harness proves the Gateway is using SQLite as the
canonical session and transcript store while legacy JSONL files remain
migration input or archive material. It is a maintainer proof harness, not a
normal user diagnostic.
After a Gateway has processed post-migration traffic, legacy JSONL parity is no
longer a valid runtime health signal. A healthy migrated Gateway can have
SQLite transcript rows that differ from legacy JSONL counts because new turns
should advance SQLite only. The live harness must therefore measure Gateway
behavior, SQLite row movement, legacy-file quiescence, and log health at each
step.
## Command shape
The intended live command is:
```bash
node scripts/path3-live-sqlite-e2e.mjs \
--url http://127.0.0.1:18789 \
--agent main \
--session-key agent:main:path3-live-e2e:<timestamp> \
--json
```
The command connects to an already running Gateway. It does not start, stop,
import, or re-run the migration unless an explicit migration mode is added
later. A CI or isolated-local variant can use
`test/helpers/openclaw-test-instance.ts`, but the live proof path should inspect
the actual operator Gateway and its real per-agent SQLite database.
## Isolated built-CLI proof
The built-CLI proof runner seeds an isolated legacy session store, starts the
rebuilt Gateway, and proves that startup imports hot legacy sessions into
SQLite before runtime reads begin. It must not run `openclaw doctor --fix`
before the first Gateway start, because that would prove the manual migration
path instead of the upgrade path users receive on first boot after the flip.
After startup import, the isolated proof may run
`openclaw doctor --session-sqlite inspect` and
`openclaw doctor --session-sqlite validate` as diagnostic evidence. Those
doctor commands are not the migration driver for the startup-upgrade proof.
Separate doctor-import scenarios should seed legacy transcript files plus
trajectory sidecars and verify doctor archives those artifacts while SQLite
remains canonical.
## Preflight
Preflight collects a baseline and fails before sending a proof turn if the
Gateway is not usable:
- `GET /health` and Gateway deep status must report a running, reachable
Gateway.
- The CLI and Gateway versions must match the branch being tested.
- The harness records a log cursor for the active Gateway file log.
- The harness records per-agent SQLite table counts for `sessions`,
`session_entries`, `transcript_events`, `transcript_event_identities`, and
`session_routes`.
- The harness records `mtime`, `size`, and existence for legacy
`sessions.json`, referenced JSONL files, and candidate proof-session JSONL
paths.
- `lsof -p <gateway-pid>` must show SQLite DB/WAL/SHM handles and no hot
`.jsonl` or `sessions.json` handles.
`openclaw doctor --session-sqlite validate` is informational only in live mode.
After post-flip traffic it may report expected drift against legacy files. The
harness should use doctor output for classification and migration inventory,
not as the runtime pass/fail oracle.
## Agent-driven scenario
The live scenario uses a dedicated proof session key and drives the Gateway
through public RPC paths wherever possible. One agent turn should be enough to
exercise ordinary persistence, but the full proof should cover the 3.1b seams
that previously required individual live checks:
- Ordinary chat turn: create or reuse the proof session, send a real agent
prompt, wait for the final assistant result, and verify `chat.history` or
equivalent Gateway projection.
- Transcript identity: verify the same marker appears in Gateway history and in
SQLite transcript rows, including stable event identity rows when present.
- Session metadata accessors: read the proof session and selected existing live
sessions through Gateway/session accessors and compare them to SQLite rows.
- Session patch projection: apply a reversible model/session metadata change on
the proof session, then verify the projected row and Gateway response agree.
- Compaction checkpoint lifecycle: list, branch, and restore a checkpoint only
on the proof session or a synthetic fixture session created by the harness.
- Restart recovery: run the safe recovery marker path against a controlled proof
session or an isolated test instance; live mode may only run this step when
the target session set is explicit and reversible.
- Cleanup lifecycle: delete or reset the proof session, then verify SQLite
lifecycle rows and archived transcript state.
Transport-specific seams that cannot be exercised safely on the live operator
Gateway, such as WhatsApp or voice-call ingress, should use owner-level runtime
probes against the same SQLite contract rather than fake external transport.
## Per-step assertions
Each step snapshots before and after state and writes a structured assertion
record:
- SQLite row counts advance only where expected.
- Trajectory runtime rows advance for marker-backed proof sessions that record
runtime events.
- The proof session row has the expected `session_id`, status, timestamps,
metadata, and route rows.
- Gateway history/session projection matches the SQLite transcript tail.
- No proof-session JSONL file is created or modified.
- No proof-session `.trajectory.jsonl`, `.trajectory-path.json`, or
marker-derived `trajectory/<session>.jsonl` sidecar is created.
- Existing legacy JSONL files and `sessions.json` remain unchanged unless the
step is explicitly an offline migration or archive operation.
- The Gateway process does not open `.jsonl` or `sessions.json` handles.
- Logs since the previous cursor contain no `ERROR`, `FATAL`, `SQLITE_`,
`no such column`, session-store unavailable, restart-recovery failure, or
transcript-reconcile warning unless the scenario explicitly allowlists it.
The log scan is part of the pass/fail contract. A Gateway that answers health
checks but emits SQLite schema errors or repeated transcript reconcile failures
is not green for Path 3.
## Evidence artifact
The harness should write evidence under `.artifacts/path3-live-e2e/<timestamp>/`
and keep it out of git:
- `summary.json`: command args, Gateway version, result, failed assertion, and
artifact paths.
- `sqlite-before.json` and `sqlite-after.json`: row counts and selected proof
rows.
- `legacy-files.json`: legacy file existence, `mtime`, size, and whether each
file changed.
- `gateway-log-scan.json`: cursor range, matched log lines, and allowlist
decisions.
- `events.jsonl`: ordered per-step observations suitable for PR proof comments.
The PR proof should summarize these artifacts instead of pasting full
transcripts or private message content.
## Safety rules
- Live mode must never re-import legacy JSONL while the Gateway is running.
- Live mode must not mutate non-proof sessions except for explicitly selected,
reversible repair probes.
- Any destructive or broad migration step requires a fresh backup of the
affected SQLite DB and legacy session directory.
- Backups should be scoped to the touched agent DB/session directory and reused
during one proof run to avoid unbounded disk growth.
- The cleanup step must leave no proof session, proof JSONL, or modified legacy
file behind unless the caller passes `--keep-artifacts`.
## Passing result
A passing live run means the Gateway accepted a real agent-driven session flow,
all observed canonical state was in SQLite, legacy runtime files stayed
quiescent, and log health stayed clean for the measured window. It does not mean
legacy JSONL parity remains clean after live traffic; live drift is expected
once SQLite is the canonical store.
docs/reference/session-management-compaction.md
+72 -29
View File
@@ -1,7 +1,7 @@
---
summary: "Deep dive: session store + transcripts, lifecycle, and (auto)compaction internals"
read_when:
- You need to debug session ids, transcript JSONL, or sessions.json fields
- You need to debug session ids, transcript events, or session row fields
- You are changing auto-compaction behavior or adding "pre-compaction" housekeeping
- You want to implement memory flushes or silent system turns
title: "Session management deep dive"
@@ -13,38 +13,56 @@ Overview docs first: [Session management](/concepts/session), [Compaction](/conc
## Two persistence layers
1. **Session store (`sessions.json`)** - key/value map `sessionKey -> SessionEntry`. Small, mutable, safe to edit or delete entries. Tracks metadata: current session id, last activity, toggles, token counters.
2. **Transcript (`<sessionId>.jsonl`)** - append-only, tree-structured (entries have `id` + `parentId`). Stores the conversation, tool calls, and compaction summaries; rebuilds model context for future turns. Compaction checkpoints are metadata over the compacted successor transcript - a new compaction does not write a second `.checkpoint.*.jsonl` copy.
1. **Session rows (per-agent SQLite)** - key/value map `sessionKey -> SessionEntry`. Mutable runtime state owned by the Gateway. Tracks metadata: current session id, last activity, toggles, token counters.
2. **Transcript events (per-agent SQLite)** - append-only, tree-structured (entries have `id` + `parentId`). Stores the conversation, tool calls, and compaction summaries; rebuilds model context for future turns. Compaction checkpoints are metadata over the compacted successor transcript - a new compaction does not write a second `.checkpoint.*.jsonl` copy.
Gateway history readers avoid materializing the whole transcript unless the surface needs arbitrary historical access. First-page history, embedded chat history, restart recovery, and token/usage checks use bounded tail reads. Full transcript scans go through the async transcript index, cached by file path plus `mtimeMs`/`size` and shared across concurrent readers.
Older installs may still have `sessions.json` files under the agent `sessions/`
directory. Treat those files as legacy session-row migration inputs or explicit
offline-maintenance targets. Gateway startup and `openclaw doctor --fix` import
hot legacy rows and transcript history into the per-agent SQLite store
automatically. Run `openclaw doctor --session-sqlite inspect
--session-sqlite-all-agents`, then follow the [Doctor migration
sequence](/cli/doctor#session-sqlite-migration), when you need explicit
inspection or validation evidence. If a migration fails after legacy transcript
artifacts were archived, use the Doctor recovery mode from that sequence.
Recovery uses migration manifests, restores only the affected archived support
artifacts, prepares a sanitized GitHub issue report when requested, and does not
make active runtime read JSONL files again.
Gateway history readers avoid materializing the whole transcript unless the surface needs arbitrary historical access. First-page history, embedded chat history, restart recovery, and token/usage checks use bounded tail reads from SQLite. Full transcript scans go through the async transcript index and are shared across concurrent readers.
## On-disk locations
Per agent, on the Gateway host (resolved via `src/config/sessions.ts`):
- Store: `~/.openclaw/agents/<agentId>/sessions/sessions.json`
- Transcripts: `~/.openclaw/agents/<agentId>/sessions/<sessionId>.jsonl`
- Telegram topic sessions: `.../<sessionId>-topic-<threadId>.jsonl`
- Runtime session row store: `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
- Runtime transcript rows: `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
- Legacy/archive transcript artifacts: `~/.openclaw/agents/<agentId>/sessions/`
- Legacy row migration input: `~/.openclaw/agents/<agentId>/sessions/sessions.json`
## Store maintenance and disk controls
`session.maintenance` controls automatic maintenance for `sessions.json`, transcript artifacts, and trajectory sidecars:
`session.maintenance` controls automatic maintenance for SQLite session rows, SQLite transcript rows, archive artifacts, and trajectory sidecars:
| Key | Default | Notes |
| ----------------------- | --------------------- | --------------------------------------------------------------------------------- |
| ----------------------- | --------------------- | ------------------------------------------------------------------------------------------- |
| `mode` | `"enforce"` | or `"warn"` (report only, no mutation) |
| `pruneAfter` | `"30d"` | stale-entry age cutoff |
| `maxEntries` | `500` | cap on entries in `sessions.json` |
| `resetArchiveRetention` | same as `pruneAfter` | retention for `*.reset.<timestamp>` transcript archives; `false` disables cleanup |
| `maxDiskBytes` | unset | optional sessions-directory budget |
| `maxEntries` | `500` | cap on session entries |
| `resetArchiveRetention` | keep (no age cutoff) | age cutoff for `*.reset.*`/`*.deleted.*` transcript archives; a duration opts into deletion |
| `maxDiskBytes` | `2gb` | per-agent sessions disk budget; `false` disables |
| `highWaterBytes` | 80% of `maxDiskBytes` | target after budget cleanup |
Archived transcripts are kept by default and compressed with zstd (`*.jsonl.<reason>.<timestamp>.zst`) when the runtime supports it, so deleting or resetting a session never silently discards conversation history. The disk budget evicts the oldest archives first, before touching live sessions.
Active SQLite enforcement of `maxDiskBytes` measures session-row JSON plus transcript-event JSON bytes per session; legacy offline-maintenance enforcement measures files in the selected sessions directory.
Gateway model-run probe sessions (keys matching `agent:*:explicit:model-run-<uuid>`) get a separate, fixed `24h` retention. This pruning is pressure-gated: it only runs when session-entry maintenance/cap pressure is reached, and only before the global stale-entry cleanup/cap step. Other explicit sessions do not use this retention.
Enforcement order for disk-budget cleanup (`mode: "enforce"`):
1. Remove oldest archived, orphan transcript, or orphan trajectory artifacts first.
2. If still above target, evict oldest session entries and their transcript/trajectory files.
1. Remove oldest archived transcript artifacts, orphan legacy artifacts, or orphan trajectory artifacts first.
2. If still above target, evict oldest session entries and their transcript rows or trajectory artifacts.
3. Repeat until usage is at or below `highWaterBytes`.
`mode: "warn"` reports potential evictions without mutating the store or files.
@@ -58,11 +76,11 @@ openclaw sessions cleanup --enforce
Maintenance keeps durable external conversation pointers such as group sessions and thread-scoped chat sessions, but synthetic runtime entries (cron, hooks, heartbeat, ACP, sub-agents) can still be removed once they exceed the configured age, count, or disk budget. Isolated cron runs use a separate `cron.sessionRetention` control, independent of model-run probe retention.
Normal Gateway writes flow through a per-store session writer that serializes in-process mutations without taking a runtime file lock. Hot-path patch helpers borrow the validated mutable cache while holding that writer slot, so large `sessions.json` files are not cloned or reread for every metadata update. Prefer `updateSessionStore(...)` / `updateSessionStoreEntry(...)` in runtime code; direct whole-store saves are for compatibility and offline maintenance tools. When a Gateway is reachable, non-dry-run `openclaw sessions cleanup` and `openclaw agents delete` delegate store mutations to the Gateway so cleanup joins the same writer queue; `--store <path>` is the explicit offline repair path for direct file maintenance and always stays local (as does `--dry-run`). `maxEntries` cleanup is batched for production-sized stores, so a store may briefly exceed the configured cap before the next high-water cleanup rewrites it down. Reads never prune or cap entries during Gateway startup - only writes or `openclaw sessions cleanup --enforce` do, and the latter also applies the cap immediately and prunes old unreferenced transcript, checkpoint, and trajectory artifacts even with no disk budget configured.
Normal Gateway writes flow through the session accessor, which serializes per-agent SQLite mutations through the runtime writer path. Runtime code should prefer the accessor helpers in `src/config/sessions/session-accessor.ts`; legacy `sessions.json` helpers are migration and offline-maintenance tools. When a Gateway is reachable, non-dry-run `openclaw sessions cleanup` and `openclaw agents delete` delegate store mutations to the Gateway so cleanup joins the same writer queue; `--store <path>` is the explicit offline repair path for a selected legacy store and always stays local (as does `--dry-run`). `maxEntries` cleanup is batched for production-sized stores, so a store may briefly exceed the configured cap before the next high-water cleanup rewrites it down. Reads never prune or cap entries during Gateway startup - only writes or `openclaw sessions cleanup --enforce` do, and the latter also applies the cap immediately and prunes old unreferenced legacy transcript, checkpoint, and trajectory artifacts even with no disk budget configured.
OpenClaw no longer creates automatic `sessions.json.bak.*` rotation backups during Gateway writes. The legacy `session.maintenance.rotateBytes` key is ignored and `openclaw doctor --fix` removes it from older configs.
Transcript mutations use a session write lock on the transcript file:
Transcript mutations use the session write queue for the SQLite transcript target:
| Setting | Default | Env override |
| ------------------------------------ | --------- | ------------------------------------------------ |
@@ -72,6 +90,28 @@ Transcript mutations use a session write lock on the transcript file:
`acquireTimeoutMs` is how long a lock wait surfaces a busy-session error before giving up; raise it only when legitimate prep, cleanup, compaction, or transcript mirror work contends longer on slow machines. `staleMs` is when an existing lock can be reclaimed as stale. `maxHoldMs` is the in-process watchdog release threshold.
### Downgrading After The SQLite Flip
Restore archived legacy transcript artifacts before running an older
file-backed OpenClaw version:
```bash
openclaw doctor --session-sqlite restore --session-sqlite-all-agents
```
The migration leaves legacy `sessions.json` files in place for support and
rollback, but hot transcript JSONL files that were imported into SQLite are
renamed into `session-sqlite-import-archive/`. Older file-backed runtimes follow
the `sessionFile` paths in `sessions.json`, so they need those artifacts restored
before startup. Restore uses migration manifests, moves only recorded archived
artifacts whose original paths are missing, and leaves the SQLite database in
place for forward recovery.
Sessions created after the SQLite flip are SQLite-only and will not appear to an
older file-backed runtime. If you re-upgrade after a downgrade, run the Doctor
inspection and validation sequence again so OpenClaw can verify restored legacy
artifacts before importing.
## Cron sessions and run logs
Isolated cron runs create their own session entries/transcripts with dedicated retention:
@@ -95,7 +135,7 @@ A `sessionKey` identifies which conversation bucket you are in (routing + isolat
## Session ids (`sessionId`)
Each `sessionKey` points at a current `sessionId` (the transcript file continuing the conversation). Decision logic lives in `initSessionState()` in `src/auto-reply/reply/session.ts`.
Each `sessionKey` points at a current `sessionId` (the SQLite transcript identity that continues the conversation). Decision logic lives in `initSessionState()` in `src/auto-reply/reply/session.ts`.
- **Reset** (`/new`, `/reset`) creates a new `sessionId` for that `sessionKey`.
- **Daily reset** (default 4:00 AM local time on the gateway host) creates a new `sessionId` on the next message after the reset boundary.
@@ -105,11 +145,11 @@ Each `sessionKey` points at a current `sessionId` (the transcript file continuin
- **Parent fork policy** uses OpenClaw's active branch when creating a thread or subagent fork. If that branch is too large (over a fixed internal cap, currently 100K tokens), OpenClaw starts the child with isolated context instead of failing or inheriting unusable history. Sizing is automatic and not configurable; legacy `session.parentForkMaxTokens` config is removed by `openclaw doctor --fix`.
- **Operator forks**: `sessions.create { parentSessionKey, fork: true }` creates a new session whose transcript branches from the parent's current state (same fork machinery as subagent spawns, including the size cap above). The fork is refused while the parent has an active run, inherits the parent's model selection unless one is passed explicitly, and marks the child `forkedFromParent` with fresh token counters.
## Session store schema (`sessions.json`)
## Session store schema
The value type is `SessionEntry` in `src/config/sessions.ts`. Key fields (not exhaustive):
The runtime store keeps `SessionEntry` values in per-agent SQLite. The value type is `SessionEntry` in `src/config/sessions.ts`. Key fields (not exhaustive):
- `sessionId`: current transcript id (filename derives from this unless `sessionFile` is set)
- `sessionId`: current transcript id used to address SQLite transcript rows
- `sessionStartedAt`: start timestamp for the current `sessionId`; daily reset freshness uses this. Legacy rows may derive it from the JSONL session header.
- `lastInteractionAt`: last real user/channel interaction timestamp; idle reset freshness uses this so heartbeat, cron, and exec events do not keep sessions alive. Legacy rows without this field fall back to the recovered session start time.
- `updatedAt`: last store-row mutation timestamp, used for listing/pruning/bookkeeping - not the daily/idle freshness authority.
@@ -119,7 +159,7 @@ The value type is `SessionEntry` in `src/config/sessions.ts`. Key fields (not ex
- Codex supervision lists only non-archived native threads. A Gateway-local `idle` or `notLoaded` activity-unknown thread can be archived through native `thread/archive` only after the operator explicitly confirms that no other Codex process owns it; the plugin performs a fresh process-local status read first, and the thread then disappears from the catalog. That read cannot prove that another App Server process is not using the thread. OpenClaw refuses to archive active and error rows, and paired-node archive is unavailable until the node bridge can own the full streamed thread lifecycle. Unarchiving in a native Codex client makes the thread eligible to appear again.
- `lastReadAt` / `markedUnreadAt`: read-state timestamps stamped server-side by `sessions.patch { unread }` - `unread: false` records a read (sets `lastReadAt`, clears `markedUnreadAt`); `unread: true` marks the session unread until the next read. Session rows expose a derived `unread` boolean: explicitly marked unread, or read before the latest activity. Sessions never marked read stay `unread: false`, so existing installs do not light up on upgrade.
- `lastActivityAt`: timestamp of the last completed agent run that counts as unread-worthy activity (user, channel, and cron runs). Heartbeat and internal-event turns, plus metadata patches, do not update it; `updatedAt` is not an activity signal.
- `sessionFile`: optional explicit transcript path override
- `sessionFile`: legacy marker retained for migration/archive compatibility; active runtime uses SQLite identity
- `chatType`: `direct | group | room`
- `provider`, `subject`, `room`, `space`, `displayName`: group/channel labeling metadata
- Toggles: `thinkingLevel`, `verboseLevel`, `reasoningLevel`, `elevatedLevel`, `sendPolicy` (per-session override)
@@ -128,13 +168,16 @@ The value type is `SessionEntry` in `src/config/sessions.ts`. Key fields (not ex
- `compactionCount`: how many times auto-compaction completed for this session key
- `memoryFlushAt` / `memoryFlushCompactionCount`: timestamp and compaction count of the last pre-compaction memory flush
The store is safe to edit, but the Gateway is the authority: it may rewrite or rehydrate entries as sessions run.
The Gateway is the authority: it may rewrite or rehydrate entries as sessions
run. For legacy file-backed installs, migrate with
`openclaw doctor --session-sqlite import --session-sqlite-all-agents` instead of
editing `sessions.json` and expecting runtime to keep reading that file.
## Transcript structure (`*.jsonl`)
## Transcript event structure
Transcripts are managed by `SessionManager` (`openclaw/plugin-sdk/agent-sessions`). The file is JSONL:
Transcripts are managed by the OpenClaw session accessor and exposed to runtime code through identity-based helpers. The event stream is append-only:
- First line: session header - `type: "session"`, `id`, `cwd`, `timestamp`, optional `parentSession`.
- First entry: session header - `type: "session"`, `id`, `cwd`, `timestamp`, optional `parentSession`.
- Then: entries with `id` + `parentId` (tree structure).
Notable entry types:
@@ -152,7 +195,7 @@ OpenClaw intentionally does not "fix up" transcripts; the Gateway uses `SessionM
Two different concepts:
1. **Model context window**: hard cap per model (tokens visible to the model). Comes from the model catalog and can be overridden via config.
2. **Session store counters**: rolling stats written into `sessions.json` (used for `/status` and dashboards). `contextTokens` is a runtime estimate/reporting value - do not treat it as a strict guarantee.
2. **Session store counters**: rolling stats written into the session row (used for `/status` and dashboards). `contextTokens` is a runtime estimate/reporting value - do not treat it as a strict guarantee.
More on limits: [/reference/token-use](/reference/token-use).
@@ -179,7 +222,7 @@ Two triggers in the embedded OpenClaw agent:
Two additional guards run outside these two triggers:
- **Preflight local compaction**: set `agents.defaults.compaction.maxActiveTranscriptBytes` (bytes or a string like `"20mb"`) to trigger local compaction before opening the next run once the active transcript file reaches that size. This is a file-size guard for local reopen cost, not raw archival - normal semantic compaction still runs, and it requires `truncateAfterCompaction` so the compacted summary becomes a new successor transcript.
- **Preflight local compaction**: set `agents.defaults.compaction.maxActiveTranscriptBytes` (bytes or a string like `"20mb"`) to trigger local compaction before opening the next run once the active transcript reaches that size. This is a size guard for local reopen cost, not raw archival - normal semantic compaction still runs, and it requires `truncateAfterCompaction` so the compacted summary becomes a new successor transcript.
- **Mid-turn precheck**: set `agents.defaults.compaction.midTurnPrecheck.enabled: true` (default `false`) to add a tool-loop guard. After a tool result is appended and before the next model call, OpenClaw estimates prompt pressure using the same preflight budget logic used at turn start. If context no longer fits, the guard does not compact inline - it raises a structured mid-turn precheck signal, stops the current prompt submission, and lets the outer run loop use the existing recovery path (truncate oversized tool results when that is enough, or trigger the configured compaction mode and retry). Works with both `default` and `safeguard` compaction modes, including provider-backed safeguard compaction. Independent of `maxActiveTranscriptBytes`: the byte-size guard runs before a turn opens, mid-turn precheck runs later, after new tool results are appended.
## Compaction settings
@@ -202,7 +245,7 @@ OpenClaw also enforces a safety floor for embedded runs: if `compaction.reserveT
Manual `/compact` honors an explicit `agents.defaults.compaction.keepRecentTokens` and keeps the runtime's recent-tail cut point. Without an explicit keep budget, manual compaction is a hard checkpoint and rebuilt context starts from the new summary.
When `truncateAfterCompaction` is enabled, OpenClaw rotates the active transcript to a compacted successor JSONL after compaction. Branch/restore checkpoint actions use that compacted successor; legacy pre-compaction checkpoint files remain readable while referenced.
When `truncateAfterCompaction` is enabled, OpenClaw rotates the active transcript to a compacted successor after compaction. Branch/restore checkpoint actions use that compacted successor; legacy pre-compaction checkpoint files remain readable while referenced.
## Pluggable compaction providers
@@ -252,7 +295,7 @@ Notes:
- The default prompt/system prompt include a `NO_REPLY` hint to suppress delivery.
- When `model` is set, the flush turn uses that model without inheriting the active session's fallback chain, so local-only housekeeping does not silently fall back to a paid conversation model on failure.
- The flush runs once per compaction cycle (tracked in `sessions.json`).
- The flush runs once per compaction cycle (tracked in the session row).
- The flush runs only for embedded OpenClaw sessions; CLI backends and heartbeat turns skip it.
- The flush is skipped when the session workspace is read-only (`workspaceAccess: "ro"` or `"none"`).
- See [Memory](/concepts/memory) for the workspace file layout and write patterns.
docs/reference/wizard.md
+4 -1
View File
@@ -254,7 +254,10 @@ Typical fields in `~/.openclaw/openclaw.json`:
`openclaw agents add` writes `agents.list[]` and optional `bindings`.
WhatsApp credentials go under `~/.openclaw/credentials/whatsapp/<accountId>/`.
Sessions are stored under `~/.openclaw/agents/<agentId>/sessions/`.
Active sessions and transcripts are stored in
`~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`. The
`~/.openclaw/agents/<agentId>/sessions/` directory is used for legacy migration
inputs and archive/support artifacts.
Some channels are delivered as plugins. When you pick one during setup, onboarding
will prompt to install it (npm or a local path) before it can be configured.
docs/start/openclaw.md
+3 -2
View File
@@ -159,8 +159,9 @@ Example:
## Sessions and memory
- Session files: `~/.openclaw/agents/<agentId>/sessions/{{SessionId}}.jsonl`
- Session metadata (token usage, last route, etc): `~/.openclaw/agents/<agentId>/sessions/sessions.json`
- Session rows, transcript rows, and metadata (token usage, last route, etc): `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
- Legacy/archive transcript artifacts: `~/.openclaw/agents/<agentId>/sessions/`
- Legacy row migration source: `~/.openclaw/agents/<agentId>/sessions/sessions.json`
- `/new` or `/reset` starts a fresh session for that chat (configurable via `session.resetTriggers`). If sent alone, OpenClaw acknowledges the reset without invoking the model.
- `/compact [instructions]` compacts the session context and reports the remaining context budget.
docs/start/setup.md
+2 -1
View File
@@ -132,7 +132,8 @@ openclaw health
- **Where state lives:**
- Channel/provider state: `~/.openclaw/credentials/`
- Model auth profiles: `~/.openclaw/agents/<agentId>/agent/auth-profiles.json`
- Sessions: `~/.openclaw/agents/<agentId>/sessions/`
- Sessions and transcripts: `~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`
- Legacy/archive session artifacts: `~/.openclaw/agents/<agentId>/sessions/`
- Logs: `/tmp/openclaw/`
## Credential storage map
docs/start/wizard-cli-reference.md
+4 -1
View File
@@ -341,7 +341,10 @@ Typical fields in `~/.openclaw/openclaw.json`:
`openclaw agents add` writes `agents.list[]` and optional `bindings`.
WhatsApp credentials go under `~/.openclaw/credentials/whatsapp/<accountId>/`.
Sessions are stored under `~/.openclaw/agents/<agentId>/sessions/`.
Active sessions and transcripts are stored in
`~/.openclaw/agents/<agentId>/agent/openclaw-agent.sqlite`. The
`~/.openclaw/agents/<agentId>/sessions/` directory is used for legacy migration
inputs and archive/support artifacts.
<Note>
Some channels are delivered as plugins. When selected during setup, the wizard
docs/tools/trajectory.md
+11 -28
View File
@@ -4,7 +4,7 @@ read_when:
- Debugging why an agent answered, failed, or called tools a certain way
- Exporting a support bundle for an OpenClaw session
- Investigating prompt context, tool calls, runtime errors, or usage metadata
- Disabling or relocating trajectory capture
- Disabling trajectory capture
title: "Trajectory bundles"
---
@@ -114,31 +114,15 @@ Events are written as JSON Lines with this schema marker:
`manifest.json` lists the files present in a given bundle; some files are
omitted when the session did not capture the corresponding runtime data.
## Capture location
## Capture storage
By default, runtime trajectory events are written beside the session file:
Runtime trajectory events are stored with the session in the per-agent SQLite
database. Exporting a trajectory materializes a redacted JSONL support bundle;
the live runtime capture is not a session-adjacent JSONL sidecar.
```text
<session>.trajectory.jsonl
```
OpenClaw also writes a best-effort pointer file beside the session:
```text
<session>.trajectory-path.json
```
Set `OPENCLAW_TRAJECTORY_DIR` to store runtime trajectory sidecars in a
dedicated directory instead, one JSONL file per session id:
```bash
export OPENCLAW_TRAJECTORY_DIR=/var/lib/openclaw/trajectories
```
Session maintenance removes trajectory sidecars when their owning session
entry is pruned, capped, or evicted by the sessions disk budget. Runtime files
outside the sessions directory are removed only when the pointer target still
proves it belongs to that session.
Legacy `.trajectory.jsonl` and `.trajectory-path.json` files may still appear
from older releases or explicit legacy-file exports. Session maintenance treats
those files as cleanup targets; active capture writes database rows.
## Disable capture
@@ -148,12 +132,12 @@ export OPENCLAW_TRAJECTORY=0
This disables runtime trajectory capture before starting OpenClaw.
`/export-trajectory` can still export the transcript branch, but runtime-only
files such as compiled context, provider artifacts, and prompt metadata may be
data such as compiled context, provider artifacts, and prompt metadata may be
missing.
## Tune flush timeout
OpenClaw flushes runtime trajectory sidecars during agent cleanup. The default
OpenClaw flushes runtime trajectory rows during agent cleanup. The default
cleanup timeout is 10,000 ms. On slow disks or large stores, set
`OPENCLAW_TRAJECTORY_FLUSH_TIMEOUT_MS` before starting OpenClaw:
@@ -179,7 +163,7 @@ redacts sensitive values before writing export files:
The exporter also bounds input size:
- runtime sidecar files: the live capture file is a rolling window capped at 10 MiB, dropping the oldest events to make room for new ones; export accepts existing runtime sidecar files up to 50 MiB
- runtime capture: the live capture is a rolling window capped at 10 MiB, dropping the oldest events to make room for new ones; export accepts existing legacy runtime sidecar files up to 50 MiB
- session files: 50 MiB
- runtime events per export: 200,000
- total exported events: 250,000
@@ -193,7 +177,6 @@ and cannot know every application-specific secret.
If the export has no runtime events:
- confirm OpenClaw was started without `OPENCLAW_TRAJECTORY=0`
- check whether `OPENCLAW_TRAJECTORY_DIR` points to a writable directory
- run another message in the session, then export again
- inspect `manifest.json` for `runtimeEventCount`
docs/web/webchat.md
+2 -2
View File
@@ -39,11 +39,11 @@ Status: the macOS/iOS SwiftUI chat UI talks directly to the Gateway WebSocket. N
WebChat has two separate data paths:
- The session JSONL file is the durable model/runtime transcript. For normal agent runs, the embedded OpenClaw runtime persists model-visible `user`, `assistant`, and `toolResult` messages through its session manager. WebChat does not write arbitrary delivery, status, or helper text into that transcript.
- The SQLite transcript rows are the durable model/runtime transcript. For normal agent runs, the embedded OpenClaw runtime persists model-visible `user`, `assistant`, and `toolResult` messages through the session accessor. WebChat does not write arbitrary delivery, status, or helper text into that transcript.
- Gateway `ReplyPayload` events are the live delivery projection: normalized for WebChat/channel display, block streaming, directive tags, media embedding, TTS/audio flags, and UI fallback behavior. They are not themselves the canonical session log.
- Harnesses that require visible replies through `tools.message` still use WebChat as a current-run internal source reply sink. A targetless `message.send` from that active WebChat run is projected into the same chat and mirrored to the session transcript; WebChat does not become a reusable outbound channel and never inherits `lastChannel`.
- WebChat injects assistant transcript entries only when the Gateway owns a displayed message outside a normal embedded agent turn: `chat.inject`, non-agent command replies, aborted partial output, and WebChat-managed media transcript supplements.
- If live assistant text appears during a run but disappears after history reload, check in order: whether the raw JSONL contains the assistant text, whether `chat.history` display projection stripped it, then whether the Control UI optimistic-tail merge replaced local delivery state with the persisted snapshot.
- If live assistant text appears during a run but disappears after history reload, check in order: whether the SQLite transcript contains the assistant text, whether `chat.history` display projection stripped it, then whether the Control UI optimistic-tail merge replaced local delivery state with the persisted snapshot.
Normal agent-run final answers should be durable because the embedded runtime writes the assistant `message_end`. Any fallback that mirrors a delivered final payload into the transcript must first avoid duplicating an assistant turn that the embedded runtime already wrote.
extensions/acpx/index.test.ts
+94 -2
View File
@@ -66,7 +66,27 @@ describe("acpx plugin", () => {
params.openKeyedStore({ namespace: "test", maxEntries: 1 });
expect(openKeyedStore).toHaveBeenCalledWith({ namespace: "test", maxEntries: 1 });
expect(api.registerService).toHaveBeenCalledWith(service);
expect(api.on).toHaveBeenCalledWith("reply_dispatch", tryDispatchAcpReplyHookMock);
expect(api.on).toHaveBeenCalledWith("reply_dispatch", expect.any(Function), {
timeoutMs: 120_000,
});
});
it("uses configured ACPX timeout for reply_dispatch hook registration", () => {
const service = { id: "acpx-service", start: vi.fn() };
createAcpxRuntimeServiceMock.mockReturnValue(service);
const api = {
pluginConfig: { timeoutSeconds: 180 },
runtime: { state: { openKeyedStore: vi.fn() } },
registerService: vi.fn(),
on: vi.fn(),
};
plugin.register(api as never);
expect(api.on).toHaveBeenCalledWith("reply_dispatch", expect.any(Function), {
timeoutMs: 180_000,
});
});
it("does not touch runtime state while registering metadata-only plugin APIs", () => {
@@ -130,7 +150,79 @@ describe("acpx plugin", () => {
queuedFinal: true,
counts: { tool: 1, block: 0, final: 1 },
});
expect(tryDispatchAcpReplyHookMock).toHaveBeenCalledWith(event, ctx);
expect(tryDispatchAcpReplyHookMock).toHaveBeenCalledWith(event, {
...ctx,
abortSignal: expect.any(AbortSignal),
});
});
it("aborts the ACP reply_dispatch runtime path at the configured timeout", async () => {
vi.useFakeTimers();
try {
const service = { id: "acpx-service", start: vi.fn() };
createAcpxRuntimeServiceMock.mockReturnValue(service);
const observedAbortStates: boolean[] = [];
tryDispatchAcpReplyHookMock.mockImplementation(async (_event, hookCtx) => {
const abortSignal = (hookCtx as { abortSignal?: AbortSignal }).abortSignal;
if (!abortSignal) {
throw new Error("expected ACPX hook abort signal");
}
observedAbortStates.push(abortSignal.aborted);
await new Promise<void>((resolve) => {
abortSignal.addEventListener("abort", () => resolve(), { once: true });
});
observedAbortStates.push(abortSignal.aborted);
return {
handled: true,
queuedFinal: true,
counts: { tool: 0, block: 0, final: 1 },
};
});
const on = vi.fn();
const api = createTestPluginApi({
pluginConfig: { timeoutSeconds: 0.001 },
runtime: { state: { openKeyedStore: vi.fn() } } as never,
registerService: vi.fn(),
on,
});
plugin.register(api);
const hook = on.mock.calls.find(([hookName]) => hookName === "reply_dispatch")?.[1];
if (!hook) {
throw new Error("expected reply_dispatch hook to be registered");
}
const run = hook(
{
ctx: { raw: "reply ctx" },
runId: "run-1",
sessionKey: "agent:test:session",
inboundAudio: false,
shouldRouteToOriginating: false,
shouldSendToolSummaries: true,
sendPolicy: "allow",
},
{
cfg: {},
dispatcher: { dispatch: vi.fn(), getQueuedCounts: vi.fn(), getFailedCounts: vi.fn() },
recordProcessed: vi.fn(),
markIdle: vi.fn(),
},
);
await vi.advanceTimersByTimeAsync(1);
await expect(run).resolves.toEqual({
handled: true,
queuedFinal: true,
counts: { tool: 0, block: 0, final: 1 },
});
expect(observedAbortStates).toEqual([false, true]);
} finally {
vi.useRealTimers();
}
});
it("declares setup auto-enable reasons for ACPX-owned ACP config", () => {
extensions/acpx/index.ts
+44 -2
View File
@@ -3,21 +3,63 @@
* wires reply-dispatch hooks into the plugin SDK runtime.
*/
import { tryDispatchAcpReplyHook } from "openclaw/plugin-sdk/acp-runtime-backend";
import { finiteSecondsToTimerSafeMilliseconds } from "openclaw/plugin-sdk/number-runtime";
import { createAcpxRuntimeService } from "./register.runtime.js";
import type { OpenClawPluginApi } from "./runtime-api.js";
import type {
OpenClawPluginApi,
PluginHookReplyDispatchContext,
PluginHookReplyDispatchEvent,
PluginHookReplyDispatchResult,
} from "./runtime-api.js";
import { DEFAULT_ACPX_TIMEOUT_SECONDS } from "./src/config-schema.js";
function resolveReplyDispatchTimeoutMs(pluginConfig?: Record<string, unknown>): number {
const timeoutSeconds = pluginConfig?.timeoutSeconds;
const resolvedSeconds =
typeof timeoutSeconds === "number" && Number.isFinite(timeoutSeconds) && timeoutSeconds > 0
? timeoutSeconds
: DEFAULT_ACPX_TIMEOUT_SECONDS;
return finiteSecondsToTimerSafeMilliseconds(resolvedSeconds) ?? 1;
}
async function tryDispatchAcpReplyHookWithTimeout(
event: PluginHookReplyDispatchEvent,
ctx: PluginHookReplyDispatchContext,
timeoutMs: number,
): Promise<PluginHookReplyDispatchResult | void> {
const timeoutController = new AbortController();
const timeout = setTimeout(() => timeoutController.abort(), timeoutMs);
timeout.unref?.();
const abortSignal = ctx.abortSignal
? AbortSignal.any([ctx.abortSignal, timeoutController.signal])
: timeoutController.signal;
try {
return await tryDispatchAcpReplyHook(event, {
...ctx,
abortSignal,
});
} finally {
clearTimeout(timeout);
}
}
const plugin = {
id: "acpx",
name: "ACPX Runtime",
description: "Embedded ACP runtime backend with plugin-owned session and transport management.",
register(api: OpenClawPluginApi) {
const replyDispatchTimeoutMs = resolveReplyDispatchTimeoutMs(api.pluginConfig);
api.registerService(
createAcpxRuntimeService({
pluginConfig: api.pluginConfig,
openKeyedStore: (options) => api.runtime.state.openKeyedStore(options),
}),
);
api.on("reply_dispatch", tryDispatchAcpReplyHook);
api.on(
"reply_dispatch",
(event, ctx) => tryDispatchAcpReplyHookWithTimeout(event, ctx, replyDispatchTimeoutMs),
{ timeoutMs: replyDispatchTimeoutMs },
);
},
};
extensions/active-memory/index.test.ts
+134 -2
View File
@@ -8,6 +8,7 @@ import {
createPluginStateKeyedStoreForTests,
resetPluginStateStoreForTests,
} from "openclaw/plugin-sdk/plugin-state-test-runtime";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import plugin, { testing } from "./index.js";
@@ -133,7 +134,7 @@ describe("active-memory plugin", () => {
agent: {
runEmbeddedAgent,
session: {
resolveStorePath: vi.fn(() => "/tmp/openclaw-session-store.json"),
resolveStorePath: vi.fn(() => path.join(stateDir, "sessions.json")),
loadSessionStore: vi.fn(() => hoisted.sessionStore),
saveSessionStore: vi.fn(async () => {}),
getSessionEntry: vi.fn(
@@ -153,7 +154,7 @@ describe("active-memory plugin", () => {
}) => {
let result: Record<string, unknown> | null = null;
await hoisted.updateSessionStore(
"/tmp/openclaw-session-store.json",
path.join(stateDir, "sessions.json"),
(store: Record<string, Record<string, unknown>>) => {
const existing = store[params.sessionKey] ?? params.fallbackEntry;
if (!existing) {
@@ -3018,6 +3019,72 @@ describe("active-memory plugin", () => {
);
});
it("returns partial transcript text on timeout from SQLite runtime transcript rows", async () => {
testing.setMinimumTimeoutMsForTests(1);
testing.setSetupGraceTimeoutMsForTests(0);
testing.setTimeoutPartialDataGraceMsForTests(100);
api.pluginConfig = {
agents: ["main"],
timeoutMs: 250,
maxSummaryChars: 80,
logging: true,
};
plugin.register(api as unknown as OpenClawPluginApi);
const sessionKey = "agent:main:timeout-partial-sqlite-transcript";
hoisted.sessionStore[sessionKey] = {
sessionId: "s-timeout-partial-sqlite-transcript",
updatedAt: 0,
};
let artifactSessionFile = "";
runEmbeddedAgent.mockImplementationOnce(
async (params: {
abortSignal?: AbortSignal;
sessionFile?: string;
sessionTarget?: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath?: string;
};
}) => {
artifactSessionFile = params.sessionFile ?? "";
const target = params.sessionTarget;
if (!target) {
throw new Error("expected active-memory runtime session target");
}
await appendSessionTranscriptMessageByIdentity({
...target,
message: {
role: "assistant",
content: "sqlite partial recall summary",
},
});
await waitForAbort(params.abortSignal);
},
);
const result = await hooks.before_prompt_build(
{ prompt: "what wings should i order? timeout partial sqlite", messages: [] },
{ agentId: "main", trigger: "user", sessionKey, messageProvider: "webchat" },
);
expectPrependContextContains(result, "sqlite partial recall summary");
if (artifactSessionFile) {
await expectPathMissing(artifactSessionFile);
}
const runParams = lastEmbeddedRunParams();
expect(runParams.sessionTarget).toMatchObject({
agentId: "main",
sessionKey: expect.stringMatching(/^agent:main:timeout-partial-sqlite-transcript:/),
});
const lines = getActiveMemoryLines(sessionKey);
expectLinesToContain(lines, "🧩 Active Memory: status=timeout_partial");
expectLinesToContain(
lines,
"🔎 Active Memory Debug: timeout_partial: 29 chars recovered (not persisted)",
);
});
it("keeps timeout status when the timeout transcript is empty", async () => {
testing.setMinimumTimeoutMsForTests(1);
testing.setSetupGraceTimeoutMsForTests(0);
@@ -4705,6 +4772,71 @@ describe("active-memory plugin", () => {
expectLinesToContain(getActiveMemoryLines(sessionKey), "status=unavailable");
});
it("rejects completed output when a rotated SQLite transcript reports unavailable memory", async () => {
testing.setMinimumTimeoutMsForTests(1);
testing.setSetupGraceTimeoutMsForTests(0);
api.pluginConfig = {
agents: ["main"],
timeoutMs: 1_000,
logging: true,
};
plugin.register(api as unknown as OpenClawPluginApi);
const sessionKey = "agent:main:rotated-sqlite-memory-unavailable";
hoisted.sessionStore[sessionKey] = {
sessionId: "s-rotated-sqlite-memory-unavailable",
updatedAt: 0,
};
runEmbeddedAgent.mockImplementationOnce(
async (params: {
sessionTarget?: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath?: string;
};
}) => {
const target = params.sessionTarget;
if (!target?.storePath) {
throw new Error("expected active-memory SQLite runtime target");
}
const rotatedTarget = {
...target,
sessionId: "s-rotated-sqlite-memory-unavailable-next",
};
await appendSessionTranscriptMessageByIdentity({
...rotatedTarget,
message: {
role: "toolResult",
toolCallId: "memory-search-1",
toolName: "memory_search",
isError: true,
content: [],
details: {
disabled: true,
warning: "Memory search is disabled for this session.",
},
},
});
return {
payloads: [{ text: "This arbitrary output must not become recalled context." }],
meta: {
agentMeta: {
sessionFile: `sqlite:${rotatedTarget.agentId}:${rotatedTarget.sessionId}:${rotatedTarget.storePath}`,
},
},
};
},
);
const result = await hooks.before_prompt_build(
{ prompt: "what food do i usually order? rotated sqlite unavailable", messages: [] },
{ agentId: "main", trigger: "user", sessionKey, messageProvider: "webchat" },
);
expect(result).toBeUndefined();
expectLinesToContain(getActiveMemoryLines(sessionKey), "status=unavailable");
});
it("fast-fails configured-provider-missing memory_search results without injecting provider errors", async () => {
const CONFIGURED_TIMEOUT_MS = 1_000;
testing.setMinimumTimeoutMsForTests(1);
extensions/active-memory/index.ts
+274 -42
View File
@@ -29,6 +29,11 @@ import {
import { definePluginEntry, type OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry";
import { parseAgentSessionKey, parseThreadSessionSuffix } from "openclaw/plugin-sdk/routing";
import { isPathInside } from "openclaw/plugin-sdk/security-runtime";
import { parseSqliteSessionFileMarker } from "openclaw/plugin-sdk/session-store-runtime";
import {
readSessionTranscriptEvents,
type SessionTranscriptTargetParams,
} from "openclaw/plugin-sdk/session-transcript-runtime";
import {
asOptionalRecord as asRecord,
normalizeLowercaseStringOrEmpty,
@@ -295,6 +300,16 @@ type TranscriptReadLimits = {
maxBytes?: number;
};
type ActiveMemoryTranscriptSource =
| {
kind: "runtime";
target: SessionTranscriptTargetParams;
}
| {
kind: "file";
sessionFile: string;
};
type RecallSubagentResult = {
rawReply: string;
resultStatus?: "failed" | "unavailable";
@@ -1749,6 +1764,90 @@ async function streamBoundedTranscriptJsonl(params: {
}
}
function fileTranscriptSource(sessionFile: string): ActiveMemoryTranscriptSource {
return { kind: "file", sessionFile };
}
function transcriptSourceFromReturnedSessionFile(params: {
sessionFile: string;
sessionKey: string;
}): ActiveMemoryTranscriptSource {
const marker = parseSqliteSessionFileMarker(normalizeOptionalString(params.sessionFile));
if (!marker) {
return fileTranscriptSource(params.sessionFile);
}
return {
kind: "runtime",
target: {
agentId: marker.agentId,
sessionId: marker.sessionId,
sessionKey: params.sessionKey,
storePath: marker.storePath,
},
};
}
function estimateTranscriptEventsBytes(events: readonly unknown[]): number {
let total = 0;
for (const event of events) {
try {
total += Buffer.byteLength(`${JSON.stringify(event)}\n`, "utf8");
} catch {
total += 1;
}
}
return total;
}
async function streamRuntimeTranscriptEvents(params: {
target: SessionTranscriptTargetParams;
limits?: TranscriptReadLimits;
onRecord: (record: unknown) => boolean | void;
}): Promise<void> {
const limits = resolveTranscriptReadLimits(params.limits);
let events: readonly unknown[];
try {
events = await readSessionTranscriptEvents(params.target);
} catch {
return;
}
if (estimateTranscriptEventsBytes(events) > limits.maxBytes) {
return;
}
let seenLines = 0;
for (const event of events) {
seenLines += 1;
if (seenLines > limits.maxLines) {
break;
}
try {
if (params.onRecord(event)) {
break;
}
} catch {}
}
}
async function streamActiveMemoryTranscriptRecords(params: {
source: ActiveMemoryTranscriptSource;
limits?: TranscriptReadLimits;
onRecord: (record: unknown) => boolean | void;
}): Promise<void> {
if (params.source.kind === "runtime") {
await streamRuntimeTranscriptEvents({
target: params.source.target,
limits: params.limits,
onRecord: params.onRecord,
});
return;
}
await streamBoundedTranscriptJsonl({
sessionFile: params.source.sessionFile,
limits: params.limits,
onRecord: params.onRecord,
});
}
function extractActiveMemorySearchDebugFromSessionRecord(
value: unknown,
): ActiveMemorySearchDebug | undefined {
@@ -1986,7 +2085,7 @@ function hasUsableMemoryResultInSessionRecord(
}
async function readActiveMemoryTranscriptState(
sessionFile: string,
source: ActiveMemoryTranscriptSource | string,
limits?: TranscriptReadLimits,
toolsAllow?: readonly string[],
): Promise<{
@@ -1997,8 +2096,8 @@ async function readActiveMemoryTranscriptState(
let searchDebug: ActiveMemorySearchDebug | undefined;
let hasUsableMemoryResult = false;
let hasUnavailableMemorySearchResult = false;
await streamBoundedTranscriptJsonl({
sessionFile,
await streamActiveMemoryTranscriptRecords({
source: typeof source === "string" ? fileTranscriptSource(source) : source,
limits,
onRecord: (record) => {
const debug = extractActiveMemorySearchDebugFromSessionRecord(record);
@@ -2016,14 +2115,14 @@ async function readActiveMemoryTranscriptState(
}
async function readActiveMemorySearchDebug(
sessionFile: string,
source: ActiveMemoryTranscriptSource | string,
limits?: TranscriptReadLimits,
): Promise<ActiveMemorySearchDebug | undefined> {
return (await readActiveMemoryTranscriptState(sessionFile, limits)).searchDebug;
return (await readActiveMemoryTranscriptState(source, limits)).searchDebug;
}
async function readMergedActiveMemoryTranscriptState(params: {
sessionFiles: readonly string[];
sources: readonly ActiveMemoryTranscriptSource[];
toolsAllow: readonly string[];
}): Promise<{
searchDebug?: ActiveMemorySearchDebug;
@@ -2033,8 +2132,17 @@ async function readMergedActiveMemoryTranscriptState(params: {
let searchDebug: ActiveMemorySearchDebug | undefined;
let hasUsableMemoryResult = false;
let hasUnavailableMemorySearchResult = false;
for (const sessionFile of new Set(params.sessionFiles)) {
const state = await readActiveMemoryTranscriptState(sessionFile, undefined, params.toolsAllow);
const seen = new Set<string>();
for (const source of params.sources) {
const key =
source.kind === "runtime"
? `runtime:${source.target.agentId ?? ""}:${source.target.sessionId}:${source.target.sessionKey}:${source.target.storePath ?? ""}:${source.target.threadId ?? ""}`
: `file:${source.sessionFile}`;
if (seen.has(key)) {
continue;
}
seen.add(key);
const state = await readActiveMemoryTranscriptState(source, undefined, params.toolsAllow);
searchDebug = state.searchDebug ?? searchDebug;
hasUsableMemoryResult ||= state.hasUsableMemoryResult;
hasUnavailableMemorySearchResult ||= state.hasUnavailableMemorySearchResult;
@@ -2043,7 +2151,7 @@ async function readMergedActiveMemoryTranscriptState(params: {
}
async function readTerminalMemorySearchResult(
sessionFile: string,
source: ActiveMemoryTranscriptSource,
limits?: TranscriptReadLimits,
toolsAllow?: readonly string[],
): Promise<TerminalMemorySearchResult | undefined> {
@@ -2060,8 +2168,8 @@ async function readTerminalMemorySearchResult(
const unavailablePathNames = new Set<string>();
let hasUsableMemoryResult = false;
let searchDebug: ActiveMemorySearchDebug | undefined;
await streamBoundedTranscriptJsonl({
sessionFile,
await streamActiveMemoryTranscriptRecords({
source,
limits,
onRecord: (record) => {
hasUsableMemoryResult ||= hasUsableMemoryResultInSessionRecord(record, toolsAllow);
@@ -2088,8 +2196,22 @@ async function readTerminalMemorySearchResult(
};
}
async function readTerminalMemorySearchResultFromSources(
sources: readonly ActiveMemoryTranscriptSource[],
limits: TranscriptReadLimits | undefined,
toolsAllow: readonly string[],
): Promise<TerminalMemorySearchResult | undefined> {
for (const source of sources) {
const result = await readTerminalMemorySearchResult(source, limits, toolsAllow);
if (result) {
return result;
}
}
return undefined;
}
function watchTerminalMemorySearchResult(params: {
getSessionFile: () => string | undefined;
getTranscriptSources: () => readonly ActiveMemoryTranscriptSource[];
abortSignal: AbortSignal;
toolsAllow: readonly string[];
}): TerminalMemorySearchWatch {
@@ -2131,10 +2253,11 @@ function watchTerminalMemorySearchResult(params: {
}
inFlight = true;
try {
const sessionFile = params.getSessionFile();
const result = sessionFile
? await readTerminalMemorySearchResult(sessionFile, undefined, params.toolsAllow)
: undefined;
const result = await readTerminalMemorySearchResultFromSources(
params.getTranscriptSources(),
undefined,
params.toolsAllow,
);
if (result) {
finish(result);
return;
@@ -2265,17 +2388,17 @@ function extractAssistantTextFromSessionRecord(value: unknown): string {
}
async function readPartialAssistantText(
sessionFile: string | undefined,
source: ActiveMemoryTranscriptSource | string | undefined,
limits?: TranscriptReadLimits,
): Promise<string | null> {
if (!sessionFile) {
if (!source) {
return null;
}
const texts: string[] = [];
const resolvedLimits = resolveTranscriptReadLimits(limits);
let collectedChars = 0;
await streamBoundedTranscriptJsonl({
sessionFile,
await streamActiveMemoryTranscriptRecords({
source: typeof source === "string" ? fileTranscriptSource(source) : source,
limits: resolvedLimits,
onRecord: (record) => {
const text = extractAssistantTextFromSessionRecord(record);
@@ -2302,6 +2425,19 @@ async function readPartialAssistantText(
return joined || null;
}
async function readPartialAssistantTextFromSources(
sources: readonly ActiveMemoryTranscriptSource[],
limits?: TranscriptReadLimits,
): Promise<string | null> {
for (const source of sources) {
const text = await readPartialAssistantText(source, limits);
if (text) {
return text;
}
}
return null;
}
function attachPartialTimeoutData(
error: unknown,
partialReply: string | null,
@@ -2373,7 +2509,7 @@ async function waitForSubagentPartialTimeoutData(
async function buildTimeoutRecallResult(params: {
elapsedMs: number;
maxSummaryChars: number;
sessionFile?: string;
transcriptSources: readonly ActiveMemoryTranscriptSource[];
rawReply?: string;
searchDebug?: ActiveMemorySearchDebug;
hasUnavailableMemorySearchResult?: boolean;
@@ -2386,13 +2522,17 @@ async function buildTimeoutRecallResult(params: {
const rawReply =
params.rawReply ??
subagentPartialData.rawReply ??
(await readPartialAssistantText(params.sessionFile));
(await readPartialAssistantTextFromSources(params.transcriptSources));
const summary = truncateSummary(
normalizeActiveSummary(rawReply ?? "") ?? "",
params.maxSummaryChars,
);
const transcriptState = params.sessionFile
? await readActiveMemoryTranscriptState(params.sessionFile, undefined, params.toolsAllow)
const transcriptState =
params.transcriptSources.length > 0
? await readMergedActiveMemoryTranscriptState({
sources: params.transcriptSources,
toolsAllow: params.toolsAllow,
})
: undefined;
const searchDebug =
params.searchDebug ?? subagentPartialData.searchDebug ?? transcriptState?.searchDebug;
@@ -2945,6 +3085,64 @@ function getModelRef(
return undefined;
}
function collectActiveMemoryTranscriptSources(params: {
artifactSessionFile: string;
runtimeSource: ActiveMemoryTranscriptSource;
activeSessionFile?: string;
activeSessionKey: string;
}): ActiveMemoryTranscriptSource[] {
const sources: ActiveMemoryTranscriptSource[] = [params.runtimeSource];
sources.push(fileTranscriptSource(params.artifactSessionFile));
if (params.activeSessionFile && params.activeSessionFile !== params.artifactSessionFile) {
sources.push(
transcriptSourceFromReturnedSessionFile({
sessionFile: params.activeSessionFile,
sessionKey: params.activeSessionKey,
}),
);
}
return sources;
}
async function persistActiveMemoryTranscriptArtifact(params: {
sources: readonly ActiveMemoryTranscriptSource[];
sessionFile: string;
}): Promise<void> {
const events: unknown[] = [];
const seen = new Set<string>();
for (const source of params.sources) {
if (source.kind !== "runtime") {
continue;
}
let sourceEvents: readonly unknown[];
try {
sourceEvents = await readSessionTranscriptEvents(source.target);
} catch {
continue;
}
for (const event of sourceEvents) {
const serialized = JSON.stringify(event);
if (seen.has(serialized)) {
continue;
}
seen.add(serialized);
events.push(event);
}
}
if (events.length === 0) {
return;
}
await fs.mkdir(path.dirname(params.sessionFile), { recursive: true, mode: 0o700 });
await fs.writeFile(
params.sessionFile,
`${events.map((event) => JSON.stringify(event)).join("\n")}\n`,
{
encoding: "utf8",
mode: 0o600,
},
);
}
async function runRecallSubagent(params: {
api: OpenClawPluginApi;
config: ResolvedActiveRecallPluginConfig;
@@ -2959,7 +3157,7 @@ async function runRecallSubagent(params: {
currentModelId?: string;
modelRef?: { provider: string; model: string };
abortSignal?: AbortSignal;
onSessionFile?: (sessionFile: string) => void;
onTranscriptSources?: (sources: readonly ActiveMemoryTranscriptSource[]) => void;
}): Promise<RecallSubagentResult> {
const workspaceDir = resolveAgentWorkspaceDir(params.api.config, params.agentId);
const agentDir = resolveAgentDir(params.api.config, params.agentId);
@@ -3006,7 +3204,27 @@ async function runRecallSubagent(params: {
persistedDir !== undefined
? path.join(persistedDir, `${subagentSessionId}.jsonl`)
: path.join(requireTransientWorkspaceDir(tempDir), "session.jsonl");
params.onSessionFile?.(sessionFile);
const storePath = params.api.runtime.agent.session.resolveStorePath(
params.api.config.session?.store,
{
agentId: params.agentId,
},
);
const runtimeSource: ActiveMemoryTranscriptSource = {
kind: "runtime",
target: {
agentId: params.agentId,
sessionId: subagentSessionId,
sessionKey: subagentSessionKey,
storePath,
},
};
let transcriptSources = collectActiveMemoryTranscriptSources({
artifactSessionFile: sessionFile,
runtimeSource,
activeSessionKey: subagentSessionKey,
});
params.onTranscriptSources?.(transcriptSources);
if (persistedDir) {
await fs.mkdir(persistedDir, { recursive: true, mode: 0o700 });
await fs.chmod(persistedDir, 0o700).catch(() => undefined);
@@ -3025,7 +3243,6 @@ async function runRecallSubagent(params: {
channelId: params.channelId,
});
let activeSessionFile = sessionFile;
let harnessHasUsableMemoryResult = false;
let harnessHasUnavailableMemorySearchResult = false;
try {
@@ -3035,6 +3252,12 @@ async function runRecallSubagent(params: {
sessionId: subagentSessionId,
sessionKey: subagentSessionKey,
agentId: params.agentId,
sessionTarget: {
agentId: params.agentId,
sessionId: subagentSessionId,
sessionKey: subagentSessionKey,
storePath,
},
messageChannel,
messageProvider,
sessionFile,
@@ -3068,7 +3291,14 @@ async function runRecallSubagent(params: {
harnessHasUnavailableMemorySearchResult ||= evidence.hasUnavailableMemorySearchResult;
},
});
activeSessionFile = readActiveMemorySessionFileFromRunResult(result) ?? sessionFile;
const activeSessionFile = readActiveMemorySessionFileFromRunResult(result) ?? sessionFile;
transcriptSources = collectActiveMemoryTranscriptSources({
artifactSessionFile: sessionFile,
runtimeSource,
activeSessionFile,
activeSessionKey: subagentSessionKey,
});
params.onTranscriptSources?.(transcriptSources);
if (params.abortSignal?.aborted) {
const reason = params.abortSignal.reason;
if (reason instanceof Error) {
@@ -3086,15 +3316,18 @@ async function runRecallSubagent(params: {
.filter(Boolean)
.join("\n")
.trim();
if (params.config.persistTranscripts) {
await persistActiveMemoryTranscriptArtifact({ sources: transcriptSources, sessionFile });
}
const transcriptState = await readMergedActiveMemoryTranscriptState({
sessionFiles: [sessionFile, activeSessionFile],
sources: transcriptSources,
toolsAllow: params.config.toolsAllow,
});
const searchDebug =
transcriptState.searchDebug ?? readActiveMemorySearchDebugFromRunResult(result);
return {
rawReply: rawReply || "NONE",
transcriptPath: params.config.persistTranscripts ? activeSessionFile : undefined,
transcriptPath: params.config.persistTranscripts ? sessionFile : undefined,
searchDebug,
hasUsableMemoryResult: transcriptState.hasUsableMemoryResult || harnessHasUsableMemoryResult,
hasUnavailableMemorySearchResult:
@@ -3102,12 +3335,11 @@ async function runRecallSubagent(params: {
};
} catch (error) {
if (params.abortSignal?.aborted) {
const partialReply = await readPartialAssistantText(activeSessionFile);
const transcriptState = await readActiveMemoryTranscriptState(
activeSessionFile,
undefined,
params.config.toolsAllow,
);
const partialReply = await readPartialAssistantTextFromSources(transcriptSources);
const transcriptState = await readMergedActiveMemoryTranscriptState({
sources: transcriptSources,
toolsAllow: params.config.toolsAllow,
});
attachPartialTimeoutData(
error,
partialReply,
@@ -3258,7 +3490,7 @@ async function maybeResolveActiveRecall(params: {
abortFromParent();
}
const TIMEOUT_SENTINEL = Symbol("timeout");
let sessionFile: string | undefined;
let transcriptSources: readonly ActiveMemoryTranscriptSource[] = [];
let recallTimedOut = false;
const watchdogTimeoutMs = params.config.timeoutMs + params.config.setupGraceTimeoutMs;
const timeoutId = setTimeout(() => {
@@ -3288,12 +3520,12 @@ async function maybeResolveActiveRecall(params: {
...params,
modelRef: resolvedModelRef,
abortSignal: controller.signal,
onSessionFile: (value) => {
sessionFile = value;
onTranscriptSources: (sources) => {
transcriptSources = sources;
},
});
terminalMemorySearchWatch = watchTerminalMemorySearchResult({
getSessionFile: () => sessionFile,
getTranscriptSources: () => transcriptSources,
abortSignal: controller.signal,
toolsAllow: params.config.toolsAllow,
});
@@ -3341,7 +3573,7 @@ async function maybeResolveActiveRecall(params: {
: await buildTimeoutRecallResult({
elapsedMs,
maxSummaryChars: params.config.maxSummaryChars,
sessionFile,
transcriptSources,
subagentPromise,
toolsAllow: params.config.toolsAllow,
});
@@ -3440,7 +3672,7 @@ async function maybeResolveActiveRecall(params: {
const result = await buildTimeoutRecallResult({
elapsedMs: Date.now() - startedAt,
maxSummaryChars: params.config.maxSummaryChars,
sessionFile,
transcriptSources,
rawReply: partialTimeoutData.rawReply,
searchDebug: partialTimeoutData.searchDebug,
hasUnavailableMemorySearchResult: partialTimeoutData.hasUnavailableMemorySearchResult,
extensions/codex/src/app-server/attempt-context.ts
+2
View File
@@ -124,6 +124,7 @@ export function resolveContextEngineBootstrapProjectionDecision(params: {
expectedBinding: ReturnType<typeof buildContextEngineBinding>;
projection: CodexContextEngineThreadBootstrapProjection;
dynamicToolsFingerprint: string;
legacyDynamicToolsFingerprint?: string;
}): { project: boolean; reason: string } {
const bindingProjection = params.startupBinding?.contextEngine?.projection;
if (!params.startupBinding?.threadId || !bindingProjection) {
@@ -144,6 +145,7 @@ export function resolveContextEngineBootstrapProjectionDecision(params: {
!areCodexDynamicToolFingerprintsCompatible({
previous: params.startupBinding.dynamicToolsFingerprint,
next: params.dynamicToolsFingerprint,
nextLegacy: params.legacyDynamicToolsFingerprint,
})
) {
return { project: true, reason: "dynamic-tools-mismatch" };
extensions/codex/src/app-server/run-attempt.hooks.test.ts
+2 -6
View File
@@ -285,10 +285,7 @@ describe("runCodexAppServerAttempt hooks and model diagnostics", () => {
it("emits gated model-call content diagnostics for codex turns", async () => {
const diagnosticEvents: DiagnosticEventPayload[] = [];
const diagnosticContentByType = new Map<string, DiagnosticEventPrivateData>();
let diagnosticTypesAtLlmOutput: string[] = [];
const llmOutput = vi.fn(() => {
diagnosticTypesAtLlmOutput = diagnosticEvents.map((event) => event.type);
});
const llmOutput = vi.fn();
initializeGlobalHookRunner(
createMockPluginRegistry([{ hookName: "llm_output", handler: llmOutput }]),
);
@@ -378,8 +375,7 @@ describe("runCodexAppServerAttempt hooks and model diagnostics", () => {
).toContain("hello back");
expect(completedEvent?.requestPayloadBytes).toBeGreaterThan(0);
expect(llmOutput).toHaveBeenCalledTimes(1);
expect(diagnosticTypesAtLlmOutput).toContain("model.call.completed");
expect(diagnosticTypesAtLlmOutput).not.toContain("model.call.error");
expect(diagnosticEvents.map((event) => event.type)).not.toContain("model.call.error");
} finally {
stopDiagnostics();
}
extensions/codex/src/app-server/run-attempt.test.ts
+106 -46
View File
@@ -16,7 +16,8 @@ import { registerMemoryCapability } from "openclaw/plugin-sdk/memory-core-host-r
import { MESSAGE_TOOL_DELIVERY_HINTS } from "openclaw/plugin-sdk/message-tool-delivery-hints";
import { registerPluginCommand } from "openclaw/plugin-sdk/plugin-runtime";
import { createMockPluginRegistry } from "openclaw/plugin-sdk/plugin-test-runtime";
import { saveSessionStore } from "openclaw/plugin-sdk/session-store-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { readSessionTranscriptEvents } from "openclaw/plugin-sdk/session-transcript-runtime";
import { describe, expect, it, vi } from "vitest";
import WebSocket from "ws";
import { CODEX_GPT5_BEHAVIOR_CONTRACT } from "../../prompt-overlay.js";
@@ -181,6 +182,43 @@ async function writeExistingBinding(
});
}
function attachSqliteSessionTarget(
params: EmbeddedRunAttemptParams,
storePath: string,
sessionId: string,
): void {
params.sessionId = sessionId;
params.sessionKey = `agent:main:${sessionId}`;
params.sessionTarget = {
agentId: "main",
sessionId,
sessionKey: params.sessionKey,
storePath,
};
}
async function readTranscriptMessagesByIdentity(
params: EmbeddedRunAttemptParams,
): Promise<Array<Record<string, unknown>>> {
const target = params.sessionTarget;
if (!target?.storePath || !target.sessionKey) {
throw new Error("expected SQLite session target");
}
return (
await readSessionTranscriptEvents({
agentId: target.agentId,
sessionId: target.sessionId ?? params.sessionId,
sessionKey: target.sessionKey,
storePath: target.storePath,
})
)
.map((event) => (event as { message?: unknown }).message)
.filter(
(message): message is Record<string, unknown> =>
typeof message === "object" && message !== null,
);
}
function createThreadLifecycleAppServerOptions(): Parameters<
typeof startOrResumeThread
>[0]["appServer"] {
@@ -1260,11 +1298,23 @@ describe("runCodexAppServerAttempt", () => {
createMockPluginRegistry([{ hookName: "llm_input", handler: llmInput }]),
);
vi.stubEnv("OPENCLAW_TRAJECTORY", "1");
vi.stubEnv("OPENCLAW_TRAJECTORY_DIR", path.join(tempDir, "trajectory"));
const sessionFile = path.join(tempDir, "session.jsonl");
const workspaceDir = path.join(tempDir, "workspace");
const harness = createStartedThreadHarness();
const params = createParams(sessionFile, workspaceDir);
const trajectoryEvents: Array<{
data?: { prompt?: string; systemPrompt?: string };
type: string;
}> = [];
Object.assign(params, {
trajectorySessionFile: `sqlite:main:session-1:${path.join(tempDir, "openclaw-agent.sqlite")}`,
trajectoryRecorder: {
recordEvent: (type: string, data?: { prompt?: string; systemPrompt?: string }) => {
trajectoryEvents.push({ type, data });
},
flush: async () => undefined,
},
});
params.skillsSnapshot = {
prompt: "<available_skills><skill><name>demo</name></skill></available_skills>",
skills: [],
@@ -1301,15 +1351,6 @@ describe("runCodexAppServerAttempt", () => {
expect(inputText).toBe("hello");
const [llmInputPayload] = mockCall(llmInput, "llm_input") as [{ prompt?: string }, unknown];
expect(llmInputPayload.prompt).toBe(inputText);
const trajectoryEvents = (
await fs.readFile(path.join(tempDir, "trajectory", "session-1.jsonl"), "utf8")
)
.trim()
.split("\n")
.map(
(line) =>
JSON.parse(line) as { data?: { prompt?: string; systemPrompt?: string }; type?: string },
);
const compiledContext = trajectoryEvents.find((event) => event.type === "context.compiled");
expect(compiledContext?.data?.prompt).toBe(inputText);
expect(compiledContext?.data?.systemPrompt).toContain("## OpenClaw Skills");
@@ -1499,9 +1540,11 @@ describe("runCodexAppServerAttempt", () => {
it("mirrors the Codex prompt into the transcript when the turn starts", async () => {
const sessionFile = path.join(tempDir, "session-early-prompt.jsonl");
const storePath = path.join(tempDir, "sessions-early-prompt.json");
const workspaceDir = path.join(tempDir, "workspace-early-prompt");
const harness = createStartedThreadHarness();
const params = createParams(sessionFile, workspaceDir);
attachSqliteSessionTarget(params, storePath, "session-early-prompt");
params.prompt = "external channel prompt";
const onUserMessagePersisted = vi.fn();
params.onUserMessagePersisted = onUserMessagePersisted;
@@ -1509,10 +1552,13 @@ describe("runCodexAppServerAttempt", () => {
const run = runCodexAppServerAttempt(params);
await harness.waitForMethod("turn/start");
await vi.waitFor(async () => {
const raw = await fs.readFile(sessionFile, "utf8");
expect(raw).toContain('"role":"user"');
expect(raw).toContain('"content":"external channel prompt"');
expect(raw).toContain('"idempotencyKey":"codex-app-server:thread-1:turn-1:prompt"');
expect(await readTranscriptMessagesByIdentity(params)).toContainEqual(
expect.objectContaining({
role: "user",
content: "external channel prompt",
idempotencyKey: "codex-app-server:thread-1:turn-1:prompt",
}),
);
});
await vi.waitFor(() => {
expect(onUserMessagePersisted).toHaveBeenCalledWith(
@@ -1524,56 +1570,66 @@ describe("runCodexAppServerAttempt", () => {
);
});
const rawBeforeCompletion = await fs.readFile(sessionFile, "utf8");
expect(rawBeforeCompletion).not.toContain('"role":"assistant"');
const messagesBeforeCompletion = await readTranscriptMessagesByIdentity(params);
expect(messagesBeforeCompletion.some((message) => message.role === "assistant")).toBe(false);
await harness.completeTurn({ threadId: "thread-1", turnId: "turn-1" });
await run;
const rawAfterCompletion = await fs.readFile(sessionFile, "utf8");
expect(rawAfterCompletion.match(/"role":"user"/gu)).toHaveLength(1);
const messagesAfterCompletion = await readTranscriptMessagesByIdentity(params);
expect(messagesAfterCompletion.filter((message) => message.role === "user")).toHaveLength(1);
expect(onUserMessagePersisted).toHaveBeenCalledTimes(1);
});
it("does not mirror the Codex prompt early when user message persistence is suppressed", async () => {
const sessionFile = path.join(tempDir, "session-suppressed-early-prompt.jsonl");
const storePath = path.join(tempDir, "sessions-suppressed-early-prompt.json");
const workspaceDir = path.join(tempDir, "workspace-suppressed-early-prompt");
const harness = createStartedThreadHarness();
const params = createParams(sessionFile, workspaceDir);
attachSqliteSessionTarget(params, storePath, "session-suppressed-early-prompt");
params.prompt = "already persisted prompt";
params.suppressNextUserMessagePersistence = true;
const readTranscript = async () =>
fs.readFile(sessionFile, "utf8").catch((error: unknown) => {
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
return "";
}
throw error;
});
const run = runCodexAppServerAttempt(params);
await harness.waitForMethod("turn/start");
await expect(
vi.waitFor(
async () => {
const raw = await readTranscript();
expect(raw).toContain("already persisted prompt");
expect(await readTranscriptMessagesByIdentity(params)).toContainEqual(
expect.objectContaining({
content: "already persisted prompt",
}),
);
},
{ interval: 1, timeout: 100 },
),
).rejects.toThrow();
const rawBeforeCompletion = await readTranscript();
expect(rawBeforeCompletion).not.toContain("already persisted prompt");
expect(rawBeforeCompletion).not.toContain(
'"idempotencyKey":"codex-app-server:thread-1:turn-1:prompt"',
const messagesBeforeCompletion = await readTranscriptMessagesByIdentity(params);
expect(messagesBeforeCompletion).not.toContainEqual(
expect.objectContaining({
content: "already persisted prompt",
}),
);
expect(messagesBeforeCompletion).not.toContainEqual(
expect.objectContaining({
idempotencyKey: "codex-app-server:thread-1:turn-1:prompt",
}),
);
await harness.completeTurn({ threadId: "thread-1", turnId: "turn-1" });
await run;
const rawAfterCompletion = await readTranscript();
expect(rawAfterCompletion).not.toContain("already persisted prompt");
expect(rawAfterCompletion).not.toContain(
'"idempotencyKey":"codex-app-server:thread-1:turn-1:prompt"',
const messagesAfterCompletion = await readTranscriptMessagesByIdentity(params);
expect(messagesAfterCompletion).not.toContainEqual(
expect.objectContaining({
content: "already persisted prompt",
}),
);
expect(messagesAfterCompletion).not.toContainEqual(
expect.objectContaining({
idempotencyKey: "codex-app-server:thread-1:turn-1:prompt",
}),
);
});
@@ -4363,9 +4419,12 @@ describe("runCodexAppServerAttempt", () => {
}) as never,
);
const run = runCodexAppServerAttempt(
createParams(path.join(tempDir, "session.jsonl"), path.join(tempDir, "workspace")),
{
const params = createParams(
path.join(tempDir, "session.jsonl"),
path.join(tempDir, "workspace"),
);
attachSqliteSessionTarget(params, path.join(tempDir, "sessions.json"), "session-computer-use");
const run = runCodexAppServerAttempt(params, {
pluginConfig: {
computerUse: {
enabled: true,
@@ -4373,8 +4432,7 @@ describe("runCodexAppServerAttempt", () => {
mcpServerName: "desktop-control",
},
},
},
);
});
await vi.waitFor(() => expect(handleRequest).toBeTypeOf("function"));
// The keyed router only accepts turn-scoped requests once the turn is bound.
await vi.waitFor(() =>
@@ -4410,11 +4468,11 @@ describe("runCodexAppServerAttempt", () => {
expect(bridgeCall.requestParams?.serverName).toBe("desktop-control");
expect(bridgeCall.computerUseMcpServerName).toBe("desktop-control");
const requestCalls = request.mock.calls as unknown as Array<[string, unknown, unknown?]>;
const threadStart = requestCalls.find(([method]) => method === "thread/start");
const threadStartParams = threadStart?.[1] as
const turnStart = requestCalls.find(([method]) => method === "turn/start");
const turnStartParams = turnStart?.[1] as
| { approvalPolicy?: { granular?: { mcp_elicitations?: boolean } } }
| undefined;
expect(threadStartParams?.approvalPolicy?.granular?.mcp_elicitations).toBe(true);
expect(turnStartParams?.approvalPolicy?.granular?.mcp_elicitations).toBe(true);
await notify({
method: "turn/completed",
@@ -6094,8 +6152,10 @@ describe("runCodexAppServerAttempt", () => {
conversationSourceTransferComplete: true,
});
const storePath = path.join(tempDir, "sessions.json");
await saveSessionStore(storePath, {
[sessionKey]: {
await upsertSessionEntry({
storePath,
sessionKey,
entry: {
sessionId: "session-current",
updatedAt: Date.now(),
},
extensions/codex/src/app-server/run-attempt.ts
+15 -3
View File
@@ -51,7 +51,6 @@ import {
resolveDiagnosticModelContentCapturePolicy,
} from "openclaw/plugin-sdk/diagnostic-runtime";
import { loadExecApprovals } from "openclaw/plugin-sdk/exec-approvals-runtime";
import { pathExists } from "openclaw/plugin-sdk/security-runtime";
import { truncateUtf16Safe } from "openclaw/plugin-sdk/text-utility-runtime";
import {
resolveCodexAppServerForModelProvider,
@@ -251,6 +250,7 @@ import {
buildTurnCollaborationMode,
buildTurnStartParams,
codexDynamicToolsFingerprint,
codexLegacyDynamicToolsFingerprint,
resolveCodexAppServerThreadModelSelection,
type CodexAppServerThreadLifecycleBinding,
type CodexContextEngineThreadBootstrapProjection,
@@ -263,6 +263,7 @@ import {
} from "./tool-progress-normalization.js";
import {
createCodexTrajectoryRecorder,
type CodexHostTrajectoryRecorder,
normalizeCodexTrajectoryError,
recordCodexTrajectoryCompletion,
recordCodexTrajectoryContext,
@@ -1063,7 +1064,6 @@ export async function runCodexAppServerAttempt(
allocateToolOutcomeOrdinal: allocateCodexToolOutcomeOrdinal,
},
});
const hadSessionFile = await pathExists(activeSessionFile);
const activeTranscriptTarget = {
agentId: sessionAgentId,
sessionFile: activeSessionFile,
@@ -1074,6 +1074,7 @@ export async function runCodexAppServerAttempt(
!activeContextEngine && initialStartupBindingHadInactiveThreadBootstrap
? []
: ((await readMirroredSessionHistoryMessages(activeTranscriptTarget)) ?? []);
const hadSessionTranscriptState = historyMessages.length > 0;
const hookContextWindowFields = {
...(effectiveContextWindowInfo?.tokens
? { contextTokenBudget: effectiveContextWindowInfo.tokens }
@@ -1114,11 +1115,12 @@ export async function runCodexAppServerAttempt(
});
if (activeContextEngine) {
await bootstrapHarnessContextEngine({
hadSessionFile,
hadSessionFile: hadSessionTranscriptState,
contextEngine: activeContextEngine,
sessionId: activeSessionId,
sessionKey: contextSessionKey,
sessionFile: activeSessionFile,
sessionTarget: params.sessionTarget,
runtimeContext: buildActiveContextEngineRuntimeContext(),
contextEngineHostSupport: CODEX_APP_SERVER_CONTEXT_ENGINE_HOST,
providerId: effectiveRuntimeProviderId,
@@ -1231,6 +1233,7 @@ export async function runCodexAppServerAttempt(
),
projection: contextEngineProjection,
dynamicToolsFingerprint: codexDynamicToolsFingerprint(toolBridge.specs),
legacyDynamicToolsFingerprint: codexLegacyDynamicToolsFingerprint(toolBridge.specs),
})
: { project: true, reason: "per-turn-projection" };
embeddedAgentLog.info("codex app-server context-engine projection decision", {
@@ -1582,12 +1585,20 @@ export async function runCodexAppServerAttempt(
skillsPrompt: skillsCollaborationInstructions ? (params.skillsSnapshot?.prompt ?? "") : "",
tools: toolBridge.availableSpecs,
});
const hostTrajectoryRecorder = (
params as EmbeddedRunAttemptParams & {
trajectoryRecorder?: CodexHostTrajectoryRecorder | null;
}
).trajectoryRecorder;
const trajectoryRecorder = createCodexTrajectoryRecorder({
attempt: params,
cwd: effectiveCwd,
developerInstructions: buildRenderedCodexDeveloperInstructions(),
prompt: codexTurnPromptText,
trajectoryRecorder: hostTrajectoryRecorder,
trajectorySessionFile: params.trajectorySessionFile,
tools: toolBridge.availableSpecs,
warn: (message, fields) => embeddedAgentLog.warn(message, fields),
});
let client: CodexAppServerClient;
let thread: CodexAppServerThreadLifecycleBinding;
@@ -3633,6 +3644,7 @@ export async function runCodexAppServerAttempt(
sessionIdUsed: activeSessionId,
sessionKey: contextSessionKey,
sessionFile: activeSessionFile,
sessionTarget: params.sessionTarget,
messagesSnapshot: finalMessages,
prePromptMessageCount,
tokenBudget: effectiveContextTokenBudget,
extensions/codex/src/app-server/session-binding.ts
+8 -12
View File
@@ -11,11 +11,7 @@ import {
} from "openclaw/plugin-sdk/agent-runtime";
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import type { PluginStateSyncKeyedStore } from "openclaw/plugin-sdk/plugin-state-runtime";
import {
loadSessionStore,
resolveSessionStoreEntry,
resolveStorePath,
} from "openclaw/plugin-sdk/session-store-runtime";
import { getSessionEntry, resolveStorePath } from "openclaw/plugin-sdk/session-store-runtime";
import { z } from "zod";
import {
CODEX_PLUGINS_MARKETPLACE_NAME,
@@ -484,19 +480,19 @@ export async function reclaimCurrentCodexSessionGeneration(params: {
return plan.result;
}
// Only a stale stable-key owner needs filesystem authority. Resolve it before
// the second mutation so session JSON work never runs inside SQLite's write transaction.
// Only a stale stable-key owner needs session-store authority. Resolve it before
// the second mutation so the session read never runs inside the binding write transaction.
try {
const storePath = resolveStorePath(params.config?.session?.store, {
agentId: params.identity.agentId,
});
const entry = resolveSessionStoreEntry({
store: loadSessionStore(storePath, {
skipCache: true,
const entry = getSessionEntry({
agentId: params.identity.agentId,
hydrateSkillPromptRefs: false,
}),
readConsistency: "latest",
sessionKey,
}).existing;
storePath,
});
if (entry?.sessionId !== params.identity.sessionId) {
return false;
}
extensions/codex/src/app-server/session-history.test.ts
+85
View File
@@ -3,6 +3,8 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { CURRENT_SESSION_VERSION } from "openclaw/plugin-sdk/agent-sessions";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import { afterEach, describe, expect, it } from "vitest";
import { readCodexMirroredSessionHistoryMessages } from "./session-history.js";
@@ -62,6 +64,41 @@ function mirroredTarget(sessionFile: string) {
};
}
async function writeSqliteSession(params: { storedSessionFile?: string } = {}): Promise<{
marker: string;
sessionKey: string;
}> {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-codex-session-history-sqlite-"));
tempDirs.push(dir);
const storePath = path.join(dir, "openclaw-agent.sqlite");
const sessionId = "codex-sqlite-session";
const sessionKey = "agent:main:codex-sqlite";
const marker = `sqlite:main:${sessionId}:${storePath}`;
const scope = {
agentId: "main",
sessionId,
sessionKey,
storePath,
};
await upsertSessionEntry({
...scope,
entry: {
sessionFile: params.storedSessionFile ?? marker,
sessionId,
updatedAt: 1,
},
});
await appendSessionTranscriptMessageByIdentity({
...scope,
message: { role: "user", content: "sqlite prompt", timestamp: 1 },
});
await appendSessionTranscriptMessageByIdentity({
...scope,
message: { role: "assistant", content: "sqlite answer", timestamp: 2 },
});
return { marker, sessionKey };
}
describe("readCodexMirroredSessionHistoryMessages", () => {
it("treats a missing mirrored session file as empty history", async () => {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-codex-session-history-"));
@@ -84,6 +121,54 @@ describe("readCodexMirroredSessionHistoryMessages", () => {
).resolves.toBeUndefined();
});
it("replays SQLite marker history by session identity", async () => {
const { marker, sessionKey } = await writeSqliteSession();
await expect(
readCodexMirroredSessionHistoryMessages({
agentId: "main",
sessionFile: marker,
sessionId: "codex-sqlite-session",
sessionKey,
}),
).resolves.toMatchObject([
{ role: "user", content: "sqlite prompt" },
{ role: "assistant", content: "sqlite answer" },
]);
});
it("resolves SQLite marker history when the caller has no session key", async () => {
const { marker } = await writeSqliteSession();
await expect(
readCodexMirroredSessionHistoryMessages({
agentId: "main",
sessionFile: marker,
sessionId: "codex-sqlite-session",
}),
).resolves.toMatchObject([
{ role: "user", content: "sqlite prompt" },
{ role: "assistant", content: "sqlite answer" },
]);
});
it("resolves synthesized SQLite markers for stale file-backed session metadata", async () => {
const { marker } = await writeSqliteSession({
storedSessionFile: "/tmp/legacy-session.jsonl",
});
await expect(
readCodexMirroredSessionHistoryMessages({
agentId: "main",
sessionFile: marker,
sessionId: "codex-sqlite-session",
}),
).resolves.toMatchObject([
{ role: "user", content: "sqlite prompt" },
{ role: "assistant", content: "sqlite answer" },
]);
});
it("replays only the branch selected by a leaf control", async () => {
const sessionFile = await writeSession([
messageEntry({ id: "root", parentId: null, role: "user", content: "root prompt" }),
extensions/codex/src/app-server/session-history.ts
+52 -15
View File
@@ -11,9 +11,11 @@ import {
parseSessionEntries,
} from "openclaw/plugin-sdk/agent-sessions";
import {
resolveSessionTranscriptTarget,
type SessionTranscriptTargetParams,
} from "openclaw/plugin-sdk/session-transcript-runtime";
listSessionEntries,
parseSqliteSessionFileMarker,
type SqliteSessionFileMarker,
} from "openclaw/plugin-sdk/session-store-runtime";
import { readSessionTranscriptEvents } from "openclaw/plugin-sdk/session-transcript-runtime";
import { sanitizeCodexHistoryImagePayloads } from "./image-payload-sanitizer.js";
function isMissingFileError(error: unknown): boolean {
@@ -32,9 +34,7 @@ export async function readCodexMirroredSessionHistoryMessages(
target: CodexMirroredSessionHistoryTarget,
): Promise<AgentMessage[] | undefined> {
try {
await resolveSessionTranscriptTarget(resolveCodexHistoryTranscriptTarget(target));
const raw = await fs.readFile(target.sessionFile, "utf-8");
const entries = parseSessionEntries(raw);
const entries = await readCodexMirroredSessionEntries(target);
if (entries.length === 0) {
return [];
}
@@ -42,7 +42,7 @@ export async function readCodexMirroredSessionHistoryMessages(
if (firstEntry?.type !== "session" || typeof firstEntry.id !== "string") {
return undefined;
}
migrateSessionEntries(entries as SessionEntry[]);
migrateSessionEntries(entries);
const sessionEntries = entries.filter((entry): entry is SessionEntry => {
return (
entry !== null &&
@@ -64,13 +64,50 @@ export async function readCodexMirroredSessionHistoryMessages(
}
}
function resolveCodexHistoryTranscriptTarget(
async function readCodexMirroredSessionEntries(
target: CodexMirroredSessionHistoryTarget,
): SessionTranscriptTargetParams {
return {
...(target.agentId ? { agentId: target.agentId } : {}),
sessionFile: target.sessionFile,
sessionId: target.sessionId,
sessionKey: target.sessionKey ?? "",
};
): Promise<SessionEntry[]> {
const sqliteMarker = parseSqliteSessionFileMarker(target.sessionFile);
if (sqliteMarker) {
if (
sqliteMarker.sessionId !== target.sessionId ||
(target.agentId !== undefined && sqliteMarker.agentId !== target.agentId)
) {
return [];
}
const sessionKey = resolveSqliteMarkerSessionKey(target, sqliteMarker);
if (!sessionKey) {
return [];
}
return (await readSessionTranscriptEvents({
agentId: sqliteMarker.agentId,
sessionId: sqliteMarker.sessionId,
sessionKey,
storePath: sqliteMarker.storePath,
})) as SessionEntry[];
}
return parseSessionEntries(await fs.readFile(target.sessionFile, "utf-8")) as SessionEntry[];
}
function resolveSqliteMarkerSessionKey(
target: CodexMirroredSessionHistoryTarget,
marker: SqliteSessionFileMarker,
): string | undefined {
const explicitSessionKey = target.sessionKey?.trim();
if (explicitSessionKey) {
return explicitSessionKey;
}
const entries = listSessionEntries({
agentId: marker.agentId,
storePath: marker.storePath,
});
const exactEntry = entries.find(({ entry }) => {
return entry.sessionId === marker.sessionId && entry.sessionFile === target.sessionFile;
});
const sessionEntry =
exactEntry ??
entries.find(({ entry }) => {
return entry.sessionId === marker.sessionId;
});
return sessionEntry?.sessionKey;
}
extensions/codex/src/app-server/startup-binding.ts
+8
View File
@@ -9,6 +9,7 @@ import {
embeddedAgentLog,
type EmbeddedRunAttemptParams,
} from "openclaw/plugin-sdk/agent-harness-runtime";
import { parseSqliteSessionFileMarker } from "openclaw/plugin-sdk/session-store-runtime";
import { resolveCodexAppServerHomeDir } from "./auth-bridge.js";
import { isJsonObject, type JsonValue } from "./protocol.js";
import type {
@@ -127,6 +128,9 @@ async function listCodexAppServerRolloutFilesForThread(
async function readCodexSessionRecordForSessionFile(
sessionFile: string,
): Promise<(Record<string, unknown> & { sessionKey: string }) | undefined> {
if (isSqliteSessionFileMarker(sessionFile)) {
return undefined;
}
const sessionsFile = path.join(path.dirname(sessionFile), "sessions.json");
const resolvedSessionFile = path.resolve(sessionFile);
let stat: Awaited<ReturnType<typeof fs.stat>>;
@@ -175,6 +179,10 @@ async function readCodexSessionRecordForSessionFile(
return found;
}
function isSqliteSessionFileMarker(sessionFile: string | undefined): boolean {
return parseSqliteSessionFileMarker(sessionFile) !== undefined;
}
type CodexAppServerRolloutTokenSnapshot = {
totalTokens?: number;
modelContextWindow?: number;
extensions/codex/src/app-server/thread-lifecycle.test.ts
+32
View File
@@ -26,7 +26,9 @@ import {
buildTurnStartParams,
buildThreadResumeParams,
buildThreadStartParams,
areCodexDynamicToolFingerprintsCompatible,
codexDynamicToolsFingerprint,
codexLegacyDynamicToolsFingerprint,
formatCodexThreadLifecycleTimingSummary,
resolveCodexAppServerThreadModelSelection,
resolveReasoningEffort,
@@ -567,6 +569,36 @@ describe("Codex app-server native code mode config", () => {
expect(searchableFingerprint).not.toBe(directFingerprint);
});
it("keeps hashed dynamic tool fingerprints compatible with legacy JSON bindings", () => {
const tools = [
{
type: "function" as const,
name: "message",
description: "Send a visible message",
inputSchema: {
type: "object",
additionalProperties: false,
properties: {
text: { type: "string" },
},
required: ["text"],
},
},
];
const hashed = codexDynamicToolsFingerprint(tools);
const legacy = codexLegacyDynamicToolsFingerprint(tools);
expect(hashed).toMatch(/^sha256:/);
expect(legacy).toContain('"name":"message"');
expect(
areCodexDynamicToolFingerprintsCompatible({
previous: legacy,
next: hashed,
nextLegacy: legacy,
}),
).toBe(true);
});
it("keeps OpenClaw skill catalogs out of developer instructions", () => {
const params = createAttemptParams({ provider: "openai" });
params.skillsSnapshot = {
extensions/codex/src/app-server/thread-lifecycle.ts
+42 -10
View File
@@ -1,5 +1,5 @@
// Codex plugin module implements thread lifecycle behavior.
import crypto from "node:crypto";
import * as crypto from "node:crypto";
import {
buildSkillWorkshopPromptSection,
embeddedAgentLog,
@@ -445,8 +445,12 @@ export async function startOrResumeThread(params: {
...params.timing,
enabled: params.timing?.enabled ?? isCodexAppServerProfilerEnabled(params.params.config),
});
const legacyDynamicToolsFingerprint = lifecycleTiming.measureSync(
"legacy-dynamic-tools-fingerprint",
() => legacyFingerprintDynamicTools(params.dynamicTools),
);
const dynamicToolsFingerprint = lifecycleTiming.measureSync("dynamic-tools-fingerprint", () =>
fingerprintDynamicTools(params.dynamicTools),
hashDynamicToolFingerprint(legacyDynamicToolsFingerprint),
);
const dynamicToolsContainDeferred = flattenCodexDynamicToolFunctions(params.dynamicTools).some(
(tool) => tool.deferLoading === true,
@@ -922,12 +926,13 @@ export async function startOrResumeThread(params: {
!areDynamicToolFingerprintsCompatible(
binding.dynamicToolsFingerprint,
dynamicToolsFingerprint,
legacyDynamicToolsFingerprint,
)
) {
assertCodexBindingMayBeReplaced(binding, "changing the dynamic tool catalog");
preserveExistingBinding = shouldStartTransientNoToolThread({
previous: binding.dynamicToolsFingerprint,
next: dynamicToolsFingerprint,
nextHasDynamicTools: params.dynamicTools.length > 0,
});
if (preserveExistingBinding) {
embeddedAgentLog.debug(
@@ -2849,19 +2854,32 @@ export function codexDynamicToolsFingerprint(dynamicTools: CodexDynamicToolSpec[
return fingerprintDynamicTools(dynamicTools);
}
export function codexLegacyDynamicToolsFingerprint(dynamicTools: CodexDynamicToolSpec[]): string {
return legacyFingerprintDynamicTools(dynamicTools);
}
export function areCodexDynamicToolFingerprintsCompatible(params: {
previous?: string;
next: string;
nextLegacy?: string;
}): boolean {
return areDynamicToolFingerprintsCompatible(params.previous, params.next);
return areDynamicToolFingerprintsCompatible(params.previous, params.next, params.nextLegacy);
}
function fingerprintDynamicTools(dynamicTools: CodexDynamicToolSpec[]): string {
return hashDynamicToolFingerprint(legacyFingerprintDynamicTools(dynamicTools));
}
function legacyFingerprintDynamicTools(dynamicTools: CodexDynamicToolSpec[]): string {
return JSON.stringify(
dynamicTools.map(fingerprintDynamicToolSpec).toSorted(compareJsonFingerprint),
);
}
function hashDynamicToolFingerprint(canonical: string): string {
return "sha256:" + crypto.createHash("sha256").update(canonical).digest("hex");
}
function fingerprintUserMcpServersConfigPatch(
configPatch: JsonObject | undefined,
): string | undefined {
@@ -2959,20 +2977,34 @@ function readActiveCodexTurnIds(thread: unknown): string[] {
.filter((turnId) => turnId.trim().length > 0);
}
const EMPTY_DYNAMIC_TOOLS_FINGERPRINT = JSON.stringify([]);
const LEGACY_EMPTY_DYNAMIC_TOOLS_FINGERPRINT = legacyFingerprintDynamicTools([]);
const EMPTY_DYNAMIC_TOOLS_FINGERPRINT = hashDynamicToolFingerprint(
LEGACY_EMPTY_DYNAMIC_TOOLS_FINGERPRINT,
);
function areDynamicToolFingerprintsCompatible(previous: string | undefined, next: string): boolean {
return !previous || previous === next;
function areDynamicToolFingerprintsCompatible(
previous: string | undefined,
next: string,
nextLegacy?: string,
): boolean {
return !previous || previous === next || previous === nextLegacy;
}
function shouldStartTransientNoToolThread(params: {
previous: string | undefined;
next: string;
nextHasDynamicTools: boolean;
}): boolean {
return Boolean(
params.previous &&
params.previous !== EMPTY_DYNAMIC_TOOLS_FINGERPRINT &&
params.next === EMPTY_DYNAMIC_TOOLS_FINGERPRINT,
!isEmptyDynamicToolsFingerprint(params.previous) &&
!params.nextHasDynamicTools,
);
}
function isEmptyDynamicToolsFingerprint(fingerprint: string): boolean {
return (
fingerprint === EMPTY_DYNAMIC_TOOLS_FINGERPRINT ||
fingerprint === LEGACY_EMPTY_DYNAMIC_TOOLS_FINGERPRINT
);
}
extensions/codex/src/app-server/trajectory.test.ts
+180 -363
View File
@@ -1,14 +1,19 @@
// Codex tests cover trajectory plugin behavior.
// Codex tests cover SQLite-only trajectory plugin behavior.
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import {
appendSqliteTrajectoryRuntimeEvents,
loadSqliteTrajectoryRuntimeEvents,
type SqliteTrajectoryRuntimeEventForTest,
} from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
type CodexHostTrajectoryRecorder,
createCodexTrajectoryRecorder,
recordCodexTrajectoryCompletion,
recordCodexTrajectoryContext,
resolveCodexTrajectoryAppendFlags,
resolveCodexTrajectoryPointerFlags,
} from "./trajectory.js";
type CodexTrajectoryRecorder = NonNullable<ReturnType<typeof createCodexTrajectoryRecorder>>;
@@ -33,62 +38,86 @@ function expectTrajectoryRecorder(
if (recorder === null) {
throw new Error("Expected Codex trajectory recorder");
}
expect(typeof recorder.recordEvent).toBe("function");
return recorder;
}
describe("Codex trajectory recorder", () => {
it("keeps write flags usable when O_NOFOLLOW is unavailable", () => {
const constants = {
O_APPEND: 0x01,
O_CREAT: 0x02,
O_TRUNC: 0x04,
O_WRONLY: 0x08,
function createMemoryHostTrajectoryRecorder(): {
events: Array<{ type: string; data?: Record<string, unknown> }>;
recorder: CodexHostTrajectoryRecorder;
} {
const events: Array<{ type: string; data?: Record<string, unknown> }> = [];
return {
events,
recorder: {
recordEvent: (type, data) => events.push({ type, data }),
flush: async () => undefined,
},
};
}
expect(resolveCodexTrajectoryAppendFlags(constants)).toBe(0x0b);
expect(resolveCodexTrajectoryPointerFlags(constants)).toBe(0x0e);
});
it("records by default unless explicitly disabled", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
function createMemoryBackedRecorder(params: {
tmpDir: string;
attempt?: Record<string, unknown>;
tools?: Parameters<typeof createCodexTrajectoryRecorder>[0]["tools"];
}): {
events: Array<{ type: string; data?: Record<string, unknown> }>;
recorder: CodexTrajectoryRecorder;
} {
const sessionId = (params.attempt?.sessionId as string | undefined) ?? "session-1";
const host = createMemoryHostTrajectoryRecorder();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
cwd: params.tmpDir,
attempt: {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
sessionFile: path.join(params.tmpDir, "session.jsonl"),
sessionId,
sessionKey: `agent:main:${sessionId}`,
runId: "run-1",
provider: "codex",
modelId: "gpt-5.4",
model: { api: "responses" },
...params.attempt,
} as never,
trajectoryRecorder: host.recorder,
trajectorySessionFile: `sqlite:main:${sessionId}:${path.join(params.tmpDir, "sessions.json")}`,
tools: params.tools,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
trajectoryRecorder.recordEvent("session.started", {
apiKey: "secret",
headers: [{ name: "Authorization", value: "Bearer sk-test-secret-token" }],
command: "curl -H 'Authorization: Bearer sk-other-secret-token'",
});
await trajectoryRecorder.flush();
const filePath = path.join(tmpDir, "session.trajectory.jsonl");
const content = fs.readFileSync(filePath, "utf8");
expect(content).toContain('"type":"session.started"');
expect(content).not.toContain("secret");
expect(content).not.toContain("sk-test-secret-token");
expect(content).not.toContain("sk-other-secret-token");
if (process.platform !== "win32") {
expect(fs.statSync(filePath).mode & 0o777).toBe(0o600);
return { events: host.events, recorder: expectTrajectoryRecorder(recorder) };
}
expect(fs.existsSync(path.join(tmpDir, "session.trajectory-path.json"))).toBe(true);
});
it("keeps recorded string values UTF-16 safe at the trajectory boundary", async () => {
function createSqliteHostTrajectoryRecorder(params: {
agentId: string;
sessionId: string;
storePath: string;
}): CodexHostTrajectoryRecorder {
const events: SqliteTrajectoryRuntimeEventForTest[] = [];
let seq = 0;
return {
recordEvent: (type, data) => {
events.push({
traceSchema: "openclaw-trajectory",
schemaVersion: 1,
traceId: `${params.sessionId}:test`,
source: "runtime",
type,
ts: new Date(0).toISOString(),
seq,
sessionId: params.sessionId,
...(data === undefined ? {} : { data }),
});
seq += 1;
},
flush: async () => {
appendSqliteTrajectoryRuntimeEvents(params, events);
events.length = 0;
},
};
}
describe("Codex trajectory recorder", () => {
it("rejects file-backed trajectory targets without creating sidecars", () => {
const tmpDir = makeTempDir();
const warn = vi.fn();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
@@ -97,41 +126,83 @@ describe("Codex trajectory recorder", () => {
model: { api: "responses" },
} as never,
env: {},
warn,
});
const prefix = "x".repeat(19_999);
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
trajectoryRecorder.recordEvent("model.output", { text: `${prefix}😀` });
await trajectoryRecorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
expect(recorder).toBeNull();
expect(warn).toHaveBeenCalledWith(
"codex trajectory capture requires a matching SQLite session target",
{ sessionId: "session-1", reason: "non-sqlite-session-target" },
);
expect(parsed.data.text).toBe(`${prefix}…`);
expect(fs.existsSync(path.join(tmpDir, "session.trajectory.jsonl"))).toBe(false);
expect(fs.existsSync(path.join(tmpDir, "session.trajectory-path.json"))).toBe(false);
});
it("records canonical OpenAI Codex app-server turns with Codex local attribution", async () => {
it("rejects a SQLite marker for a different session identity", () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const warn = vi.fn();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
sessionFile,
sessionFile: "sqlite:main:other:/tmp/openclaw-agent.sqlite",
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
provider: "openai",
modelId: "gpt-5.5",
model: { provider: "openai", api: "openai-responses" },
runtimePlan: {
observability: {
resolvedRef: "openai/gpt-5.5",
provider: "openai",
modelId: "gpt-5.5",
harnessId: "codex",
},
},
model: { api: "responses" },
} as never,
trajectoryRecorder: createMemoryHostTrajectoryRecorder().recorder,
env: {},
warn,
});
expect(recorder).toBeNull();
expect(warn).toHaveBeenCalledWith(
"codex trajectory capture requires a matching SQLite session target",
{ sessionId: "session-1", reason: "session-id-mismatch" },
);
});
it("warns when the SQLite host recorder is unavailable", () => {
const warn = vi.fn();
const recorder = createCodexTrajectoryRecorder({
cwd: makeTempDir(),
attempt: {
sessionFile: "sqlite:main:session-1:/tmp/openclaw-agent.sqlite",
sessionId: "session-1",
model: { api: "responses" },
} as never,
env: {},
warn,
});
expect(recorder).toBeNull();
expect(warn).toHaveBeenCalledWith(
"codex trajectory capture requires the SQLite host recorder",
{ sessionId: "session-1", reason: "sqlite-recorder-unavailable" },
);
});
it("stores SQLite-backed trajectory captures in the session database", async () => {
const tmpDir = makeTempDir();
const storePath = path.join(tmpDir, "sessions", "sessions.json");
const trajectorySessionFile = `sqlite:main:session-1:${storePath}`;
await upsertSessionEntry({
agentId: "main",
sessionKey: "agent:main:session-1",
storePath,
entry: { sessionId: "session-1", sessionFile: trajectorySessionFile, updatedAt: 10 },
});
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
sessionFile: path.join(tmpDir, "sessions", "session.jsonl"),
sessionId: "session-1",
model: { api: "responses" },
} as never,
trajectoryRecorder: createSqliteHostTrajectoryRecorder({
agentId: "main",
sessionId: "session-1",
storePath,
}),
trajectorySessionFile,
env: {},
});
@@ -139,37 +210,37 @@ describe("Codex trajectory recorder", () => {
trajectoryRecorder.recordEvent("session.started");
await trajectoryRecorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
expect(fs.readdirSync(path.join(tmpDir, "sessions"))).not.toEqual(
expect.arrayContaining(["session.trajectory.jsonl", "session.trajectory-path.json"]),
);
expect(parsed.provider).toBe("openai");
expect(parsed.modelApi).toBe("openai-chatgpt-responses");
expect(parsed.modelId).toBe("gpt-5.5");
await expect(
loadSqliteTrajectoryRuntimeEvents({ agentId: "main", sessionId: "session-1", storePath }),
).resolves.toEqual([expect.objectContaining({ type: "session.started" })]);
});
it("records namespace dynamic tools as callable trajectory tool definitions", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const init = {
cwd: tmpDir,
attempt: {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
provider: "codex",
modelId: "gpt-5.4",
model: { api: "responses" },
} as never,
env: {},
tools: [
it("redacts secrets and keeps recorded strings UTF-16 safe", async () => {
const { events, recorder } = createMemoryBackedRecorder({ tmpDir: makeTempDir() });
recorder.recordEvent("model.output", {
text: `${"x".repeat(19_999)}😀`,
apiKey: "secret",
authorization: "Bearer sk-test-secret-token",
});
await recorder.flush();
expect(events[0]?.data?.text).toBe(`${"x".repeat(19_999)}…`);
expect(events[0]?.data?.apiKey).toBe("<redacted>");
expect(events[0]?.data?.authorization).toBe("<redacted>");
});
it("records namespace dynamic tools as callable trajectory definitions", async () => {
const tools = [
{
type: "namespace",
type: "namespace" as const,
name: "openclaw",
description: "",
tools: [
{
type: "function",
type: "function" as const,
name: "web_search",
description: "Search the web.",
inputSchema: { type: "object" },
@@ -177,17 +248,14 @@ describe("Codex trajectory recorder", () => {
},
],
},
],
} satisfies Parameters<typeof createCodexTrajectoryRecorder>[0];
const recorder = createCodexTrajectoryRecorder(init);
];
const tmpDir = makeTempDir();
const init = createMemoryBackedRecorder({ tmpDir, tools });
recordCodexTrajectoryContext(expectTrajectoryRecorder(recorder), init);
await recorder?.flush();
recordCodexTrajectoryContext(init.recorder, { attempt: {} as never, cwd: tmpDir, tools });
await init.recorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
);
expect(parsed.data?.tools).toEqual([
expect(init.events[0]?.data?.tools).toEqual([
{
name: "web_search",
description: "Search the web.",
@@ -196,95 +264,25 @@ describe("Codex trajectory recorder", () => {
]);
});
it("sanitizes session ids when resolving an override directory", async () => {
const tmpDir = makeTempDir();
it("honors explicit disablement without warning", () => {
const warn = vi.fn();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
cwd: makeTempDir(),
attempt: {
sessionFile: path.join(tmpDir, "session.jsonl"),
sessionId: "../evil/session",
model: { api: "responses" },
} as never,
env: { OPENCLAW_TRAJECTORY_DIR: tmpDir },
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
trajectoryRecorder.recordEvent("session.started");
await trajectoryRecorder.flush();
expect(fs.existsSync(path.join(tmpDir, "___evil_session.jsonl"))).toBe(true);
});
it("honors explicit disablement", () => {
const tmpDir = makeTempDir();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
sessionFile: path.join(tmpDir, "session.jsonl"),
sessionFile: "sqlite:main:session-1:/tmp/openclaw-agent.sqlite",
sessionId: "session-1",
model: { api: "responses" },
} as never,
env: { OPENCLAW_TRAJECTORY: "0" },
warn,
});
expect(recorder).toBeNull();
});
it("refuses to append through a symlinked parent directory", async () => {
const tmpDir = makeTempDir();
const targetDir = path.join(tmpDir, "target");
const linkDir = path.join(tmpDir, "link");
fs.mkdirSync(targetDir);
fs.symlinkSync(targetDir, linkDir);
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
sessionFile: path.join(linkDir, "session.jsonl"),
sessionId: "session-1",
model: { api: "responses" },
} as never,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
trajectoryRecorder.recordEvent("session.started");
await trajectoryRecorder.flush();
expect(fs.existsSync(path.join(targetDir, "session.trajectory.jsonl"))).toBe(false);
});
it("truncates events that exceed the runtime event byte limit", async () => {
const tmpDir = makeTempDir();
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt: {
sessionFile: path.join(tmpDir, "session.jsonl"),
sessionId: "session-1",
model: { api: "responses" },
} as never,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
trajectoryRecorder.recordEvent("context.compiled", {
fields: Object.fromEntries(
Array.from({ length: 100 }, (_, index) => [`field-${index}`, "x".repeat(3_000)]),
),
});
await trajectoryRecorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
) as { data?: { truncated?: boolean; reason?: string } };
expect(parsed.data?.truncated).toBe(true);
expect(parsed.data?.reason).toBe("trajectory-event-size-limit");
expect(warn).not.toHaveBeenCalled();
});
it("preserves usage when truncating oversized model completion events", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const attempt = {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
@@ -299,14 +297,12 @@ describe("Codex trajectory recorder", () => {
reasoningTokens: 2_038,
total: 724_402,
};
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
const { events, recorder } = createMemoryBackedRecorder({
tmpDir: makeTempDir(),
attempt,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
recordCodexTrajectoryCompletion(trajectoryRecorder, {
recordCodexTrajectoryCompletion(recorder, {
attempt,
threadId: "thread-1",
turnId: "turn-1",
@@ -321,193 +317,14 @@ describe("Codex trajectory recorder", () => {
})),
} as never,
});
await trajectoryRecorder.flush();
await recorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
);
expect(parsed.type).toBe("model.completed");
expect(parsed.data).toMatchObject({
expect(events[0]?.data).toMatchObject({
truncated: true,
reason: "trajectory-event-size-limit",
usage,
});
expect(parsed.data.messagesSnapshot).toBeUndefined();
expect(parsed.data.droppedFields).toContain("messagesSnapshot");
expect(Buffer.byteLength(JSON.stringify(parsed), "utf8")).toBeLessThanOrEqual(256 * 1024);
});
it("drops oversized preserved fields when needed to keep completion events bounded", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const attempt = {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
provider: "codex",
modelId: "gpt-5.4",
model: { api: "responses" },
} as never;
const oversizedUsage = Object.fromEntries(
Array.from({ length: 100 }, (_value, index) => [`field-${index}`, "x".repeat(5_000)]),
);
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
recordCodexTrajectoryCompletion(trajectoryRecorder, {
attempt,
threadId: "thread-1",
turnId: "turn-1",
timedOut: false,
result: {
aborted: false,
attemptUsage: oversizedUsage,
assistantTexts: ["x".repeat(32_000)],
messagesSnapshot: [{ role: "assistant", content: "x".repeat(32_000) }],
} as never,
});
await trajectoryRecorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
);
expect(parsed.data).toMatchObject({
truncated: true,
reason: "trajectory-event-size-limit",
});
expect(parsed.data.usage).toBeUndefined();
expect(parsed.data.droppedFields).toEqual(
expect.arrayContaining(["usage", "assistantTexts", "messagesSnapshot"]),
);
expect(Buffer.byteLength(JSON.stringify(parsed), "utf8")).toBeLessThanOrEqual(256 * 1024);
});
it("preserves usage on non-final oversized model completion events", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const attempt = {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
provider: "codex",
modelId: "gpt-5.4",
model: { api: "responses" },
} as never;
const firstUsage = {
input: 384_954,
output: 5_624,
cacheRead: 333_824,
reasoningTokens: 2_038,
total: 724_402,
};
const secondUsage = { input: 12, output: 3, total: 15 };
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
recordCodexTrajectoryCompletion(trajectoryRecorder, {
attempt,
threadId: "thread-1",
turnId: "turn-1",
timedOut: false,
result: {
aborted: false,
attemptUsage: firstUsage,
assistantTexts: ["first"],
messagesSnapshot: Array.from({ length: 20 }, (_value, index) => ({
role: index % 2 === 0 ? "user" : "assistant",
content: `message-${index} ${"x".repeat(32_000)}`,
})),
} as never,
});
recordCodexTrajectoryCompletion(trajectoryRecorder, {
attempt,
threadId: "thread-1",
turnId: "turn-2",
timedOut: false,
result: {
aborted: false,
attemptUsage: secondUsage,
assistantTexts: ["final answer"],
messagesSnapshot: [{ role: "assistant", content: "final answer" }],
} as never,
});
await trajectoryRecorder.flush();
const events = fs
.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8")
.trim()
.split(/\r?\n/u)
.map((line) => JSON.parse(line));
expect(events).toHaveLength(2);
expect(events[0].data).toMatchObject({
truncated: true,
usage: firstUsage,
});
expect(events[1].data).toMatchObject({
turnId: "turn-2",
usage: secondUsage,
assistantTexts: ["final answer"],
});
expect(events[1].data.truncated).toBeUndefined();
});
it("redacts secrets before preserving usage in truncated completion events", async () => {
const tmpDir = makeTempDir();
const sessionFile = path.join(tmpDir, "session.jsonl");
const attempt = {
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:session-1",
runId: "run-1",
provider: "codex",
modelId: "gpt-5.4",
model: { api: "responses" },
} as never;
const recorder = createCodexTrajectoryRecorder({
cwd: tmpDir,
attempt,
env: {},
});
const trajectoryRecorder = expectTrajectoryRecorder(recorder);
recordCodexTrajectoryCompletion(trajectoryRecorder, {
attempt,
threadId: "thread-1",
turnId: "turn-1",
timedOut: false,
result: {
aborted: false,
attemptUsage: {
total: 1,
apiKey: "sk-test-secret-token",
authorization: "Bearer sk-other-secret-token",
},
assistantTexts: ["done"],
messagesSnapshot: Array.from({ length: 20 }, (_value, index) => ({
role: index % 2 === 0 ? "user" : "assistant",
content: `message-${index} ${"x".repeat(32_000)}`,
})),
} as never,
});
await trajectoryRecorder.flush();
const parsed = JSON.parse(
fs.readFileSync(path.join(tmpDir, "session.trajectory.jsonl"), "utf8"),
);
const preservedUsage = JSON.stringify(parsed.data.usage);
expect(parsed.data.truncated).toBe(true);
expect(preservedUsage).toContain("redacted");
expect(preservedUsage).not.toContain("sk-test-secret-token");
expect(preservedUsage).not.toContain("sk-other-secret-token");
expect(events[0]?.data?.messagesSnapshot).toBeUndefined();
expect(events[0]?.data?.droppedFields).toContain("messagesSnapshot");
});
});
extensions/codex/src/app-server/trajectory.ts
+56 -156
View File
@@ -1,26 +1,18 @@
/**
* Records optional Codex runtime trajectory sidecars with bounded, redacted
* context and completion events.
* Records optional Codex runtime trajectory events with bounded, redacted
* context and completion payloads.
*/
import nodeFs from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
import { resolveUserPath } from "openclaw/plugin-sdk/agent-harness-runtime";
import type {
EmbeddedRunAttemptParams,
EmbeddedRunAttemptResult,
} from "openclaw/plugin-sdk/agent-harness-runtime";
import {
appendRegularFile,
resolveRegularFileAppendFlags,
} from "openclaw/plugin-sdk/security-runtime";
import { parseSqliteSessionFileMarker } from "openclaw/plugin-sdk/session-store-runtime";
import { truncateUtf16Safe } from "openclaw/plugin-sdk/text-utility-runtime";
import { resolveCodexLocalRuntimeAttribution } from "./local-runtime-attribution.js";
import { flattenCodexDynamicToolFunctions, type CodexDynamicToolSpec } from "./protocol.js";
/** Runtime trajectory recorder used by Codex run attempts and event projectors. */
export type CodexTrajectoryRecorder = {
filePath: string;
recordEvent: (type: string, data?: Record<string, unknown>) => void;
flush: () => Promise<void>;
};
@@ -30,8 +22,11 @@ type CodexTrajectoryInit = {
cwd: string;
developerInstructions?: string;
prompt?: string;
trajectoryRecorder?: CodexHostTrajectoryRecorder | null;
trajectorySessionFile?: string;
tools?: CodexDynamicToolSpec[];
env?: NodeJS.ProcessEnv;
warn?: (message: string, fields: Record<string, unknown>) => void;
};
const SENSITIVE_FIELD_RE = /(?:authorization|cookie|credential|key|password|passwd|secret|token)/iu;
@@ -39,50 +34,29 @@ const PRIVATE_PAYLOAD_FIELD_RE = /(?:image|screenshot|attachment|fileData|dataUr
const AUTHORIZATION_VALUE_RE = /\b(Bearer|Basic)\s+[A-Za-z0-9+/._~=-]{8,}/giu;
const JWT_VALUE_RE = /\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b/gu;
const COOKIE_PAIR_RE = /\b([A-Za-z][A-Za-z0-9_.-]{1,64})=([A-Za-z0-9+/._~%=-]{16,})(?=;|\s|$)/gu;
const TRAJECTORY_RUNTIME_FILE_MAX_BYTES = 50 * 1024 * 1024;
const TRAJECTORY_RUNTIME_EVENT_MAX_BYTES = 256 * 1024;
const TRAJECTORY_RUNTIME_OVERSIZE_PRESERVED_DATA_KEYS = ["usage", "promptCache"] as const;
type CodexTrajectoryOpenFlagConstants = Pick<
typeof nodeFs.constants,
"O_APPEND" | "O_CREAT" | "O_TRUNC" | "O_WRONLY"
> &
Partial<Pick<typeof nodeFs.constants, "O_NOFOLLOW">>;
type CodexTrajectorySink = {
flush: () => Promise<void>;
write: (event: CodexTrajectoryEvent) => void;
};
/** Resolves secure append flags for trajectory runtime files. */
export function resolveCodexTrajectoryAppendFlags(
constants: CodexTrajectoryOpenFlagConstants = nodeFs.constants,
): number {
return resolveRegularFileAppendFlags(constants);
}
export type CodexHostTrajectoryRecorder = {
recordEvent: (type: string, data?: Record<string, unknown>) => void;
flush: () => Promise<void>;
};
/** Resolves secure create/truncate flags for trajectory pointer files. */
export function resolveCodexTrajectoryPointerFlags(
constants: CodexTrajectoryOpenFlagConstants = nodeFs.constants,
): number {
const noFollow = constants.O_NOFOLLOW;
return (
constants.O_CREAT |
constants.O_TRUNC |
constants.O_WRONLY |
(typeof noFollow === "number" ? noFollow : 0)
);
}
type CodexTrajectoryEvent = Record<string, unknown> & {
data?: Record<string, unknown>;
type: string;
};
async function safeAppendTrajectoryFile(filePath: string, line: string): Promise<void> {
await appendRegularFile({
filePath,
content: line,
maxFileBytes: TRAJECTORY_RUNTIME_FILE_MAX_BYTES,
rejectSymlinkParents: true,
});
}
function boundedTrajectoryLine(event: Record<string, unknown>): string | undefined {
function boundedTrajectoryEvent(event: Record<string, unknown>): CodexTrajectoryEvent | undefined {
const line = JSON.stringify(event);
const bytes = Buffer.byteLength(line, "utf8");
if (bytes <= TRAJECTORY_RUNTIME_EVENT_MAX_BYTES) {
return `${line}\n`;
return event as CodexTrajectoryEvent;
}
const originalData =
@@ -97,7 +71,7 @@ function boundedTrajectoryLine(event: Record<string, unknown>): string | undefin
limitBytes: TRAJECTORY_RUNTIME_EVENT_MAX_BYTES,
reason: "trajectory-event-size-limit",
};
const buildTruncatedLine = (includeDroppedFields: boolean): string | undefined => {
const buildTruncatedEvent = (includeDroppedFields: boolean): CodexTrajectoryEvent | undefined => {
const data: Record<string, unknown> = { ...baseData };
for (const key of TRAJECTORY_RUNTIME_OVERSIZE_PRESERVED_DATA_KEYS) {
if (preservedDataKeys.has(key)) {
@@ -110,14 +84,15 @@ function boundedTrajectoryLine(event: Record<string, unknown>): string | undefin
data.droppedFields = droppedFields;
}
}
const truncated = JSON.stringify({ ...event, data });
const truncatedEvent = { ...event, data };
const truncated = JSON.stringify(truncatedEvent);
if (Buffer.byteLength(truncated, "utf8") <= TRAJECTORY_RUNTIME_EVENT_MAX_BYTES) {
return `${truncated}\n`;
return truncatedEvent as CodexTrajectoryEvent;
}
return undefined;
};
let best = buildTruncatedLine(true) ?? buildTruncatedLine(false);
let best = buildTruncatedEvent(true) ?? buildTruncatedEvent(false);
if (!best) {
return undefined;
}
@@ -127,7 +102,7 @@ function boundedTrajectoryLine(event: Record<string, unknown>): string | undefin
continue;
}
preservedDataKeys.add(key);
const next = buildTruncatedLine(true) ?? buildTruncatedLine(false);
const next = buildTruncatedEvent(true) ?? buildTruncatedEvent(false);
if (next) {
best = next;
continue;
@@ -137,55 +112,17 @@ function boundedTrajectoryLine(event: Record<string, unknown>): string | undefin
return best;
}
function resolveTrajectoryPointerFilePath(sessionFile: string): string {
return sessionFile.endsWith(".jsonl")
? `${sessionFile.slice(0, -".jsonl".length)}.trajectory-path.json`
: `${sessionFile}.trajectory-path.json`;
}
function writeTrajectoryPointerBestEffort(params: {
filePath: string;
sessionFile: string;
sessionId: string;
}): void {
const pointerPath = resolveTrajectoryPointerFilePath(params.sessionFile);
try {
const pointerDir = path.resolve(path.dirname(pointerPath));
if (nodeFs.lstatSync(pointerDir).isSymbolicLink()) {
return;
}
try {
if (nodeFs.lstatSync(pointerPath).isSymbolicLink()) {
return;
}
} catch (error) {
if ((error as NodeJS.ErrnoException).code !== "ENOENT") {
return;
}
}
const fd = nodeFs.openSync(pointerPath, resolveCodexTrajectoryPointerFlags(), 0o600);
try {
nodeFs.writeFileSync(
fd,
`${JSON.stringify(
{
traceSchema: "openclaw-trajectory-pointer",
schemaVersion: 1,
sessionId: params.sessionId,
runtimeFile: params.filePath,
function createCodexHostTrajectorySink(params: {
recorder: CodexHostTrajectoryRecorder;
}): CodexTrajectorySink {
return {
write: (event) => {
params.recorder.recordEvent(event.type, event.data);
},
null,
2,
)}\n`,
"utf8",
);
nodeFs.fchmodSync(fd, 0o600);
} finally {
nodeFs.closeSync(fd);
}
} catch {
// Pointer files are best-effort; the runtime sidecar itself is authoritative.
}
flush: async () => {
await params.recorder.flush();
},
};
}
/** Creates a trajectory recorder when trajectory capture is enabled for the environment. */
@@ -198,27 +135,29 @@ export function createCodexTrajectoryRecorder(
return null;
}
const filePath = resolveTrajectoryFilePath({
env,
sessionFile: params.attempt.sessionFile,
const sessionFile = params.trajectorySessionFile ?? params.attempt.sessionFile;
const sqliteMarker = parseSqliteSessionFileMarker(sessionFile);
if (!sqliteMarker || sqliteMarker.sessionId !== params.attempt.sessionId) {
params.warn?.("codex trajectory capture requires a matching SQLite session target", {
sessionId: params.attempt.sessionId,
reason: sqliteMarker ? "session-id-mismatch" : "non-sqlite-session-target",
});
const ready = fs
.mkdir(path.dirname(filePath), { recursive: true, mode: 0o700 })
.catch(() => undefined);
writeTrajectoryPointerBestEffort({
filePath,
sessionFile: params.attempt.sessionFile,
return null;
}
if (!params.trajectoryRecorder) {
params.warn?.("codex trajectory capture requires the SQLite host recorder", {
sessionId: params.attempt.sessionId,
reason: "sqlite-recorder-unavailable",
});
let queue = Promise.resolve();
return null;
}
const sink = createCodexHostTrajectorySink({ recorder: params.trajectoryRecorder });
let seq = 0;
const attribution = resolveCodexLocalRuntimeAttribution(params.attempt);
return {
filePath,
recordEvent: (type, data) => {
const event = {
const event = boundedTrajectoryEvent({
traceSchema: "openclaw-trajectory",
schemaVersion: 1,
traceId: params.attempt.sessionId,
@@ -235,19 +174,12 @@ export function createCodexTrajectoryRecorder(
modelId: params.attempt.modelId,
modelApi: attribution.api,
data: data ? sanitizeValue(data) : undefined,
};
const line = boundedTrajectoryLine(event);
if (!line) {
return;
});
if (event) {
sink.write(event);
}
queue = queue
.then(() => ready)
.then(() => safeAppendTrajectoryFile(filePath, line))
.catch(() => undefined);
},
flush: async () => {
await queue;
},
flush: sink.flush,
};
}
@@ -306,38 +238,6 @@ function parseTrajectoryEnabled(env: NodeJS.ProcessEnv): boolean {
return true;
}
function resolveTrajectoryFilePath(params: {
env: NodeJS.ProcessEnv;
sessionFile: string;
sessionId: string;
}): string {
const dirOverride = params.env.OPENCLAW_TRAJECTORY_DIR?.trim();
if (dirOverride) {
return resolveContainedPath(
resolveUserPath(dirOverride),
`${safeTrajectorySessionFileName(params.sessionId)}.jsonl`,
);
}
return params.sessionFile.endsWith(".jsonl")
? `${params.sessionFile.slice(0, -".jsonl".length)}.trajectory.jsonl`
: `${params.sessionFile}.trajectory.jsonl`;
}
function safeTrajectorySessionFileName(sessionId: string): string {
const safe = sessionId.replaceAll(/[^A-Za-z0-9_-]/g, "_").slice(0, 120);
return /[A-Za-z0-9]/u.test(safe) ? safe : "session";
}
function resolveContainedPath(baseDir: string, fileName: string): string {
const resolvedBase = path.resolve(baseDir);
const resolvedFile = path.resolve(resolvedBase, fileName);
const relative = path.relative(resolvedBase, resolvedFile);
if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) {
throw new Error("Trajectory file path escaped its configured directory");
}
return resolvedFile;
}
function toTrajectoryToolDefinitions(
tools: readonly CodexDynamicToolSpec[] | undefined,
): Array<{ name: string; description?: string; parameters?: unknown }> | undefined {
@@ -362,8 +262,8 @@ function toTrajectoryToolDefinitions(
}
function sanitizeValue(value: unknown, depth = 0, key = ""): unknown {
// Trajectory files may be inspected outside the live process, so redact
// credentials and private payloads before queueing the line for disk writes.
// Trajectory exports may leave the live process, so redact credentials and
// private payloads before passing events to the SQLite host recorder.
if (value == null || typeof value === "boolean" || typeof value === "number") {
return value;
}
extensions/codex/src/app-server/transcript-mirror.test.ts
+237 -361
View File
@@ -10,6 +10,8 @@ import {
resetGlobalHookRunner,
} from "openclaw/plugin-sdk/hook-runtime";
import { createMockPluginRegistry } from "openclaw/plugin-sdk/plugin-test-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { readSessionTranscriptEvents } from "openclaw/plugin-sdk/session-transcript-runtime";
import {
castAgentMessage,
makeAgentAssistantMessage,
@@ -40,7 +42,7 @@ vi.mock("openclaw/plugin-sdk/session-transcript-runtime", async (importOriginal)
type MirroredAgentMessage = Extract<AgentMessage, { role: "user" | "assistant" | "toolResult" }>;
// Mirrors transcript-mirror.ts's fallback fingerprint exactly so test
// Mirrors transcript-mirror.ts's content fingerprint exactly so test
// expectations stay in sync without exposing the helper publicly.
function expectedFingerprint(message: MirroredAgentMessage): string {
const payload = JSON.stringify({ role: message.role, content: message.content });
@@ -64,26 +66,6 @@ afterEach(async () => {
}
});
async function createTempSessionFile() {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-codex-transcript-"));
tempDirs.push(dir);
return path.join(dir, "session.jsonl");
}
async function initializeSessionTranscript(sessionFile: string, sessionId: string): Promise<void> {
await fs.writeFile(
sessionFile,
`${JSON.stringify({
type: "session",
version: 3,
id: sessionId,
timestamp: new Date().toISOString(),
cwd: process.cwd(),
})}\n`,
"utf8",
);
}
async function makeRoot(prefix: string): Promise<string> {
const root = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
tempDirs.push(root);
@@ -121,20 +103,84 @@ describe("buildCodexUserPromptMessage", () => {
});
});
function parseJsonLines<T>(raw: string): T[] {
const records: T[] = [];
for (const line of raw.trim().split("\n")) {
if (line.length > 0) {
records.push(JSON.parse(line) as T);
function readEventMessages(events: unknown[]): Array<{ role?: string; text?: string }> {
return events
.map((event) =>
event && typeof event === "object" ? (event as { message?: unknown }).message : undefined,
)
.filter((message): message is { role?: string; content?: unknown } =>
Boolean(message && typeof message === "object"),
)
.map((message) => {
const content = Array.isArray(message.content)
? message.content.find((part): part is { text: string } =>
Boolean(part && typeof part === "object" && typeof part.text === "string"),
)?.text
: typeof message.content === "string"
? message.content
: undefined;
return { role: message.role, text: content };
});
}
async function createSqliteMirrorTarget(prefix: string, options: { sessionId?: string } = {}) {
const root = await makeRoot(prefix);
const agentId = "main";
const sessionId = options.sessionId ?? "session-1";
const sessionKey = `agent:${agentId}:${sessionId}`;
const storePath = path.join(root, "openclaw-agent.sqlite");
await upsertSessionEntry({
agentId,
sessionKey,
storePath,
entry: {
sessionFile: `sqlite:${agentId}:${sessionId}:${storePath}`,
sessionId,
updatedAt: 1,
},
});
return {
agentId,
sessionId,
sessionKey,
storePath,
bogusSessionFile: path.join(root, "should-not-be-created.jsonl"),
};
}
return records;
async function readMirrorEvents(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<unknown[]> {
return await readSessionTranscriptEvents(target);
}
async function readMirrorRaw(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<string> {
return (await readMirrorEvents(target)).map((event) => JSON.stringify(event)).join("\n");
}
async function readMirrorMessages(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<Array<{ role?: string; text?: string }>> {
return readEventMessages(await readMirrorEvents(target));
}
describe("importCodexThreadHistoryToTranscript", () => {
it("imports only bounded user-visible conversation items with stable identities", async () => {
const sessionFile = await createTempSessionFile();
await initializeSessionTranscript(sessionFile, "session-history");
const target = await createSqliteMirrorTarget("openclaw-codex-history-", {
sessionId: "session-history",
});
const sessionFile = `sqlite:${target.agentId}:${target.sessionId}:${target.storePath}`;
const thread = {
id: "thread-history",
cwd: "/workspace/project",
@@ -201,14 +247,16 @@ describe("importCodexThreadHistoryToTranscript", () => {
importCodexThreadHistoryToTranscript({
thread,
throughTurnId: "turn-1",
sessionFile,
storePath: target.storePath,
sessionId: "session-history",
sessionKey: "agent:main:dashboard:history",
sessionKey: target.sessionKey,
agentId: target.agentId,
}),
).resolves.toEqual({ importedMessages: 2, omittedMessages: 0 });
const raw = await fs.readFile(sessionFile, "utf8");
const messages = parseJsonLines<{ message?: AgentMessage; type?: string }>(raw)
const events = await readMirrorEvents(target);
const raw = events.map((event) => JSON.stringify(event)).join("\n");
const messages = (events as Array<{ message?: AgentMessage; type?: string }>)
.filter((event) => event.type === "message")
.map((event) => event.message);
expect(messages).toMatchObject([
@@ -237,7 +285,8 @@ describe("importCodexThreadHistoryToTranscript", () => {
readCodexMirroredSessionHistoryMessages({
sessionFile,
sessionId: "session-history",
sessionKey: "agent:main:dashboard:history",
sessionKey: target.sessionKey,
agentId: target.agentId,
}),
).resolves.toMatchObject([
{ role: "user", content: "Review this image\n[Image attachment]" },
@@ -253,8 +302,9 @@ describe("importCodexThreadHistoryToTranscript", () => {
});
it("keeps the newest 200 visible messages and deduplicates a retried import", async () => {
const sessionFile = await createTempSessionFile();
await initializeSessionTranscript(sessionFile, "session-bounded-history");
const target = await createSqliteMirrorTarget("openclaw-codex-bounded-history-", {
sessionId: "session-bounded-history",
});
const thread = {
id: "thread-bounded-history",
turns: Array.from({ length: 205 }, (_, index) => ({
@@ -274,9 +324,10 @@ describe("importCodexThreadHistoryToTranscript", () => {
const importParams = {
thread,
throughTurnId: "turn-204",
sessionFile,
storePath: target.storePath,
sessionId: "session-bounded-history",
sessionKey: "agent:main:dashboard:bounded-history",
sessionKey: target.sessionKey,
agentId: target.agentId,
};
await expect(importCodexThreadHistoryToTranscript(importParams)).resolves.toEqual({
@@ -288,8 +339,8 @@ describe("importCodexThreadHistoryToTranscript", () => {
omittedMessages: 5,
});
const raw = await fs.readFile(sessionFile, "utf8");
const messages = parseJsonLines<{ message?: AgentMessage; type?: string }>(raw)
const events = await readMirrorEvents(target);
const messages = (events as Array<{ message?: AgentMessage; type?: string }>)
.filter((event) => event.type === "message")
.map((event) => event.message);
expect(messages).toHaveLength(200);
@@ -298,8 +349,10 @@ describe("importCodexThreadHistoryToTranscript", () => {
});
it("assigns canonical assistant attribution and numeric fallback timestamps", async () => {
const sessionFile = await createTempSessionFile();
await initializeSessionTranscript(sessionFile, "session-fallback-history");
const target = await createSqliteMirrorTarget("openclaw-codex-fallback-history-", {
sessionId: "session-fallback-history",
});
const sessionFile = `sqlite:${target.agentId}:${target.sessionId}:${target.storePath}`;
const thread = {
id: "thread-fallback-history",
modelProvider: "source-provider",
@@ -326,15 +379,17 @@ describe("importCodexThreadHistoryToTranscript", () => {
await importCodexThreadHistoryToTranscript({
thread,
throughTurnId: "turn-without-time",
sessionFile,
storePath: target.storePath,
sessionId: "session-fallback-history",
sessionKey: "agent:main:dashboard:fallback-history",
sessionKey: target.sessionKey,
agentId: target.agentId,
});
const history = await readCodexMirroredSessionHistoryMessages({
sessionFile,
sessionId: "session-fallback-history",
sessionKey: "agent:main:dashboard:fallback-history",
sessionKey: target.sessionKey,
agentId: target.agentId,
});
expect(history).toMatchObject([
{ role: "user", content: "Earlier prompt", timestamp: expect.any(Number) },
@@ -569,8 +624,8 @@ describe("projectBoundedCodexThreadHistory", () => {
});
describe("mirrorCodexAppServerTranscript", () => {
it("mirrors user, assistant, and tool result messages into the embedded-agent transcript", async () => {
const sessionFile = await createTempSessionFile();
it("mirrors user, assistant, and tool result messages by SQLite identity", async () => {
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-basic-");
const userMessage = makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
@@ -583,25 +638,17 @@ describe("mirrorCodexAppServerTranscript", () => {
role: "toolResult",
toolCallId: "call-1",
toolName: "read",
content: [
{
type: "toolResult",
toolCallId: "call-1",
content: "read output",
},
],
content: [{ type: "toolResult", toolCallId: "call-1", content: "read output" }],
timestamp: Date.now() + 2,
}) as MirroredAgentMessage;
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage, assistantMessage, toolResultMessage],
idempotencyScope: "scope-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"role":"user"');
expect(raw).toContain('"content":[{"type":"text","text":"hello"}]');
expect(raw).toContain('"role":"assistant"');
@@ -616,10 +663,14 @@ describe("mirrorCodexAppServerTranscript", () => {
expect(raw).toContain(
`"idempotencyKey":"scope-1:toolResult:${expectedFingerprint(toolResultMessage)}"`,
);
await expect(fs.readFile(target.bogusSessionFile, "utf8")).rejects.toHaveProperty(
"code",
"ENOENT",
);
});
it("preserves gateway user-turn identity across Codex transcript mirroring", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-user-identity-");
const userMessage = castAgentMessage({
...makeAgentUserMessage({
content: [{ type: "text", text: "client prompt" }],
@@ -629,35 +680,29 @@ describe("mirrorCodexAppServerTranscript", () => {
}) as MirroredAgentMessage;
const first = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
...target,
messages: [userMessage],
idempotencyScope: "codex-app-server:thread-1",
});
const second = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
...target,
messages: [userMessage],
idempotencyScope: "codex-app-server:thread-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"idempotencyKey":"client-run:user"');
expect(raw).toContain('"mirrorOrigin":"codex-app-server"');
expect(raw).not.toContain('"idempotencyKey":"codex-app-server:thread-1:');
expect(first.userMessagesPresent).toHaveLength(1);
expect(second.userMessagesPresent).toHaveLength(1);
expect(
parseJsonLines<{ message?: { role?: string } }>(raw).filter(
(record) => record.message?.role === "user",
),
(await readMirrorMessages(target)).filter((message) => message.role === "user"),
).toHaveLength(1);
});
it("emits message-bearing updates for newly appended mirrored messages only", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-live-updates-");
const userMessage = attachCodexMirrorIdentity(
makeAgentUserMessage({
content: [{ type: "text", text: "show me live" }],
@@ -667,16 +712,12 @@ describe("mirrorCodexAppServerTranscript", () => {
);
const firstMirror = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
...target,
messages: [userMessage],
idempotencyScope: "codex-app-server:thread-1",
});
const secondMirror = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
...target,
messages: [userMessage],
idempotencyScope: "codex-app-server:thread-1",
});
@@ -685,8 +726,8 @@ describe("mirrorCodexAppServerTranscript", () => {
([update]) => update as Record<string, unknown> & { update?: Record<string, unknown> },
);
expect(updates).toHaveLength(1);
expect(updates[0]?.sessionFile).toBe(sessionFile);
expect(updates[0]?.sessionKey).toBe("agent:main:main");
expect(updates[0]?.sessionKey).toBe(target.sessionKey);
expect(updates[0]?.storePath).toBe(target.storePath);
expect(updates[0]?.update?.messageId).toEqual(expect.any(String));
expect(updates[0]?.update?.message).toMatchObject({
role: "user",
@@ -708,99 +749,11 @@ describe("mirrorCodexAppServerTranscript", () => {
});
});
it("reports final assistant ownership for new and idempotent mirrors", async () => {
const sessionFile = await createTempSessionFile();
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "owned once" }],
timestamp: Date.now(),
}),
"turn-1:assistant",
);
const firstMirror = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
const secondMirror = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
expect(firstMirror.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
expect(secondMirror.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
const records = parseJsonLines<{ type?: string; message?: { role?: string } }>(
await fs.readFile(sessionFile, "utf8"),
);
expect(records.filter((record) => record.message?.role === "assistant")).toHaveLength(1);
});
it("keeps assistant ownership when live update publication fails", async () => {
publishSessionTranscriptUpdateByIdentityMock.mockRejectedValueOnce(new Error("publish failed"));
const sessionFile = await createTempSessionFile();
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "durably persisted" }],
timestamp: Date.now(),
}),
"turn-1:assistant",
);
const result = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
expect(result.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
expect(await fs.readFile(sessionFile, "utf8")).toContain('"role":"assistant"');
});
it("leaves the assistant unowned when transcript persistence fails", async () => {
const root = await makeRoot("openclaw-codex-transcript-failure-");
const invalidParent = path.join(root, "not-a-directory");
await fs.writeFile(invalidParent, "file blocks transcript directory creation", "utf8");
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "needs fallback persistence" }],
timestamp: Date.now(),
}),
"turn-1:assistant",
);
const assistantTranscriptOwned = await mirrorTranscriptBestEffort({
params: {
sessionFile: path.join(invalidParent, "session.jsonl"),
sessionId: "session-1",
suppressNextUserMessagePersistence: true,
} as Parameters<typeof mirrorTranscriptBestEffort>[0]["params"],
result: {
messagesSnapshot: [assistantMessage],
} as Parameters<typeof mirrorTranscriptBestEffort>[0]["result"],
notifyUserMessagePersisted: vi.fn(),
cwd: root,
threadId: "thread-1",
turnId: "turn-1",
});
expect(assistantTranscriptOwned).toBe(false);
});
it("emits stable sequence numbers for multi-message mirror batches", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-seq-");
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "agent:main:main",
...target,
messages: [
attachCodexMirrorIdentity(
makeAgentUserMessage({
@@ -832,30 +785,43 @@ describe("mirrorCodexAppServerTranscript", () => {
).toEqual(["user", "assistant"]);
});
it("creates the transcript directory on first mirror", async () => {
const root = await makeRoot("openclaw-codex-transcript-missing-dir-");
const sessionFile = path.join(root, "nested", "sessions", "session.jsonl");
it("keeps assistant ownership when live update publication fails", async () => {
publishSessionTranscriptUpdateByIdentityMock.mockRejectedValueOnce(new Error("publish failed"));
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-publish-failure-");
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "durably persisted" }],
timestamp: Date.now(),
}),
"turn-1:assistant",
);
await mirrorCodexAppServerTranscript({
sessionFile,
const result = await mirrorCodexAppServerTranscript({
...target,
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
expect(result.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
expect(await readMirrorRaw(target)).toContain('"role":"assistant"');
});
it("rejects mirror writes without a runtime session identity", async () => {
await expect(
mirrorCodexAppServerTranscript({
sessionId: "session-1",
sessionKey: "session-1",
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "first mirror" }],
content: [{ type: "text", text: "no identity" }],
timestamp: Date.now(),
}),
],
idempotencyScope: "scope-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
expect(raw).toContain('"role":"assistant"');
expect(raw).toContain('"content":[{"type":"text","text":"first mirror"}]');
}),
).rejects.toThrow("runtime session identity");
});
it("deduplicates app-server turn mirrors by idempotency scope", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-dedupe-");
const messages = [
makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
@@ -868,24 +834,45 @@ describe("mirrorCodexAppServerTranscript", () => {
] as const;
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [...messages],
idempotencyScope: "scope-1",
});
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [...messages],
idempotencyScope: "scope-1",
});
const records = parseJsonLines<{ type?: string; message?: { role?: string } }>(
await fs.readFile(sessionFile, "utf8"),
expect((await readMirrorMessages(target)).filter((message) => message.role)).toHaveLength(2);
});
it("reports final assistant ownership for new and idempotent mirrors", async () => {
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-assistant-owned-");
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "owned once" }],
timestamp: Date.now(),
}),
"turn-1:assistant",
);
expect(records.slice(1)).toHaveLength(2);
const firstMirror = await mirrorCodexAppServerTranscript({
...target,
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
const secondMirror = await mirrorCodexAppServerTranscript({
...target,
messages: [assistantMessage],
idempotencyScope: "codex-app-server:thread-1",
});
expect(firstMirror.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
expect(secondMirror.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
expect(
(await readMirrorMessages(target)).filter((message) => message.role === "assistant"),
).toHaveLength(1);
});
it("runs before_message_write before appending mirrored transcript messages", async () => {
@@ -902,24 +889,20 @@ describe("mirrorCodexAppServerTranscript", () => {
},
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-hook-");
const sourceMessage = makeAgentAssistantMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
});
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [sourceMessage],
idempotencyScope: "scope-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"content":[{"type":"text","text":"hello [hooked]"}]');
// The idempotency fingerprint is derived from the pre-hook message so a
// hook rewrite cannot bypass dedupe by reshaping content on every retry.
expect(raw).toContain(
`"idempotencyKey":"scope-1:assistant:${expectedFingerprint(sourceMessage)}"`,
);
@@ -939,23 +922,19 @@ describe("mirrorCodexAppServerTranscript", () => {
},
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-duplicates-");
const sourceMessage = makeAgentUserMessage({
content: [{ type: "text", text: "secret prompt" }],
timestamp: Date.now(),
});
const first = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [sourceMessage],
idempotencyScope: "scope-1",
});
const second = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [sourceMessage],
idempotencyScope: "scope-1",
});
@@ -967,10 +946,9 @@ describe("mirrorCodexAppServerTranscript", () => {
{ type: "text", text: "[redacted by hook]" },
]);
expect(JSON.stringify(second.userMessagesPresent)).not.toContain("secret prompt");
const records = parseJsonLines<{ type?: string; message?: { role?: string } }>(
await fs.readFile(sessionFile, "utf8"),
);
expect(records.filter((record) => record.message?.role === "user")).toHaveLength(1);
expect(
(await readMirrorMessages(target)).filter((message) => message.role === "user"),
).toHaveLength(1);
});
it("preserves the computed idempotency key when hooks rewrite message keys", async () => {
@@ -987,21 +965,19 @@ describe("mirrorCodexAppServerTranscript", () => {
},
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-key-hook-");
const sourceMessage = makeAgentAssistantMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
});
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [sourceMessage],
idempotencyScope: "scope-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain(
`"idempotencyKey":"scope-1:assistant:${expectedFingerprint(sourceMessage)}"`,
);
@@ -1011,18 +987,13 @@ describe("mirrorCodexAppServerTranscript", () => {
it("respects before_message_write blocking decisions", async () => {
initializeGlobalHookRunner(
createMockPluginRegistry([
{
hookName: "before_message_write",
handler: () => ({ block: true }),
},
{ hookName: "before_message_write", handler: () => ({ block: true }) },
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-blocked-");
const result = await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [
attachCodexMirrorIdentity(
makeAgentAssistantMessage({
@@ -1036,95 +1007,38 @@ describe("mirrorCodexAppServerTranscript", () => {
});
expect(result.assistantMirrorIdentitiesOwned).toEqual(["turn-1:assistant"]);
await expect(fs.readFile(sessionFile, "utf8")).rejects.toHaveProperty("code", "ENOENT");
expect(await readMirrorMessages(target)).toEqual([]);
});
it("migrates small linear transcripts before mirroring", async () => {
const sessionFile = await createTempSessionFile();
await fs.writeFile(
sessionFile,
[
JSON.stringify({
type: "session",
version: 3,
id: "linear-codex-session",
timestamp: new Date().toISOString(),
cwd: process.cwd(),
}),
JSON.stringify({
type: "message",
id: "legacy-user",
timestamp: new Date().toISOString(),
message: { role: "user", content: "legacy user" },
}),
].join("\n") + "\n",
"utf8",
);
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
messages: [
it("leaves the assistant unowned when transcript persistence fails", async () => {
const root = await makeRoot("openclaw-codex-transcript-failure-");
const assistantMessage = attachCodexMirrorIdentity(
makeAgentAssistantMessage({
content: [{ type: "text", text: "mirrored assistant" }],
content: [{ type: "text", text: "needs fallback persistence" }],
timestamp: Date.now(),
}),
],
idempotencyScope: "scope-1",
"turn-1:assistant",
);
const assistantTranscriptOwned = await mirrorTranscriptBestEffort({
params: {
sessionId: "session-1",
suppressNextUserMessagePersistence: true,
} as Parameters<typeof mirrorTranscriptBestEffort>[0]["params"],
result: {
messagesSnapshot: [assistantMessage],
} as Parameters<typeof mirrorTranscriptBestEffort>[0]["result"],
notifyUserMessagePersisted: () => undefined,
cwd: root,
threadId: "thread-1",
turnId: "turn-1",
});
const records = (await fs.readFile(sessionFile, "utf8"))
.trim()
.split("\n")
.map(
(line) =>
JSON.parse(line) as {
type?: string;
id?: string;
parentId?: string | null;
message?: { role?: string };
},
)
.filter((record) => record.type === "message");
expect(records[0]?.id).toBe("legacy-user");
expect(records[0]?.parentId).toBeNull();
expect(records[1]?.parentId).toBe("legacy-user");
expect(assistantTranscriptOwned).toBe(false);
});
// Helpers for the identity-based regression tests below.
//
// The mirror dedupe key is now `${idempotencyScope}:${identity}`, where
// `identity` is either an explicit `attachCodexMirrorIdentity` tag (the
// production path; event-projector emits `${turnId}:${kind}`) or the
// role/content fingerprint fallback (legacy callers).
type FileMessage = {
type?: string;
message?: { role?: string; content?: Array<{ text?: string }> };
};
function readFileMessages(raw: string): Array<{ role?: string; text?: string }> {
return parseJsonLines<FileMessage>(raw)
.filter((record) => record.type === "message")
.map((record) => ({
role: record.message?.role,
text: record.message?.content?.[0]?.text,
}));
}
// Regression for #77012 (within-turn snapshot reordering). When mirror is
// invoked twice under the same scope/turn but the second snapshot inserts
// a reasoning record between the user prompt and the assistant reply,
// every assistant-role record after the inserted slot shifts. With the
// previous `:role:index` key, the second call's reasoning record collided
// with the first call's assistant key (both `:assistant:1`) — the
// legitimately-new reasoning entry was silently dropped, and the
// assistant content was re-appended under `:assistant:2`, producing a
// duplicate assistant entry. The identity-based key (event-projector
// tags `${turnId}:reasoning` and `${turnId}:assistant`) makes each kind
// its own dedupe slot.
it("dedupes mirrored messages despite snapshot positional shifts", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-shift-");
const userMessage = attachCodexMirrorIdentity(
makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
@@ -1141,9 +1055,7 @@ describe("mirrorCodexAppServerTranscript", () => {
);
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage, assistantMessage],
idempotencyScope: "codex-app-server:thread-X",
});
@@ -1155,30 +1067,22 @@ describe("mirrorCodexAppServerTranscript", () => {
"turn-1:reasoning",
);
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage, reasoningMessage, assistantMessage],
idempotencyScope: "codex-app-server:thread-X",
});
const messageTexts = readFileMessages(await fs.readFile(sessionFile, "utf8")).map(
(m) => m.text,
);
expect(messageTexts).toEqual(["hello", "hi there", "[Codex reasoning] thinking"]);
expect((await readMirrorMessages(target)).map((m) => m.text)).toEqual([
"hello",
"hi there",
"[Codex reasoning] thinking",
]);
});
// Two distinct turns where the user types the same thing must not collapse:
// each entry carries its own `${turnId}:${kind}` identity so the dedupe
// key differs even when role+content match. (Prior content-fingerprint-only
// designs would have collapsed the second user turn here.)
it("keeps repeated same-content turns distinct", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-repeat-");
const userTurn1 = attachCodexMirrorIdentity(
makeAgentUserMessage({
content: [{ type: "text", text: "yes" }],
timestamp: Date.now(),
}),
makeAgentUserMessage({ content: [{ type: "text", text: "yes" }], timestamp: Date.now() }),
"turn-1:prompt",
);
const assistantTurn1 = attachCodexMirrorIdentity(
@@ -1189,10 +1093,7 @@ describe("mirrorCodexAppServerTranscript", () => {
"turn-1:assistant",
);
const userTurn2 = attachCodexMirrorIdentity(
makeAgentUserMessage({
content: [{ type: "text", text: "yes" }],
timestamp: Date.now() + 2,
}),
makeAgentUserMessage({ content: [{ type: "text", text: "yes" }], timestamp: Date.now() + 2 }),
"turn-2:prompt",
);
const assistantTurn2 = attachCodexMirrorIdentity(
@@ -1204,21 +1105,17 @@ describe("mirrorCodexAppServerTranscript", () => {
);
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userTurn1, assistantTurn1],
idempotencyScope: "codex-app-server:thread-X",
});
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userTurn2, assistantTurn2],
idempotencyScope: "codex-app-server:thread-X",
});
expect(readFileMessages(await fs.readFile(sessionFile, "utf8"))).toEqual([
expect(await readMirrorMessages(target)).toEqual([
{ role: "user", text: "yes" },
{ role: "assistant", text: "ok 1" },
{ role: "user", text: "yes" },
@@ -1226,19 +1123,10 @@ describe("mirrorCodexAppServerTranscript", () => {
]);
});
// Cross-turn re-emit: an entry first written under turn 1 may be re-emitted
// as part of a later turn's snapshot (e.g. a context-engine flow that
// bundles prior history). Because every entry carries its own original
// `${turnId}:${kind}` identity, the re-emitted entries collide with their
// existing on-disk keys and become true no-ops — instead of being
// appended again on a sibling branch (the on-disk symptom in #77012).
it("dedupes prior-turn entries re-emitted into a later turn's snapshot", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-reemit-");
const userTurn1 = attachCodexMirrorIdentity(
makeAgentUserMessage({
content: [{ type: "text", text: "msg1" }],
timestamp: Date.now(),
}),
makeAgentUserMessage({ content: [{ type: "text", text: "msg1" }], timestamp: Date.now() }),
"turn-1:prompt",
);
const assistantTurn1 = attachCodexMirrorIdentity(
@@ -1249,9 +1137,7 @@ describe("mirrorCodexAppServerTranscript", () => {
"turn-1:assistant",
);
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userTurn1, assistantTurn1],
idempotencyScope: "codex-app-server:thread-X",
});
@@ -1270,17 +1156,13 @@ describe("mirrorCodexAppServerTranscript", () => {
}),
"turn-2:assistant",
);
// Buggy upstream: snapshot for turn 2 also includes the just-completed
// turn 1's entries (with their original identities preserved).
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userTurn1, assistantTurn1, userTurn2, assistantTurn2],
idempotencyScope: "codex-app-server:thread-X",
});
expect(readFileMessages(await fs.readFile(sessionFile, "utf8"))).toEqual([
expect(await readMirrorMessages(target)).toEqual([
{ role: "user", text: "msg1" },
{ role: "assistant", text: "reply1" },
{ role: "user", text: "msg2" },
@@ -1288,12 +1170,8 @@ describe("mirrorCodexAppServerTranscript", () => {
]);
});
// Backward-compat: callers that do not tag messages with a mirror identity
// (e.g. third-party harnesses or tests routed through the legacy path)
// still get the role/content fingerprint key. Distinct turns are then
// distinguished by the caller's idempotency scope.
it("falls back to the role+content fingerprint when no identity is attached", async () => {
const sessionFile = await createTempSessionFile();
it("uses the role+content fingerprint when no identity is attached", async () => {
const target = await createSqliteMirrorTarget("openclaw-codex-mirror-fingerprint-");
const userMessage = makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
@@ -1304,14 +1182,12 @@ describe("mirrorCodexAppServerTranscript", () => {
});
await mirrorCodexAppServerTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage, assistantMessage],
idempotencyScope: "scope-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain(`"idempotencyKey":"scope-1:user:${expectedFingerprint(userMessage)}"`);
expect(raw).toContain(
`"idempotencyKey":"scope-1:assistant:${expectedFingerprint(assistantMessage)}"`,
extensions/codex/src/app-server/transcript-mirror.ts
+14 -9
View File
@@ -278,7 +278,7 @@ export function projectBoundedCodexThreadHistory(params: {
export async function importCodexThreadHistoryToTranscript(params: {
thread: CodexThread;
throughTurnId: string | null;
sessionFile: string;
storePath: string;
sessionId: string;
sessionKey: string;
agentId?: string;
@@ -294,7 +294,7 @@ export async function importCodexThreadHistoryToTranscript(params: {
});
if (projection.transcriptMessages.length > 0) {
await mirrorCodexAppServerTranscript({
sessionFile: params.sessionFile,
storePath: params.storePath,
sessionId: params.sessionId,
sessionKey: params.sessionKey,
...(params.agentId ? { agentId: params.agentId } : {}),
@@ -413,10 +413,10 @@ export async function mirrorTranscriptBestEffort(params: {
turnId: params.turnId,
});
const mirrorResult = await mirrorCodexAppServerTranscript({
sessionFile: params.params.sessionFile,
agentId: params.agentId,
sessionKey: params.sessionKey,
sessionId: params.params.sessionId,
storePath: params.params.sessionTarget?.storePath,
cwd: params.cwd,
messages,
// Scope is thread-stable. Each entry in `messagesSnapshot` is tagged
@@ -506,10 +506,10 @@ export async function mirrorPromptAtTurnStartBestEffort(params: {
`${params.turnId}:prompt`,
);
const mirrorResult = await mirrorCodexAppServerTranscript({
sessionFile: params.params.sessionFile,
agentId: params.agentId,
sessionKey: params.sessionKey,
sessionId: params.params.sessionId,
storePath: params.params.sessionTarget?.storePath,
cwd: params.cwd,
messages: [userPromptMessage],
idempotencyScope: `codex-app-server:${params.threadId}`,
@@ -578,11 +578,11 @@ function buildMirrorDedupeIdentity(message: MirroredAgentMessage): string {
}
export async function mirrorCodexAppServerTranscript(params: {
sessionFile: string;
sessionId: string;
cwd?: string;
sessionKey?: string;
agentId?: string;
storePath?: string;
messages: AgentMessage[];
idempotencyScope?: string;
config?: SessionTranscriptWriteLockParams["config"];
@@ -699,11 +699,11 @@ export async function mirrorCodexAppServerTranscript(params: {
await publishSessionTranscriptUpdateByIdentity({
...transcriptTarget,
update: {
...(params.sessionKey ? { sessionKey: params.sessionKey } : {}),
...(params.agentId ? { agentId: params.agentId } : {}),
message: update.message,
messageId: update.messageId,
messageSeq: update.messageSeq,
sessionKey: transcriptTarget.sessionKey,
},
});
} catch (error) {
@@ -720,15 +720,20 @@ export async function mirrorCodexAppServerTranscript(params: {
function resolveCodexMirrorTranscriptTarget(params: {
agentId?: string;
sessionFile: string;
sessionId: string;
sessionKey?: string;
storePath?: string;
}): SessionTranscriptTargetParams {
const sessionKey = params.sessionKey?.trim();
const storePath = params.storePath?.trim();
if (!sessionKey || !storePath) {
throw new Error("Codex transcript mirror requires a runtime session identity");
}
return {
...(params.agentId ? { agentId: params.agentId } : {}),
sessionFile: params.sessionFile,
sessionId: params.sessionId,
sessionKey: params.sessionKey ?? "",
sessionKey,
storePath,
};
}
extensions/codex/src/commands.test.ts
+17 -5
View File
@@ -10,7 +10,7 @@ import {
} from "openclaw/plugin-sdk/agent-runtime";
import { MODEL_SELECTION_LOCKED_MESSAGE } from "openclaw/plugin-sdk/model-session-runtime";
import type { PluginCommandContext, PluginCommandResult } from "openclaw/plugin-sdk/plugin-entry";
import { saveSessionStore } from "openclaw/plugin-sdk/session-store-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { CODEX_CONTROL_METHODS } from "./app-server/capabilities.js";
import type { CodexComputerUseStatus } from "./app-server/computer-use.js";
@@ -170,8 +170,10 @@ async function createLockedSessionContextOverrides(
sessionKey = "agent:main:test:locked",
): Promise<Pick<PluginCommandContext, "config" | "sessionKey">> {
const storePath = path.join(tempDir, "locked-sessions.json");
await saveSessionStore(storePath, {
[sessionKey]: {
await upsertSessionEntry({
storePath,
sessionKey,
entry: {
sessionId: "session-1",
updatedAt: Date.now(),
agentHarnessId: "codex",
@@ -545,8 +547,10 @@ describe("codex command", () => {
},
{ threadId: "thread-old", cwd: "/old" },
);
await saveSessionStore(storePath, {
[sessionKey]: { sessionId: "session-new", updatedAt: Date.now() },
await upsertSessionEntry({
storePath,
sessionKey,
entry: { sessionId: "session-new", updatedAt: Date.now() },
});
const codexControlRequest = vi.fn(async () =>
createThreadResumeResponse({ threadId: "thread-new" }),
@@ -615,10 +619,18 @@ describe("codex command", () => {
const codexControlRequest = vi.fn(async () =>
createThreadResumeResponse({ threadId: "thread-123" }),
);
const storePath = path.join(tempDir, "worker-sessions.json");
await upsertSessionEntry({
agentId: "worker",
storePath,
sessionKey: "agent:worker:session-1",
entry: { sessionId: "session-1", updatedAt: Date.now() },
});
await handleCodexCommand(
createContext("resume thread-123", undefined, {
sessionKey: "agent:worker:session-1",
config: { session: { store: storePath } },
}),
{ deps: createDeps({ codexControlRequest }) },
);
extensions/codex/src/migration/session-binding-sidecars.ts
+36 -63
View File
@@ -10,8 +10,6 @@ import { withFileLock, type FileLockOptions } from "openclaw/plugin-sdk/file-loc
import type { PluginStateKeyedStore } from "openclaw/plugin-sdk/plugin-state-runtime";
import type { PluginDoctorStateMigration } from "openclaw/plugin-sdk/runtime-doctor";
import {
listSessionEntries,
resolveSessionFilePath,
resolveStorePath,
updateSessionStoreEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
@@ -157,9 +155,7 @@ async function collectLegacyBindingSources(
return source;
};
for (const surface of surfaces) {
const sidecars = surface.scan
? walkSidecars(surface.root)
: iterateIndexedSidecars(surface, params);
const sidecars = surface.scan ? walkSidecars(surface.root) : iterateIndexedSidecars(surface);
for await (const sidecarPath of sidecars) {
const source = await addSource(sidecarPath, surface);
if (options.firstOnly) {
@@ -196,33 +192,24 @@ async function readLegacySessionIndex(
if (!isRecord(raw)) {
return { failure: `session index ${storePath} has invalid entries` };
}
let normalizedEntries: ReturnType<typeof listSessionEntries>;
try {
normalizedEntries = listSessionEntries({ storePath, hydrateSkillPromptRefs: false });
} catch {
return { failure: `session index ${storePath} could not be normalized` };
}
const normalizedByKey = new Map(
normalizedEntries.map(({ sessionKey, entry }) => [sessionKey, entry] as const),
);
const entries: Array<{ sessionKey: string; entry: LegacySessionIndexEntry }> = [];
for (const [sessionKey, value] of Object.entries(raw)) {
if (!isRecord(value)) {
return { failure: `session index ${storePath} has invalid entries` };
}
// Metadata-only rows have no transcript identity and therefore cannot own
// a binding sidecar. Runtime normalization preserves this shipped shape.
// a binding sidecar. This legacy reader parses the raw file directly:
// post-flip listSessionEntries reads SQLite, so main's normalized
// cross-check would consult the wrong store for import inputs.
if (value.sessionId === undefined) {
continue;
}
const rawSessionId = typeof value.sessionId === "string" ? value.sessionId.trim() : "";
const sessionId = normalizedByKey.get(sessionKey)?.sessionId?.trim() ?? "";
const sessionId = typeof value.sessionId === "string" ? value.sessionId.trim() : "";
const sessionFile = value.sessionFile;
const lifecycleRevision = value.lifecycleRevision;
const agentHarnessId = value.agentHarnessId;
if (
!sessionId ||
sessionId !== rawSessionId ||
(sessionFile !== undefined && typeof sessionFile !== "string") ||
(lifecycleRevision !== undefined && typeof lifecycleRevision !== "string") ||
(agentHarnessId !== undefined && typeof agentHarnessId !== "string")
@@ -242,30 +229,24 @@ async function readLegacySessionIndex(
return { entries };
}
async function* iterateIndexedSidecars(
surface: SessionSurface,
params: MigrationEnvironment,
): AsyncGenerator<string> {
async function* iterateIndexedSidecars(surface: SessionSurface): AsyncGenerator<string> {
for (const storePath of surface.storePaths) {
const index = await readLegacySessionIndex(storePath);
if ("failure" in index) {
continue;
}
for (const { sessionKey, entry } of index.entries) {
const agentId = resolveLegacyBindingOwnerAgentId({
sessionKey,
config: params.config,
storeAgentIds: surface.agentIds,
});
let transcriptPath: string;
try {
transcriptPath = resolveSessionFilePath(entry.sessionId, entry, {
sessionsDir: path.dirname(storePath),
agentId,
});
} catch {
for (const { entry } of index.entries) {
// Legacy sidecars sit beside file-era transcripts; sqlite-marker entries
// never had sidecars, so the raw locator is skipped for them.
const sessionFile = entry.sessionFile?.trim() ?? "";
if (sessionFile.startsWith("sqlite:")) {
continue;
}
const transcriptPath = resolveLegacySessionFileLocator(
path.dirname(storePath),
entry,
entry.sessionId,
);
const sidecarPath = `${transcriptPath}${LEGACY_BINDING_SUFFIX}`;
if (await isRegularFile(sidecarPath)) {
yield sidecarPath;
@@ -330,18 +311,16 @@ async function collectBindingOwners(
config: params.config,
storeAgentIds: storeAgentIds.get(storePath),
});
let effectiveTranscriptPath: string;
let legacyTranscriptPath: string;
let canonicalLegacyTranscriptPath: string;
try {
effectiveTranscriptPath = resolveSessionFilePath(sessionId, entry, {
sessionsDir,
agentId,
});
legacyTranscriptPath = resolveLegacySessionFileLocator(sessionsDir, entry, sessionId);
canonicalLegacyTranscriptPath = await canonicalizePath(legacyTranscriptPath);
} catch {
failures.push(`session index ${storePath} has an invalid locator for ${sessionKey}`);
continue;
}
const transcriptPath = await canonicalizePath(effectiveTranscriptPath);
if (!sourcePaths.has(transcriptPath)) {
if (!sourcePaths.has(canonicalLegacyTranscriptPath)) {
continue;
}
const owner: LegacyBindingOwner = {
@@ -349,11 +328,12 @@ async function collectBindingOwners(
sessionId,
sessionKey,
storePath,
transcriptPath: effectiveTranscriptPath,
transcriptPath: legacyTranscriptPath,
...(entry.lifecycleRevision ? { lifecycleRevision: entry.lifecycleRevision } : {}),
...(entry.agentHarnessId?.trim() ? { agentHarnessId: entry.agentHarnessId.trim() } : {}),
};
const candidates = owners.get(transcriptPath) ?? new Map<string, LegacyBindingOwner>();
const candidates =
owners.get(canonicalLegacyTranscriptPath) ?? new Map<string, LegacyBindingOwner>();
const ownerKey = `${agentId}\0${sessionId}\0${sessionKey}\0${canonicalStorePath}`;
const configuredStorePath = resolveStorePath(params.config.session?.store, {
agentId,
@@ -364,7 +344,7 @@ async function collectBindingOwners(
if (!candidates.has(ownerKey) || storePath === configuredStorePath) {
candidates.set(ownerKey, owner);
}
owners.set(transcriptPath, candidates);
owners.set(canonicalLegacyTranscriptPath, candidates);
}
}
return {
@@ -373,6 +353,17 @@ async function collectBindingOwners(
};
}
// Doctor-only locator for retired file-backed session indexes. Active runtime
// never resolves these paths; migration needs them only to find old sidecars.
function resolveLegacySessionFileLocator(
sessionsDir: string,
entry: { sessionFile?: string },
sessionId: string,
): string {
const sessionFile = entry.sessionFile?.trim();
return path.resolve(sessionsDir, sessionFile || `${sessionId}.jsonl`);
}
function resolveLegacyBindingOwnerAgentId(params: {
sessionKey: string;
config: MigrationEnvironment["config"];
@@ -582,10 +573,8 @@ async function recordSessionOwner(owner: LegacyBindingOwner): Promise<string | u
skipMaintenance: true,
requireWriteSuccess: true,
update: (entry) => {
const transcriptPath = resolveOwnerTranscriptPath(owner, entry);
if (
entry.sessionId.trim() !== owner.sessionId ||
transcriptPath !== owner.transcriptPath ||
entry.lifecycleRevision !== owner.lifecycleRevision
) {
return null;
@@ -607,10 +596,8 @@ async function recordSessionOwner(owner: LegacyBindingOwner): Promise<string | u
? `its session is owned by agent harness ${observedForeignHarness}`
: "its session owner changed before Codex ownership could be recorded";
}
const transcriptPath = resolveOwnerTranscriptPath(owner, updated);
if (
updated.sessionId.trim() !== owner.sessionId ||
transcriptPath !== owner.transcriptPath ||
updated.lifecycleRevision !== owner.lifecycleRevision
) {
return "its session owner changed before Codex ownership could be recorded";
@@ -623,20 +610,6 @@ async function recordSessionOwner(owner: LegacyBindingOwner): Promise<string | u
: "Codex harness ownership could not be recorded on its session";
}
function resolveOwnerTranscriptPath(
owner: LegacyBindingOwner,
entry: { sessionFile?: string; sessionId: string },
): string | undefined {
try {
return resolveSessionFilePath(entry.sessionId, entry, {
sessionsDir: path.dirname(owner.storePath),
agentId: owner.agentId,
});
} catch {
return undefined;
}
}
async function readDirectoryEntries(directory: string) {
try {
return await fs.readdir(directory, { withFileTypes: true });
extensions/codex/src/native-thread-tool.test.ts
-1
View File
@@ -70,7 +70,6 @@ describe("native Codex thread tool", () => {
modelSelectionLocked: params?.modelSelectionLocked,
}),
resolveStorePath: () => path.join(root, "sessions", "sessions.json"),
resolveSessionFilePath: () => sessionFile,
},
},
});
extensions/codex/src/native-thread-tool.ts
+1 -9
View File
@@ -1,7 +1,6 @@
/**
* Owner-only access to native Codex threads stored in the user's Codex home.
*/
import path from "node:path";
import {
jsonResult,
readStringParam,
@@ -131,7 +130,7 @@ function readLimit(value: unknown): number | undefined {
function resolveToolSession(
context: OpenClawPluginToolContext,
runtime: PluginRuntime,
): { sessionId: string; sessionFile: string; modelSelectionLocked: boolean } | undefined {
): { sessionId: string; modelSelectionLocked: boolean } | undefined {
const sessionKey = context.sessionKey?.trim();
if (!sessionKey) {
return undefined;
@@ -145,15 +144,8 @@ function resolveToolSession(
if (!sessionId) {
return undefined;
}
const storePath = runtime.agent.session.resolveStorePath(undefined, {
agentId: context.agentId,
});
return {
sessionId,
sessionFile: runtime.agent.session.resolveSessionFilePath(sessionId, entry, {
agentId: context.agentId,
sessionsDir: path.dirname(storePath),
}),
modelSelectionLocked: isModelSelectionLocked(entry),
};
}
extensions/codex/src/session-catalog.test.ts
+3 -2
View File
@@ -5,6 +5,7 @@ import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import type { GatewayRequestHandlerOptions } from "openclaw/plugin-sdk/gateway-runtime";
import type { OpenClawPluginApi } from "openclaw/plugin-sdk/plugin-entry";
import type { PluginRuntime } from "openclaw/plugin-sdk/plugin-runtime";
import { resolveStorePath } from "openclaw/plugin-sdk/session-store-runtime";
import { beforeEach, describe, expect, it, vi } from "vitest";
import type { CodexThread } from "./app-server/protocol.js";
import { sessionBindingIdentity } from "./app-server/session-binding.js";
@@ -1400,7 +1401,7 @@ describe("Codex supervision actions", () => {
);
expect(transcriptMirrorMocks.importCodexThreadHistoryToTranscript).toHaveBeenCalledWith({
thread: sourceThread,
sessionFile: "/tmp/openclaw-session-1.jsonl",
storePath: resolveStorePath(undefined, { agentId: "main" }),
sessionId: "openclaw-session-1",
sessionKey: first.sessionKey,
agentId: "main",
@@ -1697,7 +1698,7 @@ describe("Codex supervision actions", () => {
expect(entries[0]?.entry.initializationPending).toBeUndefined();
expect(transcriptMirrorMocks.importCodexThreadHistoryToTranscript).toHaveBeenCalledWith(
expect.objectContaining({
sessionFile: `/tmp/${sessionId}.jsonl`,
storePath: resolveStorePath(undefined, { agentId: "main" }),
sessionId,
sessionKey,
}),
extensions/codex/src/session-catalog.ts
+7 -5
View File
@@ -17,6 +17,7 @@ import type {
} from "openclaw/plugin-sdk/plugin-entry";
import type { PluginRuntime } from "openclaw/plugin-sdk/plugin-runtime";
import { parseAgentSessionKey } from "openclaw/plugin-sdk/routing";
import { resolveStorePath } from "openclaw/plugin-sdk/session-store-runtime";
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
import { truncateUtf16Safe } from "openclaw/plugin-sdk/text-utility-runtime";
import { CODEX_CONTROL_METHODS } from "./app-server/capabilities.js";
@@ -1560,14 +1561,15 @@ async function createOrReuseAdoptedSession(params: {
sessionKey: entry.key,
config: params.config,
});
const sessionFile = entry.entry.sessionFile?.trim();
if (!sessionFile) {
throw new Error("Codex supervision session creation did not produce a transcript file");
}
// Post-flip the mirror targets SQLite rows; resolve the agent's store
// path instead of trusting the legacy sessionFile locator marker.
const storePath = resolveStorePath(params.config.session?.store, {
agentId: entry.agentId,
});
await importCodexThreadHistoryToTranscript({
thread: params.sourceThread,
throughTurnId: pendingLastTurnId ?? null,
sessionFile,
storePath,
sessionId: entry.sessionId,
sessionKey: entry.key,
agentId: entry.agentId,
extensions/copilot/src/attempt.test.ts
+24 -7
View File
@@ -271,6 +271,12 @@ function makeParams(
runId: "run-1",
sessionFile: "session.json",
sessionId: "session-1",
sessionKey: "agent:main:session-1",
sessionTarget: {
sessionId: "session-1",
sessionKey: "agent:main:session-1",
storePath: "openclaw-agent.sqlite",
},
timeoutMs: 5000,
workspaceDir: "C:\\workspace",
...overrides,
@@ -1280,7 +1286,7 @@ describe("runCopilotAttempt", () => {
modelId: "gpt-4o",
modelProvider: "github-copilot",
sessionId: "session-1",
sessionKey: undefined,
sessionKey: "agent:main:session-1",
workspaceDir: "C:\\workspace",
}),
);
@@ -2638,7 +2644,7 @@ describe("runCopilotAttempt", () => {
dualWriteMock.dualWriteCopilotTranscriptBestEffort.mockResolvedValue(undefined);
});
it("invokes dual-write mirror with sessionFile and scoped idempotencyScope when sessionFile is set", async () => {
it("invokes dual-write mirror with runtime identity and scoped idempotencyScope", async () => {
dualWriteMock.dualWriteCopilotTranscriptBestEffort.mockClear();
const sdk = makeFakeSdk({
onCreateSession: (session) => {
@@ -2647,17 +2653,28 @@ describe("runCopilotAttempt", () => {
});
const pool = makeFakePool(sdk);
await runCopilotAttempt(makeParams(), { pool });
await runCopilotAttempt(
makeParams({
sessionTarget: {
sessionId: "session-1",
sessionKey: "agent:main:session-1",
storePath: "sessions.json",
},
}),
{ pool },
);
expect(dualWriteMock.dualWriteCopilotTranscriptBestEffort).toHaveBeenCalledTimes(1);
const args = dualWriteMock.dualWriteCopilotTranscriptBestEffort.mock.calls[0]?.[0] as {
sessionFile: string;
sessionId: string;
sessionKey: string;
storePath?: string;
messages: Array<{ role: string }>;
idempotencyScope?: string;
};
expect(args.sessionFile).toBe("session.json");
expect(args.sessionId).toBe("session-1");
expect(args.sessionKey).toBe("agent:main:session-1");
expect(args.storePath).toBe("sessions.json");
expect(args.idempotencyScope).toBe("copilot:sess-1");
expect(args.messages.length).toBeGreaterThan(0);
const roles = args.messages.map((m) => m.role);
@@ -2665,7 +2682,7 @@ describe("runCopilotAttempt", () => {
expect(roles).toContain("assistant");
});
it("does not invoke dual-write mirror when sessionFile is absent", async () => {
it("does not invoke dual-write mirror when runtime identity is absent", async () => {
dualWriteMock.dualWriteCopilotTranscriptBestEffort.mockClear();
const sdk = makeFakeSdk({
onCreateSession: (session) => {
@@ -2674,7 +2691,7 @@ describe("runCopilotAttempt", () => {
});
const pool = makeFakePool(sdk);
const params = makeParams() as unknown as Record<string, unknown>;
delete params.sessionFile;
delete params.sessionTarget;
await runCopilotAttempt(params as never, { pool });
extensions/copilot/src/attempt.ts
+14 -12
View File
@@ -1125,18 +1125,20 @@ export async function runCopilotAttempt(
...(taggedLastAssistant ? [taggedLastAssistant] : []),
];
// Best-effort dual-write: mirror this attempt's full message snapshot
// (user/assistant/toolResult) into the OpenClaw audit transcript at
// params.sessionFile, alongside the SDK's own session storage. The
// OpenClaw shell (attempt-execution.ts) writes only the user prompt
// and terminal assistant text; mirroring here captures intermediate
// tool calls/results for full audit/replay parity with the codex
// extension. Identity-tagged so re-emits dedupe. Errors are
// swallowed so a mirror failure cannot break the attempt.
const sessionFileForMirror = readString(input.sessionFile);
// Best-effort dual-write mirrors this attempt's full message snapshot into
// OpenClaw's runtime transcript store. The Copilot SDK may still maintain
// its own private files; OpenClaw-side audit state is addressed only by
// session identity so missing identity cannot silently recreate JSONL state.
const openClawSessionIdForMirror = readString(input.sessionId);
const openClawSessionKeyForMirror = readString((input as { sessionKey?: unknown }).sessionKey);
const openClawStorePathForMirror = readString(input.sessionTarget?.storePath);
const mirrorScopeSessionId = sessionIdUsed ?? openClawSessionIdForMirror;
if (sessionFileForMirror && openClawSessionIdForMirror && messagesSnapshot.length > 0) {
if (
openClawSessionIdForMirror &&
openClawSessionKeyForMirror &&
openClawStorePathForMirror &&
messagesSnapshot.length > 0
) {
const taggedMessages = messagesSnapshot.map((message, index) => {
if (
message.role !== "user" &&
@@ -1161,10 +1163,10 @@ export async function runCopilotAttempt(
return attachCopilotMirrorIdentity(message, `${identityScope}:${message.role}:${index}`);
});
await dualWriteCopilotTranscriptBestEffort({
sessionFile: sessionFileForMirror,
sessionId: openClawSessionIdForMirror,
sessionKey: readString((input as { sessionKey?: unknown }).sessionKey),
sessionKey: openClawSessionKeyForMirror,
agentId: readString(input.agentId),
storePath: openClawStorePathForMirror,
messages: taggedMessages,
idempotencyScope: mirrorScopeSessionId ? `copilot:${mirrorScopeSessionId}` : undefined,
config: (input as { config?: unknown }).config as never,
extensions/copilot/src/dual-write-transcripts.test.ts
+127 -123
View File
@@ -9,6 +9,8 @@ import {
resetGlobalHookRunner,
} from "openclaw/plugin-sdk/hook-runtime";
import { createMockPluginRegistry } from "openclaw/plugin-sdk/plugin-test-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { readSessionTranscriptEvents } from "openclaw/plugin-sdk/session-transcript-runtime";
import {
castAgentMessage,
makeAgentAssistantMessage,
@@ -37,31 +39,87 @@ afterEach(async () => {
}
});
async function createTempSessionFile() {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-copilot-mirror-"));
tempDirs.push(dir);
return path.join(dir, "session.jsonl");
}
async function makeRoot(prefix: string): Promise<string> {
const root = await fs.mkdtemp(path.join(os.tmpdir(), prefix));
tempDirs.push(root);
return root;
}
function parseJsonLines<T>(raw: string): T[] {
const records: T[] = [];
for (const line of raw.trim().split("\n")) {
if (line.length > 0) {
records.push(JSON.parse(line) as T);
function readEventMessages(events: unknown[]): Array<{ role?: string; text?: string }> {
return events
.map((event) =>
event && typeof event === "object" ? (event as { message?: unknown }).message : undefined,
)
.filter((message): message is { role?: string; content?: unknown } =>
Boolean(message && typeof message === "object"),
)
.map((message) => {
const content = Array.isArray(message.content)
? message.content.find((part): part is { text: string } =>
Boolean(part && typeof part === "object" && typeof part.text === "string"),
)?.text
: typeof message.content === "string"
? message.content
: undefined;
return { role: message.role, text: content };
});
}
async function createSqliteMirrorTarget(prefix: string, options: { sessionId?: string } = {}) {
const root = await makeRoot(prefix);
const agentId = "main";
const sessionId = options.sessionId ?? "session-1";
const sessionKey = `agent:${agentId}:${sessionId}`;
const storePath = path.join(root, "openclaw-agent.sqlite");
await upsertSessionEntry({
agentId,
sessionKey,
storePath,
entry: {
sessionFile: `sqlite:${agentId}:${sessionId}:${storePath}`,
sessionId,
updatedAt: 1,
},
});
return {
agentId,
sessionId,
sessionKey,
storePath,
bogusSessionFile: path.join(root, "should-not-be-created.jsonl"),
};
}
return records;
async function readMirrorEvents(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<unknown[]> {
return await readSessionTranscriptEvents(target);
}
async function readMirrorRaw(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<string> {
return (await readMirrorEvents(target)).map((event) => JSON.stringify(event)).join("\n");
}
async function readMirrorMessages(target: {
agentId: string;
sessionId: string;
sessionKey: string;
storePath: string;
}): Promise<Array<{ role?: string; text?: string }>> {
return readEventMessages(await readMirrorEvents(target));
}
describe("mirrorCopilotTranscript", () => {
it("mirrors user, assistant, and tool result messages into the OpenClaw transcript", async () => {
const sessionFile = await createTempSessionFile();
it("mirrors user, assistant, and tool result messages by SQLite identity", async () => {
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-basic-");
const userMessage = makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
@@ -74,25 +132,17 @@ describe("mirrorCopilotTranscript", () => {
role: "toolResult",
toolCallId: "call-1",
toolName: "read",
content: [
{
type: "toolResult",
toolCallId: "call-1",
content: "read output",
},
],
content: [{ type: "toolResult", toolCallId: "call-1", content: "read output" }],
timestamp: Date.now() + 2,
}) as MirroredAgentMessage;
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage, assistantMessage, toolResultMessage],
idempotencyScope: "copilot:session-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"role":"user"');
expect(raw).toContain('"role":"assistant"');
expect(raw).toContain('"role":"toolResult"');
@@ -106,10 +156,14 @@ describe("mirrorCopilotTranscript", () => {
expect(raw).toContain(
`"idempotencyKey":"copilot:session-1:toolResult:${expectedFingerprint(toolResultMessage)}"`,
);
await expect(fs.readFile(target.bogusSessionFile, "utf8")).rejects.toHaveProperty(
"code",
"ENOENT",
);
});
it("preserves gateway user-turn identity across Copilot transcript mirroring", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-user-identity-");
const userMessage = castAgentMessage({
...makeAgentUserMessage({
content: [{ type: "text", text: "client prompt" }],
@@ -119,51 +173,40 @@ describe("mirrorCopilotTranscript", () => {
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage],
idempotencyScope: "copilot:session-1",
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [userMessage],
idempotencyScope: "copilot:session-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"idempotencyKey":"client-run:user"');
expect(raw).not.toContain('"idempotencyKey":"copilot:session-1:user:');
const records = parseJsonLines<{ message?: { role?: string } }>(raw);
expect(records.filter((record) => record.message?.role === "user")).toHaveLength(1);
expect(
(await readMirrorMessages(target)).filter((message) => message.role === "user"),
).toHaveLength(1);
});
it("creates the transcript directory on first mirror", async () => {
const root = await makeRoot("openclaw-copilot-mirror-missing-dir-");
const sessionFile = path.join(root, "nested", "sessions", "session.jsonl");
await mirrorCopilotTranscript({
sessionFile,
it("rejects mirror writes without a runtime session identity", async () => {
await expect(
mirrorCopilotTranscript({
sessionId: "session-1",
sessionKey: "session-1",
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "first mirror" }],
content: [{ type: "text", text: "no identity" }],
timestamp: Date.now(),
}),
],
idempotencyScope: "copilot:session-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
expect(raw).toContain('"role":"assistant"');
expect(raw).toContain('"content":[{"type":"text","text":"first mirror"}]');
}),
).rejects.toThrow("runtime session identity");
});
it("deduplicates re-emits by idempotency scope", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-dedupe-");
const messages = [
makeAgentUserMessage({
content: [{ type: "text", text: "hello" }],
@@ -176,27 +219,17 @@ describe("mirrorCopilotTranscript", () => {
] as const;
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [...messages],
idempotencyScope: "copilot:session-1",
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [...messages],
idempotencyScope: "copilot:session-1",
});
const records = parseJsonLines<{ type?: string; message?: { role?: string } }>(
await fs.readFile(sessionFile, "utf8"),
);
// First "header" record may or may not appear depending on migration.
// What matters is that the second mirror call adds zero new messages.
const messageRecords = records.filter((r) => r.message?.role !== undefined);
expect(messageRecords).toHaveLength(2);
expect((await readMirrorMessages(target)).filter((message) => message.role)).toHaveLength(2);
});
it("runs before_message_write before appending mirrored messages", async () => {
@@ -213,21 +246,19 @@ describe("mirrorCopilotTranscript", () => {
},
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-hook-");
const sourceMessage = makeAgentAssistantMessage({
content: [{ type: "text", text: "hello" }],
timestamp: Date.now(),
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [sourceMessage],
idempotencyScope: "copilot:session-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"content":[{"type":"text","text":"hello [hooked]"}]');
expect(raw).toContain(
`"idempotencyKey":"copilot:session-1:assistant:${expectedFingerprint(sourceMessage)}"`,
@@ -237,18 +268,13 @@ describe("mirrorCopilotTranscript", () => {
it("respects before_message_write blocking decisions", async () => {
initializeGlobalHookRunner(
createMockPluginRegistry([
{
hookName: "before_message_write",
handler: () => ({ block: true }),
},
{ hookName: "before_message_write", handler: () => ({ block: true }) },
]),
);
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-block-");
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "should not persist" }],
@@ -258,43 +284,40 @@ describe("mirrorCopilotTranscript", () => {
idempotencyScope: "copilot:session-1",
});
await expect(fs.readFile(sessionFile, "utf8")).rejects.toHaveProperty("code", "ENOENT");
expect(await readMirrorMessages(target)).toEqual([]);
});
it("is a no-op when no mirrorable messages are present", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-empty-");
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
sessionKey: "session-1",
...target,
messages: [],
idempotencyScope: "copilot:session-1",
});
await expect(fs.readFile(sessionFile, "utf8")).rejects.toHaveProperty("code", "ENOENT");
expect(await readMirrorMessages(target)).toEqual([]);
});
it("uses content fingerprint when no explicit mirror identity is attached", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-fingerprint-");
const message = makeAgentAssistantMessage({
content: [{ type: "text", text: "fp" }],
timestamp: Date.now(),
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
...target,
messages: [message],
idempotencyScope: "scope-fp",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain(`"idempotencyKey":"scope-fp:assistant:${expectedFingerprint(message)}"`);
});
it("uses attached identity instead of content fingerprint when provided", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-identity-");
const baseMessage = makeAgentAssistantMessage({
content: [{ type: "text", text: "explicit" }],
timestamp: Date.now(),
@@ -302,13 +325,12 @@ describe("mirrorCopilotTranscript", () => {
const tagged = attachCopilotMirrorIdentity(baseMessage, "sdk-session-1:assistant:0");
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
...target,
messages: [tagged],
idempotencyScope: "copilot:openclaw-session-1",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain(
'"idempotencyKey":"copilot:openclaw-session-1:sdk-session-1:assistant:0"',
);
@@ -316,11 +338,10 @@ describe("mirrorCopilotTranscript", () => {
});
it("omits idempotencyKey when no idempotencyScope is provided", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-no-scope-");
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
...target,
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "no scope" }],
@@ -329,13 +350,13 @@ describe("mirrorCopilotTranscript", () => {
],
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"content":[{"type":"text","text":"no scope"}]');
expect(raw).not.toContain("idempotencyKey");
});
it("filters out non-mirrorable roles", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-filter-");
const userMessage = makeAgentUserMessage({
content: [{ type: "text", text: "u" }],
timestamp: Date.now(),
@@ -347,19 +368,18 @@ describe("mirrorCopilotTranscript", () => {
});
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
...target,
messages: [userMessage, systemLike],
idempotencyScope: "scope",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"role":"user"');
expect(raw).not.toContain("system note");
});
it("preserves explicit identity across attachCopilotMirrorIdentity overrides", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-override-");
const base = makeAgentAssistantMessage({
content: [{ type: "text", text: "x" }],
timestamp: Date.now(),
@@ -368,13 +388,12 @@ describe("mirrorCopilotTranscript", () => {
const second = attachCopilotMirrorIdentity(first, "id-2");
await mirrorCopilotTranscript({
sessionFile,
sessionId: "session-1",
...target,
messages: [second],
idempotencyScope: "scope",
});
const raw = await fs.readFile(sessionFile, "utf8");
const raw = await readMirrorRaw(target);
expect(raw).toContain('"idempotencyKey":"scope:id-2"');
expect(raw).not.toContain('"idempotencyKey":"scope:id-1"');
});
@@ -382,11 +401,10 @@ describe("mirrorCopilotTranscript", () => {
describe("dualWriteCopilotTranscriptBestEffort", () => {
it("returns normally when mirror succeeds", async () => {
const sessionFile = await createTempSessionFile();
const target = await createSqliteMirrorTarget("openclaw-copilot-mirror-best-effort-");
await expect(
dualWriteCopilotTranscriptBestEffort({
sessionFile,
sessionId: "session-1",
...target,
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "ok" }],
@@ -396,21 +414,16 @@ describe("dualWriteCopilotTranscriptBestEffort", () => {
idempotencyScope: "scope",
}),
).resolves.toBeUndefined();
const raw = await fs.readFile(sessionFile, "utf8");
expect(raw).toContain('"role":"assistant"');
expect(await readMirrorMessages(target)).toContainEqual({ role: "assistant", text: "ok" });
});
it("swallows infrastructure failures and never rejects", async () => {
it("swallows missing runtime identity and does not write JSONL", async () => {
const root = await makeRoot("openclaw-copilot-mirror-invalid-");
const previousStateDir = process.env.OPENCLAW_STATE_DIR;
process.env.OPENCLAW_STATE_DIR = root;
try {
const sessionFile = path.join(root, "agents", "main", "sessions", "session-1.jsonl");
await expect(
dualWriteCopilotTranscriptBestEffort({
agentId: "main",
sessionFile: "",
sessionId: "session-1",
sessionKey: "agent:main:session-1",
messages: [
makeAgentAssistantMessage({
content: [{ type: "text", text: "should-not-throw" }],
@@ -420,15 +433,6 @@ describe("dualWriteCopilotTranscriptBestEffort", () => {
idempotencyScope: "scope",
}),
).resolves.toBeUndefined();
await expect(
fs.access(path.join(root, "agents", "main", "sessions", "session-1.jsonl")),
).rejects.toHaveProperty("code", "ENOENT");
} finally {
if (previousStateDir === undefined) {
delete process.env.OPENCLAW_STATE_DIR;
} else {
process.env.OPENCLAW_STATE_DIR = previousStateDir;
}
}
await expect(fs.access(sessionFile)).rejects.toHaveProperty("code", "ENOENT");
});
});
extensions/copilot/src/dual-write-transcripts.ts
+22 -16
View File
@@ -34,9 +34,9 @@ import {
type AgentMessage,
} from "openclaw/plugin-sdk/agent-harness-runtime";
import {
publishSessionTranscriptUpdateByIdentity,
withSessionTranscriptWriteLock,
type SessionTranscriptTargetParams,
type SessionTranscriptWriteLockContext,
type SessionTranscriptWriteLockParams,
} from "openclaw/plugin-sdk/session-transcript-runtime";
@@ -94,10 +94,10 @@ function buildMirrorDedupeIdentity(message: MirroredAgentMessage): string {
}
interface MirrorCopilotTranscriptParams {
sessionFile: string;
sessionId: string;
sessionKey?: string;
agentId?: string;
storePath?: string;
messages: AgentMessage[];
/**
* Stable per-harness/per-thread scope. The codex equivalent uses
@@ -122,7 +122,7 @@ export async function mirrorCopilotTranscript(
}
const transcriptTarget = resolveCopilotMirrorTranscriptTarget(params);
const didAppend = await withSessionTranscriptWriteLock(
await withCopilotMirrorTranscriptWriteLock(
{ ...transcriptTarget, config: params.config },
async (transcript) => {
let didAppendMessage = false;
@@ -175,36 +175,42 @@ export async function mirrorCopilotTranscript(
existingIdempotencyKeys.add(idempotencyKey);
}
}
if (didAppendMessage) {
await transcript.publishUpdate(
params.sessionKey ? { sessionKey: params.sessionKey } : undefined,
);
}
return didAppendMessage;
},
);
if (didAppend) {
await publishSessionTranscriptUpdateByIdentity({
...transcriptTarget,
update: params.sessionKey ? { sessionKey: params.sessionKey } : undefined,
});
}
}
function resolveCopilotMirrorTranscriptTarget(params: {
agentId?: string;
sessionFile: string;
sessionId: string;
sessionKey?: string;
storePath?: string;
}): SessionTranscriptTargetParams {
const sessionFile = params.sessionFile.trim();
if (!sessionFile) {
throw new Error("Copilot transcript mirror requires a sessionFile target");
const sessionKey = params.sessionKey?.trim();
const storePath = params.storePath?.trim();
if (!sessionKey || !storePath) {
throw new Error("Copilot transcript mirror requires a runtime session identity");
}
return {
...(params.agentId ? { agentId: params.agentId } : {}),
sessionFile,
sessionId: params.sessionId,
sessionKey: params.sessionKey ?? "",
sessionKey,
storePath,
};
}
function withCopilotMirrorTranscriptWriteLock<T>(
params: SessionTranscriptTargetParams & { config?: SessionTranscriptWriteLockParams["config"] },
run: (context: SessionTranscriptWriteLockContext) => Promise<T> | T,
): Promise<T> {
return withSessionTranscriptWriteLock(params, run);
}
function readTranscriptIdempotencyKeys(events: unknown[]): Set<string> {
const keys = new Set<string>();
for (const event of events) {
extensions/discord/src/monitor/native-command.model-picker.test.ts
+33 -42
View File
@@ -12,7 +12,8 @@ import type { ModelsProviderData } from "openclaw/plugin-sdk/command-auth-native
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import * as globalsModule from "openclaw/plugin-sdk/runtime-env";
import {
loadSessionStore,
getSessionEntry,
listSessionEntries,
resolveStorePath,
upsertSessionEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
@@ -442,16 +443,13 @@ describe("Discord model picker interactions", () => {
dispatchSpy,
model: "openai/gpt-4o",
});
const store = loadSessionStore(
resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
{
skipCache: true,
},
);
const entry = Object.values(store).find(
const entries = listSessionEntries({
storePath: resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
});
const entry = entries.find(
(candidate) =>
candidate.providerOverride === "openai" && candidate.modelOverride === "gpt-4o",
);
candidate.entry.providerOverride === "openai" && candidate.entry.modelOverride === "gpt-4o",
)?.entry;
expect(typeof entry?.sessionId).toBe("string");
expect(entry?.sessionId).not.toBe("");
expect(entry?.agentRuntimeOverride).toBe("codex");
@@ -489,17 +487,14 @@ describe("Discord model picker interactions", () => {
dispatchSpy,
model: "anthropic/claude-sonnet-4-5",
});
const store = loadSessionStore(
resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
{
skipCache: true,
},
);
const entry = Object.values(store).find(
const entries = listSessionEntries({
storePath: resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
});
const entry = entries.find(
(candidate) =>
candidate.providerOverride === "anthropic" &&
candidate.modelOverride === "claude-sonnet-4-5",
);
candidate.entry.providerOverride === "anthropic" &&
candidate.entry.modelOverride === "claude-sonnet-4-5",
)?.entry;
expect(entry?.agentRuntimeOverride).toBeUndefined();
});
@@ -534,17 +529,14 @@ describe("Discord model picker interactions", () => {
dispatchSpy,
model: "anthropic/claude-sonnet-4-5",
});
const store = loadSessionStore(
resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
{
skipCache: true,
},
);
const entry = Object.values(store).find(
const entries = listSessionEntries({
storePath: resolveStorePath(context.cfg.session?.store, { agentId: "main" }),
});
const entry = entries.find(
(candidate) =>
candidate.providerOverride === "anthropic" &&
candidate.modelOverride === "claude-sonnet-4-5",
);
candidate.entry.providerOverride === "anthropic" &&
candidate.entry.modelOverride === "claude-sonnet-4-5",
)?.entry;
expect(entry?.agentRuntimeOverride).toBeUndefined();
});
@@ -840,12 +832,10 @@ describe("Discord model picker interactions", () => {
mi: "1",
});
const store = loadSessionStore(storePath, { skipCache: true });
expect(store["agent:worker:subagent:bound"]?.providerOverride).toBe("lmstudio");
expect(store["agent:worker:subagent:bound"]?.modelOverride).toBe(
"unsloth/gemma-4-26b-a4b-it@iq4_xs",
);
expect(store["agent:worker:subagent:bound"]?.liveModelSwitchPending).toBe(true);
const entry = getSessionEntry({ storePath, sessionKey: "agent:worker:subagent:bound" });
expect(entry?.providerOverride).toBe("lmstudio");
expect(entry?.modelOverride).toBe("unsloth/gemma-4-26b-a4b-it@iq4_xs");
expect(entry?.liveModelSwitchPending).toBe(true);
expectDispatchedModelSelection({
dispatchSpy,
model: "lmstudio/unsloth/gemma-4-26b-a4b-it@iq4_xs",
@@ -893,9 +883,9 @@ describe("Discord model picker interactions", () => {
createModelsViewSubmitData(),
);
const store = loadSessionStore(storePath, { skipCache: true });
expect(store["agent:worker:subagent:bound"]?.providerOverride).toBeUndefined();
expect(store["agent:worker:subagent:bound"]?.modelOverride).toBeUndefined();
const entry = getSessionEntry({ storePath, sessionKey: "agent:worker:subagent:bound" });
expect(entry?.providerOverride).toBeUndefined();
expect(entry?.modelOverride).toBeUndefined();
expect(
JSON.stringify(firstMockArg(submitInteraction.followUp, "interaction.followUp")),
).toContain("❌ Failed to apply openai/gpt-4o.");
@@ -950,14 +940,15 @@ describe("Discord model picker interactions", () => {
r: "openclaw",
});
const store = loadSessionStore(storePath, { skipCache: true });
expect(store["agent:worker:subagent:bound"]).toMatchObject({
expect(getSessionEntry({ storePath, sessionKey: "agent:worker:subagent:bound" })).toMatchObject(
{
providerOverride: "openai",
modelOverride: "gpt-5.5",
agentHarnessId: "codex",
agentRuntimeOverride: "codex",
modelSelectionLocked: true,
});
},
);
expect(
JSON.stringify(firstMockArg(submitInteraction.followUp, "interaction.followUp")),
).toContain("❌ Model selection is locked for this session.");
extensions/discord/src/monitor/native-command.think-autocomplete.test.ts
+7 -12
View File
@@ -9,8 +9,7 @@ import {
} from "openclaw/plugin-sdk/plugin-test-runtime";
import {
clearSessionStoreCacheForTest,
saveSessionStore,
type SessionEntry,
upsertSessionEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
import { afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { ChannelType, type AutocompleteInteraction } from "../internal/discord.js";
@@ -140,21 +139,17 @@ async function saveSessionOverride(params: {
agentRuntimeOverride?: string;
}): Promise<void> {
fs.mkdirSync(path.dirname(STORE_PATH), { recursive: true });
await saveSessionStore(
STORE_PATH,
{
[SESSION_KEY]: {
await upsertSessionEntry({
storePath: STORE_PATH,
sessionKey: SESSION_KEY,
entry: {
sessionId: "main",
updatedAt: Date.now(),
providerOverride: params.providerOverride,
modelOverride: params.modelOverride,
...(params.agentRuntimeOverride
? { agentRuntimeOverride: params.agentRuntimeOverride }
: {}),
...(params.agentRuntimeOverride ? { agentRuntimeOverride: params.agentRuntimeOverride } : {}),
},
} satisfies Record<string, SessionEntry>,
{ skipMaintenance: true },
);
});
}
function installProviderThinkingRegistryForTest(): void {
extensions/feishu/src/doctor.test.ts
+276 -38
View File
@@ -2,7 +2,17 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { loadSessionStore } from "openclaw/plugin-sdk/session-store-runtime";
import { DatabaseSync } from "node:sqlite";
import {
formatSqliteSessionFileMarker,
listSessionEntries,
type SessionEntry,
upsertSessionEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
import {
appendSessionTranscriptMessageByIdentity,
readSessionTranscriptEvents,
} from "openclaw/plugin-sdk/session-transcript-runtime";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import type { OpenClawConfig } from "../runtime-api.js";
import { isFeishuSessionStoreKey, runFeishuDoctorSequence } from "./doctor.js";
@@ -59,13 +69,40 @@ function storePath(agentId = "main"): string {
return path.join(sessionsDir(agentId), "sessions.json");
}
function writeStore(entries: Record<string, unknown>, agentId = "main"): string {
function sqliteStorePath(agentId = "main"): string {
return path.join(stateDir(), "agents", agentId, "agent", "openclaw-agent.sqlite");
}
function corruptTranscriptEventJson(agentId: string, sessionId: string): void {
const database = new DatabaseSync(sqliteStorePath(agentId));
try {
database
.prepare("UPDATE transcript_events SET event_json = ? WHERE session_id = ?")
.run("{", sessionId);
} finally {
database.close();
}
}
async function writeStore(entries: Record<string, unknown>, agentId = "main"): Promise<string> {
const target = storePath(agentId);
fs.mkdirSync(path.dirname(target), { recursive: true });
fs.writeFileSync(target, JSON.stringify(entries, null, 2));
for (const [sessionKey, entry] of Object.entries(entries as Record<string, SessionEntry>)) {
await upsertSessionEntry({ agentId, storePath: target, sessionKey, entry });
}
return target;
}
function readStoreEntries(target: string, agentId = "main"): Record<string, SessionEntry> {
return Object.fromEntries(
listSessionEntries({ agentId, storePath: target }).map(({ sessionKey, entry }) => [
sessionKey,
entry,
]),
);
}
function writeTranscript(sessionId: string, lines: unknown[], agentId = "main"): string {
const target = path.join(sessionsDir(agentId), `${sessionId}.jsonl`);
fs.mkdirSync(path.dirname(target), { recursive: true });
@@ -131,7 +168,7 @@ describe("Feishu doctor state repair", () => {
fs.writeFileSync(path.join(feishuDedupDir, "default.json"), JSON.stringify({ msg1: 1 }));
writeTranscript("sess-ok", [sessionHeader("sess-ok"), userMessage("hello")]);
writeStore({
await writeStore({
"agent:main:feishu:direct:ou_user": {
sessionId: "sess-ok",
sessionFile: "sess-ok.jsonl",
@@ -149,22 +186,21 @@ describe("Feishu doctor state repair", () => {
});
it("keeps custom-store sessions with canonical absolute transcripts", async () => {
const customStorePath = path.join(stateDir(), "custom-sessions", "sessions.json");
const transcriptPath = writeTranscript("sess-abs", [
sessionHeader("sess-abs"),
userMessage("hello"),
]);
const customStorePath = path.join(stateDir(), "custom-sessions", "sessions.json");
fs.mkdirSync(path.dirname(customStorePath), { recursive: true });
fs.writeFileSync(
customStorePath,
JSON.stringify({
"agent:main:feishu:direct:ou_user": {
await upsertSessionEntry({
agentId: "main",
storePath: customStorePath,
sessionKey: "agent:main:feishu:direct:ou_user",
entry: {
sessionId: "sess-abs",
sessionFile: transcriptPath,
updatedAt: Date.now(),
},
}),
);
});
const result = await runFeishuDoctorSequence({
cfg: {
@@ -178,6 +214,109 @@ describe("Feishu doctor state repair", () => {
expect(result).toEqual({ changeNotes: [], warningNotes: [] });
});
it("keeps SQLite-backed Feishu session rows without file inspection", async () => {
await writeStore({
"agent:main:feishu:direct:ou_user": {
sessionId: "sess-sqlite",
sessionFile: `sqlite:main:sess-sqlite:${storePath()}`,
updatedAt: Date.now(),
},
});
const result = await runFeishuDoctorSequence({
cfg: feishuConfig(),
env: process.env,
shouldRepair: false,
});
expect(result).toEqual({ changeNotes: [], warningNotes: [] });
});
it("repairs SQLite-backed Feishu sessions with repeated blank user messages", async () => {
const targetStorePath = storePath();
const sessionKey = "agent:main:feishu:direct:ou_sqlite_blank";
const sessionId = "sess-sqlite-blank";
await upsertSessionEntry({
agentId: "main",
storePath: targetStorePath,
sessionKey,
entry: {
sessionId,
sessionFile: formatSqliteSessionFileMarker({
agentId: "main",
sessionId,
storePath: targetStorePath,
}),
updatedAt: Date.now(),
},
});
for (const content of ["", "", ""]) {
await appendSessionTranscriptMessageByIdentity({
agentId: "main",
sessionId,
sessionKey,
storePath: targetStorePath,
message: { role: "user", content },
});
}
const result = await runFeishuDoctorSequence({
cfg: feishuConfig(),
env: process.env,
shouldRepair: true,
});
expect(result.warningNotes).toEqual([]);
expect(result.changeNotes.join("\n")).toContain("Removed 1 Feishu-scoped session entry");
expect(readStoreEntries(targetStorePath)[sessionKey]).toBeUndefined();
await expect(
readSessionTranscriptEvents({
agentId: "main",
sessionId,
sessionKey,
storePath: targetStorePath,
}),
).resolves.toEqual([]);
});
it("repairs SQLite-backed Feishu sessions with corrupt transcript rows", async () => {
const targetStorePath = storePath();
const sessionKey = "agent:main:feishu:direct:ou_sqlite_corrupt";
const sessionId = "sess-sqlite-corrupt";
await upsertSessionEntry({
agentId: "main",
storePath: targetStorePath,
sessionKey,
entry: {
sessionId,
sessionFile: formatSqliteSessionFileMarker({
agentId: "main",
sessionId,
storePath: targetStorePath,
}),
updatedAt: Date.now(),
},
});
await appendSessionTranscriptMessageByIdentity({
agentId: "main",
sessionId,
sessionKey,
storePath: targetStorePath,
message: { role: "user", content: "bad row follows" },
});
corruptTranscriptEventJson("main", sessionId);
const result = await runFeishuDoctorSequence({
cfg: feishuConfig(),
env: process.env,
shouldRepair: true,
});
expect(result.warningNotes).toEqual([]);
expect(result.changeNotes.join("\n")).toContain("Removed 1 Feishu-scoped session entry");
expect(readStoreEntries(targetStorePath)[sessionKey]).toBeUndefined();
});
it("keeps Feishu sessions with separated blank user messages", async () => {
writeTranscript("sess-separated-blanks", [
sessionHeader("sess-separated-blanks"),
@@ -187,7 +326,7 @@ describe("Feishu doctor state repair", () => {
userMessage("world"),
userMessage(""),
]);
writeStore({
await writeStore({
"agent:main:feishu:direct:ou_user": {
sessionId: "sess-separated-blanks",
sessionFile: "sess-separated-blanks.jsonl",
@@ -230,7 +369,7 @@ describe("Feishu doctor state repair", () => {
sessionHeader("sess-ok"),
userMessage("hello"),
]);
const targetStorePath = writeStore({
const targetStorePath = await writeStore({
"agent:main:feishu:direct:ou_user": {
sessionId: "sess-ok",
sessionFile: "sess-ok.jsonl",
@@ -248,7 +387,7 @@ describe("Feishu doctor state repair", () => {
expect(result.changeNotes.join("\n")).toContain("Rebuilt Feishu runtime state: yes");
expect(result.changeNotes.join("\n")).toContain("Removed 0 Feishu-scoped session entries");
const store = loadSessionStore(targetStorePath, { skipCache: true });
const store = readStoreEntries(targetStorePath);
expect(store["agent:main:feishu:direct:ou_user"]).toBeDefined();
expect(fs.existsSync(transcriptPath)).toBe(true);
@@ -286,7 +425,7 @@ describe("Feishu doctor state repair", () => {
userMessage(""),
]);
const targetStorePath = writeStore({
const targetStorePath = await writeStore({
"agent:main:feishu:direct:ou_user": {
sessionId: "sess-bad",
sessionFile: "sess-bad.jsonl",
@@ -325,8 +464,11 @@ describe("Feishu doctor state repair", () => {
expect(fs.existsSync(path.join(backupDir, "session-stores", "main", "sessions.json"))).toBe(
true,
);
expect(
fs.existsSync(path.join(backupDir, "session-stores", "main", "openclaw-agent.sqlite")),
).toBe(true);
const store = loadSessionStore(targetStorePath, { skipCache: true });
const store = readStoreEntries(targetStorePath);
expect(store["agent:main:feishu:direct:ou_user"]).toBeUndefined();
expect(store["agent:codex:acp:binding:feishu:default:abc123"]).toBeDefined();
expect(store["agent:main:discord:direct:user"]).toBeDefined();
@@ -346,29 +488,26 @@ describe("Feishu doctor state repair", () => {
});
it("preserves locked harness sessions while repairing ordinary Feishu sessions", async () => {
const lockedTranscriptPath = writeTranscript("sess-codex-locked", [
sessionHeader("sess-codex-locked"),
userMessage(""),
userMessage(""),
userMessage(""),
]);
const ordinaryTranscriptPath = writeTranscript("sess-feishu-bad", [
sessionHeader("sess-feishu-bad"),
userMessage(""),
userMessage(""),
userMessage(""),
]);
const targetStorePath = writeStore({
"agent:main:ordinary-codex-locked": {
const targetStorePath = storePath();
await upsertSessionEntry({
agentId: "main",
storePath: targetStorePath,
sessionKey: "agent:main:ordinary-codex-locked",
entry: {
sessionId: "sess-codex-locked",
sessionFile: "sess-codex-locked.jsonl",
agentHarnessId: "codex",
modelSelectionLocked: true,
route: { channel: "feishu", target: { to: "ou_user", chatType: "direct" } },
updatedAt: 1,
},
"agent:main:feishu:direct:ou_user": {
});
await upsertSessionEntry({
agentId: "main",
storePath: targetStorePath,
sessionKey: "agent:main:feishu:direct:ou_user",
entry: {
sessionId: "sess-feishu-bad",
sessionFile: "sess-feishu-bad.jsonl",
updatedAt: 1,
},
});
@@ -380,11 +519,110 @@ describe("Feishu doctor state repair", () => {
expect(result.warningNotes).toEqual([]);
expect(result.changeNotes.join("\n")).toContain("Removed 1 Feishu-scoped session entry");
const store = loadSessionStore(targetStorePath, { skipCache: true });
const store = readStoreEntries(targetStorePath);
expect(store["agent:main:ordinary-codex-locked"]).toBeDefined();
expect(store["agent:main:feishu:direct:ou_user"]).toBeUndefined();
expect(fs.existsSync(lockedTranscriptPath)).toBe(true);
expect(fs.existsSync(ordinaryTranscriptPath)).toBe(false);
});
it("backs up SQLite session stores before removing migrated Feishu sessions", async () => {
const targetStorePath = storePath();
const sessionKey = "agent:main:feishu:direct:ou_migrated";
await upsertSessionEntry({
agentId: "main",
storePath: targetStorePath,
sessionKey,
entry: {
sessionId: "sess-migrated-bad",
updatedAt: Date.now(),
},
});
expect(fs.existsSync(targetStorePath)).toBe(false);
expect(fs.existsSync(sqliteStorePath())).toBe(true);
const result = await runFeishuDoctorSequence({
cfg: feishuConfig(),
env: process.env,
shouldRepair: true,
});
expect(result.warningNotes).toEqual([]);
expect(result.changeNotes.join("\n")).toContain("Removed 1 Feishu-scoped session entry");
const backups = listBackupDirs();
expect(backups).toHaveLength(1);
const backupDir = path.join(stateDir(), "backups", backups[0] ?? "");
expect(fs.existsSync(path.join(backupDir, "session-stores", "main", "sessions.json"))).toBe(
false,
);
expect(
fs.existsSync(path.join(backupDir, "session-stores", "main", "openclaw-agent.sqlite")),
).toBe(true);
expect(readStoreEntries(targetStorePath)[sessionKey]).toBeUndefined();
});
it("backs up and repairs Feishu sessions in an agent-scoped custom SQLite store", async () => {
const customStorePath = path.join(stateDir(), "custom-sessions", "sessions.json");
const customSqlitePath = path.join(
path.dirname(customStorePath),
"openclaw-agent.support.sqlite",
);
const sessionKey = "agent:support:feishu:direct:ou_migrated";
await upsertSessionEntry({
agentId: "support",
storePath: customStorePath,
sessionKey,
entry: {
sessionId: "sess-support-bad",
updatedAt: Date.now(),
},
});
await appendSessionTranscriptMessageByIdentity({
agentId: "support",
sessionId: "sess-support-bad",
sessionKey,
storePath: customStorePath,
message: { role: "user", content: "unhealthy migrated Feishu session" },
});
expect(fs.existsSync(customStorePath)).toBe(false);
expect(fs.existsSync(customSqlitePath)).toBe(true);
const result = await runFeishuDoctorSequence({
cfg: {
...feishuConfig(),
agents: { list: [{ id: "support", default: true }] },
session: { store: customStorePath },
} as OpenClawConfig,
env: process.env,
shouldRepair: true,
});
expect(result.warningNotes).toEqual([]);
expect(result.changeNotes.join("\n")).toContain("Removed 1 Feishu-scoped session entry");
const backups = listBackupDirs();
expect(backups).toHaveLength(1);
const backupDir = path.join(stateDir(), "backups", backups[0] ?? "");
expect(fs.existsSync(path.join(backupDir, "session-stores", "support", "sessions.json"))).toBe(
false,
);
expect(
fs.existsSync(
path.join(backupDir, "session-stores", "support", "openclaw-agent.support.sqlite"),
),
).toBe(true);
expect(readStoreEntries(customStorePath, "support")[sessionKey]).toBeUndefined();
await expect(
readSessionTranscriptEvents({
agentId: "support",
sessionId: "sess-support-bad",
sessionKey,
storePath: customStorePath,
}),
).resolves.toEqual([]);
});
it("archives unhealthy default-scope sessions when metadata identifies Feishu", async () => {
@@ -394,7 +632,7 @@ describe("Feishu doctor state repair", () => {
userMessage(""),
userMessage(""),
]);
const targetStorePath = writeStore({
const targetStorePath = await writeStore({
"agent:main:main": {
sessionId: "sess-default-feishu-bad",
sessionFile: "sess-default-feishu-bad.jsonl",
@@ -416,7 +654,7 @@ describe("Feishu doctor state repair", () => {
});
expect(result.warningNotes).toEqual([]);
const store = loadSessionStore(targetStorePath, { skipCache: true });
const store = readStoreEntries(targetStorePath);
expect(store["agent:main:main"]).toBeUndefined();
expect(store["agent:main:main-non-feishu"]).toBeDefined();
expect(fs.existsSync(transcriptPath)).toBe(false);
extensions/feishu/src/doctor.ts
+186 -100
View File
@@ -10,10 +10,12 @@ import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import { normalizeAgentId } from "openclaw/plugin-sdk/routing";
import {
isValidAgentHarnessSessionStoreEntry,
loadSessionStore,
resolveSessionFilePath,
deleteSessionEntry,
listSessionEntries,
loadTranscriptEventsSync,
parseSqliteSessionFileMarker,
resolveSessionStoreBackupPaths,
resolveStorePath,
updateSessionStore,
} from "openclaw/plugin-sdk/session-store-runtime";
import { resolveStateDir } from "openclaw/plugin-sdk/state-paths";
import { isRecord } from "openclaw/plugin-sdk/string-coerce-runtime";
@@ -116,6 +118,14 @@ function existsFile(filePath: string): boolean {
}
}
function resolveFeishuAgentSessionsDir(agentId: string): string {
return path.join(resolveStateDir(), "agents", normalizeAgentId(agentId), "sessions");
}
function isSqliteTranscriptMarker(value: string): boolean {
return parseSqliteSessionFileMarker(value) !== undefined;
}
function safeReadDir(dir: string): fs.Dirent[] {
try {
return fs.readdirSync(dir, { withFileTypes: true });
@@ -238,9 +248,11 @@ function collectFeishuSessionTargets(params: {
}): FeishuSessionTarget[] {
const byStorePath = new Map<string, FeishuSessionTarget>();
const addTarget = (target: FeishuSessionTarget) => {
byStorePath.set(path.resolve(target.storePath), {
const resolvedStorePath = path.resolve(target.storePath);
byStorePath.set(`${normalizeAgentId(target.agentId)}\0${resolvedStorePath}`, {
...target,
storePath: path.resolve(target.storePath),
agentId: normalizeAgentId(target.agentId),
storePath: resolvedStorePath,
});
};
@@ -315,29 +327,39 @@ function resolveSessionTranscriptCandidates(params: {
}): string[] {
const candidates = new Set<string>();
const sessionsDir = path.dirname(params.storePath);
const addSafeCandidate = (candidate: string) => {
const agentSessionsDir = resolveFeishuAgentSessionsDir(params.agentId);
const addSafeCandidate = (candidate: string): boolean => {
const resolved = path.isAbsolute(candidate)
? path.resolve(candidate)
: path.resolve(sessionsDir, candidate);
if (resolved === sessionsDir || !isPathWithinRoot(resolved, sessionsDir)) {
return;
const isStoreCandidate = isPathWithinRoot(resolved, sessionsDir);
const isAgentSessionCandidate = isPathWithinRoot(resolved, agentSessionsDir);
if (
resolved === sessionsDir ||
resolved === agentSessionsDir ||
(!isStoreCandidate && !isAgentSessionCandidate)
) {
return false;
}
candidates.add(resolved);
return true;
};
if (
typeof params.entry.sessionId === "string" &&
/^[a-z0-9][a-z0-9._-]{0,127}$/i.test(params.entry.sessionId)
) {
candidates.add(
resolveSessionFilePath(
params.entry.sessionId,
typeof params.entry.sessionFile === "string"
? { sessionFile: params.entry.sessionFile }
: undefined,
{ agentId: params.agentId, sessionsDir },
),
);
let addedExplicitCandidate = false;
if (typeof params.entry.sessionFile === "string" && params.entry.sessionFile.trim()) {
const explicitSessionFile = params.entry.sessionFile.trim();
if (isSqliteTranscriptMarker(explicitSessionFile)) {
return [];
}
addedExplicitCandidate = addSafeCandidate(explicitSessionFile);
}
if (!addedExplicitCandidate) {
candidates.add(path.join(sessionsDir, `${params.entry.sessionId}.jsonl`));
}
return [...candidates].toSorted();
}
@@ -375,6 +397,72 @@ function isUserMessage(value: unknown): boolean {
);
}
function inspectTranscriptEntries(params: {
sessionKey: string;
storePath: string;
transcriptPath: string;
entries: unknown[];
allowMissingSessionHeader?: boolean;
malformedLines?: number;
}): FeishuDoctorFinding | null {
let blankUserMessageRun = 0;
let maxBlankUserMessageRun = 0;
for (const entry of params.entries) {
if (isBlankUserMessage(entry)) {
blankUserMessageRun += 1;
maxBlankUserMessageRun = Math.max(maxBlankUserMessageRun, blankUserMessageRun);
} else if (isUserMessage(entry)) {
blankUserMessageRun = 0;
}
}
if (params.entries.length === 0) {
if (params.allowMissingSessionHeader) {
return null;
}
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: "empty transcript",
};
}
const firstEntry = params.entries[0];
if (
!isSessionHeader(firstEntry) &&
(!params.allowMissingSessionHeader ||
(!isUserMessage(firstEntry) && !isBlankUserMessage(firstEntry)))
) {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: "invalid session header",
};
}
if ((params.malformedLines ?? 0) > 0) {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: `${params.malformedLines} malformed JSONL line(s)`,
};
}
if (maxBlankUserMessageRun >= BLANK_USER_MESSAGE_REPAIR_THRESHOLD) {
return {
kind: "blank-user-message-run",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
count: maxBlankUserMessageRun,
};
}
return null;
}
function inspectSessionTranscript(params: {
sessionKey: string;
storePath: string;
@@ -414,8 +502,6 @@ function inspectSessionTranscript(params: {
const entries: unknown[] = [];
let malformedLines = 0;
let blankUserMessageRun = 0;
let maxBlankUserMessageRun = 0;
for (const line of raw.split(/\r?\n/)) {
if (!line.trim()) {
continue;
@@ -423,55 +509,56 @@ function inspectSessionTranscript(params: {
try {
const entry = JSON.parse(line);
entries.push(entry);
if (isBlankUserMessage(entry)) {
blankUserMessageRun += 1;
maxBlankUserMessageRun = Math.max(maxBlankUserMessageRun, blankUserMessageRun);
} else if (isUserMessage(entry)) {
blankUserMessageRun = 0;
}
} catch {
malformedLines += 1;
}
}
if (entries.length === 0) {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: "empty transcript",
};
}
if (!isSessionHeader(entries[0])) {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: "invalid session header",
};
}
if (malformedLines > 0) {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
reason: `${malformedLines} malformed JSONL line(s)`,
};
}
if (maxBlankUserMessageRun >= BLANK_USER_MESSAGE_REPAIR_THRESHOLD) {
return {
kind: "blank-user-message-run",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.transcriptPath,
count: maxBlankUserMessageRun,
};
return inspectTranscriptEntries({ ...params, entries, malformedLines });
}
function inspectSqliteSessionTranscript(params: {
agentId: string;
sessionKey: string;
storePath: string;
entry: FeishuSessionEntry;
}): FeishuDoctorFinding | null {
if (typeof params.entry.sessionFile !== "string") {
return null;
}
const marker = parseSqliteSessionFileMarker(params.entry.sessionFile);
if (!marker) {
return null;
}
const sessionId =
typeof params.entry.sessionId === "string" && params.entry.sessionId.trim()
? params.entry.sessionId.trim()
: marker.sessionId;
let entries: unknown[];
try {
entries = loadTranscriptEventsSync({
agentId: marker.agentId,
sessionId,
sessionKey: params.sessionKey,
storePath: marker.storePath,
});
} catch {
return {
kind: "invalid-session-transcript",
sessionKey: params.sessionKey,
storePath: params.storePath,
path: params.entry.sessionFile,
reason: "unreadable",
};
}
return inspectTranscriptEntries({
sessionKey: params.sessionKey,
storePath: params.storePath,
transcriptPath: params.entry.sessionFile,
allowMissingSessionHeader: true,
entries,
});
}
function collectFeishuSessionFindings(params: {
agentId: string;
@@ -479,6 +566,10 @@ function collectFeishuSessionFindings(params: {
storePath: string;
entry: FeishuSessionEntry;
}): FeishuDoctorFinding[] {
const sqliteFinding = inspectSqliteSessionTranscript(params);
if (sqliteFinding) {
return [sqliteFinding];
}
const transcriptCandidates = resolveSessionTranscriptCandidates(params);
const existing = transcriptCandidates.filter(existsFile);
if (transcriptCandidates.length > 0 && existing.length === 0) {
@@ -556,18 +647,16 @@ function inspectFeishuDoctorState(params: {
const sessionEntries: FeishuDoctorInspection["sessionEntries"] = [];
for (const target of collectFeishuSessionTargets({ cfg: params.cfg, env, stateDir })) {
const store = loadSessionStore(target.storePath, { skipCache: true });
for (const [key, entry] of Object.entries(store).toSorted(([left], [right]) =>
left.localeCompare(right),
)) {
// Harness ownership supersedes channel-derived route metadata. Feishu
// doctor must not diagnose or clean up another runtime's locked row.
if (isRecord(entry) && isValidAgentHarnessSessionStoreEntry(key, entry)) {
continue;
}
if (!isFeishuSessionEntry(key, entry)) {
continue;
}
for (const { sessionKey: key, entry } of listSessionEntries({
agentId: target.agentId,
storePath: target.storePath,
})
.filter(
({ sessionKey, entry: sessionEntry }) =>
!isValidAgentHarnessSessionStoreEntry(sessionKey, sessionEntry) &&
isFeishuSessionEntry(sessionKey, sessionEntry),
)
.toSorted((left, right) => left.sessionKey.localeCompare(right.sessionKey))) {
const sessionEntry = toFeishuSessionEntry(entry);
sessionEntries.push({
key,
@@ -628,17 +717,18 @@ function movePathToBackup(params: {
}
function copyStoreBackup(params: { storePath: string; backupDir: string; agentId: string }) {
if (!existsFile(params.storePath)) {
return;
const targetDir = path.join(params.backupDir, "session-stores", params.agentId);
for (const sourcePath of resolveSessionStoreBackupPaths({
agentId: params.agentId,
storePath: params.storePath,
})) {
if (!existsFile(sourcePath)) {
continue;
}
const targetPath = path.join(
params.backupDir,
"session-stores",
params.agentId,
path.basename(params.storePath),
);
const targetPath = path.join(targetDir, path.basename(sourcePath));
fs.mkdirSync(path.dirname(targetPath), { recursive: true, mode: 0o700 });
fs.copyFileSync(params.storePath, resolveUniquePath(targetPath));
fs.copyFileSync(sourcePath, resolveUniquePath(targetPath));
}
}
function collectSessionArtifactPaths(params: {
@@ -739,32 +829,28 @@ async function repairFeishuDoctorState(params: {
try {
copyStoreBackup({ storePath, backupDir, agentId: group.agentId });
const keys = new Set(group.entries.map((entry) => entry.key));
const removedEntries = await updateSessionStore(
storePath,
(store) => {
const removed: typeof group.entries = [];
const removedEntries: typeof group.entries = [];
for (const key of keys) {
const currentEntry = store[key];
// Recheck under the store update lock because a harness can claim
// a previously flagged row after inspection but before repair.
if (
!Object.hasOwn(store, key) ||
(currentEntry && isValidAgentHarnessSessionStoreEntry(key, currentEntry))
) {
const currentEntry = listSessionEntries({ agentId: group.agentId, storePath }).find(
(candidate) => candidate.sessionKey === key,
)?.entry;
if (!currentEntry || isValidAgentHarnessSessionStoreEntry(key, currentEntry)) {
continue;
}
const deleted = await deleteSessionEntry({
agentId: group.agentId,
archiveTranscript: true,
sessionKey: key,
storePath,
});
if (!deleted) {
continue;
}
delete store[key];
const entry = group.entries.find((candidate) => candidate.key === key);
if (entry) {
removed.push(entry);
removedEntries.push(entry);
}
}
return removed;
},
{
skipMaintenance: true,
},
);
const removed = removedEntries.length;
removedSessionEntries += removed;
if (removed > 0) {
extensions/matrix/src/exec-approvals.test.ts
+14 -7
View File
@@ -3,7 +3,8 @@ import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import type { OpenClawConfig } from "openclaw/plugin-sdk/config-contracts";
import { saveSessionStore } from "openclaw/plugin-sdk/session-store-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { closeOpenClawAgentDatabasesForTest } from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { afterEach, describe, expect, it } from "vitest";
import {
getMatrixExecApprovalApprovers,
@@ -24,6 +25,7 @@ type MatrixExecApprovalRequest = Parameters<
>[0]["request"];
afterEach(() => {
closeOpenClawAgentDatabasesForTest();
for (const dir of tempDirs.splice(0)) {
fs.rmSync(dir, { recursive: true, force: true });
}
@@ -346,23 +348,28 @@ describe("matrix exec approvals", () => {
it("scopes non-matrix turn sources to the stored matrix account", async () => {
const tmpDir = createTempDir();
const storePath = path.join(tmpDir, "sessions.json");
await saveSessionStore(
await upsertSessionEntry({
storePath,
{
"agent:ops-agent:matrix:channel:!room:example.org": {
sessionKey: "agent:ops-agent:matrix:channel:!room:example.org",
entry: {
sessionId: "main",
updatedAt: 1,
origin: {
provider: "matrix",
accountId: "ops",
to: "room:!room:example.org",
nativeChannelId: "!room:example.org",
},
deliveryContext: {
channel: "matrix",
to: "room:!room:example.org",
accountId: "ops",
},
lastChannel: "slack",
lastTo: "channel:C999",
lastAccountId: "work",
},
},
{ skipMaintenance: true },
);
});
const cfg = buildMultiAccountMatrixConfig({ sessionStorePath: storePath });
const request = makeForeignChannelApprovalRequest({
id: "req-3",
extensions/matrix/src/matrix/monitor/handler.test.ts
+9 -17
View File
@@ -7,11 +7,7 @@ import {
testing as sessionBindingTesting,
registerSessionBindingAdapter,
} from "openclaw/plugin-sdk/session-binding-runtime";
import {
getSessionEntry,
saveSessionStore,
upsertSessionEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
import { getSessionEntry, upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { installMatrixMonitorTestRuntime } from "../../test-runtime.js";
import { MATRIX_OPENCLAW_FINALIZED_PREVIEW_KEY } from "../send/types.js";
@@ -1468,10 +1464,10 @@ describe("matrix monitor handler pairing account scope", () => {
it("skips the shared-session notice when Matrix DMs are isolated per room", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "matrix-dm-room-scope-"));
const storePath = path.join(tempDir, "sessions.json");
await saveSessionStore(
await upsertSessionEntry({
storePath,
{
"agent:ops:main": {
sessionKey: "agent:ops:main",
entry: {
sessionId: "sess-main",
updatedAt: Date.now(),
deliveryContext: {
@@ -1480,9 +1476,7 @@ describe("matrix monitor handler pairing account scope", () => {
accountId: "ops",
},
},
},
{ skipMaintenance: true },
);
});
const sendNotice = vi.fn(async () => "$notice");
try {
@@ -1515,10 +1509,10 @@ describe("matrix monitor handler pairing account scope", () => {
it("skips the shared-session notice when a Matrix DM is explicitly bound", async () => {
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "matrix-dm-bound-notice-"));
const storePath = path.join(tempDir, "sessions.json");
await saveSessionStore(
await upsertSessionEntry({
storePath,
{
"agent:bound:session-1": {
sessionKey: "agent:bound:session-1",
entry: {
sessionId: "sess-bound",
updatedAt: Date.now(),
deliveryContext: {
@@ -1527,9 +1521,7 @@ describe("matrix monitor handler pairing account scope", () => {
accountId: "ops",
},
},
},
{ skipMaintenance: true },
);
});
const sendNotice = vi.fn(async () => "$notice");
const touch = vi.fn();
registerSessionBindingAdapter({
extensions/matrix/src/session-route.test.ts
+5 -2
View File
@@ -2,7 +2,8 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { saveSessionStore, type SessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import type { SessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { afterEach, describe, expect, it } from "vitest";
import type { OpenClawConfig } from "./runtime-api.js";
import { resolveMatrixOutboundSessionRoute } from "./session-route.js";
@@ -32,7 +33,9 @@ async function createTempStore(entries: Record<string, SessionEntry>): Promise<s
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "matrix-session-route-"));
tempDirs.add(tempDir);
const storePath = path.join(tempDir, "sessions.json");
await saveSessionStore(storePath, entries, { skipMaintenance: true });
for (const [sessionKey, entry] of Object.entries(entries)) {
await upsertSessionEntry({ sessionKey, storePath, entry });
}
return storePath;
}
extensions/mattermost/src/mattermost/model-picker.test.ts
+33 -15
View File
@@ -2,6 +2,7 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { describe, expect, it } from "vitest";
import type { OpenClawConfig } from "../../runtime-api.js";
import {
@@ -215,37 +216,52 @@ describe("Mattermost model picker", () => {
}
});
it("resolves current and parent model overrides from targeted session entries", () => {
it("resolves current and parent model overrides from targeted session entries", async () => {
const testDir = fs.mkdtempSync(path.join(os.tmpdir(), "mm-model-picker-"));
try {
const storePath = path.join(testDir, "{agentId}.json");
const supportStorePath = path.join(testDir, "support.json");
const storePath = path.join(testDir, "agents", "{agentId}", "sessions", "sessions.json");
const supportStorePath = path.join(testDir, "agents", "support", "sessions", "sessions.json");
const parentSessionKey = "agent:support:mattermost:default:channel-1";
const childSessionKey = "agent:support:mattermost:default:child-with-explicit-parent";
const directSessionKey = "agent:support:mattermost:default:direct-1";
fs.writeFileSync(
supportStorePath,
JSON.stringify(
{
[parentSessionKey]: {
await upsertSessionEntry({
agentId: "support",
storePath: supportStorePath,
sessionKey: parentSessionKey,
entry: {
providerOverride: "anthropic",
modelOverride: "claude-sonnet-4-5",
chatType: "channel",
channel: "channel-1",
sessionId: "parent-session",
updatedAt: 1,
},
[childSessionKey]: {
});
await upsertSessionEntry({
agentId: "support",
storePath: supportStorePath,
sessionKey: childSessionKey,
entry: {
parentSessionKey,
chatType: "channel",
channel: "child-with-explicit-parent",
sessionId: "child-session",
updatedAt: 2,
},
[directSessionKey]: {
});
await upsertSessionEntry({
agentId: "support",
storePath: supportStorePath,
sessionKey: directSessionKey,
entry: {
providerOverride: "openai",
modelOverride: "gpt-5",
chatType: "channel",
channel: "direct-1",
sessionId: "direct-session",
updatedAt: 3,
},
},
null,
2,
),
);
});
const cfg: OpenClawConfig = {
session: {
store: storePath,
@@ -260,6 +276,7 @@ describe("Mattermost model picker", () => {
sessionKey: directSessionKey,
},
data,
readConsistency: "latest",
}),
).toBe("openai/gpt-5");
expect(
@@ -270,6 +287,7 @@ describe("Mattermost model picker", () => {
sessionKey: childSessionKey,
},
data,
readConsistency: "latest",
}),
).toBe("anthropic/claude-sonnet-4-5");
} finally {
extensions/memory-core/src/dreaming-narrative.test.ts
+90 -25
View File
@@ -9,7 +9,13 @@ import {
import { resolveGlobalMap } from "openclaw/plugin-sdk/global-singleton";
import * as memoryCoreHostRuntimeCoreModule from "openclaw/plugin-sdk/memory-core-host-runtime-core";
import * as runtimeConfigSnapshotModule from "openclaw/plugin-sdk/runtime-config-snapshot";
import * as sessionStoreRuntimeModule from "openclaw/plugin-sdk/session-store-runtime";
import {
listSessionEntries,
loadTranscriptEventsSync,
upsertSessionEntry,
type SessionEntry,
} from "openclaw/plugin-sdk/session-store-runtime";
import { appendSqliteSessionTranscriptEventForTest } from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
appendNarrativeEntry,
@@ -87,6 +93,40 @@ function expectLogExcludes(source: MockCallSource, text: string): void {
expect(logIncludes(source, text), `Expected log not to include ${text}`).toBe(false);
}
async function seedSessionStore(
storePath: string,
entries: Record<string, SessionEntry>,
): Promise<void> {
for (const [sessionKey, entry] of Object.entries(entries)) {
await upsertSessionEntry({ storePath, sessionKey, entry });
}
}
function readSessionStoreEntries(storePath: string): Record<string, SessionEntry> {
return Object.fromEntries(
listSessionEntries({ storePath }).map(({ sessionKey, entry }) => [sessionKey, entry]),
);
}
async function seedDreamingTranscriptEvent(params: {
sessionId: string;
storePath: string;
timestampMs: number;
runId?: string;
}): Promise<void> {
await appendSqliteSessionTranscriptEventForTest({
agentId: "main",
sessionId: params.sessionId,
sessionKey: `agent:main:dreaming-narrative-fixture:${params.sessionId}`,
storePath: params.storePath,
event: {
type: "metadata",
timestamp: params.timestampMs,
runId: params.runId ?? `dreaming-narrative-${params.sessionId}`,
},
});
}
async function flushNarrativeSettleTimers<T>(operation: Promise<T>): Promise<T> {
await vi.runAllTimersAsync();
return operation;
@@ -1223,15 +1263,14 @@ describe("generateAndAppendDreamNarrative", () => {
const livePath = path.join(sessionsDir, "still-live.jsonl");
const normalTranscriptPath = path.join(sessionsDir, "normal-user-session.jsonl");
const updatedAt = Date.now();
await sessionStoreRuntimeModule.saveSessionStore(
storePath,
{
await seedSessionStore(storePath, {
"agent:main:dreaming-narrative-light-1": {
sessionId: "missing",
updatedAt,
},
"agent:main:kept-session": {
sessionId: "still-live",
sessionFile: livePath,
updatedAt,
},
"agent:main:telegram:group:dreaming-narrative-room": {
@@ -1240,11 +1279,22 @@ describe("generateAndAppendDreamNarrative", () => {
},
"agent:main:dreaming-narrative-corrupt-normal": {
sessionId: "normal-user-session",
sessionFile: normalTranscriptPath,
updatedAt,
},
},
{ skipMaintenance: true },
);
});
await seedDreamingTranscriptEvent({
sessionId: "orphan",
storePath,
timestampMs: Date.now() - 600_000,
runId: "dreaming-narrative-light-123",
});
await seedDreamingTranscriptEvent({
sessionId: "still-live",
storePath,
timestampMs: Date.now(),
runId: "dreaming-narrative-light-keep",
});
await fs.writeFile(orphanPath, '{"runId":"dreaming-narrative-light-123"}\n', "utf-8");
await fs.writeFile(livePath, '{"runId":"dreaming-narrative-light-keep"}\n', "utf-8");
await fs.writeFile(normalTranscriptPath, '{"runId":"ordinary-user-session"}\n', "utf-8");
@@ -1269,15 +1319,18 @@ describe("generateAndAppendDreamNarrative", () => {
logger,
});
const updatedStore = sessionStoreRuntimeModule.loadSessionStore(storePath, {
skipCache: true,
}) as Record<string, unknown>;
const updatedStore = readSessionStoreEntries(storePath) as Record<string, unknown>;
expect(updatedStore).not.toHaveProperty("agent:main:dreaming-narrative-light-1");
expect(updatedStore).not.toHaveProperty("agent:main:dreaming-narrative-corrupt-normal");
expect(updatedStore).toHaveProperty("agent:main:kept-session");
expect(updatedStore).toHaveProperty("agent:main:telegram:group:dreaming-narrative-room");
expect(loadTranscriptEventsSync({ agentId: "main", sessionId: "orphan", storePath })).toEqual(
[],
);
expect(
loadTranscriptEventsSync({ agentId: "main", sessionId: "still-live", storePath }),
).not.toEqual([]);
const sessionFiles = await fs.readdir(sessionsDir);
expect(sessionFiles.filter((file) => file.startsWith("orphan.jsonl.deleted."))).not.toEqual([]);
expect(sessionFiles).toContain("still-live.jsonl");
expect(sessionFiles).toContain("normal-user-session.jsonl");
expectLogIncludes(logger.info, "dreaming cleanup scrubbed");
@@ -1296,24 +1349,35 @@ describe("generateAndAppendDreamNarrative", () => {
// must be preserved.
const liveTranscript = path.join(sessionsDir, "live-dreaming.jsonl");
const updatedAt = Date.now();
await sessionStoreRuntimeModule.saveSessionStore(
storePath,
{
await seedSessionStore(storePath, {
"agent:main:dreaming-narrative-deep-orphan": {
sessionId: "orphan-dreaming",
sessionFile: orphanTranscript,
updatedAt,
},
"agent:main:dreaming-narrative-deep-live": {
sessionId: "live-dreaming",
sessionFile: liveTranscript,
updatedAt,
},
"agent:main:kept-session": {
sessionId: "still-live",
sessionFile: path.join(sessionsDir, "still-live.jsonl"),
updatedAt,
},
},
{ skipMaintenance: true },
);
});
await seedDreamingTranscriptEvent({
sessionId: "orphan-dreaming",
storePath,
timestampMs: Date.now() - 600_000,
runId: "dreaming-narrative-deep-orphan",
});
await seedDreamingTranscriptEvent({
sessionId: "live-dreaming",
storePath,
timestampMs: Date.now(),
runId: "dreaming-narrative-deep-live",
});
await fs.writeFile(orphanTranscript, '{"runId":"dreaming-narrative-deep-orphan"}\n', "utf-8");
await fs.writeFile(liveTranscript, '{"runId":"dreaming-narrative-deep-live"}\n', "utf-8");
await fs.writeFile(path.join(sessionsDir, "still-live.jsonl"), "{}\n", "utf-8");
@@ -1338,21 +1402,22 @@ describe("generateAndAppendDreamNarrative", () => {
logger,
});
const updatedStore = sessionStoreRuntimeModule.loadSessionStore(storePath, {
skipCache: true,
}) as Record<string, unknown>;
const updatedStore = readSessionStoreEntries(storePath) as Record<string, unknown>;
// The aged orphan dreaming row is reclaimed even though its transcript existed.
expect(updatedStore).not.toHaveProperty("agent:main:dreaming-narrative-deep-orphan");
// The fresh dreaming row and the non-dreaming row survive.
expect(updatedStore).toHaveProperty("agent:main:dreaming-narrative-deep-live");
expect(updatedStore).toHaveProperty("agent:main:kept-session");
expect(
loadTranscriptEventsSync({ agentId: "main", sessionId: "orphan-dreaming", storePath }),
).toEqual([]);
expect(
loadTranscriptEventsSync({ agentId: "main", sessionId: "live-dreaming", storePath }),
).not.toEqual([]);
const sessionFiles = await fs.readdir(sessionsDir);
// The orphan transcript is archived; the live transcript stays.
expect(
sessionFiles.filter((file) => file.startsWith("orphan-dreaming.jsonl.deleted.")),
).not.toEqual([]);
expect(sessionFiles).not.toContain("orphan-dreaming.jsonl");
// SQLite transcript state is archived while legacy JSONL support files are left alone.
expect(sessionFiles).toContain("orphan-dreaming.jsonl");
expect(sessionFiles).toContain("live-dreaming.jsonl");
expectLogIncludes(logger.info, "dreaming cleanup scrubbed");
});
extensions/memory-core/src/dreaming-phases.test.ts
+213 -369
View File
@@ -10,7 +10,10 @@ import {
resolveMemoryLightDreamingConfig,
resolveMemoryRemDreamingConfig,
} from "openclaw/plugin-sdk/memory-core-host-status";
import { saveSessionStore } from "openclaw/plugin-sdk/session-store-runtime";
import { clearRuntimeConfigSnapshot } from "openclaw/plugin-sdk/runtime-config-snapshot";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import { formatSqliteSessionFileMarker } from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
testing,
@@ -64,6 +67,7 @@ const LIGHT_DREAMING_TEST_CONFIG: OpenClawConfig = {
function setDreamingTestEnv(stateDir: string): void {
Reflect.set(process.env, "OPENCLAW_TEST_FAST", "1");
Reflect.set(process.env, "OPENCLAW_STATE_DIR", stateDir);
clearRuntimeConfigSnapshot();
}
function restoreDreamingTestEnv(): void {
@@ -77,6 +81,7 @@ function restoreDreamingTestEnv(): void {
} else {
Reflect.set(process.env, "OPENCLAW_STATE_DIR", originalDreamingStateDir);
}
clearRuntimeConfigSnapshot();
}
afterEach(() => {
@@ -141,6 +146,61 @@ function requireFirstIngestionEntry(sessionIngestion: {
return firstEntry;
}
async function seedDreamingSessionTranscript(params: {
agentId?: string;
messages: Array<{
role: "assistant" | "user";
content: unknown;
timestamp: number | string;
}>;
sessionId: string;
sessionKey?: string;
}): Promise<void> {
const agentId = params.agentId ?? "main";
const sessionsDir = resolveSessionTranscriptsDirForAgent(agentId);
const storePath = path.join(sessionsDir, "sessions.json");
const sessionKey = params.sessionKey ?? `agent:${agentId}:dreaming:${params.sessionId}`;
const timestamps = params.messages
.map((message) =>
typeof message.timestamp === "number" ? message.timestamp : Date.parse(message.timestamp),
)
.filter((timestamp) => Number.isFinite(timestamp));
// Accessor writes run normal maintenance; keep fixture entries fresh while
// retaining per-message timestamps as the dreaming corpus clock.
const updatedAt = Math.max(Date.now(), ...timestamps);
await fs.mkdir(sessionsDir, { recursive: true });
const sessionFile = formatSqliteSessionFileMarker({
agentId,
sessionId: params.sessionId,
storePath,
});
await upsertSessionEntry({
agentId,
sessionKey,
storePath,
entry: { sessionFile, sessionId: params.sessionId, updatedAt },
});
for (const message of params.messages) {
await appendSessionTranscriptMessageByIdentity({
agentId,
sessionId: params.sessionId,
sessionKey,
storePath,
message: {
role: message.role,
content: message.content,
timestamp: message.timestamp,
},
});
}
await upsertSessionEntry({
agentId,
sessionKey,
storePath,
entry: { sessionFile, sessionId: params.sessionId, updatedAt },
});
}
function createHarness(
config: OpenClawConfig,
workspaceDir?: string,
@@ -1039,62 +1099,41 @@ describe("memory-core dreaming phases", () => {
it("checkpoints session transcript ingestion and skips unchanged transcripts", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptName = `dreaming-${"x".repeat(48)}.jsonl`;
const transcriptPath = path.join(sessionsDir, transcriptName);
const snippetTranscriptName = "snippet-boundary.jsonl";
const snippetTranscriptPath = path.join(sessionsDir, snippetTranscriptName);
const transcriptName = `dreaming-${"x".repeat(48)}`;
const snippetTranscriptName = "snippet-boundary";
const renderedSource = `[main/sessions/main/${transcriptName}#L4] `;
const renderedPadding = "r".repeat(343 - renderedSource.length - "User: ".length);
const snippetPadding = "s".repeat(273);
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "session",
id: "dreaming-main",
timestamp: "2026-04-05T18:00:00.000Z",
}),
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: transcriptName,
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: "Move backups to S3 Glacier." }],
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-05T18:02:00.000Z",
content: [{ type: "text", text: "Set retention to 365 days." }],
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "user",
timestamp: "2026-04-05T18:03:00.000Z",
content: [{ type: "text", text: `${renderedPadding}🎉 omitted tail` }],
},
}),
].join("\n") + "\n",
"utf-8",
);
await fs.writeFile(
snippetTranscriptPath,
JSON.stringify({
type: "message",
message: {
],
});
await seedDreamingSessionTranscript({
sessionId: snippetTranscriptName,
messages: [
{
role: "user",
timestamp: "2026-04-05T18:04:00.000Z",
content: [{ type: "text", text: `${snippetPadding}🌍 omitted tail` }],
},
}) + "\n",
"utf-8",
);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1126,25 +1165,23 @@ describe("memory-core dreaming phases", () => {
workspaceDir,
);
const readSpy = vi.spyOn(fs, "readFile");
let transcriptReadCount;
let firstSessionIngestion;
try {
await withDreamingTestClock(async () => {
await triggerLightDreaming(beforeAgentReply, workspaceDir, 5);
firstSessionIngestion = await testing.readSessionIngestionState(workspaceDir);
await triggerLightDreaming(beforeAgentReply, workspaceDir, 6);
});
} finally {
transcriptReadCount = readSpy.mock.calls.filter(
([target]) => typeof target === "string" && target === transcriptPath,
).length;
readSpy.mockRestore();
restoreDreamingTestEnv();
}
expect(transcriptReadCount).toBeLessThanOrEqual(1);
const sessionIngestion = await testing.readSessionIngestionState(workspaceDir);
expect(firstSessionIngestion).toStrictEqual(sessionIngestion);
expect(Object.keys(sessionIngestion.files)).toContain(`main:sessions/main/${transcriptName}`);
expect(Object.keys(sessionIngestion.seenMessages)).toContain(
`main:sessions/main/${transcriptName}`,
);
const corpusPath = path.join(
workspaceDir,
"memory",
@@ -1157,7 +1194,7 @@ describe("memory-core dreaming phases", () => {
expect(corpus).toContain("Set retention to 365 days.");
expect(corpus).toContain(`${renderedSource}User: ${renderedPadding}\n`);
expect(corpus).toContain(
`[main/sessions/main/${snippetTranscriptName}#L1] User: ${snippetPadding}\n`,
`[main/sessions/main/${snippetTranscriptName}#L2] User: ${snippetPadding}\n`,
);
expect(corpus).not.toContain("🎉");
expect(corpus).not.toContain("🌍");
@@ -1169,12 +1206,12 @@ describe("memory-core dreaming phases", () => {
minUniqueQueries: 0,
nowMs: Date.parse("2026-04-05T19:00:00.000Z"),
});
expect(ranked.map((candidate) => candidate.path)).not.toContain(
expect(ranked.map((candidate) => candidate.path)).toContain(
"memory/.dreams/session-corpus/2026-04-05.txt",
);
const snippets = ranked.map((candidate) => candidate.snippet);
expectNotIncludesSubstring(snippets, "Move backups to S3 Glacier.");
expectNotIncludesSubstring(snippets, "Set retention to 365 days.");
expectIncludesSubstring(snippets, "Move backups to S3 Glacier.");
expectIncludesSubstring(snippets, "Set retention to 365 days.");
});
it("keeps primary session transcripts out of configured subagent workspaces", async () => {
@@ -1182,38 +1219,27 @@ describe("memory-core dreaming phases", () => {
const subagentWorkspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const mainSessionsDir = resolveSessionTranscriptsDirForAgent("main");
const subagentSessionsDir = resolveSessionTranscriptsDirForAgent("agi-ceo");
await fs.mkdir(mainSessionsDir, { recursive: true });
await fs.mkdir(subagentSessionsDir, { recursive: true });
await fs.writeFile(
path.join(mainSessionsDir, "main-session.jsonl"),
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "main-session",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: "Main workspace should stay in main dreams." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
await fs.writeFile(
path.join(subagentSessionsDir, "subagent-session.jsonl"),
[
JSON.stringify({
type: "message",
message: {
],
});
await seedDreamingSessionTranscript({
agentId: "agi-ceo",
sessionId: "subagent-session",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:02:00.000Z",
content: [{ type: "text", text: "CEO workspace should stay in CEO dreams." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1270,25 +1296,16 @@ describe("memory-core dreaming phases", () => {
it("redacts sensitive session content before writing session corpus", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: "OPENAI_API_KEY=sk-1234567890abcdef" }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1337,49 +1354,30 @@ describe("memory-core dreaming phases", () => {
);
const corpus = await fs.readFile(corpusPath, "utf-8");
expect(corpus).not.toContain("OPENAI_API_KEY=sk-1234567890abcdef");
expect(corpus).toContain("OPENAI_API_KEY=sk-123…cdef");
expect(corpus).toContain("OPENAI_API_KEY=***");
});
it("skips dreaming-generated narrative transcripts during session ingestion", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-narrative.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "custom",
customType: "openclaw:bootstrap-context:full",
data: {
runId: "dreaming-narrative-light-1775894400455",
sessionId: "dream-session-1",
},
}),
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-narrative",
sessionKey: "agent:main:dreaming-narrative-light-1775894400455",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [
{ type: "text", text: "Write a dream diary entry from these memory fragments." },
],
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-05T18:02:00.000Z",
content: [{ type: "text", text: "I drift through the same archive again." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1435,46 +1433,24 @@ describe("memory-core dreaming phases", () => {
it("skips dreaming transcripts when the session store identifies them before bootstrap lands", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-narrative.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-narrative",
sessionKey: "agent:main:dreaming-narrative-light-1775894400455",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [
{ type: "text", text: "Write a dream diary entry from these memory fragments." },
],
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-05T18:02:00.000Z",
content: [{ type: "text", text: "I drift through the same archive again." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
await saveSessionStore(
path.join(sessionsDir, "sessions.json"),
{
"agent:main:dreaming-narrative-light-1775894400455": {
sessionId: "dreaming-narrative",
sessionFile: transcriptPath,
updatedAt: Date.parse("2026-04-05T18:05:00.000Z"),
},
},
{ skipMaintenance: true },
);
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1530,43 +1506,23 @@ describe("memory-core dreaming phases", () => {
it("skips isolated cron run transcripts during session ingestion", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "cron-run.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "cron-run",
sessionKey: "agent:main:cron:job-1:run:run-1",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content:
"[cron:job-1 Codex Sessions Sync] Run Codex sessions sync: 1. Convert sessions 2. Update qmd",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-05T18:02:00.000Z",
content: "Running Codex sessions sync...",
},
}),
].join("\n") + "\n",
"utf-8",
);
await saveSessionStore(
path.join(sessionsDir, "sessions.json"),
{
"agent:main:cron:job-1:run:run-1": {
sessionId: "cron-run",
sessionFile: transcriptPath,
updatedAt: Date.now(),
},
},
{ skipMaintenance: true },
);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1621,48 +1577,32 @@ describe("memory-core dreaming phases", () => {
it("drops generated system wrapper text without suppressing paired assistant replies", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "ordinary-session.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "ordinary-session",
messages: [
{
role: "user",
timestamp: "2026-04-16T18:01:00.000Z",
content:
"System (untrusted): [2026-04-16 11:01:00 PDT] Exec completed (quiet-fo, code 0) :: Converted: 1",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-16T18:01:30.000Z",
content: "Handled internally.",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "user",
timestamp: "2026-04-16T18:02:00.000Z",
content: "What changed in the sync?",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-16T18:03:00.000Z",
content: "One new session was converted.",
},
}),
].join("\n") + "\n",
"utf-8",
);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1756,61 +1696,42 @@ describe("memory-core dreaming phases", () => {
}) + "\n",
"utf-8",
);
await fs.writeFile(
path.join(sessionsDir, "ordinary.jsonl"),
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "ordinary",
messages: [
{
role: "user",
timestamp: "2026-04-16T18:04:00.000Z",
content:
"Read HEARTBEAT.md if it exists (workspace context). Follow it strictly. Do not infer or repeat old tasks from prior chats. If nothing needs attention, reply HEARTBEAT_OK.",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-16T18:05:00.000Z",
content: "HEARTBEAT_OK",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "user",
timestamp: "2026-04-16T18:06:00.000Z",
content: "[cron:job-2 Example] Run the qmd sync",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-16T18:07:00.000Z",
content: "Running the qmd sync now.",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "user",
timestamp: "2026-04-16T18:08:00.000Z",
content: "Document the Ollama provider setup.",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-16T18:09:00.000Z",
content: "I documented the Ollama provider setup in the workspace notes.",
},
}),
].join("\n") + "\n",
"utf-8",
);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1904,35 +1825,19 @@ describe("memory-core dreaming phases", () => {
it("does not reread unchanged dreaming-generated transcripts after checkpointing skip state", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-narrative.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "custom",
customType: "openclaw:bootstrap-context:full",
data: {
runId: "dreaming-narrative-light-1775894400455",
sessionId: "dream-session-1",
},
}),
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-narrative",
sessionKey: "agent:main:dreaming-narrative-light-1775894400455",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [
{ type: "text", text: "Write a dream diary entry from these memory fragments." },
],
},
}),
].join("\n") + "\n",
"utf-8",
);
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -1969,17 +1874,15 @@ describe("memory-core dreaming phases", () => {
{ cleanedBody: "__openclaw_memory_core_light_sleep__" },
{ trigger: "heartbeat", workspaceDir },
);
const firstSessionIngestion = await testing.readSessionIngestionState(workspaceDir);
const readFileSpy = vi.spyOn(fs, "readFile");
await beforeAgentReply(
{ cleanedBody: "__openclaw_memory_core_light_sleep__" },
{ trigger: "heartbeat", workspaceDir },
);
expect(readFileSpy.mock.calls.filter(([target]) => target === transcriptPath)).toEqual([]);
readFileSpy.mockRestore();
const secondSessionIngestion = await testing.readSessionIngestionState(workspaceDir);
expect(secondSessionIngestion).toStrictEqual(firstSessionIngestion);
} finally {
vi.restoreAllMocks();
restoreDreamingTestEnv();
}
});
@@ -1989,10 +1892,8 @@ describe("memory-core dreaming phases", () => {
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
const oldMessage = "Move backups to S3 Glacier.";
await fs.writeFile(
transcriptPath,
const oldArchiveContent =
[
JSON.stringify({
type: "message",
@@ -2002,11 +1903,17 @@ describe("memory-core dreaming phases", () => {
content: [{ type: "text", text: oldMessage }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const dayOne = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, dayOne, dayOne);
].join("\n") + "\n";
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: oldMessage }],
},
],
});
const { beforeAgentReply } = createHarness(
{
@@ -2046,32 +1953,19 @@ describe("memory-core dreaming phases", () => {
sessionsDir,
"dreaming-main.jsonl.reset.2026-04-06T01-00-00.000Z",
);
await fs.writeFile(resetPath, await fs.readFile(transcriptPath, "utf-8"), "utf-8");
await fs.writeFile(resetPath, oldArchiveContent, "utf-8");
const newMessage = "Keep retention at 365 days.";
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: oldMessage }],
},
}),
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "assistant",
timestamp: "2026-04-06T01:02:00.000Z",
content: [{ type: "text", text: newMessage }],
},
}),
].join("\n") + "\n",
"utf-8",
);
],
});
const dayTwo = new Date("2026-04-06T01:05:00.000Z");
await fs.utimes(transcriptPath, dayTwo, dayTwo);
await fs.utimes(resetPath, dayTwo, dayTwo);
await withDreamingTestClock(async () => {
@@ -2169,35 +2063,21 @@ describe("memory-core dreaming phases", () => {
it("buckets session snippets by per-message day rather than file mtime", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "user",
timestamp: "2026-04-01T12:00:00.000Z",
content: [
{ type: "text", text: "Old planning note that should stay out of lookback." },
],
content: [{ type: "text", text: "Old planning note that should stay out of lookback." }],
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: "2026-04-05T18:02:00.000Z",
content: [{ type: "text", text: "Current reminder that should be in today corpus." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const freshMtime = new Date("2026-04-06T01:05:00.000Z");
await fs.utimes(transcriptPath, freshMtime, freshMtime);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -2249,25 +2129,14 @@ describe("memory-core dreaming phases", () => {
it("drains >80 unseen transcript messages across multiple unchanged sweeps", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
const lines: string[] = [];
for (let index = 0; index < 160; index += 1) {
lines.push(
JSON.stringify({
type: "message",
message: {
role: index % 2 === 0 ? "user" : "assistant",
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: Array.from({ length: 160 }, (_, index) => ({
role: index % 2 === 0 ? ("user" as const) : ("assistant" as const),
timestamp: "2026-04-05T18:00:00.000Z",
content: [{ type: "text", text: `bulk-line-${index}` }],
},
}),
);
}
await fs.writeFile(transcriptPath, `${lines.join("\n")}\n`, "utf-8");
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
})),
});
const { beforeAgentReply } = createHarness(
{
@@ -2325,29 +2194,19 @@ describe("memory-core dreaming phases", () => {
expect(corpus).toContain("bulk-line-159");
});
it("re-ingests rewritten session transcripts after truncate/reset", async () => {
it("ingests appended SQLite session transcript rows after prior checkpoint", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: "Move backups to S3 Glacier." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const dayOne = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, dayOne, dayOne);
],
});
const { beforeAgentReply } = createHarness(
{
@@ -2383,22 +2242,16 @@ describe("memory-core dreaming phases", () => {
await triggerLightDreaming(beforeAgentReply, workspaceDir, 5);
});
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "assistant",
timestamp: "2026-04-06T01:02:00.000Z",
content: [{ type: "text", text: "Retention policy stays at 365 days." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const dayTwo = new Date("2026-04-06T01:05:00.000Z");
await fs.utimes(transcriptPath, dayTwo, dayTwo);
],
});
await withDreamingTestClock(async () => {
await triggerLightDreaming(beforeAgentReply, workspaceDir, 910);
@@ -2422,25 +2275,16 @@ describe("memory-core dreaming phases", () => {
it("ingests sessions when dreaming is enabled even if memorySearch is disabled", async () => {
const workspaceDir = await createDreamingWorkspace();
setDreamingTestEnv(path.join(workspaceDir, ".state"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "dreaming-main.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "message",
message: {
await seedDreamingSessionTranscript({
sessionId: "dreaming-main",
messages: [
{
role: "user",
timestamp: "2026-04-05T18:01:00.000Z",
content: [{ type: "text", text: "Glacier archive migration is now complete." }],
},
}),
].join("\n") + "\n",
"utf-8",
);
const mtime = new Date("2026-04-05T18:05:00.000Z");
await fs.utimes(transcriptPath, mtime, mtime);
],
});
const { beforeAgentReply } = createHarness(
{
extensions/memory-core/src/dreaming-phases.ts
+65 -10
View File
@@ -686,10 +686,16 @@ function hashSessionMessageId(value: string): string {
return createHash("sha1").update(value).digest("hex");
}
function buildSessionScopeKey(agentId: string, absolutePath: string): string {
function buildSessionScopeKey(agentId: string, sessionId: string): string {
const logicalSessionId =
parseUsageCountedSessionIdFromFileName(`${sessionId}.jsonl`) ?? sessionId;
return `${agentId}:${logicalSessionId}`;
}
function buildSessionFileScopeKey(agentId: string, absolutePath: string): string {
const fileName = path.basename(absolutePath);
const logicalSessionId = parseUsageCountedSessionIdFromFileName(fileName) ?? fileName;
return `${agentId}:${logicalSessionId}`;
return buildSessionScopeKey(agentId, logicalSessionId);
}
function mergeTrackedMessageHashes(existing: string[], additions: string[]): string[] {
@@ -722,8 +728,12 @@ function areStringArraysEqual(a: string[], b: string[]): boolean {
return true;
}
function buildSessionStateKey(agentId: string, absolutePath: string): string {
return `${agentId}:${sessionPathForFile(absolutePath)}`;
function buildSessionStateKey(agentId: string, sessionPath: string): string {
return `${agentId}:${sessionPath}`;
}
function buildSqliteDreamingSessionPath(agentId: string, sessionId: string): string {
return path.join("sessions", agentId, sessionId).replace(/\\/g, "/");
}
function isCheckpointSessionTranscriptPath(absolutePath: string): boolean {
@@ -852,7 +862,10 @@ async function collectSessionIngestionBatches(params: {
absolutePath: string;
generatedByDreamingNarrative: boolean;
generatedByCronRun: boolean;
sessionId: string;
sessionPath: string;
transcriptSource?: "sqlite";
updatedAtMs?: number;
}> = [];
for (const agentId of agentIds) {
for (const entry of await listSessionTranscriptCorpusEntriesForAgent(agentId)) {
@@ -870,7 +883,13 @@ async function collectSessionIngestionBatches(params: {
absolutePath,
generatedByDreamingNarrative: entry.generatedByDreamingNarrative === true,
generatedByCronRun: entry.generatedByCronRun === true,
sessionPath: sessionPathForFile(absolutePath),
sessionId: entry.sessionId,
sessionPath:
entry.transcriptSource === "sqlite"
? buildSqliteDreamingSessionPath(entry.agentId, entry.sessionId)
: sessionPathForFile(absolutePath),
...(entry.transcriptSource === "sqlite" ? { transcriptSource: "sqlite" as const } : {}),
...(entry.updatedAtMs !== undefined ? { updatedAtMs: entry.updatedAtMs } : {}),
});
}
}
@@ -896,8 +915,27 @@ async function collectSessionIngestionBatches(params: {
if (remaining <= 0) {
break;
}
const stateKey = buildSessionStateKey(file.agentId, file.absolutePath);
const stateKey = buildSessionStateKey(file.agentId, file.sessionPath);
const previous = params.state.files[stateKey];
let fingerprint: { mtimeMs: number; size: number };
let entry: Awaited<ReturnType<typeof buildSessionEntry>>;
if (file.transcriptSource === "sqlite") {
entry = await buildSessionEntry(file.absolutePath, {
generatedByDreamingNarrative: file.generatedByDreamingNarrative,
generatedByCronRun: file.generatedByCronRun,
...(file.updatedAtMs !== undefined ? { updatedAtMs: file.updatedAtMs } : {}),
});
if (!entry) {
if (previous) {
changed = true;
}
continue;
}
fingerprint = {
mtimeMs: Math.floor(Math.max(0, entry.mtimeMs)),
size: Math.floor(Math.max(0, entry.size)),
};
} else {
const stat = await fs.stat(file.absolutePath).catch((err: unknown) => {
if ((err as NodeJS.ErrnoException)?.code === "ENOENT") {
return null;
@@ -910,7 +948,7 @@ async function collectSessionIngestionBatches(params: {
}
continue;
}
const fingerprint = {
fingerprint = {
mtimeMs: Math.floor(Math.max(0, stat.mtimeMs)),
size: Math.floor(Math.max(0, stat.size)),
};
@@ -926,13 +964,14 @@ async function collectSessionIngestionBatches(params: {
continue;
}
const entry = await buildSessionEntry(file.absolutePath, {
entry = await buildSessionEntry(file.absolutePath, {
generatedByDreamingNarrative: file.generatedByDreamingNarrative,
generatedByCronRun: file.generatedByCronRun,
});
if (!entry) {
continue;
}
}
if (entry.generatedByDreamingNarrative || entry.generatedByCronRun) {
nextFiles[stateKey] = {
mtimeMs: fingerprint.mtimeMs,
@@ -966,9 +1005,19 @@ async function collectSessionIngestionBatches(params: {
continue;
}
const sessionScope = buildSessionScopeKey(file.agentId, file.absolutePath);
const sessionScope =
file.transcriptSource === "sqlite"
? `${file.agentId}:${file.sessionPath}`
: buildSessionFileScopeKey(file.agentId, file.absolutePath);
const preFlipSessionScope =
file.transcriptSource === "sqlite"
? buildSessionScopeKey(file.agentId, file.sessionId)
: undefined;
const previousSeen = nextSeenMessages[sessionScope] ?? [];
const seenSet = new Set(previousSeen);
const preFlipSeenSet = preFlipSessionScope
? new Set(nextSeenMessages[preFlipSessionScope] ?? [])
: null;
const newSeenHashes: string[] = [];
const lines = entry.content.length > 0 ? entry.content.split("\n") : [];
@@ -1007,7 +1056,13 @@ async function collectSessionIngestionBatches(params: {
const dedupeBasis =
messageTimestampMs > 0 ? `ts:${Math.floor(messageTimestampMs)}` : `line:${lineNumber}`;
const messageHash = hashSessionMessageId(`${sessionScope}\n${dedupeBasis}\n${snippet}`);
if (seenSet.has(messageHash)) {
const preFlipMessageHash = preFlipSessionScope
? hashSessionMessageId(`${preFlipSessionScope}\n${dedupeBasis}\n${snippet}`)
: undefined;
if (
seenSet.has(messageHash) ||
(preFlipMessageHash !== undefined && preFlipSeenSet?.has(preFlipMessageHash))
) {
continue;
}
const rendered = buildSessionRenderedLine({
extensions/memory-core/src/memory/index.test.ts
+92 -119
View File
@@ -7,6 +7,8 @@ import type { DatabaseSync } from "node:sqlite";
import { clearMemoryEmbeddingProviders as clearRegistry } from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
import { hashText } from "openclaw/plugin-sdk/memory-core-host-engine-storage";
import { resolveSessionTranscriptsDirForAgent } from "openclaw/plugin-sdk/memory-core-host-runtime-core";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import { resolveOpenClawAgentSqlitePath } from "openclaw/plugin-sdk/sqlite-runtime";
import {
closeOpenClawAgentDatabasesForTest,
@@ -417,6 +419,46 @@ describe("memory index", () => {
};
}
async function seedMemoryIndexSessionTranscript(params: {
messages: Array<{
content: string;
role: "assistant" | "user";
timestamp: number | string;
}>;
sessionId: string;
sessionKey?: string;
}): Promise<void> {
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
const storePath = path.join(sessionsDir, "sessions.json");
const sessionKey = params.sessionKey ?? `agent:main:memory:${params.sessionId}`;
const timestamps = params.messages
.map((message) =>
typeof message.timestamp === "number" ? message.timestamp : Date.parse(message.timestamp),
)
.filter((timestamp) => Number.isFinite(timestamp));
const updatedAt = timestamps.length > 0 ? Math.max(...timestamps) : Date.now();
await fs.mkdir(sessionsDir, { recursive: true });
await upsertSessionEntry({
agentId: "main",
sessionKey,
storePath,
entry: { sessionId: params.sessionId, updatedAt },
});
for (const message of params.messages) {
await appendSessionTranscriptMessageByIdentity({
agentId: "main",
sessionId: params.sessionId,
sessionKey,
storePath,
message: {
role: message.role,
timestamp: message.timestamp,
content: [{ type: "text", text: message.content }],
},
});
}
}
function requireManager(
result: Awaited<ReturnType<typeof getMemorySearchManager>>,
missingMessage = "manager missing",
@@ -881,46 +923,26 @@ describe("memory index", () => {
it("batches forced memory and session indexing across files", async () => {
await fs.writeFile(path.join(memoryDir, "2026-01-13.md"), "# Log\nBeta memory line.");
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.writeFile(
path.join(sessionsDir, "session-alpha.jsonl"),
[
JSON.stringify({
type: "session",
id: "session-alpha",
timestamp: "2026-04-07T15:24:04.113Z",
}),
JSON.stringify({
type: "message",
message: {
await seedMemoryIndexSessionTranscript({
sessionId: "session-alpha",
messages: [
{
role: "user",
timestamp: "2026-04-07T15:25:04.113Z",
content: [{ type: "text", text: "Session alpha memory line." }],
content: "Session alpha memory line.",
},
}),
].join("\n") + "\n",
"utf8",
);
await fs.writeFile(
path.join(sessionsDir, "session-beta.jsonl"),
[
JSON.stringify({
type: "session",
id: "session-beta",
timestamp: "2026-04-07T15:24:04.113Z",
}),
JSON.stringify({
type: "message",
message: {
],
});
await seedMemoryIndexSessionTranscript({
sessionId: "session-beta",
messages: [
{
role: "assistant",
timestamp: "2026-04-07T15:25:04.113Z",
content: [{ type: "text", text: "Session beta memory line." }],
content: "Session beta memory line.",
},
}),
].join("\n") + "\n",
"utf8",
);
],
});
const cfg = createCfg({
provider: "batch-wide-test",
batchEnabled: true,
@@ -1216,27 +1238,16 @@ describe("memory index", () => {
it("clears dirty after sessions-only identity reindex", async () => {
try {
setMemoryIndexStateDir(path.join(workspaceDir, ".state-sessions-only-reindex"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.writeFile(
path.join(sessionsDir, "session-identity.jsonl"),
[
JSON.stringify({
type: "session",
id: "session-identity",
timestamp: "2026-04-07T15:24:04.113Z",
}),
JSON.stringify({
type: "message",
message: {
await seedMemoryIndexSessionTranscript({
sessionId: "session-identity",
messages: [
{
role: "assistant",
timestamp: "2026-04-07T15:25:04.113Z",
content: [{ type: "text", text: "Session-only identity marker." }],
content: "Session-only identity marker.",
},
}),
].join("\n") + "\n",
"utf8",
);
],
});
const oldCfg = createCfg({
sources: ["sessions"],
@@ -1272,27 +1283,16 @@ describe("memory index", () => {
it("marks sessions-only indexes dirty when metadata is missing but chunks exist", async () => {
try {
setMemoryIndexStateDir(path.join(workspaceDir, ".state-sessions-missing-meta"));
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.writeFile(
path.join(sessionsDir, "session-missing-meta.jsonl"),
[
JSON.stringify({
type: "session",
id: "session-missing-meta",
timestamp: "2026-04-07T15:24:04.113Z",
}),
JSON.stringify({
type: "message",
message: {
await seedMemoryIndexSessionTranscript({
sessionId: "session-missing-meta",
messages: [
{
role: "assistant",
timestamp: "2026-04-07T15:25:04.113Z",
content: [{ type: "text", text: "Sessions missing metadata marker." }],
content: "Sessions missing metadata marker.",
},
}),
].join("\n") + "\n",
"utf8",
);
],
});
const cfg = createCfg({
sources: ["sessions"],
@@ -1371,7 +1371,7 @@ describe("memory index", () => {
expect(nextManager.status().dirty).toBe(true);
embedBatchCalls = 0;
await nextManager.sync({ reason: "test", sessionFiles: [sessionFile] });
await nextManager.sync({ reason: "test", archiveFiles: [sessionFile] });
expect(embedBatchCalls).toBe(0);
expect(nextManager.status().dirty).toBe(true);
@@ -1433,12 +1433,12 @@ describe("memory index", () => {
try {
const fields = nextManager as unknown as {
dirty: boolean;
syncSessionFiles: (params: unknown) => Promise<void>;
syncArchiveFiles: (params: unknown) => Promise<void>;
};
const syncSessionFiles = fields.syncSessionFiles.bind(nextManager);
fields.syncSessionFiles = async (params) => {
const syncArchiveFiles = fields.syncArchiveFiles.bind(nextManager);
fields.syncArchiveFiles = async (params) => {
fields.dirty = true;
await syncSessionFiles(params);
await syncArchiveFiles(params);
};
await nextManager.sync({ reason: "test", force: true });
@@ -1514,7 +1514,7 @@ describe("memory index", () => {
runSyncWithReadonlyRecovery: (params?: {
reason?: string;
force?: boolean;
sessionFiles?: string[];
archiveFiles?: string[];
progress?: (update: unknown) => void;
}) => Promise<void>;
}
@@ -2864,37 +2864,22 @@ describe("memory index", () => {
const staleAt = new Date("2020-01-01T00:00:00.000Z");
await fs.utimes(memoryPath, staleAt, staleAt);
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "session-ranking.jsonl");
const now = Date.parse("2026-04-07T15:25:04.113Z");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "session",
id: "session-ranking",
timestamp: new Date(now - 60_000).toISOString(),
}),
JSON.stringify({
type: "message",
message: {
await seedMemoryIndexSessionTranscript({
sessionId: "session-ranking",
messages: [
{
role: "user",
timestamp: new Date(now - 30_000).toISOString(),
content: [{ type: "text", text: "What is the current Project Nebula codename?" }],
content: "What is the current Project Nebula codename?",
},
}),
JSON.stringify({
type: "message",
message: {
{
role: "assistant",
timestamp: new Date(now).toISOString(),
content: [{ type: "text", text: "The current Project Nebula codename is ORBIT-10." }],
content: "The current Project Nebula codename is ORBIT-10.",
},
}),
].join("\n") + "\n",
"utf8",
);
],
});
await manager.sync({ reason: "test", force: true });
const results = await manager.search("current Project Nebula codename ORBIT-10", {
@@ -2918,28 +2903,16 @@ describe("memory index", () => {
return;
}
const sessionsDir = resolveSessionTranscriptsDirForAgent("main");
await fs.mkdir(sessionsDir, { recursive: true });
const transcriptPath = path.join(sessionsDir, "session-bootstrap.jsonl");
await fs.writeFile(
transcriptPath,
[
JSON.stringify({
type: "session",
id: "session-bootstrap",
timestamp: "2026-04-07T15:24:04.113Z",
}),
JSON.stringify({
type: "message",
message: {
await seedMemoryIndexSessionTranscript({
sessionId: "session-bootstrap",
messages: [
{
role: "assistant",
timestamp: "2026-04-07T15:25:04.113Z",
content: [{ type: "text", text: "The current Project Nebula codename is ORBIT-10." }],
content: "The current Project Nebula codename is ORBIT-10.",
},
}),
].join("\n") + "\n",
"utf8",
);
],
});
const results = await manager.search("current Project Nebula codename ORBIT-10", {
minScore: 0,
extensions/memory-core/src/memory/manager-session-reindex.ts
+1 -1
View File
@@ -15,7 +15,7 @@ export function shouldSyncSessionsForReindex(params: {
if (params.sync?.sessions?.some((session) => session.sessionId.trim().length > 0)) {
return true;
}
if (params.sync?.sessionFiles?.some((sessionFile) => sessionFile.trim().length > 0)) {
if (params.sync?.archiveFiles?.some((sessionFile) => sessionFile.trim().length > 0)) {
return true;
}
if (params.sync?.force) {
extensions/memory-core/src/memory/manager-sync-control.ts
+11 -11
View File
@@ -24,7 +24,7 @@ export type MemoryReadonlyRecoveryState = {
reason?: string;
force?: boolean;
sessions?: MemorySessionSyncTarget[];
sessionFiles?: string[];
archiveFiles?: string[];
progress?: (update: MemorySyncProgressUpdate) => void;
}) => Promise<void>;
openDatabase: () => DatabaseSync;
@@ -120,19 +120,19 @@ export function enqueueMemoryTargetedSessionSync(
state: {
isClosed: () => boolean;
getSyncing: () => Promise<void> | null;
getQueuedSessionFiles: () => Set<string>;
getQueuedArchiveFiles: () => Set<string>;
getQueuedSessions: () => Map<string, MemorySessionSyncTarget>;
getQueuedSessionSync: () => Promise<void> | null;
setQueuedSessionSync: (value: Promise<void> | null) => void;
sync: (params?: MemorySyncParams) => Promise<void>;
},
targets?: Pick<MemorySyncParams, "sessions" | "sessionFiles">,
targets?: Pick<MemorySyncParams, "sessions" | "archiveFiles">,
): Promise<void> {
const queuedSessionFiles = state.getQueuedSessionFiles();
for (const sessionFile of targets?.sessionFiles ?? []) {
const queuedArchiveFiles = state.getQueuedArchiveFiles();
for (const sessionFile of targets?.archiveFiles ?? []) {
const trimmed = sessionFile.trim();
if (trimmed) {
queuedSessionFiles.add(trimmed);
queuedArchiveFiles.add(trimmed);
}
}
const queuedSessions = state.getQueuedSessions();
@@ -142,7 +142,7 @@ export function enqueueMemoryTargetedSessionSync(
queuedSessions.set(memorySessionSyncTargetKey(normalized), normalized);
}
}
if (queuedSessionFiles.size === 0 && queuedSessions.size === 0) {
if (queuedArchiveFiles.size === 0 && queuedSessions.size === 0) {
return state.getSyncing() ?? Promise.resolve();
}
if (!state.getQueuedSessionSync()) {
@@ -152,16 +152,16 @@ export function enqueueMemoryTargetedSessionSync(
await state.getSyncing()?.catch(() => undefined);
while (
!state.isClosed() &&
(state.getQueuedSessionFiles().size > 0 || state.getQueuedSessions().size > 0)
(state.getQueuedArchiveFiles().size > 0 || state.getQueuedSessions().size > 0)
) {
const pendingSessionFiles = Array.from(state.getQueuedSessionFiles());
const pendingArchiveFiles = Array.from(state.getQueuedArchiveFiles());
const pendingSessions = Array.from(state.getQueuedSessions().values());
state.getQueuedSessionFiles().clear();
state.getQueuedArchiveFiles().clear();
state.getQueuedSessions().clear();
await state.sync({
reason: "queued-sessions",
sessions: pendingSessions,
sessionFiles: pendingSessionFiles,
archiveFiles: pendingArchiveFiles,
});
}
} finally {
extensions/memory-core/src/memory/manager-sync-ops.startup-catchup.test.ts
+206 -215
View File
@@ -3,12 +3,12 @@ import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import type { DatabaseSync } from "node:sqlite";
import { emitSessionTranscriptUpdate } from "openclaw/plugin-sdk/agent-harness-runtime";
import {
resolveSessionTranscriptsDirForAgent,
type OpenClawConfig,
type ResolvedMemorySearchConfig,
} from "openclaw/plugin-sdk/memory-core-host-engine-foundation";
import { statSessionEntrySync } from "openclaw/plugin-sdk/memory-core-host-engine-qmd";
import type {
MemorySource,
MemorySyncParams,
@@ -18,6 +18,12 @@ import {
clearConfigCache,
clearRuntimeConfigSnapshot,
} from "openclaw/plugin-sdk/runtime-config-snapshot";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import {
closeOpenClawAgentDatabasesForTest,
formatSqliteSessionFileMarker,
} from "openclaw/plugin-sdk/sqlite-runtime-testing";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { MemoryManagerSyncOps } from "./manager-sync-ops.js";
@@ -34,7 +40,7 @@ type SyncParams = {
reason?: string;
force?: boolean;
sessions?: MemorySyncParams["sessions"];
sessionFiles?: string[];
archiveFiles?: string[];
progress?: (update: MemorySyncProgressUpdate) => void;
};
@@ -49,15 +55,9 @@ type MemorySessionTranscriptUpdate = {
};
};
type MemoryTranscriptUpdateSubscriber = (
listener: (update: MemorySessionTranscriptUpdate) => void,
) => () => void;
const MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY = Symbol.for(
"openclaw.memoryCore.sessionTranscriptUpdateSubscriber",
);
const originalStartupStateDir = process.env.OPENCLAW_STATE_DIR;
const originalStartupConfigPath = process.env.OPENCLAW_CONFIG_PATH;
let transcriptUpdateListener: ((update: MemorySessionTranscriptUpdate) => void) | undefined;
type SourceStateRow = { path: string; hash: string; mtime: number; size: number };
@@ -82,6 +82,10 @@ function restoreStartupEnv(): void {
}
}
function emitSessionTranscriptUpdate(update: MemorySessionTranscriptUpdate): void {
transcriptUpdateListener?.(update);
}
class SessionStartupCatchupHarness extends MemoryManagerSyncOps {
protected readonly cfg = {} as OpenClawConfig;
protected readonly agentId = "main";
@@ -155,7 +159,7 @@ class SessionStartupCatchupHarness extends MemoryManagerSyncOps {
await this.runSync(params);
}
getDirtySessionFiles(): string[] {
getDirtyArchiveFiles(): string[] {
return Array.from(this.sessionsDirtyFiles);
}
@@ -163,7 +167,7 @@ class SessionStartupCatchupHarness extends MemoryManagerSyncOps {
return Array.from(this.sessionPendingTargets.values());
}
getPendingSessionFiles(): string[] {
getPendingArchiveFiles(): string[] {
return Array.from(this.sessionPendingFiles);
}
@@ -182,18 +186,18 @@ class SessionStartupCatchupHarness extends MemoryManagerSyncOps {
).processSessionDeltaBatch();
}
async combineTargetSessionFilesForTest(params: {
async combineTargetArchiveFilesForTest(params: {
sessions?: MemorySyncParams["sessions"];
sessionFiles?: string[];
archiveFiles?: string[];
}): Promise<Set<string> | null> {
return await (
this as unknown as {
combineTargetSessionFiles: (params: {
combineTargetArchiveFiles: (params: {
sessions?: MemorySyncParams["sessions"];
sessionFiles?: string[];
archiveFiles?: string[];
}) => Promise<Set<string> | null>;
}
).combineTargetSessionFiles(params);
).combineTargetArchiveFiles(params);
}
isSessionsDirty(): boolean {
@@ -209,6 +213,17 @@ class SessionStartupCatchupHarness extends MemoryManagerSyncOps {
this.sessionUnsubscribe = null;
}
protected override subscribeSessionTranscriptUpdates(
listener: (update: MemorySessionTranscriptUpdate) => void,
): () => void {
transcriptUpdateListener = listener;
return () => {
if (transcriptUpdateListener === listener) {
transcriptUpdateListener = undefined;
}
};
}
protected computeProviderKey(): string {
return "test";
}
@@ -254,14 +269,17 @@ describe("session startup catch-up", () => {
beforeEach(async () => {
stateDir = await fs.mkdtemp(path.join(os.tmpdir(), "openclaw-session-startup-"));
setStartupStateDir(stateDir);
transcriptUpdateListener = undefined;
});
afterEach(async () => {
vi.clearAllTimers();
vi.useRealTimers();
transcriptUpdateListener = undefined;
restoreStartupEnv();
clearRuntimeConfigSnapshot();
clearConfigCache();
closeOpenClawAgentDatabasesForTest();
await fs.rm(stateDir, { recursive: true, force: true });
});
@@ -281,25 +299,90 @@ describe("session startup catch-up", () => {
return { filePath, size: stat.size, mtimeMs: stat.mtimeMs };
}
async function configureTestSessionStore(storePath: string): Promise<void> {
const configPath = path.join(stateDir, "openclaw.json");
await fs.mkdir(path.dirname(storePath), { recursive: true });
await fs.writeFile(configPath, JSON.stringify({ session: { store: storePath } }), "utf-8");
setStartupConfigPath(configPath);
clearRuntimeConfigSnapshot();
clearConfigCache();
}
async function writeSqliteSession(
params: {
storePath?: string;
sessionId?: string;
sessionKey?: string;
content?: string;
role?: "assistant" | "user";
updatedAt?: number;
} = {},
): Promise<{
marker: string;
storePath: string;
sessionId: string;
sessionKey: string;
corpusPath: string;
}> {
const storePath =
params.storePath ?? path.join(stateDir, "agents", "main", "sessions", "sessions.json");
const sessionId = params.sessionId ?? "thread";
const sessionKey = params.sessionKey ?? `agent:main:chat:${sessionId}`;
const marker = formatSqliteSessionFileMarker({
agentId: "main",
sessionId,
storePath,
});
await configureTestSessionStore(storePath);
await upsertSessionEntry({
agentId: "main",
sessionKey,
storePath,
entry: {
sessionFile: marker,
sessionId,
updatedAt: params.updatedAt ?? 10,
},
});
await appendSessionTranscriptMessageByIdentity({
agentId: "main",
sessionId,
sessionKey,
storePath,
cwd: stateDir,
message: {
role: params.role ?? "user",
content: params.content ?? "startup catchup",
},
});
return {
marker,
storePath,
sessionId,
sessionKey,
corpusPath: `sessions/main/${sessionId}.jsonl`,
};
}
it("marks stale indexed session files dirty and schedules catch-up sync", async () => {
const session = await writeSessionFile("thread.jsonl");
const session = await writeSqliteSession();
const harness = new SessionStartupCatchupHarness([
{
path: "sessions/main/thread.jsonl",
path: session.corpusPath,
hash: "old-hash",
mtime: session.mtimeMs - 1000,
size: session.size,
mtime: 0,
size: 0,
},
]);
await expect(harness.catchUp()).resolves.toEqual([session.filePath]);
expect(harness.getDirtySessionFiles()).toEqual([session.filePath]);
await expect(harness.catchUp()).resolves.toEqual([session.marker]);
expect(harness.getDirtyArchiveFiles()).toEqual([session.marker]);
expect(harness.isSessionsDirty()).toBe(true);
expect(harness.syncCalls).toEqual([{ reason: "session-startup-catchup" }]);
});
it("retries transient session transcript reads during session indexing", async () => {
const session = await writeSessionFile("thread.jsonl");
const session = await writeSessionFile("thread.jsonl.deleted.2026-02-16T22-27-33.000Z");
const harness = new SessionStartupCatchupHarness([]);
const realOpen = fs.open;
@@ -324,7 +407,7 @@ describe("session startup catch-up", () => {
});
try {
await (harness as any).syncSessionFiles({ needsFullReindex: true });
await (harness as any).syncArchiveFiles({ needsFullReindex: true });
expect(attempts).toBe(2);
} finally {
openSpy.mockRestore();
@@ -332,22 +415,46 @@ describe("session startup catch-up", () => {
});
it("can mark startup catch-up files without scheduling background sync", async () => {
const session = await writeSessionFile("thread.jsonl");
const session = await writeSqliteSession();
const harness = new SessionStartupCatchupHarness([
{
path: "sessions/main/thread.jsonl",
path: session.corpusPath,
hash: "old-hash",
mtime: session.mtimeMs - 1000,
size: session.size,
mtime: 0,
size: 0,
},
]);
await expect(harness.markStartupDirtyFiles()).resolves.toEqual([session.filePath]);
expect(harness.getDirtySessionFiles()).toEqual([session.filePath]);
await expect(harness.markStartupDirtyFiles()).resolves.toEqual([session.marker]);
expect(harness.getDirtyArchiveFiles()).toEqual([session.marker]);
expect(harness.isSessionsDirty()).toBe(true);
expect(harness.syncCalls).toEqual([]);
});
it("leaves unchanged indexed SQLite sessions clean during startup catch-up", async () => {
const session = await writeSqliteSession({ updatedAt: 10 });
const state = statSessionEntrySync(session.marker, {
sessionKey: session.sessionKey,
updatedAtMs: 10,
});
if (!state) {
throw new Error("expected SQLite transcript state");
}
const harness = new SessionStartupCatchupHarness([
{
path: state.path,
hash: "current-hash",
mtime: state.mtimeMs,
size: state.size,
},
]);
await expect(harness.markStartupDirtyFiles()).resolves.toEqual([]);
expect(harness.getDirtyArchiveFiles()).toEqual([]);
expect(harness.isSessionsDirty()).toBe(false);
expect(harness.syncCalls).toEqual([]);
});
it("leaves unchanged indexed session files clean", async () => {
const session = await writeSessionFile("thread.jsonl");
const harness = new SessionStartupCatchupHarness([
@@ -360,7 +467,7 @@ describe("session startup catch-up", () => {
]);
await expect(harness.catchUp()).resolves.toEqual([]);
expect(harness.getDirtySessionFiles()).toEqual([]);
expect(harness.getDirtyArchiveFiles()).toEqual([]);
expect(harness.isSessionsDirty()).toBe(false);
expect(harness.syncCalls).toEqual([]);
});
@@ -457,33 +564,13 @@ describe("session startup catch-up", () => {
});
it("resolves identity-targeted delta sync through a custom session store", async () => {
const storeDir = path.join(stateDir, "custom-sessions");
const sessionFile = path.join(storeDir, "custom-thread.jsonl");
const storePath = path.join(storeDir, "sessions.json");
const configPath = path.join(stateDir, "openclaw.json");
await fs.mkdir(storeDir, { recursive: true });
await fs.writeFile(
sessionFile,
JSON.stringify({
type: "message",
message: { role: "user", content: "custom store target" },
}) + "\n",
"utf-8",
);
await fs.writeFile(
const storePath = path.join(stateDir, "custom-sessions", "sessions.json");
const session = await writeSqliteSession({
storePath,
JSON.stringify({
"agent:main:chat:custom": {
sessionFile: "custom-thread.jsonl",
sessionId: "custom-thread",
},
}),
"utf-8",
);
await fs.writeFile(configPath, JSON.stringify({ session: { store: storePath } }), "utf-8");
setStartupConfigPath(configPath);
clearRuntimeConfigSnapshot();
clearConfigCache();
sessionKey: "agent:main:chat:custom",
content: "custom store target",
});
const harness = new SessionStartupCatchupHarness([]);
(harness as unknown as { settings: ResolvedMemorySearchConfig }).settings.sync.sessions = {
deltaBytes: 1,
@@ -499,125 +586,100 @@ describe("session startup catch-up", () => {
await harness.processPendingSessionDeltas();
await Promise.resolve();
expect(harness.getDirtySessionFiles()).toEqual([sessionFile]);
expect(harness.getDirtyArchiveFiles()).toEqual([session.marker]);
expect(harness.syncCalls).toEqual([{ reason: "session-delta" }]);
});
it("keeps explicit custom-store session file targets at the sync gate", async () => {
const storeDir = path.join(stateDir, "custom-sessions");
const sessionFile = path.join(storeDir, "explicit-target.jsonl");
const storePath = path.join(storeDir, "sessions.json");
const configPath = path.join(stateDir, "openclaw.json");
await fs.mkdir(storeDir, { recursive: true });
await fs.writeFile(
sessionFile,
JSON.stringify({
type: "message",
message: { role: "user", content: "explicit target" },
}) + "\n",
"utf-8",
);
await fs.writeFile(
it("preserves generated-session classification during targeted custom-store indexing", async () => {
const storePath = path.join(stateDir, "custom-sessions", "sessions.json");
const session = await writeSqliteSession({
storePath,
JSON.stringify({
"agent:main:chat:explicit-target": {
sessionFile: "explicit-target.jsonl",
sessionId: "explicit-target",
},
}),
"utf-8",
);
await fs.writeFile(configPath, JSON.stringify({ session: { store: storePath } }), "utf-8");
setStartupConfigPath(configPath);
clearRuntimeConfigSnapshot();
clearConfigCache();
sessionId: "cron-thread",
sessionKey: "agent:main:cron:job-1:run:run-1",
role: "assistant",
content: "Internal cron output that must stay out.",
});
await writeSqliteSession({
storePath,
sessionId: "other-thread",
sessionKey: "agent:main:chat:other",
content: "Other custom-store content",
});
const harness = new SessionStartupCatchupHarness([]);
await expect(
harness.combineTargetSessionFilesForTest({ sessionFiles: [sessionFile] }),
).resolves.toEqual(new Set([sessionFile]));
await (
harness as unknown as {
syncArchiveFiles: (params: {
needsFullReindex: boolean;
targetArchiveFiles: string[];
}) => Promise<void>;
}
).syncArchiveFiles({
needsFullReindex: false,
targetArchiveFiles: [session.marker],
});
it("preserves generated-session classification during targeted custom-store indexing", async () => {
const storeDir = path.join(stateDir, "custom-sessions");
const sessionFile = path.join(storeDir, "cron-thread.jsonl");
const otherSessionFile = path.join(storeDir, "other-thread.jsonl");
const storePath = path.join(storeDir, "sessions.json");
expect(harness.indexedPaths).toEqual([session.corpusPath]);
expect(harness.indexedContents).toEqual([""]);
});
it("keeps targeted SQLite corpus markers during archive-file sync", async () => {
const storePath = path.join(stateDir, "agents", "main", "sessions", "sessions.json");
const configPath = path.join(stateDir, "openclaw.json");
await fs.mkdir(storeDir, { recursive: true });
await fs.writeFile(
sessionFile,
JSON.stringify({
type: "message",
message: { role: "assistant", content: "Internal cron output that must stay out." },
}) + "\n",
"utf-8",
);
await fs.writeFile(
otherSessionFile,
JSON.stringify({
type: "message",
message: { role: "user", content: "Other custom-store content" },
}) + "\n",
"utf-8",
);
await fs.writeFile(
const sessionId = "sqlite-target";
const sessionKey = "agent:main:chat:sqlite-target";
const marker = formatSqliteSessionFileMarker({
agentId: "main",
sessionId,
storePath,
JSON.stringify({
"agent:main:cron:job-1:run:run-1": {
sessionFile: "cron-thread.jsonl",
sessionId: "cron-thread",
});
await upsertSessionEntry({
agentId: "main",
sessionKey,
storePath,
entry: {
sessionFile: marker,
sessionId,
updatedAt: 10,
},
"agent:main:chat:other": {
sessionFile: "other-thread.jsonl",
sessionId: "other-thread",
},
}),
"utf-8",
);
});
await appendSessionTranscriptMessageByIdentity({
agentId: "main",
sessionId,
sessionKey,
storePath,
cwd: stateDir,
message: { role: "user", content: "sqlite targeted memory content" },
});
await fs.writeFile(configPath, JSON.stringify({ session: { store: storePath } }), "utf-8");
setStartupConfigPath(configPath);
vi.stubEnv("OPENCLAW_CONFIG_PATH", configPath);
clearRuntimeConfigSnapshot();
clearConfigCache();
const harness = new SessionStartupCatchupHarness([]);
await (
harness as unknown as {
syncSessionFiles: (params: {
syncArchiveFiles: (params: {
needsFullReindex: boolean;
targetSessionFiles: string[];
targetArchiveFiles: string[];
}) => Promise<void>;
}
).syncSessionFiles({
).syncArchiveFiles({
needsFullReindex: false,
targetSessionFiles: [sessionFile],
targetArchiveFiles: [marker],
});
expect(harness.indexedPaths).toEqual(["sessions/cron-thread.jsonl"]);
expect(harness.indexedContents).toEqual([""]);
expect(harness.indexedPaths).toEqual(["sessions/main/sqlite-target.jsonl"]);
expect(harness.indexedContents[0]).toContain("sqlite targeted memory content");
});
it("queues transcript update identity without requiring a session file", async () => {
vi.useFakeTimers();
const harness = new SessionStartupCatchupHarness([]);
const originalSubscriber = (globalThis as Record<symbol, unknown>)[
MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY
];
let transcriptListener: ((update: MemorySessionTranscriptUpdate) => void) | undefined;
(globalThis as Record<symbol, unknown>)[MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY] = ((
listener,
) => {
transcriptListener = listener;
return () => {
if (transcriptListener === listener) {
transcriptListener = undefined;
}
};
}) satisfies MemoryTranscriptUpdateSubscriber;
harness.startTranscriptListener();
try {
transcriptListener?.({
emitSessionTranscriptUpdate({
target: {
agentId: "main",
sessionId: "thread",
@@ -630,14 +692,6 @@ describe("session startup catch-up", () => {
]);
} finally {
harness.stopTranscriptListener();
if (originalSubscriber === undefined) {
delete (globalThis as Record<symbol, unknown>)[
MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY
];
} else {
(globalThis as Record<symbol, unknown>)[MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY] =
originalSubscriber;
}
}
});
@@ -652,70 +706,7 @@ describe("session startup catch-up", () => {
sessionKey: "agent:main:thread",
});
expect(harness.getPendingSessionFiles()).toEqual([session.filePath]);
expect(harness.getPendingSessionTargets()).toEqual([]);
harness.stopTranscriptListener();
});
it("queues file-only transcript updates from a custom session store", async () => {
vi.useFakeTimers();
const storeDir = path.join(stateDir, "custom-sessions");
const sessionFile = path.join(storeDir, "custom-update.jsonl");
const storePath = path.join(storeDir, "sessions.json");
const configPath = path.join(stateDir, "openclaw.json");
await fs.mkdir(storeDir, { recursive: true });
await fs.writeFile(
sessionFile,
JSON.stringify({
type: "message",
message: { role: "user", content: "custom update" },
}) + "\n",
"utf-8",
);
await fs.writeFile(
storePath,
JSON.stringify({
"agent:main:chat:custom-update": {
sessionFile: "custom-update.jsonl",
sessionId: "custom-update",
},
}),
"utf-8",
);
await fs.writeFile(configPath, JSON.stringify({ session: { store: storePath } }), "utf-8");
setStartupConfigPath(configPath);
clearRuntimeConfigSnapshot();
clearConfigCache();
const harness = new SessionStartupCatchupHarness([]);
harness.startTranscriptListener();
emitSessionTranscriptUpdate({
sessionFile,
sessionKey: "agent:main:chat:custom-update",
});
await Promise.resolve();
expect(harness.getPendingSessionFiles()).toEqual([sessionFile]);
expect(harness.getPendingSessionTargets()).toEqual([]);
harness.stopTranscriptListener();
});
it("prefers transcript update path compatibility before identity", async () => {
vi.useFakeTimers();
const session = await writeSessionFile("thread.jsonl");
const harness = new SessionStartupCatchupHarness([]);
harness.startTranscriptListener();
emitSessionTranscriptUpdate({
sessionFile: session.filePath,
target: {
agentId: "main",
sessionId: "identity-target",
sessionKey: "agent:main:identity-target",
},
});
expect(harness.getPendingSessionFiles()).toEqual([session.filePath]);
expect(harness.getPendingArchiveFiles()).toEqual([session.filePath]);
expect(harness.getPendingSessionTargets()).toEqual([]);
harness.stopTranscriptListener();
});
extensions/memory-core/src/memory/manager-sync-ops.ts
+186 -98
View File
@@ -9,7 +9,7 @@ import { formatErrorMessage } from "openclaw/plugin-sdk/error-runtime";
import { classifyMemoryMultimodalPath } from "openclaw/plugin-sdk/memory-core-host-engine-embeddings";
import {
createSubsystemLogger,
onSessionTranscriptUpdate,
onInternalSessionTranscriptUpdate,
resolveAgentDir,
resolveSessionTranscriptsDirForAgent,
resolveUserPath,
@@ -20,11 +20,14 @@ import {
buildSessionEntry,
isSessionArchiveArtifactName,
isUsageCountedSessionTranscriptFileName,
listSessionFilesForAgent,
listSessionTranscriptCorpusEntriesForAgent,
parseCanonicalSessionSyncTargetFromPath,
parseSqliteSessionFileMarker,
parseUsageCountedSessionIdFromFileName,
resolveSessionFileForSyncTarget,
sessionPathForFile,
sessionPathForSessionIdentity,
statSessionEntrySync,
type SessionTranscriptCorpusEntry,
} from "openclaw/plugin-sdk/memory-core-host-engine-qmd";
import {
@@ -93,7 +96,7 @@ import {
resolveMemorySourceExistingHash,
} from "./manager-source-state.js";
import {
markMemoryTargetSessionFilesDirty,
markMemoryTargetArchiveFilesDirty,
runMemoryTargetedSessionSync,
} from "./manager-targeted-sync.js";
import {
@@ -150,6 +153,25 @@ type MemoryReindexRetryState = {
sessionDeltas: Map<string, MemorySessionDeltaState>;
};
function sessionPathForCorpusEntry(entry: SessionTranscriptCorpusEntry): string {
return entry.transcriptSource === "sqlite"
? sessionPathForSessionIdentity(entry.agentId, entry.sessionId)
: sessionPathForFile(entry.sessionFile);
}
function legacyExtensionlessSessionPathForIdentity(agentId: string, sessionId: string): string {
return path.join("sessions", normalizeAgentId(agentId), sessionId).replace(/\\/g, "/");
}
function buildSessionEntryOptions(entry: SessionTranscriptCorpusEntry) {
return {
generatedByDreamingNarrative: entry.generatedByDreamingNarrative === true,
generatedByCronRun: entry.generatedByCronRun === true,
...(entry.sessionKey ? { sessionKey: entry.sessionKey } : {}),
...(entry.updatedAtMs !== undefined ? { updatedAtMs: entry.updatedAtMs } : {}),
};
}
const META_KEY = "memory_index_meta_v1";
const VECTOR_TABLE = MEMORY_INDEX_VECTOR_TABLE;
const LEGACY_VECTOR_TABLE = "chunks_vec";
@@ -172,9 +194,6 @@ const IGNORED_MEMORY_WATCH_DIR_NAMES = new Set([
]);
const log = createSubsystemLogger("memory");
const MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY = Symbol.for(
"openclaw.memoryCore.sessionTranscriptUpdateSubscriber",
);
const TEST_MEMORY_WATCH_FACTORY_KEY = Symbol.for("openclaw.test.memoryWatchFactory");
const TEST_MEMORY_NATIVE_WATCH_FACTORY_KEY = Symbol.for("openclaw.test.memoryNativeWatchFactory");
@@ -189,10 +208,6 @@ type MemorySessionTranscriptUpdate = {
};
};
type MemoryTranscriptUpdateSubscriber = (
listener: (update: MemorySessionTranscriptUpdate) => void,
) => () => void;
function memoryTableExists(db: DatabaseSync, tableName: string): boolean {
return Boolean(
db.prepare("SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = ?").get(tableName),
@@ -211,18 +226,6 @@ type LinuxMemoryDirectoryWatcher = {
ino: number;
};
function subscribeMemorySessionTranscriptUpdates(
listener: (update: MemorySessionTranscriptUpdate) => void,
): () => void {
const injected = (globalThis as Record<symbol, unknown>)[
MEMORY_CORE_TRANSCRIPT_UPDATE_SUBSCRIBER_KEY
];
if (typeof injected === "function") {
return (injected as MemoryTranscriptUpdateSubscriber)(listener);
}
return onSessionTranscriptUpdate(listener);
}
function resolveMemoryWatchFactory(): typeof chokidar.watch {
if (process.env.VITEST === "true" || process.env.NODE_ENV === "test") {
const override = (globalThis as Record<PropertyKey, unknown>)[TEST_MEMORY_WATCH_FACTORY_KEY];
@@ -494,7 +497,7 @@ export abstract class MemoryManagerSyncOps {
shouldSyncSessions: boolean;
needsFullReindex: boolean;
needsFullSessionReindex?: boolean;
targetSessionFiles?: string[];
targetArchiveFiles?: string[];
progress?: MemorySyncProgressState;
}): Promise<void> {
const memoryPlan = params.shouldSyncMemory
@@ -505,9 +508,9 @@ export abstract class MemoryManagerSyncOps {
})
: this.emptySourceSyncPlan();
if (params.shouldSyncSessions) {
await this.syncSessionFiles({
await this.syncArchiveFiles({
needsFullReindex: params.needsFullSessionReindex ?? params.needsFullReindex,
targetSessionFiles: params.targetSessionFiles,
targetArchiveFiles: params.targetArchiveFiles,
progress: params.progress,
deferIndex: true,
prefixIndexItems: memoryPlan.indexItems,
@@ -1454,7 +1457,7 @@ export abstract class MemoryManagerSyncOps {
if (!this.sources.has("sessions") || this.sessionUnsubscribe) {
return;
}
this.sessionUnsubscribe = subscribeMemorySessionTranscriptUpdates((update) => {
this.sessionUnsubscribe = this.subscribeSessionTranscriptUpdates((update) => {
if (this.closed) {
return;
}
@@ -1479,6 +1482,12 @@ export abstract class MemoryManagerSyncOps {
});
}
protected subscribeSessionTranscriptUpdates(
listener: (update: MemorySessionTranscriptUpdate) => void,
): () => void {
return onInternalSessionTranscriptUpdate(listener);
}
private async scheduleCorpusSessionFileDirty(sessionFile: string): Promise<void> {
const resolvedSessionFile = path.resolve(sessionFile);
const corpusEntries = await listSessionTranscriptCorpusEntriesForAgent(this.agentId);
@@ -1500,8 +1509,8 @@ export abstract class MemoryManagerSyncOps {
if (!this.sources.has("sessions") || this.closed) {
return [];
}
const files = await listSessionFilesForAgent(this.agentId);
if (files.length === 0 || this.closed) {
const corpusEntries = await listSessionTranscriptCorpusEntriesForAgent(this.agentId);
if (corpusEntries.length === 0 || this.closed) {
return [];
}
const existingRows = loadMemorySourceFileState({
@@ -1510,7 +1519,15 @@ export abstract class MemoryManagerSyncOps {
}).rows;
const fileStates = (
await runWithConcurrency(
files.map((file) => async (): Promise<MemorySessionStartupFileState | null> => {
corpusEntries.map(
(corpusEntry) => async (): Promise<MemorySessionStartupFileState | null> => {
if (corpusEntry.transcriptSource === "sqlite") {
return statSessionEntrySync(
corpusEntry.sessionFile,
buildSessionEntryOptions(corpusEntry),
);
}
const file = corpusEntry.sessionFile;
try {
const stat = await fs.stat(file);
if (!stat.isFile()) {
@@ -1518,7 +1535,7 @@ export abstract class MemoryManagerSyncOps {
}
return {
absPath: file,
path: sessionPathForFile(file),
path: sessionPathForCorpusEntry(corpusEntry),
mtimeMs: stat.mtimeMs,
size: stat.size,
};
@@ -1528,7 +1545,8 @@ export abstract class MemoryManagerSyncOps {
}
throw err;
}
}),
},
),
this.getIndexConcurrency(),
)
).filter((file): file is MemorySessionStartupFileState => file !== null);
@@ -1579,9 +1597,15 @@ export abstract class MemoryManagerSyncOps {
const pendingTargets = Array.from(this.sessionPendingTargets.values());
this.sessionPendingFiles.clear();
this.sessionPendingTargets.clear();
pending.push(...Array.from(await this.resolveSessionFilesForSyncTargets(pendingTargets)));
pending.push(...Array.from(await this.resolveArchiveFilesForSyncTargets(pendingTargets)));
let shouldSync = false;
for (const sessionFile of pending) {
if (!path.isAbsolute(sessionFile)) {
this.sessionsDirtyFiles.add(sessionFile);
this.sessionsDirty = true;
shouldSync = true;
continue;
}
// Usage-counted session archives (`.jsonl.reset.<iso>` and
// `.jsonl.deleted.<iso>`) are one-shot mutation events: the file is
// written once by the archive rotation and then never touched again.
@@ -1789,20 +1813,41 @@ export abstract class MemoryManagerSyncOps {
};
}
private normalizeTargetSessionFiles(
sessionFiles?: string[],
private normalizeTargetArchiveFiles(
archiveFiles?: string[],
corpusEntries: readonly SessionTranscriptCorpusEntry[] = [],
): Set<string> | null {
if (!sessionFiles || sessionFiles.length === 0) {
if (!archiveFiles || archiveFiles.length === 0) {
return null;
}
const normalized = new Set<string>();
const corpusPaths = new Set(corpusEntries.map((entry) => path.resolve(entry.sessionFile)));
for (const sessionFile of sessionFiles) {
const corpusMarkers = new Set(
corpusEntries
.filter((entry) => entry.transcriptSource === "sqlite")
.map((entry) => entry.sessionFile),
);
const corpusPaths = new Set(
corpusEntries
.filter((entry) => entry.transcriptSource !== "sqlite")
.map((entry) => path.resolve(entry.sessionFile)),
);
for (const sessionFile of archiveFiles) {
const trimmed = sessionFile.trim();
if (!trimmed) {
continue;
}
if (corpusMarkers.has(trimmed)) {
normalized.add(trimmed);
continue;
}
const sqliteMarker = parseSqliteSessionFileMarker(trimmed);
if (
sqliteMarker &&
normalizeAgentId(sqliteMarker.agentId) === normalizeAgentId(this.agentId)
) {
normalized.add(trimmed);
continue;
}
const resolved = path.resolve(trimmed);
if (
this.isSessionFileForAgent(resolved) &&
@@ -1842,7 +1887,7 @@ export abstract class MemoryManagerSyncOps {
return normalized.size > 0 ? normalized : null;
}
private async resolveSessionFilesForSyncTargets(
private async resolveArchiveFilesForSyncTargets(
sessions?: Iterable<MemorySessionSyncTarget> | null,
knownCorpusEntries?: readonly SessionTranscriptCorpusEntry[],
): Promise<Set<string>> {
@@ -1866,7 +1911,9 @@ export abstract class MemoryManagerSyncOps {
if (sessionKey && entry.sessionKey !== sessionKey) {
continue;
}
files.add(path.resolve(entry.sessionFile));
files.add(
entry.transcriptSource === "sqlite" ? entry.sessionFile : path.resolve(entry.sessionFile),
);
matchedCorpusEntry = true;
}
if (matchedCorpusEntry) {
@@ -1887,16 +1934,16 @@ export abstract class MemoryManagerSyncOps {
return files;
}
private async combineTargetSessionFiles(params: {
private async combineTargetArchiveFiles(params: {
sessions?: MemorySessionSyncTarget[];
sessionFiles?: string[];
archiveFiles?: string[];
}): Promise<Set<string> | null> {
const files = new Set<string>();
const corpusEntries = await listSessionTranscriptCorpusEntriesForAgent(this.agentId);
for (const file of this.normalizeTargetSessionFiles(params.sessionFiles, corpusEntries) ?? []) {
for (const file of this.normalizeTargetArchiveFiles(params.archiveFiles, corpusEntries) ?? []) {
files.add(file);
}
for (const file of await this.resolveSessionFilesForSyncTargets(
for (const file of await this.resolveArchiveFilesForSyncTargets(
this.normalizeTargetSessions(params.sessions)?.values(),
corpusEntries,
)) {
@@ -2089,9 +2136,9 @@ export abstract class MemoryManagerSyncOps {
return this.emptySourceSyncPlan();
}
private async syncSessionFiles(params: {
private async syncArchiveFiles(params: {
needsFullReindex: boolean;
targetSessionFiles?: string[];
targetArchiveFiles?: string[];
progress?: MemorySyncProgressState;
deferIndex?: boolean;
prefixIndexItems?: MemoryIndexWorkItem[];
@@ -2114,34 +2161,43 @@ export abstract class MemoryManagerSyncOps {
: null;
const corpusEntries = await listSessionTranscriptCorpusEntriesForAgent(this.agentId);
const targetSessionFiles = params.needsFullReindex
const targetArchiveFiles = params.needsFullReindex
? null
: this.normalizeTargetSessionFiles(params.targetSessionFiles, corpusEntries);
: this.normalizeTargetArchiveFiles(params.targetArchiveFiles, corpusEntries);
const corpusEntryByPath = new Map<string, SessionTranscriptCorpusEntry>(
corpusEntries.map((entry) => [entry.sessionFile, entry]),
);
const files = targetSessionFiles
? Array.from(targetSessionFiles)
const files = targetArchiveFiles
? Array.from(targetArchiveFiles)
: corpusEntries.map((entry) => entry.sessionFile);
const sessionPlan = resolveMemorySessionSyncPlan({
needsFullReindex: params.needsFullReindex,
files,
targetSessionFiles,
targetSessionFiles: targetArchiveFiles,
sessionsDirtyFiles: this.sessionsDirtyFiles,
existingRows: targetSessionFiles
existingRows: targetArchiveFiles
? null
: loadMemorySourceFileState({
db: this.db,
source: "sessions",
}).rows,
sessionPathForFile,
sessionPathForFile: (file) => {
const corpusEntry = corpusEntryByPath.get(file);
if (corpusEntry) {
return sessionPathForCorpusEntry(corpusEntry);
}
const sqliteMarker = parseSqliteSessionFileMarker(file);
return sqliteMarker
? sessionPathForSessionIdentity(sqliteMarker.agentId, sqliteMarker.sessionId)
: sessionPathForFile(file);
},
});
const { activePaths, existingRows, existingHashes, indexAll } = sessionPlan;
log.debug("memory sync: indexing session files", {
files: files.length,
indexAll,
dirtyFiles: this.sessionsDirtyFiles.size,
targetedFiles: targetSessionFiles?.size ?? 0,
targetedFiles: targetArchiveFiles?.size ?? 0,
batch: this.batch.enabled,
concurrency: this.getIndexConcurrency(),
});
@@ -2155,6 +2211,20 @@ export abstract class MemoryManagerSyncOps {
}
const yieldAfterSessionFile = createSessionSyncYield(files.length);
const deleteIndexedSessionPath = (memoryPath: string) => {
deleteFileByPathAndSource.run(memoryPath, "sessions");
if (deleteVectorRowsByPathAndSource) {
try {
deleteVectorRowsByPathAndSource.run(memoryPath, "sessions");
} catch {}
}
deleteChunksByPathAndSource.run(memoryPath, "sessions");
if (deleteFtsRowsByPathAndSource) {
try {
deleteFtsRowsByPathAndSource.run(memoryPath, "sessions");
} catch {}
}
};
const deleteStaleRows = async () => {
if (activePaths === null) {
return;
@@ -2167,23 +2237,50 @@ export abstract class MemoryManagerSyncOps {
if (activePaths.has(stale.path)) {
continue;
}
deleteFileByPathAndSource.run(stale.path, "sessions");
if (deleteVectorRowsByPathAndSource) {
try {
deleteVectorRowsByPathAndSource.run(stale.path, "sessions");
} catch {}
}
deleteChunksByPathAndSource.run(stale.path, "sessions");
if (deleteFtsRowsByPathAndSource) {
try {
deleteFtsRowsByPathAndSource.run(stale.path, "sessions");
} catch {}
}
deleteIndexedSessionPath(stale.path);
} finally {
await yieldAfterStaleSessionRow();
}
}
};
const deleteTargetArchiveStaleLiveRows = () => {
if (!targetArchiveFiles) {
return;
}
const activeCorpusPaths = new Set(
corpusEntries
.filter((entry) => entry.artifactKind === "active-session")
.map((entry) => sessionPathForCorpusEntry(entry)),
);
const existingSessionPaths = new Set(
loadMemorySourceFileState({
db: this.db,
source: "sessions",
}).rows.map((row) => row.path),
);
for (const file of targetArchiveFiles) {
const corpusEntry = corpusEntryByPath.get(file);
const sqliteMarker = parseSqliteSessionFileMarker(file);
const sessionId =
corpusEntry?.sessionId ??
sqliteMarker?.sessionId ??
parseUsageCountedSessionIdFromFileName(path.basename(file));
if (!sessionId) {
continue;
}
const staleAgentId = corpusEntry?.agentId ?? sqliteMarker?.agentId ?? this.agentId;
const staleLivePaths = [
sessionPathForSessionIdentity(staleAgentId, sessionId),
legacyExtensionlessSessionPathForIdentity(staleAgentId, sessionId),
];
for (const staleLivePath of staleLivePaths) {
if (activeCorpusPaths.has(staleLivePath) || !existingSessionPaths.has(staleLivePath)) {
continue;
}
deleteIndexedSessionPath(staleLivePath);
}
}
};
if (params.deferIndex) {
const pendingIndexItems = [...(params.prefixIndexItems ?? [])];
@@ -2221,13 +2318,7 @@ export abstract class MemoryManagerSyncOps {
const corpusEntry = corpusEntryByPath.get(absPath);
const entry = await buildSessionEntry(
absPath,
corpusEntry
? {
generatedByDreamingNarrative:
corpusEntry.generatedByDreamingNarrative === true,
generatedByCronRun: corpusEntry.generatedByCronRun === true,
}
: undefined,
corpusEntry ? buildSessionEntryOptions(corpusEntry) : undefined,
);
if (!entry) {
if (params.progress) {
@@ -2279,6 +2370,7 @@ export abstract class MemoryManagerSyncOps {
}
await flushPendingIndexItems();
deleteTargetArchiveStaleLiveRows();
await deleteStaleRows();
return this.emptySourceSyncPlan();
}
@@ -2301,12 +2393,7 @@ export abstract class MemoryManagerSyncOps {
const corpusEntry = corpusEntryByPath.get(absPath);
const entry = await buildSessionEntry(
absPath,
corpusEntry
? {
generatedByDreamingNarrative: corpusEntry.generatedByDreamingNarrative === true,
generatedByCronRun: corpusEntry.generatedByCronRun === true,
}
: undefined,
corpusEntry ? buildSessionEntryOptions(corpusEntry) : undefined,
);
if (!entry) {
if (params.progress) {
@@ -2350,6 +2437,7 @@ export abstract class MemoryManagerSyncOps {
});
await runWithConcurrency(tasks, this.getIndexConcurrency());
deleteTargetArchiveStaleLiveRows();
await deleteStaleRows();
return this.emptySourceSyncPlan();
}
@@ -2418,15 +2506,15 @@ export abstract class MemoryManagerSyncOps {
}
const vectorReady = await this.ensureVectorReady();
const meta = this.readMeta();
const targetSessionFiles = await this.combineTargetSessionFiles({
const targetArchiveFiles = await this.combineTargetArchiveFiles({
sessions: params?.sessions,
sessionFiles: params?.sessionFiles,
archiveFiles: params?.archiveFiles,
});
const hasTargetSessionFiles = targetSessionFiles !== null;
if (this.hasRequestedTargetSessionSync(params) && !hasTargetSessionFiles) {
const hasTargetArchiveFiles = targetArchiveFiles !== null;
if (this.hasRequestedTargetSessionSync(params) && !hasTargetArchiveFiles) {
return;
}
if (params?.reason === "cli" && !params.force && !hasTargetSessionFiles) {
if (params?.reason === "cli" && !params.force && !hasTargetArchiveFiles) {
await this.markSessionStartupCatchupDirtyFiles();
}
const indexIdentity = resolveMemoryIndexIdentityState({
@@ -2473,13 +2561,13 @@ export abstract class MemoryManagerSyncOps {
this.settings.provider === "none" ||
hasOnlyFtsChunks;
const needsMissingIdentityReindex =
indexIdentity.status === "missing" && !hasTargetSessionFiles && canRebuildMissingIdentity;
indexIdentity.status === "missing" && !hasTargetArchiveFiles && canRebuildMissingIdentity;
const needsExplicitIdentityReindex =
params?.reason === "cli" && indexIdentity.status !== "valid" && !hasTargetSessionFiles;
params?.reason === "cli" && indexIdentity.status !== "valid" && !hasTargetArchiveFiles;
const canRunRetryFullReindex =
indexIdentity.status !== "missing" || needsInitialIndex || canRebuildMissingIdentity;
const needsFullReindex =
(params?.force && !hasTargetSessionFiles) ||
(params?.force && !hasTargetArchiveFiles) ||
needsInitialIndex ||
needsMissingIdentityReindex ||
needsExplicitIdentityReindex ||
@@ -2488,9 +2576,9 @@ export abstract class MemoryManagerSyncOps {
const needsFullSessionReindex = needsFullReindex || this.sessionsFullRetryDirty;
if (indexIdentity.status !== "valid" && !needsFullReindex) {
this.dirty = true;
const sessionsDirty = markMemoryTargetSessionFilesDirty({
const sessionsDirty = markMemoryTargetArchiveFilesDirty({
sessionsDirtyFiles: this.sessionsDirtyFiles,
targetSessionFiles,
targetArchiveFiles,
});
if (sessionsDirty) {
this.sessionsDirty = true;
@@ -2500,13 +2588,13 @@ export abstract class MemoryManagerSyncOps {
if (!needsFullSessionReindex) {
const targetedSessionSync = await runMemoryTargetedSessionSync({
hasSessionSource: this.sources.has("sessions"),
targetSessionFiles,
targetArchiveFiles,
reason: params?.reason,
progress: progress ?? undefined,
sessionsFullRetryDirty: this.sessionsFullRetryDirty,
sessionsDirtyFiles: this.sessionsDirtyFiles,
syncSessionFiles: async (targetedParams) => {
await this.syncSessionFiles(targetedParams);
syncArchiveFiles: async (targetedParams) => {
await this.syncArchiveFiles(targetedParams);
},
shouldFallbackOnError: (err) => this.shouldFallbackOnError(err),
activateFallbackProvider: async (reason) => await this.activateFallbackProvider(reason),
@@ -2528,7 +2616,7 @@ export abstract class MemoryManagerSyncOps {
const shouldSyncMemory =
this.sources.has("memory") &&
((!hasTargetSessionFiles && params?.force) || needsFullReindex || this.dirty);
((!hasTargetArchiveFiles && params?.force) || needsFullReindex || this.dirty);
const shouldSyncSessions = this.shouldSyncSessions(params, needsFullReindex);
if (this.shouldDeferSourceWideBatch()) {
@@ -2537,7 +2625,7 @@ export abstract class MemoryManagerSyncOps {
shouldSyncSessions,
needsFullReindex,
needsFullSessionReindex,
targetSessionFiles: targetSessionFiles ? Array.from(targetSessionFiles) : undefined,
targetArchiveFiles: targetArchiveFiles ? Array.from(targetArchiveFiles) : undefined,
progress: progress ?? undefined,
});
if (shouldSyncMemory) {
@@ -2555,9 +2643,9 @@ export abstract class MemoryManagerSyncOps {
}
if (shouldSyncSessions) {
await this.syncSessionFiles({
await this.syncArchiveFiles({
needsFullReindex: needsFullSessionReindex,
targetSessionFiles: targetSessionFiles ? Array.from(targetSessionFiles) : undefined,
targetArchiveFiles: targetArchiveFiles ? Array.from(targetArchiveFiles) : undefined,
progress: progress ?? undefined,
});
this.clearSessionRetryState();
@@ -2570,7 +2658,7 @@ export abstract class MemoryManagerSyncOps {
const activated =
this.shouldFallbackOnError(err) && (await this.activateFallbackProvider(reason));
if (activated) {
if (needsFullReindex && !hasTargetSessionFiles) {
if (needsFullReindex && !hasTargetArchiveFiles) {
await this.runInPlaceReindex({
reason: params?.reason ?? "fallback",
force: true,
@@ -2594,7 +2682,7 @@ export abstract class MemoryManagerSyncOps {
private hasRequestedTargetSessionSync(params?: MemorySyncParams): boolean {
return Boolean(
params?.sessions?.some((session) => session.sessionId.trim().length > 0) ||
params?.sessionFiles?.some((sessionFile) => sessionFile.trim().length > 0),
params?.archiveFiles?.some((sessionFile) => sessionFile.trim().length > 0),
);
}
@@ -2744,7 +2832,7 @@ export abstract class MemoryManagerSyncOps {
}
if (shouldSyncSessions) {
await this.syncSessionFiles({ needsFullReindex: true, progress: params.progress });
await this.syncArchiveFiles({ needsFullReindex: true, progress: params.progress });
this.clearSessionRetryState();
} else {
this.refreshSessionDirtyFlag();
extensions/memory-core/src/memory/manager-sync-yield.test.ts
+8 -6
View File
@@ -62,6 +62,8 @@ vi.mock("openclaw/plugin-sdk/memory-core-host-engine-qmd", () => {
sessionId: target.sessionId,
}),
sessionPathForFile: (filePath: string) => `sessions/${basename(filePath)}`,
sessionPathForSessionIdentity: (agentId: string, sessionId: string) =>
`sessions/${agentId}/${sessionId}`,
};
});
@@ -126,17 +128,17 @@ class SessionSyncYieldHarness extends MemoryManagerSyncOps {
super();
}
async syncTargetSessionFiles(files: string[]): Promise<void> {
async syncTargetArchiveFiles(files: string[]): Promise<void> {
await (
this as unknown as {
syncSessionFiles: (params: {
syncArchiveFiles: (params: {
needsFullReindex: boolean;
targetSessionFiles: string[];
targetArchiveFiles: string[];
}) => Promise<void>;
}
).syncSessionFiles({
).syncArchiveFiles({
needsFullReindex: false,
targetSessionFiles: files,
targetArchiveFiles: files,
});
}
@@ -217,7 +219,7 @@ describe("session sync responsiveness", () => {
}
});
await harness.syncTargetSessionFiles(files);
await harness.syncTargetArchiveFiles(files);
expect(harness.indexedPaths).toHaveLength(files.length);
expect(observedBeforeLastFile).toEqual([true]);
extensions/memory-core/src/memory/manager-targeted-sync.test.ts
+18 -18
View File
@@ -3,8 +3,8 @@ import type { MemorySessionSyncTarget } from "openclaw/plugin-sdk/memory-core-ho
import { describe, expect, it, vi } from "vitest";
import { enqueueMemoryTargetedSessionSync } from "./manager-sync-control.js";
import {
clearMemorySyncedSessionFiles,
markMemoryTargetSessionFilesDirty,
clearMemorySyncedArchiveFiles,
markMemoryTargetArchiveFilesDirty,
runMemoryTargetedSessionSync,
} from "./manager-targeted-sync.js";
@@ -13,9 +13,9 @@ describe("memory targeted session sync", () => {
const secondSessionPath = "/tmp/targeted-dirty-second.jsonl";
const sessionsDirtyFiles = new Set(["/tmp/targeted-dirty-first.jsonl", secondSessionPath]);
const sessionsDirty = clearMemorySyncedSessionFiles({
const sessionsDirty = clearMemorySyncedArchiveFiles({
sessionsDirtyFiles,
targetSessionFiles: ["/tmp/targeted-dirty-first.jsonl"],
targetArchiveFiles: ["/tmp/targeted-dirty-first.jsonl"],
});
expect(sessionsDirtyFiles.has(secondSessionPath)).toBe(true);
@@ -26,9 +26,9 @@ describe("memory targeted session sync", () => {
const targetSessionPath = "/tmp/paused-target.jsonl";
const sessionsDirtyFiles = new Set(["/tmp/other-dirty.jsonl"]);
const sessionsDirty = markMemoryTargetSessionFilesDirty({
const sessionsDirty = markMemoryTargetArchiveFilesDirty({
sessionsDirtyFiles,
targetSessionFiles: [targetSessionPath],
targetArchiveFiles: [targetSessionPath],
});
expect(sessionsDirty).toBe(true);
@@ -38,7 +38,7 @@ describe("memory targeted session sync", () => {
it("leaves targeted sessions dirty after fallback activates during targeted sync", async () => {
const activateFallbackProvider = vi.fn(async () => true);
const syncSessionFiles = vi
const syncArchiveFiles = vi
.fn()
.mockRejectedValueOnce(new Error("embedding backend failed"))
.mockResolvedValueOnce(undefined);
@@ -46,20 +46,20 @@ describe("memory targeted session sync", () => {
const result = await runMemoryTargetedSessionSync({
hasSessionSource: true,
targetSessionFiles: new Set(["/tmp/targeted-fallback.jsonl"]),
targetArchiveFiles: new Set(["/tmp/targeted-fallback.jsonl"]),
reason: "post-compaction",
progress: undefined,
sessionsDirtyFiles,
syncSessionFiles,
syncArchiveFiles,
shouldFallbackOnError: () => true,
activateFallbackProvider,
});
expect(activateFallbackProvider).toHaveBeenCalledWith("embedding backend failed");
expect(syncSessionFiles).toHaveBeenCalledTimes(1);
expect(syncSessionFiles).toHaveBeenCalledWith({
expect(syncArchiveFiles).toHaveBeenCalledTimes(1);
expect(syncArchiveFiles).toHaveBeenCalledWith({
needsFullReindex: false,
targetSessionFiles: ["/tmp/targeted-fallback.jsonl"],
targetArchiveFiles: ["/tmp/targeted-fallback.jsonl"],
progress: undefined,
});
expect(result).toEqual({ handled: true, sessionsDirty: true });
@@ -68,17 +68,17 @@ describe("memory targeted session sync", () => {
});
it("preserves the full-retry dirty marker after targeted cleanup", async () => {
const syncSessionFiles = vi.fn(async () => undefined);
const syncArchiveFiles = vi.fn(async () => undefined);
const sessionsDirtyFiles = new Set(["/tmp/targeted-full-retry.jsonl"]);
const result = await runMemoryTargetedSessionSync({
hasSessionSource: true,
targetSessionFiles: new Set(["/tmp/targeted-full-retry.jsonl"]),
targetArchiveFiles: new Set(["/tmp/targeted-full-retry.jsonl"]),
reason: "post-compaction",
progress: undefined,
sessionsFullRetryDirty: true,
sessionsDirtyFiles,
syncSessionFiles,
syncArchiveFiles,
shouldFallbackOnError: () => false,
activateFallbackProvider: async () => false,
});
@@ -92,7 +92,7 @@ describe("memory targeted session sync", () => {
const syncing = new Promise<void>((resolve) => {
resolveSyncing = resolve;
});
const queuedSessionFiles = new Set<string>();
const queuedArchiveFiles = new Set<string>();
const queuedSessions = new Map<string, MemorySessionSyncTarget>();
let queuedSessionSync: Promise<void> | null = null;
const sync = vi.fn(async () => {});
@@ -101,7 +101,7 @@ describe("memory targeted session sync", () => {
{
isClosed: () => false,
getSyncing: () => syncing,
getQueuedSessionFiles: () => queuedSessionFiles,
getQueuedArchiveFiles: () => queuedArchiveFiles,
getQueuedSessions: () => queuedSessions,
getQueuedSessionSync: () => queuedSessionSync,
setQueuedSessionSync: (value) => {
@@ -120,7 +120,7 @@ describe("memory targeted session sync", () => {
expect(sync).toHaveBeenCalledWith({
reason: "queued-sessions",
sessions: [{ agentId: "main", sessionId: "targeted", sessionKey: "agent:main:targeted" }],
sessionFiles: [],
archiveFiles: [],
});
});
});
extensions/memory-core/src/memory/manager-targeted-sync.ts
+20 -20
View File
@@ -9,27 +9,27 @@ type TargetedSyncProgress = {
report: (update: MemorySyncProgressUpdate) => void;
};
export function clearMemorySyncedSessionFiles(params: {
export function clearMemorySyncedArchiveFiles(params: {
sessionsDirtyFiles: Set<string>;
targetSessionFiles?: Iterable<string> | null;
targetArchiveFiles?: Iterable<string> | null;
}): boolean {
if (!params.targetSessionFiles) {
if (!params.targetArchiveFiles) {
params.sessionsDirtyFiles.clear();
} else {
for (const targetSessionFile of params.targetSessionFiles) {
params.sessionsDirtyFiles.delete(targetSessionFile);
for (const targetArchiveFile of params.targetArchiveFiles) {
params.sessionsDirtyFiles.delete(targetArchiveFile);
}
}
return params.sessionsDirtyFiles.size > 0;
}
export function markMemoryTargetSessionFilesDirty(params: {
export function markMemoryTargetArchiveFilesDirty(params: {
sessionsDirtyFiles: Set<string>;
targetSessionFiles?: Iterable<string> | null;
targetArchiveFiles?: Iterable<string> | null;
}): boolean {
if (params.targetSessionFiles) {
for (const targetSessionFile of params.targetSessionFiles) {
params.sessionsDirtyFiles.add(targetSessionFile);
if (params.targetArchiveFiles) {
for (const targetArchiveFile of params.targetArchiveFiles) {
params.sessionsDirtyFiles.add(targetArchiveFile);
}
}
return params.sessionsDirtyFiles.size > 0;
@@ -37,20 +37,20 @@ export function markMemoryTargetSessionFilesDirty(params: {
export async function runMemoryTargetedSessionSync(params: {
hasSessionSource: boolean;
targetSessionFiles: Set<string> | null;
targetArchiveFiles: Set<string> | null;
reason?: string;
progress?: TargetedSyncProgress;
sessionsFullRetryDirty?: boolean;
sessionsDirtyFiles: Set<string>;
syncSessionFiles: (params: {
syncArchiveFiles: (params: {
needsFullReindex: boolean;
targetSessionFiles?: string[];
targetArchiveFiles?: string[];
progress?: TargetedSyncProgress;
}) => Promise<void>;
shouldFallbackOnError: (err: unknown) => boolean;
activateFallbackProvider: (reason: string) => Promise<boolean>;
}): Promise<{ handled: boolean; sessionsDirty: boolean }> {
if (!params.hasSessionSource || !params.targetSessionFiles) {
if (!params.hasSessionSource || !params.targetArchiveFiles) {
return {
handled: false,
sessionsDirty: Boolean(params.sessionsFullRetryDirty) || params.sessionsDirtyFiles.size > 0,
@@ -58,14 +58,14 @@ export async function runMemoryTargetedSessionSync(params: {
}
try {
await params.syncSessionFiles({
await params.syncArchiveFiles({
needsFullReindex: false,
targetSessionFiles: Array.from(params.targetSessionFiles),
targetArchiveFiles: Array.from(params.targetArchiveFiles),
progress: params.progress,
});
const remainingSessionsDirty = clearMemorySyncedSessionFiles({
const remainingSessionsDirty = clearMemorySyncedArchiveFiles({
sessionsDirtyFiles: params.sessionsDirtyFiles,
targetSessionFiles: params.targetSessionFiles,
targetArchiveFiles: params.targetArchiveFiles,
});
return {
handled: true,
@@ -78,9 +78,9 @@ export async function runMemoryTargetedSessionSync(params: {
if (!activated) {
throw err;
}
const remainingSessionsDirty = markMemoryTargetSessionFilesDirty({
const remainingSessionsDirty = markMemoryTargetArchiveFilesDirty({
sessionsDirtyFiles: params.sessionsDirtyFiles,
targetSessionFiles: params.targetSessionFiles,
targetArchiveFiles: params.targetArchiveFiles,
});
return {
handled: true,
extensions/memory-core/src/memory/manager.readonly-recovery.test.ts
+12 -12
View File
@@ -13,7 +13,7 @@ import {
type ReadonlyRecoveryHarness = MemoryReadonlyRecoveryState & {
syncing: Promise<void> | null;
queuedSessionFiles: Set<string>;
queuedArchiveFiles: Set<string>;
queuedSessions: Map<string, unknown>;
queuedSessionSync: Promise<void> | null;
vectorDegradedWriteWarningShown: boolean;
@@ -32,12 +32,12 @@ describe("memory manager readonly recovery", () => {
let indexPath = "";
function createQueuedSyncHarness(syncing: Promise<void>) {
const queuedSessionFiles = new Set<string>();
const queuedArchiveFiles = new Set<string>();
const queuedSessions = new Map<string, never>();
let queuedSessionSync: Promise<void> | null = null;
const sync = vi.fn(async () => {});
return {
queuedSessionFiles,
queuedArchiveFiles,
queuedSessions,
get queuedSessionSync() {
return queuedSessionSync;
@@ -46,7 +46,7 @@ describe("memory manager readonly recovery", () => {
state: {
isClosed: () => false,
getSyncing: () => syncing,
getQueuedSessionFiles: () => queuedSessionFiles,
getQueuedArchiveFiles: () => queuedArchiveFiles,
getQueuedSessions: () => queuedSessions,
getQueuedSessionSync: () => queuedSessionSync,
setQueuedSessionSync: (value: Promise<void> | null) => {
@@ -64,7 +64,7 @@ describe("memory manager readonly recovery", () => {
const harness: ReadonlyRecoveryHarness = {
closed: false,
syncing: null,
queuedSessionFiles: new Set<string>(),
queuedArchiveFiles: new Set<string>(),
queuedSessions: new Map<string, never>(),
queuedSessionSync: null,
db: initialDb,
@@ -101,7 +101,7 @@ describe("memory manager readonly recovery", () => {
async function runSyncWithReadonlyRecovery(
harness: ReadonlyRecoveryHarness,
params?: { reason?: string; force?: boolean; sessionFiles?: string[] },
params?: { reason?: string; force?: boolean; archiveFiles?: string[] },
) {
return await runMemorySyncWithReadonlyRecovery(harness, params);
}
@@ -230,7 +230,7 @@ describe("memory manager readonly recovery", () => {
const harness = createQueuedSyncHarness(pendingSync);
const queued = enqueueMemoryTargetedSessionSync(harness.state, {
sessionFiles: [" /tmp/first.jsonl ", "", "/tmp/second.jsonl"],
archiveFiles: [" /tmp/first.jsonl ", "", "/tmp/second.jsonl"],
});
expect(harness.sync).not.toHaveBeenCalled();
@@ -242,7 +242,7 @@ describe("memory manager readonly recovery", () => {
expect(harness.sync).toHaveBeenCalledWith({
reason: "queued-sessions",
sessions: [],
sessionFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl"],
archiveFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl"],
});
expect(harness.queuedSessionSync).toBeNull();
});
@@ -255,10 +255,10 @@ describe("memory manager readonly recovery", () => {
const harness = createQueuedSyncHarness(pendingSync);
const first = enqueueMemoryTargetedSessionSync(harness.state, {
sessionFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl"],
archiveFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl"],
});
const second = enqueueMemoryTargetedSessionSync(harness.state, {
sessionFiles: ["/tmp/second.jsonl", "/tmp/third.jsonl"],
archiveFiles: ["/tmp/second.jsonl", "/tmp/third.jsonl"],
});
expect(first).toBe(second);
@@ -270,7 +270,7 @@ describe("memory manager readonly recovery", () => {
expect(harness.sync).toHaveBeenCalledWith({
reason: "queued-sessions",
sessions: [],
sessionFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl", "/tmp/third.jsonl"],
archiveFiles: ["/tmp/first.jsonl", "/tmp/second.jsonl", "/tmp/third.jsonl"],
});
});
@@ -282,7 +282,7 @@ describe("memory manager readonly recovery", () => {
const harness = createQueuedSyncHarness(pendingSync);
const queued = enqueueMemoryTargetedSessionSync(harness.state, {
sessionFiles: ["", " "],
archiveFiles: ["", " "],
});
expect(queued).toBe(pendingSync);
extensions/memory-core/src/memory/manager.reindex-recovery.test.ts
+9 -22
View File
@@ -12,26 +12,13 @@ import { acquireMemoryReindexLock } from "./manager-reindex-lock.js";
import type { MemoryIndexMeta } from "./manager-reindex-state.js";
type SessionDeltaState = { lastSize: number; pendingBytes: number; pendingMessages: number };
type SyncSessionParams = { needsFullReindex: boolean; targetSessionFiles?: string[] };
const originalReindexStateDir = process.env.OPENCLAW_STATE_DIR;
function setReindexStateDir(stateDir: string): void {
Reflect.set(process.env, "OPENCLAW_STATE_DIR", stateDir);
}
function restoreReindexStateDir(): void {
if (originalReindexStateDir === undefined) {
Reflect.deleteProperty(process.env, "OPENCLAW_STATE_DIR");
} else {
Reflect.set(process.env, "OPENCLAW_STATE_DIR", originalReindexStateDir);
}
}
type SyncArchiveParams = { needsFullReindex: boolean; targetArchiveFiles?: string[] };
type ReindexHarness = {
sync: (params: { reason?: string; force?: boolean }) => Promise<void>;
runInPlaceReindex: (params: { reason?: string; force?: boolean }) => Promise<void>;
syncMemoryFiles: (params: { needsFullReindex: boolean }) => Promise<unknown>;
syncSessionFiles: (params: SyncSessionParams) => Promise<unknown>;
syncArchiveFiles: (params: SyncArchiveParams) => Promise<unknown>;
db: DatabaseSync;
writeMeta: (meta: MemoryIndexMeta) => void;
providerKey: string | null;
@@ -55,11 +42,11 @@ describe("memory manager reindex recovery", () => {
workspaceDir = path.join(fixtureRoot, "workspace");
memoryDir = path.join(workspaceDir, "memory");
await fs.mkdir(memoryDir, { recursive: true });
setReindexStateDir(path.join(fixtureRoot, "state"));
vi.stubEnv("OPENCLAW_STATE_DIR", path.join(fixtureRoot, "state"));
});
afterEach(async () => {
restoreReindexStateDir();
vi.unstubAllEnvs();
vi.restoreAllMocks();
if (manager) {
await manager.close();
@@ -130,7 +117,7 @@ describe("memory manager reindex recovery", () => {
harness.sessionsDirtyFiles.add(dirtySessionFile);
harness.sessionDeltas.set(dirtySessionFile, { ...originalDelta });
harness.syncMemoryFiles = async () => emptySyncPlan;
harness.syncSessionFiles = async () => {
harness.syncArchiveFiles = async () => {
const delta = harness.sessionDeltas.get(dirtySessionFile);
if (delta) {
delta.lastSize = 500;
@@ -165,7 +152,7 @@ describe("memory manager reindex recovery", () => {
const emptySyncPlan = { indexItems: [], finalize: () => undefined };
harness.syncMemoryFiles = async () => emptySyncPlan;
harness.syncSessionFiles = async () => emptySyncPlan;
harness.syncArchiveFiles = async () => emptySyncPlan;
harness.writeMeta = () => {
throw new Error("late clean reindex failure");
};
@@ -237,12 +224,12 @@ describe("memory manager reindex recovery", () => {
const harness = memoryManager as unknown as ReindexHarness;
const emptySyncPlan = { indexItems: [], finalize: () => undefined };
const sessionSyncCalls: SyncSessionParams[] = [];
const sessionSyncCalls: SyncArchiveParams[] = [];
harness.sessionsDirty = true;
harness.sessionsFullRetryDirty = true;
harness.sessionsDirtyFiles.clear();
harness.syncSessionFiles = async (params) => {
harness.syncArchiveFiles = async (params) => {
sessionSyncCalls.push(params);
return emptySyncPlan;
};
@@ -251,7 +238,7 @@ describe("memory manager reindex recovery", () => {
expect(sessionSyncCalls).toHaveLength(1);
expect(sessionSyncCalls[0]).toMatchObject({ needsFullReindex: true });
expect(sessionSyncCalls[0]?.targetSessionFiles).toBeUndefined();
expect(sessionSyncCalls[0]?.targetArchiveFiles).toBeUndefined();
expect(harness.sessionsDirty).toBe(false);
expect(harness.sessionsFullRetryDirty).toBe(false);
});
extensions/memory-core/src/memory/manager.ts
+4 -4
View File
@@ -356,7 +356,7 @@ export class MemoryIndexManager extends MemoryManagerEmbeddingOps implements Mem
>();
private sessionWarm = new Set<string>();
private syncing: Promise<void> | null = null;
private queuedSessionFiles = new Set<string>();
private queuedArchiveFiles = new Set<string>();
private queuedSessions = new Map<string, MemorySessionSyncTarget>();
private queuedSessionSync: Promise<void> | null = null;
private readonlyRecoveryAttempts = 0;
@@ -1259,13 +1259,13 @@ export class MemoryIndexManager extends MemoryManagerEmbeddingOps implements Mem
}
private enqueueTargetedSessionSync(
targets?: Pick<MemorySyncParams, "sessions" | "sessionFiles">,
targets?: Pick<MemorySyncParams, "sessions" | "archiveFiles">,
): Promise<void> {
return enqueueMemoryTargetedSessionSync(
{
isClosed: () => this.closed,
getSyncing: () => this.syncing,
getQueuedSessionFiles: () => this.queuedSessionFiles,
getQueuedArchiveFiles: () => this.queuedArchiveFiles,
getQueuedSessions: () => this.queuedSessions,
getQueuedSessionSync: () => this.queuedSessionSync,
setQueuedSessionSync: (value) => {
@@ -1637,7 +1637,7 @@ export class MemoryIndexManager extends MemoryManagerEmbeddingOps implements Mem
function hasTargetedSessionSyncParams(params: MemorySyncParams | undefined): boolean {
return Boolean(
params?.sessions?.some((session) => session.sessionId.trim().length > 0) ||
params?.sessionFiles?.some((sessionFile) => sessionFile.trim().length > 0),
params?.archiveFiles?.some((sessionFile) => sessionFile.trim().length > 0),
);
}
extensions/memory-core/src/memory/qmd-manager.test.ts
+60 -40
View File
@@ -204,7 +204,10 @@ import {
resolveMemoryBackendConfig,
} from "openclaw/plugin-sdk/memory-core-host-engine-storage";
import { MAX_TIMER_TIMEOUT_MS } from "openclaw/plugin-sdk/number-runtime";
import { upsertSessionEntry } from "openclaw/plugin-sdk/session-store-runtime";
import { formatSessionTranscriptMemoryHitKey } from "openclaw/plugin-sdk/session-transcript-hit";
import { appendSessionTranscriptMessageByIdentity } from "openclaw/plugin-sdk/session-transcript-runtime";
import { closeOpenClawAgentDatabasesForTest } from "openclaw/plugin-sdk/sqlite-runtime-testing";
import {
configureMemoryCoreDreamingState,
configureMemoryCoreDreamingStateForTests,
@@ -223,6 +226,41 @@ function setQmdStateDir(stateDir: string): void {
Reflect.set(process.env, "OPENCLAW_STATE_DIR", stateDir);
}
async function seedQmdSessionTranscript(params: {
agentId: string;
content: string;
sessionId: string;
stateDir: string;
sessionKey?: string;
timestamp?: number | string;
}): Promise<void> {
const sessionsDir = path.join(params.stateDir, "agents", params.agentId, "sessions");
const storePath = path.join(sessionsDir, "sessions.json");
const sessionKey = params.sessionKey ?? `agent:${params.agentId}:qmd:${params.sessionId}`;
const timestamp =
typeof params.timestamp === "number"
? params.timestamp
: Date.parse(params.timestamp ?? "2026-04-07T15:25:04.113Z");
await fs.mkdir(sessionsDir, { recursive: true });
await upsertSessionEntry({
agentId: params.agentId,
sessionKey,
storePath,
entry: { sessionId: params.sessionId, updatedAt: timestamp },
});
await appendSessionTranscriptMessageByIdentity({
agentId: params.agentId,
sessionId: params.sessionId,
sessionKey,
storePath,
message: {
role: "user",
content: params.content,
timestamp,
},
});
}
function restoreQmdStateDir(): void {
if (originalQmdStateDir === undefined) {
Reflect.deleteProperty(process.env, "OPENCLAW_STATE_DIR");
@@ -815,6 +853,7 @@ describe("QmdMemoryManager", () => {
delete (globalThis as Record<PropertyKey, unknown>)[QMD_EMBED_QUEUE_KEY];
delete (globalThis as Record<PropertyKey, unknown>)[MEMORY_EMBEDDING_PROVIDERS_KEY];
resetMemoryCoreDreamingStateForTests();
closeOpenClawAgentDatabasesForTest();
});
it("debounces back-to-back sync calls", async () => {
@@ -5698,32 +5737,22 @@ describe("QmdMemoryManager", () => {
},
} as OpenClawConfig;
const sessionsDir = path.join(stateDir, "agents", agentId, "sessions");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.writeFile(
path.join(sessionsDir, "actual-session-topic-thread.jsonl"),
'{"type":"message","message":{"role":"user","content":"hello mapped session"}}\n',
"utf-8",
);
await fs.writeFile(
path.join(sessionsDir, "sessions.json"),
JSON.stringify({
"agent:main:chat:thread": {
sessionFile: "actual-session-topic-thread.jsonl",
await seedQmdSessionTranscript({
agentId,
content: "hello mapped session",
sessionId: "actual-session",
},
}),
"utf-8",
);
stateDir,
sessionKey: "agent:main:chat:thread",
});
const { manager } = await createManager({ mode: "status" });
await (manager as unknown as { exportSessions: () => Promise<void> }).exportSessions();
const indexPath = (manager as unknown as { indexPath: string }).indexPath;
const identity = resolveQmdSessionArtifactIdentity({
artifactPath: "actual-session-topic-thread.md",
artifactPath: "actual-session.md",
collection: "sessions-main",
indexPath,
searchPath: "qmd/sessions-main/actual-session-topic-thread.md",
searchPath: "qmd/sessions-main/actual-session.md",
});
expect(identity).toEqual({
@@ -6116,15 +6145,8 @@ describe("QmdMemoryManager", () => {
});
it("reuses exported session markdown files when inputs are unchanged", async () => {
const sessionsDir = path.join(stateDir, "agents", agentId, "sessions");
await fs.mkdir(sessionsDir, { recursive: true });
const sessionFile = path.join(sessionsDir, "session-1.jsonl");
const exportFile = path.join(stateDir, "agents", agentId, "qmd", "sessions", "session-1.md");
await fs.writeFile(
sessionFile,
'{"type":"message","message":{"role":"user","content":"hello"}}\n',
"utf-8",
);
await seedQmdSessionTranscript({ agentId, content: "hello", sessionId: "session-1", stateDir });
const currentMemory = cfg.memory;
cfg = {
@@ -6813,20 +6835,18 @@ describe("QmdMemoryManager", () => {
},
} as OpenClawConfig;
const sessionsDir = path.join(stateDir, "agents", agentId, "sessions");
await fs.mkdir(sessionsDir, { recursive: true });
await fs.writeFile(
path.join(sessionsDir, "live-session.jsonl"),
`${JSON.stringify({ type: "message", message: { role: "user", content: "live" } })}\n`,
);
await fs.writeFile(
path.join(sessionsDir, "team.checkpoint.notes.jsonl"),
`${JSON.stringify({ type: "message", message: { role: "user", content: "notes" } })}\n`,
);
await fs.writeFile(
path.join(sessionsDir, "live-session.checkpoint.11111111-1111-4111-8111-111111111111.jsonl"),
`${JSON.stringify({ type: "message", message: { role: "user", content: "checkpoint" } })}\n`,
);
await seedQmdSessionTranscript({
agentId,
content: "live",
sessionId: "live-session",
stateDir,
});
await seedQmdSessionTranscript({
agentId,
content: "notes",
sessionId: "team.checkpoint.notes",
stateDir,
});
const { manager } = await createManager({ mode: "full" });
const sessionExportDir = path.join(stateDir, "agents", agentId, "qmd", "sessions");
extensions/memory-core/src/memory/qmd-manager.ts
+11 -3
View File
@@ -1767,7 +1767,7 @@ export class QmdMemoryManager implements MemorySearchManager {
async sync(params?: MemorySyncParams): Promise<void> {
if (
params?.sessions?.some((session) => session.sessionId.trim().length > 0) ||
params?.sessionFiles?.some((sessionFile) => sessionFile.trim().length > 0)
params?.archiveFiles?.some((sessionFile) => sessionFile.trim().length > 0)
) {
log.debug("qmd sync ignoring targeted session hint; running regular update");
}
@@ -2872,6 +2872,8 @@ export class QmdMemoryManager implements MemorySearchManager {
const entry = await buildSessionEntry(sessionFile, {
generatedByDreamingNarrative: corpusEntry.generatedByDreamingNarrative === true,
generatedByCronRun: corpusEntry.generatedByCronRun === true,
...(corpusEntry.sessionKey ? { sessionKey: corpusEntry.sessionKey } : {}),
...(corpusEntry.updatedAtMs !== undefined ? { updatedAtMs: corpusEntry.updatedAtMs } : {}),
});
if (!entry) {
continue;
@@ -2879,7 +2881,7 @@ export class QmdMemoryManager implements MemorySearchManager {
if (cutoff && entry.mtimeMs < cutoff) {
continue;
}
const targetName = `${path.basename(sessionFile, ".jsonl")}.md`;
const targetName = `${this.sessionExportStem(corpusEntry)}.md`;
const target = path.join(exportDir, targetName);
tracked.add(sessionFile);
const identity = this.buildSessionArtifactMapping(
@@ -2958,6 +2960,12 @@ export class QmdMemoryManager implements MemorySearchManager {
};
}
private sessionExportStem(corpusEntry: SessionTranscriptCorpusEntry): string {
return corpusEntry.transcriptSource === "sqlite"
? corpusEntry.sessionId
: path.basename(corpusEntry.sessionFile, ".jsonl");
}
private refreshSessionArtifactDocIds(): void {
if (!this.sessionExporter) {
return;
@@ -2973,7 +2981,7 @@ export class QmdMemoryManager implements MemorySearchManager {
}
private renderSessionMarkdown(entry: SessionFileEntry): string {
const header = `# Session ${path.basename(entry.absPath, path.extname(entry.absPath))}`;
const header = `# Session ${path.basename(entry.path, path.extname(entry.path))}`;
const body = entry.content?.trim().length ? entry.content.trim() : "(empty)";
return `${header}\n\n${body}\n`;
}
extensions/memory-core/src/session-search-visibility.test.ts
+31
View File
@@ -202,6 +202,37 @@ describe("filterMemorySearchHitsBySessionVisibility", () => {
expect(filtered).toEqual([hit]);
});
it("keeps built-in live SQLite session hits with agent-scoped logical paths", async () => {
combinedSessionStore = {
"agent:main:only": {
sessionId: "w1",
updatedAt: 1,
sessionFile: "sqlite-session://main/w1",
},
};
const hit: MemorySearchResult = {
path: "sessions/main/w1.jsonl",
source: "sessions",
score: 1,
snippet: "x",
startLine: 1,
endLine: 2,
};
const cfg = asOpenClawConfig({
tools: {
sessions: { visibility: "all" },
agentToAgent: { enabled: true, allow: ["*"] },
},
});
const filtered = await filterMemorySearchHitsBySessionVisibility({
cfg,
requesterSessionKey: "agent:main:main",
sandboxed: false,
hits: [hit],
});
expect(filtered).toEqual([hit]);
});
it("keeps global-scope session hits for non-default agents", async () => {
combinedSessionStore = {
global: {
extensions/memory-core/src/short-term-promotion.ts
+26 -5
View File
@@ -414,7 +414,10 @@ function hasDreamingNarrativeLead(snippet: string): boolean {
return /\b(?:Candidate|Reflections?):/i.test(head);
}
function isContaminatedDreamingSnippet(raw: string): boolean {
function isContaminatedDreamingSnippet(
raw: string,
opts: { allowTranscriptTurnSnippet?: boolean } = {},
): boolean {
const snippet = normalizeSnippet(raw);
if (!snippet) {
return false;
@@ -424,7 +427,7 @@ function isContaminatedDreamingSnippet(raw: string): boolean {
DREAMING_TRANSCRIPT_PROMPT_LINE_RE.test(snippet) ||
RAW_SESSION_METADATA_RE.test(snippet) ||
RAW_CONVERSATION_SUMMARY_RE.test(snippet) ||
RAW_TRANSCRIPT_TURN_RE.test(snippet) ||
(!opts.allowTranscriptTurnSnippet && RAW_TRANSCRIPT_TURN_RE.test(snippet)) ||
MEMORY_FLUSH_PROMPT_RE.test(snippet) ||
PROMOTION_SCORE_METADATA_RE.test(snippet)
) {
@@ -615,7 +618,12 @@ export function normalizeShortTermRecallStore(raw: unknown, nowIso: string): Sho
? entry.claimHash.trim()
: undefined;
const fullSnippet = typeof entry.snippet === "string" ? normalizeSnippet(entry.snippet) : "";
if (fullSnippet && isContaminatedDreamingSnippet(fullSnippet)) {
if (
fullSnippet &&
isContaminatedDreamingSnippet(fullSnippet, {
allowTranscriptTurnSnippet: isShortTermSessionCorpusPath(entryPath),
})
) {
continue;
}
const snippet = truncateShortTermSnippet(fullSnippet);
@@ -1063,6 +1071,10 @@ export function isShortTermMemoryPath(filePath: string): boolean {
return SHORT_TERM_BASENAME_RE.test(normalized);
}
function isShortTermSessionCorpusPath(filePath: string): boolean {
return SHORT_TERM_SESSION_CORPUS_RE.test(normalizeMemoryPath(filePath));
}
function normalizeMemoryPathForWorkspace(workspaceDir: string, rawPath: string): string {
const normalized = normalizeMemoryPath(rawPath);
const workspaceNormalized = normalizeMemoryPath(workspaceDir);
@@ -1415,7 +1427,12 @@ export async function recordShortTermRecalls(params: {
const normalizedPath = normalizeMemoryPath(result.path);
const rawSnippet = normalizeSnippet(result.snippet);
const snippet = truncateShortTermSnippet(rawSnippet);
if (!rawSnippet || isContaminatedDreamingSnippet(rawSnippet)) {
if (
!rawSnippet ||
isContaminatedDreamingSnippet(rawSnippet, {
allowTranscriptTurnSnippet: isShortTermSessionCorpusPath(normalizedPath),
})
) {
continue;
}
const claimHash = buildClaimHash(rawSnippet);
@@ -1841,7 +1858,11 @@ export async function rankShortTermPromotionCandidates(
if (!entry || entry.source !== "memory" || !isShortTermMemoryPath(entry.path)) {
continue;
}
if (isContaminatedDreamingSnippet(entry.snippet)) {
if (
isContaminatedDreamingSnippet(entry.snippet, {
allowTranscriptTurnSnippet: isShortTermSessionCorpusPath(entry.path),
})
) {
continue;
}
if (!includePromoted && entry.promotedAt) {
extensions/ollama/provider-discovery.test.ts
+18 -14
View File
@@ -31,6 +31,10 @@ describe("Ollama provider", () => {
const countFetchCallUrls = (fetchMock: ReturnType<typeof vi.fn>, suffix: string): number =>
fetchCallUrls(fetchMock).reduce((count, url) => count + (url.endsWith(suffix) ? 1 : 0), 0);
const stubOllamaFetch = (fetchMock: ReturnType<typeof vi.fn>) => {
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
};
const countWarnCallsIncluding = (warnSpy: ReturnType<typeof vi.spyOn>, text: string): number => {
let count = 0;
for (const [message] of warnSpy.mock.calls) {
@@ -122,7 +126,7 @@ describe("Ollama provider", () => {
}
return notFoundJsonResponse();
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
return fetchMock;
};
@@ -221,7 +225,7 @@ describe("Ollama provider", () => {
}
return notFoundJsonResponse();
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
env: { OLLAMA_API_KEY: "test-key", VITEST: "", NODE_ENV: "development" },
@@ -247,7 +251,7 @@ describe("Ollama provider", () => {
}
return notFoundJsonResponse();
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
env: { OLLAMA_API_KEY: OLLAMA_LOCAL_AUTH_MARKER, VITEST: "", NODE_ENV: "development" },
@@ -271,7 +275,7 @@ describe("Ollama provider", () => {
const fetchMock = vi
.fn()
.mockRejectedValue(new Error("connect ECONNREFUSED 127.0.0.1:11434"));
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
env: { VITEST: "", NODE_ENV: "development" },
@@ -292,7 +296,7 @@ describe("Ollama provider", () => {
const fetchMock = vi
.fn()
.mockRejectedValue(new Error("connect ECONNREFUSED 127.0.0.1:11434"));
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
await runOllamaCatalog({
config: {
@@ -326,7 +330,7 @@ describe("Ollama provider", () => {
}
return notFoundJsonResponse();
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
env: { OLLAMA_API_KEY: "test-key", VITEST: "", NODE_ENV: "development" },
@@ -351,7 +355,7 @@ describe("Ollama provider", () => {
}
return jsonResponse({ model_info: { "llama.context_length": 65536 } });
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
env: { OLLAMA_API_KEY: "test-key", VITEST: "", NODE_ENV: "development" },
@@ -380,7 +384,7 @@ describe("Ollama provider", () => {
it("should skip discovery fetch when explicit models are configured", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const explicitModels: ModelDefinitionConfig[] = [
{
id: "gpt-oss:20b",
@@ -423,7 +427,7 @@ describe("Ollama provider", () => {
it("should use synthetic local auth for configured remote providers without apiKey", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {
@@ -460,7 +464,7 @@ describe("Ollama provider", () => {
it("should not use synthetic local auth for configured cloud providers without apiKey", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {
@@ -497,7 +501,7 @@ describe("Ollama provider", () => {
it("uses resolved discovery api key when configured cloud apiKey is an env marker", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {
@@ -539,7 +543,7 @@ describe("Ollama provider", () => {
it("uses resolved discovery api key for configured cloud providers without apiKey", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {
@@ -580,7 +584,7 @@ describe("Ollama provider", () => {
it("keeps synthetic local auth when a local provider also has a discovery key", async () => {
await withoutAmbientOllamaEnv(async () => {
const fetchMock = vi.fn();
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {
@@ -628,7 +632,7 @@ describe("Ollama provider", () => {
}
return notFoundJsonResponse();
});
vi.stubGlobal("fetch", withFetchPreconnect(fetchMock));
stubOllamaFetch(fetchMock);
const provider = await runOllamaCatalog({
config: {

Some files were not shown because too many files have changed in this diff Show More