* feat(sqlite): migrate persisted media to canonical facts and stop legacy writes
PR 3 of the media legacy retirement program — the operator-approved
canonical cutover.
- openclaw doctor --fix owns one idempotent migration: active
transcript_events rows canonicalize to __openclaw.media (facts-first
gap-fill, bare legacy kinds to fact.kind, transcribed indexes and
workspace dirs onto per-fact fields) via the transcript replacement
owner; cold plain/.zst archives rewrite through temp-file + codec
readback + event/id verification + atomic replace; trajectory
runtime snapshots canonicalize IN PLACE (telemetry preserved, never
row deletion). Invalid JSON, genuinely ambiguous legacy-only sparse
alignment, or a changed source aborts that owner without partial work;
reruns are no-ops.
- Per-agent schema advances to v16 as a pure downgrade guard (main
independently took v15 for board/session-sharing tables; no
columns/tables/indexes change here, shared-state DB untouched). v15
databases repair canonical indexes before the version assertion so
repairable installations never strand.
- The user-turn builder stops writing top-level legacy Media* fields;
shouldPersistStructuredMediaEntries and the aligned projection mode
are deleted; the generic transcript append boundary canonicalizes
every message role so SDK/mirror writers cannot mint new legacy rows.
- Internal persisted-reader legacy fallbacks are removed; the public
SDK projection stays until retirement PR 4's window expires.
Hardening from three adversarial review rounds, each with fixture
regressions: in-place trajectory canonicalization instead of row
deletion; repair-before-assert on the v15 path; all-roles append
canonicalization; duplicate-preserving exact row rewrites; v0-v15
reopen guards; complete canonical facts bypass compact legacy
projections (PR-1 dual-write rows migrate cleanly); SQLite LIKE
underscore escaped so populated foreign databases are never claimed.
* fix(sqlite): align schema-support metadata and gates with the v16 cutover
package.json agent schema support advances to 16; verifier and board
parity fixtures run doctor migration before steady-state access (the
production guards were correct); two test-only exports removed; the
migration module registered in the doctor raw-SQLite allowlist.
* refactor(media): read persisted media facts first across all consumers
PR 2 of the media legacy retirement program. Gateway display projection,
Control UI (history filtering, attachment rendering, initial-turn
handoff), embedded and plugin-harness hydration, replay/prune, late-media
reconstruction, hooks, media understanding, and trajectory capture now
read nested __openclaw.media facts first. Legacy top-level Media* fields
survive only in the narrow persisted-message compatibility reader (rows
predating the PR-1 dual-write) and the public SDK projection.
Non-goals pinned by tests: rendered batches, reply-payload dedupe,
durable session/outbound queues, and lowercase ReplyPayload.mediaUrl(s)
are untouched contracts.
Consumer matrix covers {legacy-only, facts-only, both-equal,
both-conflict, sparse, type-only, media-only} inputs across every
consumer family; prompt/cache goldens byte-identical; 131-file legacy
consumer sweep run individually (two failures reproduce unchanged on
clean main); 1,047 broad media tests and the full UI lane (5,553) green.
* refactor(media): drop caller-less staging predicate and localize empty-user-row classification
The consumer flip left hasStagedMediaProjection without production
callers (facts-only staging uses hasStagedMediaFacts); delete it and
retest the staged merge matrix against the facts predicate. Unexport
LEGACY_MEDIA_CONTEXT_KEYS (internal only; the derived type stays public).
Move isEmptyUserTextOnlyMessage beside the other message-shape
classifiers in ui/src/lib/chat/message-extract.ts where chat-history
imports it cross-module.
* chore(ui): drop unused hasTranscriptMediaFacts import after predicate move
* chore(ui): localize hasTranscriptMediaFacts after its last external consumer moved in
* style: format staged-media matrix assertion
* feat(sdk): always persist media facts and ship facts-first replacements for legacy Media* surfaces
PR 1 of the media legacy retirement program (audit-frozen, 4 PRs).
- Every media-bearing user turn now persists normalized __openclaw.media
facts unconditionally while continuing to emit the legacy top-level
Media* projection byte-identically (dual-write bridge; the conditional
shouldPersistStructuredMediaEntries gate now always includes media).
- New replacement APIs, shipped before any removal: typed hook media
facts (media[], originalMedia[], mediaStagingPending) on message
events; {{AttachmentPath}}/{{AttachmentUrl}}/{{AttachmentContentType}}/
{{AttachmentDir}}/{{AttachmentIndex}} template variables; focused
openclaw/plugin-sdk/media-local-roots subpath split out of the
deprecated agent-media-payload facade.
- Every legacy surface carries @deprecated naming its replacement, under
one named compatibility record media-legacy-projection with the
operator-approved removeAfter 2026-10-01 (two release trains; deletion
additionally gates on a clean published-plugin artifact sweep).
- Generic transcript append invariant documented; SDK migration, hooks,
and configuration docs updated to the facts-first path.
Writer golden matrix proves legacy bytes and model prompt bytes are
unchanged while nested facts become unconditional. 2,189 broad media
tests green; SDK api-baseline regenerated on fresh-env Testbox.
* feat(sdk): register media-local-roots subpath exports and deprecation metadata
Completes PR 1: package export map for openclaw/plugin-sdk/media-local-roots
plus the deprecated-subpath inventory and doc metadata entries for the
media-legacy-projection record.
* chore(sdk): track media-local-roots entrypoint and deprecated-export budgets
* fix(sdk): keep deprecated MSTeams buildMediaPayload re-export through the compat window
Deleting shipped runtime-api re-exports belongs to retirement PR 4 after
the media-legacy-projection window; PR 1 only deprecates. Also formats
the migration-guide schedule table.
* docs: regenerate docs map for media migration additions
* refactor(media): hydrate and prune images from structured facts, delete marker-text parsing
Embedded, CLI, host-context replay, and plugin-harness image hydration now
consume the persisted/runtime MediaFact carrier instead of parsing
OpenClaw-authored [media attached:]/[Image: source:] marker text. The
prompt-text attachment parser family (splitPromptAndAttachmentRefs, leading/
trailing marker extractors, ref-count consumption, text-based prune scrubbing)
is deleted; explicit user-authored paths and file URLs still hydrate.
- Described-image suppression is a declared MediaFact.hydrationSuppressed
field carried across copy/reprojection boundaries; the runtime symbol
plumbing is removed.
- Persisted facts plus mediaImageLayout are authoritative at every hydration
entry point; image blocks reconcile by structural fact index, never
byte/MIME equality.
- Offloaded attachments (image and non-image) persist their media://inbound
claim-check alias so ownership-aware history pruning can scrub markers
after persistence; factless pre-MediaFact rows keep narrow legacy redaction.
- History pruning removes facts, layout, and block provenance structurally
before replay.
Prompt bytes are golden-equal throughout; goldens from the fact-carrier PR
are untouched. 1,303 tests across 44 files; delegated check:changed green
(run 29980534186, pinned to the declared merge-base).
* fix(media): fail missing inline image slots
* test(agents): carry offloaded CLI retry image as a structured fact
The retry-image reliability test modeled an offloaded attachment with only
its [media attached:] marker text; offloaded attachments are carried as
structured media facts and marker text is presentation-only, never parsed
for hydration.
Media-facts program PR 5 — the planned additive bridge. Structured
facts travel alongside existing prompt text through prompt prelude,
GetReplyOptions, FollowupRun, queue collect/defer/retry, active
steering, embedded/CLI runs, Gateway offloads (offloadedRefs no longer
discarded; AgentContentPhaseResult carries media), Tlon, and late-media
projection — with prompt bytes golden-equal everywhere. Facts follow
the same adoption/idempotency/late-append lifecycle as prompt text.
Intentionally +175 prod LOC per the audit; the facts-first
hydration/prune deletion PR consumes this carrier next.
Media-facts program PR 4: legacy Media* fields are emitted only by the
user-turn transcript writer (persisted rows byte-identical, no
migration); reads normalize into ordered facts; documented singular
template variables project per attachment with per-attachment CLI
execution. Review fix: normalized attachments carry their original fact
index so sparse aligned slots cannot shift CLI url/type projections.
Final sweep of the media-placeholder program: with all eleven producers
migrated, the consumer regexes are dead input and are hard-deleted —
media-understanding token un-mixing (bodies are pure user text now),
the persisted-user-turn placeholder drop, the session-memory strip, and
the inbound-event mediaPlaceholder field with its replacement branch
(an intentional removal of shipped SDK surface per the program's
hard-delete decision: plugins still minting placeholder bodies render
them as literal text until updated). formatMediaPlaceholderText remains
the single owner of the <media:kind> vocabulary, used only for
text-only carriers. Legacy tokens in pre-migration history remain
ordinary text.
* refactor(sessions): keep late-media attachment text out of persisted transcripts
Late-resolved media appended a second user turn whose content was literal
"[media attached: <path>]" lines, so clients rendered wire scaffolding
under the image — same leak class as #111204. Persist those turns bare
(media fields + a protected __openclaw.lateMedia marker, image blocks
kept), and project the attachment lines at the LLM boundary and the
history-image-prune stage instead, byte-identical for providers across
string, array, and legacy forms. Also dedupes media-field sniffing into
a shared hasPersistedMedia helper.
* fix(sessions): satisfy type gates for late-media boundary projection
* fix(agents): narrow prune content cast to array element type
* test(sessions): split transcript persistence suite under max-lines cap
* feat(codex): fork upstream-linked sessions at a message via thread/fork
* fix(gateway): fail closed for rewind and branch switch on upstream-linked sessions
* fix(codex): fail closed on first-message forks, image-only prompts, and orphan archival
* fix(codex): baseline retained history and reject paginated threads on upstream fork
* fix(codex): validate the full fork prefix and fail closed across crash windows
* fix(codex): treat all non-text inputs as unverifiable in fork drift checks
* fix(codex): support first-message forks as empty-history upstream cuts
* fix(codex): reject source-id reuse and unverifiable hidden inputs in fork boundaries
* refactor(codex): materialize upstream forks from verified thread read-back
* fix(codex): satisfy strict type lanes and knip for upstream fork
* fix(sessions): stop persisting media-only caption placeholder into transcripts
Media-only user turns persisted the literal "[User sent media without
caption]" as message content, so every client rendered the model-facing
scaffolding as a visible subtitle under the image. Persist bare content
instead and inject the placeholder at the single LLM-boundary stamping
site (like per-message timestamps, #3658), keeping provider-visible
bytes identical for current turns, history replay, and legacy
transcripts.
* fix(agents): satisfy lint and test-type gates for boundary media injection
* refactor: delete dead infra and config exports
* refactor: preserve live infra and config contracts
* refactor(config): remove obsolete file-store lifecycle APIs
* refactor(infra): finish current-main dead export cleanup
* refactor(deadcode): trim auto-reply and CLI exports
* refactor(deadcode): trim cron and task exports
* refactor(deadcode): trim fleet and process exports
* test(deadcode): exercise live task and process seams
* test(fleet): cover stream redaction through owner module
* refactor(security): trim dead internal exports
* refactor(secrets): trim dead internal exports
* refactor(deadcode): trim remaining src exports
* refactor(deadcode): remove test-only runtime exports
* refactor(deadcode): trim pairing test exports
* refactor(deadcode): reconcile refreshed baseline
* test(auto-reply): deduplicate queue state imports
* fix(core): make indexed access explicit in auto-reply, infra, and config
Part 1/3 of the src NUIA phase-3b burn-down (#104600): iteration and
destructuring over index reads, boundary guards on parsed input, and
named invariants. Config path walkers bind the path head once; SQLite
migration key handling is hoisted without query-shape changes.
* fix(core): make indexed access explicit in cli, gateway, commands, security, shared
Part 2/3: argv/token selection restructured, gateway event/attachment
invariants named, security parsers stay fail-closed (invariant
violations throw), edit-distance matrices access checked entries.
* fix(core): make indexed access explicit across remaining src surfaces
Part 3/3: channels, plugins, process, cron, plugin-sdk, media, logging,
tui, hooks, daemon, and small directories. Latent bug fixed: a tailnet
resolver could leak undefined through a string|null contract and now
fails with a descriptive local error.
* fix(core): keep optional boundaries optional after per-commit review
Review findings: expectDefined misused where absence is a legitimate
state. CLI --profile/route-args missing next tokens take their existing
miss paths; help normalization compares --help against the last
positional again; first-time plugin install spreads absent cfg.plugins;
denylist scan iterates manifest dependency entries instead of throwing
on omitted sections; tailnet resolver returns a guaranteed string at
the source instead of a caller-side undefined throw.
* refactor(core): closed-key provider labels and honest optional passthroughs
PROVIDER_LABELS becomes a satisfies-typed closed record (static reads
provably defined; dynamic lookups go through providerUsageLabel with
honest string|undefined). Status-scan overview passes its optional
params through unchanged instead of asserting them.
* fix(channels): make getChatChannelMeta honestly optional
The original signature claimed ChatChannelMeta while leaking undefined
on bundled channel id metadata drift; three of four callers already
handled absence. The return type now says so, and the one assuming
caller falls back to the raw channel label.
* fix(core): index-safety for post-rebase main drift
Covers the sqlite-sessions flip and auth-source-plan code that landed
mid-phase, plus the channel-validation test consuming the now honestly
optional getChatChannelMeta.
* refactor(channels): split chat-meta accessors along the SDK contract
getChatChannelMeta keeps its shipped plugin-SDK signature (defined for
bundled ids, fail-loud on impossible misses); new findChatChannelMeta
carries the drift-tolerant optional contract for core auto-enable and
formatting paths.
* fix(qa-channel): own channel metadata instead of a guaranteed-undefined catalog lookup
qa-channel spread getChatChannelMeta over an id that is never in the
bundled catalog, shipping an empty setup meta by accident; the fail-loud
SDK accessor exposed it. The channel now declares its metadata once.
* fix(gateway): heartbeat projection lookahead is optional at the transcript tail
expectDefined wrapped messages[i + 1] whose absence on the final message
is the normal case; the adjacent ternary already handled it. Restores
the plain optional read with an explicit guard in the pair condition.
* fix(plugin-sdk): channel plugin factory tolerates non-bundled channel ids again
createChannelPluginBase spreads bundled catalog meta for ANY channel id,
where absence is the normal case for external plugins; the resolver is
honestly optional again while the exported bundled-id accessor keeps the
fail-loud contract.
* fix(core): spreads of optional config sections stay optional
Fresh-setup and first-install paths (crestodian setup inference, hook
installs, agent config base, target agent models) legitimately lack the
section being rebuilt; spreading undefined is the shipped {} semantics.
Removes the remaining gratuitous assertion wraps found by tree audit.
* refactor(sessions): migrate runtime storage to sqlite
* test(sessions): fix sqlite CI regressions
* test(sessions): align remaining sqlite fixtures
* fix(codex): require sqlite trajectory recorder
* test(sessions): align orphan recovery sqlite fixture
* test(sessions): align sqlite rebase fixtures
* fix(sessions): finish current-main integration of the sqlite flip
Resolve the whole-store SDK removal across its owner boundary: drop the
loadSessionStore re-export and the registry whole-store wrappers, wire
hasTrackedActiveSessionRun into gateway chat, complete the
preserveLockedHarnessIds cleanup contract, flip the codex thread-history
import to storePath targets, and port remaining main-side tests from
file-store helpers to session accessor reads.
* chore: drop committed pebbles log, revert plugin-inspector bump, refresh generated docs
Remove the 1.8k-line .pebbles/events.jsonl work log from the branch, restore
the plugin-inspector advisory lane to main's pinned 0.3.10 so the supply-chain
bump gets its own review, and regenerate docs_map, the plugin SDK API baseline,
and the export-surface ratchet for the merged tree.
* feat(sessions): keep archived transcripts by default with zstd cold storage
Codex-style retention: deleting or resetting a session archives its
transcript as a zstd-compressed JSONL artifact (plain when the runtime
lacks node:zlib zstd) and keeps it until the disk budget evicts oldest
first. resetArchiveRetention now governs both deleted and reset archives
and defaults to keep; maxDiskBytes defaults to 2gb so retention stays
bounded, with archives evicted before live sessions. The cron reaper
follows the same knob instead of deleting archives on its own timer.
* fix(state): converge agent DB migration lineages and bound database growth
Merge coherence: run both structure-gated legacy memory-schema repairs
(flip-lineage drop, main-lineage identity rebuild) before the flip
migration so pre-flip v1/v2 and pre-merge flip v1/v4 databases all
converge, and hoist foreign_keys=OFF outside the schema transaction
where the pragma was silently ignored and the v1 sessions rebuild
cascade-deleted session_entries.
Growth guards: fresh agent DBs enable auto_vacuum=INCREMENTAL, WAL
maintenance releases freed pages in bounded passes (never a blocking
full VACUUM), and doctor reports state/agent DB bloat from freelist
stats.
* fix(codex): resolve the store path for thread-history import via the SDK
The supervision catalog passed the legacy sessionFile locator to the
storePath-targeted transcript mirror; resolve the agent store path with
the session-store SDK helper instead of a runtime-object seam so test
fakes and headless callers need no extra surface. Drop the obsolete
missing-session-id preprocessing case: sessions rows are NOT NULL on
session_id and upsert repairs id-less patches at write time.
* fix(sessions): fail safe on malformed disk-budget config and doctor stat errors
A malformed explicit maxDiskBytes disables the budget instead of
falling back to the destructive 2gb default the user never chose, and
the doctor bloat check skips databases whose paths stat-fail instead of
aborting doctor.
* fix(sessions): complete sqlite conflict translations
* test(sqlite): align hardening checks with maintenance
* test(sessions): inspect compressed transcript archives
* fix(tests): await session seeds and drop unused helpers flagged by CI lint
The five unawaited writeSessionStoreSeed calls raced their SQLite seeds
against the assertions, failing compact shards; the bloat probe drops a
useless initializer and the merged tests drop now-unused helpers.
* test(sessions): type legacy proof events directly
* test(sessions): align hardening contracts
* perf(sessions): read usage transcript sizes from SQL aggregates
Usage/cost scans walked every session and materialized every transcript
event just to re-stringify it for a byte estimate — the #86718 stall
class reborn on the DB. readTranscriptStatsSync sums stored JSON bytes
in SQLite without loading a single row.
* fix(sessions): re-root foreign-root transcript paths onto the current sessions dir
Restored backups, moved OPENCLAW_STATE_DIR, and rehearsal copies carry
absolute sessionFile paths from the old root; the containment fallback
kept those foreign paths, so migration read (and would archive) files in
the original root and reported local copies missing. Re-root the
canonical agents/<id>/sessions suffix onto the current dir when the file
exists there; genuine cross-root layouts still fall through unchanged.
* test(agents): seed harness admission through sqlite
* fix(sqlite): close agent db on pragma setup failure
* fix(doctor): compact and retrofit incremental auto-vacuum after session import
The migration is the sanctioned offline window: post-import compact
reclaims import churn and applies auto_vacuum=INCREMENTAL to databases
created before the fresh-DB pragma existed, so runtime maintenance can
release pages in bounded passes on every install.
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
* feat(sessions): durable session state events with parent invalidation
Parents no longer act on stale child-session state: a durable, typed
signal log (session_state_events + per-agent heads) records direct human
messages to children, child spawn/terminal outcomes, goal changes, and
compaction at the seams where those facts are created. A frozen-watermark
cursor protocol (session_watch_cursors) delivers one coalesced system
notice to the parent through the existing system-event + heartbeat wake
idiom, acknowledged at the shared generic drain, with a deferred startup
sweep for restart recovery. session_status gains stateVersion and
changesSince (with exact pruned-history gap signaling); sessions_list
rows gain stateVersion.
Closes#104565
* chore: regenerate docs map and prompt snapshots for session_status changesSince
* fix(agents): keep model fallback turn-local instead of persisting over user pins
* fix(telegram): use live config snapshots per operation
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* test(telegram): fix config snapshot type coverage
---------
Co-authored-by: Ayaan Zaidi <hi@obviy.us>
* fix(agents): isolated cron busts prompt prefix cache via per-run session id
Isolated cron runs carry a per-run :run:<id> session scope (#91685) rendered
verbatim into the cached system-prompt Runtime line, re-busting byte-exact
prefix caching for the tool catalog after it every run (#96677, #43148 class).
buildRuntimeLine now renders the stable base session key and drops the per-run
id the run scope duplicates; parseCronRunScopeSuffix is gated to the
isolated-cron key shape so a :run: segment in any other session key is never
truncated.
* fix(agents): preserve mixed-case cron run markers
* test(agents): cover rotated cron session identity
---------
Co-authored-by: Peter Steinberger <steipete@gmail.com>
Session-scoped tool-result projection freeze (shrink-only, stable
fallback identities for ambiguous tool ids) and a sent-watermark so
late-resolving inbound media appends a new user turn instead of
rewriting an already-sent slot. Stops Anthropic prompt-cache tail
invalidation on busy/media-heavy embedded sessions.
Peter Steinberger