Commit Graph

2 Commits

Author SHA1 Message Date
Patrick Buckley 7da07e2350 perf(attachments): per-send wire-part memo to stop re-rasterizing every round-trip
_resolve_attachments re-runs on every agentic round-trip (and per fallback model), each time re-fetching every attachment across the full history and re-rasterizing / re-base64'ing it. A 10-page PDF in a 10-cycle tool turn was rendered dozens of times.

Add a per-send memo (self._wire_part_cache) keyed by (attachment_id, caps-signature): the materialized wire part is computed at most once per send. The cache is None outside a send (display/export paths unaffected) and reset per send to bound the heavy rasterized-page parts and pick up any mid-session capability change. Skip the DB fetch entirely when every id is already cached.

Also peek the perception (alias, content_hash) memo before building parts in _perception_fallback_part, so a cross-send describe hit no longer wastes a PDF rasterize. Leaves pdf.py's deliberate no-module-cache stance intact — the per-send scope addresses the round-trip amplification without the durable store it defers.

Adds describe_peek() + per-send-cache and peek tests.
2026-06-16 00:48:14 -07:00
Patrick Buckley 558ddadc79 feat(attachments): universal perception fallback for non-native modalities
Add a `perception.model_alias` model role: when the primary model can't ingest
an attachment natively and can't be shown a degraded-but-native form, a
configured perception model perceives it and its output is carried as text.
Mirrors the STT role — a role alias plus a module-level memo so the extra LLM
round-trip runs once per attachment, not once per conversation turn. The call
goes through the provider abstraction's create_completion (the path the intent
judge uses), so any vision/omni provider works.

Bottom-tier, universal ladder — perception only fills the remaining gap:
- pdf  : native supports_pdf -> rasterize-to-vision-primary -> perception
         -> extracted text -> placeholder
- image: native vision -> perception (non-vision primary) -> native image_url
- audio: native supports_audio_input -> STT -> perception (omni) -> placeholder

Folds in two review findings the role subsumes:
- bug-1: thread the active attempt's capabilities into _resolve_attachments
  (bound in _try_stream) so a model fallback materializes attachments against
  the fallback model's caps, not the primary's.
- bug-2: charge a by-reference pdf/audio a bounded budget min(size_bytes, 16K)
  instead of zero, so a large-attachment turn isn't budgeted as ~empty (the
  exact materialized size isn't known until wire build).
2026-06-16 00:48:14 -07:00