feat: enable prompt caching for Anthropic and OpenAI providers (#104)

* feat: enable prompt caching for Anthropic and OpenAI providers

Activate automatic prompt caching on both LLM providers to reduce input
token costs on multi-turn conversations. Anthropic gets cache_control:
ephemeral (90% savings on cache hits), OpenAI GPT-5.x gets 24h extended
cache retention (free). Cache metrics flow end-to-end through the entire
data pipeline: provider → session → server SSE → MQ protocol → storage →
Prometheus metrics → admin Usage tab.

- AnthropicProvider: top-level cache_control on all requests, extract
  cache_creation_input_tokens and cache_read_input_tokens from streaming
  and non-streaming responses
- OpenAIProvider: prompt_cache_retention=24h for GPT-5.x, extract
  cached_tokens from usage.prompt_tokens_details
- UsageInfo: new cache_creation_tokens and cache_read_tokens fields
- Migration 020: add cache columns to usage_events table
- Storage: record_usage_event and query_usage updated (sqlite + pg)
- Metrics: turnstone_tokens_total{type="cache_creation|cache_read"}
- Server: on_status passes cache tokens to SSE, storage, and metrics
- MQ: StatusEvent carries cache fields through bridge
- SDKs: Python and TypeScript StatusEvent types updated
- OpenAPI: UsageBreakdownItem schema includes cache fields
- Console UI: Usage tab shows cache write/read as secondary readouts
  with visual separator, dimmed when zero
- 16 new tests, docs and 3 diagrams updated

* fix: address Copilot review feedback

- Fix MQ protocol diagram clipping by switching to vertical package
  layout (inbound on top, outbound below) with package aliases
- Regenerate OpenAPI snapshots to include cache_creation_tokens and
  cache_read_tokens on UsageBreakdownItem
- Replace fragile MagicMock(spec=[]) + del pattern with
  types.SimpleNamespace in cache metrics missing-attributes test
This commit is contained in:
Patrick Buckley
2026-03-16 01:59:55 -07:00
committed by GitHub
parent 471bf89c8b
commit 80e1924d7f
32 changed files with 1099 additions and 36 deletions
+2 -2
View File
@@ -1,3 +1,3 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01fbb3338df6426cefc2811541a865f268673b4febf32f524c264d120bc068fa
size 589546
oid sha256:0e605963c649574c7bf987b2d338257c78bc1fc68b524ac8276bb25365035e06
size 594096