Every streamed delta saves a snapshot of the in-progress response so a reconnecting client can resume it, and each save rebuilt the assistant text from scratch. On the Chat Completions path that re-joined every accumulated chunk, including on saves carrying no new text, so a long answer followed by a large tool call re-joined the whole answer once per argument chunk. The Responses API path never collects those chunks and reads the text back out of the output items instead, where the blank check copied it in full every time.
The joined string is now kept and reused until another chunk arrives, since content_parts is only ever appended to; the nonlocal declaration that suggested otherwise was already dead and is dropped, and inlining the single-use helper removes an unreachable branch with it. The blank check in get_output_text now tests the text rather than allocating a stripped copy of it, which is equivalent for all twelve of its callers. Text streaming on the Chat Completions path is unchanged, since a text delta always appends before it saves.
| stream | before | after |
| --- | --- | --- |
| 20k-char answer, 2000 tool-argument chunks | 21.4 ms | 0.06 ms |
| Responses API, 40k deltas, 200k chars | 80.7 ms | 50.5 ms |
Without Redis nothing extra is retained, since the snapshot store already held that string; with Redis one copy of the response text stays alive while the stream runs.
Tag detection for the code interpreter ran regardless of the tool-calling mode, so a model in Native (Agentic) Mode that emitted <code_interpreter> blocks in ordinary reply text had that code sent to the executor. Native mode never teaches the tag format and exposes execute_code as a builtin tool, so the parser had nothing legitimate to pick up there.
Gates detection on the legacy mode, matching the condition that already decides whether the tag prompt is injected at all. The five authorization checks are unchanged, and native mode keeps executing through the tool.
Deployments on native mode whose models emit the tags unprompted will now see them rendered as text.
Streamed responses are scanned for reasoning and code interpreter tags. To work out where the last complete tag ended, the scanner searched backwards from the start of the accumulated text on every chunk, once per tag set. Ordinary prose contains no angle bracket, so that search never stopped early and read the entire response back every time. The cost grows with the square of the response length, and this scanning is on unless a model turns it off.
The two positions are now carried forward as the text grows, so each chunk only scans the characters it added.
Measured on CPython 3.12, a 270 KB response streamed in 27000 chunks:
| response text | before | after |
|---|---|---|
| no newlines | 7690 ms | 40.6 ms |
| with newlines | 5695 ms | 41.7 ms |
The carried positions match a full rescan at every step of 36282 randomized replays, covering text with no markers, newlines only, dense markers, real tags and truncation part way through.
A stream filter function, or a provider that puts something other than a string in a delta, makes the streaming handler concatenate a string with a non-string. That raises TypeError, and the broad handler wrapped around the whole per-chunk block swallows it at debug level and moves on. The chunk's text never reaches the message the user sees, and nothing above debug level says why.
The content and reasoning fields are now coerced to text once, where they are read off the delta, ahead of every consumer. The coercion is guarded on truthiness, so falsy values such as an empty list still skip the block exactly as before, and the accumulated content receives byte for byte what it received previously.
Checked against 14 delta shapes covering strings, empty values, numbers, booleans, None, lists, dicts and a content array: the truthiness gate and the accumulated content are identical before and after.
A chat attached via the "+" menu or dropped from the sidebar references an
existing chat by id and carries no url. add_file_context() filtered on
`file.get('url')`, so the reference was dropped from <attached_files>
entirely and the model was never told it existed.
When the RAG file-context path is enabled the chat content still reaches
the model as <source> context, which masked this. With file_context
disabled that path is skipped, and get_attached_knowledge() only promotes
collection/note items into <attached_knowledge> - so an attached chat was
visible in the UI but invisible to the model, which then reported having
no chat attachments despite having a view_chat tool available.
Keep chat references and emit their id so the model can resolve them with
view_chat. The url attribute is now conditional, since a chat has none;
the id guard it replaces was dead once the filter guarantees a url or a
chat id.
Co-authored-by: Claude <noreply@anthropic.com>
Listing skills ran one database query per skill in the instance. A non-admin opening the list on a workspace with 500 skills issued over 500 queries, the paginated list re-resolved the caller's group membership once per row, and every chat message carrying a skill loaded every skill the user can read, full body and owner included, to use the two or three it actually referenced.
Skills now arrive already filtered: the owner-or-grant check runs in the query as an EXISTS subquery, the same way prompts and the search endpoints already do it, the per-item write flag uses the existing batch grant lookup, and the chat path asks only for the skill ids the request names.
Measured with 500 skills of which 3 are visible to the caller: 504 queries and ~300 ms before, 4 queries and ~2.6 ms after. The resulting set is unchanged for owner, public, direct-user, group and multi-grant entries, for both read and write.
Assembling the builtin tools for a chat message fetched the chat row a second time to answer one question: whether this is a note chat. The caller had loaded that same row a few lines earlier, from the same id in the same metadata dict, and had already evaluated the same predicate for its own note handling. So every message with builtin tools enabled read the whole conversation blob twice.
The caller now works the flag out once and passes it down. Tool assembly no longer touches a chat model at all, so the two files cannot drift apart when the shape of that metadata changes.
Measured with a stub request across five chat shapes, a note chat, a plain chat, an internal chat that is not a note, a chat id with no row behind it, and an unsaved chat id: the returned tool set is identical in every case and the query count drops from six to five. The note tools are still enabled for a note chat with the notes feature switched off, which is the only thing that predicate decides.
The legacy function-calling path acted on the client-supplied `features` dict after checking only the per-user permission, so a user who still held `features.web_search` or `features.image_generation` could keep triggering web searches and image generation after an administrator had switched those off instance-wide. The native function-calling path already gates the equivalent builtin tools on `web.search.enable` and `image_generation.enable` in `get_builtin_tools`, so the two paths disagreed and the admin-level switch did not actually stop the outbound provider calls it was turned off to stop.
Gate the legacy web search handler on `web.search.enable` at its call site, and gate `chat_image_generation_handler` on the two image switches internally. The image handler needs the check inside it because `image_generation.enable` and `images.edit.enable` are independent: editing stays available when generation is disabled, matching the `/images/generations` and `/images/edit` routes and the native `generate_image`/`edit_image` tools. The handler calls `image_generations`/`image_edits` directly and so bypasses the route guards, which is why the check has to live at the caller.
The "Creating image" status event moves below the new guard so a disabled configuration returns without leaving an unresolved progress indicator in the chat.
* fix: enforce the image generation flag on the legacy chat feature path
* fix: refresh the config store when image generation is disabled in admin settings
* fix: hide active feature pills when the feature is no longer available
Revoking a user's `features.memories` permission removed their access to the memories API and to the native function-calling memory tools, but their stored memories were still injected into the system context on the legacy function-calling path.
The branch in `process_chat_payload` only checked the client-supplied `features['memory']` flag plus the global `memories.system_context.enable` switch, with no user-permission check. `add_memory_context` did not compensate: it only checks `model_allows_memory`, which is a model capability rather than a permission, and the one call inside it that does check the permission (`query_memory`) has its 403 swallowed by a `try/except`, so `Memories.get_memories_by_user_id` and the neighbourhood scan still fed the system prompt.
Gate the branch with the same permission check the native path already performs in `get_builtin_tools`, matching the neighbouring `web_search` and `image_generation` branches.
Only the caller's own memories were injected into the caller's own context, so there was no cross-user exposure. The practical effect was that the permission toggle did not do what its name implies: an admin who revoked it still got memory content injected for that user.
The Responses API handler had a branch for response.output_item.done whose own comment said it was handled specifically below, but it never ran. The generic branch matching any response.*.done event came first in the chain and matched this event too, so it fell through and returned the accumulated output unchanged, leaving the dedicated branch below unreachable since the feature was added.
Moving the dedicated branch above the generic one makes the event apply. On a compliant stream this changes nothing, since response.completed replaces the whole output with the same data straight afterwards. It matters when a provider is less tidy: one that never sends response.content_part.added leaves the assistant's own reply unextractable from the next turn's context, and one that omits response.content_part.done drops the annotations that only arrive with the finished item. Both are repaired by honouring the event.
Worth knowing: the item replaces whatever the deltas accumulated, with no guard against a provider sending back less than it streamed. A reasoning item arriving without its content would therefore lose the reasoning body, which is the same shape of provider brokenness that #27800 already needed a guard for.
The generic response.*.delta branch could leave the streaming handler in two states that crash the caller. It bound its result only inside the guard that checks the target item exists, but returned that result outside the guard, so a delta arriving before its output item, or carrying an index past the end, raised UnboundLocalError. Separately, an event name with only two dot-separated parts failed the length check and fell off the end of the branch, so the function returned None and both call sites raised TypeError unpacking it.
Where the response is streamed to a browser both crashes were swallowed at debug level and cost a chunk. On the direct API path there is no handler between here and the server, so the caller kept its 200 while the body was cut short with no [DONE], and the outlet filters never ran.
The return now sits inside the guard with a branch-level fallback that hands back the accumulated output untouched, which is what the sibling done branch and every other skip path in this function already do. Deltas whose item exists behave exactly as before.
Dropping an orphan delta is deliberate rather than synthesizing the missing item: response.output_item.added appends without regard to output_index, so a placeholder would be duplicated when the real item arrives, and a fabricated function_call would have no name or call id.
The `response.completed` handler replaced the accumulated output with the terminal event's `output` whenever that key was present, guarded only by `is not None`. An empty array satisfies that guard, so a provider that finishes the stream with `"output": []` wiped everything collected from `response.output_item.added`, `response.output_text.delta` and `response.output_item.done`.
The assistant message was then persisted with `output: []` and empty content, which shows up as a reply that renders correctly while streaming and disappears the moment the stream ends.
Fall back to the accumulated output when the terminal array is empty. A spec-compliant `response.completed` still wins, since a populated array is truthy, and when nothing was streamed the accumulated output is empty too, so the fallback cannot invent content.
Fixes#27789
GLOBAL_LOG_LEVEL defaults to INFO, so every log.debug(...) in the backend is discarded, but the message is built first: 187 call sites interpolate their payload into an f-string before the logging call runs, so the work happens on every request and the result is thrown away. The worst one sits in process_chat_payload and stringifies the whole request body, full conversation history included, once per chat completion.
That one line with DEBUG disabled, CPython 3.12:
| conversation | payload | before | after |
| ------------ | ------- | -------- | ------- |
| 4 messages | 1.2 kB | 3.4 us | 0.07 us |
| 20 messages | 17 kB | 24.8 us | 0.07 us |
| 60 messages | 123 kB | 216.6 us | 0.07 us |
The lazy form log.debug('form_data: %s', form_data) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. With DEBUG enabled the emitted lines are byte-identical, f'{x=}' sites included: those map to %r. MistralLoader._debug_log callers get the same treatment, since that wrapper already forwards *args.
`request.app.state.MODELS` is a `RedisDict` when Redis is configured. Unpacking it with `{**pool}` makes Python call `keys()` and then `__getitem__` once per key, which is one HKEYS plus one HGET per model, issued sequentially through a synchronous client. At 200 models that is 201 blocking Redis round trips per call.
`RedisDict.items()` is a single HGETALL, so `dict(pool.items())` fetches the same data in one round trip. `utils/chat.py:184` already does exactly this and carries a comment explaining why; these ten call sites were missed.
They are on the direct-connection branch of the task endpoints (title, tags, follow-up, autocomplete, query generation and the rest), of `chat_completed`, and of context compaction, so they run for background tasks fired on ordinary chat turns.
Behaviour is unchanged. The merged mapping is identical, the explicitly added direct model still overrides any pool entry with the same id, and when Redis is not configured the pool is a plain dict where `dict(d.items())` and `{**d}` are equivalent.
It also closes a race. `RedisDict.set` writes with HSET and then HDELs the stale keys, so a key returned by HKEYS could be deleted before its HGET arrived, raising `KeyError` out of the dict literal and failing the request mid model refresh. The old path could likewise observe a mix of pre- and post-refresh entries. HGETALL is atomic, so the caller now always sees one coherent snapshot.
Swap the JSON encoder/decoder used across the backend from stdlib json to
orjson when ENABLE_ORJSON is set — HTTP request bodies, JSONResponse
bodies, upstream provider responses, SSE chunks, and socket.io/Redis
payloads.
The flag defaults to off, in which case the app uses stdlib json and
engineio's codec verbatim, so default behaviour is unchanged.
- json_codec exports JSONCodec (stdlib json or the orjson codec) and
SOCKETIO_JSON (engineio's codec or the orjson codec); call sites import
JSONCodec and stay implementation-agnostic
- apply_orjson_http_json() is a no-op when the flag is off, leaving
starlette's Request.json / JSONResponse.render untouched
- the orjson codec falls back to the stdlib for inputs orjson rejects
(non-str dict keys, ints beyond 64 bits, NaN literals)
- orjson is imported only when the flag is on
- FastAPI(default_response_class=...) is deliberately not used: an
explicit default disables the Pydantic direct-to-bytes fast path for
response_model routes
Timothy Jaeryang Baek