The previous regular expression used manual surrogate-pair ranges to
match emojis and missed a large category of commonly used symbols:
/[\uD800-\uDBFF][\uDC00-\uDFFF]|\uD83C[\uDC00-\uDFFF]|\uD83D[\uDC00-\uDE4F]/g
This approach only covers emojis encoded as surrogate pairs (U+1F000 –
U+1F4FF range), but silently skips BMP emojis that use a text-
presentation code point followed by the variation selector U+FE0F,
such as ❤️ (U+2764 U+FE0F), ☀️, ✅, ⚡, ⭐, and keycap sequences
like 1️⃣, as well as ZWJ family sequences (👨👩👧👦) and flag sequences.
Replace with the Unicode property escape \p{RGI_Emoji} using the 'v'
(unicodeSets) flag introduced in ES2024. This single pattern covers
every standardised emoji sequence defined by Unicode, including all
the cases above.
Browser support: Chrome 112+, Firefox 116+, Safari 17+, Node.js 20+.
All browsers targeted by open-webui already support this syntax.
Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: joaoback <156559121+joaoback@users.noreply.github.com>
Co-authored-by: yoloni <yoloni@tencent.com>
getTimeRange returns month names that are used as i18n translation keys
(consumed via \.t(chat.time_range) in the sidebar, search modal, etc.).
The keys must be exact English strings like 'January', 'February', etc.
Previously, toLocaleString('default', { month: 'long' }) was used to
generate these keys. The 'default' locale defers to the browser's locale
resolution, which in Firefox with intl.regional_prefs.use_os_locales=true
picks up OS regional settings instead of the browser language. This caused
German month names (e.g. 'Februar', 'Januar') to appear in the sidebar for
users whose OS region is set to Germany, even when both browser and app
language are set to English. Chrome was unaffected because it ignores OS
regional settings for the 'default' locale.
Since i18n has no translation key for 'Februar', the German string passed
through untranslated. Replace toLocaleString with a static MONTH_NAMES
array lookup to make the intent explicit and eliminate any browser/OS
locale dependency.
Replace the recursive spread-based implementation with an iterative
push+reverse approach. The recursive version created a new array at
each level of recursion via spread, resulting in O(d^2) array copies
where d is the conversation depth. The iterative version walks from
the target message to the root, pushes each message, and reverses
once at the end for O(d) total work.
No behavioral change - same input produces the same output array.
removeAllDetails() uses replaceOutsideCode() which splits content on
triple-backtick code blocks before applying the details-removal regex.
When thinking/reasoning content inside a <details> block contained
code blocks (backticks survive html.escape), the <details> opening
and </details> closing tags ended up in different split segments,
making the regex unable to match either. This caused thinking content
to leak through to TTS playback.
Fix: add a direct <details> strip (without code-block splitting) as
the first step of getMessageContentParts(), which is the TTS-specific
entry point. This catches the edge case while keeping removeAllDetails
safe for copy-to-clipboard (where legitimate <details> inside code
blocks should be preserved).
Fixes#22197
codeHighlight.ts had a top-level static import of shiki that pulled
the entire highlighter engine (~5-10MB of JavaScript including all
language grammars) into any page that imported the module - even if
only the lightweight isCodeFile() function was used.
Replace the static shiki import with:
- A static set of ~85 common language IDs for synchronous extension
checks (isCodeFile, extToLang) - no shiki dependency needed
- A dynamic import('shiki') inside highlightCode(), which is already
async so callers are completely unaffected
The static language set covers all commonly-used file extensions.
Obscure extensions not in the set simply won't be detected by
isCodeFile() (the file still opens fine, just won't show the code
file indicator). Highlighting itself still works for all shiki
languages since the full bundle loads on demand.
* perf: pre-compile KaTeX Unicode regex at module load time
The katexStart() function was creating a new RegExp with Unicode
property escapes (\p{Script=Han}, \p{Script=Hiragana}, etc.) on
every invocation. Unicode property escapes are extremely expensive
to compile as the regex engine must build character class tables
covering tens of thousands of code points.
Since marked calls the start() function at every character position
while scanning source text, this meant hundreds of regex compilations
per marked.lexer() call, and lexer runs ~60 times/sec during streaming.
Profiling showed KaTeX regex consuming 87% (320ms/365ms) of total
markdown rendering time.
Changes:
- Pre-compile SURROUNDING_CHARS_REGEX once at module load time
- Use .test() instead of .match() to avoid array allocations
- Fix delimiter search to find earliest match, not last match
* perf: replace katexStart with single-pass character scan
The katexStart() function was the dominant cost in marked's lexer,
consuming 55-58% of total markdown rendering time per profiling.
It was called at every character position by marked and each call:
- Looped through 3-5 delimiters, each doing indexOf() on the full
remaining source (3-5 x O(n) string scans per call)
- Ran the complex ruleReg regex with Unicode lookaheads for validation
- On failed validation, created substrings and looped again
Replace with a single linear character scan using charCodeAt that:
- Checks only for $ (charCode 36) or backslash (charCode 92)
- Filters backslash hits by next character to avoid false positives
- Preserves the surrounding-character validation
- Returns immediately on first valid candidate
- Lets the tokenizer handle full validation (it already does this)
This reduces start() from O(n * delimiters * retries) to O(n) with
a very small constant factor per call.
* Update katex-extension.ts
Timothy Jaeryang Baek