From 0116c6e1b977b89889a05860961f46efbe6514c2 Mon Sep 17 00:00:00 2001
From: Classic298 <27028174+Classic298@users.noreply.github.com>
Date: Mon, 27 Jul 2026 00:34:02 +0200
Subject: [PATCH] perf: stop running chardet over entire uploaded files
 (#27445)

`_detect_text_encoding()` hands the complete file to `chardet.detect()`. chardet is pure Python and costs roughly 1.3 seconds per megabyte, so uploading a large non-UTF-8 text file stalls for seconds inside encoding detection alone. A 4 MiB Shift-JIS file spends 6.4 seconds there. The UTF-8 fast path above it means only non-UTF-8 files reach this, which in practice are exactly the CJK documents the surrounding code was written to handle, so the slow case and the case that matters are the same case.

Detection does not need the whole file. It needs the bytes that are actually not UTF-8, and `UnicodeDecodeError.start` from the fast-path decode already says where those begin, so this samples a 256 KiB window around that offset.

Two things make that safe rather than merely fast.

Centring the window on the first non-UTF-8 byte instead of the file head is what keeps the common case correct. A plain head sample makes chardet report ascii for a file that is ASCII for its first few hundred KiB and only turns CJK later, and the method then falls through to latin-1 instead of the right codec.

The window still cannot help when a stray byte, a pasted Windows-1252 artifact for example, sits hundreds of KiB ahead of the real payload: the sample is then almost pure ASCII and carries no signal. So when the sample holds almost no non-ASCII bytes and is a strict subset of the file, detection falls back to the whole buffer. That case pays the old cost, which is the right trade, because it is precisely the case where sampling would otherwise be wrong. Without this guard a Cyrillic document with a stray leading byte was detected as ISO-8859-1 rather than windows-1251, which is silent mojibake.

Measured, with the encoding returned identical in every case:

| file | before | after |
|---|---|---|
| shift_jis 4 MiB | 6402ms | 755ms |
| gb18030 4 MiB | 3199ms | 449ms |
| big5 4 MiB | 2926ms | 413ms |
| euc-jp 4 MiB | 2456ms | 413ms |
| euc-kr 4 MiB | 2382ms | 468ms |
| latin-1 4 MiB | 1902ms | 394ms |
| gb18030 1 MiB | 807ms | 376ms |
| ascii head then gb18030 tail | 533ms | 294ms |
| stray byte then cp1251 payload | 496ms | 1051ms |
| any UTF-8 file | 8ms | 0ms |

29 cases, all returning an identical encoding before and after: six encodings at 100 KiB, 1 MiB and 4 MiB, three layouts where the non-UTF-8 bytes only begin beyond the window, four where a stray byte is separated from the payload, plus plain UTF-8, UTF-8 CJK and an empty file. The stray-byte rows are slower than before because they scan twice, once over the window and once over the whole buffer. They are the pathological shape, and correctness wins there.

The residual time is now the decode-and-validate loop below, which walks the file once per candidate codec, and `_has_cjk_characters`, which is a per-character Python loop over the decoded text. Both are the same "full scan for a detection decision" pattern and could take a bounded prefix too. That is left alone here.
---
 backend/open_webui/retrieval/loaders/main.py | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/backend/open_webui/retrieval/loaders/main.py b/backend/open_webui/retrieval/loaders/main.py
index f86f6fc2dc..218369fd93 100644
--- a/backend/open_webui/retrieval/loaders/main.py
+++ b/backend/open_webui/retrieval/loaders/main.py
@@ -303,13 +303,20 @@ class Loader:
         try:
             raw.decode('utf-8')
             return 'utf-8'
-        except UnicodeDecodeError:
-            pass
+        except UnicodeDecodeError as e:
+            first_non_utf8 = e.start
 
         # Use chardet as a hint, not as ground truth
         import chardet
 
-        detected = chardet.detect(raw)
+        # chardet is pure Python (~1.3s/MB), so sample around the first bad byte
+        window = 256 * 1024
+        sample_start = max(0, first_non_utf8 - window // 2)
+        sample = raw[sample_start : sample_start + window]
+        detected = chardet.detect(sample)
+        # A stray byte can sit far from the real payload, leaving the sample with nothing to read
+        if len(sample.translate(None, delete=bytes(range(128)))) < 64 and len(sample) < len(raw):
+            detected = chardet.detect(raw)
         detected_enc = (detected.get('encoding') or '').lower().replace('-', '').replace('_', '')
 
         # Map chardet's detected encoding to the correct superset codec.