ewilliams28/open-webui
1
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-08-13 01:02:25 -06:00
Code Issues Packages Projects Releases Wiki Activity

perf: parse scraped web pages with lxml, not html.parser (#27439)

Browse Source 
Every page pulled in by web search and web RAG is parsed with BeautifulSoup's `html.parser`, a pure-Python parser. It is the slowest option bs4 offers, and it is being handed 300 KiB to 1.5 MiB documents, several per query. `SafeWebBaseLoader` inherits `default_parser = "html.parser"` from langchain's `WebBaseLoader` and never overrides it, so this is an upstream default carried by accident, not a decision anyone made for Open WebUI.

`default_parser` is the single chokepoint for both the sync `_scrape()` path and the async `ascrape_all()` path, so one `setdefault` covers everything and an explicit caller override still wins.

lxml is already in the tree as a transitive hard dependency of ddgs, python-pptx and unstructured, so nothing new enters the image and `uv.lock` already resolves it at 6.1.1. The pin makes it explicit and closes a latent failure: bs4's `"xml"` feature, already used for `.xml` URLs in `_unpack_fetch_results()`, requires lxml and would raise `FeatureNotFound` the day that transitive dependency moves.

## Benchmarks

37 real pages, 13.8 MiB of HTML, median of 5 runs each. The timed operation is `BeautifulSoup(html, parser)` plus `get_text()` plus `extract_metadata()`, which is exactly what the loader does per page. bs4 4.14.3, lxml 6.1.1, CPython 3.12.

| | html.parser | lxml | |
|---|---|---|---|
| 37 pages, 13.8 MiB total | 1611.0ms | 1151.3ms | 1.4x faster, 460ms saved |

Largest pages:

| page | size | html.parser | lxml | speedup |
|---|---|---|---|---|
| pypi.org/project/aiohttp/ | 1259 KiB | 243.75ms | 180.51ms | 1.4x |
| gnu.org/software/bash/manual/bash.html | 1017 KiB | 257.97ms | 178.99ms | 1.4x |
| rfc-editor.org/rfc/rfc9110.html | 1157 KiB | 205.94ms | 154.87ms | 1.3x |
| docs.aiohttp.org/en/stable/client_reference.html | 403 KiB | 108.62ms | 84.93ms | 1.3x |
| ollama.com/library | 779 KiB | 117.55ms | 73.64ms | 1.6x |
| theregister.com | 1052 KiB | 88.32ms | 60.12ms | 1.5x |
| kubernetes.io/docs/concepts/services-networking/service/ | 563 KiB | 72.18ms | 43.43ms | 1.7x |
| docs.python.org/3/library/socket.html | 301 KiB | 71.88ms | 49.04ms | 1.5x |

Ranges from 1.1x to 1.7x, and the win grows with page size. A ten result web search sheds roughly 125ms of parsing. Because the async path builds its soups inline in `_unpack_fetch_results()`, that is 125ms the event loop spends parsing HTML instead of serving other users' streams. Pages under about 10 KiB are marginally slower under lxml due to fixed setup cost, which is worth nothing either way.

## Output verification

The risk in changing parser is silently different extracted text, so that was measured rather than assumed. Across all 37 real pages:

- **Zero characters of text were lost.** Every diff opcode against html.parser output was an insertion. Not one page dropped content under lxml.
- 659 characters were added, all on one page (docs.docker.com), where an inline Alpine.js `@click` handler containing a regex confuses libxml2's attribute handling and leaks a 73-character JS fragment into the text nine times. That is 659 characters of script noise in 27,206 characters of extracted text, with no content affected.
- Metadata (`title`, `description`, `language`) was identical on 35 of 37 pages. The two exceptions are 141-byte Wikipedia bot-block stubs with no `<html>` element, where lxml's fragment auto-wrapping adds `language: "No language found."`. Both parsers extract the same text from them.

Large documents were checked separately because libxml2 carries internal size caps. A 12 MiB single text node, 12 MiB spread across 400k nodes, a 3 MiB attribute value and 50k sibling elements with a trailing marker all produced byte-identical text under both parsers, with no truncation.

Malformed markup was checked too. lxml and html.parser diverge on unterminated comments, bare CDATA and duplicated `<html>` elements, all cases where both parsers are guessing and neither is correct. None of those shapes appeared in the 37 page corpus.

`backend/open_webui/env.py:184` also uses `html.parser`, on the local CHANGELOG at import time. That is trivial input on a startup path and is deliberately left alone.
This commit is contained in:
Classic298
2026-07-27 00:19:09 +02:00
committed by GitHub
parent 9c352e37b8
commit fb1f1a3c92
4 changed files with 6 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backend/open_webui/retrieval/web/utils.py
+2
View File
@@ -729,6 +729,8 @@ class SafeWebBaseLoader(WebBaseLoader):
trust_env (bool, optional): set to True if using proxy to make web requests, for example
using http(s)_proxy environment variables. Defaults to False.
"""
# lxml parses scraped pages far faster than the html.parser default
kwargs.setdefault('default_parser', 'lxml')
super().__init__(*args, **kwargs)
self.trust_env = trust_env
backend/requirements.txt
+1
View File
@@ -77,6 +77,7 @@ unstructured==0.22.31
nltk==3.9.4
Markdown==3.10.2
beautifulsoup4==4.14.3
lxml==6.1.1
pypandoc==1.17
pandas==3.0.3
openpyxl==3.1.5
pyproject.toml
+1
View File
@@ -84,6 +84,7 @@ dependencies = [
"nltk==3.9.4",
"Markdown==3.10.2",
"beautifulsoup4==4.14.3",
"lxml==6.1.1",
"pypandoc==1.17",
"pandas==3.0.3",
"openpyxl==3.1.5",
uv.lock
Generated
+2
View File
@@ -3036,6 +3036,7 @@ dependencies = [
{ name = "langchain-text-splitters" },
{ name = "ldap3" },
{ name = "loguru" },
{ name = "lxml" },
{ name = "markdown" },
{ name = "mcp" },
{ name = "msoffcrypto-tool" },
@@ -3175,6 +3176,7 @@ requires-dist = [
{ name = "langchain-text-splitters", specifier = "==1.1.2" },
{ name = "ldap3", specifier = "==2.9.1" },
{ name = "loguru", specifier = "==0.7.3" },
{ name = "lxml", specifier = "==6.1.1" },
{ name = "mariadb", marker = "extra == 'mariadb'", specifier = "==1.1.14" },
{ name = "markdown", specifier = "==3.10.2" },
{ name = "mcp", specifier = "==1.27.2" },