Commit Graph

136 Commits

Author SHA1 Message Date
Classic298 e7ba8978c6 fix: reject parser-confusing chars in validate_url to close SSRF bypass (#24534)
urllib.parse.urlparse and requests/aiohttp disagree on how to split URLs
containing backslash, tab, CR, or LF in or around the netloc. urlparse
treats backslash as part of userinfo and uses what follows '@' as the
host; requests treats backslash as the start of the path and connects
to whatever precedes it. The same URL therefore passes the private-IP
filter (urlparse sees a public host) but reaches an internal target
(requests connects to e.g. 127.0.0.1). End result is an SSRF that the
existing IP block list cannot catch because it's evaluating the wrong
host.

PoC: http://127.0.0.1:6666\@1.1.1.1 — urlparse hostname is 1.1.1.1
(global, passes), requests reaches 127.0.0.1 (loopback).

Reject up front any URL containing one of the four documented parser-
confusing characters before either parser gets a chance to interpret
it. None of these characters is valid in an unencoded URL (\ should
always be %5C, whitespace should be %09 / %0A / %0D), so this is a
pure defensive rejection with no legitimate-input false positives.

Reported by Fushuling and RacerZ-fighting in GHSA-8w7q-q5jp-jvgx.

Co-authored-by: Fushuling <Fushuling@users.noreply.github.com>
Co-authored-by: RacerZ-fighting <RacerZ-fighting@users.noreply.github.com>
2026-05-11 00:57:48 +09:00
Timothy Jaeryang Baek df42d96c95 refac 2026-05-09 21:05:49 +09:00
Classic298 8854541508 fix: prevent redirect-based SSRF in web-fetch and image-load call sites (#24491)
validate_url() in retrieval/web/utils.py only validates the initial URL.
The HTTP clients used downstream (sync requests, sync requests via the
parent WebBaseLoader._scrape, aiohttp via load_url_image) followed 3xx
redirects by default and did not re-validate the redirect target against
the private-IP / metadata-IP block list. An authenticated user could
submit a public URL that 302-redirected to an internal address (RFC1918,
127.0.0.1, 169.254.169.254, etc.) and the redirected response was returned
to them, enabling SSRF reads of internal services and cloud metadata.

Three call sites needed allow_redirects=False to match the policy already
enforced on the async _fetch() path:

- SafeWebBaseLoader: override requests_kwargs in __init__ so that the
  inherited synchronous _scrape() path passes allow_redirects=False to
  self.session.get() (the parent WebBaseLoader uses requests' default
  allow_redirects=True).
- get_content_from_url (retrieval/utils.py): pass allow_redirects=False
  on the streamed requests.get(...) call.
- load_url_image (routers/images.py, image-edits endpoint): pass
  allow_redirects=False on the aiohttp session.get(...) call.

Reports consolidated under GHSA-rh5x-h6pp-cjj6:
- GHSA-rh5x-h6pp-cjj6 (tenbbughunters / Tenable) - sync _scrape
- GHSA-5vxg-6gmv-m2qr (YLChen-007) - load_url_image
- GHSA-hf76-c83f-63w2 (tempcollab) - aiohttp _fetch (already fixed)
- GHSA-h55f-h5fh-mvm4 (sneaXOR) - get_content_from_url
2026-05-09 21:01:45 +09:00
Timothy Jaeryang Baek 7bcc0e2e5c chore: format 2026-05-09 15:25:27 +09:00
Timothy Jaeryang Baek bb0e6cb108 refac 2026-05-09 06:41:42 +09:00
Timothy Jaeryang Baek 6700f7bb72 feat: brave search llm context 2026-05-09 06:34:25 +09:00
Timothy Jaeryang Baek 9386fc83a3 refac 2026-05-09 04:49:18 +09:00
Timothy Jaeryang Baek 6dff85b9d2 refac 2026-05-09 02:08:08 +09:00
Timothy Jaeryang Baek 9adc0c442a refac 2026-05-09 02:02:02 +09:00
RomualdYT e0d6074cd2 refactor(firecrawl): use v2 API directly (#23934)
Co-authored-by: Tim Baek <tim@openwebui.com>
2026-04-24 18:32:08 +09:00
Timothy Jaeryang Baek 0e311a95a7 refac 2026-04-24 15:16:37 +09:00
Timothy Jaeryang Baek fd25152076 refac 2026-04-20 08:34:15 +09:00
Timothy Jaeryang Baek 9c64d84ad9 refac 2026-04-13 15:03:22 -05:00
Classic298 0753409e7b fix: use ipaddress stdlib for IPv6 SSRF protection (#23453)
The validators.ipv6(ip, private=True) call always returns a falsy ValidationError because validators==0.35.0 does not support the private kwarg for IPv6. This means any hostname resolving to a private IPv6 address (::1, fd00::*, ::ffff:169.254.169.254) bypasses SSRF protection entirely, circumventing the fix for CVE-2025-65958.

Replace both the IPv4 and IPv6 validators-based private checks with Python's stdlib ipaddress module using an allowlist approach (not addr.is_global). This blocks all non-globally-routable addresses — private, loopback, link-local, reserved, multicast, and unspecified — for both IPv4 and IPv6, including IPv4-mapped IPv6 addresses.
2026-04-12 16:34:13 -05:00
biebiep f593f92f18 FIX: serper.dev API - Change snippet key from 'description' to 'snippet' (#22869)
This has apparently been broken since forever and native tool calling made it a lot more apparent.
2026-03-20 18:54:16 -05:00
Timothy Jaeryang Baek de3317e26b refac 2026-03-17 17:58:01 -05:00
lazariv 5759917f54 feat: Adding You.com as a web search provider (#21599)
* Add ydc.py provider implementation

* Add PersistentConfig entry for you.com

* Add Youcom search function import

* Update you.com configuration

* Add you.com as a web search engine option in frontend

* Add YOUCOM_API_KEY to main.py
2026-02-21 14:51:56 -06:00
Timothy Jaeryang Baek f376d4f378 chore: format 2026-02-11 16:24:11 -06:00
Tim Baek a214ec40ea fix 2026-02-06 03:34:21 +04:00
Tim Baek 2c37daef86 refac 2026-02-06 03:23:37 +04:00
Danil c5c4aef7b1 Yandex web search (#20922)
Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: joaoback <156559121+joaoback@users.noreply.github.com>
2026-01-26 07:31:44 -05:00
Classic298 614cb56420 feat: Add configurable DDGS backend selection with UI support (#20366)
* init

* Update WebSearch.svelte

* reorder
2026-01-05 03:05:56 +04:00
Timothy Jaeryang Baek f7f8a263b9 feat: JINA_API_BASE_URL 2026-01-01 02:17:47 +04:00
Timothy Jaeryang Baek 89ad1c68d1 enh: FIRECRAWL_TIMEOUT 2026-01-01 02:07:22 +04:00
Classic298 464846d4a3 fix: retry Brave Search on HTTP 429 rate limit with 1s delay (#20255)
* Update brave.py

* Update brave.py
2025-12-30 17:37:44 +04:00
Timothy Jaeryang Baek d4bd938a77 refac: ollama web 2025-12-20 17:54:13 +04:00
Timothy Jaeryang Baek b5bd8704fe fix: bocha filter list
Co-Authored-By: zengxy <11961641+zengxy@users.noreply.github.com>
2025-12-20 17:24:24 +04:00
Classic298 823b9a6dd9 chore/perf: Remove old SRC level log env vars with no impact (#20045)
* Update openai.py

* Update env.py

* Merge pull request open-webui#19030 from open-webui/dev (#119)

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-20 08:16:14 -05:00
Boris Bocquet bc681f8258 feat : new environment variable SEARXNG_LANGUAGE , in the persistent config, that you can also edit in Admin > Web Search pannel in case you choose Searxng. This is used in the request to searxng as the "search language" (arguement "language"). Before this feature, it was set to en-US only. Now default is "all". (#19909) 2025-12-14 12:38:47 -05:00
Timothy Jaeryang Baek b02397e460 feat: WEB_LOADER_TIMEOUT 2025-12-08 11:49:27 -05:00
Poccia 6e531679f4 fix/adjust web search to properly block domains (#19670)
Co-authored-by: Tim Baek <tim@openwebui.com>
2025-12-02 04:17:32 -05:00
Timothy Jaeryang Baek 743199f2d0 feat/enh: tool server function name filter list 2025-11-25 02:31:34 -05:00
Timothy Jaeryang Baek 2af4c4b3c7 refac 2025-11-18 04:42:09 -05:00
Timothy Jaeryang Baek 02238d3113 feat/security: Add SSRF protection with configurable blocklist
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2025-11-18 04:40:55 -05:00
Timothy Jaeryang Baek ee10f372a0 refac/enh: web search domain allow/block filter 2025-11-16 13:52:09 -05:00
Sang Lê 64747f7f79 Add Azure Search (#19104)
Co-authored-by: Tim Baek <tim@openwebui.com>
2025-11-13 19:12:34 -05:00
Classic298 ad17d35ac4 feat: Add custom API endpoint and user info headers for Perplexity Search (#31) (#19147)
Co-authored-by: Claude <noreply@anthropic.com>
2025-11-12 22:53:54 -05:00
Timothy Jaeryang Baek 25c7f101f2 enh: optionally add user headers external websearch
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2025-11-09 16:09:29 -05:00
Timothy Jaeryang Baek 7faf19dad9 refac 2025-11-06 15:21:06 -05:00
Timothy Jaeryang Baek c38f878e1e fix: firecrawl import 2025-11-06 15:19:08 -05:00
Timothy Jaeryang Baek a4fd26b478 enh/fix: google pse referer header 2025-11-04 13:50:07 -05:00
Omar Aburub 3bcf9a442a fix: make SSL verification async 2025-10-29 16:14:53 +03:00
wei840222 7a3f4d85f6 refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader 2025-10-26 15:05:35 +08:00
Timothy Jaeryang Baek e000494e48 refac 2025-10-07 11:53:30 -05:00
Timothy Jaeryang Baek 7f411dd5cc feat/enh: perplexity search support 2025-09-25 14:02:46 -05:00
Timothy Jaeryang Baek fe65fe0b97 refac: ollama cloud web search count support 2025-09-24 15:58:56 -05:00
Timothy Jaeryang Baek e06489d92b enh: search_ollama_cloud 2025-09-24 15:19:05 -05:00
Timothy Jaeryang Baek 0214c1e66c refac 2025-09-09 16:48:59 +04:00
_00_ 093af754e7 FIX: Playwright Timeout (ms) interpreted as seconds
Fix for Playwright Timeout (ms) interpreted as seconds.

To address https://github.com/open-webui/open-webui/issues/16801

In Frontend Playwright Timeout is setted as (ms), but in backend is interpreted as (s) doing a time conversion for playwright_timeout var (that have to be in ms).

& as  _Originally posted by @rawbby in [#16801](https://github.com/open-webui/open-webui/issues/16801#issuecomment-3216782565)_

> I personally think milliseconds are a reasonable choice for the timeout. Maybe the conversion should be fixed, not the label.
> This would further not break existing configurations from users that rely on their current config.
>
2025-08-23 14:15:00 +02:00
Timothy Jaeryang Baek 02479425a5 refac 2025-08-21 12:51:41 +04:00