mirror of
https://github.com/open-webui/open-webui.git
synced 2026-08-27 07:54:55 -06:00
Classic298
e3e4bd87df
* fix: apply the SSRF checks to redirect targets on every web fetch path Two guards protect server-side fetches: a private-IP check and the operator's `WEB_FETCH_FILTER_LIST`. Neither reached a redirect hop on the aiohttp paths, and the filter list never reached one on the requests paths either. aiohttp answers IP-literal hosts itself without consulting a resolver, so `_SSRFSafeResolver` was never invoked for a hop such as `http://169.254.169.254/` and the private-IP check simply did not run. With redirect following enabled, a submitted public URL that redirects to an IP literal reached loopback, RFC1918 and cloud-metadata addresses, and the response body was returned to the caller. The filter list was consulted only in `validate_url`, on the originally submitted URL, so a redirect to a filter-listed host was fetched without it ever being applied. `_SSRFSafeResolver` is replaced by `_SSRFSafeConnector`, which hooks `_resolve_host` so the IP check also covers the IP-literal shortcut and both DNS cache paths. The filter list moves to a per-request hook on each transport, `connect()` for aiohttp and `send()` for the requests adapter, because those see the request destination: at the connection layer a proxied request presents the proxy's host, and a pooled connection skips resolution entirely. This covers every hop, including redirects, on all five aiohttp call sites and both requests sessions. The Playwright loader already validated each hop and is unchanged. Both gaps required `AIOHTTP_CLIENT_ALLOW_REDIRECTS=true`, which is not the default. Two behaviour changes for operators. The filter list now applies to redirect targets rather than only to submitted URLs. Under a forward proxy it is evaluated against the request destination instead of the proxy, which also fixes allowlist entries rejecting every fetch in proxied deployments. * refac: match the web fetch filter list against resolved addresses The filter list is now evaluated against the hostname together with the addresses it resolves to, at URL validation and on each connection, on both transports. An IPv6 address is also matched by the IPv4 address it carries. * refac: screen outbound fetch addresses against reserved ranges ipaddress misses `ipaddress.is_global` was the only test behind the web-fetch address check, and it answers a narrower question than "may we fetch this". Several special-purpose ranges are globally routable by registry while nothing on them is a legitimate destination, so they passed. Classification now screens those ranges on top of `is_global`, and applies the same screen to the IPv4 address embedded in an IPv6 transition encoding rather than only to the literal. All three checkpoints share the predicate, so they all inherit it. The range list is the exact complement of what CPython's `ipaddress` already models, checked entry by entry against both IANA special-purpose registries. Prefixes IANA marks globally reachable are deliberately left out, so no real destination changes behaviour. Verified against 31 addresses covering every entry, their transition-encoded forms, and public controls in both families: 31/31 expected after, 18/31 before. * refac: match web fetch filter entries that name an address or a range A filter entry that parses as an address or a CIDR range is matched by containment rather than by DNS label suffix, so a range covers the addresses inside it and an address matches however it is spelled. A range entry previously matched nothing at all, silently. The built-in list gains the special-purpose networks that ipaddress.is_global reports as reachable while nothing on them is a legitimate destination, so taking an address out of reach is a WEB_FETCH_FILTER_LIST change rather than a release. Those entries hold whether or not local web fetch is enabled; the private-address rule still follows the toggle.
51 lines
1.4 KiB
Python
51 lines
1.4 KiB
Python
from __future__ import annotations
|
|
|
|
from urllib.parse import urlparse
|
|
|
|
import validators
|
|
from open_webui.retrieval.web.utils import resolve_hostname
|
|
from open_webui.utils.misc import as_network, get_allow_block_lists, is_host_allowed
|
|
from pydantic import BaseModel
|
|
|
|
|
|
def get_filtered_results(results, filter_list):
|
|
if not filter_list:
|
|
return results
|
|
|
|
allow_list, block_list = get_allow_block_lists(filter_list)
|
|
# Only worth a lookup when an entry names an address, since a hostname entry matches by name.
|
|
resolve_ips = any(as_network(entry) is not None for entry in allow_list + block_list)
|
|
|
|
filtered_results = []
|
|
|
|
for result in results:
|
|
url = result.get('url') or result.get('link', '') or result.get('href', '')
|
|
if not validators.url(url):
|
|
continue
|
|
|
|
domain = urlparse(url).hostname
|
|
if not domain:
|
|
continue
|
|
|
|
hostnames = [domain]
|
|
|
|
if resolve_ips:
|
|
try:
|
|
ipv4_addresses, ipv6_addresses = resolve_hostname(domain)
|
|
hostnames.extend(ipv4_addresses)
|
|
hostnames.extend(ipv6_addresses)
|
|
except Exception:
|
|
pass
|
|
|
|
if is_host_allowed(hostnames, filter_list):
|
|
filtered_results.append(result)
|
|
continue
|
|
|
|
return filtered_results
|
|
|
|
|
|
class SearchResult(BaseModel):
|
|
link: str
|
|
title: str | None
|
|
snippet: str | None
|