Docker compose list-form environment syntax passes quotes through verbatim, so WEB_FETCH_FILTER_LIST="" reaches the backend as two literal quote characters rather than an empty string. Config parsing turned that into the filter entry '""', which has no "!" prefix and therefore landed in the allow list. A non-empty allow list requires every host to match one of its entries, and a quotes-only pattern can never match a hostname, so every fetch_url and web loader request was rejected with "URL blocked by filter list" and surfaced to the user as "The URL you provided is invalid".
get_allow_block_lists now strips surrounding quote characters from each entry and drops entries that are empty after normalisation. Quoted but otherwise valid entries such as "example.com" or !"example.com" now behave as their unquoted forms, and garbage entries no longer convert the default blocklist into a match-nothing allowlist that blocks everything.
Fixes#26908
is_string_allowed does endswith() matching and was called with the full URL
(retrieval/web/utils.py) against WEB_FETCH_FILTER_LIST, so a blocklisted host with any
path (https://blocked.example/x) ended with /x, not the host, and slipped through; the
allowlist direction false-rejected legitimate URLs and admitted attacker URLs ending in
an allowed string. The same endswith caused label confusion at the hostname call site
(retrieval/web/main.py): corp.com matched evilcorp.com, 10.0.0.1 matched 110.0.0.1.
Add is_host_allowed(host, ...) matching on DNS label boundaries (host == pattern or
host.endswith('.' + pattern)), called with the parsed hostname at both web-fetch call
sites. is_string_allowed is left unchanged for the unrelated function-name filters
(utils/middleware.py, utils/tools.py).
The separate is_global guard (validate_url / _ssrf_safe_new_conn, active when
ENABLE_RAG_LOCAL_WEB_FETCH is off) already blocks RFC1918/loopback/link-local, so this
restores the admin's intended blocking of specific public hosts.
Co-authored-by: addcontent <59762500+addcontent@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After migration to async db operations, the throttle decorator also
needs to support async. Since the decorator is only used for async funcs
now, we can just switch it to async instead of supporting sync and async
at the same time.
Signed-off-by: Adam Tao <tcx4c70@gmail.com>
Timothy Jaeryang Baek