From 1e0ab847176a681a61acbd0bc9b761a98d4fccee Mon Sep 17 00:00:00 2001 From: Classic298 <27028174+Classic298@users.noreply.github.com> Date: Sun, 26 Jul 2026 23:55:28 +0200 Subject: [PATCH] fix: unshadow the time module so the web loader rate limiter can sleep (#27528) `from datetime import datetime, time, timedelta` shadows the `time` module, so `RateLimitMixin._sync_wait_for_rate_limit` calls `datetime.time.sleep` and raises `AttributeError: type object 'datetime.time' has no attribute 'sleep'` whenever it actually has to wait. Every synchronous loader path that paces requests hits this. `SafeFireCrawlLoader.lazy_load` calls the limiter directly, and Tavily, Microsoft Web IQ and Playwright reach it through `_safe_process_url_sync`. The exception is raised inside their per-URL `try`, so with `continue_on_failure=True` (the default) the URL is logged as a per-URL failure and dropped instead of being scraped. This is live by default: `WEB_LOADER_CONCURRENT_REQUESTS` is passed as `requests_per_second` and defaults to 10, so any URL whose predecessor finished within 100ms takes the sleep branch and is lost. Tavily and Microsoft Web IQ report it as "SSL verification failed", which points at the wrong cause. `_wait_for_rate_limit` uses `asyncio.sleep` and is unaffected, but `SafeMicrosoftWebIQLoader.alazy_load` runs `lazy_load` in a threadpool, so its async entry point is affected too. `datetime.time` is not used anywhere in the file, so importing the `time` module instead is enough. The per-URL `continue` half of #26079 landed in 6f8221df5, which also added the `_sync_wait_for_rate_limit()` call to the Firecrawl loop. This makes that call work rather than throw. Fixes #26079 --- backend/open_webui/retrieval/web/utils.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/backend/open_webui/retrieval/web/utils.py b/backend/open_webui/retrieval/web/utils.py index 4222aa71be..e1f6eaf58d 100644 --- a/backend/open_webui/retrieval/web/utils.py +++ b/backend/open_webui/retrieval/web/utils.py @@ -3,9 +3,10 @@ import ipaddress import logging import socket import ssl +import time import urllib.parse import urllib.request -from datetime import datetime, time, timedelta +from datetime import datetime, timedelta from typing import ( Any, AsyncIterator,