ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat(web-search): replace Tavily/DuckDuckGo backends with self-hosted SearxNG

Browse Source 
Drop the Tavily and DuckDuckGo (ddgs) web_search backends for a single
self-hosted SearxNG service bundled into the docker-compose stacks.

Core:
- New SearXNGClient + _format_searxng; rewrite resolve_web_search_client to
  (backend, searxng_url, searxng_engines, ...). MCP backend + oauth_user guard
  unchanged. _resolve_search_client follows storage -> toml -> env -> default
  precedence (explicit "" disables, via ConfigStore.stored_keys()).
- Drop the Tavily-era topic=finance (no SearxNG category); topic is now
  general/news.

Settings/config:
- Remove tools.tavily_api_key, get_tavily_key, $TAVILY_API_KEY, [api].tavily_key.
- Add tools.searxng_url (default http://searxng:8080) + tools.searxng_engines,
  with get_searxng_url/get_searxng_engines.

Compose + bundled config:
- Internal-only searxng service (no published API port, :ro config, /healthz
  healthcheck, persistent searxng-cache volume) in both stacks; bundle
  turnstone/deploy/searxng/settings.yml (JSON output on, limiter off).
- Caddy serves the SearxNG web UI on :8444 (dev: localhost-only; prod: opt-in).
- bootstrap extractor + wheel packaging updated.

Deps: drop the ddg extra + ddgs mypy override (regenerates uv.lock, removing the
lxml/h2/brotli transitives).

Docs: tools/docker/architecture/openshell + diagrams + config example + CHANGELOG;
docs/docker.md carries the AGPL-3.0 §13 operator note.

BREAKING: tools.web_search_backend no longer accepts "tavily"/"ddg";
tools.tavily_api_key and the ddg extra are removed. Run the bundled SearxNG (ships
in the compose stacks) or set TURNSTONE_SEARXNG_URL to an external instance.

Closes #545
This commit is contained in:
Patrick Buckley
2026-05-31 18:05:16 -07:00
parent effc60297c
commit 1728a4c0af
32 changed files with 785 additions and 616 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tools.md
+4 -4
View File
@@ -132,7 +132,7 @@ Each item's `execute` callable is invoked:
- `edit_file` -- modifies file content
- `math` -- sandboxed computation (confirmation required despite being sandboxed)
- `web_fetch` -- fetches a URL (SSRF-protected, but makes network requests)
- `web_search` -- web search via Tavily API (makes network requests)
- `web_search` -- web search via self-hosted SearxNG (makes network requests)
- `task` -- spawns an autonomous sub-agent
- `plan` -- spawns a planning sub-agent, plus post-execution review gate
@@ -344,12 +344,12 @@ Search the web using a text query.
|---------------|---------|----------|-------------|
| `query` | string | yes | The search query. |
| `max_results` | integer | no | Max results to return (default 5, max 20). |
| `topic` | string | no | Search topic: `general`, `news`, or `finance` (default `general`). |
| `topic` | string | no | Search topic: `general` or `news` (default `general`). |
- **What it does**: Searches the web and returns ranked results with titles, URLs, and content snippets. Uses provider-native search when available:
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No Tavily key needed.
- **Anthropic**: Replaced at the API boundary with Anthropic's `web_search_20250305` server-side tool. Claude decides when to search; the API executes it and returns results with citations inline. No backend needed.
- **OpenAI search models** (`gpt-5-search-api`): Replaced with `web_search_options` parameter. The model always searches and returns `url_citation` annotations.
- **Local/vLLM models**: Falls back to the Tavily API. Requires `tavily_key` in `config.toml` or `$TAVILY_API_KEY`.
- **Local/vLLM models**: Falls back to a self-hosted [SearxNG](https://searxng.org) instance. Set `searxng_url` in `config.toml` `[tools]` or `$TURNSTONE_SEARXNG_URL` (the docker-compose stack bundles a `searxng` service and points at it by default). Operators with a custom MCP search server can instead set `web_search_backend = "mcp:server:tool"`.
- **Auto-approve**: Yes (auto-approved for all tool dispatch paths).
- **Agent availability**: `agent` and `task_agent`.