mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
feat(providers): anthropic-compatible lane for local /v1/messages servers
Add provider id "anthropic-compatible": the existing AnthropicProvider pointed at Anthropic-compatible local servers (vLLM /v1/messages), mirroring the openai/openai-compatible split. Registry-only — configured via the admin Models tab or [models.*] toml, not exposed on the bare --provider flag, so the CLI/server prod-URL defaults are unreachable for the lane and real-Anthropic behavior is untouched. Lane behavior (live-verified against vLLM 0.22.1rc1 + DeepSeek-V4-Flash): - Capability defaults replace the Claude static table: token_param max_tokens, thinking_mode none, web_search/tool_search/vision off, reasoning replay on. vLLM rejects Anthropic server-side tool types (tools require input_schema) and ignores the thinking request param, so neither is sent; thinking blocks still stream back and round-trip through the native lane verbatim. - Reasoning toggles via server_compat extra_body chat_template_kwargs (first-class vLLM request field; request-level keys beat server defaults). _build_thinking_and_kwargs forwards non-internal extra_params as SDK extra_body; thinking_budget_tokens stays internal. - No temperature force: thinking_mode none skips the Claude-only temperature=1.0 requirement. Admin UI: provider option + URL placeholder (base_url without /v1 — the SDK appends /v1/messages); the server-compat section shows only the extra-body field for the lane. thinking_mode round-trips through the form dropdown for every provider except anthropic-compatible, where it stays in the raw capabilities JSON — the edit-load lift and save restore use the same predicate so stored overrides are never silently dropped. Docs: architecture.md gains the lane subsection incl. verified quirks (thinking param dropped by vLLM; stop_sequences cut inside thinking and report end_turn; usage has no cache fields; images need a multimodal model; mid-conversation system turns are per-model opt-in). Negative-tested: removing the _INTERNAL_EXTRA_PARAMS exclusion fails test_internal_keys_not_leaked; the live test drives a streamed turn with the chat_template_kwargs toggle and asserts no reasoning deltas.
This commit is contained in:
+56
-1
@@ -702,7 +702,7 @@ agent_model = "claude"
|
||||
|
||||
Each `[models.*]` entry produces a `ModelConfig` with a `provider` field
|
||||
(default: `"openai"`). Supported values: `"openai"`, `"anthropic"`, `"google"`,
|
||||
and `"openai-compatible"`.
|
||||
`"openai-compatible"`, and `"anthropic-compatible"`.
|
||||
|
||||
**Per-model sampling overrides:** Each model can specify `temperature`,
|
||||
`max_tokens`, and `reasoning_effort` to override the global defaults from
|
||||
@@ -765,6 +765,61 @@ model = "qwen-3.5-vl"
|
||||
supports_vision = true
|
||||
```
|
||||
|
||||
**Anthropic-compatible local servers (vLLM `/v1/messages`):** the
|
||||
`"anthropic-compatible"` provider drives local servers that expose
|
||||
Anthropic's Messages API for arbitrary checkpoints — vLLM's
|
||||
`/v1/messages` endpoint, which requires a release with thinking-block
|
||||
support in the Anthropic endpoint (post-2026-02-28; verified against
|
||||
v0.22.1rc1). The lane reuses `AnthropicProvider` in compat mode: same
|
||||
wire translation as the real Anthropic lane, but every model resolves to
|
||||
the `_ANTHROPIC_COMPAT_DEFAULT` capabilities (200K context, 64K output,
|
||||
`token_param=max_tokens`, `thinking_mode=none`, no native
|
||||
web_search/tool_search, no vision) — the static Claude table never
|
||||
applies to local checkpoints. `base_url` is the server root WITHOUT
|
||||
`/v1` (the Anthropic SDK appends `/v1/messages`); a trailing `/v1`
|
||||
pasted out of openai-compatible habit is stripped automatically. Set a
|
||||
placeholder `api_key` (e.g. `"dummy"`) for unauthenticated servers. Tool calling
|
||||
needs the server started with `--enable-auto-tool-choice
|
||||
--tool-call-parser <family>` plus the matching reasoning parser.
|
||||
Per-model capability overrides opt in to what the checkpoint actually
|
||||
supports:
|
||||
|
||||
```toml
|
||||
[models.vllm-claude]
|
||||
provider = "anthropic-compatible"
|
||||
base_url = "http://localhost:8000" # no /v1 — the SDK appends /v1/messages
|
||||
api_key = "dummy"
|
||||
model = "deepseek-ai/DeepSeek-V4-Flash"
|
||||
|
||||
[models.vllm-claude.capabilities]
|
||||
supports_vision = true # multimodal checkpoints only
|
||||
supports_mid_conversation_system = true # template-dependent
|
||||
context_window = 131072
|
||||
```
|
||||
|
||||
The reasoning toggle does NOT use Anthropic's `thinking` request param.
|
||||
Toggle it through the chat template instead: set `{"chat_template_kwargs":
|
||||
{"thinking": false}}` as extra body params in the admin Models
|
||||
server-compat section (for this provider the section shows only the
|
||||
extra-body field — server type, API surface, and thinking mode are
|
||||
openai-compatible-only knobs); the provider forwards it via the SDK's
|
||||
`extra_body`.
|
||||
|
||||
Verified quirks of vLLM's Anthropic endpoint:
|
||||
|
||||
* The `thinking` request param is silently dropped — use
|
||||
`chat_template_kwargs` (above) to control reasoning.
|
||||
* `stop_sequences` cut the raw stream wherever the text appears —
|
||||
including inside thinking — and report `end_turn` with
|
||||
`stop_sequence=None`. Turnstone does not send stop sequences from
|
||||
this provider.
|
||||
* No cache telemetry: `usage` carries input/output token counts only
|
||||
(no `cache_creation_input_tokens` / `cache_read_input_tokens`).
|
||||
* Images require a multimodal checkpoint — text-only models return a
|
||||
500 on image blocks, so `supports_vision` stays opt-in per model.
|
||||
* Mid-conversation `role: "system"` turns are template-dependent —
|
||||
opt in per model via `supports_mid_conversation_system`.
|
||||
|
||||
**Database model definitions:** On server entry points, models can also be
|
||||
defined in the `model_definitions` table (admin Models tab). DB models support
|
||||
the same per-model sampling overrides. Config.toml models override DB models
|
||||
|
||||
@@ -164,6 +164,25 @@ class TestProbeModelEndpoint:
|
||||
assert result["server_type"] == "anthropic"
|
||||
assert result["context_window"] == 1000000
|
||||
|
||||
@patch("turnstone.core.providers.create_client")
|
||||
def test_anthropic_compatible_server_type(self, mock_cc: MagicMock) -> None:
|
||||
m = _mock_model("deepseek-ai/DeepSeek-V4-Flash")
|
||||
mock_cc.return_value = _mock_client(m)
|
||||
|
||||
result = probe_model_endpoint("anthropic-compatible", "http://localhost:8000", "dummy")
|
||||
assert result["reachable"] is True
|
||||
assert result["server_type"] == "anthropic-compatible"
|
||||
assert result["context_window"] is None
|
||||
|
||||
@patch("turnstone.core.providers.create_client")
|
||||
def test_anthropic_compatible_max_model_len(self, mock_cc: MagicMock) -> None:
|
||||
m = _mock_model("deepseek-ai/DeepSeek-V4-Flash", max_model_len=131072)
|
||||
mock_cc.return_value = _mock_client(m)
|
||||
|
||||
result = probe_model_endpoint("anthropic-compatible", "http://localhost:8000", "dummy")
|
||||
assert result["context_window"] == 131072
|
||||
assert result["server_type"] == "anthropic-compatible"
|
||||
|
||||
@patch("turnstone.core.providers.create_client")
|
||||
def test_connection_failure(self, mock_cc: MagicMock) -> None:
|
||||
mock_cc.side_effect = OSError("Connection refused")
|
||||
|
||||
@@ -142,6 +142,24 @@ def test_create_emits_models_changed(storage: SQLiteBackend) -> None:
|
||||
assert collector.emit_models_changed.call_count == 1
|
||||
|
||||
|
||||
def test_create_accepts_anthropic_compatible_provider(storage: SQLiteBackend) -> None:
|
||||
"""anthropic-compatible passes the _MODEL_PROVIDERS enum check."""
|
||||
client, collector = _make_client(storage)
|
||||
resp = client.post(
|
||||
"/v1/api/admin/model-definitions",
|
||||
json={
|
||||
"alias": "vllm-messages",
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"provider": "anthropic-compatible",
|
||||
"base_url": "http://localhost:8000",
|
||||
"api_key": "dummy",
|
||||
"context_window": 131072,
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert collector.emit_models_changed.call_count == 1
|
||||
|
||||
|
||||
def test_update_emits_models_changed(storage: SQLiteBackend) -> None:
|
||||
_seed(storage, definition_id="m1", alias="local")
|
||||
client, collector = _make_client(storage)
|
||||
|
||||
@@ -0,0 +1,347 @@
|
||||
"""Tests for the ``anthropic-compatible`` provider lane.
|
||||
|
||||
Local servers (vLLM) expose Anthropic's ``/v1/messages`` wire surface for
|
||||
arbitrary checkpoints. The lane reuses ``AnthropicProvider`` with
|
||||
``compat=True``: identical message translation, but capabilities come from
|
||||
``_ANTHROPIC_COMPAT_DEFAULT`` for every model (the static Claude table
|
||||
never applies), native server-side tools are not injected, and operator
|
||||
``server_compat["extra_body"]`` overrides ride the Anthropic SDK's
|
||||
``extra_body`` — the channel for vLLM's ``chat_template_kwargs`` reasoning
|
||||
toggle.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from tests._session_helpers import make_session as _make_session
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _capture_client() -> MagicMock:
|
||||
"""Build a fake Anthropic client whose ``messages.stream`` records kwargs."""
|
||||
stream_ctx = MagicMock()
|
||||
stream_ctx.__enter__ = MagicMock(return_value=iter([]))
|
||||
stream_ctx.__exit__ = MagicMock(return_value=False)
|
||||
client = MagicMock()
|
||||
client.messages.stream.return_value = stream_ctx
|
||||
return client
|
||||
|
||||
|
||||
_WEB_SEARCH_FUNCTION_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "web_search",
|
||||
"description": "Search the web",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"query": {"type": "string"}},
|
||||
"required": ["query"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestCompatCapabilities
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
class TestCompatCapabilities:
|
||||
"""Capability resolution on the compat lane."""
|
||||
|
||||
def test_compat_capability_defaults(self) -> None:
|
||||
provider = AnthropicProvider(compat=True)
|
||||
caps = provider.get_capabilities("deepseek-ai/DeepSeek-V4-Flash")
|
||||
assert caps.token_param == "max_tokens"
|
||||
assert caps.thinking_mode == "none"
|
||||
assert caps.supports_web_search is False
|
||||
assert caps.supports_tool_search is False
|
||||
assert caps.supports_vision is False
|
||||
assert caps.supports_reasoning_replay is True
|
||||
assert caps.supports_temperature is True
|
||||
|
||||
def test_claude_id_does_not_pick_up_static_table(self) -> None:
|
||||
"""A Claude-named local checkpoint must not inherit Claude API caps."""
|
||||
provider = AnthropicProvider(compat=True)
|
||||
caps = provider.get_capabilities("claude-opus-4-6")
|
||||
assert caps.context_window == 200000
|
||||
assert caps.thinking_mode == "none"
|
||||
assert caps.supports_web_search is False
|
||||
# The real lane still resolves the static entry.
|
||||
real_caps = AnthropicProvider().get_capabilities("claude-opus-4-6")
|
||||
assert real_caps.context_window == 1000000
|
||||
assert real_caps.thinking_mode == "adaptive"
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestCompatWireShape
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
class TestCompatWireShape:
|
||||
"""Body-inspecting tests on the kwargs handed to ``messages.stream``."""
|
||||
|
||||
def setup_method(self) -> None:
|
||||
self.provider = AnthropicProvider(compat=True)
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_compat_no_web_search_swap_no_temp_force(self, mock_ensure: MagicMock) -> None:
|
||||
"""No native web_search swap, no temperature=1 forcing, max_tokens param."""
|
||||
client = _capture_client()
|
||||
list(
|
||||
self.provider.create_streaming(
|
||||
client=client,
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
tools=[_WEB_SEARCH_FUNCTION_TOOL],
|
||||
temperature=0.6,
|
||||
)
|
||||
)
|
||||
kwargs = client.messages.stream.call_args[1]
|
||||
sent_tools = kwargs["tools"]
|
||||
assert sent_tools == [
|
||||
{
|
||||
"name": "web_search",
|
||||
"description": "Search the web",
|
||||
"input_schema": {
|
||||
"type": "object",
|
||||
"properties": {"query": {"type": "string"}},
|
||||
"required": ["query"],
|
||||
},
|
||||
}
|
||||
]
|
||||
assert all(t.get("type") != "web_search_20250305" for t in sent_tools)
|
||||
assert kwargs["temperature"] == 0.6
|
||||
assert "thinking" not in kwargs
|
||||
assert "max_tokens" in kwargs
|
||||
assert "max_completion_tokens" not in kwargs
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_extra_params_passthrough_to_extra_body(self, mock_ensure: MagicMock) -> None:
|
||||
"""server_compat extra_body (chat_template_kwargs) reaches the SDK."""
|
||||
client = _capture_client()
|
||||
list(
|
||||
self.provider.create_streaming(
|
||||
client=client,
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
extra_params={"chat_template_kwargs": {"thinking": False}},
|
||||
)
|
||||
)
|
||||
kwargs = client.messages.stream.call_args[1]
|
||||
assert kwargs["extra_body"] == {"chat_template_kwargs": {"thinking": False}}
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_internal_keys_not_leaked(self, mock_ensure: MagicMock) -> None:
|
||||
"""Real-lane request bodies stay byte-identical with thinking overrides.
|
||||
|
||||
``thinking_budget_tokens`` is consumed by ``_reasoning_params`` and
|
||||
must never surface as wire ``extra_body`` — a leaked key would change
|
||||
every real-Anthropic request that threads a thinking override.
|
||||
Negative-tested: fails when the ``_INTERNAL_EXTRA_PARAMS`` exclusion
|
||||
is removed from ``_build_thinking_and_kwargs``.
|
||||
"""
|
||||
provider = AnthropicProvider()
|
||||
client = _capture_client()
|
||||
list(
|
||||
provider.create_streaming(
|
||||
client=client,
|
||||
model="claude-sonnet-4-5",
|
||||
messages=[{"role": "user", "content": "hi"}],
|
||||
extra_params={"thinking_budget_tokens": 2048},
|
||||
)
|
||||
)
|
||||
kwargs = client.messages.stream.call_args[1]
|
||||
assert "extra_body" not in kwargs
|
||||
assert kwargs["thinking"] == {"type": "enabled", "budget_tokens": 2048}
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestCompatFactory
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
class TestCompatFactory:
|
||||
"""create_provider / create_client routing for the compat lane."""
|
||||
|
||||
def test_create_provider_anthropic_compatible(self) -> None:
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
provider = create_provider("anthropic-compatible")
|
||||
assert provider.provider_name == "anthropic-compatible"
|
||||
assert provider is not create_provider("anthropic")
|
||||
assert create_provider("anthropic-compatible") is provider
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_create_client_anthropic_compatible(self, mock_ensure: MagicMock) -> None:
|
||||
"""base_url forwards verbatim; empty api_key is omitted entirely."""
|
||||
from turnstone.core.providers import create_client
|
||||
|
||||
mock_anthropic_cls = MagicMock()
|
||||
mock_mod = MagicMock()
|
||||
mock_mod.Anthropic = mock_anthropic_cls
|
||||
mock_ensure.return_value = mock_mod
|
||||
|
||||
create_client("anthropic-compatible", base_url="http://vllm-host:8000", api_key="")
|
||||
mock_anthropic_cls.assert_called_once_with(base_url="http://vllm-host:8000")
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_create_client_strips_v1_suffix(self, mock_ensure: MagicMock) -> None:
|
||||
"""A /v1-suffixed base_url (openai-compatible muscle memory) is
|
||||
normalized for the compat lane — the SDK appends /v1/... itself,
|
||||
so the verbatim URL would request /v1/v1/messages and 404."""
|
||||
from turnstone.core.providers import create_client
|
||||
|
||||
mock_anthropic_cls = MagicMock()
|
||||
mock_mod = MagicMock()
|
||||
mock_mod.Anthropic = mock_anthropic_cls
|
||||
mock_ensure.return_value = mock_mod
|
||||
|
||||
for suffixed in ("http://vllm-host:8000/v1", "http://vllm-host:8000/v1/"):
|
||||
mock_anthropic_cls.reset_mock()
|
||||
create_client("anthropic-compatible", base_url=suffixed, api_key="")
|
||||
mock_anthropic_cls.assert_called_once_with(base_url="http://vllm-host:8000")
|
||||
|
||||
# A base_url that strips to nothing stays verbatim so the typo
|
||||
# fails loudly in httpx instead of silently targeting the SDK's
|
||||
# prod default.
|
||||
mock_anthropic_cls.reset_mock()
|
||||
create_client("anthropic-compatible", base_url="/v1", api_key="")
|
||||
mock_anthropic_cls.assert_called_once_with(base_url="/v1")
|
||||
|
||||
@patch("turnstone.core.providers._anthropic._ensure_anthropic")
|
||||
def test_create_client_real_lane_base_url_untouched(self, mock_ensure: MagicMock) -> None:
|
||||
"""The real anthropic lane forwards base_url verbatim — the /v1
|
||||
normalization is compat-lane-only."""
|
||||
from turnstone.core.providers import create_client
|
||||
|
||||
mock_anthropic_cls = MagicMock()
|
||||
mock_mod = MagicMock()
|
||||
mock_mod.Anthropic = mock_anthropic_cls
|
||||
mock_ensure.return_value = mock_mod
|
||||
|
||||
create_client("anthropic", base_url="http://proxy:9000/v1", api_key="k")
|
||||
mock_anthropic_cls.assert_called_once_with(api_key="k", base_url="http://proxy:9000/v1")
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestCliScope
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
class TestCliScope:
|
||||
def test_cli_rejects_compat_provider_id(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The lane is registry-only — the CLI --provider flag does not grow."""
|
||||
from turnstone import cli
|
||||
|
||||
monkeypatch.setattr(sys, "argv", ["turnstone", "--provider", "anthropic-compatible"])
|
||||
with pytest.raises(SystemExit) as excinfo:
|
||||
cli.main()
|
||||
assert excinfo.value.code == 2
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestCompatSessionPlumbing
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
class TestCompatSessionPlumbing:
|
||||
"""ChatSession capability merge + extra_params gate for the lane."""
|
||||
|
||||
def test_per_model_capability_override_merge(self, tmp_db: Any) -> None:
|
||||
"""Per-model capabilities win over _ANTHROPIC_COMPAT_DEFAULT fields."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
cfg = ModelConfig(
|
||||
alias="vllm-messages",
|
||||
base_url="http://localhost:8000",
|
||||
api_key="dummy",
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
provider="anthropic-compatible",
|
||||
capabilities={"supports_mid_conversation_system": True, "context_window": 131072},
|
||||
)
|
||||
registry = ModelRegistry(models={"vllm-messages": cfg}, default="vllm-messages")
|
||||
session = _make_session(registry=registry, model_alias="vllm-messages")
|
||||
provider = create_provider("anthropic-compatible")
|
||||
caps = session._resolve_capabilities(
|
||||
provider, "deepseek-ai/DeepSeek-V4-Flash", "vllm-messages"
|
||||
)
|
||||
assert caps.supports_mid_conversation_system is True
|
||||
assert caps.context_window == 131072
|
||||
# Untouched fields keep the compat-lane defaults.
|
||||
assert caps.token_param == "max_tokens"
|
||||
assert caps.thinking_mode == "none"
|
||||
assert caps.supports_web_search is False
|
||||
assert caps.supports_vision is False
|
||||
|
||||
def test_session_extra_params_gate(self, tmp_db: Any) -> None:
|
||||
"""server_compat extra_body forwards for the compat lane, not real Anthropic."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
session = _make_session(reasoning_effort="medium")
|
||||
cfg = ModelConfig(
|
||||
alias="vllm-messages",
|
||||
base_url="http://localhost:8000",
|
||||
api_key="dummy",
|
||||
model="deepseek-ai/DeepSeek-V4-Flash",
|
||||
provider="anthropic-compatible",
|
||||
server_compat={"extra_body": {"chat_template_kwargs": {"thinking": False}}},
|
||||
)
|
||||
session._registry = ModelRegistry(models={"vllm-messages": cfg}, default="vllm-messages")
|
||||
session._model_alias = "vllm-messages"
|
||||
|
||||
session._provider = create_provider("anthropic-compatible")
|
||||
assert session._provider_extra_params() == {"chat_template_kwargs": {"thinking": False}}
|
||||
|
||||
session._provider = create_provider("anthropic")
|
||||
assert session._provider_extra_params() is None
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# TestLiveCompatStream
|
||||
# ===========================================================================
|
||||
|
||||
|
||||
@pytest.mark.live
|
||||
@pytest.mark.skipif(
|
||||
not os.environ.get("TURNSTONE_LIVE_ANTHROPIC_COMPAT_URL"),
|
||||
reason="TURNSTONE_LIVE_ANTHROPIC_COMPAT_URL not set",
|
||||
)
|
||||
class TestLiveCompatStream:
|
||||
"""One real streamed turn against a vLLM /v1/messages endpoint."""
|
||||
|
||||
def test_live_compat_streamed_turn(self) -> None:
|
||||
from turnstone.core.providers import create_client, create_provider
|
||||
|
||||
base_url = os.environ["TURNSTONE_LIVE_ANTHROPIC_COMPAT_URL"]
|
||||
model = os.environ.get(
|
||||
"TURNSTONE_LIVE_ANTHROPIC_COMPAT_MODEL", "deepseek-ai/DeepSeek-V4-Flash"
|
||||
)
|
||||
client = create_client("anthropic-compatible", base_url=base_url, api_key="dummy")
|
||||
provider = create_provider("anthropic-compatible")
|
||||
chunks = list(
|
||||
provider.create_streaming(
|
||||
client=client,
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "Reply with the single word: pong"}],
|
||||
max_tokens=64,
|
||||
extra_params={"chat_template_kwargs": {"thinking": False}},
|
||||
)
|
||||
)
|
||||
content = "".join(c.content_delta or "" for c in chunks)
|
||||
assert content.strip()
|
||||
assert any(c.finish_reason for c in chunks)
|
||||
assert any(c.usage is not None for c in chunks)
|
||||
assert not any(c.reasoning_delta for c in chunks)
|
||||
@@ -10210,7 +10210,9 @@ async def admin_import_mcp_config(request: Request) -> JSONResponse:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_MODEL_ALIAS_RE = re.compile(r"^[a-zA-Z0-9._-]+$")
|
||||
_MODEL_PROVIDERS = frozenset({"openai", "anthropic", "openai-compatible", "google", "xai"})
|
||||
_MODEL_PROVIDERS = frozenset(
|
||||
{"openai", "anthropic", "openai-compatible", "anthropic-compatible", "google", "xai"}
|
||||
)
|
||||
_REASONING_EFFORT_CHOICES = frozenset(
|
||||
{"", "none", "minimal", "low", "medium", "high", "xhigh", "max"}
|
||||
)
|
||||
|
||||
@@ -5596,7 +5596,8 @@ function _renderModels(items) {
|
||||
? "model-provider-anthropic"
|
||||
: m.provider === "google"
|
||||
? "model-provider-google"
|
||||
: m.provider === "openai-compatible"
|
||||
: m.provider === "openai-compatible" ||
|
||||
m.provider === "anthropic-compatible"
|
||||
? "model-provider-compat"
|
||||
: "model-provider-openai";
|
||||
|
||||
@@ -5896,12 +5897,16 @@ function showEditModelModal(definitionId) {
|
||||
? capsObj.server_compat
|
||||
: {};
|
||||
// Only extract thinking_mode into the dropdown when the UI can
|
||||
// represent it ("manual" or ""). Values like "adaptive" (Anthropic-
|
||||
// only) stay in the raw capabilities JSON so they aren't silently
|
||||
// lost on save.
|
||||
// represent it ("manual" or "") AND the provider round-trips the
|
||||
// dropdown on save (every provider except anthropic-compatible —
|
||||
// see submitCreateModel). Unrepresentable values like "adaptive"
|
||||
// and anthropic-compatible rows keep thinking_mode in the raw
|
||||
// capabilities JSON so it isn't silently lost on save.
|
||||
const tmVal = capsObj.thinking_mode || "";
|
||||
const tmRepresentable = tmVal === "" || tmVal === "manual";
|
||||
if (tmRepresentable) {
|
||||
const tmCaptured =
|
||||
tmRepresentable && (m.provider || "openai") !== "anthropic-compatible";
|
||||
if (tmCaptured) {
|
||||
document.getElementById("model-thinking-mode").value = tmVal;
|
||||
document.getElementById("model-thinking-param").value =
|
||||
capsObj.thinking_param || "";
|
||||
@@ -5945,7 +5950,7 @@ function showEditModelModal(definitionId) {
|
||||
// Remove structured fields from capabilities display — only delete
|
||||
// thinking_mode/thinking_param when the UI successfully captured them.
|
||||
delete capsObj.server_compat;
|
||||
if (tmRepresentable) {
|
||||
if (tmCaptured) {
|
||||
delete capsObj.thinking_mode;
|
||||
delete capsObj.thinking_param;
|
||||
}
|
||||
@@ -6017,10 +6022,16 @@ function submitCreateModel() {
|
||||
}
|
||||
}
|
||||
|
||||
// Thinking mode → capabilities (provider uses this to inject
|
||||
// the correct chat_template_kwargs param automatically).
|
||||
const providerVal = document.getElementById("model-provider").value;
|
||||
|
||||
// Thinking mode → capabilities. thinking_mode round-trips through the
|
||||
// dropdown for every provider EXCEPT anthropic-compatible, where it
|
||||
// stays in the raw capabilities JSON (mirroring the edit-load lift):
|
||||
// that lane hides the dropdown row and drives reasoning via extra-body
|
||||
// chat_template_kwargs, so a lingering dropdown value must never be
|
||||
// persisted.
|
||||
const thinkingMode = document.getElementById("model-thinking-mode").value;
|
||||
if (thinkingMode) {
|
||||
if (providerVal !== "anthropic-compatible" && thinkingMode) {
|
||||
caps.thinking_mode = thinkingMode;
|
||||
// Preserve thinking_param so Granite/DeepSeek "thinking" key
|
||||
// isn't silently reverted to the default "enable_thinking".
|
||||
@@ -6028,20 +6039,25 @@ function submitCreateModel() {
|
||||
if (savedParam) caps.thinking_param = savedParam;
|
||||
}
|
||||
|
||||
// Build server_compat from structured fields. Only meaningful for
|
||||
// openai-compatible aliases — for other providers the section is hidden
|
||||
// Build server_compat from structured fields. Only meaningful for the
|
||||
// compat lanes (openai-compatible: all fields; anthropic-compatible: the
|
||||
// extra-body JSON only) — for other providers the section is hidden
|
||||
// but the form values can linger after a provider switch, so gate the
|
||||
// whole block on the active provider to keep persisted state honest.
|
||||
const serverCompat = {};
|
||||
const providerVal = document.getElementById("model-provider").value;
|
||||
const ebEl = document.getElementById("model-extra-body");
|
||||
ebEl.removeAttribute("aria-invalid");
|
||||
ebEl.style.borderColor = "";
|
||||
if (providerVal === "openai-compatible") {
|
||||
const serverType = document.getElementById("model-server-type").value;
|
||||
if (serverType) serverCompat.server_type = serverType;
|
||||
const apiSurface = document.getElementById("model-api-surface").value;
|
||||
if (apiSurface) serverCompat.api_surface = apiSurface;
|
||||
if (
|
||||
providerVal === "openai-compatible" ||
|
||||
providerVal === "anthropic-compatible"
|
||||
) {
|
||||
if (providerVal === "openai-compatible") {
|
||||
const serverType = document.getElementById("model-server-type").value;
|
||||
if (serverType) serverCompat.server_type = serverType;
|
||||
const apiSurface = document.getElementById("model-api-surface").value;
|
||||
if (apiSurface) serverCompat.api_surface = apiSurface;
|
||||
}
|
||||
const ebText = ebEl.value.trim();
|
||||
if (ebText) {
|
||||
try {
|
||||
@@ -6516,7 +6532,11 @@ function _modelCapsRefreshBaseline() {
|
||||
const provider = document.getElementById("model-provider").value;
|
||||
const modelName = document.getElementById("model-name").value.trim();
|
||||
const banner = document.getElementById("model-autofill");
|
||||
if (!modelName || provider === "openai-compatible") {
|
||||
if (
|
||||
!modelName ||
|
||||
provider === "openai-compatible" ||
|
||||
provider === "anthropic-compatible"
|
||||
) {
|
||||
_modelCapsBaseline = {};
|
||||
banner.hidden = true;
|
||||
_modelRenderTiles();
|
||||
@@ -6578,6 +6598,10 @@ const _providerDefaults = {
|
||||
urlPlaceholder: "e.g. https://your-provider.com/v1",
|
||||
modelPlaceholder: "GLM5",
|
||||
},
|
||||
"anthropic-compatible": {
|
||||
urlPlaceholder: "e.g. http://your-vllm-host:8000",
|
||||
modelPlaceholder: "deepseek-ai/DeepSeek-V4-Flash",
|
||||
},
|
||||
};
|
||||
|
||||
/* Update placeholders when provider changes. */
|
||||
@@ -6592,8 +6616,18 @@ function _applyProviderDefaults() {
|
||||
if (scSection) {
|
||||
// hidden attr, not style.display — `.hatch [hidden]` is !important and
|
||||
// an inline display can never un-hide it.
|
||||
scSection.hidden = provider !== "openai-compatible";
|
||||
scSection.hidden =
|
||||
provider !== "openai-compatible" && provider !== "anthropic-compatible";
|
||||
}
|
||||
// Within the section, server type / API surface / thinking mode are
|
||||
// openai-compatible knobs — the anthropic-compatible lane is configured
|
||||
// through the extra-body JSON alone, so collapse the section to just
|
||||
// that field.
|
||||
const hideOpenaiOnlyRows = provider === "anthropic-compatible";
|
||||
["model-server-fields-row", "model-thinking-mode-row"].forEach(function (id) {
|
||||
const row = document.getElementById(id);
|
||||
if (row) row.hidden = hideOpenaiOnlyRows;
|
||||
});
|
||||
}
|
||||
|
||||
/* Populate the model name datalist with known model prefixes for the
|
||||
|
||||
@@ -1577,6 +1577,7 @@
|
||||
<option value="anthropic">anthropic</option>
|
||||
<option value="google">google</option>
|
||||
<option value="openai-compatible">openai-compatible</option>
|
||||
<option value="anthropic-compatible">anthropic-compatible</option>
|
||||
</select>
|
||||
</div>
|
||||
<div>
|
||||
@@ -1673,7 +1674,7 @@
|
||||
|
||||
<div id="model-server-compat-section" hidden>
|
||||
<div class="sh-section">Server compatibility</div>
|
||||
<div class="field-pair">
|
||||
<div class="field-pair" id="model-server-fields-row">
|
||||
<div>
|
||||
<label for="model-server-type"
|
||||
>Server type
|
||||
@@ -1702,28 +1703,32 @@
|
||||
</select>
|
||||
</div>
|
||||
</div>
|
||||
<label for="model-thinking-mode"
|
||||
>Thinking mode
|
||||
<span class="label-hint">chat-template reasoning</span></label
|
||||
>
|
||||
<select id="model-thinking-mode">
|
||||
<option value="">None</option>
|
||||
<option value="manual">Enabled</option>
|
||||
</select>
|
||||
<div id="model-thinking-param-row" hidden>
|
||||
<label for="model-thinking-param"
|
||||
>Template param name
|
||||
<div id="model-thinking-mode-row">
|
||||
<label for="model-thinking-mode"
|
||||
>Thinking mode
|
||||
<span class="label-hint"
|
||||
>Granite/DeepSeek use "thinking"</span
|
||||
>chat-template reasoning</span
|
||||
></label
|
||||
>
|
||||
<input
|
||||
type="text"
|
||||
id="model-thinking-param"
|
||||
class="sh-mono"
|
||||
value="enable_thinking"
|
||||
placeholder="enable_thinking"
|
||||
/>
|
||||
<select id="model-thinking-mode">
|
||||
<option value="">None</option>
|
||||
<option value="manual">Enabled</option>
|
||||
</select>
|
||||
<div id="model-thinking-param-row" hidden>
|
||||
<label for="model-thinking-param"
|
||||
>Template param name
|
||||
<span class="label-hint"
|
||||
>Granite/DeepSeek use "thinking"</span
|
||||
></label
|
||||
>
|
||||
<input
|
||||
type="text"
|
||||
id="model-thinking-param"
|
||||
class="sh-mono"
|
||||
value="enable_thinking"
|
||||
placeholder="enable_thinking"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
<label for="model-extra-body"
|
||||
>Extra body params
|
||||
|
||||
@@ -817,6 +817,14 @@ def probe_model_endpoint(
|
||||
if known is not None:
|
||||
result["context_window"] = known["context_window"]
|
||||
result["server_type"] = "xai"
|
||||
elif provider == "anthropic-compatible":
|
||||
# No static table for local models; vLLM exposes max_model_len
|
||||
# as an extra field the Anthropic SDK preserves (extra="allow").
|
||||
if inspect_obj is not None:
|
||||
max_len = inspect_obj.model_dump().get("max_model_len")
|
||||
if isinstance(max_len, int) and max_len > 0:
|
||||
result["context_window"] = max_len
|
||||
result["server_type"] = "anthropic-compatible"
|
||||
else:
|
||||
# OpenAI-compatible path
|
||||
_detect_openai_compat(result, inspect_obj, inspect_id, base_url)
|
||||
|
||||
@@ -46,6 +46,7 @@ _openai_provider = OpenAIResponsesProvider()
|
||||
_openai_compat_provider = OpenAIChatCompletionsProvider()
|
||||
_xai_provider = XAIProvider()
|
||||
_anthropic_provider: LLMProvider | None = None
|
||||
_anthropic_compat_provider: LLMProvider | None = None
|
||||
_google_provider: LLMProvider | None = None
|
||||
|
||||
|
||||
@@ -67,8 +68,13 @@ def create_provider(
|
||||
endpoints like Mistral cloud, or local servers that expose the
|
||||
Responses surface).
|
||||
|
||||
Ignored for non-OpenAI providers. ``provider_name="openai"`` always
|
||||
uses the Responses API regardless of *api_surface*.
|
||||
Ignored for non-OpenAI providers — both Anthropic lanes
|
||||
(``"anthropic"`` and ``"anthropic-compatible"``) talk to the
|
||||
Messages API regardless of *api_surface*.
|
||||
``provider_name="anthropic-compatible"`` returns the Anthropic
|
||||
adapter in compat mode (local servers exposing ``/v1/messages``,
|
||||
e.g. vLLM). ``provider_name="openai"`` always uses the Responses
|
||||
API regardless of *api_surface*.
|
||||
|
||||
Note: the ``OpenAIResponsesProvider`` singleton is reused for both
|
||||
cloud OpenAI and ``openai-compatible`` + responses, so its
|
||||
@@ -77,7 +83,7 @@ def create_provider(
|
||||
must read ``ModelConfig.provider`` and ``server_compat["api_surface"]``
|
||||
rather than ``provider.provider_name``.
|
||||
"""
|
||||
global _anthropic_provider, _google_provider # noqa: PLW0603
|
||||
global _anthropic_provider, _anthropic_compat_provider, _google_provider # noqa: PLW0603
|
||||
if provider_name == "openai":
|
||||
return _openai_provider
|
||||
if provider_name == "openai-compatible":
|
||||
@@ -98,6 +104,13 @@ def create_provider(
|
||||
|
||||
_anthropic_provider = AnthropicProvider()
|
||||
return _anthropic_provider
|
||||
if provider_name == "anthropic-compatible":
|
||||
with _provider_lock:
|
||||
if _anthropic_compat_provider is None:
|
||||
from turnstone.core.providers._anthropic import AnthropicProvider
|
||||
|
||||
_anthropic_compat_provider = AnthropicProvider(compat=True)
|
||||
return _anthropic_compat_provider
|
||||
if provider_name == "google":
|
||||
with _provider_lock:
|
||||
if _google_provider is None:
|
||||
@@ -107,7 +120,7 @@ def create_provider(
|
||||
return _google_provider
|
||||
raise ValueError(
|
||||
f"Unknown provider: {provider_name!r}. "
|
||||
"Supported: openai, anthropic, google, openai-compatible, xai"
|
||||
"Supported: openai, anthropic, google, openai-compatible, anthropic-compatible, xai"
|
||||
)
|
||||
|
||||
|
||||
@@ -133,19 +146,28 @@ def create_client(provider_name: str, *, base_url: str, api_key: str) -> Any:
|
||||
if base_url:
|
||||
return OpenAI(base_url=base_url, api_key=resolved_key)
|
||||
return OpenAI(api_key=resolved_key)
|
||||
if provider_name == "anthropic":
|
||||
if provider_name in ("anthropic", "anthropic-compatible"):
|
||||
from turnstone.core.providers._anthropic import _ensure_anthropic
|
||||
|
||||
anthropic = _ensure_anthropic()
|
||||
kwargs: dict[str, str] = {}
|
||||
if resolved_key is not None:
|
||||
kwargs["api_key"] = resolved_key
|
||||
if provider_name == "anthropic-compatible" and base_url:
|
||||
# The Anthropic SDK appends /v1/... to base_url, so a
|
||||
# /v1-suffixed URL (the openai-compatible convention) would
|
||||
# request /v1/v1/messages and 404. Tolerate the suffix.
|
||||
# Keep the verbatim value when stripping would empty it
|
||||
# (base_url of exactly "/v1") so the typo still fails loudly
|
||||
# instead of silently retargeting the SDK's prod default.
|
||||
stripped = base_url.rstrip("/").removesuffix("/v1")
|
||||
base_url = stripped or base_url
|
||||
if base_url and base_url != "https://api.anthropic.com":
|
||||
kwargs["base_url"] = base_url
|
||||
return anthropic.Anthropic(**kwargs)
|
||||
raise ValueError(
|
||||
f"Unknown provider: {provider_name!r}. "
|
||||
"Supported: openai, anthropic, google, openai-compatible, xai"
|
||||
"Supported: openai, anthropic, google, openai-compatible, anthropic-compatible, xai"
|
||||
)
|
||||
|
||||
|
||||
@@ -153,11 +175,12 @@ def lookup_model_capabilities(provider: str, model: str) -> dict[str, Any] | Non
|
||||
"""Return static capabilities for a known model, or ``None`` if unknown.
|
||||
|
||||
The returned dict has JSON-friendly values (tuples converted to lists).
|
||||
Returns ``None`` for ``openai-compatible`` (no static table for local models).
|
||||
Returns ``None`` for ``openai-compatible`` and ``anthropic-compatible``
|
||||
(no static table for local models).
|
||||
"""
|
||||
import dataclasses
|
||||
|
||||
if provider == "openai-compatible":
|
||||
if provider in ("openai-compatible", "anthropic-compatible"):
|
||||
return None
|
||||
prov = create_provider(provider)
|
||||
caps = prov.get_capabilities(model)
|
||||
|
||||
@@ -71,6 +71,11 @@ _WEB_SEARCH_TOOL_TYPE = "web_search_20250305"
|
||||
# Tool search: server-side BM25 tool discovery for deferred tools
|
||||
_TOOL_SEARCH_TOOL_TYPE = "tool_search_tool_bm25"
|
||||
|
||||
# extra_params keys consumed internally (``_reasoning_params``) — never
|
||||
# forwarded to the wire ``extra_body``. Keeps real-Anthropic request
|
||||
# bodies byte-identical when a caller threads thinking overrides.
|
||||
_INTERNAL_EXTRA_PARAMS = frozenset({"thinking_budget_tokens"})
|
||||
|
||||
# -- model capabilities -------------------------------------------------------
|
||||
|
||||
_ANTHROPIC_DEFAULT = ModelCapabilities(
|
||||
@@ -83,6 +88,27 @@ _ANTHROPIC_DEFAULT = ModelCapabilities(
|
||||
supports_reasoning_replay=True,
|
||||
)
|
||||
|
||||
# Anthropic-compatible local servers (vLLM's /v1/messages endpoint):
|
||||
# token_param must be "max_tokens" (the only token param the endpoint
|
||||
# accepts); the "thinking" request param is not consumed by vLLM — the
|
||||
# reasoning toggle rides chat_template_kwargs via extra_body, so
|
||||
# thinking_mode stays "none"; supports_reasoning_replay stays True even
|
||||
# so, because the endpoint emits and round-trips thinking blocks whenever
|
||||
# the chat template enables reasoning (the request param is simply not
|
||||
# the switch); native web_search / tool_search server-tool types 400 on
|
||||
# vLLM (tools require input_schema); vision is opt-in per model.
|
||||
# supports_temperature stays True via the dataclass default.
|
||||
_ANTHROPIC_COMPAT_DEFAULT = ModelCapabilities(
|
||||
context_window=200000,
|
||||
max_output_tokens=64000,
|
||||
token_param="max_tokens",
|
||||
thinking_mode="none",
|
||||
supports_web_search=False,
|
||||
supports_tool_search=False,
|
||||
supports_vision=False,
|
||||
supports_reasoning_replay=True,
|
||||
)
|
||||
|
||||
_ANTHROPIC_CAPABILITIES: dict[str, ModelCapabilities] = {
|
||||
# Fable 5: same wire surface as opus-4-8 (adaptive-only thinking, no
|
||||
# sampling params, prefill rejected) with one extra constraint — an
|
||||
@@ -239,13 +265,24 @@ ANTHROPIC_REASONING_BLOCK_TYPES = frozenset({"thinking", "redacted_thinking"})
|
||||
|
||||
|
||||
class AnthropicProvider:
|
||||
"""Provider for Anthropic's Messages API with native streaming."""
|
||||
"""Provider for Anthropic's Messages API with native streaming.
|
||||
|
||||
``compat=True`` serves Anthropic-compatible local servers (vLLM's
|
||||
``/v1/messages``): same wire translation, but capabilities come from
|
||||
``_ANTHROPIC_COMPAT_DEFAULT`` for every model — the static Claude
|
||||
table never applies to local checkpoints.
|
||||
"""
|
||||
|
||||
def __init__(self, *, compat: bool = False) -> None:
|
||||
self._compat = compat
|
||||
|
||||
@property
|
||||
def provider_name(self) -> str:
|
||||
return "anthropic"
|
||||
return "anthropic-compatible" if self._compat else "anthropic"
|
||||
|
||||
def get_capabilities(self, model: str) -> ModelCapabilities:
|
||||
if self._compat:
|
||||
return _ANTHROPIC_COMPAT_DEFAULT
|
||||
return _lookup_capabilities(model, _ANTHROPIC_CAPABILITIES, _ANTHROPIC_DEFAULT)
|
||||
|
||||
# -- web search tool injection -------------------------------------------
|
||||
@@ -352,6 +389,13 @@ class AnthropicProvider:
|
||||
if effort:
|
||||
kwargs["output_config"] = {"effort": effort}
|
||||
|
||||
# Operator server_compat extra_body overrides (e.g. chat_template_kwargs
|
||||
# for anthropic-compatible local servers) ride the SDK's extra_body.
|
||||
if extra_params:
|
||||
wire_extra = {k: v for k, v in extra_params.items() if k not in _INTERNAL_EXTRA_PARAMS}
|
||||
if wire_extra:
|
||||
kwargs["extra_body"] = wire_extra
|
||||
|
||||
return kwargs
|
||||
|
||||
# -- message conversion --------------------------------------------------
|
||||
|
||||
@@ -3135,7 +3135,10 @@ class ChatSession:
|
||||
|
||||
Forwards operator-supplied ``server_compat["extra_body"]`` overrides
|
||||
(``skip_special_tokens``, ``reasoning_format``, or explicit
|
||||
``chat_template_kwargs``) to the OpenAI SDK ``extra_body``. Operators
|
||||
``chat_template_kwargs``) to the OpenAI SDK ``extra_body`` on the
|
||||
OpenAI-shaped lanes, and to the Anthropic SDK ``extra_body`` on the
|
||||
anthropic-compatible lane (the channel for vLLM's
|
||||
``chat_template_kwargs`` reasoning toggle). Operators
|
||||
running gpt-oss-style local templates that consume ``reasoning_effort``
|
||||
from ``chat_template_kwargs`` should set it explicitly under
|
||||
``server_compat["extra_body"]["chat_template_kwargs"]``.
|
||||
@@ -3151,9 +3154,11 @@ class ChatSession:
|
||||
from turnstone.core.server_compat import merge_server_compat
|
||||
|
||||
prov = provider or self._provider
|
||||
# Only OpenAI-shaped providers consume extra_body. Anthropic/Google
|
||||
# have their own param paths handled inside their providers.
|
||||
if prov.provider_name not in ("openai", "openai-compatible"):
|
||||
# extra_body consumers: the OpenAI-shaped providers, plus the
|
||||
# anthropic-compatible lane (server_compat extra_body rides the
|
||||
# Anthropic SDK's extra_body). Real Anthropic and Google keep
|
||||
# their own param paths handled inside their providers.
|
||||
if prov.provider_name not in ("openai", "openai-compatible", "anthropic-compatible"):
|
||||
return None
|
||||
extra = merge_server_compat(None, self._get_server_compat(model_alias))
|
||||
return extra or None
|
||||
|
||||
Reference in New Issue
Block a user