feat(providers): api_surface toggle + mistral medium reasoning fix (#469)

* feat(providers): api_surface toggle + mistral medium reasoning fix

Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions.  The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).

Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
  and thread it through `create_provider` / `model_registry.get_provider`.
  `openai-compatible` defaults to Chat Completions; operators can flip
  individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
  on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
  `chat_template_kwargs`.  Operators running gpt-oss-style local
  templates that consume `reasoning_effort` from the chat template now
  opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
  server-side at create/update time; pre-filled by Detect via the
  profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
  api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
  for capability and server_compat resolution; previously the fallback
  passed `alias=None`, which silently dropped per-model caps on the
  agent path.

Tests: 5117 passed (-m "not live"); ruff + mypy clean.

* fix(providers): don't auto-suggest Responses for Mistral medium

vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls.  Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.

Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile.  Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.

* fix(providers): address Copilot review on PR #469

- providers/__init__.py: drop the redundant *_responses_provider /
  *_chat_provider names; have create_provider use _openai_provider and
  _openai_compat_provider directly so they're not flagged as unused
  globals.
- console/server.py: tighten _validate_api_surface to a strict equality
  match against the canonical {"chat", "responses"} set.  The previous
  strip().lower() membership check accepted ' Responses '/'CHAT' but
  stored the raw string verbatim, which then failed to round-trip
  through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
  type, api_surface, extra_body) on provider == "openai-compatible" at
  save time so toggling provider away can't leave a stale hidden surface
  selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
  longer flags the call as a wrong-name keyword (the point of the test
  is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
  for the api_surface validation on both create and update — covers the
  bogus-value rejection, non-canonical-string rejection, and the happy
  path persisting through to the refreshed registry.
This commit is contained in:
Patrick Buckley
2026-05-03 13:37:50 -07:00
committed by Patrick Buckley
parent 2fd0c29a92
commit ec74334e74
12 changed files with 524 additions and 169 deletions
@@ -352,6 +352,90 @@ def test_update_endpoint_skips_refresh_on_empty_body(
assert calls == [] # gate held: empty body did not trigger a refresh
def test_create_rejects_invalid_api_surface(storage: SQLiteBackend) -> None:
"""POST with a bogus server_compat.api_surface returns 400 rather than
persisting a value that would make get_provider() raise on every later
ChatSession init for the alias."""
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
registry = _make_registry(alias="local", model="m")
client = _make_client(storage, registry)
resp = client.post(
"/v1/api/admin/model-definitions",
json={
"alias": "bad",
"model": "x",
"provider": "openai-compatible",
"base_url": "http://localhost:9000/v1",
"api_key": "sk-x",
"capabilities": {"server_compat": {"api_surface": "BOGUS"}},
},
)
assert resp.status_code == 400, resp.text
assert "api_surface" in resp.json()["error"]
# And the alias is not persisted
assert not registry.has_alias("bad")
def test_create_rejects_non_canonical_api_surface(storage: SQLiteBackend) -> None:
"""Strict validation: ' Responses ' / 'CHAT' don't round-trip through the
admin <select>, so they're rejected even though they'd survive a
case-insensitive membership check."""
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
registry = _make_registry(alias="local", model="m")
client = _make_client(storage, registry)
for bad in (" responses ", "RESPONSES", "Chat"):
resp = client.post(
"/v1/api/admin/model-definitions",
json={
"alias": "noncanon",
"model": "x",
"provider": "openai-compatible",
"base_url": "http://localhost:9000/v1",
"api_key": "sk-x",
"capabilities": {"server_compat": {"api_surface": bad}},
},
)
assert resp.status_code == 400, f"{bad!r}: {resp.text}"
def test_create_accepts_valid_api_surface(storage: SQLiteBackend) -> None:
"""Canonical 'chat' / 'responses' / unset are all accepted and persisted."""
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
registry = _make_registry(alias="local", model="m")
client = _make_client(storage, registry)
resp = client.post(
"/v1/api/admin/model-definitions",
json={
"alias": "responses-alias",
"model": "x",
"provider": "openai-compatible",
"base_url": "http://localhost:9000/v1",
"api_key": "sk-x",
"capabilities": {"server_compat": {"api_surface": "responses"}},
},
)
assert resp.status_code == 200, resp.text
assert registry.has_alias("responses-alias")
def test_update_rejects_invalid_api_surface(storage: SQLiteBackend) -> None:
"""PUT path also gates the validation, so an admin can't smuggle a bad
value into an existing alias."""
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
registry = _make_registry(alias="local", model="m")
client = _make_client(storage, registry)
resp = client.put(
"/v1/api/admin/model-definitions/m1",
json={"capabilities": {"server_compat": {"api_surface": "junk"}}},
)
assert resp.status_code == 400, resp.text
assert "api_surface" in resp.json()["error"]
def test_delete_endpoint_refreshes_registry(storage: SQLiteBackend) -> None:
"""DELETE drops the alias from the in-process registry too — a
coord session that tried to resolve the deleted alias would
+78 -6
View File
@@ -1126,24 +1126,57 @@ class TestSessionAgentModel:
def _captured_effort(captured: dict[str, Any]) -> str | None:
"""Pull reasoning_effort out of provider-specific shapes.
openai-compatible servers receive it via extra_body.chat_template_kwargs;
commercial providers receive it as a top-level kwarg.
Chat Completions delivers it as a top-level ``reasoning_effort`` kwarg
(when the model's caps permit it). Operators who route reasoning_effort
through ``chat_template_kwargs`` (gpt-oss-style local templates) get
it inside ``extra_body.chat_template_kwargs``.
"""
if "reasoning_effort" in captured:
return captured["reasoning_effort"]
eb = captured.get("extra_body") or {}
ctk = eb.get("chat_template_kwargs") or {}
return ctk.get("reasoning_effort") or captured.get("reasoning_effort")
return ctk.get("reasoning_effort")
@staticmethod
def _effort_caps() -> dict[str, Any]:
"""Capabilities that allow Chat-Completions reasoning_effort to flow."""
return {
"reasoning_effort_values": [
"minimal",
"low",
"medium",
"high",
"max",
],
}
def _three_model_registry(self, **kwargs: Any) -> ModelRegistry:
caps = self._effort_caps()
return ModelRegistry(
models={
"main": ModelConfig(
"main", "http://m/v1", "k", "main-model", provider="openai-compatible"
"main",
"http://m/v1",
"k",
"main-model",
provider="openai-compatible",
capabilities=dict(caps),
),
"smart": ModelConfig(
"smart", "http://s/v1", "k", "smart-model", provider="openai-compatible"
"smart",
"http://s/v1",
"k",
"smart-model",
provider="openai-compatible",
capabilities=dict(caps),
),
"fast": ModelConfig(
"fast", "http://f/v1", "k", "fast-model", provider="openai-compatible"
"fast",
"http://f/v1",
"k",
"fast-model",
provider="openai-compatible",
capabilities=dict(caps),
),
},
default="main",
@@ -1249,6 +1282,45 @@ class TestSessionAgentModel:
session._run_agent([{"role": "user", "content": "x"}], label="plan", agent_alias="fast")
assert captured["model"] == "fast-model"
def test_session_fallback_inherits_primary_alias_for_caps(self) -> None:
"""When _run_agent has no registry agent route, it must fall back to
the session's primary alias for capability and server_compat lookup —
otherwise per-model caps (reasoning_effort_values, server_compat) get
silently dropped on the agent path."""
reg = self._three_model_registry() # no agent_model / plan_model set
session = _make_session(registry=reg, model_alias="main")
# Probe what _run_agent passes to _provider_extra_params and
# _resolve_capabilities by recording the model_alias on each call.
captured_extra_alias: list[str | None] = []
captured_resolve_alias: list[str | None] = []
original_extra = session._provider_extra_params
original_resolve = session._resolve_capabilities
def spy_extra(*args: Any, **kwargs: Any) -> Any:
captured_extra_alias.append(kwargs.get("model_alias"))
return original_extra(*args, **kwargs)
def spy_resolve(*args: Any, **kwargs: Any) -> Any:
# _resolve_capabilities(provider, model, alias)
alias = args[2] if len(args) >= 3 else kwargs.get("alias")
captured_resolve_alias.append(alias)
return original_resolve(*args, **kwargs)
session._provider_extra_params = spy_extra # type: ignore[method-assign]
session._resolve_capabilities = spy_resolve # type: ignore[method-assign]
self._capture_on(session.client) # patch client.chat.completions.create
session._run_agent([{"role": "user", "content": "x"}], label="plan")
assert captured_extra_alias and captured_extra_alias[-1] == "main", (
f"agent fallback path did not inherit primary alias for extra_params: "
f"{captured_extra_alias!r}"
)
assert captured_resolve_alias and captured_resolve_alias[-1] == "main", (
f"agent fallback path did not inherit primary alias for caps: "
f"{captured_resolve_alias!r}"
)
def test_invalid_alias_raises_in_run_agent(self) -> None:
"""Defence-in-depth: _prepare_* validates first, but _run_agent
rejects unknown aliases too rather than silently falling back."""
+28
View File
@@ -1383,6 +1383,34 @@ class TestProviderFactory:
p2 = create_provider("openai")
assert p1 is p2
def test_create_provider_compat_responses_surface(self) -> None:
"""openai-compatible + api_surface=responses returns the Responses provider."""
from turnstone.core.providers import OpenAIResponsesProvider, create_provider
provider = create_provider("openai-compatible", api_surface="responses")
assert isinstance(provider, OpenAIResponsesProvider)
def test_create_provider_compat_chat_surface_default(self) -> None:
"""openai-compatible defaults to Chat Completions."""
from turnstone.core.providers import create_provider
for surface in (None, "", "chat"):
provider = create_provider("openai-compatible", api_surface=surface)
assert isinstance(provider, OpenAIChatCompletionsProvider)
def test_create_provider_invalid_api_surface(self) -> None:
from turnstone.core.providers import create_provider
with pytest.raises(ValueError, match="Unknown api_surface"):
create_provider("openai-compatible", api_surface="bogus")
def test_create_provider_openai_ignores_api_surface(self) -> None:
"""Cloud OpenAI is always Responses regardless of api_surface."""
from turnstone.core.providers import OpenAIResponsesProvider, create_provider
provider = create_provider("openai", api_surface="chat")
assert isinstance(provider, OpenAIResponsesProvider)
# -- Google provider -------------------------------------------------------
def test_create_provider_google(self) -> None:
+80 -35
View File
@@ -89,6 +89,27 @@ class TestSuggestProfile:
p = suggest_profile("vllm", "Google/GEMMA-4-31B-IT")
assert p["capabilities"]["thinking_mode"] == "manual"
def test_vllm_mistral_medium_not_auto_suggested(self) -> None:
"""Mistral medium falls back to the generic vLLM profile.
We don't auto-suggest the Responses surface for Mistral medium because
vLLM's Responses API tool-call parser isn't wired up for it yet —
operators who want per-request reasoning effort must pick "Responses
API" manually in the admin UI and accept the tool-calling limitation.
"""
p = suggest_profile("vllm", "mistralai/Mistral-Medium-3-Instruct")
assert p["server_compat"]["server_type"] == "vllm"
assert "api_surface" not in p["server_compat"]
assert "capabilities" not in p
def test_vllm_mistral_medium_profile_still_available(self) -> None:
"""The vllm-mistral-medium profile remains in _PROFILES so an operator
who explicitly opts in via the admin UI gets the Responses surface."""
from turnstone.core.server_compat import _PROFILES
assert "vllm-mistral-medium" in _PROFILES
assert _PROFILES["vllm-mistral-medium"]["server_compat"]["api_surface"] == "responses"
def test_holo_requires_holo2(self) -> None:
"""Short 'holo' prefix shouldn't false-match; 'holo2' should match."""
p_short = suggest_profile("vllm", "some-org/hologram-7b")
@@ -110,32 +131,47 @@ class TestSuggestProfile:
class TestMergeServerCompat:
def test_empty_compat_returns_base_only(self) -> None:
def test_empty_base_and_compat_is_empty(self) -> None:
"""No base, no compat → no extra_body needed."""
assert merge_server_compat(None, {}) == {}
assert merge_server_compat({}, {}) == {}
def test_explicit_base_passes_through(self) -> None:
"""Explicit chat_template_kwargs base is forwarded as-is."""
base = {"reasoning_effort": "medium"}
result = merge_server_compat(base, {})
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
def test_extra_body_merged_top_level(self) -> None:
base = {"reasoning_effort": "medium"}
compat = {"extra_body": {"skip_special_tokens": False}}
result = merge_server_compat(base, compat)
assert result["skip_special_tokens"] is False
assert "chat_template_kwargs" in result
def test_extra_body_merged_top_level_no_base(self) -> None:
"""Server-level overrides forward without a chat_template_kwargs wrapper."""
result = merge_server_compat(None, {"extra_body": {"skip_special_tokens": False}})
assert result == {"skip_special_tokens": False}
def test_full_vllm_gemma_compat(self) -> None:
base = {"reasoning_effort": "medium"}
def test_full_vllm_gemma_compat_no_base(self) -> None:
"""vLLM workaround forwards on its own."""
compat = {
"server_type": "vllm",
"extra_body": {"skip_special_tokens": False},
}
result = merge_server_compat(base, compat)
result = merge_server_compat(None, compat)
assert result == {"skip_special_tokens": False}
def test_operator_chat_template_kwargs_only(self) -> None:
"""Operator can set chat_template_kwargs explicitly without seeding the base."""
compat = {
"extra_body": {
"chat_template_kwargs": {"reasoning_effort": "high"},
"skip_special_tokens": False,
},
}
result = merge_server_compat(None, compat)
assert result == {
"chat_template_kwargs": {"reasoning_effort": "medium"},
"chat_template_kwargs": {"reasoning_effort": "high"},
"skip_special_tokens": False,
}
def test_extra_body_chat_template_kwargs_deep_merged(self) -> None:
"""chat_template_kwargs in extra_body is deep-merged, operator wins."""
def test_extra_body_chat_template_kwargs_deep_merged_with_base(self) -> None:
"""Operator chat_template_kwargs deep-merges over the seeded base."""
base = {"reasoning_effort": "medium"}
compat = {
"extra_body": {
@@ -144,17 +180,15 @@ class TestMergeServerCompat:
},
}
result = merge_server_compat(base, compat)
# Operator values win over base
assert result["chat_template_kwargs"]["custom_flag"] is True
# Operator value wins over seeded base
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
assert result["skip_special_tokens"] is False
def test_extra_body_chat_template_kwargs_non_dict_ignored(self) -> None:
"""Non-dict chat_template_kwargs in extra_body is safely ignored."""
base = {"reasoning_effort": "medium"}
compat = {"extra_body": {"chat_template_kwargs": "bad"}}
result = merge_server_compat(base, compat)
assert result["chat_template_kwargs"] == {"reasoning_effort": "medium"}
assert merge_server_compat(None, compat) == {}
def test_base_not_mutated(self) -> None:
base = {"reasoning_effort": "medium"}
@@ -164,9 +198,7 @@ class TestMergeServerCompat:
def test_non_dict_extra_body_ignored(self) -> None:
"""Gracefully handle malformed server_compat."""
base = {"reasoning_effort": "medium"}
result = merge_server_compat(base, {"extra_body": 42})
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
assert merge_server_compat(None, {"extra_body": 42}) == {}
# ---------------------------------------------------------------------------
@@ -178,45 +210,58 @@ class TestEndToEndRequestShaping:
"""Compose both layers — session builds extra_params, provider applies thinking."""
def test_vllm_gemma_full_flow(self) -> None:
"""Session merges server workarounds, provider adds thinking param."""
"""Session forwards server workarounds, provider adds thinking param."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
base_ctk = {"reasoning_effort": "medium"}
server_compat = {
"server_type": "vllm",
"extra_body": {"skip_special_tokens": False},
}
# Step 1: session merges
extra_params = merge_server_compat(base_ctk, server_compat)
# Step 2: provider finalises
# Step 1: session forwards (no auto-injection of reasoning_effort).
extra_params = merge_server_compat(None, server_compat)
# Step 2: provider injects thinking param into chat_template_kwargs.
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
assert extra_body == {
"chat_template_kwargs": {
"reasoning_effort": "medium",
"enable_thinking": True,
},
"chat_template_kwargs": {"enable_thinking": True},
"skip_special_tokens": False,
}
def test_granite_thinking_key(self) -> None:
"""Granite uses 'thinking' instead of 'enable_thinking'."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="thinking")
extra_params = merge_server_compat({"reasoning_effort": "low"}, {})
extra_params = merge_server_compat(None, {})
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
assert extra_body["chat_template_kwargs"]["thinking"] is True
assert "enable_thinking" not in extra_body["chat_template_kwargs"]
assert extra_body == {"chat_template_kwargs": {"thinking": True}}
def test_non_thinking_model_no_injection(self) -> None:
"""Non-thinking model gets no thinking params."""
"""Non-thinking model gets no chat_template_kwargs at all."""
caps = ModelCapabilities() # thinking_mode="none"
extra_params = merge_server_compat({"reasoning_effort": "medium"}, {})
extra_params = merge_server_compat(None, {})
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
assert extra_body == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
assert extra_body == {}
def test_operator_reasoning_effort_passthrough(self) -> None:
"""Operator-supplied reasoning_effort under chat_template_kwargs is preserved."""
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
compat = {
"server_type": "vllm",
"extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}},
}
extra_params = merge_server_compat(None, compat)
extra_body = dict(extra_params)
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
assert extra_body == {
"chat_template_kwargs": {
"reasoning_effort": "high",
"enable_thinking": True,
},
}
# ---------------------------------------------------------------------------
+44 -66
View File
@@ -1182,7 +1182,7 @@ class TestAgentOutputGuard:
class TestProviderExtraParams:
"""Tests for _provider_extra_params — local-only chat_template_kwargs."""
"""Tests for _provider_extra_params — server_compat passthrough only."""
def _session_with_provider(self, provider_name: str, tmp_db) -> ChatSession:
from turnstone.core.providers import create_provider
@@ -1191,68 +1191,36 @@ class TestProviderExtraParams:
session._provider = create_provider(provider_name)
return session
def test_openai_compatible_returns_chat_template_kwargs(self, tmp_db):
def test_openai_compatible_no_compat_returns_none(self, tmp_db):
"""No server_compat → no extra_body needed (no auto-injection)."""
session = self._session_with_provider("openai-compatible", tmp_db)
result = session._provider_extra_params()
assert result is not None
assert "chat_template_kwargs" in result
assert result["chat_template_kwargs"]["reasoning_effort"] == "medium"
assert session._provider_extra_params() is None
def test_openai_commercial_returns_none(self, tmp_db):
def test_openai_commercial_no_compat_returns_none(self, tmp_db):
"""Cloud OpenAI without server_compat → None."""
session = self._session_with_provider("openai", tmp_db)
result = session._provider_extra_params()
assert result is None
assert session._provider_extra_params() is None
def test_anthropic_returns_none(self, tmp_db):
session = self._session_with_provider("anthropic", tmp_db)
result = session._provider_extra_params()
assert result is None
assert session._provider_extra_params() is None
def test_reasoning_effort_override(self, tmp_db):
def test_no_reasoning_effort_kwarg(self, tmp_db):
"""reasoning_effort is not part of the surface; passing it should TypeError.
Splatted via ``**kwargs`` so static analyzers (CodeQL "wrong-name
argument" / mypy) don't flag the call — the point of this test is the
runtime contract, not the static type.
"""
import pytest
bad_kwargs = {"reasoning_effort": "high"}
session = self._session_with_provider("openai-compatible", tmp_db)
result = session._provider_extra_params(reasoning_effort="high")
assert result is not None
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
with pytest.raises(TypeError):
session._provider_extra_params(**bad_kwargs)
def test_explicit_openai_provider_overrides_session(self, tmp_db):
"""Passing an explicit commercial OpenAI provider returns None even
when the session's own provider is openai-compatible."""
from turnstone.core.providers import create_provider
session = self._session_with_provider("openai-compatible", tmp_db)
openai_prov = create_provider("openai")
result = session._provider_extra_params(provider=openai_prov)
assert result is None
def test_server_compat_extra_body_merged(self, tmp_db):
"""server_compat.extra_body workarounds are merged into extra_params."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
session = self._session_with_provider("openai-compatible", tmp_db)
cfg = ModelConfig(
alias="test",
base_url="http://localhost:8000/v1",
api_key="none",
model="google/gemma-4-31B-it",
server_compat={
"extra_body": {"skip_special_tokens": False},
},
)
session._registry = ModelRegistry(models={"test": cfg}, default="test")
session._model_alias = "test"
result = session._provider_extra_params()
assert result is not None
assert result["chat_template_kwargs"]["reasoning_effort"] == "medium"
assert result["skip_special_tokens"] is False
def test_empty_server_compat_backwards_compatible(self, tmp_db):
"""Empty server_compat produces same output as before."""
session = self._session_with_provider("openai-compatible", tmp_db)
result = session._provider_extra_params()
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
def test_server_compat_with_reasoning_effort_override(self, tmp_db):
"""reasoning_effort override works alongside server_compat."""
def test_server_compat_extra_body_passes_through(self, tmp_db):
"""server_compat.extra_body workarounds forward as extra_params."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
session = self._session_with_provider("openai-compatible", tmp_db)
@@ -1265,10 +1233,25 @@ class TestProviderExtraParams:
)
session._registry = ModelRegistry(models={"test": cfg}, default="test")
session._model_alias = "test"
result = session._provider_extra_params(reasoning_effort="high")
assert result is not None
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
assert result["skip_special_tokens"] is False
result = session._provider_extra_params()
assert result == {"skip_special_tokens": False}
def test_operator_chat_template_kwargs_pass_through(self, tmp_db):
"""Operator-set chat_template_kwargs (e.g. for gpt-oss) forwards verbatim."""
from turnstone.core.model_registry import ModelConfig, ModelRegistry
session = self._session_with_provider("openai-compatible", tmp_db)
cfg = ModelConfig(
alias="test",
base_url="http://localhost:8000/v1",
api_key="none",
model="openai/gpt-oss-120b",
server_compat={"extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}}},
)
session._registry = ModelRegistry(models={"test": cfg}, default="test")
session._model_alias = "test"
result = session._provider_extra_params()
assert result == {"chat_template_kwargs": {"reasoning_effort": "high"}}
def test_model_alias_resolves_target_compat(self, tmp_db):
"""model_alias parameter selects compat from the target, not the primary."""
@@ -1297,14 +1280,9 @@ class TestProviderExtraParams:
session._model_alias = "primary"
# Primary alias → gets Gemma workaround
result_primary = session._provider_extra_params()
assert result_primary is not None
assert result_primary["skip_special_tokens"] is False
# Fallback alias → no compat, just base kwargs
result_fallback = session._provider_extra_params(model_alias="fallback")
assert result_fallback == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
assert "skip_special_tokens" not in result_fallback
assert session._provider_extra_params() == {"skip_special_tokens": False}
# Fallback alias → no compat at all
assert session._provider_extra_params(model_alias="fallback") is None
class TestSafePrepareTool:
+33
View File
@@ -8090,6 +8090,33 @@ _MODEL_PROVIDERS = frozenset({"openai", "anthropic", "openai-compatible", "googl
_REASONING_EFFORT_CHOICES = frozenset(
{"", "none", "minimal", "low", "medium", "high", "xhigh", "max"}
)
# Keep in sync with turnstone.core.providers._VALID_API_SURFACES.
_API_SURFACE_CHOICES = frozenset({"chat", "responses"})
def _validate_api_surface(caps: Any) -> str | None:
"""Return an error message if ``caps["server_compat"]["api_surface"]`` is invalid.
Strict equality match (no strip/lower normalisation): the persisted value
is bound directly to the admin ``<select>`` whose options are the canonical
``"chat"`` / ``"responses"`` strings, so anything else fails to round-trip
through edit/save. The provider factory raises ``ValueError`` at request
time for an unknown surface; validating here turns that into a 400 at
write time so an admin can't poison a model alias via direct API calls.
"""
if not isinstance(caps, dict):
return None
sc = caps.get("server_compat")
if not isinstance(sc, dict):
return None
raw = sc.get("api_surface")
if raw is None or raw == "":
return None
if not isinstance(raw, str) or raw not in _API_SURFACE_CHOICES:
return f"Invalid server_compat.api_surface: {raw!r}"
return None
# Keep in sync with turnstone.core.providers._google.GOOGLE_DEFAULT_BASE_URL
_PROVIDER_DEFAULT_URLS: dict[str, str] = {
"openai": "https://api.openai.com/v1",
@@ -8391,6 +8418,9 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
ctx_raw = body.get("context_window", 32768)
context_window = max(0, int(ctx_raw)) if isinstance(ctx_raw, (int, float)) else 0
caps = body.get("capabilities", {})
err_msg = _validate_api_surface(caps)
if err_msg:
return JSONResponse({"error": err_msg}, status_code=400)
capabilities = json.dumps(caps) if isinstance(caps, dict) else "{}"
enabled = bool(body.get("enabled", True))
@@ -8545,6 +8575,9 @@ async def admin_update_model_definition(request: Request) -> JSONResponse:
updates["context_window"] = max(0, int(ctx_raw)) if isinstance(ctx_raw, (int, float)) else 0
if "capabilities" in body:
caps = body["capabilities"]
err_msg = _validate_api_surface(caps)
if err_msg:
return JSONResponse({"error": err_msg}, status_code=400)
updates["capabilities"] = json.dumps(caps) if isinstance(caps, dict) else "{}"
if "enabled" in body:
updates["enabled"] = bool(body["enabled"])
+33 -16
View File
@@ -5001,6 +5001,7 @@ function showCreateModelModal() {
document.getElementById("model-max-tokens").value = "";
document.getElementById("model-reasoning-effort").value = "";
document.getElementById("model-server-type").value = "";
document.getElementById("model-api-surface").value = "";
document.getElementById("model-thinking-mode").value = "";
document.getElementById("model-thinking-param").value = "";
document.getElementById("model-thinking-param-row").style.display = "none";
@@ -5077,8 +5078,9 @@ function showEditModelModal(definitionId) {
document.getElementById("model-thinking-param").value = "";
}
_toggleThinkingParam();
// Server compat: server_type and extra_body workarounds
// Server compat: server_type, api_surface, and extra_body workarounds
document.getElementById("model-server-type").value = sc.server_type || "";
document.getElementById("model-api-surface").value = sc.api_surface || "";
var eb = sc.extra_body || {};
var ebText = JSON.stringify(eb, null, 2);
document.getElementById("model-extra-body").value =
@@ -5160,26 +5162,34 @@ function submitCreateModel() {
if (savedParam) caps.thinking_param = savedParam;
}
// Build server_compat from structured fields
// Build server_compat from structured fields. Only meaningful for
// openai-compatible aliases — for other providers the section is hidden
// but the form values can linger after a provider switch, so gate the
// whole block on the active provider to keep persisted state honest.
var serverCompat = {};
var serverType = document.getElementById("model-server-type").value;
if (serverType) serverCompat.server_type = serverType;
var providerVal = document.getElementById("model-provider").value;
var ebEl = document.getElementById("model-extra-body");
var ebText = ebEl.value.trim();
ebEl.removeAttribute("aria-invalid");
ebEl.style.borderColor = "";
if (ebText) {
try {
var ebParsed = JSON.parse(ebText);
if (!_isPlainObject(ebParsed)) {
throw new Error("not an object");
if (providerVal === "openai-compatible") {
var serverType = document.getElementById("model-server-type").value;
if (serverType) serverCompat.server_type = serverType;
var apiSurface = document.getElementById("model-api-surface").value;
if (apiSurface) serverCompat.api_surface = apiSurface;
var ebText = ebEl.value.trim();
if (ebText) {
try {
var ebParsed = JSON.parse(ebText);
if (!_isPlainObject(ebParsed)) {
throw new Error("not an object");
}
serverCompat.extra_body = ebParsed;
} catch (e) {
ebEl.setAttribute("aria-invalid", "true");
ebEl.style.borderColor = "var(--red)";
_showModelError("Extra body params must be a JSON object");
return;
}
serverCompat.extra_body = ebParsed;
} catch (e) {
ebEl.setAttribute("aria-invalid", "true");
ebEl.style.borderColor = "var(--red)";
_showModelError("Extra body params must be a JSON object");
return;
}
}
if (Object.keys(serverCompat).length > 0) {
@@ -5410,6 +5420,13 @@ function detectModel() {
stOpts2.indexOf(ssc.server_type) !== -1
)
stEl2.value = ssc.server_type;
// Restrict to the known set so a hostile detect response can't
// smuggle a non-listed value into the form.
var _SURFACE_SUGGESTABLE = { chat: 1, responses: 1 };
if (ssc.api_surface && _SURFACE_SUGGESTABLE[ssc.api_surface]) {
var asEl = document.getElementById("model-api-surface");
if (!asEl.value) asEl.value = ssc.api_surface;
}
if (ssc.extra_body) {
var ebEl2 = document.getElementById("model-extra-body");
if (!ebEl2.value.trim()) {
+11
View File
@@ -3920,6 +3920,17 @@
<option value="llama.cpp">llama.cpp</option>
<option value="openai-compatible">Other OpenAI-compatible</option>
</select>
<label for="model-api-surface"
>API Surface
<span style="font-weight: 400; text-transform: none"
>(Chat Completions vs Responses)</span
></label
>
<select id="model-api-surface">
<option value="">Inherit (Chat Completions)</option>
<option value="chat">Chat Completions (pinned)</option>
<option value="responses">Responses API</option>
</select>
<label for="model-thinking-mode"
>Thinking Mode
<span style="font-weight: 400; text-transform: none"
+25 -5
View File
@@ -44,6 +44,19 @@ class ModelConfig:
server_compat: dict[str, Any] = field(default_factory=dict)
def _api_surface_of(cfg: ModelConfig) -> str | None:
"""Extract the operator-pinned api_surface from *cfg*, or ``None``.
Used both at provider-cache lookup time and at reload-eviction time so the
two sites stay in sync. Returns ``None`` when the field is absent, blank,
or not a string matching the "inherit provider default" semantics.
"""
raw = cfg.server_compat.get("api_surface") if isinstance(cfg.server_compat, dict) else None
if isinstance(raw, str) and raw.strip():
return raw
return None
# ---------------------------------------------------------------------------
# Registry
# ---------------------------------------------------------------------------
@@ -127,7 +140,9 @@ class ModelRegistry:
raise ValueError(f"Unknown model alias: {alias}")
if alias not in self._providers:
cfg = self._models[alias]
self._providers[alias] = create_provider(cfg.provider)
self._providers[alias] = create_provider(
cfg.provider, api_surface=_api_surface_of(cfg)
)
return self._providers[alias]
def get_config(self, alias: str) -> ModelConfig:
@@ -257,13 +272,18 @@ class ModelRegistry:
if hasattr(client, "close"):
client.close()
del self._clients[alias]
# Providers are keyed on alias but only depend on
# ``cfg.provider`` — drop only when the provider string
# changed or the alias was removed.
# Providers are keyed on alias and depend on (cfg.provider,
# cfg.server_compat["api_surface"]) — drop when either changes
# or the alias was removed.
for alias in list(self._providers.keys()):
old_cfg = old_models.get(alias)
new_cfg = self._models.get(alias)
if new_cfg is None or old_cfg is None or old_cfg.provider != new_cfg.provider:
if (
new_cfg is None
or old_cfg is None
or old_cfg.provider != new_cfg.provider
or _api_surface_of(old_cfg) != _api_surface_of(new_cfg)
):
del self._providers[alias]
def shutdown(self) -> None:
+38 -3
View File
@@ -33,7 +33,9 @@ __all__ = [
"lookup_model_capabilities",
]
# Singleton instances (stateless, safe to share)
# Singleton instances (stateless, safe to share). ``_openai_provider``
# is reused for both cloud OpenAI and ``openai-compatible`` with
# ``api_surface="responses"`` — see the ``create_provider`` docstring.
_provider_lock = threading.Lock()
_openai_provider = OpenAIResponsesProvider()
_openai_compat_provider = OpenAIChatCompletionsProvider()
@@ -41,12 +43,45 @@ _anthropic_provider: LLMProvider | None = None
_google_provider: LLMProvider | None = None
def create_provider(provider_name: str) -> LLMProvider:
"""Return a provider adapter for the given provider name. Thread-safe."""
_VALID_API_SURFACES = ("chat", "responses")
def create_provider(
provider_name: str,
*,
api_surface: str | None = None,
) -> LLMProvider:
"""Return a provider adapter for the given provider name. Thread-safe.
*api_surface* selects the OpenAI-compatible API surface for
``provider_name="openai-compatible"``:
- ``"chat"`` (default) Chat Completions (vLLM, llama.cpp, SGLang).
- ``"responses"`` Responses API (commercial OpenAI-compat
endpoints like Mistral cloud, or local servers that expose the
Responses surface).
Ignored for non-OpenAI providers. ``provider_name="openai"`` always
uses the Responses API regardless of *api_surface*.
Note: the ``OpenAIResponsesProvider`` singleton is reused for both
cloud OpenAI and ``openai-compatible`` + responses, so its
``provider_name`` reports ``"openai"`` even when serving an
openai-compatible config. Code that needs to distinguish the two
must read ``ModelConfig.provider`` and ``server_compat["api_surface"]``
rather than ``provider.provider_name``.
"""
global _anthropic_provider, _google_provider # noqa: PLW0603
if provider_name == "openai":
return _openai_provider
if provider_name == "openai-compatible":
normalised = (api_surface or "").strip().lower()
if normalised and normalised not in _VALID_API_SURFACES:
raise ValueError(
f"Unknown api_surface: {api_surface!r}. Supported: {', '.join(_VALID_API_SURFACES)}"
)
if normalised == "responses":
return _openai_provider
return _openai_compat_provider
if provider_name == "anthropic":
with _provider_lock:
+44 -10
View File
@@ -1,14 +1,20 @@
"""Server compatibility profiles for OpenAI-compatible backends.
Different local model servers (vLLM, llama.cpp, SGLang) need different
request shaping. This module separates two concerns:
request shaping. This module separates three concerns:
1. **Model capabilities** ``thinking_mode`` and ``thinking_param`` are
properties of the *model* (Gemma thinks, Llama doesn't). These go
into the ``capabilities`` dict and flow through ``ModelCapabilities``
so the provider can act on them (just like Anthropic's thinking mode).
2. **Server workarounds** ``extra_body`` overrides like
2. **API surface** ``api_surface`` selects which OpenAI-compatible
API surface the provider talks to: ``"chat"`` (Chat Completions,
the default) or ``"responses"`` (Responses API, native reasoning).
Stored under ``server_compat`` because it's an endpoint property,
not a model property.
3. **Server workarounds** ``extra_body`` overrides like
``skip_special_tokens=false`` are properties of the *server* (vLLM
bug workaround). These stay in ``server_compat`` and get merged
into the request's ``extra_body`` at call time.
@@ -82,6 +88,23 @@ _PROFILES: dict[str, dict[str, Any]] = {
"server_type": "vllm",
},
},
"vllm-mistral-medium": {
# Mistral medium open-weights served by vLLM can deliver reasoning
# via either surface, but the trade-off is asymmetric:
# * Chat Completions — tool calling works (``--tool-call-parser
# mistral``); reasoning is enabled via the vLLM CLI
# (``--reasoning-parser``) rather than per-request.
# * Responses API — reasoning effort is per-request and clean,
# but as of vLLM 0.x the tool-call parser is not wired up on
# this surface so tool calls leak as ``[TOOL_CALLS]`` text.
# We do **not** auto-suggest this profile from Detect; an operator
# who needs per-request effort and accepts the tool-calling
# limitation can pick "Responses API" manually in the admin UI.
"server_compat": {
"server_type": "vllm",
"api_surface": "responses",
},
},
"vllm": {
"server_compat": {
"server_type": "vllm",
@@ -125,6 +148,9 @@ _VLLM_MODEL_PROFILES: list[tuple[str, str]] = [
("granite3", "vllm-granite-thinking"),
("deepseek-r1", "vllm-deepseek-thinking"),
("holo2", "vllm-holo-thinking"),
# Mistral medium intentionally omitted — see ``vllm-mistral-medium``
# profile docstring for the Chat-vs-Responses trade-off; operator
# picks manually rather than letting Detect auto-suggest Responses.
]
# llama.cpp model-family → profile key mapping.
@@ -175,31 +201,39 @@ def suggest_profile(server_type: str, model_id: str) -> dict[str, Any]:
def merge_server_compat(
base_chat_template_kwargs: dict[str, Any],
base_chat_template_kwargs: dict[str, Any] | None,
server_compat: dict[str, Any],
) -> dict[str, Any]:
"""Build the ``extra_body`` dict by merging server compat into base kwargs.
*base_chat_template_kwargs* always contains at least ``reasoning_effort``.
*server_compat* comes from ``ModelConfig.server_compat``.
*base_chat_template_kwargs* is an explicit ``chat_template_kwargs`` dict
to seed the request with, or ``None``/empty to skip seeding. Operator-
supplied entries in ``server_compat["extra_body"]["chat_template_kwargs"]``
are deep-merged on top. Top-level ``extra_body`` keys (``skip_special_tokens``,
``reasoning_format``, etc.) are forwarded as-is.
Note: thinking-mode params (``enable_thinking``, ``thinking``) are **not**
merged here the provider handles those via ``ModelCapabilities``.
This function only merges server workarounds from ``extra_body``.
This function only merges what the operator stored in ``server_compat``.
Returns the complete dict to pass as ``extra_body`` to the OpenAI client.
May be empty when there is nothing to send.
"""
extra: dict[str, Any] = {"chat_template_kwargs": dict(base_chat_template_kwargs)}
extra: dict[str, Any] = {}
if base_chat_template_kwargs:
extra["chat_template_kwargs"] = dict(base_chat_template_kwargs)
# Merge top-level extra_body overrides (skip_special_tokens, etc.)
compat_eb = server_compat.get("extra_body")
if isinstance(compat_eb, dict):
for key, value in compat_eb.items():
if key == "chat_template_kwargs":
# Deep-merge: operator values in extra_body win over the
# base dict (which has reasoning_effort). This lets
# operators intentionally extend chat_template_kwargs.
# Deep-merge with operator values winning so an operator
# can intentionally extend chat_template_kwargs (e.g. set
# ``reasoning_effort`` for gpt-oss-style local templates).
if isinstance(value, dict):
if "chat_template_kwargs" not in extra:
extra["chat_template_kwargs"] = {}
extra["chat_template_kwargs"].update(value)
continue
extra[key] = value
+26 -28
View File
@@ -1458,11 +1458,6 @@ class ChatSession:
"""
new_system_messages: list[dict[str, Any]] = []
# -- Chat template kwargs --
self._chat_template_kwargs_base: dict[str, Any] = {
"reasoning_effort": self.reasoning_effort,
}
# -- Developer message --
if self.creative_mode:
dev_parts = [
@@ -1844,48 +1839,48 @@ class ChatSession:
def _provider_extra_params(
self,
reasoning_effort: str | None = None,
provider: LLMProvider | None = None,
model_alias: str | None = None,
) -> dict[str, Any] | None:
"""Build provider-specific extra parameters.
``chat_template_kwargs`` is only meaningful for local model servers
(``openai-compatible``). Commercial OpenAI rejects it as an unknown
parameter, and handles ``reasoning_effort`` natively.
Forwards operator-supplied ``server_compat["extra_body"]`` overrides
(``skip_special_tokens``, ``reasoning_format``, or explicit
``chat_template_kwargs``) to the OpenAI SDK ``extra_body``. Operators
running gpt-oss-style local templates that consume ``reasoning_effort``
from ``chat_template_kwargs`` should set it explicitly under
``server_compat["extra_body"]["chat_template_kwargs"]``.
Merges server workarounds (``skip_special_tokens``, etc.) from
``ModelConfig.server_compat`` into the request's ``extra_body``.
Thinking-mode params (``enable_thinking``) are handled separately
by the provider based on ``ModelCapabilities.thinking_mode``.
Thinking-mode params (``enable_thinking``, ``thinking``) are added
separately by ``OpenAIChatCompletionsProvider._apply_thinking_mode``
based on ``ModelCapabilities.thinking_mode`` the Responses API
surface handles reasoning natively and ignores ``extra_body``.
*model_alias* controls which model config supplies server compat
*model_alias* selects which stored config supplies server compat
settings. When ``None``, defaults to the session's primary alias.
"""
from turnstone.core.server_compat import merge_server_compat
prov = provider or self._provider
if prov.provider_name == "openai-compatible":
ctk_base = dict(self._chat_template_kwargs_base)
if reasoning_effort:
ctk_base["reasoning_effort"] = reasoning_effort
return merge_server_compat(
ctk_base,
self._get_server_compat(model_alias),
)
return None
# Only OpenAI-shaped providers consume extra_body. Anthropic/Google
# have their own param paths handled inside their providers.
if prov.provider_name not in ("openai", "openai-compatible"):
return None
extra = merge_server_compat(None, self._get_server_compat(model_alias))
return extra or None
def _get_server_compat(self, model_alias: str | None = None) -> dict[str, Any]:
"""Get server compatibility settings from a model config.
*model_alias* selects the config to read. Falls back to the
session's primary alias when ``None``.
session's primary alias when ``None``. The returned dict is the
live ``ModelConfig.server_compat`` reference callers must not
mutate it. ``merge_server_compat`` reads only.
"""
alias = model_alias or self._model_alias
if self._registry and alias:
try:
cfg = self._registry.get_config(alias)
return dict(cfg.server_compat)
return self._registry.get_config(alias).server_compat
except (ValueError, KeyError):
pass
return {}
@@ -1914,7 +1909,7 @@ class ChatSession:
max_tokens=clamped,
temperature=temperature,
reasoning_effort=reasoning_effort,
extra_params=self._provider_extra_params(reasoning_effort=reasoning_effort),
extra_params=self._provider_extra_params(),
capabilities=caps,
)
@@ -8037,6 +8032,10 @@ class ChatSession:
agent_client = self.client
agent_model = self.model
agent_provider = self._provider
# When falling through to the session's primary model, use the
# session's primary alias for capability and server_compat
# resolution so the agent sees the same caps as the main loop.
agent_alias = self._model_alias
# Per-kind reasoning effort. Explicit caller arg wins; otherwise
# delegate to the registry which knows the per-kind default (plan
@@ -8059,7 +8058,6 @@ class ChatSession:
# Build extra params for agent calls — resolve server compat from the
# agent's own model alias, not the session's primary model.
agent_extra = self._provider_extra_params(
reasoning_effort=reasoning_effort,
provider=agent_provider,
model_alias=agent_alias,
)