mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
feat(providers): api_surface toggle + mistral medium reasoning fix (#469)
* feat(providers): api_surface toggle + mistral medium reasoning fix
Mistral medium open-weights served by vLLM expects reasoning_effort via
the Responses API (`reasoning.effort`), not as a `chat_template_kwargs`
entry on Chat Completions. The session was unconditionally injecting
`{"reasoning_effort": ...}` into `chat_template_kwargs` for every
openai-compatible request, which corrupted the prompt rendering for any
backend whose chat template didn't consume that key (Mistral medium,
Mistral cloud, Groq, OpenRouter).
Changes:
- Add `api_surface` ("chat" | "responses") to `ModelConfig.server_compat`
and thread it through `create_provider` / `model_registry.get_provider`.
`openai-compatible` defaults to Chat Completions; operators can flip
individual aliases to Responses for endpoints that support it.
- New `vllm-mistral-medium` profile that pre-fills api_surface=responses
on Detect for known Mistral medium model ids.
- Drop the unconditional `reasoning_effort` injection into
`chat_template_kwargs`. Operators running gpt-oss-style local
templates that consume `reasoning_effort` from the chat template now
opt in via `server_compat.extra_body.chat_template_kwargs`.
- New "API Surface" select in the Models admin tab; allowlist-validated
server-side at create/update time; pre-filled by Detect via the
profile suggestion.
- Evict the cached provider singleton in `ModelRegistry.reload()` when
api_surface changes (previously only cfg.provider triggered eviction).
- Fix `_run_agent` fallback path to inherit the session's primary alias
for capability and server_compat resolution; previously the fallback
passed `alias=None`, which silently dropped per-model caps on the
agent path.
Tests: 5117 passed (-m "not live"); ruff + mypy clean.
* fix(providers): don't auto-suggest Responses for Mistral medium
vLLM's Responses API surface for Mistral medium open-weights doesn't
wire up the Mistral tool-call parser as of vLLM 0.x — tool calls leak
into the response as ``[TOOL_CALLS]<name>{...}`` text instead of
structured tool_calls. Chat Completions on the same engine handles
tools cleanly via ``--tool-call-parser mistral``, and reasoning can be
turned on via the vLLM CLI ``--reasoning-parser`` flag.
Drop the auto-suggest mapping so Detect falls back to the generic
``vllm`` profile. Keep the ``vllm-mistral-medium`` profile definition
in place so an operator who specifically wants per-request effort and
accepts the tool-calling limitation can still pick "Responses API"
manually in the admin UI.
* fix(providers): address Copilot review on PR #469
- providers/__init__.py: drop the redundant *_responses_provider /
*_chat_provider names; have create_provider use _openai_provider and
_openai_compat_provider directly so they're not flagged as unused
globals.
- console/server.py: tighten _validate_api_surface to a strict equality
match against the canonical {"chat", "responses"} set. The previous
strip().lower() membership check accepted ' Responses '/'CHAT' but
stored the raw string verbatim, which then failed to round-trip
through the admin <select>.
- console/static/admin.js: gate the entire server_compat block (server
type, api_surface, extra_body) on provider == "openai-compatible" at
save time so toggling provider away can't leave a stale hidden surface
selection in the persisted capabilities JSON.
- tests/test_session.py: splat the bad kwarg via **dict so CodeQL no
longer flags the call as a wrong-name keyword (the point of the test
is the runtime contract, not the static type).
- tests/test_admin_model_registry_refresh.py: add endpoint-level tests
for the api_surface validation on both create and update — covers the
bogus-value rejection, non-canonical-string rejection, and the happy
path persisting through to the refreshed registry.
This commit is contained in:
committed by
Patrick Buckley
parent
2fd0c29a92
commit
ec74334e74
@@ -352,6 +352,90 @@ def test_update_endpoint_skips_refresh_on_empty_body(
|
||||
assert calls == [] # gate held: empty body did not trigger a refresh
|
||||
|
||||
|
||||
def test_create_rejects_invalid_api_surface(storage: SQLiteBackend) -> None:
|
||||
"""POST with a bogus server_compat.api_surface returns 400 rather than
|
||||
persisting a value that would make get_provider() raise on every later
|
||||
ChatSession init for the alias."""
|
||||
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
|
||||
registry = _make_registry(alias="local", model="m")
|
||||
client = _make_client(storage, registry)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/admin/model-definitions",
|
||||
json={
|
||||
"alias": "bad",
|
||||
"model": "x",
|
||||
"provider": "openai-compatible",
|
||||
"base_url": "http://localhost:9000/v1",
|
||||
"api_key": "sk-x",
|
||||
"capabilities": {"server_compat": {"api_surface": "BOGUS"}},
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
assert "api_surface" in resp.json()["error"]
|
||||
# And the alias is not persisted
|
||||
assert not registry.has_alias("bad")
|
||||
|
||||
|
||||
def test_create_rejects_non_canonical_api_surface(storage: SQLiteBackend) -> None:
|
||||
"""Strict validation: ' Responses ' / 'CHAT' don't round-trip through the
|
||||
admin <select>, so they're rejected even though they'd survive a
|
||||
case-insensitive membership check."""
|
||||
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
|
||||
registry = _make_registry(alias="local", model="m")
|
||||
client = _make_client(storage, registry)
|
||||
|
||||
for bad in (" responses ", "RESPONSES", "Chat"):
|
||||
resp = client.post(
|
||||
"/v1/api/admin/model-definitions",
|
||||
json={
|
||||
"alias": "noncanon",
|
||||
"model": "x",
|
||||
"provider": "openai-compatible",
|
||||
"base_url": "http://localhost:9000/v1",
|
||||
"api_key": "sk-x",
|
||||
"capabilities": {"server_compat": {"api_surface": bad}},
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 400, f"{bad!r}: {resp.text}"
|
||||
|
||||
|
||||
def test_create_accepts_valid_api_surface(storage: SQLiteBackend) -> None:
|
||||
"""Canonical 'chat' / 'responses' / unset are all accepted and persisted."""
|
||||
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
|
||||
registry = _make_registry(alias="local", model="m")
|
||||
client = _make_client(storage, registry)
|
||||
|
||||
resp = client.post(
|
||||
"/v1/api/admin/model-definitions",
|
||||
json={
|
||||
"alias": "responses-alias",
|
||||
"model": "x",
|
||||
"provider": "openai-compatible",
|
||||
"base_url": "http://localhost:9000/v1",
|
||||
"api_key": "sk-x",
|
||||
"capabilities": {"server_compat": {"api_surface": "responses"}},
|
||||
},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
assert registry.has_alias("responses-alias")
|
||||
|
||||
|
||||
def test_update_rejects_invalid_api_surface(storage: SQLiteBackend) -> None:
|
||||
"""PUT path also gates the validation, so an admin can't smuggle a bad
|
||||
value into an existing alias."""
|
||||
_seed_model_def(storage, definition_id="m1", alias="local", model="m")
|
||||
registry = _make_registry(alias="local", model="m")
|
||||
client = _make_client(storage, registry)
|
||||
|
||||
resp = client.put(
|
||||
"/v1/api/admin/model-definitions/m1",
|
||||
json={"capabilities": {"server_compat": {"api_surface": "junk"}}},
|
||||
)
|
||||
assert resp.status_code == 400, resp.text
|
||||
assert "api_surface" in resp.json()["error"]
|
||||
|
||||
|
||||
def test_delete_endpoint_refreshes_registry(storage: SQLiteBackend) -> None:
|
||||
"""DELETE drops the alias from the in-process registry too — a
|
||||
coord session that tried to resolve the deleted alias would
|
||||
|
||||
@@ -1126,24 +1126,57 @@ class TestSessionAgentModel:
|
||||
def _captured_effort(captured: dict[str, Any]) -> str | None:
|
||||
"""Pull reasoning_effort out of provider-specific shapes.
|
||||
|
||||
openai-compatible servers receive it via extra_body.chat_template_kwargs;
|
||||
commercial providers receive it as a top-level kwarg.
|
||||
Chat Completions delivers it as a top-level ``reasoning_effort`` kwarg
|
||||
(when the model's caps permit it). Operators who route reasoning_effort
|
||||
through ``chat_template_kwargs`` (gpt-oss-style local templates) get
|
||||
it inside ``extra_body.chat_template_kwargs``.
|
||||
"""
|
||||
if "reasoning_effort" in captured:
|
||||
return captured["reasoning_effort"]
|
||||
eb = captured.get("extra_body") or {}
|
||||
ctk = eb.get("chat_template_kwargs") or {}
|
||||
return ctk.get("reasoning_effort") or captured.get("reasoning_effort")
|
||||
return ctk.get("reasoning_effort")
|
||||
|
||||
@staticmethod
|
||||
def _effort_caps() -> dict[str, Any]:
|
||||
"""Capabilities that allow Chat-Completions reasoning_effort to flow."""
|
||||
return {
|
||||
"reasoning_effort_values": [
|
||||
"minimal",
|
||||
"low",
|
||||
"medium",
|
||||
"high",
|
||||
"max",
|
||||
],
|
||||
}
|
||||
|
||||
def _three_model_registry(self, **kwargs: Any) -> ModelRegistry:
|
||||
caps = self._effort_caps()
|
||||
return ModelRegistry(
|
||||
models={
|
||||
"main": ModelConfig(
|
||||
"main", "http://m/v1", "k", "main-model", provider="openai-compatible"
|
||||
"main",
|
||||
"http://m/v1",
|
||||
"k",
|
||||
"main-model",
|
||||
provider="openai-compatible",
|
||||
capabilities=dict(caps),
|
||||
),
|
||||
"smart": ModelConfig(
|
||||
"smart", "http://s/v1", "k", "smart-model", provider="openai-compatible"
|
||||
"smart",
|
||||
"http://s/v1",
|
||||
"k",
|
||||
"smart-model",
|
||||
provider="openai-compatible",
|
||||
capabilities=dict(caps),
|
||||
),
|
||||
"fast": ModelConfig(
|
||||
"fast", "http://f/v1", "k", "fast-model", provider="openai-compatible"
|
||||
"fast",
|
||||
"http://f/v1",
|
||||
"k",
|
||||
"fast-model",
|
||||
provider="openai-compatible",
|
||||
capabilities=dict(caps),
|
||||
),
|
||||
},
|
||||
default="main",
|
||||
@@ -1249,6 +1282,45 @@ class TestSessionAgentModel:
|
||||
session._run_agent([{"role": "user", "content": "x"}], label="plan", agent_alias="fast")
|
||||
assert captured["model"] == "fast-model"
|
||||
|
||||
def test_session_fallback_inherits_primary_alias_for_caps(self) -> None:
|
||||
"""When _run_agent has no registry agent route, it must fall back to
|
||||
the session's primary alias for capability and server_compat lookup —
|
||||
otherwise per-model caps (reasoning_effort_values, server_compat) get
|
||||
silently dropped on the agent path."""
|
||||
reg = self._three_model_registry() # no agent_model / plan_model set
|
||||
session = _make_session(registry=reg, model_alias="main")
|
||||
# Probe what _run_agent passes to _provider_extra_params and
|
||||
# _resolve_capabilities by recording the model_alias on each call.
|
||||
captured_extra_alias: list[str | None] = []
|
||||
captured_resolve_alias: list[str | None] = []
|
||||
original_extra = session._provider_extra_params
|
||||
original_resolve = session._resolve_capabilities
|
||||
|
||||
def spy_extra(*args: Any, **kwargs: Any) -> Any:
|
||||
captured_extra_alias.append(kwargs.get("model_alias"))
|
||||
return original_extra(*args, **kwargs)
|
||||
|
||||
def spy_resolve(*args: Any, **kwargs: Any) -> Any:
|
||||
# _resolve_capabilities(provider, model, alias)
|
||||
alias = args[2] if len(args) >= 3 else kwargs.get("alias")
|
||||
captured_resolve_alias.append(alias)
|
||||
return original_resolve(*args, **kwargs)
|
||||
|
||||
session._provider_extra_params = spy_extra # type: ignore[method-assign]
|
||||
session._resolve_capabilities = spy_resolve # type: ignore[method-assign]
|
||||
|
||||
self._capture_on(session.client) # patch client.chat.completions.create
|
||||
session._run_agent([{"role": "user", "content": "x"}], label="plan")
|
||||
|
||||
assert captured_extra_alias and captured_extra_alias[-1] == "main", (
|
||||
f"agent fallback path did not inherit primary alias for extra_params: "
|
||||
f"{captured_extra_alias!r}"
|
||||
)
|
||||
assert captured_resolve_alias and captured_resolve_alias[-1] == "main", (
|
||||
f"agent fallback path did not inherit primary alias for caps: "
|
||||
f"{captured_resolve_alias!r}"
|
||||
)
|
||||
|
||||
def test_invalid_alias_raises_in_run_agent(self) -> None:
|
||||
"""Defence-in-depth: _prepare_* validates first, but _run_agent
|
||||
rejects unknown aliases too rather than silently falling back."""
|
||||
|
||||
@@ -1383,6 +1383,34 @@ class TestProviderFactory:
|
||||
p2 = create_provider("openai")
|
||||
assert p1 is p2
|
||||
|
||||
def test_create_provider_compat_responses_surface(self) -> None:
|
||||
"""openai-compatible + api_surface=responses returns the Responses provider."""
|
||||
from turnstone.core.providers import OpenAIResponsesProvider, create_provider
|
||||
|
||||
provider = create_provider("openai-compatible", api_surface="responses")
|
||||
assert isinstance(provider, OpenAIResponsesProvider)
|
||||
|
||||
def test_create_provider_compat_chat_surface_default(self) -> None:
|
||||
"""openai-compatible defaults to Chat Completions."""
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
for surface in (None, "", "chat"):
|
||||
provider = create_provider("openai-compatible", api_surface=surface)
|
||||
assert isinstance(provider, OpenAIChatCompletionsProvider)
|
||||
|
||||
def test_create_provider_invalid_api_surface(self) -> None:
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
with pytest.raises(ValueError, match="Unknown api_surface"):
|
||||
create_provider("openai-compatible", api_surface="bogus")
|
||||
|
||||
def test_create_provider_openai_ignores_api_surface(self) -> None:
|
||||
"""Cloud OpenAI is always Responses regardless of api_surface."""
|
||||
from turnstone.core.providers import OpenAIResponsesProvider, create_provider
|
||||
|
||||
provider = create_provider("openai", api_surface="chat")
|
||||
assert isinstance(provider, OpenAIResponsesProvider)
|
||||
|
||||
# -- Google provider -------------------------------------------------------
|
||||
|
||||
def test_create_provider_google(self) -> None:
|
||||
|
||||
+80
-35
@@ -89,6 +89,27 @@ class TestSuggestProfile:
|
||||
p = suggest_profile("vllm", "Google/GEMMA-4-31B-IT")
|
||||
assert p["capabilities"]["thinking_mode"] == "manual"
|
||||
|
||||
def test_vllm_mistral_medium_not_auto_suggested(self) -> None:
|
||||
"""Mistral medium falls back to the generic vLLM profile.
|
||||
|
||||
We don't auto-suggest the Responses surface for Mistral medium because
|
||||
vLLM's Responses API tool-call parser isn't wired up for it yet —
|
||||
operators who want per-request reasoning effort must pick "Responses
|
||||
API" manually in the admin UI and accept the tool-calling limitation.
|
||||
"""
|
||||
p = suggest_profile("vllm", "mistralai/Mistral-Medium-3-Instruct")
|
||||
assert p["server_compat"]["server_type"] == "vllm"
|
||||
assert "api_surface" not in p["server_compat"]
|
||||
assert "capabilities" not in p
|
||||
|
||||
def test_vllm_mistral_medium_profile_still_available(self) -> None:
|
||||
"""The vllm-mistral-medium profile remains in _PROFILES so an operator
|
||||
who explicitly opts in via the admin UI gets the Responses surface."""
|
||||
from turnstone.core.server_compat import _PROFILES
|
||||
|
||||
assert "vllm-mistral-medium" in _PROFILES
|
||||
assert _PROFILES["vllm-mistral-medium"]["server_compat"]["api_surface"] == "responses"
|
||||
|
||||
def test_holo_requires_holo2(self) -> None:
|
||||
"""Short 'holo' prefix shouldn't false-match; 'holo2' should match."""
|
||||
p_short = suggest_profile("vllm", "some-org/hologram-7b")
|
||||
@@ -110,32 +131,47 @@ class TestSuggestProfile:
|
||||
|
||||
|
||||
class TestMergeServerCompat:
|
||||
def test_empty_compat_returns_base_only(self) -> None:
|
||||
def test_empty_base_and_compat_is_empty(self) -> None:
|
||||
"""No base, no compat → no extra_body needed."""
|
||||
assert merge_server_compat(None, {}) == {}
|
||||
assert merge_server_compat({}, {}) == {}
|
||||
|
||||
def test_explicit_base_passes_through(self) -> None:
|
||||
"""Explicit chat_template_kwargs base is forwarded as-is."""
|
||||
base = {"reasoning_effort": "medium"}
|
||||
result = merge_server_compat(base, {})
|
||||
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
|
||||
|
||||
def test_extra_body_merged_top_level(self) -> None:
|
||||
base = {"reasoning_effort": "medium"}
|
||||
compat = {"extra_body": {"skip_special_tokens": False}}
|
||||
result = merge_server_compat(base, compat)
|
||||
assert result["skip_special_tokens"] is False
|
||||
assert "chat_template_kwargs" in result
|
||||
def test_extra_body_merged_top_level_no_base(self) -> None:
|
||||
"""Server-level overrides forward without a chat_template_kwargs wrapper."""
|
||||
result = merge_server_compat(None, {"extra_body": {"skip_special_tokens": False}})
|
||||
assert result == {"skip_special_tokens": False}
|
||||
|
||||
def test_full_vllm_gemma_compat(self) -> None:
|
||||
base = {"reasoning_effort": "medium"}
|
||||
def test_full_vllm_gemma_compat_no_base(self) -> None:
|
||||
"""vLLM workaround forwards on its own."""
|
||||
compat = {
|
||||
"server_type": "vllm",
|
||||
"extra_body": {"skip_special_tokens": False},
|
||||
}
|
||||
result = merge_server_compat(base, compat)
|
||||
result = merge_server_compat(None, compat)
|
||||
assert result == {"skip_special_tokens": False}
|
||||
|
||||
def test_operator_chat_template_kwargs_only(self) -> None:
|
||||
"""Operator can set chat_template_kwargs explicitly without seeding the base."""
|
||||
compat = {
|
||||
"extra_body": {
|
||||
"chat_template_kwargs": {"reasoning_effort": "high"},
|
||||
"skip_special_tokens": False,
|
||||
},
|
||||
}
|
||||
result = merge_server_compat(None, compat)
|
||||
assert result == {
|
||||
"chat_template_kwargs": {"reasoning_effort": "medium"},
|
||||
"chat_template_kwargs": {"reasoning_effort": "high"},
|
||||
"skip_special_tokens": False,
|
||||
}
|
||||
|
||||
def test_extra_body_chat_template_kwargs_deep_merged(self) -> None:
|
||||
"""chat_template_kwargs in extra_body is deep-merged, operator wins."""
|
||||
def test_extra_body_chat_template_kwargs_deep_merged_with_base(self) -> None:
|
||||
"""Operator chat_template_kwargs deep-merges over the seeded base."""
|
||||
base = {"reasoning_effort": "medium"}
|
||||
compat = {
|
||||
"extra_body": {
|
||||
@@ -144,17 +180,15 @@ class TestMergeServerCompat:
|
||||
},
|
||||
}
|
||||
result = merge_server_compat(base, compat)
|
||||
# Operator values win over base
|
||||
assert result["chat_template_kwargs"]["custom_flag"] is True
|
||||
# Operator value wins over seeded base
|
||||
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
|
||||
assert result["skip_special_tokens"] is False
|
||||
|
||||
def test_extra_body_chat_template_kwargs_non_dict_ignored(self) -> None:
|
||||
"""Non-dict chat_template_kwargs in extra_body is safely ignored."""
|
||||
base = {"reasoning_effort": "medium"}
|
||||
compat = {"extra_body": {"chat_template_kwargs": "bad"}}
|
||||
result = merge_server_compat(base, compat)
|
||||
assert result["chat_template_kwargs"] == {"reasoning_effort": "medium"}
|
||||
assert merge_server_compat(None, compat) == {}
|
||||
|
||||
def test_base_not_mutated(self) -> None:
|
||||
base = {"reasoning_effort": "medium"}
|
||||
@@ -164,9 +198,7 @@ class TestMergeServerCompat:
|
||||
|
||||
def test_non_dict_extra_body_ignored(self) -> None:
|
||||
"""Gracefully handle malformed server_compat."""
|
||||
base = {"reasoning_effort": "medium"}
|
||||
result = merge_server_compat(base, {"extra_body": 42})
|
||||
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
|
||||
assert merge_server_compat(None, {"extra_body": 42}) == {}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -178,45 +210,58 @@ class TestEndToEndRequestShaping:
|
||||
"""Compose both layers — session builds extra_params, provider applies thinking."""
|
||||
|
||||
def test_vllm_gemma_full_flow(self) -> None:
|
||||
"""Session merges server workarounds, provider adds thinking param."""
|
||||
"""Session forwards server workarounds, provider adds thinking param."""
|
||||
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
|
||||
base_ctk = {"reasoning_effort": "medium"}
|
||||
server_compat = {
|
||||
"server_type": "vllm",
|
||||
"extra_body": {"skip_special_tokens": False},
|
||||
}
|
||||
# Step 1: session merges
|
||||
extra_params = merge_server_compat(base_ctk, server_compat)
|
||||
# Step 2: provider finalises
|
||||
# Step 1: session forwards (no auto-injection of reasoning_effort).
|
||||
extra_params = merge_server_compat(None, server_compat)
|
||||
# Step 2: provider injects thinking param into chat_template_kwargs.
|
||||
extra_body = dict(extra_params)
|
||||
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
|
||||
|
||||
assert extra_body == {
|
||||
"chat_template_kwargs": {
|
||||
"reasoning_effort": "medium",
|
||||
"enable_thinking": True,
|
||||
},
|
||||
"chat_template_kwargs": {"enable_thinking": True},
|
||||
"skip_special_tokens": False,
|
||||
}
|
||||
|
||||
def test_granite_thinking_key(self) -> None:
|
||||
"""Granite uses 'thinking' instead of 'enable_thinking'."""
|
||||
caps = ModelCapabilities(thinking_mode="manual", thinking_param="thinking")
|
||||
extra_params = merge_server_compat({"reasoning_effort": "low"}, {})
|
||||
extra_params = merge_server_compat(None, {})
|
||||
extra_body = dict(extra_params)
|
||||
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
|
||||
|
||||
assert extra_body["chat_template_kwargs"]["thinking"] is True
|
||||
assert "enable_thinking" not in extra_body["chat_template_kwargs"]
|
||||
assert extra_body == {"chat_template_kwargs": {"thinking": True}}
|
||||
|
||||
def test_non_thinking_model_no_injection(self) -> None:
|
||||
"""Non-thinking model gets no thinking params."""
|
||||
"""Non-thinking model gets no chat_template_kwargs at all."""
|
||||
caps = ModelCapabilities() # thinking_mode="none"
|
||||
extra_params = merge_server_compat({"reasoning_effort": "medium"}, {})
|
||||
extra_params = merge_server_compat(None, {})
|
||||
extra_body = dict(extra_params)
|
||||
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
|
||||
|
||||
assert extra_body == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
|
||||
assert extra_body == {}
|
||||
|
||||
def test_operator_reasoning_effort_passthrough(self) -> None:
|
||||
"""Operator-supplied reasoning_effort under chat_template_kwargs is preserved."""
|
||||
caps = ModelCapabilities(thinking_mode="manual", thinking_param="enable_thinking")
|
||||
compat = {
|
||||
"server_type": "vllm",
|
||||
"extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}},
|
||||
}
|
||||
extra_params = merge_server_compat(None, compat)
|
||||
extra_body = dict(extra_params)
|
||||
OpenAIChatCompletionsProvider._apply_thinking_mode(extra_body, caps)
|
||||
|
||||
assert extra_body == {
|
||||
"chat_template_kwargs": {
|
||||
"reasoning_effort": "high",
|
||||
"enable_thinking": True,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
+44
-66
@@ -1182,7 +1182,7 @@ class TestAgentOutputGuard:
|
||||
|
||||
|
||||
class TestProviderExtraParams:
|
||||
"""Tests for _provider_extra_params — local-only chat_template_kwargs."""
|
||||
"""Tests for _provider_extra_params — server_compat passthrough only."""
|
||||
|
||||
def _session_with_provider(self, provider_name: str, tmp_db) -> ChatSession:
|
||||
from turnstone.core.providers import create_provider
|
||||
@@ -1191,68 +1191,36 @@ class TestProviderExtraParams:
|
||||
session._provider = create_provider(provider_name)
|
||||
return session
|
||||
|
||||
def test_openai_compatible_returns_chat_template_kwargs(self, tmp_db):
|
||||
def test_openai_compatible_no_compat_returns_none(self, tmp_db):
|
||||
"""No server_compat → no extra_body needed (no auto-injection)."""
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
result = session._provider_extra_params()
|
||||
assert result is not None
|
||||
assert "chat_template_kwargs" in result
|
||||
assert result["chat_template_kwargs"]["reasoning_effort"] == "medium"
|
||||
assert session._provider_extra_params() is None
|
||||
|
||||
def test_openai_commercial_returns_none(self, tmp_db):
|
||||
def test_openai_commercial_no_compat_returns_none(self, tmp_db):
|
||||
"""Cloud OpenAI without server_compat → None."""
|
||||
session = self._session_with_provider("openai", tmp_db)
|
||||
result = session._provider_extra_params()
|
||||
assert result is None
|
||||
assert session._provider_extra_params() is None
|
||||
|
||||
def test_anthropic_returns_none(self, tmp_db):
|
||||
session = self._session_with_provider("anthropic", tmp_db)
|
||||
result = session._provider_extra_params()
|
||||
assert result is None
|
||||
assert session._provider_extra_params() is None
|
||||
|
||||
def test_reasoning_effort_override(self, tmp_db):
|
||||
def test_no_reasoning_effort_kwarg(self, tmp_db):
|
||||
"""reasoning_effort is not part of the surface; passing it should TypeError.
|
||||
|
||||
Splatted via ``**kwargs`` so static analyzers (CodeQL "wrong-name
|
||||
argument" / mypy) don't flag the call — the point of this test is the
|
||||
runtime contract, not the static type.
|
||||
"""
|
||||
import pytest
|
||||
|
||||
bad_kwargs = {"reasoning_effort": "high"}
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
result = session._provider_extra_params(reasoning_effort="high")
|
||||
assert result is not None
|
||||
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
|
||||
with pytest.raises(TypeError):
|
||||
session._provider_extra_params(**bad_kwargs)
|
||||
|
||||
def test_explicit_openai_provider_overrides_session(self, tmp_db):
|
||||
"""Passing an explicit commercial OpenAI provider returns None even
|
||||
when the session's own provider is openai-compatible."""
|
||||
from turnstone.core.providers import create_provider
|
||||
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
openai_prov = create_provider("openai")
|
||||
result = session._provider_extra_params(provider=openai_prov)
|
||||
assert result is None
|
||||
|
||||
def test_server_compat_extra_body_merged(self, tmp_db):
|
||||
"""server_compat.extra_body workarounds are merged into extra_params."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
cfg = ModelConfig(
|
||||
alias="test",
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="none",
|
||||
model="google/gemma-4-31B-it",
|
||||
server_compat={
|
||||
"extra_body": {"skip_special_tokens": False},
|
||||
},
|
||||
)
|
||||
session._registry = ModelRegistry(models={"test": cfg}, default="test")
|
||||
session._model_alias = "test"
|
||||
result = session._provider_extra_params()
|
||||
assert result is not None
|
||||
assert result["chat_template_kwargs"]["reasoning_effort"] == "medium"
|
||||
assert result["skip_special_tokens"] is False
|
||||
|
||||
def test_empty_server_compat_backwards_compatible(self, tmp_db):
|
||||
"""Empty server_compat produces same output as before."""
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
result = session._provider_extra_params()
|
||||
assert result == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
|
||||
|
||||
def test_server_compat_with_reasoning_effort_override(self, tmp_db):
|
||||
"""reasoning_effort override works alongside server_compat."""
|
||||
def test_server_compat_extra_body_passes_through(self, tmp_db):
|
||||
"""server_compat.extra_body workarounds forward as extra_params."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
@@ -1265,10 +1233,25 @@ class TestProviderExtraParams:
|
||||
)
|
||||
session._registry = ModelRegistry(models={"test": cfg}, default="test")
|
||||
session._model_alias = "test"
|
||||
result = session._provider_extra_params(reasoning_effort="high")
|
||||
assert result is not None
|
||||
assert result["chat_template_kwargs"]["reasoning_effort"] == "high"
|
||||
assert result["skip_special_tokens"] is False
|
||||
result = session._provider_extra_params()
|
||||
assert result == {"skip_special_tokens": False}
|
||||
|
||||
def test_operator_chat_template_kwargs_pass_through(self, tmp_db):
|
||||
"""Operator-set chat_template_kwargs (e.g. for gpt-oss) forwards verbatim."""
|
||||
from turnstone.core.model_registry import ModelConfig, ModelRegistry
|
||||
|
||||
session = self._session_with_provider("openai-compatible", tmp_db)
|
||||
cfg = ModelConfig(
|
||||
alias="test",
|
||||
base_url="http://localhost:8000/v1",
|
||||
api_key="none",
|
||||
model="openai/gpt-oss-120b",
|
||||
server_compat={"extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}}},
|
||||
)
|
||||
session._registry = ModelRegistry(models={"test": cfg}, default="test")
|
||||
session._model_alias = "test"
|
||||
result = session._provider_extra_params()
|
||||
assert result == {"chat_template_kwargs": {"reasoning_effort": "high"}}
|
||||
|
||||
def test_model_alias_resolves_target_compat(self, tmp_db):
|
||||
"""model_alias parameter selects compat from the target, not the primary."""
|
||||
@@ -1297,14 +1280,9 @@ class TestProviderExtraParams:
|
||||
session._model_alias = "primary"
|
||||
|
||||
# Primary alias → gets Gemma workaround
|
||||
result_primary = session._provider_extra_params()
|
||||
assert result_primary is not None
|
||||
assert result_primary["skip_special_tokens"] is False
|
||||
|
||||
# Fallback alias → no compat, just base kwargs
|
||||
result_fallback = session._provider_extra_params(model_alias="fallback")
|
||||
assert result_fallback == {"chat_template_kwargs": {"reasoning_effort": "medium"}}
|
||||
assert "skip_special_tokens" not in result_fallback
|
||||
assert session._provider_extra_params() == {"skip_special_tokens": False}
|
||||
# Fallback alias → no compat at all
|
||||
assert session._provider_extra_params(model_alias="fallback") is None
|
||||
|
||||
|
||||
class TestSafePrepareTool:
|
||||
|
||||
@@ -8090,6 +8090,33 @@ _MODEL_PROVIDERS = frozenset({"openai", "anthropic", "openai-compatible", "googl
|
||||
_REASONING_EFFORT_CHOICES = frozenset(
|
||||
{"", "none", "minimal", "low", "medium", "high", "xhigh", "max"}
|
||||
)
|
||||
# Keep in sync with turnstone.core.providers._VALID_API_SURFACES.
|
||||
_API_SURFACE_CHOICES = frozenset({"chat", "responses"})
|
||||
|
||||
|
||||
def _validate_api_surface(caps: Any) -> str | None:
|
||||
"""Return an error message if ``caps["server_compat"]["api_surface"]`` is invalid.
|
||||
|
||||
Strict equality match (no strip/lower normalisation): the persisted value
|
||||
is bound directly to the admin ``<select>`` whose options are the canonical
|
||||
``"chat"`` / ``"responses"`` strings, so anything else fails to round-trip
|
||||
through edit/save. The provider factory raises ``ValueError`` at request
|
||||
time for an unknown surface; validating here turns that into a 400 at
|
||||
write time so an admin can't poison a model alias via direct API calls.
|
||||
"""
|
||||
if not isinstance(caps, dict):
|
||||
return None
|
||||
sc = caps.get("server_compat")
|
||||
if not isinstance(sc, dict):
|
||||
return None
|
||||
raw = sc.get("api_surface")
|
||||
if raw is None or raw == "":
|
||||
return None
|
||||
if not isinstance(raw, str) or raw not in _API_SURFACE_CHOICES:
|
||||
return f"Invalid server_compat.api_surface: {raw!r}"
|
||||
return None
|
||||
|
||||
|
||||
# Keep in sync with turnstone.core.providers._google.GOOGLE_DEFAULT_BASE_URL
|
||||
_PROVIDER_DEFAULT_URLS: dict[str, str] = {
|
||||
"openai": "https://api.openai.com/v1",
|
||||
@@ -8391,6 +8418,9 @@ async def admin_create_model_definition(request: Request) -> JSONResponse:
|
||||
ctx_raw = body.get("context_window", 32768)
|
||||
context_window = max(0, int(ctx_raw)) if isinstance(ctx_raw, (int, float)) else 0
|
||||
caps = body.get("capabilities", {})
|
||||
err_msg = _validate_api_surface(caps)
|
||||
if err_msg:
|
||||
return JSONResponse({"error": err_msg}, status_code=400)
|
||||
capabilities = json.dumps(caps) if isinstance(caps, dict) else "{}"
|
||||
enabled = bool(body.get("enabled", True))
|
||||
|
||||
@@ -8545,6 +8575,9 @@ async def admin_update_model_definition(request: Request) -> JSONResponse:
|
||||
updates["context_window"] = max(0, int(ctx_raw)) if isinstance(ctx_raw, (int, float)) else 0
|
||||
if "capabilities" in body:
|
||||
caps = body["capabilities"]
|
||||
err_msg = _validate_api_surface(caps)
|
||||
if err_msg:
|
||||
return JSONResponse({"error": err_msg}, status_code=400)
|
||||
updates["capabilities"] = json.dumps(caps) if isinstance(caps, dict) else "{}"
|
||||
if "enabled" in body:
|
||||
updates["enabled"] = bool(body["enabled"])
|
||||
|
||||
@@ -5001,6 +5001,7 @@ function showCreateModelModal() {
|
||||
document.getElementById("model-max-tokens").value = "";
|
||||
document.getElementById("model-reasoning-effort").value = "";
|
||||
document.getElementById("model-server-type").value = "";
|
||||
document.getElementById("model-api-surface").value = "";
|
||||
document.getElementById("model-thinking-mode").value = "";
|
||||
document.getElementById("model-thinking-param").value = "";
|
||||
document.getElementById("model-thinking-param-row").style.display = "none";
|
||||
@@ -5077,8 +5078,9 @@ function showEditModelModal(definitionId) {
|
||||
document.getElementById("model-thinking-param").value = "";
|
||||
}
|
||||
_toggleThinkingParam();
|
||||
// Server compat: server_type and extra_body workarounds
|
||||
// Server compat: server_type, api_surface, and extra_body workarounds
|
||||
document.getElementById("model-server-type").value = sc.server_type || "";
|
||||
document.getElementById("model-api-surface").value = sc.api_surface || "";
|
||||
var eb = sc.extra_body || {};
|
||||
var ebText = JSON.stringify(eb, null, 2);
|
||||
document.getElementById("model-extra-body").value =
|
||||
@@ -5160,26 +5162,34 @@ function submitCreateModel() {
|
||||
if (savedParam) caps.thinking_param = savedParam;
|
||||
}
|
||||
|
||||
// Build server_compat from structured fields
|
||||
// Build server_compat from structured fields. Only meaningful for
|
||||
// openai-compatible aliases — for other providers the section is hidden
|
||||
// but the form values can linger after a provider switch, so gate the
|
||||
// whole block on the active provider to keep persisted state honest.
|
||||
var serverCompat = {};
|
||||
var serverType = document.getElementById("model-server-type").value;
|
||||
if (serverType) serverCompat.server_type = serverType;
|
||||
var providerVal = document.getElementById("model-provider").value;
|
||||
var ebEl = document.getElementById("model-extra-body");
|
||||
var ebText = ebEl.value.trim();
|
||||
ebEl.removeAttribute("aria-invalid");
|
||||
ebEl.style.borderColor = "";
|
||||
if (ebText) {
|
||||
try {
|
||||
var ebParsed = JSON.parse(ebText);
|
||||
if (!_isPlainObject(ebParsed)) {
|
||||
throw new Error("not an object");
|
||||
if (providerVal === "openai-compatible") {
|
||||
var serverType = document.getElementById("model-server-type").value;
|
||||
if (serverType) serverCompat.server_type = serverType;
|
||||
var apiSurface = document.getElementById("model-api-surface").value;
|
||||
if (apiSurface) serverCompat.api_surface = apiSurface;
|
||||
var ebText = ebEl.value.trim();
|
||||
if (ebText) {
|
||||
try {
|
||||
var ebParsed = JSON.parse(ebText);
|
||||
if (!_isPlainObject(ebParsed)) {
|
||||
throw new Error("not an object");
|
||||
}
|
||||
serverCompat.extra_body = ebParsed;
|
||||
} catch (e) {
|
||||
ebEl.setAttribute("aria-invalid", "true");
|
||||
ebEl.style.borderColor = "var(--red)";
|
||||
_showModelError("Extra body params must be a JSON object");
|
||||
return;
|
||||
}
|
||||
serverCompat.extra_body = ebParsed;
|
||||
} catch (e) {
|
||||
ebEl.setAttribute("aria-invalid", "true");
|
||||
ebEl.style.borderColor = "var(--red)";
|
||||
_showModelError("Extra body params must be a JSON object");
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (Object.keys(serverCompat).length > 0) {
|
||||
@@ -5410,6 +5420,13 @@ function detectModel() {
|
||||
stOpts2.indexOf(ssc.server_type) !== -1
|
||||
)
|
||||
stEl2.value = ssc.server_type;
|
||||
// Restrict to the known set so a hostile detect response can't
|
||||
// smuggle a non-listed value into the form.
|
||||
var _SURFACE_SUGGESTABLE = { chat: 1, responses: 1 };
|
||||
if (ssc.api_surface && _SURFACE_SUGGESTABLE[ssc.api_surface]) {
|
||||
var asEl = document.getElementById("model-api-surface");
|
||||
if (!asEl.value) asEl.value = ssc.api_surface;
|
||||
}
|
||||
if (ssc.extra_body) {
|
||||
var ebEl2 = document.getElementById("model-extra-body");
|
||||
if (!ebEl2.value.trim()) {
|
||||
|
||||
@@ -3920,6 +3920,17 @@
|
||||
<option value="llama.cpp">llama.cpp</option>
|
||||
<option value="openai-compatible">Other OpenAI-compatible</option>
|
||||
</select>
|
||||
<label for="model-api-surface"
|
||||
>API Surface
|
||||
<span style="font-weight: 400; text-transform: none"
|
||||
>(Chat Completions vs Responses)</span
|
||||
></label
|
||||
>
|
||||
<select id="model-api-surface">
|
||||
<option value="">Inherit (Chat Completions)</option>
|
||||
<option value="chat">Chat Completions (pinned)</option>
|
||||
<option value="responses">Responses API</option>
|
||||
</select>
|
||||
<label for="model-thinking-mode"
|
||||
>Thinking Mode
|
||||
<span style="font-weight: 400; text-transform: none"
|
||||
|
||||
@@ -44,6 +44,19 @@ class ModelConfig:
|
||||
server_compat: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
def _api_surface_of(cfg: ModelConfig) -> str | None:
|
||||
"""Extract the operator-pinned api_surface from *cfg*, or ``None``.
|
||||
|
||||
Used both at provider-cache lookup time and at reload-eviction time so the
|
||||
two sites stay in sync. Returns ``None`` when the field is absent, blank,
|
||||
or not a string — matching the "inherit provider default" semantics.
|
||||
"""
|
||||
raw = cfg.server_compat.get("api_surface") if isinstance(cfg.server_compat, dict) else None
|
||||
if isinstance(raw, str) and raw.strip():
|
||||
return raw
|
||||
return None
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Registry
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -127,7 +140,9 @@ class ModelRegistry:
|
||||
raise ValueError(f"Unknown model alias: {alias}")
|
||||
if alias not in self._providers:
|
||||
cfg = self._models[alias]
|
||||
self._providers[alias] = create_provider(cfg.provider)
|
||||
self._providers[alias] = create_provider(
|
||||
cfg.provider, api_surface=_api_surface_of(cfg)
|
||||
)
|
||||
return self._providers[alias]
|
||||
|
||||
def get_config(self, alias: str) -> ModelConfig:
|
||||
@@ -257,13 +272,18 @@ class ModelRegistry:
|
||||
if hasattr(client, "close"):
|
||||
client.close()
|
||||
del self._clients[alias]
|
||||
# Providers are keyed on alias but only depend on
|
||||
# ``cfg.provider`` — drop only when the provider string
|
||||
# changed or the alias was removed.
|
||||
# Providers are keyed on alias and depend on (cfg.provider,
|
||||
# cfg.server_compat["api_surface"]) — drop when either changes
|
||||
# or the alias was removed.
|
||||
for alias in list(self._providers.keys()):
|
||||
old_cfg = old_models.get(alias)
|
||||
new_cfg = self._models.get(alias)
|
||||
if new_cfg is None or old_cfg is None or old_cfg.provider != new_cfg.provider:
|
||||
if (
|
||||
new_cfg is None
|
||||
or old_cfg is None
|
||||
or old_cfg.provider != new_cfg.provider
|
||||
or _api_surface_of(old_cfg) != _api_surface_of(new_cfg)
|
||||
):
|
||||
del self._providers[alias]
|
||||
|
||||
def shutdown(self) -> None:
|
||||
|
||||
@@ -33,7 +33,9 @@ __all__ = [
|
||||
"lookup_model_capabilities",
|
||||
]
|
||||
|
||||
# Singleton instances (stateless, safe to share)
|
||||
# Singleton instances (stateless, safe to share). ``_openai_provider``
|
||||
# is reused for both cloud OpenAI and ``openai-compatible`` with
|
||||
# ``api_surface="responses"`` — see the ``create_provider`` docstring.
|
||||
_provider_lock = threading.Lock()
|
||||
_openai_provider = OpenAIResponsesProvider()
|
||||
_openai_compat_provider = OpenAIChatCompletionsProvider()
|
||||
@@ -41,12 +43,45 @@ _anthropic_provider: LLMProvider | None = None
|
||||
_google_provider: LLMProvider | None = None
|
||||
|
||||
|
||||
def create_provider(provider_name: str) -> LLMProvider:
|
||||
"""Return a provider adapter for the given provider name. Thread-safe."""
|
||||
_VALID_API_SURFACES = ("chat", "responses")
|
||||
|
||||
|
||||
def create_provider(
|
||||
provider_name: str,
|
||||
*,
|
||||
api_surface: str | None = None,
|
||||
) -> LLMProvider:
|
||||
"""Return a provider adapter for the given provider name. Thread-safe.
|
||||
|
||||
*api_surface* selects the OpenAI-compatible API surface for
|
||||
``provider_name="openai-compatible"``:
|
||||
|
||||
- ``"chat"`` (default) → Chat Completions (vLLM, llama.cpp, SGLang).
|
||||
- ``"responses"`` → Responses API (commercial OpenAI-compat
|
||||
endpoints like Mistral cloud, or local servers that expose the
|
||||
Responses surface).
|
||||
|
||||
Ignored for non-OpenAI providers. ``provider_name="openai"`` always
|
||||
uses the Responses API regardless of *api_surface*.
|
||||
|
||||
Note: the ``OpenAIResponsesProvider`` singleton is reused for both
|
||||
cloud OpenAI and ``openai-compatible`` + responses, so its
|
||||
``provider_name`` reports ``"openai"`` even when serving an
|
||||
openai-compatible config. Code that needs to distinguish the two
|
||||
must read ``ModelConfig.provider`` and ``server_compat["api_surface"]``
|
||||
rather than ``provider.provider_name``.
|
||||
"""
|
||||
global _anthropic_provider, _google_provider # noqa: PLW0603
|
||||
if provider_name == "openai":
|
||||
return _openai_provider
|
||||
if provider_name == "openai-compatible":
|
||||
normalised = (api_surface or "").strip().lower()
|
||||
if normalised and normalised not in _VALID_API_SURFACES:
|
||||
raise ValueError(
|
||||
f"Unknown api_surface: {api_surface!r}. Supported: {', '.join(_VALID_API_SURFACES)}"
|
||||
)
|
||||
if normalised == "responses":
|
||||
return _openai_provider
|
||||
return _openai_compat_provider
|
||||
if provider_name == "anthropic":
|
||||
with _provider_lock:
|
||||
|
||||
@@ -1,14 +1,20 @@
|
||||
"""Server compatibility profiles for OpenAI-compatible backends.
|
||||
|
||||
Different local model servers (vLLM, llama.cpp, SGLang) need different
|
||||
request shaping. This module separates two concerns:
|
||||
request shaping. This module separates three concerns:
|
||||
|
||||
1. **Model capabilities** — ``thinking_mode`` and ``thinking_param`` are
|
||||
properties of the *model* (Gemma thinks, Llama doesn't). These go
|
||||
into the ``capabilities`` dict and flow through ``ModelCapabilities``
|
||||
so the provider can act on them (just like Anthropic's thinking mode).
|
||||
|
||||
2. **Server workarounds** — ``extra_body`` overrides like
|
||||
2. **API surface** — ``api_surface`` selects which OpenAI-compatible
|
||||
API surface the provider talks to: ``"chat"`` (Chat Completions,
|
||||
the default) or ``"responses"`` (Responses API, native reasoning).
|
||||
Stored under ``server_compat`` because it's an endpoint property,
|
||||
not a model property.
|
||||
|
||||
3. **Server workarounds** — ``extra_body`` overrides like
|
||||
``skip_special_tokens=false`` are properties of the *server* (vLLM
|
||||
bug workaround). These stay in ``server_compat`` and get merged
|
||||
into the request's ``extra_body`` at call time.
|
||||
@@ -82,6 +88,23 @@ _PROFILES: dict[str, dict[str, Any]] = {
|
||||
"server_type": "vllm",
|
||||
},
|
||||
},
|
||||
"vllm-mistral-medium": {
|
||||
# Mistral medium open-weights served by vLLM can deliver reasoning
|
||||
# via either surface, but the trade-off is asymmetric:
|
||||
# * Chat Completions — tool calling works (``--tool-call-parser
|
||||
# mistral``); reasoning is enabled via the vLLM CLI
|
||||
# (``--reasoning-parser``) rather than per-request.
|
||||
# * Responses API — reasoning effort is per-request and clean,
|
||||
# but as of vLLM 0.x the tool-call parser is not wired up on
|
||||
# this surface so tool calls leak as ``[TOOL_CALLS]`` text.
|
||||
# We do **not** auto-suggest this profile from Detect; an operator
|
||||
# who needs per-request effort and accepts the tool-calling
|
||||
# limitation can pick "Responses API" manually in the admin UI.
|
||||
"server_compat": {
|
||||
"server_type": "vllm",
|
||||
"api_surface": "responses",
|
||||
},
|
||||
},
|
||||
"vllm": {
|
||||
"server_compat": {
|
||||
"server_type": "vllm",
|
||||
@@ -125,6 +148,9 @@ _VLLM_MODEL_PROFILES: list[tuple[str, str]] = [
|
||||
("granite3", "vllm-granite-thinking"),
|
||||
("deepseek-r1", "vllm-deepseek-thinking"),
|
||||
("holo2", "vllm-holo-thinking"),
|
||||
# Mistral medium intentionally omitted — see ``vllm-mistral-medium``
|
||||
# profile docstring for the Chat-vs-Responses trade-off; operator
|
||||
# picks manually rather than letting Detect auto-suggest Responses.
|
||||
]
|
||||
|
||||
# llama.cpp model-family → profile key mapping.
|
||||
@@ -175,31 +201,39 @@ def suggest_profile(server_type: str, model_id: str) -> dict[str, Any]:
|
||||
|
||||
|
||||
def merge_server_compat(
|
||||
base_chat_template_kwargs: dict[str, Any],
|
||||
base_chat_template_kwargs: dict[str, Any] | None,
|
||||
server_compat: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
"""Build the ``extra_body`` dict by merging server compat into base kwargs.
|
||||
|
||||
*base_chat_template_kwargs* always contains at least ``reasoning_effort``.
|
||||
*server_compat* comes from ``ModelConfig.server_compat``.
|
||||
*base_chat_template_kwargs* is an explicit ``chat_template_kwargs`` dict
|
||||
to seed the request with, or ``None``/empty to skip seeding. Operator-
|
||||
supplied entries in ``server_compat["extra_body"]["chat_template_kwargs"]``
|
||||
are deep-merged on top. Top-level ``extra_body`` keys (``skip_special_tokens``,
|
||||
``reasoning_format``, etc.) are forwarded as-is.
|
||||
|
||||
Note: thinking-mode params (``enable_thinking``, ``thinking``) are **not**
|
||||
merged here — the provider handles those via ``ModelCapabilities``.
|
||||
This function only merges server workarounds from ``extra_body``.
|
||||
This function only merges what the operator stored in ``server_compat``.
|
||||
|
||||
Returns the complete dict to pass as ``extra_body`` to the OpenAI client.
|
||||
May be empty when there is nothing to send.
|
||||
"""
|
||||
extra: dict[str, Any] = {"chat_template_kwargs": dict(base_chat_template_kwargs)}
|
||||
extra: dict[str, Any] = {}
|
||||
if base_chat_template_kwargs:
|
||||
extra["chat_template_kwargs"] = dict(base_chat_template_kwargs)
|
||||
|
||||
# Merge top-level extra_body overrides (skip_special_tokens, etc.)
|
||||
compat_eb = server_compat.get("extra_body")
|
||||
if isinstance(compat_eb, dict):
|
||||
for key, value in compat_eb.items():
|
||||
if key == "chat_template_kwargs":
|
||||
# Deep-merge: operator values in extra_body win over the
|
||||
# base dict (which has reasoning_effort). This lets
|
||||
# operators intentionally extend chat_template_kwargs.
|
||||
# Deep-merge with operator values winning so an operator
|
||||
# can intentionally extend chat_template_kwargs (e.g. set
|
||||
# ``reasoning_effort`` for gpt-oss-style local templates).
|
||||
if isinstance(value, dict):
|
||||
if "chat_template_kwargs" not in extra:
|
||||
extra["chat_template_kwargs"] = {}
|
||||
extra["chat_template_kwargs"].update(value)
|
||||
continue
|
||||
extra[key] = value
|
||||
|
||||
+26
-28
@@ -1458,11 +1458,6 @@ class ChatSession:
|
||||
"""
|
||||
new_system_messages: list[dict[str, Any]] = []
|
||||
|
||||
# -- Chat template kwargs --
|
||||
self._chat_template_kwargs_base: dict[str, Any] = {
|
||||
"reasoning_effort": self.reasoning_effort,
|
||||
}
|
||||
|
||||
# -- Developer message --
|
||||
if self.creative_mode:
|
||||
dev_parts = [
|
||||
@@ -1844,48 +1839,48 @@ class ChatSession:
|
||||
|
||||
def _provider_extra_params(
|
||||
self,
|
||||
reasoning_effort: str | None = None,
|
||||
provider: LLMProvider | None = None,
|
||||
model_alias: str | None = None,
|
||||
) -> dict[str, Any] | None:
|
||||
"""Build provider-specific extra parameters.
|
||||
|
||||
``chat_template_kwargs`` is only meaningful for local model servers
|
||||
(``openai-compatible``). Commercial OpenAI rejects it as an unknown
|
||||
parameter, and handles ``reasoning_effort`` natively.
|
||||
Forwards operator-supplied ``server_compat["extra_body"]`` overrides
|
||||
(``skip_special_tokens``, ``reasoning_format``, or explicit
|
||||
``chat_template_kwargs``) to the OpenAI SDK ``extra_body``. Operators
|
||||
running gpt-oss-style local templates that consume ``reasoning_effort``
|
||||
from ``chat_template_kwargs`` should set it explicitly under
|
||||
``server_compat["extra_body"]["chat_template_kwargs"]``.
|
||||
|
||||
Merges server workarounds (``skip_special_tokens``, etc.) from
|
||||
``ModelConfig.server_compat`` into the request's ``extra_body``.
|
||||
Thinking-mode params (``enable_thinking``) are handled separately
|
||||
by the provider based on ``ModelCapabilities.thinking_mode``.
|
||||
Thinking-mode params (``enable_thinking``, ``thinking``) are added
|
||||
separately by ``OpenAIChatCompletionsProvider._apply_thinking_mode``
|
||||
based on ``ModelCapabilities.thinking_mode`` — the Responses API
|
||||
surface handles reasoning natively and ignores ``extra_body``.
|
||||
|
||||
*model_alias* controls which model config supplies server compat
|
||||
*model_alias* selects which stored config supplies server compat
|
||||
settings. When ``None``, defaults to the session's primary alias.
|
||||
"""
|
||||
from turnstone.core.server_compat import merge_server_compat
|
||||
|
||||
prov = provider or self._provider
|
||||
if prov.provider_name == "openai-compatible":
|
||||
ctk_base = dict(self._chat_template_kwargs_base)
|
||||
if reasoning_effort:
|
||||
ctk_base["reasoning_effort"] = reasoning_effort
|
||||
return merge_server_compat(
|
||||
ctk_base,
|
||||
self._get_server_compat(model_alias),
|
||||
)
|
||||
return None
|
||||
# Only OpenAI-shaped providers consume extra_body. Anthropic/Google
|
||||
# have their own param paths handled inside their providers.
|
||||
if prov.provider_name not in ("openai", "openai-compatible"):
|
||||
return None
|
||||
extra = merge_server_compat(None, self._get_server_compat(model_alias))
|
||||
return extra or None
|
||||
|
||||
def _get_server_compat(self, model_alias: str | None = None) -> dict[str, Any]:
|
||||
"""Get server compatibility settings from a model config.
|
||||
|
||||
*model_alias* selects the config to read. Falls back to the
|
||||
session's primary alias when ``None``.
|
||||
session's primary alias when ``None``. The returned dict is the
|
||||
live ``ModelConfig.server_compat`` reference — callers must not
|
||||
mutate it. ``merge_server_compat`` reads only.
|
||||
"""
|
||||
alias = model_alias or self._model_alias
|
||||
if self._registry and alias:
|
||||
try:
|
||||
cfg = self._registry.get_config(alias)
|
||||
return dict(cfg.server_compat)
|
||||
return self._registry.get_config(alias).server_compat
|
||||
except (ValueError, KeyError):
|
||||
pass
|
||||
return {}
|
||||
@@ -1914,7 +1909,7 @@ class ChatSession:
|
||||
max_tokens=clamped,
|
||||
temperature=temperature,
|
||||
reasoning_effort=reasoning_effort,
|
||||
extra_params=self._provider_extra_params(reasoning_effort=reasoning_effort),
|
||||
extra_params=self._provider_extra_params(),
|
||||
capabilities=caps,
|
||||
)
|
||||
|
||||
@@ -8037,6 +8032,10 @@ class ChatSession:
|
||||
agent_client = self.client
|
||||
agent_model = self.model
|
||||
agent_provider = self._provider
|
||||
# When falling through to the session's primary model, use the
|
||||
# session's primary alias for capability and server_compat
|
||||
# resolution so the agent sees the same caps as the main loop.
|
||||
agent_alias = self._model_alias
|
||||
|
||||
# Per-kind reasoning effort. Explicit caller arg wins; otherwise
|
||||
# delegate to the registry which knows the per-kind default (plan
|
||||
@@ -8059,7 +8058,6 @@ class ChatSession:
|
||||
# Build extra params for agent calls — resolve server compat from the
|
||||
# agent's own model alias, not the session's primary model.
|
||||
agent_extra = self._provider_extra_params(
|
||||
reasoning_effort=reasoning_effort,
|
||||
provider=agent_provider,
|
||||
model_alias=agent_alias,
|
||||
)
|
||||
|
||||
Reference in New Issue
Block a user