mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
29c42c1427
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.
Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
effective_port, sanitize_log_text moved out of oidc.py into a shared
oauth_ssrf module; oidc.py re-exports for compatibility. The shared
helpers also expose async wrappers (validate_url_no_ssrf_async,
validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
from async handlers — does not block the event loop on the
synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
DELETE...RETURNING)
Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
existing refresh value is preserved at the OAuth-flow layer rather
than cleared, so production ASes (Google, Auth0 default, Okta) don't
force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
MCP server URL, not the audience. Audience and resource are distinct
concepts; using audience as resource would mismatch the AS RS
allowlist.
Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
accepted_audiences = {server_url, oauth_audience} and validates
against the set, so Auth0-style ASes that honor `audience=` (not
RFC 8707 `resource=`) issue tokens that pass audience-bound
validation
- build_authorize_url emits both `resource=` (RFC 8707) and
`audience=` (Auth0-style) per server config; comment documents which
AS implementations need which form
Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
request Host header — closes the same Host-header injection PR #476
fixed for OIDC. Both /start and /callback return 503 with operator-
actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
inside the lock, so concurrent /start callers don't both register
and overwrite each other's client_id (the second user's code is no
longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
so a leaked state can't be replayed against a separately-obtained
code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
through sanitize_log_text before reaching exception messages or
audit details. AS error responses are parsed for the standard
RFC 6749 fields (error, error_description, error_uri), each
capped at 80 chars and run through redact_credentials to defend
against ASes that echo the request body back into their error
payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
tokens and pending OAuth state for the previous server name
(delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
mutable server_name; without this purge, a future server with the
same name (and an attacker-controlled URL) would silently rebind
prior user tokens. A future schema migration will replace the
server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
installed key can decrypt the row, e.g. after key rotation) and
falls through to None so dispatch surfaces a re-consent rather than
crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
path. Auto-connecting them at startup with empty headers fails the
AS check and trips the circuit breaker; per-user tokens come online
lazily once the user has consented.
Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
token_revoked, dcr_registered. _audit_event is async and wraps
record_audit in asyncio.to_thread so the audit write doesn't block
the event loop. resource_id on the audit row is the immutable
server_id (PK UUID) so admin-driven server renames don't break
event correlation; server_name is exposed in detail for cross-
reference. dcr_registered detail.has_secret reflects whether the
DCR-issued secret was actually persisted (the prior code reported
has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
server_name (which is what the column is — the table's PK was
always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
the audit-action taxonomy.
Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.
Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
limitation in oauth_ssrf module docstring
Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.
236 lines
9.1 KiB
Python
236 lines
9.1 KiB
Python
"""Tests for ``turnstone.core.mcp_crypto`` cipher + config loading.
|
|
|
|
Covers token-at-rest encryption for OAuth-MCP.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import base64
|
|
|
|
import pytest
|
|
from cryptography.fernet import Fernet
|
|
|
|
from turnstone.core.mcp_crypto import (
|
|
MCPTokenCipher,
|
|
MCPTokenCipherConfig,
|
|
MCPTokenDecryptError,
|
|
MCPTokenKeyConfigError,
|
|
_key_fingerprint,
|
|
_validate_key,
|
|
load_mcp_token_cipher_config,
|
|
)
|
|
|
|
|
|
def _new_raw_key() -> bytes:
|
|
"""Return a fresh 32-byte Fernet key as raw bytes (post-base64-decode)."""
|
|
return base64.urlsafe_b64decode(Fernet.generate_key())
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Cipher round-trip
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCipherRoundTrip:
|
|
def test_round_trip_single_key(self) -> None:
|
|
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
|
|
plaintext = b"access_token_12345"
|
|
ct = cipher.encrypt(plaintext)
|
|
assert ct != plaintext
|
|
assert cipher.decrypt(ct) == plaintext
|
|
|
|
def test_round_trip_unicode_token(self) -> None:
|
|
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
|
|
# Tokens may legitimately carry UTF-8 bytes (e.g. JWT with
|
|
# non-ASCII claim values). Round-trip a multi-byte sequence.
|
|
plaintext = "tok_é中💯".encode()
|
|
ct = cipher.encrypt(plaintext)
|
|
assert cipher.decrypt(ct) == plaintext
|
|
|
|
def test_wrong_key_raises_decrypt_error(self) -> None:
|
|
cipher_a = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
|
|
cipher_b = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
|
|
ct = cipher_a.encrypt(b"secret")
|
|
with pytest.raises(MCPTokenDecryptError) as exc_info:
|
|
cipher_b.decrypt(ct)
|
|
# Audit-trail correlation: error must carry the fingerprints of
|
|
# the keys actually attempted, not a placeholder.
|
|
assert exc_info.value.key_fingerprints_attempted
|
|
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Rotation (MultiFernet behavior)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestRotation:
|
|
def test_rotation_forward(self) -> None:
|
|
"""Encrypt with a new-only cipher, decrypt with a [v2, v1] cluster.
|
|
|
|
Mirrors the operational situation where a node already has the
|
|
rotated key list installed and a peer just wrote a row under v2.
|
|
"""
|
|
v1 = _new_raw_key()
|
|
v2 = _new_raw_key()
|
|
new_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2,)))
|
|
cluster = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
|
|
ct = new_only.encrypt(b"hello")
|
|
assert cluster.decrypt(ct) == b"hello"
|
|
|
|
def test_rotation_backward_keeps_old_decryptable(self) -> None:
|
|
"""A row written under the OLD key (v1) must still decrypt after
|
|
rotation places v2 first and keeps v1 as fallback."""
|
|
v1 = _new_raw_key()
|
|
v2 = _new_raw_key()
|
|
old_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1,)))
|
|
rotated = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
|
|
ct = old_only.encrypt(b"legacy")
|
|
assert rotated.decrypt(ct) == b"legacy"
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Config loader
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
def _patch_load_config(monkeypatch: pytest.MonkeyPatch, payload: dict) -> None:
|
|
"""Override ``turnstone.core.config.load_config`` to return ``payload``
|
|
when the ``"security"`` section is requested."""
|
|
|
|
def fake(section: str | None = None) -> dict:
|
|
if section == "security":
|
|
return payload
|
|
return {}
|
|
|
|
import turnstone.core.config as cfg_mod
|
|
|
|
monkeypatch.setattr(cfg_mod, "load_config", fake)
|
|
|
|
|
|
class TestLoadConfig:
|
|
def test_load_singular_key(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
key = Fernet.generate_key().decode()
|
|
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": key})
|
|
cfg = load_mcp_token_cipher_config()
|
|
assert cfg is not None
|
|
assert len(cfg.keys) == 1
|
|
|
|
def test_load_plural_overrides_singular(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
plural = [Fernet.generate_key().decode(), Fernet.generate_key().decode()]
|
|
_patch_load_config(
|
|
monkeypatch,
|
|
{
|
|
"mcp_token_encryption_keys": plural,
|
|
"mcp_token_encryption_key": Fernet.generate_key().decode(),
|
|
},
|
|
)
|
|
cfg = load_mcp_token_cipher_config()
|
|
assert cfg is not None
|
|
assert len(cfg.keys) == 2 # plural wins, singular ignored
|
|
|
|
def test_load_returns_none_when_absent(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
_patch_load_config(monkeypatch, {})
|
|
assert load_mcp_token_cipher_config() is None
|
|
|
|
def test_load_empty_plural_falls_through_to_singular(
|
|
self, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Operator wrote ``mcp_token_encryption_keys = []`` AND set a
|
|
singular value: empty plural is treated as absent."""
|
|
key = Fernet.generate_key().decode()
|
|
_patch_load_config(
|
|
monkeypatch,
|
|
{"mcp_token_encryption_keys": [], "mcp_token_encryption_key": key},
|
|
)
|
|
cfg = load_mcp_token_cipher_config()
|
|
assert cfg is not None
|
|
assert len(cfg.keys) == 1
|
|
|
|
def test_load_invalid_base64_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": "###not-base64###"})
|
|
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
|
|
load_mcp_token_cipher_config()
|
|
# Operator-facing hint is part of every error message.
|
|
assert "regenerate with:" in str(exc_info.value)
|
|
|
|
def test_load_wrong_length_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
# 24 raw bytes → 32 base64 chars; not 32 raw bytes after decode.
|
|
short_key = base64.urlsafe_b64encode(b"\x00" * 24).decode()
|
|
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": short_key})
|
|
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
|
|
load_mcp_token_cipher_config()
|
|
assert "32 bytes" in str(exc_info.value)
|
|
|
|
def test_load_non_list_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": "single-string-not-list"})
|
|
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
|
|
load_mcp_token_cipher_config()
|
|
assert "list" in str(exc_info.value).lower()
|
|
|
|
def test_load_non_string_in_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
|
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": [12345]})
|
|
with pytest.raises(MCPTokenKeyConfigError):
|
|
load_mcp_token_cipher_config()
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Fingerprint stability
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestFingerprint:
|
|
def test_key_fingerprint_stable_and_short(self) -> None:
|
|
key = _new_raw_key()
|
|
fp1 = _key_fingerprint(key)
|
|
fp2 = _key_fingerprint(key)
|
|
assert fp1 == fp2
|
|
# 8 bytes -> 16 hex characters.
|
|
assert len(fp1) == 16
|
|
assert all(c in "0123456789abcdef" for c in fp1)
|
|
|
|
def test_different_keys_have_different_fingerprints(self) -> None:
|
|
fp1 = _key_fingerprint(_new_raw_key())
|
|
fp2 = _key_fingerprint(_new_raw_key())
|
|
assert fp1 != fp2
|
|
|
|
def test_cipher_fingerprints_match_keys(self) -> None:
|
|
v1 = _new_raw_key()
|
|
v2 = _new_raw_key()
|
|
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1, v2)))
|
|
assert cipher.key_fingerprints == (
|
|
_key_fingerprint(v1),
|
|
_key_fingerprint(v2),
|
|
)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# Direct ``_validate_key`` — exercises edge cases not reachable via loader
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestValidateKey:
|
|
def test_empty_string_rejected(self) -> None:
|
|
with pytest.raises(MCPTokenKeyConfigError):
|
|
_validate_key("", label="x")
|
|
|
|
def test_whitespace_only_rejected(self) -> None:
|
|
with pytest.raises(MCPTokenKeyConfigError):
|
|
_validate_key(" ", label="x")
|
|
|
|
def test_label_propagated_in_error(self) -> None:
|
|
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
|
|
_validate_key("###", label="my_label_42")
|
|
assert "my_label_42" in str(exc_info.value)
|
|
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# MCPTokenCipher constructor guard
|
|
# ---------------------------------------------------------------------------
|
|
|
|
|
|
class TestCipherConstructorGuard:
|
|
def test_empty_keys_rejected(self) -> None:
|
|
with pytest.raises(MCPTokenKeyConfigError):
|
|
MCPTokenCipher(MCPTokenCipherConfig(keys=()))
|