Files
turnstone/tests/test_mcp_crypto.py
T
Patrick Buckley 29c42c1427 feat(mcp): per-(user, server) OAuth 2.1 + PKCE flow
Lands the OAuth flow that uses the token-at-rest store from the prior
commit: discovery (RFC 9728 PRM + RFC 8414 AS metadata with operator-
override precedence), PKCE S256 (mandatory — refuse AS without it),
RFC 8707 resource indicator on every authorize and token request,
RFC 7591 minimal one-shot dynamic client registration, authorization-
code exchange, refresh-token grant with re-read-after-acquire single-
flight lock, and the /v1/api/mcp/oauth/{start,callback} endpoints
mounted on both server and console.

Refactored:
- validate_url_no_ssrf, validate_discovered_endpoint, is_localhost,
  effective_port, sanitize_log_text moved out of oidc.py into a shared
  oauth_ssrf module; oidc.py re-exports for compatibility. The shared
  helpers also expose async wrappers (validate_url_no_ssrf_async,
  validate_discovered_endpoint_async) so OAuth-MCP discovery — invoked
  from async handlers — does not block the event loop on the
  synchronous socket.getaddrinfo call.
- MCPTokenStore.get_oauth_client_secret reader path added (the prior
  commit was write-only)
- Storage protocol gains create/pop/cleanup_*_mcp_oauth_pending_state
  and get_mcp_oauth_client_secret_ct (mirror OIDC pending-state
  pattern: SQLite BEGIN IMMEDIATE select-then-delete, Postgres atomic
  DELETE...RETURNING)

Refresh-grant correctness:
- When the AS omits refresh_token (RFC 6749 §6 — MAY rotate), the
  existing refresh value is preserved at the OAuth-flow layer rather
  than cleared, so production ASes (Google, Auth0 default, Okta) don't
  force re-consent every hour
- expires_in accepts int, float, str-with-decimal — earlier int-coerce
  through str() failed on float and silently dropped expiry tracking
- The refresh-grant `resource=` parameter (RFC 8707) is the canonical
  MCP server URL, not the audience. Audience and resource are distinct
  concepts; using audience as resource would mismatch the AS RS
  allowlist.

Audience handling:
- _validate_token_audience accepts str or tuple; the callback resolves
  accepted_audiences = {server_url, oauth_audience} and validates
  against the set, so Auth0-style ASes that honor `audience=` (not
  RFC 8707 `resource=`) issue tokens that pass audience-bound
  validation
- build_authorize_url emits both `resource=` (RFC 8707) and
  `audience=` (Auth0-style) per server config; comment documents which
  AS implementations need which form

Security hardening:
- redirect_uri pinned to oidc_config.redirect_base instead of the
  request Host header — closes the same Host-header injection PR #476
  fixed for OIDC. Both /start and /callback return 503 with operator-
  actionable hint when redirect_base is unset
- DCR registration runs under per-server asyncio.Lock with re-fetch
  inside the lock, so concurrent /start callers don't both register
  and overwrite each other's client_id (the second user's code is no
  longer rejected on callback)
- /callback error branch pops the pending state row before redirecting
  so a leaked state can't be replayed against a separately-obtained
  code in the 60s cleanup window
- WWW-Authenticate Bearer parser handles RFC 7235 quoted-string
  escapes (\" and \\) instead of the naive [^"]+ regex
- AS-controlled response bodies and error_description query params go
  through sanitize_log_text before reaching exception messages or
  audit details. AS error responses are parsed for the standard
  RFC 6749 fields (error, error_description, error_uri), each
  capped at 80 chars and run through redact_credentials to defend
  against ASes that echo the request body back into their error
  payload.
- oauth_as_issuer_cached is re-validated against the SSRF guard on
  read; on rejection the column is cleared and PRM rediscovery runs
- DCR / token-endpoint / refresh-endpoint response bodies cap at 64
  KiB (PRM/AS metadata cap stays at 256 KiB) so a hostile or
  malfunctioning AS can't exhaust client memory.
- oauth_client_secret operator input capped at 1024 chars at the
  admin-form boundary; longer plaintext rejected with 400.
- /start and /callback responses stamp `X-Frame-Options: DENY` so the
  redirected pages can't be framed by attacker sites.
- delete_user cascades to mcp_user_tokens and mcp_oauth_pending so
  user deletion no longer leaves dangling per-user OAuth state.
- Renaming or deleting an oauth_user MCP server purges per-user
  tokens and pending OAuth state for the previous server name
  (delete_mcp_oauth_rows_by_server_name). The OAuth tables key on the
  mutable server_name; without this purge, a future server with the
  same name (and an attacker-controlled URL) would silently rebind
  prior user tokens. A future schema migration will replace the
  server_name key with a server_id FK + ON DELETE CASCADE.
- get_user_access_token catches MCPTokenDecryptError (raised when no
  installed key can decrypt the row, e.g. after key rotation) and
  falls through to None so dispatch surfaces a re-consent rather than
  crashing.
- oauth_user MCP server rows are skipped in the static auto-connect
  path. Auto-connecting them at startup with empty headers fails the
  AS check and trips the circuit breaker; per-user tokens come online
  lazily once the user has consented.

Audit (mcp_server.oauth.* prefix):
- consent_started, consent_completed, consent_failed, token_refreshed,
  token_revoked, dcr_registered. _audit_event is async and wraps
  record_audit in asyncio.to_thread so the audit write doesn't block
  the event loop. resource_id on the audit row is the immutable
  server_id (PK UUID) so admin-driven server renames don't break
  event correlation; server_name is exposed in detail for cross-
  reference. dcr_registered detail.has_secret reflects whether the
  DCR-issued secret was actually persisted (the prior code reported
  has_secret=true even on persistence failure).
- _admin_mcp_action audits the immutable server_id, not the mutable
  server_name (which is what the column is — the table's PK was
  always server_id).
- All OAuth-flow log keys use the mcp_server.oauth.* prefix to match
  the audit-action taxonomy.

Lifespan close-order in turnstone.server and turnstone.console.server
is reversed (LIFO) — mcp_oauth → mcp_crypto → oidc — to match init
order.

Deferred until the upcoming per-user pool integration:
- Multi-node refresh-lock contention via pg_advisory_lock
- DCR re-register on token-endpoint 401 (the dispatch path surfaces
  those 401s)
- TTL-LRU caching of decrypted plaintext access tokens
- DNS-rebinding hardening (httpx Transport pin) — documented as
  limitation in oauth_ssrf module docstring

Tests: 7 new test files / ~85 new tests covering discovery precedence
+ PRM quoted-string parsing, PKCE round-trip, SSRF helper extraction,
authorize/callback handlers including 503-on-no-redirect-base + DCR
concurrency + JWT audience polymorphism + callback-error-pops-pending,
refresh single-flight lock, refresh resource-vs-audience regression,
decrypt-error fallthrough, _db_servers_to_config skipping oauth_user,
pending-state CRUD round-trip.
2026-05-04 22:00:23 -07:00

236 lines
9.1 KiB
Python

"""Tests for ``turnstone.core.mcp_crypto`` cipher + config loading.
Covers token-at-rest encryption for OAuth-MCP.
"""
from __future__ import annotations
import base64
import pytest
from cryptography.fernet import Fernet
from turnstone.core.mcp_crypto import (
MCPTokenCipher,
MCPTokenCipherConfig,
MCPTokenDecryptError,
MCPTokenKeyConfigError,
_key_fingerprint,
_validate_key,
load_mcp_token_cipher_config,
)
def _new_raw_key() -> bytes:
"""Return a fresh 32-byte Fernet key as raw bytes (post-base64-decode)."""
return base64.urlsafe_b64decode(Fernet.generate_key())
# ---------------------------------------------------------------------------
# Cipher round-trip
# ---------------------------------------------------------------------------
class TestCipherRoundTrip:
def test_round_trip_single_key(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
plaintext = b"access_token_12345"
ct = cipher.encrypt(plaintext)
assert ct != plaintext
assert cipher.decrypt(ct) == plaintext
def test_round_trip_unicode_token(self) -> None:
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
# Tokens may legitimately carry UTF-8 bytes (e.g. JWT with
# non-ASCII claim values). Round-trip a multi-byte sequence.
plaintext = "tok_é中💯".encode()
ct = cipher.encrypt(plaintext)
assert cipher.decrypt(ct) == plaintext
def test_wrong_key_raises_decrypt_error(self) -> None:
cipher_a = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
cipher_b = MCPTokenCipher(MCPTokenCipherConfig(keys=(_new_raw_key(),)))
ct = cipher_a.encrypt(b"secret")
with pytest.raises(MCPTokenDecryptError) as exc_info:
cipher_b.decrypt(ct)
# Audit-trail correlation: error must carry the fingerprints of
# the keys actually attempted, not a placeholder.
assert exc_info.value.key_fingerprints_attempted
assert exc_info.value.key_fingerprints_attempted == cipher_b.key_fingerprints
# ---------------------------------------------------------------------------
# Rotation (MultiFernet behavior)
# ---------------------------------------------------------------------------
class TestRotation:
def test_rotation_forward(self) -> None:
"""Encrypt with a new-only cipher, decrypt with a [v2, v1] cluster.
Mirrors the operational situation where a node already has the
rotated key list installed and a peer just wrote a row under v2.
"""
v1 = _new_raw_key()
v2 = _new_raw_key()
new_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2,)))
cluster = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = new_only.encrypt(b"hello")
assert cluster.decrypt(ct) == b"hello"
def test_rotation_backward_keeps_old_decryptable(self) -> None:
"""A row written under the OLD key (v1) must still decrypt after
rotation places v2 first and keeps v1 as fallback."""
v1 = _new_raw_key()
v2 = _new_raw_key()
old_only = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1,)))
rotated = MCPTokenCipher(MCPTokenCipherConfig(keys=(v2, v1)))
ct = old_only.encrypt(b"legacy")
assert rotated.decrypt(ct) == b"legacy"
# ---------------------------------------------------------------------------
# Config loader
# ---------------------------------------------------------------------------
def _patch_load_config(monkeypatch: pytest.MonkeyPatch, payload: dict) -> None:
"""Override ``turnstone.core.config.load_config`` to return ``payload``
when the ``"security"`` section is requested."""
def fake(section: str | None = None) -> dict:
if section == "security":
return payload
return {}
import turnstone.core.config as cfg_mod
monkeypatch.setattr(cfg_mod, "load_config", fake)
class TestLoadConfig:
def test_load_singular_key(self, monkeypatch: pytest.MonkeyPatch) -> None:
key = Fernet.generate_key().decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": key})
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_plural_overrides_singular(self, monkeypatch: pytest.MonkeyPatch) -> None:
plural = [Fernet.generate_key().decode(), Fernet.generate_key().decode()]
_patch_load_config(
monkeypatch,
{
"mcp_token_encryption_keys": plural,
"mcp_token_encryption_key": Fernet.generate_key().decode(),
},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 2 # plural wins, singular ignored
def test_load_returns_none_when_absent(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {})
assert load_mcp_token_cipher_config() is None
def test_load_empty_plural_falls_through_to_singular(
self, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Operator wrote ``mcp_token_encryption_keys = []`` AND set a
singular value: empty plural is treated as absent."""
key = Fernet.generate_key().decode()
_patch_load_config(
monkeypatch,
{"mcp_token_encryption_keys": [], "mcp_token_encryption_key": key},
)
cfg = load_mcp_token_cipher_config()
assert cfg is not None
assert len(cfg.keys) == 1
def test_load_invalid_base64_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": "###not-base64###"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
# Operator-facing hint is part of every error message.
assert "regenerate with:" in str(exc_info.value)
def test_load_wrong_length_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
# 24 raw bytes → 32 base64 chars; not 32 raw bytes after decode.
short_key = base64.urlsafe_b64encode(b"\x00" * 24).decode()
_patch_load_config(monkeypatch, {"mcp_token_encryption_key": short_key})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "32 bytes" in str(exc_info.value)
def test_load_non_list_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": "single-string-not-list"})
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
load_mcp_token_cipher_config()
assert "list" in str(exc_info.value).lower()
def test_load_non_string_in_plural_raises(self, monkeypatch: pytest.MonkeyPatch) -> None:
_patch_load_config(monkeypatch, {"mcp_token_encryption_keys": [12345]})
with pytest.raises(MCPTokenKeyConfigError):
load_mcp_token_cipher_config()
# ---------------------------------------------------------------------------
# Fingerprint stability
# ---------------------------------------------------------------------------
class TestFingerprint:
def test_key_fingerprint_stable_and_short(self) -> None:
key = _new_raw_key()
fp1 = _key_fingerprint(key)
fp2 = _key_fingerprint(key)
assert fp1 == fp2
# 8 bytes -> 16 hex characters.
assert len(fp1) == 16
assert all(c in "0123456789abcdef" for c in fp1)
def test_different_keys_have_different_fingerprints(self) -> None:
fp1 = _key_fingerprint(_new_raw_key())
fp2 = _key_fingerprint(_new_raw_key())
assert fp1 != fp2
def test_cipher_fingerprints_match_keys(self) -> None:
v1 = _new_raw_key()
v2 = _new_raw_key()
cipher = MCPTokenCipher(MCPTokenCipherConfig(keys=(v1, v2)))
assert cipher.key_fingerprints == (
_key_fingerprint(v1),
_key_fingerprint(v2),
)
# ---------------------------------------------------------------------------
# Direct ``_validate_key`` — exercises edge cases not reachable via loader
# ---------------------------------------------------------------------------
class TestValidateKey:
def test_empty_string_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key("", label="x")
def test_whitespace_only_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
_validate_key(" ", label="x")
def test_label_propagated_in_error(self) -> None:
with pytest.raises(MCPTokenKeyConfigError) as exc_info:
_validate_key("###", label="my_label_42")
assert "my_label_42" in str(exc_info.value)
# ---------------------------------------------------------------------------
# MCPTokenCipher constructor guard
# ---------------------------------------------------------------------------
class TestCipherConstructorGuard:
def test_empty_keys_rejected(self) -> None:
with pytest.raises(MCPTokenKeyConfigError):
MCPTokenCipher(MCPTokenCipherConfig(keys=()))