mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
feat: replace Redis MQ with direct HTTP transport (Phase 1)
Delete the entire turnstone/mq/ package (broker, bridge, protocol, client) and turnstone/sim/ package. Remove Redis as a dependency. Channel gateway and console now communicate with server nodes via direct HTTP (httpx + httpx-sse) instead of Redis pub/sub and queues. Single-node deployments work with zero infrastructure beyond the database. Key changes: - Channel adapters use httpx POST for create/send/approve/close and httpx-sse for per-workstream event streaming - Console collector discovers nodes via services table instead of Redis SCAN - Console scheduler dispatches tasks via HTTP POST with DB-based leader election - Server registers in services table with 30s heartbeat - Server accepts optional ws_id in create request (for Phase 2 console-generated routing) - SDK events gain IntentVerdictEvent and OutputWarningEvent types - All docs, examples, bootstrap wizard updated 63 files changed, -5968 net lines (Redis transport fully removed)
This commit is contained in:
Patrick Buckley
committed by
Patrick Buckley
Patrick Buckley
parent
0e02d1b52c
commit
2bb55590bf
@@ -1,10 +1,10 @@
|
||||
# MCP Cluster Ops
|
||||
|
||||
An MCP server that exposes tools for executing commands across a [Turnstone](https://github.com/turnstonelabs/turnstone) cluster. Serves as a reference implementation for both MCP server patterns and Turnstone MQ client SDK usage.
|
||||
An MCP server that exposes tools for executing commands across a [Turnstone](https://github.com/turnstonelabs/turnstone) cluster. Serves as a reference implementation for both MCP server patterns and Turnstone SDK usage.
|
||||
|
||||
## How it works
|
||||
|
||||
This server uses Turnstone's MQ client (`TurnstoneClient`) to dispatch shell commands to specific nodes via Redis. Remote agents execute the command and the raw bash output is captured directly from the `ToolResultEvent` stream — bypassing the costly "agent reads output → re-generates output as completion tokens" round-trip.
|
||||
This server uses Turnstone's SDK client (`TurnstoneServer`) to dispatch shell commands to specific nodes via HTTP. Remote agents execute the command and the raw bash output is captured directly from the `ToolResultEvent` stream — bypassing the costly "agent reads output → re-generates output as completion tokens" round-trip.
|
||||
|
||||
Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time is bounded by the slowest node rather than the sum.
|
||||
|
||||
@@ -19,8 +19,7 @@ Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time i
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- A running Turnstone cluster (at least one `turnstone-server` + `turnstone-bridge`)
|
||||
- Redis accessible from wherever this MCP server runs
|
||||
- A running Turnstone cluster (at least one `turnstone-server`)
|
||||
- Python 3.11+
|
||||
|
||||
## Installation
|
||||
@@ -28,10 +27,6 @@ Multi-node dispatches run in parallel via `asyncio.gather`, so total wall time i
|
||||
```bash
|
||||
# From the turnstone repo root:
|
||||
pip install -e ./examples/mcp-cluster-ops
|
||||
|
||||
# Or install turnstone with MQ support first, then the example:
|
||||
pip install -e ".[mq]"
|
||||
pip install -e ./examples/mcp-cluster-ops
|
||||
```
|
||||
|
||||
## Configuration
|
||||
@@ -40,9 +35,8 @@ pip install -e ./examples/mcp-cluster-ops
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `REDIS_HOST` | `localhost` | Redis host |
|
||||
| `REDIS_PORT` | `6379` | Redis port |
|
||||
| `REDIS_PASSWORD` | _(none)_ | Redis password (use env vars, not config files) |
|
||||
| `TURNSTONE_SERVER_URL` | `http://localhost:8080` | Server URL |
|
||||
| `TURNSTONE_API_TOKEN` | _(none)_ | API token for authentication |
|
||||
| `MCP_CLUSTER_OPS_TIMEOUT` | `120` | Default command timeout (seconds, clamped 5-3600) |
|
||||
| `MCP_CLUSTER_OPS_MAX_OUTPUT` | `8192` | Max output bytes per node (0 = unlimited) |
|
||||
| `MCP_CLUSTER_OPS_MAX_NODES` | `32` | Max concurrent node dispatches |
|
||||
@@ -57,7 +51,7 @@ pip install -e ./examples/mcp-cluster-ops
|
||||
command = "mcp-cluster-ops"
|
||||
|
||||
[mcp.servers.cluster-ops.env]
|
||||
REDIS_HOST = "redis.example.com"
|
||||
TURNSTONE_SERVER_URL = "http://turnstone.example.com:8080"
|
||||
```
|
||||
|
||||
**JSON** (via `--mcp-config`):
|
||||
@@ -68,7 +62,7 @@ REDIS_HOST = "redis.example.com"
|
||||
"cluster-ops": {
|
||||
"command": "mcp-cluster-ops",
|
||||
"env": {
|
||||
"REDIS_HOST": "redis.example.com"
|
||||
"TURNSTONE_SERVER_URL": "http://turnstone.example.com:8080"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -90,10 +84,6 @@ node-2: /dev/sda1 500G 410G 90G 82% /
|
||||
node-3: /dev/sda1 1.0T 200G 800G 20% /
|
||||
```
|
||||
|
||||
## Why MQ client instead of HTTP SDK?
|
||||
|
||||
The HTTP SDK (`TurnstoneServer`) talks to a single server instance. The MQ client (`TurnstoneClient`) routes through Redis with `target_node` support, which is the entire point of cross-node cluster operations.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
**This MCP server grants the calling agent shell access to cluster nodes.**
|
||||
@@ -104,8 +94,8 @@ The HTTP SDK (`TurnstoneServer`) talks to a single server instance. The MQ clien
|
||||
is returned through the MCP tool result and becomes part of the LLM context.
|
||||
- The security boundary is at the MCP host layer -- use Turnstone's tool
|
||||
policy system to restrict which agents can invoke these tools.
|
||||
- Set `REDIS_PASSWORD` via your environment or a secrets manager -- avoid
|
||||
hardcoding passwords in config files.
|
||||
- Set `TURNSTONE_API_TOKEN` via your environment or a secrets manager -- avoid
|
||||
hardcoding tokens in config files.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
"""MCP server for Turnstone cluster operations.
|
||||
|
||||
Exposes tools to execute commands on specific nodes in a Turnstone cluster.
|
||||
Uses the MQ client (``TurnstoneClient``) for direct node targeting via Redis.
|
||||
Uses the SDK client (``TurnstoneServer``) for direct node targeting via HTTP.
|
||||
|
||||
Usage::
|
||||
|
||||
@@ -14,22 +14,20 @@ Configure in ``~/.config/turnstone/config.toml``::
|
||||
command = "mcp-cluster-ops"
|
||||
|
||||
[mcp.servers.cluster-ops.env]
|
||||
REDIS_HOST = "redis.example.com"
|
||||
TURNSTONE_SERVER_URL = "http://localhost:8080"
|
||||
|
||||
Environment variables
|
||||
---------------------
|
||||
REDIS_HOST Redis host (default: localhost)
|
||||
REDIS_PORT Redis port (default: 6379)
|
||||
REDIS_PASSWORD Redis password (default: none)
|
||||
TURNSTONE_SERVER_URL Server URL (default: http://localhost:8080)
|
||||
TURNSTONE_API_TOKEN API token for authentication (default: none)
|
||||
MCP_CLUSTER_OPS_TIMEOUT Default command timeout in seconds (default: 120)
|
||||
MCP_CLUSTER_OPS_MAX_OUTPUT Max output bytes per node (default: 8192, 0=unlimited)
|
||||
|
||||
Performance notes
|
||||
-----------------
|
||||
Remote agents are told to reply with only "ok" or "failed" — the raw bash
|
||||
output is captured directly from the ToolResultEvent that already flows
|
||||
through Redis, bypassing the costly "agent reads output then re-generates
|
||||
output as completion tokens" round-trip.
|
||||
output is captured directly from the ToolResultEvent, bypassing the costly
|
||||
"agent reads output then re-generates output as completion tokens" round-trip.
|
||||
|
||||
All multi-node dispatches run in parallel via ``asyncio.gather`` so total
|
||||
wall time is bounded by the slowest node, not the sum of all nodes.
|
||||
@@ -45,7 +43,7 @@ from contextlib import asynccontextmanager
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
from mcp.server.fastmcp import Context, FastMCP
|
||||
from turnstone.mq.client import TurnResult, TurnstoneClient
|
||||
from turnstone.sdk import TurnResult, TurnstoneServer
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import AsyncIterator
|
||||
@@ -68,19 +66,14 @@ _MAX_TIMEOUT = 3600
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _redis_kwargs() -> dict[str, Any]:
|
||||
"""Build Redis connection kwargs from environment variables.
|
||||
|
||||
Follows the same env var convention as ``turnstone.mq.broker.add_redis_args``:
|
||||
``REDIS_HOST``, ``REDIS_PORT``, ``REDIS_PASSWORD``.
|
||||
"""
|
||||
kwargs: dict[str, Any] = {"host": os.environ.get("REDIS_HOST", "localhost")}
|
||||
port = os.environ.get("REDIS_PORT")
|
||||
if port is not None:
|
||||
kwargs["port"] = int(port)
|
||||
password = os.environ.get("REDIS_PASSWORD")
|
||||
if password:
|
||||
kwargs["password"] = password
|
||||
def _server_kwargs() -> dict[str, Any]:
|
||||
"""Build TurnstoneServer connection kwargs from environment variables."""
|
||||
kwargs: dict[str, Any] = {
|
||||
"base_url": os.environ.get("TURNSTONE_SERVER_URL", "http://localhost:8080"),
|
||||
}
|
||||
token = os.environ.get("TURNSTONE_API_TOKEN")
|
||||
if token:
|
||||
kwargs["token"] = token
|
||||
return kwargs
|
||||
|
||||
|
||||
@@ -172,12 +165,12 @@ def _format_node_result(
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Core dispatch functions (testable with mocked TurnstoneClient)
|
||||
# Core dispatch functions (testable with mocked TurnstoneServer)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _exec_on_node_sync(
|
||||
redis_kw: dict[str, Any],
|
||||
server_kw: dict[str, Any],
|
||||
node_id: str,
|
||||
command: str,
|
||||
timeout: float,
|
||||
@@ -185,11 +178,11 @@ def _exec_on_node_sync(
|
||||
"""Dispatch *command* to *node_id* and block until complete.
|
||||
|
||||
Runs inside ``asyncio.to_thread`` so it does not block the event loop.
|
||||
Each call creates its own ``TurnstoneClient`` to avoid Redis pub/sub
|
||||
subscription conflicts between concurrent dispatches.
|
||||
Each call creates its own ``TurnstoneServer`` client to avoid state
|
||||
conflicts between concurrent dispatches.
|
||||
"""
|
||||
prompt = _exec_prompt(command)
|
||||
with TurnstoneClient(**redis_kw) as client:
|
||||
with TurnstoneServer(**server_kw) as client:
|
||||
result = client.send_and_wait(
|
||||
message=prompt,
|
||||
target_node=node_id,
|
||||
@@ -200,7 +193,7 @@ def _exec_on_node_sync(
|
||||
|
||||
|
||||
async def _dispatch_parallel(
|
||||
redis_kw: dict[str, Any],
|
||||
server_kw: dict[str, Any],
|
||||
node_ids: list[str],
|
||||
command: str,
|
||||
timeout: float,
|
||||
@@ -211,7 +204,7 @@ async def _dispatch_parallel(
|
||||
Total wall time is bounded by the slowest node.
|
||||
"""
|
||||
tasks = [
|
||||
asyncio.to_thread(_exec_on_node_sync, redis_kw, nid, command, timeout) for nid in node_ids
|
||||
asyncio.to_thread(_exec_on_node_sync, server_kw, nid, command, timeout) for nid in node_ids
|
||||
]
|
||||
outcomes = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
|
||||
@@ -227,16 +220,16 @@ async def _dispatch_parallel(
|
||||
return results
|
||||
|
||||
|
||||
def _list_nodes_sync(redis_kw: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
def _list_nodes_sync(server_kw: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""List active cluster nodes (blocking)."""
|
||||
with TurnstoneClient(**redis_kw) as client:
|
||||
with TurnstoneServer(**server_kw) as client:
|
||||
nodes: list[dict[str, Any]] = client.list_nodes()
|
||||
return nodes
|
||||
|
||||
|
||||
async def _list_nodes_impl(redis_kw: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
async def _list_nodes_impl(server_kw: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
"""List active cluster nodes."""
|
||||
return await asyncio.to_thread(_list_nodes_sync, redis_kw)
|
||||
return await asyncio.to_thread(_list_nodes_sync, server_kw)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -246,9 +239,9 @@ async def _list_nodes_impl(redis_kw: dict[str, Any]) -> list[dict[str, Any]]:
|
||||
|
||||
@asynccontextmanager
|
||||
async def _lifespan(server: FastMCP[dict[str, Any]]) -> AsyncIterator[dict[str, Any]]:
|
||||
"""Lifespan context — stores Redis kwargs for tool handlers."""
|
||||
kw = _redis_kwargs()
|
||||
yield {"redis_kwargs": kw}
|
||||
"""Lifespan context — stores server connection kwargs for tool handlers."""
|
||||
kw = _server_kwargs()
|
||||
yield {"server_kwargs": kw}
|
||||
|
||||
|
||||
mcp = FastMCP(
|
||||
@@ -270,8 +263,8 @@ async def list_nodes(ctx: Context[Any, Any, Any]) -> str:
|
||||
Call this before dispatching work to discover available node IDs.
|
||||
Returns a JSON array of node metadata objects.
|
||||
"""
|
||||
redis_kw: dict[str, Any] = ctx.request_context.lifespan_context["redis_kwargs"]
|
||||
nodes = await _list_nodes_impl(redis_kw)
|
||||
server_kw: dict[str, Any] = ctx.request_context.lifespan_context["server_kwargs"]
|
||||
nodes = await _list_nodes_impl(server_kw)
|
||||
return json.dumps(nodes, indent=2)
|
||||
|
||||
|
||||
@@ -298,12 +291,12 @@ async def run_on_node(
|
||||
if cmd_err:
|
||||
return json.dumps({"error": cmd_err})
|
||||
|
||||
redis_kw: dict[str, Any] = ctx.request_context.lifespan_context["redis_kwargs"]
|
||||
server_kw: dict[str, Any] = ctx.request_context.lifespan_context["server_kwargs"]
|
||||
max_output = _DEFAULT_MAX_OUTPUT
|
||||
|
||||
log.info("run_on_node node=%s cmd=%r", node_id, command)
|
||||
_, result = await asyncio.to_thread(
|
||||
_exec_on_node_sync, redis_kw, node_id, command, _clamp_timeout(timeout)
|
||||
_exec_on_node_sync, server_kw, node_id, command, _clamp_timeout(timeout)
|
||||
)
|
||||
formatted = _format_node_result(node_id, result, max_output)
|
||||
return json.dumps(formatted, indent=2)
|
||||
@@ -330,7 +323,7 @@ async def run_on_nodes(
|
||||
if cmd_err:
|
||||
return json.dumps({"error": cmd_err})
|
||||
|
||||
redis_kw: dict[str, Any] = ctx.request_context.lifespan_context["redis_kwargs"]
|
||||
server_kw: dict[str, Any] = ctx.request_context.lifespan_context["server_kwargs"]
|
||||
max_output = _DEFAULT_MAX_OUTPUT
|
||||
|
||||
clean_ids = list(dict.fromkeys(nid.strip() for nid in node_ids if nid.strip()))
|
||||
@@ -343,7 +336,7 @@ async def run_on_nodes(
|
||||
|
||||
log.info("run_on_nodes nodes=%s cmd=%r", clean_ids, command)
|
||||
results = await _dispatch_parallel(
|
||||
redis_kw, clean_ids, command, _clamp_timeout(timeout), max_output
|
||||
server_kw, clean_ids, command, _clamp_timeout(timeout), max_output
|
||||
)
|
||||
return json.dumps(results, indent=2)
|
||||
|
||||
@@ -368,10 +361,10 @@ async def run_on_all_nodes(
|
||||
if cmd_err:
|
||||
return json.dumps({"error": cmd_err})
|
||||
|
||||
redis_kw: dict[str, Any] = ctx.request_context.lifespan_context["redis_kwargs"]
|
||||
server_kw: dict[str, Any] = ctx.request_context.lifespan_context["server_kwargs"]
|
||||
max_output = _DEFAULT_MAX_OUTPUT
|
||||
|
||||
nodes = await _list_nodes_impl(redis_kw)
|
||||
nodes = await _list_nodes_impl(server_kw)
|
||||
if not nodes:
|
||||
return json.dumps({"error": "No active nodes found in cluster"})
|
||||
|
||||
@@ -388,7 +381,7 @@ async def run_on_all_nodes(
|
||||
)
|
||||
log.info("run_on_all_nodes nodes=%s cmd=%r", node_ids, command)
|
||||
results = await _dispatch_parallel(
|
||||
redis_kw, node_ids, command, _clamp_timeout(timeout), max_output
|
||||
server_kw, node_ids, command, _clamp_timeout(timeout), max_output
|
||||
)
|
||||
return json.dumps(results, indent=2)
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ description = "MCP server for Turnstone cluster operations — reference impleme
|
||||
requires-python = ">=3.11"
|
||||
license = "BUSL-1.1"
|
||||
dependencies = [
|
||||
"turnstone[mq]",
|
||||
"turnstone",
|
||||
"mcp>=1.6",
|
||||
]
|
||||
|
||||
@@ -18,7 +18,7 @@ mcp-cluster-ops = "mcp_cluster_ops.server:main"
|
||||
|
||||
[project.optional-dependencies]
|
||||
test = ["pytest>=9.0"]
|
||||
dev = ["ruff>=0.9", "mypy>=1.14", "types-redis>=4.6"]
|
||||
dev = ["ruff>=0.9", "mypy>=1.14"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from turnstone.mq.client import TurnResult
|
||||
from turnstone.sdk import TurnResult
|
||||
|
||||
from mcp_cluster_ops.server import (
|
||||
_clamp_timeout,
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Tests for MCP tool handlers with mocked TurnstoneClient."""
|
||||
"""Tests for MCP tool handlers with mocked TurnstoneServer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -6,7 +6,7 @@ import asyncio
|
||||
from typing import Any
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from turnstone.mq.client import TurnResult
|
||||
from turnstone.sdk import TurnResult
|
||||
|
||||
from mcp_cluster_ops.server import (
|
||||
_dispatch_parallel,
|
||||
@@ -22,7 +22,7 @@ from mcp_cluster_ops.server import (
|
||||
class TestListNodesImpl:
|
||||
def test_returns_nodes(self):
|
||||
nodes = [{"node_id": "a", "model": "gpt-5"}, {"node_id": "b", "model": "gpt-5"}]
|
||||
with patch("mcp_cluster_ops.server.TurnstoneClient") as mock_cls:
|
||||
with patch("mcp_cluster_ops.server.TurnstoneServer") as mock_cls:
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_nodes.return_value = nodes
|
||||
mock_cls.return_value.__enter__ = MagicMock(return_value=mock_client)
|
||||
@@ -32,7 +32,7 @@ class TestListNodesImpl:
|
||||
assert result == nodes
|
||||
|
||||
def test_empty_cluster(self):
|
||||
with patch("mcp_cluster_ops.server.TurnstoneClient") as mock_cls:
|
||||
with patch("mcp_cluster_ops.server.TurnstoneServer") as mock_cls:
|
||||
mock_client = MagicMock()
|
||||
mock_client.list_nodes.return_value = []
|
||||
mock_cls.return_value.__enter__ = MagicMock(return_value=mock_client)
|
||||
@@ -52,7 +52,7 @@ class TestExecOnNodeSync:
|
||||
turn_result = TurnResult(
|
||||
tool_results=[("bash", "hello world")],
|
||||
)
|
||||
with patch("mcp_cluster_ops.server.TurnstoneClient") as mock_cls:
|
||||
with patch("mcp_cluster_ops.server.TurnstoneServer") as mock_cls:
|
||||
mock_client = MagicMock()
|
||||
mock_client.send_and_wait.return_value = turn_result
|
||||
mock_cls.return_value.__enter__ = MagicMock(return_value=mock_client)
|
||||
@@ -70,7 +70,7 @@ class TestExecOnNodeSync:
|
||||
|
||||
def test_timeout(self):
|
||||
turn_result = TurnResult(timed_out=True)
|
||||
with patch("mcp_cluster_ops.server.TurnstoneClient") as mock_cls:
|
||||
with patch("mcp_cluster_ops.server.TurnstoneServer") as mock_cls:
|
||||
mock_client = MagicMock()
|
||||
mock_client.send_and_wait.return_value = turn_result
|
||||
mock_cls.return_value.__enter__ = MagicMock(return_value=mock_client)
|
||||
@@ -88,7 +88,7 @@ class TestExecOnNodeSync:
|
||||
|
||||
class TestDispatchParallel:
|
||||
def test_parallel_success(self):
|
||||
def fake_exec(redis_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
def fake_exec(server_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
return (node_id, TurnResult(tool_results=[("bash", f"output-{node_id}")]))
|
||||
|
||||
with patch("mcp_cluster_ops.server._exec_on_node_sync", side_effect=fake_exec):
|
||||
@@ -108,9 +108,9 @@ class TestDispatchParallel:
|
||||
assert outputs["b"] == "output-b"
|
||||
|
||||
def test_partial_failure(self):
|
||||
def fake_exec(redis_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
def fake_exec(server_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
if node_id == "bad":
|
||||
raise ConnectionError("Redis down")
|
||||
raise ConnectionError("connection refused")
|
||||
return (node_id, TurnResult(tool_results=[("bash", "ok")]))
|
||||
|
||||
with patch("mcp_cluster_ops.server._exec_on_node_sync", side_effect=fake_exec):
|
||||
@@ -128,10 +128,10 @@ class TestDispatchParallel:
|
||||
bad = next(r for r in results if r["node"] == "bad")
|
||||
assert good["ok"] is True
|
||||
assert bad["ok"] is False
|
||||
assert "Redis down" in bad["error"]
|
||||
assert "connection refused" in bad["error"]
|
||||
|
||||
def test_all_fail(self):
|
||||
def fake_exec(redis_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
def fake_exec(server_kw: Any, node_id: str, command: str, timeout: float) -> Any:
|
||||
raise RuntimeError(f"fail-{node_id}")
|
||||
|
||||
with patch("mcp_cluster_ops.server._exec_on_node_sync", side_effect=fake_exec):
|
||||
|
||||
Reference in New Issue
Block a user