turnstone

ewilliams28/turnstone

Fork 0

mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00

Commit Graph

Author	SHA1	Message	Date
Patrick Buckley	be165c1971	feat: MCP resource and prompt discovery with read_resource tool (#44 ) * feat: MCP resource and prompt discovery with read_resource tool Extends MCPClientManager with resource and prompt discovery alongside existing tool support. Resources and prompts are discovered on connect, cached per-server with copy-on-write rebuilds, and refreshed via push notifications, periodic polling, or manual /mcp refresh. New read_resource built-in tool reads MCP resources by URI. Requires user approval (same as MCP tool calls) since resources are served by external MCP servers. Resource catalog injected into system message with XML delimiters. Error messages sanitized to prevent leaking server internals to the model. Prompt discovery stores prefixed names (mcp__server__prompt) and exposes get_prompt_sync() for future use_prompt tool (Chunk D). /mcp command now shows tools, resources, and prompts. Docs and diagrams updated. * feat: MCP prompt governance sync with origin tracking and readonly guards Migration 009 adds origin, mcp_server, and readonly columns to prompt_templates. MCP prompts discovered by MCPClientManager are automatically synced into the governance table as read-only templates with origin="mcp". Sync engine handles: create on connect, update on prompt refresh, delete when prompts are removed from server. Manual templates take precedence on name collision (MCP prompt skipped with warning). Admin API returns 403 on update/delete of readonly templates. Console UI shows MCP origin badge and disables edit/delete buttons. Storage backends gain get_prompt_template_by_name, list_prompt_templates_by_origin, and delete_prompt_templates_by_server methods. Also addresses PR #44 review feedback: concurrent.futures.TimeoutError handling in sync dispatch, XML-escape resource catalog descriptions, resource template entries excluded from _resource_map, URI collision warnings, needs_periodic capability-aware computation, malformed JSON primary key fallback for read_resource. * feat: use_prompt tool, prompt catalog, and PR review hardening New use_prompt built-in tool invokes MCP prompt templates by name, expanding them into messages. Requires user approval (external MCP servers). Prompt catalog injected into system message with XML delimiters (up to 30 prompts, HTML-escaped). Prompt listener registered in session for catalog rebuild on changes. Addresses PR #44 review feedback: - _init_system_messages() now uses copy-on-write (build locally, assign atomically) so background thread callbacks never see partial system messages - sync_prompts_to_storage() serialized behind _sync_lock to prevent races between set_storage() (main thread) and MCP background thread - shutdown() clears listener lists to release callback references Docs and diagrams updated for 18 built-in tools. * feat: granular tool policies for MCP resources, prompts, and tools Policy evaluation now uses approval_label (falling back to func_name) for fnmatch pattern matching, enabling fine-grained per-URI and per-server policies: - read_resource: mcp_resource__{normalized_uri} - use_prompt: mcp__{server}__{prompt} (prefixed name) - MCP tools: mcp__{server}__{tool} (was static "mcp_tool") URI normalization resolves .. path segments to prevent traversal bypasses in policy matching. Resource templates filtered from system message catalog (not directly readable). use_prompt arguments validated as dict with string coercion. TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly fields. Governance docs updated with MCP policy patterns. * feat: MCP visibility in server and console UIs Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts counts. Server UI status bar shows magenta MCP indicator with tooltip. Console cluster status bar shows MCP metrics with magenta LED dot. Console node detail view shows per-node MCP summary. Console collector aggregates MCP counts across nodes in overview. Uses var(--magenta) design token with new --magenta-glow for theme adaptation. ARIA roles on MCP status elements. Tooltips on console MCP metric labels. Node MCP summary hidden on mobile (< 700px). New diagram: 20-mcp-architecture.puml covering full MCP lifecycle (connection, discovery, refresh, governance sync, policy, UI). * fix: McpStatus in health schema, count properties, catalog name fidelity Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so typed clients see the mcp field from /health. Addresses Copilot review feedback: - resource_count/prompt_count properties avoid list allocation on /health and /metrics polls - get_tools/resources/prompts return shallow-copied dicts to prevent callers from mutating internal cache - Prompt names and arg names in system message catalog are NOT HTML-escaped (model must use exact strings in use_prompt calls); only descriptions are escaped * fix: OpenAPI spec McpStatus + diagram approval column accuracy Adds McpStatus schema and optional mcp field to HealthResponse in openapi-server.json, matching the Python schema and TypeScript types. Fixes tool pipeline diagram: math, web_fetch, web_search correctly shown as auto-approve (not "Yes" for approval).	2026-03-12 14:49:58 -07:00
Patrick Buckley	7492816ab2	feat: governance — RBAC, tool policies, prompt templates, usage track… (#39 ) * feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging Add comprehensive governance layer for the admin console: - RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer), custom role CRUD, user-role assignment with privilege escalation prevention - Tool policies with glob pattern matching, priority-ordered evaluation (allow/deny/ask), enforced before auto-approve in WebUI.approve_tools() - Prompt templates with variable substitution, categories, default flag - Usage tracking: per-LLM-request token/tool metrics, aggregated queries (group by day/model/user), automatic 90-day pruning via scheduler - Audit logging: append-only event trail for all admin mutations, filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For aware IP extraction - require_permission() enforced on all 35+ admin endpoints (users, tokens, channels, schedules, watches, roles, orgs, policies, templates, usage, audit) - Field allowlists on storage update methods prevent mass-assignment bugs - Self-deletion guard on admin_delete_user, delete_user cascades user_roles - _row_to_dict helper eliminates ~400 lines of fragile positional row mapping - _audit_context helper deduplicates 18 instances of audit boilerplate - Migration 008: 7 new tables, 3 builtin roles, org_id on users - Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit) with permission-gated visibility, 7 modal dialogs, full keyboard accessibility - Python + TypeScript SDK methods for all governance endpoints - 120+ new tests (1554 total) * fix: address PR #39 review feedback - Rebuild serialized items after policy evaluation so denied/allowed verdicts are reflected in tool_info/approve_request SSE payloads - Make `since` query param optional in usage OpenAPI spec (handler already defaults to last 7 days) - Add response_model=StatusResponse to DELETE role/policy/template and POST/DELETE role assignment endpoints in OpenAPI spec - Add missing org_id/created/updated fields to UserRoleInfo schema - Add missing created field to AuditEventInfo schema - Show "no permissions" empty state instead of loading inaccessible tab when all admin tabs are permission-gated - Fix "13 permissions" → "15 permissions" in architecture.md and security.md - Fix import sorting in test_audit.py and test_tool_policy.py * fix: address PR #39 round 2 review feedback - Clear stale permissions from sessionStorage on config-token login (auth.js _storePermissions) - Only trust X-Forwarded-For when behind a proxy that sets X-Forwarded-Proto (conditional on is_secure_request trust model) - Thread user_id from auth into WebUI.on_status for usage events - Add created field to TS AuditEventInfo type - Return typed Pydantic models from all SDK governance methods instead of dict[str, Any] — both async and sync clients - Validate group_by param against allowed enum in admin_usage handler - Add deterministic secondary sort (event_id DESC) to list_audit_events in both SQLite and PostgreSQL backends	2026-03-10 19:32:37 -07:00

Author

SHA1

Message

Date

Patrick Buckley

be165c1971

feat: MCP resource and prompt discovery with read_resource tool (#44 )

* feat: MCP resource and prompt discovery with read_resource tool

Extends MCPClientManager with resource and prompt discovery alongside
existing tool support. Resources and prompts are discovered on connect,
cached per-server with copy-on-write rebuilds, and refreshed via push
notifications, periodic polling, or manual /mcp refresh.

New read_resource built-in tool reads MCP resources by URI. Requires
user approval (same as MCP tool calls) since resources are served by
external MCP servers. Resource catalog injected into system message
with XML delimiters. Error messages sanitized to prevent leaking
server internals to the model.

Prompt discovery stores prefixed names (mcp__server__prompt) and
exposes get_prompt_sync() for future use_prompt tool (Chunk D).

/mcp command now shows tools, resources, and prompts. Docs and
diagrams updated.

* feat: MCP prompt governance sync with origin tracking and readonly guards

Migration 009 adds origin, mcp_server, and readonly columns to
prompt_templates. MCP prompts discovered by MCPClientManager are
automatically synced into the governance table as read-only templates
with origin="mcp".

Sync engine handles: create on connect, update on prompt refresh,
delete when prompts are removed from server. Manual templates take
precedence on name collision (MCP prompt skipped with warning).

Admin API returns 403 on update/delete of readonly templates. Console
UI shows MCP origin badge and disables edit/delete buttons. Storage
backends gain get_prompt_template_by_name, list_prompt_templates_by_origin,
and delete_prompt_templates_by_server methods.

Also addresses PR #44 review feedback: concurrent.futures.TimeoutError
handling in sync dispatch, XML-escape resource catalog descriptions,
resource template entries excluded from _resource_map, URI collision
warnings, needs_periodic capability-aware computation, malformed JSON
primary key fallback for read_resource.

* feat: use_prompt tool, prompt catalog, and PR review hardening

New use_prompt built-in tool invokes MCP prompt templates by name,
expanding them into messages. Requires user approval (external MCP
servers). Prompt catalog injected into system message with XML
delimiters (up to 30 prompts, HTML-escaped).

Prompt listener registered in session for catalog rebuild on changes.

Addresses PR #44 review feedback:
- _init_system_messages() now uses copy-on-write (build locally,
  assign atomically) so background thread callbacks never see
  partial system messages
- sync_prompts_to_storage() serialized behind _sync_lock to prevent
  races between set_storage() (main thread) and MCP background thread
- shutdown() clears listener lists to release callback references

Docs and diagrams updated for 18 built-in tools.

* feat: granular tool policies for MCP resources, prompts, and tools

Policy evaluation now uses approval_label (falling back to func_name)
for fnmatch pattern matching, enabling fine-grained per-URI and
per-server policies:
- read_resource: mcp_resource__{normalized_uri}
- use_prompt: mcp__{server}__{prompt} (prefixed name)
- MCP tools: mcp__{server}__{tool} (was static "mcp_tool")

URI normalization resolves .. path segments to prevent traversal
bypasses in policy matching. Resource templates filtered from system
message catalog (not directly readable). use_prompt arguments
validated as dict with string coercion.

TypeScript SDK PromptTemplateInfo gains origin, mcp_server, readonly
fields. Governance docs updated with MCP policy patterns.

* feat: MCP visibility in server and console UIs

Server health endpoint includes mcp.servers, mcp.resources, mcp.prompts
counts. Server UI status bar shows magenta MCP indicator with tooltip.
Console cluster status bar shows MCP metrics with magenta LED dot.
Console node detail view shows per-node MCP summary. Console collector
aggregates MCP counts across nodes in overview.

Uses var(--magenta) design token with new --magenta-glow for theme
adaptation. ARIA roles on MCP status elements. Tooltips on console
MCP metric labels. Node MCP summary hidden on mobile (< 700px).

New diagram: 20-mcp-architecture.puml covering full MCP lifecycle
(connection, discovery, refresh, governance sync, policy, UI).

* fix: McpStatus in health schema, count properties, catalog name fidelity

Adds McpStatus model to HealthResponse (Python + TypeScript SDKs) so
typed clients see the mcp field from /health.

Addresses Copilot review feedback:
- resource_count/prompt_count properties avoid list allocation on
  /health and /metrics polls
- get_tools/resources/prompts return shallow-copied dicts to prevent
  callers from mutating internal cache
- Prompt names and arg names in system message catalog are NOT
  HTML-escaped (model must use exact strings in use_prompt calls);
  only descriptions are escaped

* fix: OpenAPI spec McpStatus + diagram approval column accuracy

Adds McpStatus schema and optional mcp field to HealthResponse in
openapi-server.json, matching the Python schema and TypeScript types.

Fixes tool pipeline diagram: math, web_fetch, web_search correctly
shown as auto-approve (not "Yes" for approval).

2026-03-12 14:49:58 -07:00

Patrick Buckley

7492816ab2

feat: governance — RBAC, tool policies, prompt templates, usage track… (#39 )

* feat: governance — RBAC, tool policies, prompt templates, usage tracking, audit logging

Add comprehensive governance layer for the admin console:

- RBAC with 15 granular permissions, 3 builtin roles (admin, operator, viewer),
  custom role CRUD, user-role assignment with privilege escalation prevention
- Tool policies with glob pattern matching, priority-ordered evaluation
  (allow/deny/ask), enforced before auto-approve in WebUI.approve_tools()
- Prompt templates with variable substitution, categories, default flag
- Usage tracking: per-LLM-request token/tool metrics, aggregated queries
  (group by day/model/user), automatic 90-day pruning via scheduler
- Audit logging: append-only event trail for all admin mutations,
  filterable/paginated queries, automatic 365-day pruning, X-Forwarded-For
  aware IP extraction
- require_permission() enforced on all 35+ admin endpoints (users, tokens,
  channels, schedules, watches, roles, orgs, policies, templates, usage, audit)
- Field allowlists on storage update methods prevent mass-assignment bugs
- Self-deletion guard on admin_delete_user, delete_user cascades user_roles
- _row_to_dict helper eliminates ~400 lines of fragile positional row mapping
- _audit_context helper deduplicates 18 instances of audit boilerplate
- Migration 008: 7 new tables, 3 builtin roles, org_id on users
- Console admin panel: 5 new tabs (Roles, Policies, Templates, Usage, Audit)
  with permission-gated visibility, 7 modal dialogs, full keyboard accessibility
- Python + TypeScript SDK methods for all governance endpoints
- 120+ new tests (1554 total)

* fix: address PR #39 review feedback

- Rebuild serialized items after policy evaluation so denied/allowed
  verdicts are reflected in tool_info/approve_request SSE payloads
- Make `since` query param optional in usage OpenAPI spec (handler
  already defaults to last 7 days)
- Add response_model=StatusResponse to DELETE role/policy/template
  and POST/DELETE role assignment endpoints in OpenAPI spec
- Add missing org_id/created/updated fields to UserRoleInfo schema
- Add missing created field to AuditEventInfo schema
- Show "no permissions" empty state instead of loading inaccessible
  tab when all admin tabs are permission-gated
- Fix "13 permissions" → "15 permissions" in architecture.md and
  security.md
- Fix import sorting in test_audit.py and test_tool_policy.py

* fix: address PR #39 round 2 review feedback

- Clear stale permissions from sessionStorage on config-token login
  (auth.js _storePermissions)
- Only trust X-Forwarded-For when behind a proxy that sets
  X-Forwarded-Proto (conditional on is_secure_request trust model)
- Thread user_id from auth into WebUI.on_status for usage events
- Add created field to TS AuditEventInfo type
- Return typed Pydantic models from all SDK governance methods instead
  of dict[str, Any] — both async and sync clients
- Validate group_by param against allowed enum in admin_usage handler
- Add deterministic secondary sort (event_id DESC) to
  list_audit_events in both SQLite and PostgreSQL backends

2026-03-10 19:32:37 -07:00

2 Commits