2 Commits

Author	SHA1	Message	Date
Patrick Buckley	d0f5f50650	feat(node): auto-detect node capabilities via kernel interfaces (#445) ... * feat(node): auto-detect node capabilities via kernel interfaces Closes the operator-burden gap the harness shakedown surfaced — the list_nodes capability/region/role filtering surface that nodes were launching with empty. Auto-detection runs at server startup and populates ``node_metadata`` rows with sensible defaults that operators can still override via the ``[metadata]`` section of config.toml (operator-config writes win on the per-key upsert). What's detected, all from kernel interfaces (no userspace binaries on PATH — works the same way regardless of whether nvidia-smi / rocm-smi / lspci is installed): - ``gpu_count`` / ``gpu_vendor`` / ``gpu_vendors`` / ``gpus`` — walks ``/sys/class/drm/cardN/device/{vendor,device}`` and decodes PCI vendor IDs to friendly names (NVIDIA / AMD / Intel / Apple). Heterogeneous-GPU nodes get the first KNOWN vendor in the flat ``gpu_vendor`` key — never ``"unknown"`` when known vendors are present — so a coord filtering on ``gpu_vendor=nvidia`` matches nodes whose first card happened to be exotic. - ``memory_gb`` — reads ``/proc/meminfo``, rounds GiB down so ``filters={"memory_gb": 32}`` doesn't match a 31.5 GiB node. - ``cpu_model`` — first ``model name`` line from ``/proc/cpuinfo``. - ``cloud_provider`` / ``cloud_region`` / ``cloud_zone`` / ``cloud_instance_type`` / ``cloud_instance_id`` — DMI sysfs identifies the cloud provider from BIOS/SMBIOS strings (no network call) and only THEN does the IMDS probe fire. Baremetal hosts pay zero startup latency on the cloud path. Hardening highlights: - IMDS probes target the link-local IP literal ``169.254.169.254`` for AWS, GCP, AND Azure — no DNS-resolvable hostname for any vendor, so a host with attacker-controlled DNS can't redirect the probe even when its DMI claims a cloud provider. - Response bodies capped at 64 KiB on read; per-field strings capped at 256 chars and stripped of control characters before persistence. Stops a hostile IMDS responder from spraying multi-megabyte / newline-injected payloads into ``node_metadata`` and from there into coord-LLM ``list_nodes`` context. - ``isinstance(doc, dict)`` guards on every JSON IMDS response so a non-conformant body (list / scalar / null) returns clean ``{}`` instead of raising. - ``collect_node_info()`` runs via ``asyncio.to_thread`` from the server's lifespan handler so the IMDS probe latency never blocks the event loop. - GCP fans the three zone/machine-type/id probes concurrently so a misidentified host's worst case is one timeout window (~1 s) instead of three (~3 s). - Operator opt-out via ``TURNSTONE_AUTO_CLOUD_METADATA=0`` skips the IMDS phase entirely; the DMI-derived ``cloud_provider`` still populates because that's a kernel interface. Tests: 4807 pass (+11 net, 73 in test_node_info.py). Ruff + mypy clean on every modified file. New tests pin the heterogeneous-GPU flat-key fix, the IMDS hardening (non-dict JSON, control-char sanitisation, body cap, per-field cap), and the GCP IP-literal property. * fix(node): filter synthetic display adapters + per-vendor GPU flags PR review on 68cb0ab flagged two real issues with the GPU surface: 1. Hyper-V synthetic display adapter (vendor 0x1414, device 0x06) registers a /sys/class/drm/cardN entry on Linux but is NOT a compute GPU. A CI runner reproduced this and came back with gpu_count=1 on a CPU-only VM. Same hazard for AWS Nitro VGA, QEMU virtio-gpu, and any other hypervisor synthetic display adapter. Fix: ``_detect_gpus`` filters DRM cards by PCI vendor against the GPU allow-list (NVIDIA / AMD / Intel / Apple); cards from other vendors are skipped entirely. Operators with exotic accelerators that don't match any known vendor can still set ``gpu_count`` + the relevant flags via [metadata] config. 2. ``gpu_vendor`` (singular flat key) was sorted-alphabetical-first of the unique known vendors. list_nodes() filtering does exact-equality JSON matching, so a mixed AMD+NVIDIA node ended up with ``gpu_vendor=amd`` and was invisible to a coord filtering ``gpu_vendor=nvidia``. Fix: drop the singular key entirely; emit per-vendor booleans (``gpu_has_nvidia=true``, ``gpu_has_amd=true``) so multi-vendor nodes match EITHER vendor. Also add ``has_gpu=true`` for "any compute GPU at all" filtering. Tests: 4809 pass (+2 net). New / updated tests pin both behaviors: - _detect_gpus: Hyper-V synthetic + arbitrary unknown-vendor card now filter out; mixed-known-and-unknown keeps only the known card. - collect_node_info integration: multi-vendor node has both ``gpu_has_amd`` and ``gpu_has_nvidia`` set; ``gpu_vendor`` (singular) is asserted absent so a future regression that re-introduces it fails loudly. Ruff + mypy clean.	2026-04-28 13:49:31 -07:00
Patrick Buckley	24f59a6c53	feat: add per-node metadata with auto-collection, admin API, and cons… (#318) ... * feat: add per-node metadata with auto-collection, admin API, and console UI Adds a normalized node_metadata table for structured per-node key/value metadata with source tracking (auto/user/config). Auto-populated fields (hostname, OS, arch, interfaces, cpu_count) are collected at server startup via stdlib; user-defined fields are managed through the admin API, CLI, or config.toml [metadata] section. Storage: migration 035, 7 new protocol methods (get, get_all, set, set_bulk, delete, delete_by_source, filter), both SQLite and PostgreSQL backends. Filtering uses single-query GROUP BY/HAVING for efficiency. Console API: GET/PUT/DELETE endpoints under /admin/nodes/{node_id}/metadata with auto-source protection. cluster_nodes gains meta.* query param filtering; cluster_node_detail attaches metadata to responses. Frontend: new Nodes admin tab with collapsible per-node sections, inline add form, delete with confirmation. Read-only metadata panel in node detail drill-down. Proper design token usage, accessibility (ARIA, keyboard nav, screen reader labels), and mobile responsiveness. CLI: turnstone-admin list-node-metadata, set-node-metadata, and delete-node-metadata subcommands. 64 tests (25 storage, 19 node_info, 20 existing unaffected). * fix: resolve CI typecheck and test failures - Fix mypy error: use %-style format string instead of structlog kwargs for standard Logger.warning() in console server - Fix test_get_nodes assertion to include new node_ids=None parameter - Add debug logging to _collect_interfaces empty except block * fix: address Copilot review feedback on node metadata - Clear stale auto/config metadata before upserting on startup - Wrap metadata filter in try/except with graceful fallback - Add metadata field to NodeDetailResponse schema - Use _VALID_NODE_ID regex for consistent node_id validation - Defensive JSON decode in admin_get_node_metadata - Switch to read_json_or_400 and require_storage_or_503 helpers - Add SetNodeMetadataValueRequest for single-key PUT endpoint - Add bulk GET /admin/node-metadata endpoint (replaces N+1 fetches) - Update frontend to use single bulk metadata fetch * feat: add admin.nodes permission scope for node metadata - Add admin.nodes to builtin-admin role via migration 035 - Switch all node metadata handlers from admin.settings to admin.nodes - Register admin.nodes in the admin panel permission set - Node detail metadata panel fetches from cluster endpoint (no admin permission needed) instead of admin endpoint * fix: address second round of Copilot feedback - Replace inline onclick handlers with data-* attributes and event delegation to prevent JS string context XSS - Move NodeMetadataEntry before NodeDetailResponse and use it as the typed metadata field (was list[dict[str, Any]]) - Clean up config metadata on shutdown (was only cleaning auto)	2026-04-06 03:08:19 -07:00