Commit Graph

11 Commits

Author SHA1 Message Date
Patrick Buckley 62ff3217d0 fix: TLS Docker end-to-end testing fixes (#185)
* fix: TLS Docker end-to-end testing fixes

Fixes discovered during Docker Compose TLS integration testing:

- Dockerfile: use --extra all (prevents missing optional deps)
- lacme 1.0.3: fixes CACertificateIssued event logging crash
- chmod PermissionError: guard for Docker volume mounts
- socket import: moved to top of main() (was inside TLS conditional,
  caused NameError in _default_node_id)
- redis.SSLConnection: ConnectionPool needs explicit connection_class,
  not ssl=True (which only works on Redis() directly)
- Empty redis password: pass None instead of "" to avoid AUTH error
- TURNSTONE_CONSOLE_URL: env var for Docker service discovery
  (0.0.0.0 bind address isn't reachable from other containers)
- HTTP01Handler: ACME client needs a challenge handler even when
  server auto-approves
- Docker overlay: tls-init as root with chmod, Redis conditional
  password, console Redis TLS flags, TURNSTONE_CONSOLE_URL

* feat: full mTLS end-to-end with lacme 1.0.4

Completes the mTLS chain across all services:

lacme 1.0.4:
- Dual EKU certs (serverAuth + clientAuth) — fixes mTLS rejection
- Configurable CA name (name="turnstone") — consistent store key

Bootstrap CA import:
- Console imports bootstrap CA from /certs volume on first boot
- Single trust root: bootstrap CA → console → all service certs

Bridge mTLS:
- TLSClient init when TURNSTONE_TLS_ENABLED set
- Auto-upgrades server URL from http:// to https://
- SSLContext passed to all 3 httpx clients via verify=

Console collector mTLS:
- upgrade_tls() method replaces httpx client with mTLS context
- Called in lifespan after cert issuance alongside proxy upgrade
- Fixes "Failed to poll node" when server serves HTTPS

Docker overlay:
- TURNSTONE_TLS_SANS on all services (Docker service names as SANs)
- TURNSTONE_TLS_ENABLED on bridge
- Channel service with Redis TLS flags
- TURNSTONE_CONSOLE_URL for service discovery
- Server healthcheck disabled (mTLS healthcheck deferred)
- Redis conditional password from env

Verified end-to-end: bootstrap → console CA → server HTTPS →
bridge mTLS → Redis TLS → channel Redis TLS → console collector
polls server over mTLS → workstream creation works through bridge

* fix: lint + copilot feedback on TLS Docker e2e

- SIM105: contextlib.suppress(PermissionError) for chmod
- F401: remove unused get_storage import in bridge
- Redis healthcheck: pass password when REDIS_PASSWORD is set

* fix: sort imports in admin.py and bridge.py

* fix: tls-init key permissions, healthcheck env, collector race

- tls-init: add set -e, chown to turnstone:turnstone with restrictive
  perms (keys 0600, certs 0640, dirs 0750) instead of world-readable
- Redis healthcheck: use container runtime $$REDIS_PASSWORD instead of
  Compose-time interpolation for consistency with --requirepass block
- collector upgrade_tls(): don't close old httpx client while concurrent
  poll threads may still be using it — let GC handle cleanup
2026-03-26 14:24:44 -07:00
renovate[bot] b20548583d chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.11.1 (#176)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-25 11:57:44 -07:00
Patrick Buckley a07172b0c0 fix: include ddg extra in Docker image for free web search fallback 2026-03-23 15:23:12 -07:00
renovate[bot] e0d17e0f99 chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.10.12 2026-03-23 12:55:08 -07:00
renovate[bot] 4e4ae2a91d chore(deps): update ghcr.io/astral-sh/uv docker tag to v0.10.11 2026-03-18 13:20:47 -07:00
Patrick Buckley c11991819e fix: apt-get upgrade in Dockerfile to resolve CVE-2026-0861
Trivy scan fails on HIGH for libc-bin/libc6 (2.41-12+deb13u1).
The fix (2.41-12+deb13u2) is available in Debian repos but the
base python:3.14-slim image hasn't been rebuilt yet. Adding
apt-get upgrade pulls in all pending security patches at build time.
2026-03-15 16:40:03 -07:00
renovate[bot] 8b94d553e4 chore(deps): update docker images (#87)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:55:53 -07:00
Patrick Buckley 22402e89de feat: add dependency management with Renovate, uv.lock, and security … (#83)
* feat: add dependency management with Renovate, uv.lock, and security scanning

Adds automated dependency update detection and vulnerability scanning
across all dependency layers (Python, vendored JS, TypeScript SDK, Docker,
GitHub Actions).

- Renovate config with 10 package groups and custom regex managers for
  vendored JS (KaTeX, Highlight.js, Mermaid) tracking via npm registry
- uv.lock for reproducible builds (80 packages)
- Dockerfile switched to uv sync --frozen with layer caching
- CI: pip-audit (via lock file), npm audit, lock-check jobs
- CI: lint job uses pre-commit for ruff version consistency
- Docker security scan workflow (weekly Trivy, HIGH/CRITICAL)
- Helper script for vendored JS library updates

* fix: resolve CI failures and address review feedback

- Update pre-commit hooks: ruff v0.9.10 -> v0.15.6 (fixes deprecated
  UP038 rule), mypy v1.14.1 -> v1.19.1
- Add per-file-ignore for N802 on sandbox.py (ast visitor convention)
- Fix pip-audit: install into uv venv so uv run can find it
- Pin uv-version in CI to match lock file generator (0.9.18)
- Upgrade vitest ^2.0 -> ^4.1 to fix esbuild GHSA-67mh-4wv8-2f99
- Vendored JS script: use grep -rl for auto-discovery of version refs
  (catches docs/architecture.md), fix LICENSE comment, portable grep
2026-03-15 15:46:48 -07:00
Patrick Buckley a6e929b0a0 Add channel integrations with Discord adapter and atomic session resu… (#24)
* Add channel integrations with Discord adapter and atomic session resume (#24)

Bidirectional channel adapter framework connecting external messaging
platforms to turnstone workstreams via Redis MQ. Discord ships as the
first adapter; the protocol supports future Slack/Teams integrations.

Channel framework:
- ChannelAdapter protocol and ChannelRouter for channel↔workstream mapping
- AsyncRedisBroker with single dispatch loop and per-channel ordered workers
- channel_routes table (migration 003) for persistent route storage
- 9 new StorageBackend methods (4 channel_user + 5 channel_route CRUD)
- Unified turnstone-channel gateway entry point, loads adapters by config
- Message chunking, approval formatting, plan review formatting

Discord adapter:
- discord.py v2.4+ bot with thread-per-@mention model
- Slash commands: /link (modal), /unlink, /ask, /status, /close
- Persistent button views for tool approval and plan review
- Streaming responses via edit-in-place (1.5s interval)
- Stale route detection and atomic session resume via resume_session field
- SessionResumedEvent confirmation back to channel
- Auto-approve support (blanket + per-tool list)

Atomic session resume:
- resume_session field on CreateWorkstreamMessage for single-request resume
- Server resumes session during POST /v1/api/workstreams/new atomically
- Bridge emits SessionResumedEvent to per-workstream channel
- WorkstreamCreatedEvent extended with resumed/session_id/message_count
- Server UI dashboardResumeSession simplified to single request
- Pruned sessions fall back gracefully to fresh start

Service auth:
- Bridge and console auto-mint service JWTs from TURNSTONE_JWT_SECRET
- Bridge: approve scope (1 week). Console collector: read. Proxy: write.

Console admin:
- Channels tab with per-user view, force-link modal, unlink
- 3 admin API endpoints for channel user management
- Styled confirm modals replacing browser confirm() dialogs

Bug fixes:
- AsyncRedisBroker: replaced per-channel listener tasks with single
  dispatch loop + per-channel queue workers (fixes message stealing race)
- Bridge: approval/plan review dedup guard prevents SSE reconnect duplicates
- Bridge: _active_sends tracked for initial messages (fixes missing
  TurnCompleteEvent and unfinalized streaming messages)
- Bridge: HTTP calls moved outside lock scope in approval handlers
- Bridge: _handle_send cleans up _active_sends on HTTP/server errors
- Formatter: reads server SSE format (func_name/preview) with fallback

Docs, SDK, tests:
- docs/channels.md setup guide, architecture diagram 16
- Updated api-reference.md, architecture.md, console.md, docker.md
- Python SDK: resume_session param on create_workstream (async + sync)
- TypeScript SDK: updated CreateWorkstreamRequest/Response interfaces
- OpenAPI schema: resume_session request, resumed/message_count response
- 91 new tests (19 storage, 15 broker, 22 protocol, 6 routing,
  18 discord, 12 resume flow) — 1120 total passing

* Fix CI lint/typecheck failures and address Copilot review feedback (#24)

Lint: fix import ordering, remove unused imports, use contextlib.suppress.
Mypy: explicit postgresql dialect import, add discord module overrides for
optional-dependency CI environments.
Copilot: fix double-escaping in admin confirm modals, return resolved
session_id from server resume response, fix channel_routes diagram schema,
use atomic setdefault for routing locks, add post-insert race guard in
admin channel create, support SSE format in auto-approve check, update
identity linking note in architecture diagram.

* Fix remaining mypy call-arg errors for discord.py optional dependency

Add type: ignore[call-arg] on Modal(title=) and Cog(name=) class
definitions that fail when discord.py is not installed in CI.
2026-03-04 13:02:58 -08:00
Patrick Buckley 2b58c127b1 Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
2026-03-03 22:57:34 -08:00
Patrick Buckley 0d6252dd7d Initial commit — turnstone multi-node AI orchestration platform. 2026-03-02 00:33:37 -08:00