Commit Graph

10 Commits

Author SHA1 Message Date
Patrick Buckley a315cabe71 chore: polish — remove dead code, update diagrams and docs
Remove stale Redis/Bridge/MQ references found via vulture scan and
manual grep:
- bot.py docstring: remove Redis MQ reference
- server.py trusted_sources: remove "bridge"
- tls.py docstring: remove "bridge" from service list

Delete 4 obsolete diagram pairs (puml + png):
- 06-mq-protocol, 07-message-routing, 08-redis-key-schema,
  10-simulator-architecture

Update 7 diagrams to reflect direct HTTP architecture:
- system-context, package-structure, workstream-states,
  console-data-flow, deployment, channel-architecture,
  settings-architecture

Redraw architecture-overview.svg: Console router replaces Redis MQ,
direct SSE data plane, hash ring routing.
2026-03-30 20:30:05 -07:00
Patrick Buckley 62ff3217d0 fix: TLS Docker end-to-end testing fixes (#185)
* fix: TLS Docker end-to-end testing fixes

Fixes discovered during Docker Compose TLS integration testing:

- Dockerfile: use --extra all (prevents missing optional deps)
- lacme 1.0.3: fixes CACertificateIssued event logging crash
- chmod PermissionError: guard for Docker volume mounts
- socket import: moved to top of main() (was inside TLS conditional,
  caused NameError in _default_node_id)
- redis.SSLConnection: ConnectionPool needs explicit connection_class,
  not ssl=True (which only works on Redis() directly)
- Empty redis password: pass None instead of "" to avoid AUTH error
- TURNSTONE_CONSOLE_URL: env var for Docker service discovery
  (0.0.0.0 bind address isn't reachable from other containers)
- HTTP01Handler: ACME client needs a challenge handler even when
  server auto-approves
- Docker overlay: tls-init as root with chmod, Redis conditional
  password, console Redis TLS flags, TURNSTONE_CONSOLE_URL

* feat: full mTLS end-to-end with lacme 1.0.4

Completes the mTLS chain across all services:

lacme 1.0.4:
- Dual EKU certs (serverAuth + clientAuth) — fixes mTLS rejection
- Configurable CA name (name="turnstone") — consistent store key

Bootstrap CA import:
- Console imports bootstrap CA from /certs volume on first boot
- Single trust root: bootstrap CA → console → all service certs

Bridge mTLS:
- TLSClient init when TURNSTONE_TLS_ENABLED set
- Auto-upgrades server URL from http:// to https://
- SSLContext passed to all 3 httpx clients via verify=

Console collector mTLS:
- upgrade_tls() method replaces httpx client with mTLS context
- Called in lifespan after cert issuance alongside proxy upgrade
- Fixes "Failed to poll node" when server serves HTTPS

Docker overlay:
- TURNSTONE_TLS_SANS on all services (Docker service names as SANs)
- TURNSTONE_TLS_ENABLED on bridge
- Channel service with Redis TLS flags
- TURNSTONE_CONSOLE_URL for service discovery
- Server healthcheck disabled (mTLS healthcheck deferred)
- Redis conditional password from env

Verified end-to-end: bootstrap → console CA → server HTTPS →
bridge mTLS → Redis TLS → channel Redis TLS → console collector
polls server over mTLS → workstream creation works through bridge

* fix: lint + copilot feedback on TLS Docker e2e

- SIM105: contextlib.suppress(PermissionError) for chmod
- F401: remove unused get_storage import in bridge
- Redis healthcheck: pass password when REDIS_PASSWORD is set

* fix: sort imports in admin.py and bridge.py

* fix: tls-init key permissions, healthcheck env, collector race

- tls-init: add set -e, chown to turnstone:turnstone with restrictive
  perms (keys 0600, certs 0640, dirs 0750) instead of world-readable
- Redis healthcheck: use container runtime $$REDIS_PASSWORD instead of
  Compose-time interpolation for consistency with --requirepass block
- collector upgrade_tls(): don't close old httpx client while concurrent
  poll threads may still be using it — let GC handle cleanup
2026-03-26 14:24:44 -07:00
Patrick Buckley b086390558 fix: TLS deferred work — wiring, security, tests, Docker overlay (#184)
* fix: TLS deferred work — wiring, security, tests, Docker overlay

Security fixes:
- PostgreSQL SSL: validate sslmode against known values, urlencode
  all params to prevent URL injection
- ConfigStore env seeding: removed redundant type coercion, delegate
  to validate_value() which handles all coercion correctly

Functional wiring:
- Database SSL: init_storage() passes SSL params to PostgreSQL URL
- Server: env var fallbacks for DB SSL (TURNSTONE_DB_SSLMODE etc.)
- Proxy mTLS: re-create proxy clients after TLS cert issuance
- Channel gateway: --ssl-certfile/keyfile/ca-certs CLI args, HTTPS
  advertise URL when SSL configured
- ConfigStore env seeding: TURNSTONE_{SECTION}_{KEY} seeds on first boot
- Console deregistration on shutdown (with debug logging)

Specs, tests, Docker:
- OpenAPI: 5 TLS admin endpoints in console_spec.py
- Auth enforcement test (401 without auth)
- SDK ValueError test (mismatched cert/key)
- Docker overlay: TURNSTONE_TLS_ENABLED, bridge --redis-tls, Redis
  healthcheck with client cert
- Removed stale type:ignore comments (lacme 1.0.2 type stubs)

* review: address copilot feedback on TLS deferred work

- ConfigStore env seeding: use config_store.set() instead of
  storage.set_system_setting() (correct API, updates cache)
- Remove unused defn variable (iterate SETTINGS keys only)
- Fix structlog call-arg error (positional args, not kwargs)
- Channel gateway: validate cert+key provided together
- Restore type:ignore[no-any-return] for CI mypy (lacme 1.0.2
  type stubs not in CI's mypy overrides yet)

* fix: rename _VALID_SSLMODES to lowercase (N806)
2026-03-25 22:43:35 -07:00
Patrick Buckley d08a57dfc2 feat: SDK TLS support, Docker Compose overlay, TLS docs (#183)
Python SDK:
- ca_cert, client_cert, client_key on all 4 client classes
- ValueError if only one of client_cert/client_key provided
- Passed to httpx verify=/cert=

TypeScript SDK:
- TlsOptions type exported (zero runtime code)
- Fix picomatch vulnerability (npm audit fix)

Docker Compose:
- deploy/docker-compose.tls.yml overlay with tls-init bootstrap
- Notes it's an overlay requiring a base compose file

Documentation:
- docs/tls.md: architecture, config, CLI, SDK examples, troubleshooting
- Fixed package name (@turnstone/sdk), Node.js 18+ note
2026-03-25 20:43:34 -07:00
Patrick Buckley 10165bb8a1 feat: add OpenShell sandbox policy for turnstone-server (#128)
* feat: add OpenShell sandbox policy for turnstone-server

Curated policy for running turnstone-server inside an OpenShell sandbox
with kernel-enforced security boundaries (Landlock, netns, seccomp).

- Filesystem: workdir read-write, /usr+/etc read-only, /tmp+/dev/null
  read-write, Landlock best_effort compatibility
- Network: default-deny with allowlisted LLM APIs (OpenAI, Anthropic),
  Tavily, skills.sh, GitHub (read-only L7), MCP registry (read-only L7),
  Redis localhost, package registries, curated web_fetch domains
- Git: L7-enforced read-only (info/refs + git-upload-pack only)
- Process: privilege drop to sandbox:sandbox
- Inference routing template for credential isolation (real API keys
  never enter the sandbox, resolved at proxy layer)

* fix: address PR #128 review feedback + add integration guide

Review fixes:
- Use python3 (not python) in usage examples to match binary allowlist
- Fix network_policy → network_policies in comment
- Remove /usr/bin/git from github_api (git uses github.com not
  api.github.com; already covered by git_operations policy)
- Remove pip/uv from bash_network_tools (package_registries already
  covers their PyPI access; no need for StackOverflow/Wikipedia reach)
- Restructure routes.yaml so commented blocks are indented under
  routes: key (uncomment without restructuring YAML)

New: docs/openshell.md covering policy customization, inference routing,
domain allowlisting, MCP subprocess inheritance, and the dual-layer
security model.
2026-03-18 18:09:47 -07:00
renovate[bot] f494553020 chore(deps): update helm release redis to v25 (#93)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:41 -07:00
renovate[bot] c766c81f25 chore(deps): update helm release postgresql to v18 (#92)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:58:38 -07:00
renovate[bot] 387ef06da7 chore(deps): update helm release redis to ~20.13.0 (#89)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:13 -07:00
renovate[bot] e4e2200c33 chore(deps): update helm release postgresql to ~16.7.0 (#88)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2026-03-15 15:56:11 -07:00
Patrick Buckley 2b58c127b1 Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)
* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
2026-03-03 22:57:34 -08:00