Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging (#20)

* Add pluggable storage backend (SQLite + PostgreSQL) and deployment packaging

Database abstraction: StorageBackend protocol with 21 methods, SQLAlchemy Core
schema, SQLite backend (FTS5), PostgreSQL backend (tsvector/ILIKE), Alembic
migrations, singleton registry. memory.py reduced to thin facade. Session.py
open_db() calls replaced with generic KV methods. [database] config section
with env var support.

Deployment: Docker Compose production profile with PostgreSQL, Dockerfile with
postgres extras and migration entrypoint, Helm chart with bitnami subcharts,
Terraform AWS ECS/Fargate module with RDS + ElastiCache + ALB.

39 new storage tests (934 total). mypy strict clean. Docs and diagrams updated.

* Address PR #20 review feedback (16 items)

- Backends only call create_all() when Alembic migrations are disabled
- Helm configmap uses correct TURNSTONE_DB_BACKEND env var; DB URL
  constructed via env expansion with secret reference instead of ConfigMap
- Migration errors fail fast for PostgreSQL (only non-fatal for SQLite)
- save_memory/delete_memory wrapped in exception handling like other facade fns
- pool_size passed through from config/env to init_storage() in cli + server
- Terraform: DB URL moved to Secrets Manager, auth enabled flag set,
  optional TLS listeners with certificate_arn, Redis transit encryption on
- Docker entrypoint no longer suppresses migration output
- Diagram fixes: removed StaticPool claim, removed non-existent migration ref
- compose.yaml/README: clarified production profile requires DB env vars
This commit is contained in:
Patrick Buckley
2026-03-03 22:57:34 -08:00
committed by GitHub
parent 5ee539c983
commit 2b58c127b1
64 changed files with 4083 additions and 847 deletions
+11 -2
View File
@@ -25,22 +25,31 @@ FROM python:3.13-slim
LABEL org.opencontainers.image.title="turnstone" \
org.opencontainers.image.description="Multi-node AI orchestration platform"
# System dependencies for psycopg (PostgreSQL client library)
RUN apt-get update && apt-get install -y --no-install-recommends libpq5 \
&& rm -rf /var/lib/apt/lists/*
# Non-root user
RUN useradd --create-home --shell /bin/bash turnstone
# Install the wheel with all optional extras (redis for mq/console/sim)
# Install the wheel with all optional extras
COPY --from=builder /build/wheels/*.whl /tmp/wheels/
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim]" \
RUN pip install --no-cache-dir "$(ls /tmp/wheels/*.whl)[mq,console,sim,postgres]" \
&& rm -rf /tmp/wheels
# Health check script (stdlib only, no pip deps needed)
COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
# Entrypoint script — runs migrations before starting
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
# Data directory — SQLite DB is created in CWD
WORKDIR /data
RUN chown turnstone:turnstone /data
USER turnstone
ENTRYPOINT ["entrypoint.sh"]
# Default command (overridden per service in compose.yaml)
CMD ["turnstone-server", "--host", "0.0.0.0", "--port", "8080"]