mirror of
https://github.com/turnstonelabs/turnstone.git
synced 2026-08-12 23:12:23 -06:00
Patrick Buckley
c339615e39
* Bound search tool output against pathological inputs
Replaces the per-line truncation with a fully bounded pipeline so the
search tool can no longer overflow the LLM context — or OOM the parent —
on minified bundles, multi-GB JSONL records, or huge result sets.
Backend:
- Prefer ripgrep when on PATH; grep is the fallback. Detection is
cached via functools.cache.
- ripgrep flags do most of the bounding natively: --max-columns 1024
+ --max-columns-preview, --max-filesize 10M, --max-count 100,
--no-config, --no-messages, plus negative globs for the same
noisy directories grep has been excluding.
- ripgrep added to the Dockerfile.
Streaming subprocess (_search_capture):
- subprocess.Popen with a streaming, byte-capped stdout read (4 MB).
Defends against single-line files (training data, minified bundles)
that would have OOM'd the previous subprocess.run capture.
- threading.Timer watchdog enforces tool_timeout even when the
pipe read is blocked in the kernel — proc.wait(timeout=…) alone
was insufficient because the read sat ahead of it.
- Stderr drained in a daemon thread to avoid pipe-deadlock when the
child writes to stderr while we're still reading stdout. Cap on
captured stderr keeps a hostile child from growing the buffer.
Tier-based formatter (_format_search_results):
- Tier 1: full path:line:content output, stream-emitted with a
running-cost short-circuit so we never materialize past the budget.
- Tier 2: K samples per file with overflow notes; K is computed
analytically from budget / file_count / avg-line-length so we hit
the right ladder rung in a single pass.
- Tier 3: per-file counts only, also budget-bounded with a tail line
reporting the omitted files. Sorted by descending count.
- Total output budget (32 KB) is well under tool_truncation, so the
head+tail _truncate_output strategy never silently drops middle
files in a search result.
Argument injection fix:
- The ripgrep arg list was missing the `--` separator that the grep
branch already had. With auto_approve on the search tool, that was
exploitable: path='--pre=COMMAND' would have made ripgrep run the
script as a per-file preprocessor and surface its stdout. Added
`--` and a regression test.
State-machine cleanup in _exec_search:
- rc < 0 (signal-killed by something other than us) now surfaces a
dedicated 'killed by signal N' message instead of being parsed as
success.
- capped + zero parsed records (e.g. one multi-MB line with no \n)
now returns a dedicated byte-cap message instead of the malformed-
output message that previously masked the real cause.
- _report_tool_result descriptions now match the returned payload
(no more 'no matches' tag on a 'malformed' payload).
Defence-in-depth on env scrub:
- RIPGREP_CONFIG_PATH, GIT_CONFIG, GIT_CONFIG_GLOBAL, GIT_CONFIG_SYSTEM
added to _EXPLICIT_SCRUB. We pass --no-config on the rg CLI today,
but if a future caller forgets the flag, an attacker who can set
one of these env vars could plant a config containing --pre=… and
recreate the same RCE shape.
Tests:
- TestSearchLineTruncation rewritten to mock _search_capture instead
of subprocess.run (the previous tests passed ChatSession kwargs
that no longer satisfy the constructor).
- TestSearchBackendSelection covers rg/grep detection and arg
construction, including the --pre flag-injection regression.
- TestSearchOutputBudget exercises Tier 1/2/3 directly.
- TestSearchCaptureStreaming spawns real Python subprocess writers
to exercise the byte-cap trim, mega-line-no-newline edge case, the
watchdog timeout when the child writes nothing, and the stderr
drain under load.
- test_env_scrub picks up the new tool-config keys.
* Address Copilot review on #473
- Budget the Tier 2/3 header up front so the formatter's emission stays
strictly within _SEARCH_OUTPUT_BUDGET. Previously the fit checks only
counted body bytes, letting the final string overflow by ~120 chars
(header + separator) and triggering _truncate_output's head+tail
dropout — exactly the shape this code was trying to avoid.
- Restore the (5, 3, 1) ladder in Tier 2: the analytical K from perf-2
is kept as a starting estimate, but if that K's actual emission
doesn't fit (the estimate ignores the header and overweights shared-
path compression) we step down through the ladder before falling
through to Tier 3. The previous one-shot K could collapse to counts-
only when 3/file or 1/file would have fit.
- Only normalise rc to 0 in the capped-output path when rc < 0 (our
SIGKILL). There's a narrow race where the child can exit naturally
between our read and our kill; preserving a non-negative rc means
rg's rc=2 ('matches found but some files had errors') no longer
silently turns into a clean success when the byte cap also fires.
- Clarify _MAX_SEARCH_LINE_LENGTH doc: the cap applies to the content
portion (after path:lineno:), not the whole emitted line.
- Add explanatory comments on the two intentional `except Exception:
pass` blocks in _search_capture (stderr drain, pipe close in the
cleanup finally) so static analysis and future readers can see the
silence is deliberate.
- Tighten the budget tests: now assert strict `<= _SEARCH_OUTPUT_BUDGET`
instead of the +512-char slack that was masking the header overflow.
- New regression tests:
- Tier 2 ladder step-down (K=5 over budget, K=3 fits, no Tier 3 fall-through)
- capped + rc=2 surfaces stderr instead of being normalised to success
- capped + rc<0 (our SIGKILL) flows through as a partial-result success
* chore(search): post-review cleanup
Follow-up to the Copilot-review fixes in 39d2aa2 — these are all small
quality items (no behaviour change, no new tests).
- q-1: collapse the Tier 2 candidates filter to a single expression.
Drops the redundant inner ``max(estimated_k, 1)`` and the unreachable
``if not candidates`` branch (the ladder ends in 1 and ``estimated_k``
is already floored at 1, so the comprehension always yields ≥ ``[1]``).
``or [...]`` is kept as defence against future ladder changes.
- q-2: update _format_search_results docstring to match the new ladder
semantics (analytical seed → step down through (5, 3, 1) from the
highest rung ≤ the estimate). The previous wording suggested every
Tier 2 attempt started at 5.
- q-3: combine the two ``from turnstone.core.session import ...``
statements in test_tier2_steps_down_ladder_before_falling_to_tier3
into a single top-of-function import (matches the surrounding tests).
- q-4: shorten the explanatory comments on the two best-effort cleanup
paths in _search_capture to one line each. Both sites now read with
the same shape ("# best-effort: pipe may be torn down by ...").
- q-5: trim the _MAX_SEARCH_LINE_LENGTH comment from 7 lines back to 3.
Keeps the load-bearing semantic (cap is on the content portion only)
and the pathological-line defence; drops the paths-aren't-bounded
parenthetical, which was background reading rather than WHY.
70 lines
2.7 KiB
Docker
70 lines
2.7 KiB
Docker
# =============================================================================
|
|
# Turnstone — Docker build with uv for reproducible, locked installs
|
|
# Single image for all services: server, console, channel, eval
|
|
# =============================================================================
|
|
|
|
FROM python:3.14-slim
|
|
|
|
LABEL org.opencontainers.image.title="turnstone" \
|
|
org.opencontainers.image.description="Multi-node AI orchestration platform"
|
|
|
|
COPY --from=ghcr.io/astral-sh/uv:0.11.8 /uv /usr/local/bin/uv
|
|
|
|
# Remove the slim image's man page exclusion so man-db has actual content
|
|
RUN rm -f /etc/dpkg/dpkg.cfg.d/docker
|
|
|
|
# System dependencies: psycopg (libpq5), developer tooling for agent workflows.
|
|
# ripgrep is the preferred backend for the search tool — natively bounds
|
|
# per-line, per-file, and per-filesize so pathological inputs (minified
|
|
# bundles, training-data JSONL with multi-MB single records) can't OOM us.
|
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y --no-install-recommends \
|
|
libpq5 git curl jq man-db manpages procps file ripgrep \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Node.js LTS (for npx-based MCP servers like @modelcontextprotocol/server-github)
|
|
COPY --from=node:24-slim /usr/local/bin/node /usr/local/bin/node
|
|
COPY --from=node:24-slim /usr/local/lib/node_modules /usr/local/lib/node_modules
|
|
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm \
|
|
&& ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/local/bin/npx
|
|
|
|
# Non-root user
|
|
RUN useradd --create-home --shell /bin/bash turnstone
|
|
|
|
WORKDIR /app
|
|
|
|
# Install dependencies first (cached layer — only re-runs when deps change)
|
|
COPY pyproject.toml uv.lock README.md LICENSE ./
|
|
RUN uv sync --frozen --no-install-project --no-dev \
|
|
--no-compile --extra all
|
|
|
|
# Install the project itself
|
|
COPY turnstone/ turnstone/
|
|
RUN uv sync --frozen --no-dev \
|
|
--no-compile --extra all
|
|
|
|
# Compile bytecode in a separate step (avoids fd exhaustion during install)
|
|
RUN python -m compileall -q .venv turnstone/
|
|
|
|
# Add venv to PATH so entry points are found
|
|
ENV PATH="/app/.venv/bin:$PATH"
|
|
|
|
# Health check script (stdlib only, no pip deps needed)
|
|
COPY docker/healthcheck.py /usr/local/bin/healthcheck.py
|
|
|
|
# Entrypoint script — runs migrations before starting
|
|
COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
|
|
|
|
# Data directory — SQLite DB is created in CWD
|
|
WORKDIR /data
|
|
RUN chown turnstone:turnstone /data
|
|
|
|
# Workspace mount point — bind-mount a host directory here
|
|
RUN mkdir -p /workspace && chown turnstone:turnstone /workspace
|
|
|
|
USER turnstone
|
|
|
|
ENTRYPOINT ["entrypoint.sh"]
|
|
|
|
# Default command (overridden per service in compose.yaml)
|
|
CMD ["turnstone-server", "--host", "0.0.0.0", "--port", "8080"]
|