ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

feat: add vision/image support to read_file tool (#33)

Browse Source 
* feat: add vision/image support to read_file tool

read_file now detects image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO)
and returns base64-encoded content parts for vision-capable models.
Non-vision models receive a text description instead. A new
supports_vision flag on ModelCapabilities gates the feature, with
config.toml [models.*.capabilities] overrides for local models
(vLLM, llama.cpp, NIM).

* fix: address PR review feedback

- Discard _read_files on no-vision OSError path, include exception detail
- Discard _read_files on oversized image error (not a successful read)
- Validate capabilities type from config.toml (reject non-dict)
- Clarify tool description re: vision behavior and offset/limit scope
- Remove unused os import in tests, fix import sort order
- Handle list content (image tool results) in eval.py tool result loop
This commit is contained in:
Patrick Buckley
2026-03-08 23:43:42 -07:00
committed by GitHub
parent cc9afe94cd
commit 6cc1b3a5bd
18 changed files with 556 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tools.md
+6 -4
View File
@@ -189,15 +189,17 @@ Execute a bash command and return stdout + stderr.
### read_file
Read the contents of a file, returning numbered lines.
Read the contents of a file, returning numbered lines for text files or
base64-encoded image data for supported image formats.
| Parameter | Type | Required | Description |
|-----------|---------|----------|-------------|
| `path` | string | yes | Absolute or relative file path. |
| `offset` | integer | no | Line number to start from (1-based, default: 1). |
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. |
| `offset` | integer | no | Line number to start from (1-based, default: 1). Text files only. |
| `limit` | integer | no | Maximum number of lines to read. Omit for full file. Text files only. |
- **What it does**: Reads the file and returns content with line numbers. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **What it does**: For text files, reads and returns content with line numbers. For image files (PNG, JPEG, GIF, WebP, BMP, TIFF, ICO), returns image data as multi-part content when the model supports vision, or a text description when it does not. SVG files are read as text. Images larger than 4 MB are rejected. Must be called before `edit_file` on the same path (the session tracks which files have been read).
- **Vision support**: Controlled by `ModelCapabilities.supports_vision`. All commercial OpenAI and Anthropic models have vision enabled. Local models (vLLM, llama.cpp, NIM) default to off — enable via `[models.*.capabilities] supports_vision = true` in config.toml.
- **Auto-approve**: Yes.
- **Agent availability**: `agent` and `task_agent`.