Files
turnstone/tests
Patrick Buckley 7b8cc157f7 fix(web): preserve block structure in strip_html, remove ReDoS risk
strip_html deleted every HTML tag with no separator, gluing paragraphs,
headings, list items, and table cells into a structureless run of text
("<p>a</p><p>b</p>" -> "ab"). This degrades web_fetch, which feeds the
cleaned page to a summarising agent — and it flattens the structure any
downstream chunking/retrieval would rely on.

Block-level tags and <br> now become newlines so structure survives
("<p>a</p><p>b</p>" -> "a\n\nb"); inline tags are still dropped.

The conversion is a single linear tag scan: one pass over `<[^>]++>` with
a possessive quantifier, dispatching each tag name against a frozenset.
This replaces three full-document passes plus a 24-way alternation, and:

- Removes catastrophic backtracking (ReDoS). The earlier `<\s*/?\s*` and
  `<\s*br\s*/?\s*>` patterns were quadratic on '<' + a long whitespace
  run (~2s at 4k chars); the scan is now linear (~3ms at 1M chars) on the
  untrusted, up-to-10MB web_fetch input. The possessive quantifier also
  neutralises the pre-existing quadratic in the old `<[^>]+>` pass.
- Matches <br> carrying attributes (e.g. `<br clear="all">`), which the
  first cut missed.

Tests cover block separation, inline-tag joining, uppercase tags, <br>
with attributes, lookalike tag names, and a pathological-whitespace
regression guard.

Note (pre-existing, not changed here): in _exec_web_fetch the 10 MB cap is
applied after strip_html, so the stripper sees the full fetched body. With
the scan now linear this is no longer a CPU concern; capping the raw input
before stripping remains a worthwhile defence-in-depth follow-up.
2026-05-30 01:11:48 -07:00
..