Patrick Buckley
7b8cc157f7
fix(web): preserve block structure in strip_html, remove ReDoS risk
...
strip_html deleted every HTML tag with no separator, gluing paragraphs,
headings, list items, and table cells into a structureless run of text
("<p>a</p><p>b</p>" -> "ab"). This degrades web_fetch, which feeds the
cleaned page to a summarising agent — and it flattens the structure any
downstream chunking/retrieval would rely on.
Block-level tags and <br> now become newlines so structure survives
("<p>a</p><p>b</p>" -> "a\n\nb"); inline tags are still dropped.
The conversion is a single linear tag scan: one pass over `<[^>]++>` with
a possessive quantifier, dispatching each tag name against a frozenset.
This replaces three full-document passes plus a 24-way alternation, and:
- Removes catastrophic backtracking (ReDoS). The earlier `<\s*/?\s*` and
`<\s*br\s*/?\s*>` patterns were quadratic on '<' + a long whitespace
run (~2s at 4k chars); the scan is now linear (~3ms at 1M chars) on the
untrusted, up-to-10MB web_fetch input. The possessive quantifier also
neutralises the pre-existing quadratic in the old `<[^>]+>` pass.
- Matches <br> carrying attributes (e.g. `<br clear="all">`), which the
first cut missed.
Tests cover block separation, inline-tag joining, uppercase tags, <br>
with attributes, lookalike tag names, and a pathological-whitespace
regression guard.
Note (pre-existing, not changed here): in _exec_web_fetch the 10 MB cap is
applied after strip_html, so the stripper sees the full fetched body. With
the scan now linear this is no longer a CPU concern; capping the raw input
before stripping remains a worthwhile defence-in-depth follow-up.
2026-05-30 01:11:48 -07:00
..
2026-03-02 00:33:37 -08:00
2026-04-30 23:20:14 -07:00
2026-05-06 16:16:57 -07:00
2026-04-28 10:27:26 -07:00
2026-05-09 02:45:13 -07:00
2026-05-04 22:00:23 -07:00
2026-05-05 15:27:14 -07:00
2026-05-19 08:09:33 -07:00
2026-05-29 21:11:16 -07:00
2026-05-06 23:29:08 -07:00
2026-04-26 22:14:22 -07:00
2026-05-28 17:14:50 -07:00
2026-04-27 22:51:31 -07:00
2026-04-26 22:14:22 -07:00
2026-05-28 21:02:51 -07:00
2026-03-13 21:21:09 -07:00
2026-04-04 12:26:10 -07:00
2026-05-24 17:49:27 -07:00
2026-05-23 14:20:36 -07:00
2026-04-18 05:49:54 -07:00
2026-04-18 05:49:54 -07:00
2026-03-30 20:30:05 -07:00
2026-04-18 05:49:54 -07:00
2026-04-18 05:49:54 -07:00
2026-03-23 11:11:40 -07:00
2026-05-12 13:15:09 -07:00
2026-05-11 01:06:47 -07:00
2026-04-30 23:20:14 -07:00
2026-04-30 23:20:14 -07:00
2026-05-24 18:31:23 -07:00
2026-04-16 11:08:26 -07:00
2026-05-19 08:09:33 -07:00
2026-05-09 17:06:47 -07:00
2026-05-11 01:06:47 -07:00
2026-04-27 21:52:57 -07:00
2026-04-16 13:30:25 -07:00
2026-04-18 19:02:52 -07:00
2026-05-24 18:31:23 -07:00
2026-05-09 17:06:47 -07:00
2026-05-11 16:42:07 -07:00
2026-05-29 14:40:10 -07:00
2026-04-30 23:20:14 -07:00
2026-05-11 01:06:47 -07:00
2026-05-28 21:02:51 -07:00
2026-04-24 16:56:06 -07:00
2026-05-11 01:06:47 -07:00
2026-05-29 21:11:16 -07:00
2026-04-24 16:56:06 -07:00
2026-05-06 12:02:27 -07:00
2026-05-28 17:14:50 -07:00
2026-04-16 21:40:50 -07:00
2026-05-26 17:07:10 -07:00
2026-03-13 21:21:09 -07:00
2026-04-04 17:53:20 -07:00
2026-03-02 00:33:37 -08:00
2026-05-04 00:46:43 -07:00
2026-05-29 21:11:16 -07:00
2026-03-03 22:57:34 -08:00
2026-05-24 18:31:23 -07:00
2026-05-24 18:31:23 -07:00
2026-04-04 23:44:29 -07:00
2026-05-29 11:17:12 -07:00
2026-05-29 17:27:44 -07:00
2026-05-30 01:11:48 -07:00
2026-05-24 17:49:27 -07:00
2026-05-06 12:02:27 -07:00
2026-04-25 22:52:39 -07:00
2026-04-05 01:53:33 -07:00
2026-04-05 01:53:33 -07:00
2026-05-18 21:00:24 -07:00
2026-05-09 15:38:16 -07:00
2026-03-04 06:47:30 -08:00
2026-03-02 16:55:12 -08:00
2026-05-05 15:27:14 -07:00
2026-05-12 13:15:09 -07:00
2026-05-12 13:15:09 -07:00
2026-05-07 13:59:50 -07:00
2026-05-04 22:00:23 -07:00
2026-05-04 22:00:23 -07:00
2026-05-06 11:29:25 -07:00
2026-05-04 22:00:23 -07:00
2026-05-07 13:59:50 -07:00
2026-05-07 13:59:50 -07:00
2026-05-12 13:15:09 -07:00
2026-05-04 22:00:23 -07:00
2026-05-04 22:00:23 -07:00
2026-05-05 15:27:14 -07:00
2026-05-07 13:59:50 -07:00
2026-05-07 13:59:50 -07:00
2026-05-12 13:15:09 -07:00
2026-05-12 13:15:09 -07:00
2026-05-12 13:15:09 -07:00
2026-05-07 13:59:50 -07:00
2026-05-07 13:59:50 -07:00
2026-05-06 22:09:47 -07:00
2026-05-07 13:59:50 -07:00
2026-03-12 16:57:31 -07:00
2026-03-21 16:12:50 -07:00
2026-03-20 20:01:26 -07:00
2026-03-23 11:11:40 -07:00
2026-05-07 13:59:50 -07:00
2026-05-04 22:00:23 -07:00
2026-05-06 22:09:47 -07:00
2026-05-07 13:59:50 -07:00
2026-03-15 14:28:03 -07:00
2026-05-02 23:52:49 -07:00
2026-05-06 16:16:57 -07:00
2026-05-04 22:00:23 -07:00
2026-05-09 02:45:13 -07:00
2026-03-30 04:19:24 -07:00
2026-05-24 17:49:27 -07:00
2026-05-09 15:38:16 -07:00
2026-04-28 13:49:31 -07:00
2026-04-06 03:08:19 -07:00
2026-04-06 09:54:03 -07:00
2026-05-11 01:23:54 -07:00
2026-04-05 17:18:26 -07:00
2026-03-07 12:49:07 -08:00
2026-05-14 15:13:00 -07:00
2026-05-04 22:00:23 -07:00
2026-05-04 14:27:19 -07:00
2026-05-04 14:27:19 -07:00
2026-05-04 14:27:19 -07:00
2026-05-28 21:02:51 -07:00
2026-05-24 17:49:27 -07:00
2026-05-24 17:49:27 -07:00
2026-05-29 11:17:12 -07:00
2026-04-27 11:41:14 -07:00
2026-05-24 17:49:27 -07:00
2026-04-28 19:51:00 -07:00
2026-05-09 02:45:13 -07:00
2026-05-09 02:45:13 -07:00
2026-05-09 17:19:40 -07:00
2026-05-24 18:26:54 -07:00
2026-04-15 13:30:22 -07:00
2026-05-28 19:08:26 -07:00
2026-04-26 22:14:22 -07:00
2026-05-28 17:14:50 -07:00
2026-05-06 23:34:29 -07:00
2026-05-23 11:43:14 -07:00
2026-04-18 19:02:52 -07:00
2026-03-30 20:30:05 -07:00
2026-05-24 17:49:27 -07:00
2026-05-28 21:02:51 -07:00
2026-04-05 01:53:33 -07:00
2026-03-02 00:33:37 -08:00
2026-03-02 16:55:12 -08:00
2026-03-10 19:32:37 -07:00
2026-03-23 11:11:40 -07:00
2026-03-30 20:30:05 -07:00
2026-03-02 16:55:12 -08:00
2026-05-28 21:02:51 -07:00
2026-05-29 11:17:12 -07:00
2026-04-04 22:19:03 -07:00
2026-04-16 13:30:25 -07:00
2026-05-29 16:05:17 -07:00
2026-04-26 22:14:22 -07:00
2026-04-26 13:11:07 -07:00
2026-05-04 01:25:34 -07:00
2026-05-24 18:31:23 -07:00
2026-05-29 16:05:17 -07:00
2026-05-29 13:09:18 -07:00
2026-05-03 13:37:50 -07:00
2026-05-04 22:00:23 -07:00
2026-05-24 17:49:27 -07:00
2026-05-02 20:22:25 -07:00
2026-05-22 15:13:39 -07:00
2026-03-30 20:30:05 -07:00
2026-05-04 01:25:34 -07:00
2026-05-10 16:43:52 -07:00
2026-05-20 22:41:40 -07:00
2026-05-06 12:02:27 -07:00
2026-05-07 13:59:50 -07:00
2026-05-09 17:19:40 -07:00
2026-05-28 21:02:51 -07:00
2026-05-20 22:41:40 -07:00
2026-05-24 17:49:27 -07:00
2026-04-26 22:14:22 -07:00
2026-05-29 14:40:10 -07:00
2026-05-07 22:46:04 -07:00
2026-04-05 13:09:36 -07:00
2026-04-16 11:08:26 -07:00
2026-05-24 14:56:43 -07:00
2026-05-24 14:56:43 -07:00
2026-05-24 14:56:43 -07:00
2026-05-24 17:49:27 -07:00
2026-03-17 01:26:44 -07:00
2026-03-23 11:11:40 -07:00
2026-03-16 15:45:21 -07:00
2026-05-08 13:58:02 -07:00
2026-05-24 14:56:43 -07:00
2026-05-24 17:49:27 -07:00
2026-05-29 18:35:16 -07:00
2026-04-25 20:11:47 -07:00
2026-04-16 13:39:32 -07:00
2026-05-11 01:23:54 -07:00
2026-04-04 17:53:20 -07:00
2026-05-24 14:56:43 -07:00
2026-04-18 10:20:19 -07:00
2026-05-06 23:34:29 -07:00
2026-05-29 16:05:17 -07:00
2026-05-02 23:52:49 -07:00
2026-05-02 23:52:49 -07:00
2026-05-24 14:56:43 -07:00
2026-04-01 19:38:24 -07:00
2026-03-31 11:28:39 -07:00
2026-04-01 19:38:24 -07:00
2026-03-25 16:07:05 -07:00
2026-05-07 17:32:23 -07:00
2026-03-30 20:30:05 -07:00
2026-03-12 14:49:58 -07:00
2026-03-17 17:02:25 -07:00
2026-04-04 19:11:07 -07:00
2026-05-26 17:07:10 -07:00
2026-03-23 11:11:40 -07:00
2026-05-14 15:13:00 -07:00
2026-05-14 15:13:00 -07:00
2026-05-14 15:13:00 -07:00
2026-05-06 23:34:29 -07:00
2026-05-23 11:33:36 -07:00
2026-05-06 15:02:58 -07:00
2026-04-27 15:51:01 -07:00
2026-04-30 23:20:14 -07:00
2026-05-29 21:11:16 -07:00
2026-05-26 17:07:10 -07:00