ewilliams28/turnstone
1
0
Fork 0
mirror of https://github.com/turnstonelabs/turnstone.git synced 2026-08-12 23:12:23 -06:00
Code Issues Packages Projects Releases Wiki Activity

fix(renderer): mermaid streaming parser errors + progressive hljs (#510)

Browse Source 
* fix(renderer): mermaid streaming parser errors + progressive hljs

Live streaming was rendering mermaid diagrams with `Parse error,
got 'PS'` messages — bare `(`, `[`, `{` inside unquoted edge / node
labels re-entered Mermaid's shape parser. Two unrelated streaming-
specific issues in the renderer pile-up here; this commit addresses
both plus a follow-on UX improvement for code highlighting.

## Mermaid label autoquoter

`_normalizeMermaidSource` wraps two label forms that Mermaid rejects
when they contain bare shape-delimiter chars:

  1. Edge labels:  `|content|`  →  `|"content"|`
  2. Rectangle node labels:  `ID[content]`  →  `ID["content"]`

Shapes whose syntax already nests delimiters — cylinders `[(...)`,
subroutines `[[...]]`, trapezoids `[/.../]` `[\...\]`, circles
`((...))`, hexagons `{{...}}`, diamonds `{...}` — are intentionally
left alone (their inner delimiters are part of the shape syntax;
quoting would corrupt them). Labels already wrapped in `"..."` are
also left alone. The rewrite is idempotent and runs before the
mermaid SVG cache lookup so identical malformed input hits the
cache on re-render rather than re-quoting per tick.

## Markdown fence-pair regex

The old fence regex `/(```+)([^\s`]*)\n([\s\S]*?)\1/g` would, mid-
stream, pair an unclosed ```mermaid open with the OPENING backticks
of a later ```python fence as the "close", handing mermaid a
truncated source. New regex:

    /(```+)([^\s`]*)\n((?:(?!\1)[\s\S])*?)\1[ \t]*(?=\n|$)/g

Two constraints close the gap:

  - `(?!\1)` inside the content quantifier blocks the lazy matcher
    from extending across another N-backtick run. Smaller inner
    counts (e.g. 3-backtick inner inside a 4-backtick outer) still
    pass since `\1` is the open's actual count.
  - `[ \t]*(?=\n|$)` after `\1` forces the close to a line
    boundary, so ```python (open with a language tag) can't
    masquerade as a previous fence's close.

Together: an unclosed fence stays as plain markdown until its true
close arrives, so neither mermaid nor hljs ever sees a mid-stream
truncated source.

## Progressive hljs

Extracted `postRenderHljs` from `postRenderMarkdown` with a source-
keyed `_hljsCache` (FIFO, cap 64, keyed on `language:source`) and
wired it into `_streamingRenderApply`. Closed code fences are now
syntax-highlighted as they stream in, matching the progressive
mermaid pattern from #426. Per-tick cost stays cheap because the
cache returns the pre-tokenized HTML synchronously on hit; only
unique (language, source) pairs pay `hljs.highlightElement`.

## Internal cleanup from the review pipeline

  - `_cacheFifoEntry(cache, key, value, max)` replaces the duplicated
    `_cacheHljsEntry` and `_cacheMermaidEntry`. Single tested
    implementation across four caches (hljs, mermaid svg, mermaid
    error, mermaid normalize memo). The "don't evict on overwrite"
    invariant is pinned per-cache in tests.
  - `_mermaidNormalizeCache` memoizes raw textContent → normalized
    output so the per-rAF-tick autoquoter split + regex doesn't
    repeat for unchanged diagrams. Eviction shares
    `_MERMAID_CACHE_MAX` with the SVG cache it feeds.

## Tests

The fake DOM in tests/test_renderer_js.py grew a few capabilities
to drive these paths:
  - `classList` is now array-like (length + indexed access) so the
    hljs language-extraction loop works.
  - `textContent` setter mirrors the real-DOM side effect of
    entity-escaping into innerHTML, so `escapeHtml()` round-trips
    (otherwise every `renderMarkdown` returns empty `<p>` tags).
  - `querySelectorAll` handles both `pre code.language-mermaid`
    and `pre code[class*='language-']`.

Added: 6 fence-pairing regression cases, 9 hljs-progressive cases
(cache hit / distinct sources / language separation / NO_HIGHLIGHT
langs / terminal class / eviction / overwrite / postRenderMarkdown
wraps hljs / _streamingRenderApply invokes hljs), 11 autoquoter
cases including both diagram sources from the live screenshot
encoded verbatim as parametrized regressions, and 3 normalize-memo
cases (populates on first call, consulted before normalize via
sentinel pre-seed, distinct sources cache separately).

Total: 104 renderer tests pass (was 67).

* fix(renderer): apply Copilot review feedback on #510

Two doc / harness adjustments from the PR review — no behavior
change in production code.

- The `_mermaidNormalizeCache` comment claimed eviction "stays in
  lockstep with the SVG cache". That was misleading: the two
  caches key on different things (raw textContent vs normalized
  source) and evict independently. Updated the comment to describe
  what they actually share (the cap, for memory footprint) and
  what they don't (positional coupling), and to note that the memo
  deliberately survives `_initMermaid` since normalize output is
  theme-independent.

- The fake DOM in tests/test_renderer_js.py had `innerHTML` setter
  clear `children` but leave `_textContent` intact, so subsequent
  `textContent` reads could return stale data after an innerHTML
  mutation (real DOM invalidates textContent on innerHTML write).
  No current test triggered this, but it would mask future bugs
  that depend on innerHTML/textContent consistency. Setter now
  clears `_textContent`; the children-derived fallback in the
  getter returns `''` after the wholesale replace.

All 104 renderer tests still pass; ruff + mypy clean.
This commit is contained in:
Patrick Buckley
2026-05-11 15:57:46 -07:00
committed by GitHub
parent 8a847f5288
commit f8f076cf20
2 changed files with 920 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_renderer_js.py
+668 -19
View File
@@ -255,12 +255,17 @@ function makeEl(tag) {
setAttribute(k, v) { this._attrs[k] = v; },
getAttribute(k) { return this._attrs[k] !== undefined ? this._attrs[k] : null; },
get classList() {
// Real DOMTokenList is array-like (length + indexed access) AND
// exposes add/remove/contains. The hljs language-extraction
// loop reads .length + [j], so we return a fresh Array snapshot
// each get + bolt the mutator methods on. add/remove operate on
// the live _classes set so subsequent reads see updates.
const self = this;
return {
add(...c) { c.forEach(x => self._classes.add(x)); },
remove(...c) { c.forEach(x => self._classes.delete(x)); },
contains(c) { return self._classes.has(c); },
};
const arr = Array.from(self._classes);
arr.add = (...c) => c.forEach((x) => self._classes.add(x));
arr.remove = (...c) => c.forEach((x) => self._classes.delete(x));
arr.contains = (c) => self._classes.has(c);
return arr;
},
get className() { return Array.from(this._classes).join(' '); },
set className(v) {
@@ -269,9 +274,33 @@ function makeEl(tag) {
get textContent() {
return this._textContent || this.children.map(c => c.textContent || '').join('');
},
set textContent(v) { this._textContent = v; this.children = []; },
set textContent(v) {
// Real DOM: assigning textContent ALSO replaces innerHTML with
// an entity-escaped representation of the same text. escapeHtml
// (utils.js) round-trips via this side effect — without it,
// every escapeHtml() call returns '' and renderMarkdown emits
// empty <p> tags.
this._textContent = v;
this.children = [];
this._innerHTML = String(v)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;');
},
get innerHTML() { return this._innerHTML; },
set innerHTML(v) { this._innerHTML = v; this.children = []; },
set innerHTML(v) {
// Real DOM invalidates the previous textContent when innerHTML
// is replaced — leaving _textContent intact would return stale
// data from subsequent textContent reads and mask bugs that
// depend on innerHTML/textContent consistency. We don't HTML-
// parse here, so the cheap correct behavior is to clear
// _textContent and let the children-derived fallback in the
// textContent getter (which is empty after this children = [])
// take over.
this._innerHTML = v;
this.children = [];
this._textContent = '';
},
get isConnected() {
// In real DOM this checks attachment to the document; for the
// test harness we approximate via the parent chain. After
@@ -304,17 +333,28 @@ function makeEl(tag) {
this.parent = null;
},
querySelectorAll(selector) {
// Only supports the literal "pre code.language-mermaid"
// selector that postRenderMermaid uses.
// Supports the two selectors the post-render passes use:
// "pre code.language-mermaid" (postRenderMermaid)
// "pre code[class*='language-']" (postRenderHljs)
const out = [];
const wantsMermaid = selector === "pre code.language-mermaid";
function matchesLangAttr(el) {
for (const cls of el._classes) {
if (cls.startsWith('language-')) return true;
}
return false;
}
function walk(node) {
for (const c of (node.children || [])) {
if (
const isCodeInPre =
c.tagName === 'CODE' &&
c.parent && c.parent.tagName === 'PRE' &&
c._classes.has('language-mermaid')
) {
out.push(c);
c.parent && c.parent.tagName === 'PRE';
if (isCodeInPre) {
if (wantsMermaid) {
if (c._classes.has('language-mermaid')) out.push(c);
} else if (matchesLangAttr(c)) {
out.push(c);
}
}
walk(c);
}
@@ -352,6 +392,20 @@ global.mermaid = {
},
};
// hljs stub. highlightElement mutates the element in place: replaces
// innerHTML with a deterministic synthetic span keyed by the source,
// and adds the hljs class — same surface postRenderHljs depends on.
// hljsHighlightCallCount lets tests assert "ran N times" semantics.
let hljsHighlightCallCount = 0;
global.hljs = {
configure: () => {},
highlightElement: (el) => {
hljsHighlightCallCount++;
el._classes.add('hljs');
el._innerHTML = '<span class="hljs-tok">' + el._textContent + '</span>';
},
};
vm.runInThisContext(fs.readFileSync(%(utils)s, 'utf8'));
vm.runInThisContext(fs.readFileSync(%(renderer)s, 'utf8'));
@@ -514,7 +568,7 @@ def test_mermaid_cache_evicts_oldest_at_cap() -> None:
scenario = """
const cap = _MERMAID_CACHE_MAX;
for (let i = 0; i < cap + 5; i++) {
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
}
process.stdout.write(JSON.stringify({
size: _mermaidSvgCache.size,
@@ -536,10 +590,10 @@ def test_mermaid_overwrite_does_not_evict() -> None:
const cap = _MERMAID_CACHE_MAX;
// Fill exactly to cap.
for (let i = 0; i < cap; i++) {
_cacheMermaidEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-' + i, {svg: 'svg-' + i, bindFunctions: null}, cap);
}
// Overwrite an existing entry — must not evict src-0.
_cacheMermaidEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null});
_cacheFifoEntry(_mermaidSvgCache, 'src-5', {svg: 'svg-updated', bindFunctions: null}, cap);
process.stdout.write(JSON.stringify({
size: _mermaidSvgCache.size,
hasOldest: _mermaidSvgCache.has('src-0'),
@@ -558,8 +612,8 @@ def test_mermaid_cache_cleared_on_init() -> None:
— the rendered output depends on themeVariables which change
on init."""
scenario = """
_cacheMermaidEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null});
_cacheMermaidEntry(_mermaidErrorCache, 'src-bad', 'old error');
_cacheFifoEntry(_mermaidSvgCache, 'src-1', {svg: 'old', bindFunctions: null}, _MERMAID_CACHE_MAX);
_cacheFifoEntry(_mermaidErrorCache, 'src-bad', 'old error', _MERMAID_CACHE_MAX);
_initMermaid();
process.stdout.write(JSON.stringify({
svgSize: _mermaidSvgCache.size,
@@ -624,3 +678,598 @@ def test_streaming_render_invokes_mermaid_post_render() -> None:
"_streamingRenderApply must call postRenderMermaid for "
"progressive diagram rendering during streaming"
)
# ---------------------------------------------------------------------------
# _normalizeMermaidSource — autoquote labels with bare shape-delimiter
# chars. Mermaid rejects unquoted ( ) [ ] { } inside other labels with
# a "got 'PS'" parse error (paren-start in shape context). The two
# diagrams in the screenshot regression case are encoded here verbatim.
# ---------------------------------------------------------------------------
def _run_normalize(source: str) -> str:
"""Drive _normalizeMermaidSource against the JS harness and return
its output. The function is pure, so no container / mermaid stub
setup is required."""
scenario = f"""
const input = {json.dumps(source)};
const output = _normalizeMermaidSource(input);
process.stdout.write(JSON.stringify({{ output: output }}));
"""
out = _run_mermaid_scenario(scenario)
return str(out["output"])
# Diagram 1 from the screenshot regression — unquoted edge labels with
# parens and <br/> markers. Mermaid rejects both edge labels with
# "got 'PS'"; quoting them resolves it.
_SCREENSHOT_DIAGRAM_1_IN = (
"flowchart LR\n"
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
" -->|22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)|"
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
" B -->|13 jasl patches<br/>(Python only:<br/>tunings, kernels,"
"<br/>warmup, etc.)|"
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
)
_SCREENSHOT_DIAGRAM_1_OUT = (
"flowchart LR\n"
' A["vllm-openai:nightly<br/>commit 5536fc0c0<br/>2026-05-11 11:59"]'
' -->|"22 upstream<br/>main commits<br/>(10 csrc, but<br/>no new bindings)"|'
' B["fork merge_base<br/>7863fff6e5<br/>2026-05-12 00:27"]\n'
' B -->|"13 jasl patches<br/>(Python only:<br/>tunings, kernels,'
'<br/>warmup, etc.)"|'
' C["ds4-sm120-preview-dev<br/>acc3455b1e"]'
)
# Diagram 2 from the screenshot regression — unquoted RECTANGLE node
# label `D[untouched<br/>(.so, _version.py,<br/>install-vendored)]`.
# Same parser failure mode; quoting the bracket label fixes it.
_SCREENSHOT_DIAGRAM_2_IN = (
"flowchart LR\n"
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
" B -->|not in archive| D[untouched<br/>(.so, _version.py,"
"<br/>install-vendored)]\n"
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
)
_SCREENSHOT_DIAGRAM_2_OUT = (
"flowchart LR\n"
" A[nightly's vllm/<br/>installed package] --> B{tar -xf<br/>fork-vllm.tar}\n"
" B -->|in archive| C[overwritten with<br/>fork's version]\n"
' B -->|not in archive| D["untouched<br/>(.so, _version.py,'
'<br/>install-vendored)"]\n'
" E[explicit rm of 1 file<br/>deleted upstream] --> B"
)
@pytest.mark.parametrize(
("source", "expected"),
[
(_SCREENSHOT_DIAGRAM_1_IN, _SCREENSHOT_DIAGRAM_1_OUT),
(_SCREENSHOT_DIAGRAM_2_IN, _SCREENSHOT_DIAGRAM_2_OUT),
],
)
def test_mermaid_autoquote_fixes_screenshot_diagrams(source: str, expected: str) -> None:
"""The two exact diagrams from the screenshot regression. If
these stop being rewritten with quoted labels, mermaid will
again reject them with `Expecting ... got 'PS'` during live
streaming."""
assert _run_normalize(source) == expected
@pytest.mark.parametrize(
"source",
[
# Clean diagram — no shape delimiters in any label.
"graph TD\n A[foo] --> B[bar]",
# Edge label with no special chars.
"A --> B\nA -->|plain text| B",
# Already-correctly-quoted node label.
'A["already (quoted)"] --> B',
# Already-correctly-quoted edge label.
'A -->|"already (quoted)"| B',
# Cylinder shape — inner () is part of the shape syntax.
"A[(database)] --> B",
# Subroutine shape — inner [] is part of the shape syntax.
"A[[subroutine]] --> B",
# Trapezoid shape — inner / is part of the shape syntax.
"A[/trapezoid/] --> B",
# Reverse trapezoid.
"A[\\trap\\] --> B",
# Mermaid directive — braces here are config, not a label.
'%%{init: {"theme": "dark"}}%%\ngraph TD\n A --> B',
# <br/> tags on their own don't trip quoting.
"A[line1<br/>line2] --> B",
# Sequence diagram — different grammar; we only target labels
# in shape/edge syntax that match the regex anchors.
"sequenceDiagram\n A->>B: hello",
],
)
def test_mermaid_autoquote_leaves_valid_source_alone(source: str) -> None:
"""The autoquoter must not rewrite syntactically valid Mermaid —
a false positive here would break a working diagram. Each case
covers a syntax form whose delimiters are intentional and must
not be wrapped."""
assert _run_normalize(source) == source
def test_mermaid_autoquote_edge_label_with_parens() -> None:
"""Bare-parens edge label gets wrapped. The bare `(` would
otherwise re-enter Mermaid's shape parser."""
src = "A -->|note (with parens)| B"
assert _run_normalize(src) == 'A -->|"note (with parens)"| B'
def test_mermaid_autoquote_node_label_with_parens() -> None:
"""Bare-parens node label gets wrapped."""
src = "D[label (foo, bar)]"
assert _run_normalize(src) == 'D["label (foo, bar)"]'
def test_mermaid_autoquote_node_label_with_braces() -> None:
"""Bare-braces in a rectangle label get wrapped. (Diamond {}
shapes are left alone — only single-bracket [] labels are
rewritten.)"""
src = "A[config {key: value}]"
assert _run_normalize(src) == 'A["config {key: value}"]'
def test_mermaid_autoquote_preserves_br_tag_with_parens() -> None:
"""`<br/>` inside a label that also has parens stays — only the
quoting needs to be added around the whole label."""
src = "A[line1<br/>(line2)] --> B"
assert _run_normalize(src) == 'A["line1<br/>(line2)"] --> B'
def test_mermaid_autoquote_skips_label_with_internal_quote() -> None:
"""If a label contains a literal `"`, wrapping would produce
nested unescaped quotes. The autoquoter must punt — leaving the
parse error to surface, rather than silently producing a worse
one."""
src = 'A[he said "hi" (lol)]'
assert _run_normalize(src) == src
def test_mermaid_autoquote_multiple_edges_on_one_line() -> None:
"""Both edge labels on a single line get rewritten independently."""
src = "A -->|first (paren)| B -->|second (paren)| C"
expected = 'A -->|"first (paren)"| B -->|"second (paren)"| C'
assert _run_normalize(src) == expected
def test_mermaid_autoquote_normalized_source_hits_cache() -> None:
"""The SVG cache keys on the normalized source — same malformed
input that the LLM streamed earlier still hits the cache on
re-render rather than re-invoking mermaid.render every tick."""
bad = "A[label (with parens)] --> B"
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {
const container2 = buildContainer(sources);
postRenderMermaid(container2);
setTimeout(() => {
process.stdout.write(JSON.stringify({
renderCalls: renderCallCount,
normalized: container.children[0]._attrs['data-mermaid-source'],
}));
}, 0);
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["renderCalls"] == 1, "second render bypassed the cache"
assert out["normalized"] == 'A["label (with parens)"] --> B'
def test_mermaid_normalize_memo_populates_on_first_call() -> None:
"""First postRenderMermaid call populates _mermaidNormalizeCache
with a raw→normalized entry. A second call on identical raw
textContent then hits the memo (size stays at 1, no second
normalize call), which is the perf-1 fix — avoids re-running
split + per-line regex per rAF tick when the diagram hasn't
changed."""
bad = "A[label (with parens)] --> B"
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
const sizeAfterFirst = _mermaidNormalizeCache.size;
const cachedNorm = _mermaidNormalizeCache.get(sources[0]);
// Re-render on a fresh container with the same source.
const container2 = buildContainer(sources);
postRenderMermaid(container2);
setTimeout(() => setTimeout(() => {
process.stdout.write(JSON.stringify({
sizeAfterFirst: sizeAfterFirst,
cachedNorm: cachedNorm,
sizeAfterSecond: _mermaidNormalizeCache.size,
}));
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["sizeAfterFirst"] == 1, "first call didn't populate normalize memo"
assert out["cachedNorm"] == 'A["label (with parens)"] --> B'
assert out["sizeAfterSecond"] == 1, (
"second call added a new entry — memo missed on identical source"
)
def test_mermaid_normalize_memo_is_consulted_before_normalize() -> None:
"""Pre-seed _mermaidNormalizeCache with a sentinel value for a
raw source. postRenderMermaid must use the sentinel rather than
re-running _normalizeMermaidSource. Catches a regression where
the memo gets populated but the lookup path is skipped."""
bad = "A[label (with parens)] --> B"
sentinel = "SENTINEL_FROM_MEMO --> X"
raw_js = json.dumps(bad)
sentinel_js = json.dumps(sentinel)
scenario = (
_build_mermaid_container_js([bad])
+ _MERMAID_DRAIN_JS
+ f"""
_mermaidNormalizeCache.set({raw_js}, {sentinel_js});
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {{
process.stdout.write(JSON.stringify({{
sourceAttr: container.children[0]._attrs['data-mermaid-source'],
}}));
}}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["sourceAttr"] == sentinel, (
"postRenderMermaid bypassed the normalize memo and re-ran normalize"
)
def test_mermaid_normalize_memo_distinct_sources_cache_separately() -> None:
"""Two distinct raw sources produce two memo entries. Confirms
the memo keys on raw textContent, not on something coarser like
container identity."""
bad1 = "A[label (with parens)] --> B"
bad2 = "C[other (label)] --> D"
scenario = (
_build_mermaid_container_js([bad1, bad2])
+ _MERMAID_DRAIN_JS
+ """
postRenderMermaid(container);
setTimeout(() => setTimeout(() => {
process.stdout.write(JSON.stringify({
size: _mermaidNormalizeCache.size,
hasBad1: _mermaidNormalizeCache.has(sources[0]),
hasBad2: _mermaidNormalizeCache.has(sources[1]),
}));
}, 0), 0);
"""
)
out = _run_mermaid_scenario(scenario)
assert out["size"] == 2
assert out["hasBad1"] is True
assert out["hasBad2"] is True
# ---------------------------------------------------------------------------
# Code-fence pairing — close requires \n / EOS, content can't cross
# another close-pattern. Repros the streaming bug where ```mermaid +
# later ```python were paired by the regex, handing mermaid a
# truncated source.
# ---------------------------------------------------------------------------
def _render_md(source: str) -> str:
"""Drive renderMarkdown against the JS harness and return the
rendered HTML. The function is a pure string transform; no DOM
container scaffolding is required."""
scenario = f"""
const input = {json.dumps(source)};
const output = renderMarkdown(input);
process.stdout.write(JSON.stringify({{ output: output }}));
"""
out = _run_mermaid_scenario(scenario)
return str(out["output"])
_FENCE = "```"
def test_fence_partial_open_emits_no_code_block() -> None:
"""While a fence is still open and there's no other ``` later in
the buffer, no <code> block is emitted — the open fence stays as
plain markdown text until the real close arrives."""
src = "Intro\n" + _FENCE + 'mermaid\nA["x"] -->|note (with parens)| B["y"]\nstill streaming'
html = _render_md(src)
assert "<code" not in html, f"open fence should not emit <code> mid-stream: {html!r}"
def test_fence_partial_with_later_open_does_not_pair_wrongly() -> None:
"""Before the fence-pair fix: an unclosed ```mermaid followed by
a ```python (also unclosed) would have paired up as
<code class=mermaid>...</code>python..., handing mermaid a
truncated source. With the new regex, neither fence emits a
block until its OWN closing line arrives."""
src = "Intro\n" + _FENCE + "mermaid\nA --> B\n" + _FENCE + 'python\nprint("hi")'
html = _render_md(src)
assert 'class="language-mermaid"' not in html, (
f"mermaid fence should not emit while open: {html!r}"
)
assert 'class="language-python"' not in html, (
f"python fence should not emit while open: {html!r}"
)
def test_fence_close_paired_with_next_open_is_rejected() -> None:
"""Repro of the live-streaming failure: mermaid fence open, then
```python opens and ``` closes the python block. Without the
fix, the regex paired mermaid's open with python's *open* (or
backtracked all the way to python's close), producing
<code class=mermaid>truncated</code>. With the fix mermaid stays
open (content can't cross another \\1 run; close must be at line
boundary) and only python's pair matches."""
src = (
"Intro\n"
+ _FENCE
+ 'mermaid\nA["x"] -->|note (with parens)| B["y"]\n'
+ _FENCE
+ 'python\nprint("hi")\n'
+ _FENCE
)
html = _render_md(src)
assert 'class="language-mermaid"' not in html, f"mermaid fence misparing reintroduced: {html!r}"
assert 'class="language-python"' in html, f"python fence on its own should match: {html!r}"
def test_fence_closed_emits_code_block() -> None:
"""Baseline: a properly closed fence with its close on its own
line emits the <code> block as expected — the anchor doesn't
break the normal case."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + "\nAfter"
html = _render_md(src)
assert 'class="language-python"' in html
assert "import os" in html
def test_fence_close_at_end_of_buffer_emits() -> None:
"""A fence that closes at the very end of the buffer (no trailing
newline) still emits — the anchor accepts end-of-string as a
valid line boundary, so the rehydration / static-render path
where the buffer ends cleanly at ``` still works."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE
html = _render_md(src)
assert 'class="language-python"' in html
assert "import os" in html
def test_fence_close_with_trailing_whitespace_emits() -> None:
"""A close followed only by spaces / tabs before \\n still counts
— CommonMark allows trailing whitespace on the close line."""
src = "Intro\n" + _FENCE + "python\nimport os\n" + _FENCE + " \nAfter"
html = _render_md(src)
assert 'class="language-python"' in html
# ---------------------------------------------------------------------------
# postRenderHljs — progressive syntax highlighting + source-keyed cache
# ---------------------------------------------------------------------------
def _build_hljs_container_js(blocks: list[tuple[str, str]]) -> str:
"""Build a container with <pre><code class="language-LANG"> blocks.
``blocks`` is a list of ``(language, source)`` tuples — the language
becomes the ``language-X`` class, the source becomes textContent."""
arr = "[" + ", ".join(f"[{json.dumps(lang)}, {json.dumps(src)}]" for lang, src in blocks) + "]"
return f"""
function buildHljsContainer(blocks) {{
const container = document.createElement('div');
for (const [lang, src] of blocks) {{
const pre = document.createElement('pre');
const code = document.createElement('code');
code.classList.add('language-' + lang);
code.textContent = src;
pre.appendChild(code);
container.appendChild(pre);
}}
return container;
}}
const blocks = {arr};
const container = buildHljsContainer(blocks);
"""
def test_hljs_cache_hit_skips_highlight_call() -> None:
"""Two postRenderHljs calls on identical source must invoke
hljs.highlightElement exactly once — the second call hits the
cache and applies the stored markup synchronously. Mirrors the
mermaid SVG-cache invariant that lets streamingRender fire on
every rAF tick without re-tokenizing every code block."""
scenario = (
_build_hljs_container_js([("python", "import os")])
+ """
postRenderHljs(container);
const container2 = buildHljsContainer(blocks);
postRenderHljs(container2);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
firstHtml: container.children[0].children[0]._innerHTML,
secondHtml: container2.children[0].children[0]._innerHTML,
secondHasHljsClass: container2.children[0].children[0]._classes.has('hljs'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1, (
"second postRenderHljs call invoked highlightElement — cache miss"
)
assert out["cacheSize"] == 1
assert out["firstHtml"] == out["secondHtml"]
assert out["secondHasHljsClass"] is True
def test_hljs_distinct_sources_highlight_independently() -> None:
"""Distinct sources each trigger one highlight and cache one entry.
Cache key includes the source string, not e.g. just the language."""
scenario = (
_build_hljs_container_js([("python", "import os"), ("python", "print('hi')")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 2
assert out["cacheSize"] == 2
def test_hljs_cache_separates_by_language() -> None:
"""Same source text under different language fences must NOT
collide in the cache — language is part of the key. Otherwise a
`python` block of `foo` and a `ruby` block of `foo` would share
a single (wrongly-highlighted) cache entry."""
scenario = (
_build_hljs_container_js([("python", "foo"), ("ruby", "foo")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 2
assert out["cacheSize"] == 2
def test_hljs_skips_no_highlight_langs() -> None:
"""language-mermaid / language-text / language-plaintext etc. must
get the `nohighlight` class without invoking hljs.highlightElement.
Highlighting plaintext or mermaid source would be both wasteful
and ugly."""
scenario = (
_build_hljs_container_js(
[("mermaid", "graph TD\\nA-->B"), ("text", "plain"), ("plaintext", "p")]
)
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
cacheSize: _hljsCache.size,
mermaidNoHighlight: container.children[0].children[0]._classes.has('nohighlight'),
textNoHighlight: container.children[1].children[0]._classes.has('nohighlight'),
plaintextNoHighlight: container.children[2].children[0]._classes.has('nohighlight'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 0
assert out["cacheSize"] == 0
assert out["mermaidNoHighlight"] is True
assert out["textNoHighlight"] is True
assert out["plaintextNoHighlight"] is True
def test_hljs_terminal_lang_marks_pre_for_terminal_styling() -> None:
"""Shell-family languages (bash / sh / zsh / console / terminal)
must add the `code-terminal` class to the parent <pre>, so the
stylesheet can give them the terminal look-and-feel."""
scenario = (
_build_hljs_container_js([("bash", "echo hi")])
+ """
postRenderHljs(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
preHasTerminalClass: container.children[0]._classes.has('code-terminal'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1
assert out["preHasTerminalClass"] is True
def test_hljs_cache_evicts_oldest_at_cap() -> None:
"""FIFO eviction at _HLJS_CACHE_MAX. Mirrors the mermaid cache —
prevents unbounded growth on long sessions with many distinct
code blocks."""
scenario = """
const cap = _HLJS_CACHE_MAX;
for (let i = 0; i < cap + 5; i++) {
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
}
process.stdout.write(JSON.stringify({
size: _hljsCache.size,
hasOldest: _hljsCache.has('key-0'),
hasNewest: _hljsCache.has('key-' + (cap + 4)),
}));
"""
out = _run_mermaid_scenario(scenario)
assert out["size"] == 64
assert out["hasOldest"] is False
assert out["hasNewest"] is True
def test_hljs_overwrite_does_not_evict() -> None:
"""Overwriting an existing key is an in-place update, not a new
insertion — must not evict the oldest unrelated entry. Same
invariant as the mermaid cache."""
scenario = """
const cap = _HLJS_CACHE_MAX;
for (let i = 0; i < cap; i++) {
_cacheFifoEntry(_hljsCache, 'key-' + i, 'val-' + i, cap);
}
_cacheFifoEntry(_hljsCache, 'key-5', 'val-updated', cap);
process.stdout.write(JSON.stringify({
size: _hljsCache.size,
hasOldest: _hljsCache.has('key-0'),
updated: _hljsCache.get('key-5'),
}));
"""
out = _run_mermaid_scenario(scenario)
assert out["size"] == 64
assert out["hasOldest"] is True, "overwrite evicted oldest unnecessarily"
assert out["updated"] == "val-updated"
def test_post_render_markdown_invokes_hljs() -> None:
"""postRenderMarkdown is the public end-of-stream entry point and
must still run syntax highlighting after the postRenderHljs
refactor — regression guard for the public API surface that
app.js / coordinator code already call."""
scenario = (
_build_hljs_container_js([("python", "import os")])
+ """
postRenderMarkdown(container);
process.stdout.write(JSON.stringify({
highlightCalls: hljsHighlightCallCount,
hasHljsClass: container.children[0].children[0]._classes.has('hljs'),
}));
"""
)
out = _run_mermaid_scenario(scenario)
assert out["highlightCalls"] == 1
assert out["hasHljsClass"] is True
def test_streaming_render_invokes_hljs() -> None:
"""_streamingRenderApply must call postRenderHljs so closed code
fences appear progressively (syntax-highlighted) during streaming,
not only at stream_end via streamingRenderFinalize. The cache
keeps the per-tick cost down to a synchronous lookup."""
body = _RENDERER_JS.read_text(encoding="utf-8")
start = body.index("function _streamingRenderApply")
hljs_call = body.find("postRenderHljs(el)", start, start + 4000)
assert hljs_call != -1, (
"_streamingRenderApply must call postRenderHljs for progressive "
"syntax highlighting during streaming"
)