Skip to content
Lightning YAML

Deep-dive: yaml (eemeli/yaml) v2.9.0 — pipeline anatomy, allocation profile, and why it loses to js-yaml

Produced by a multi-agent research session (Claude Code). All local numbers were measured on the session container — Node v22.22.2 / V8 12.4, 4-vCPU Xeon 2.80 GHz, 16 GB RAM; absolute MB/s are machine-specific, ratios are the durable signal. Referenced scratchpad/*.mjs scripts were session throwaways and are not committed.

Package inspected: /home/user/lightning-yaml/node_modules/yaml → /home/user/lightning-yaml/node_modules/.pnpm/yaml@2.9.0/node_modules/yaml (v2.9.0, dist/ compiled JS). All line refs are into that dist/. All measurements: Node v22.22.2, this machine, fresh 10.0MB JSON fixture (60,315 records, flow-style — same class of input as the repo’s fixtures).

0. The top-level parse() pipeline

Section titled “0. The top-level parse() pipeline”

YAML.parse(src) (dist/public-api.js:60-79) = parseDocument(src) → doc.toJS(...). parseDocument (public-api.js:40-53) wires four stages as three nested generators:

Composer.compose( Parser.parse(source), true, source.length )   // public-api.js:46
   Parser.parse: for (const lexeme of this.lexer.lex(source)) yield* this.next(lexeme)  // parser.js:165-172

Stages: Lexer (string → lexeme strings) → Parser (lexemes → CST token objects) → Composer (CST → Document node tree: Scalar/Pair/YAMLMap/YAMLSeq) → doc.toJS() (node tree → plain JS). Also by default prettyErrors creates a LineCounter and the Parser calls onNewLine per line (public-api.js:12-14, parser.js:204-205).

Critical structural fact: the Parser only yields a top-level CST token when its stack empties (parser.js:279-288 — pop() yields only if (this.stack.length === 0)). A single-document file therefore yields exactly one giant 'document' CST token at EOF, and the Composer (composer.js:151-160 → compose-doc.js:34-36) only starts composing after the entire CST is materialized in memory. The “streaming” generator architecture streams nothing within a document; the CST and the composed Document tree are then simultaneously alive during composition.

1. What is allocated per input token

Section titled “1. What is allocated per input token”

Lexer (dist/parse/lexer.js): a class-based state machine whose *lex() generator (lexer.js:152-163) yields one JS string per lexical token — including every space run, every ,, :, {, [. Emission goes through pushCount → this.buffer.substr(this.pos, n) (lexer.js:616-623), pushToIndex → this.buffer.slice(...) (lexer.js:624-634), pushSpaces → substr (lexer.js:701-713). Scalars are announced by first yielding a '\x1f' SCALAR control char (lexer.js:568, 612; cst.js:14), then the scalar source slice. In V8, slice/substr results ≥13 chars are SlicedStrings (SlicedString::kMinLength = 13, v8/src/objects/string.h — https://github.com/v8/v8/blob/main/src/objects/string.h) that pin the entire parent source string; shorter ones are fresh copies. Measured: the 10MB fixture produces 3,860,166 lexemes (one per 2.6 input bytes), total 10.48MB of lexeme text.

Parser (dist/parse/parser.js): for each lexeme, next() calls cst.tokenType(source) — a string-switch re-classifying the lexeme from its text (parser.js:186, cst.js:40-98). Every consumed lexeme becomes a sourceToken object {type, offset, indent, source} allocated in the sourceToken getter (parser.js:231-239) or in flowScalar() (parser.js:862-876), and is pushed into arrays (start/sep/end/items) hanging off composite CST tokens (document, block-map, block-seq, flow-collection — e.g. parser.js:311, 336, 444-449, 781-820). Flow/block collection items are records {start:[], key, sep:[], value} — two array allocations per item (parser.js:311, 336, 781). So the CST carries the full source text as substrings on every token plus an object per token. Measured for 10MB: ~3.68M CST token objects + ~1.03M collection-item records; the CST alone retains 733MB — 73× the input (forced-GC heapUsed delta with tokens array retained).

Composer (dist/compose/*): walks the CST recursively. Per map/seq entry it calls resolveProps() twice (key props + value props), each returning a fresh 10-field object (resolve-block-map.js:21-28, 66-73; resolve-flow-collection.js:29-37, 113-121; resolve-props.js:134-145). Per scalar, composeScalar (compose-scalar.js:8-46) calls resolveFlowScalar which returns another {value, type, comment, range:[3-number array]} object (resolve-flow-scalar.js:36-41), then allocates a Scalar node carrying range (3-elem array), source (decoded value string), value, type (compose-scalar.js:35-44); NodeBase’s constructor does an Object.defineProperty(this, NODE_TYPE, ...) per node (nodes/Node.js:9), as do Pair (nodes/Pair.js:14-18) and Collection (a second one for schema, nodes/Collection.js:35-41). (I microbenched defineProperty-in-constructor vs plain store on Node 22: only ~7% slower — V8 handles it; the cost is the number of objects, not defineProperty.) Every map entry gets a Pair object; and inside flow sequences, a key: value entry gets an extra single-entry YAMLMap wrapper (resolve-flow-collection.js:169-175). srcToken back-links to the CST are only attached when keepSourceTokens: true (compose-node.js:72-73, resolve-block-map.js:91-92) — off by default, so the CST can be GC’d after compose, but it is fully alive during compose. Measured composed tree for 10MB: 1,387,246 Scalars + 663,466 Pairs + 301,577 collections = 2.35M node objects, retaining 330MB (33× input) after the CST is freed.

toJS (nodes/toJS.js:15-37, Document.js:293-302): third full traversal. Dispatches through value.toJSON(arg, ctx) per node; YAMLMap.toJSON → addPairToJSMap per pair, which does toJS(key) + stringifyKey + stringKey in map + assignment (YAMLMap.js:119-126, addPairToJSMap.js:16-35). Output (plain JS) retains ~42MB — comparable to js-yaml’s result (51MB) and JSON.parse’s (22MB).

2. Generator/yield control-flow cost

Section titled “2. Generator/yield control-flow cost”

Three generator layers are nested: Lexer (*lex → *parseNext → *parseDocument/*parseFlowCollection → *pushCount/*pushToIndex/*pushSpaces, all yield*-delegating — lexer.js:152-163, 225-244, 616-720), Parser (*parse → *next → *step → *blockMap/*flowCollection/*pop, parser.js:165-172, 240-275), Composer (*compose → *next, composer.js:132-136). Every one of the 3.86M lexemes crosses the lexer→parser generator boundary (an IteratorResult per .next()), and re-enters a fresh delegated generator chain in Parser.next. Microbench on this machine: yielding 3.86M strings through a 3-deep yield* chain costs 223ms vs 12ms for a plain loop (~19×, ~58ns/item) — so generator plumbing alone plausibly burns ~0.5s+ of the parse, before any real work. Symptom: the Lexer alone takes 1.56s on 10MB — 3× the time js-yaml needs to fully parse the same input to finished JS objects (0.53s).

3. Where the memory goes (mechanistic account of 2.63GB vs 495MB vs 282MB)

Section titled “3. Where the memory goes (mechanistic account of 2.63GB vs 495MB vs 282MB)”

Staged measurements, 10MB fixture, isolated processes (process.resourceUsage().maxRSS / forced-GC heap deltas):

stage time (1 iter) peak RSS retained (after GC)
JSON.parse 0.09s 84MB 22MB
js-yaml.load 0.53s 153MB 51MB
yaml Lexer only (discard lexemes) 1.56s 69MB ~0
yaml Parser → CST (retained) 8.8s 839MB 733MB
YAML.parseDocument (Document retained) 10.8s 1182MB 330MB (Document alone)
YAML.parse (full) 9.7s 1217MB (heapUsed 1113MB) 42MB (JS out)
YAML.parse × 25 loop 268s (10.7s/iter) 4302MB —
js-yaml.load × 25 loop 12.3s (0.49s/iter) 564MB —
JSON.parse × 25 loop 2.6s (0.105s/iter) 643MB —

So the harness’s 2.63GB is reproduced (4.3GB here — heap-limit/machine dependent). The mechanism, in order of magnitude:

  1. The CST is the whale: 733MB for 10MB of input. ~4.7M objects/records + 3.9M substrings, each token a {type, offset, indent, source} object inside item records with two arrays each. It is fully materialized before composition starts (parser.js:286-288), by design: the docs guarantee the CST “retains every character of the input, including whitespace” (https://github.com/eemeli/yaml/blob/main/docs/07_parsing_yaml.md).
  2. CST + Document tree coexist at compose time: 733MB + 330MB (+ source + churn) ≈ the measured 1.1GB single-iteration live peak. Four representations of the same data exist across the parse: source string → CST → node tree → JS output.
  3. GC pressure converts allocation rate into RSS. Only ~1.15GB is ever live, but a 25-iteration loop ratchets maxRSS to 4.3GB: V8 grows the old generation rather than collecting eagerly when ~15M+ objects/iteration are being thrown at it. Sheer garbage volume (resolveProps ×~1.7M, resolveFlowScalar result objects ×1.4M, range arrays ×2.35M, rope nodes from res += ch) inflates peak RSS even though it’s all transient. js-yaml allocates roughly an order of magnitude fewer objects (final values only) → 564MB over 25 iters.
  4. Sliced-string pinning is real but secondary here: CST token.source slices ≥13 chars are V8 SlicedStrings pinning the whole 10MB buffer, but since the CST itself is 73× the input, pinning is noise during parse. It matters if a user retains a CST or uses keepSourceTokens. Note composed Scalar.source is the decoded value (compose-scalar.js:36), a fresh string, not a slice — but for double-quoted input it’s built as a rope (see below).

Speed-side hot spots (why 10.7s vs js-yaml 0.49s ≈ 22×): (a) per-lexeme re-classification + object churn in Parser (Parser→CST stage is ~7s of the ~10s); (b) doubleQuotedValue builds every double-quoted string one res += ch per character (resolve-flow-scalar.js:118-173, plain chars at :167) — JSON-as-YAML input is ~100% double-quoted strings; js-yaml instead captures whole segments between escapes (captureSegment, js-yaml/lib/loader.js:354, used at :706-743); (c) plain scalars run a sequential regex battery — schema.tags.find(tag => tag.test?.test(value)) over null/bool/int-oct/int/int-hex/float-nan/float-exp/float (compose-scalar.js:72-77; dist/schema/core/int.js, float.js, bool.js, common/null.js); (d) duplicate-key checking is items.some(...) per inserted key → O(n²) per mapping (util-map-includes.js:12, called from resolve-block-map.js:63 and resolve-flow-collection.js:164); (e) the toJS third pass re-walks everything through polymorphic toJSON dispatch.

4. The maintainer’s own statements

Section titled “4. The maintainer’s own statements”
  • Discussion #358 “Performance yaml 1.10, yaml 2 and js-yaml” (https://github.com/eemeli/yaml/discussions/358), Feb 2022. User benchmark on a ~500k-line file: read — JSON 94ms, js-yaml 504ms, yaml@1.10 2108ms, yaml@2 9584ms (v2 is slower than v1 at reading and ~19× slower than js-yaml — matching my 18-22× measurements); write — yaml@2 2442ms beats js-yaml 7946ms. eemeli: the library is “built for power rather than speed, and simply does more with its input than e.g. js-yaml”; “yaml handles its input in stages: lexing, parsing to a CST, composing an AST, and then converting to JS”; and he discounts the benchmark: “most real-world files are at most a few hundred lines long, and most real-world usage patterns have significant other work happening around the parsing.” No commitment to structural speed work — the multi-stage lossless design is intentional and defended.
  • PR #612 (https://github.com/eemeli/yaml/pull/612, shipped in 2.8.0, 2025): the one recent perf fix, and it’s alias resolution only — Alias.resolve() previously re-traversed the whole document per alias; a cached anchor/alias list gave 12-64× on alias-heavy docs (eemeli verified ~12×). Confirms perf fixes are accepted when contributed, but hot-path architecture is untouched.
  • Downstream evidence of the pain: redhat-developer/vscode-yaml#1063 (“Performance issue in used yaml parser makes usage prohibitively slow”, anchors/aliases) — https://github.com/redhat-developer/vscode-yaml/issues/1063.
  • Official docs state the CST-retention guarantee explicitly: the CST “retains every character of the input, including whitespace” and the Lexer/Parser “should never throw” (error tolerance) — https://github.com/eemeli/yaml/blob/main/docs/07_parsing_yaml.md.

5. Top-5 reasons it loses to js-yaml, ranked, and lessons for lightning-yaml

Section titled “5. Top-5 reasons it loses to js-yaml, ranked, and lessons for lightning-yaml”
  1. A fully materialized, lossless CST before any value exists (733MB / 73× input; ~4.7M objects; built completely before compose starts — parser.js:286-288). js-yaml has zero intermediate representation: an index-cursor recursive-descent loader writing directly into state.result (js-yaml/lib/loader.js). → Lesson: one pass, integer cursors into the source string, emit final JS values directly. No token objects, no token strings, ever, in the default parse path.
  2. A second full intermediate tree (Document/AST) + a third conversion pass (2.35M Scalar/Pair/YAMLMap objects, each with a range array and source string; 330MB; then toJS re-walks it). → Lesson: the rich Document API (ranges, comments, round-trip, srcToken) must be a separate opt-in API; parse() must never build nodes it will immediately throw away.
  3. Everything-is-a-string generator lexing: 3.86M yielded substrings for 10MB, re-classified by tokenType(source) string-switching, flowing through 3 nested generator layers (~58ns/item just for yield* plumbing; Lexer alone is 3× js-yaml’s total parse time). → Lesson: no generators in the hot path; a flat switch-on-charCode state machine with direct calls; represent token type/extent as local integers, not strings.
  4. Per-scalar composition overhead: two 10-field resolveProps objects per pair, a result object + 3-number range array per scalar, res += ch per-character decoding of double-quoted strings (rope churn on exactly the JSON-shaped inputs we benchmark), and a sequential ~8-regex tag battery per plain scalar. → Lesson: decode scalars by segment-capture with a zero-copy fast path (no escapes → single slice); type plain scalars by first-char dispatch + hand-rolled numeric scan, not regex arrays; allocate nothing per scalar except the value itself. (Caveat: slices pin the source via V8 SlicedString for ≥13 chars — for retained results consider flattening; for peak RSS slices win.)
  5. Error-tolerance/round-trip bookkeeping always-on: never-throw lexer/parser and full error recovery force every space/comment/newline to be tokenized and threaded through resolveProps/parsePrelude even in comment-free input; LineCounter line-index built by default for pretty errors; O(n²) duplicate-key mapIncludes; comment re-attachment logic in every collection resolver. → Lesson: fail fast on first error like JSON.parse; treat whitespace as something to skip (advance cursor), never to emit; duplicate keys are detected for free by the target object/Map insertion; line/col computed lazily only when throwing.

Meta-lesson for the memory benchmark specifically: peak RSS tracks allocation rate, not just live-set — yaml’s 25-iter loop hit 4.3GB RSS with only ~1.15GB ever live. Minimizing transient garbage (objects/strings per input byte) is itself the peak-RSS strategy; js-yaml wins memory mostly by allocating ~10× fewer objects, and JSON.parse wins by allocating almost nothing but the output.

Scratch artifacts (fixture + measurement scripts, reproducible): /tmp/claude-0/-home-user-lightning-yaml/5b5afb90-7c63-5a56-af0b-8798d3e9b488/scratchpad/{fix10mb.json,stage.js,retained.js}.

KEY FACTS

Section titled “KEY FACTS”
  • yaml v2.9.0 parse() pipeline = Lexer -> Parser(CST) -> Composer(Document) -> doc.toJS(), wired as 3 nested generators; Composer only starts after the ENTIRE single-document CST is materialized because Parser.pop() yields only when its stack empties (dist/parse/parser.js:279-288; dist/public-api.js:46,78)
  • Lexer yields one JS string per lexical token via buffer.substr/slice — including every space run, comma, colon, bracket (dist/parse/lexer.js:616-634,701-713); measured 3,860,166 lexemes for a 10.0MB JSON fixture (1 per 2.6 bytes)
  • Every lexeme becomes a CST sourceToken object {type, offset, indent, source} pushed into start/sep/end/items arrays (dist/parse/parser.js:231-239,862-876); measured CST = ~3.68M token objects + ~1.03M item records retaining 733MB for 10MB input (73x)
  • Composed Document tree for 10MB input = 1,387,246 Scalars + 663,466 Pairs + 301,577 collections (2.35M nodes, each with a 3-number range array, source string, defineProperty’d NODE_TYPE) retaining 330MB (33x); CST and Document coexist during compose -> ~1.1GB live per parse (compose-scalar.js:35-44; nodes/Node.js:9)
  • Measured on Node v22.22.2, 10MB JSON: YAML.parse 9.7-10.7s vs js-yaml 0.49-0.53s (~20x) vs JSON.parse 0.09-0.105s (~100x); 25-iteration loop maxRSS: yaml 4302MB vs js-yaml 564MB vs JSON 643MB — reproduces the harness’s yaml-vs-js-yaml memory blowout; peak RSS tracks allocation rate, only ~1.15GB ever live
  • yaml’s Lexer stage alone takes 1.56s on 10MB — 3x js-yaml’s entire parse; a 3-deep yield* generator chain costs ~58ns/item (223ms vs 12ms for 3.86M items) — generators and per-lexeme tokenType() string-switch are pure overhead (dist/parse/cst.js:40-98)
  • Hot-path pathologies: doubleQuotedValue builds strings one ‘res += ch’ per character (dist/compose/resolve-flow-scalar.js:118-173) vs js-yaml’s segment capture (js-yaml/lib/loader.js:354,688-743); plain scalars run a sequential ~8-regex tag battery (dist/compose/compose-scalar.js:72-77); duplicate-key check is O(n^2) items.some per mapping (dist/compose/util-map-includes.js:12); resolveProps allocates a 10-field object twice per map entry (dist/compose/resolve-props.js:134-145)
  • Maintainer eemeli (GitHub discussion #358, Feb 2022): yaml is ‘built for power rather than speed, and simply does more with its input than e.g. js-yaml’; ‘most real-world files are at most a few hundred lines long’; benchmark in-thread: read 500k-line file JSON 94ms / js-yaml 504ms / yaml@1.10 2108ms / yaml@2 9584ms (v2 slower than v1 at reading; faster than js-yaml at writing) — no plan to change the architecture; only targeted fix is PR #612 (2.8.0) caching anchor/alias lists for 12-64x on alias-heavy docs (https://github.com/eemeli/yaml/discussions/358, https://github.com/eemeli/yaml/pull/612)
  • The CST losslessness is a documented guarantee — it ‘retains every character of the input, including whitespace’ and lexer/parser ‘should never throw’ (https://github.com/eemeli/yaml/blob/main/docs/07_parsing_yaml.md) — i.e., round-trip/comments/error-recovery are the structural cost, and must be kept out of lightning-yaml’s parse() hot path
  • Lessons for lightning-yaml: single pass with integer cursors direct to JS values (js-yaml-style, no CST/AST); no generators; no per-token strings or objects; zero-copy slice fast path for escape-free scalars (mind V8 SlicedString pinning >=13 chars, v8 src/objects/string.h kMinLength=13); first-char dispatch instead of regex batteries; dup-key detection via target-object insertion; lazy line/col only on error