Skip to content
Lightning YAML

js-yaml loader deep-dive (installed: js-yaml@4.3.0, NOT 4.1.0)

Produced by a multi-agent research session (Claude Code). All local numbers were measured on the session container — Node v22.22.2 / V8 12.4, 4-vCPU Xeon 2.80 GHz, 16 GB RAM; absolute MB/s are machine-specific, ratios are the durable signal. Referenced scratchpad/*.mjs scripts were session throwaways and are not committed.

Version note: repo declares "js-yaml": "^4.1.0" (/home/user/lightning-yaml/package.json:23) but pnpm resolved 4.3.0 (/home/user/lightning-yaml/node_modules/js-yaml → node_modules/.pnpm/js-yaml@4.3.0/…; lib/package.json "version": "4.3.0"). 4.3.0 adds per-node state-snapshot allocations and DoS guards (maxDepth/maxTotalMergeKeys) not in 4.1.0 — this is what the benchmark actually measures. All paths below are under /home/user/lightning-yaml/node_modules/js-yaml/lib/.

Update — 2026-07-20. The repo now pins js-yaml 5.2.1 (package.json declares "js-yaml": "^5.2.1"; the lockfile resolves 5.2.1), a Rollup-bundled rewrite that ships only dist/js-yaml.mjs — there is no lib/ tree, so the lib/…:NNN coordinates throughout this note are 4.3.0’s. This deep-dive still stands as a portrait of the 4.3.0 loader, but one finding it leans on has changed in 5.2.1 and is worth flagging.

Implicit scalar typing is first-char-indexed in 5.2.1. Each implicit scalar tag now declares an implicitFirstChars set (e.g. bool → t/f, int → - and 0–9, merge → <, or null meaning “any first char”). The Schema constructor compiles these into a Map<firstChar, tag[]> (implicitScalarByFirstChar) plus an implicitScalarAnyFirstChar fallback bucket, and the hot path (constructScalar) resolves a plain scalar with implicitScalarByFirstChar.get(source.charAt(0)) ?? implicitScalarAnyFirstChar, trying only that shortlist. So a scalar whose first character no type claims — hello, say — misses the map and becomes a string with zero resolve() calls, not the six that §3, §6 (#2), and KEY FACTS describe. Read those three as 4.3.0-only; the 5.2.1 cost is one Map lookup plus a short, first-char-relevant candidate list.

Weakness #2 (“replace the 6-resolver loop with a first-char switch”) is now resolved on both sides. js-yaml adopted the first-char index in v5, and lightning-yaml already implements the same idea, more aggressively, in resolvePlain (src/index.ts:2096): a switch on the integer charCodeAt (a jump table — no one-character-string allocation, no Map hash) over src offsets, so no substring is materialized unless the scalar really is a string, with the number grammar inlined (Smi accumulation) instead of a polymorphic tag.resolve + regex. For this project’s fixed YAML-1.2-core schema the hardcoded switch strictly dominates js-yaml’s schema-driven map; js-yaml’s structure only pays off if you need pluggable custom scalar tags, which we deliberately don’t.

1. Parser architecture

Section titled “1. Parser architecture”
  • Single-pass recursive-descent composer with NO tokenizer and NO AST. There is no token/event layer at all: composeNode (loader.js:1410) recursively calls the readers (readBlockSequence loader.js:1011, readBlockMapping :1078, readFlowCollection :759, readBlockScalar :869, quoted/plain scalar readers :644/:688/:541), and each reader directly builds the final plain JS value (_result = [] loader.js:1014/781, _result = {} :1085/785, scalars accumulate into state.result string). _result.push(state.result) (loader.js:1056) and setProperty(_result, keyNode, valueNode) (loader.js:445) write straight into the output objects.
  • One mutable god-object State (loader.js:142-184) threaded through every function: holds input, position, line, lineStart, lineIndent, firstTabInLine, depth, plus per-node “return channel” fields tag/anchor/kind/result (commented out in the constructor at loader.js:175-183, assigned later — deliberate, but means hidden-class transition after construction and a polymorphic result field that alternates string/array/object/null). Readers “return” values by mutating state.result/state.kind and returning boolean.
  • Speculative parse with save/restore, not re-scan: readBlockMapping parses any node as a potential implicit key via composeNode(state, flowIndent, CONTEXT_FLOW_OUT, false, true) (loader.js:1145) and only then looks for : (loader.js:1158); if absent it returns keeping the composed result (loader.js:1178-1188) — so scalars are scanned once, but every block-level node pays an extra readBlockMapping+composeNode frame. True backtracking exists only for property edge-cases: tryReadBlockMappingFromProperty (loader.js:1387-1408) snapshots/restores full state and uses an anchor-transaction stack (loader.js:210-262).
  • Trial order per node (loader.js:1506-1538): blockSeq → blockMap → flowCollection → blockScalar → singleQuoted → doubleQuoted → alias → plainScalar; each failed probe is a cheap charCodeAt+compare except readBlockMapping (see above).
  • Entry: load → loadDocuments (loader.js:1720-1759) → readDocument (:1620) → composeNode. Input is NUL-terminated (state.input += '\0' loader.js:1747) so all loops use ch !== 0 instead of bounds checks — “Use 0 as string terminator. That significantly simplifies bounds check” (loader.js:1746).

2. Scanning technique

Section titled “2. Scanning technique”
  • Pure charCodeAt loops everywhere; state.position is the single cursor. Char classification via tiny predicate functions of 2-4 comparisons: isEol (loader.js:31), isWhiteSpace (:35), isWsOrEol (:39), isFlowIndicator (:46) — comparison chains, not lookup tables. The only table is the 256-entry simpleEscapeCheck/simpleEscapeMap for double-quote escapes (loader.js:135-140).
  • Scalar capture = deferred slice + += concat: readers track captureStart/captureEnd and flush via captureSegment (loader.js:354-372): state.input.slice(start, end) then state.result += _result (:370). Single-line scalars = 1 slice + 1 concat onto '' (V8 fast path); multi-line/escaped scalars = one slice+concat per line/segment, plus fold chars via writeFoldedLines (:533-539) using common.repeat — itself a += loop, not native String.prototype.repeat (common.js:31-39).
  • Every captured segment gets a second validation pass: quoted scalars (checkJson=true) run a charCodeAt loop over the whole slice (loader.js:358-365); plain/block scalars run PATTERN_NON_PRINTABLE.test(_result) — a regex over every scalar’s content (loader.js:366, pattern at :20).
  • Plain-scalar per-char loop (loader.js:583-631): checks :+lookahead charCodeAt (:584-590), #+look-behind charCodeAt (:591-595), position === lineStart && testDocumentSeparator (:597), EOL, whitespace — up to ~3 charCodeAt calls and 5+ branches per character. Entry gate is a 13-way comparison chain (:553-565).
  • Position/line tracking is always-on but cheap: state.line++, lineStart = position, firstTabInLine = -1 only in readLineBreak (loader.js:452-469); indent counted in skipSeparationSpace (:471-510); column computed lazily only on error (position - lineStart, loader.js:192). No per-char line bookkeeping — a good design to copy.

3. Implicit typing (core/default schema)

Section titled “3. Implicit typing (core/default schema)”

Describes the 4.3.0 loader. 5.2.1 first-char-indexes this step (implicitScalarByFirstChar) — see the Update at the top.

  • Core schema is JSON schema in js-yaml (schema/core.js:9), default = core + timestamp,merge implicit (schema/default.js:9-13). compiledImplicit order: null, bool, int, float, timestamp, merge (schema/json.js:11-16 concat’d per schema.js:99).
  • Resolution (composeNode, loader.js:1566-1577): only plain scalars (tag set to '?' at :1535-1537) loop over all 6 type.resolve(state.result) in order until one matches; quoted scalars skip entirely (tag stays null → :1551). On match: state.result = type.construct(state.result) (:1570).
  • Cost per plain string (e.g. hello): 6 dynamic type.resolve calls through Type objects (type.js:51) — null (string compares, type/null.js:5-12), bool (compares, type/bool.js:5-12), int (char loop, type/int.js:20-90), float: anchored-regex YAML_FLOAT_PATTERN.test on every scalar (type/float.js:6-15,27), timestamp: up to TWO RegExp.exec calls (type/timestamp.js:24-25), merge (=== '<<'). Fail-fast anchors help, but that’s 3 regex-engine invocations + 6 calls per unmatched plain scalar.
  • Numbers are parsed 2-3x: int resolve does full char validation loop + isFinite(parseYamlInteger(data)) (type/int.js:52,63,74,89 — parseInt inside), then construct calls parseYamlInteger again (type/int.js:115-117). Float resolve does regex test + parseFloat (type/float.js:27-31); construct does data.toLowerCase() (a fresh string allocation per float) + slice + parseFloat again (type/float.js:38-51). And before any of this the number was already materialized as a JS string via slice+non-printable regex (loader.js:356,366). JSON.parse does none of this — digits → double directly.

4. Allocation profile per node/scalar

Section titled “4. Allocation profile per node/scalar”
  • Strings: ≥1 slice per scalar segment (loader.js:356); rope concat per extra segment; toLowerCase copy per float (type/float.js:39); String(keyNode) per mapping key (loader.js:421, no-op for strings); numeric strings are garbage immediately after construct.
  • Objects: final {}/[] per collection (loader.js:781/785/1014/1085); Object.create(null) overridableKeys per mapping — block AND flow (loader.js:1086, 771); NEW in 4.3.0: one 9-field snapshotState object per composeNode with indentStatus===1 — allocated at loader.js:1457 on the first property-loop iteration even when the node has no tag/anchor, i.e. per almost every node; tagMap + anchorMap Object.create(null) per document (loader.js:1627-1628); documents array (:172); anchorMapTransactions array (:173).
  • Per mapping pair: hasOwnProperty ×2 for dup-key check (loader.js:437-438, skipped when json:true option), __proto__ string compare in setProperty (:121-133), and a delete overridableKeys[keyNode] on every pair (loader.js:446) — a dictionary-mode delete per key even when merge keys never occur.
  • Errors/Marks: allocated only on throw/warn — generateError copies mark fields plus state.input.slice(0, -1) (the whole document!) and builds a snippet by regexing the full buffer (loader.js:186-198, snippet.js:41-54; exception.js:24-41 with captureStackTrace). Zero cost on happy path — good design.
  • Closures: none per node; all functions are module-level. Listener is two !== null checks per node (loader.js:1427-1429, 1612-1614).
  • Input copies before parsing: String(input) (loader.js:1721), input += '\n' if no trailing newline (:1726-1729), BOM slice (:1732-1734), full-input indexOf('\0') scan (:1739), state.input += '\0' (:1747 — rope, flattened to a full copy on first charCodeAt). For a 10MB doc: ~1-2 full copies + 1 full scan before byte one is parsed.

5. Why js-yaml beats eemeli/yaml (architecture)

Section titled “5. Why js-yaml beats eemeli/yaml (architecture)”

js-yaml: 1 pass, 0 intermediate representations, values composed in place. eemeli/yaml@2.9.0 (node_modules/yaml/dist/): 4 stages — (a) generator-based Lexer yielding substring tokens via yield* chains (parse/lexer.js:207-216, 228-242), (b) Parser building CST token objects {type, offset, source} pushed into arrays per item (parse/parser.js:231-248, 496-524), (c) Composer wrapping everything in new Scalar/new Pair/new YAMLMap AST nodes with per-node range arrays (compose/compose-scalar.js:28-35, compose/resolve-block-map.js:90,113), then (d) doc.toJS() walking the AST to build plain JS values — the tree is built twice (public-api.js parse() → doc.toJS). Plus a LineCounter is on by default via prettyErrors !== false (public-api.js parseOptions). Maintainer confirmation (eemeli, github.com/eemeli/yaml/discussions/358): “yaml handles its input in stages: lexing, parsing to a CST, composing an AST, and then converting to JS. It all adds up” — built “for power rather than speed”. Reading benchmarks there: js-yaml 504ms vs yaml v2 9,584ms; Stripe’s anchor-heavy 14k-line file: js-yaml@4.1.0 50ms vs yaml@2.3.4 6,900ms (aliases in yaml v2 re-resolve/count nodes per alias). Generator yield* per token is itself expensive in V8. Lesson: the intermediate-representation tax dominates; js-yaml’s direct composition is the right baseline to beat, and the remaining headroom vs JSON.parse is the per-node constant factor below.

6. WEAKNESSES a from-scratch parser can exploit

Section titled “6. WEAKNESSES a from-scratch parser can exploit”
  1. Every scalar is double-scanned for validity — slice then charCodeAt re-loop (quoted, loader.js:358-365) or PATTERN_NON_PRINTABLE.test regex (plain/block, loader.js:366). Fold validation into the main scan loop → one pass, zero regex.
  2. [Resolved in 5.2.1, and already implemented in lightning-yaml — see the Update at the top.] Implicit typing = 6 polymorphic resolver calls + 3 regex invocations per plain scalar (loader.js:1566-1577; type/float.js:27; type/timestamp.js:24-25). Replace with a first-char switch (digit/‘-’/‘t’/‘f’/‘n’/‘~’…) and inline scanning — most plain scalars are decided in O(1) dispatch. Timestamp/merge can be gated on first-char being digit / ‘<’.
  3. Numbers materialized as strings then parsed 2-3x (slice loader.js:356 → int: validate loop + parseInt in resolve int.js:52/89 + parseInt again in construct int.js:115-117; float: regex + parseFloat ×2 + toLowerCase() alloc float.js:27-51). Parse digits to a number directly from the input buffer during the scan, JSON.parse-style; never allocate the intermediate string.
  4. Per-node snapshot allocation (4.3.0): 9-field object from snapshotState allocated for virtually every node before even checking whether a !/& property exists (loader.js:1455-1457, 264-276). Peek the char first; snapshot lazily.
  5. Per-pair overhead in mappings: two hasOwnProperty dup-key checks (loader.js:437-438), String(keyNode) (:421), __proto__ compare (:123), and an unconditional delete overridableKeys[keyNode] (:446) + per-mapping Object.create(null) (:1086,771). A fast parser can skip dup detection (JSON.parse semantics; js-yaml itself skips it with json:true) and handle __proto__/merge lazily only when actually seen.
  6. Flow collections (the JSON path!) are fully generic: per element — 3× skipSeparationSpace (loader.js:797,834,854), full composeNode recursion with tag/anchor probes + snapshot alloc, then pair-detection logic (state.line === _line && ch === ':', :838-844) even for plain array elements, plus storeMappingPair’s machinery for map entries (:846-852). A dedicated tight flow-mode scanner (switch on [ { " digit t f n) would collapse ~10 function calls per element to ~1.
  7. Failed-probe chain per node: blockSeq→blockMap→flowColl→blockScalar→’→“→*→plain (loader.js:1506-1532), and every block-level scalar is parsed inside an extra readBlockMapping “implicit key” attempt frame (:1145,1178-1188) — two composeNode frames (and two snapshot objects) per top-level scalar. Dispatch on current char instead of trying readers in sequence.
  8. Char classification via comparison-chain function calls (loader.js:31-52) rather than a 256-byte flags table; plain-scalar loop does lookahead/look-behind charCodeAt per char (:584-595). A single table lookup FLAGS[ch] & STOP per char is measurably cheaper on 10MB inputs.
  9. Result strings built by repeated += (loader.js:370; block scalars: one slice+concat per line, :1005, plus common.repeat’s own += loop, common.js:31-39). Fine for V8 ropes but flattening cost + garbage; for single-segment scalars prefer returning the slice directly (js-yaml already effectively does via '' + s); for multi-segment, collect parts and join.
  10. Up-front input copies + full scan: trailing-\n concat, NUL append (full-copy on flatten), and whole-input indexOf('\0') (loader.js:1726-1747). A from-scratch parser can bounds-check with pos < len (or pad once into a scratch buffer) and skip the NUL scan.
  11. Always-on generality even when unused: listener checks ×2/node (loader.js:1427,1612), depth guard (:1421-1425), tag/anchor probes ×2/node (:1467), anchor-null checks ×4/node (:1540,1552,1572,1606), anchorMap/tagMap per doc (:1627-1628), multi-doc loop (:1754-1756). Individually cheap; collectively part of the ~10-20x constant factor vs JSON.parse. Design features as pay-on-first-use (e.g. only enter anchor machinery after the first &).
  12. State object polymorphism: state.result alternates string/object/array/null and tag/kind/anchor are added post-construction (loader.js:175-183, 1431-1434) — ICs at state.result += and reader call-sites go polymorphic. Keep scalar accumulation in a dedicated local/monomorphic slot.

7. Maintainer insights (quick pass)

Section titled “7. Maintainer insights (quick pass)”
  • eemeli/yaml discussion #358 (“Performance yaml 1.10, yaml 2 and js-yaml”, https://github.com/eemeli/yaml/discussions/358): reading js-yaml 504ms vs yaml@2 9,584ms; maintainer attributes it to the lex→CST→AST→JS staging; Stripe anchor-heavy file 50ms (js-yaml) vs 6,900ms (yaml 2.3.4) — validates that alias handling must be O(1) reference sharing like js-yaml’s anchorMap[alias] (loader.js:1382).
  • nodeca/js-yaml#627 (https://github.com/nodeca/js-yaml/issues/627): core-schema number resolution deviates from YAML 1.2 (js-yaml accepts 0b/sexagesimal-era leftovers via its regexes; type/int.js:43-75) — a fresh parser targeting YAML 1.2 core can use the stricter, simpler 1.2 rules.
  • js-yaml README markets “Very fast” (https://github.com/nodeca/js-yaml); no open loader-perf issue found — its loader is considered the JS speed baseline; the exploitable headroom is the per-scalar resolver/regex/double-parse and per-node allocation overheads enumerated above, not algorithmic complexity.

Sources: github.com/eemeli/yaml/discussions/358, github.com/nodeca/js-yaml, github.com/nodeca/js-yaml/issues/627

KEY FACTS

Section titled “KEY FACTS”
  • Installed js-yaml is 4.3.0, not 4.1.0 (node_modules/.pnpm/js-yaml@4.3.0; repo package.json:23 declares ^4.1.0) — 4.3.0 adds a per-node snapshotState allocation (loader.js:1455-1457) absent in 4.1.0
  • Architecture: single-pass recursive descent with NO tokenizer/AST — readers compose final JS values directly into state.result via one mutable State object (loader.js:142-184, 1410, 1014, 1085)
  • Scanning: charCodeAt loops with predicate functions (isWsOrEol etc., loader.js:31-52, no lookup tables); scalars captured via captureSegment = input.slice + state.result += (loader.js:354-372)
  • Every captured scalar segment gets a redundant second validation pass: charCodeAt re-loop for quoted (loader.js:358-365) or PATTERN_NON_PRINTABLE regex for plain/block scalars (loader.js:366)
  • Implicit typing: every PLAIN scalar (tag ‘?’) runs up to 6 Type.resolve calls in order null,bool,int,float,timestamp,merge (loader.js:1566-1577; order from schema/json.js:11-16 + schema/default.js:10-13), including a float regex test (type/float.js:27) and two timestamp RegExp.exec (type/timestamp.js:24-25); quoted scalars skip resolvers entirely (loader.js:1551,1535) — 4.3.0 only: 5.2.1 shortlists resolvers by first char via implicitScalarByFirstChar (see Update at top)
  • Numbers are string-materialized then parsed 2-3x: int resolve isFinite(parseYamlInteger) then construct parseYamlInteger again (type/int.js:52,89,115-117); float regex + parseFloat in resolve then toLowerCase() allocation + parseFloat again in construct (type/float.js:27-51)
  • Per mapping pair: 2 hasOwnProperty dup-checks (skipped with json:true), String(key), proto compare, and an unconditional delete overridableKeys[keyNode] (loader.js:436-446, 121-133); per mapping an Object.create(null) overridableKeys (loader.js:1086, 771)
  • Flow collections (JSON input path) are generic: 3x skipSeparationSpace per entry (loader.js:797,834,854) + full composeNode with tag/anchor probes + pair-detection logic even for plain array elements (loader.js:838-851)
  • Pre-parse input costs: String(input), possible ‘+\n’ copy, full indexOf(‘\0’) scan, and state.input += '\0' NUL-terminator copy (loader.js:1720-1752); errors are pay-on-throw only but then slice the entire input into the Mark (loader.js:186-198)
  • js-yaml beats eemeli/yaml@2.9 because yaml runs 4 stages (generator lexer yielding substring tokens, CST token objects, Scalar/Pair/YAMLMap AST with range arrays, then doc.toJS rebuild — dist/parse/lexer.js:207-216, dist/parse/parser.js:231-248, dist/compose/compose-scalar.js:28-35, public-api.js); maintainer-confirmed at https://github.com/eemeli/yaml/discussions/358 (reading: js-yaml 504ms vs yaml v2 9,584ms; Stripe file 50ms vs 6,900ms)