Completeness critique: gaps in the lightning-yaml research + designs (ranked)

Produced by a multi-agent research session (Claude Code). All local numbers were measured on the session container — Node v22.22.2 / V8 12.4, 4-vCPU Xeon 2.80 GHz, 16 GB RAM; absolute MB/s are machine-specific, ratios are the durable signal. Referenced scratchpad/*.mjs scripts were session throwaways and are not committed.

TIER 1 — Contradicts the harness or leaves the core claim unmeasured

1. stringify is a mandatory, benchmarked op that ALL THREE designs ignore

The Candidate interface requires stringify: (value: unknown) => string (/home/user/lightning-yaml/bench/candidates.ts:26-27), so registering group ours (the CLAUDE.md-mandated integration point) will not even typecheck without one. Worse, bench:self — which CLAUDE.md requires before every commit — runs stringify.bench.ts (bench/report.ts:57) and the memory matrix runs OPS = ["parse","stringify"] for every dataset (bench/memory/run.ts:30,71-77). So the README “Our implementation” block will publish stringify speed + stringify peak-RSS rows from day one. No design specifies a dumper, a delegation choice (JSON.stringify output is valid flow YAML, but then the table advertises native numbers as “ours” — Design C’s optics problem squared), or a harness change (which would contradict CLAUDE.md’s “candidates.ts is the single source of truth”). Competitive twist the dossier itself recorded but no design used: stringify is the one axis where eemeli/yaml beats js-yaml (2442 ms vs 7946 ms, github.com/eemeli/yaml/discussions/358) — the write path has different competitive dynamics than the read path and is currently a total design void.

2. Native N-API addon route was never evaluated — despite the dossier’s own precedent being N-API, not wasm

The transcode-to-JSON+JSON.parse trick that anchors Design B is sourced from swc/oxc (https://github.com/oxc-project/oxc/issues/2409) — those are N-API native addons, not wasm. A napi-rs/prebuildify addon (ryml or saphyr → JSON string → JSON.parse) pays no 1.45-2.5× wasm derate, no emscripten +20% memory overshoot, and no never-shrink linear memory (malloc’d native memory is returned to the OS), i.e. it strictly dominates Design B on both benchmark axes for the stated primary target (Node ≥20). Its real costs — per-platform prebuilds, install-fallback complexity, no browser story without a second build — were never scored against B’s costs (WIP emscripten port, base64+33% package, Chrome M114 sync floor). Other unconsidered strategies: (a) shape-specialized codegen (new Function object-literal allocator per observed mapping shape — flagged as “the extreme version” in the V8 dossier T6, never carried into any design or measured); (b) worker-thread offload (legitimately excluded by the sync API, but no design says so); (c) simdjson-style structural pre-scan in wasm feeding a JS materializer (mentioned once as a hypothetical, never costed); (d) fork-and-strip js-yaml as a low-risk starting point that inherits its conformance behavior and test expectations — never considered as an alternative to from-scratch.

3. The actual current competitor, js-yaml 5.2.1, remains unmeasured — and the dossier contradicts itself about what was measured

Verified now: npm latest is js-yaml 5.2.1 (registry.npmjs.org/js-yaml/latest; yaml is current at 2.9.0). Every ratio in all three designs targets 4.x. Installing and benchmarking 5.2.1 is a minutes-long task that no research pass did; if the 2026 TS rewrite changed loader throughput either way, every headline (“3× js-yaml”) and every semantics-mirroring decision is stale at ship time. Additionally, the dossier is internally inconsistent about the 4.x baseline itself: the “Local micro-benchmarks” section states “Deps: js-yaml 4.1.0” while the js-yaml source analysis (and my verification: node_modules/js-yaml → .pnpm/js-yaml@4.3.0, package.json “version”: “4.3.0”) shows only 4.3.0 is installed — so the a-parsers numbers labeled 4.1.0 were actually measured against 4.3.0 (which has extra per-node snapshot allocations, i.e. is plausibly slower than 4.1.0, flattering “ours vs js-yaml” ratios).

4. Block-mode throughput — the product’s real win condition — is 100% extrapolation

Designs A and C both admit no block-mode analog of minijson exists; the 55-80 MB/s figures rest on invented multipliers (“≤1.5× flow penalty”, “10-25% generality tax”). The committed benchmark is flow-JSON, so the only number that proves “clearly beats js-yaml on YAML” is the one nobody prototyped. A 1-2 day spike (indent tracking + plain-scalar end detection + block-map composition on a yaml.stringify’d fixture) would convert the largest open risk into data before architecture lock-in. Until then, C’s own stated failure mode (block lands at 1.5-2× js-yaml → win condition fails while the fixture table “looks spectacular”) is live and untested.

TIER 2 — YAML semantic surface that can sink the designs

5. No design has a compatibility matrix vs js-yaml’s DEFAULT_SCHEMA behavior

Under-examined features js-yaml users actively depend on: merge keys << (default schema); timestamp→Date — js-yaml resolves unquoted 2026-07-12 to a Date by default; Design C explicitly drops this, A is silent — a guaranteed migration surprise for config files with bare dates; binary/omap/pairs/set types (in js-yaml’s default schema; !!binary → Buffer); non-string keys (js-yaml does String(keyNode) → '[object Object]' — match it or diverge?); unknown-tag behavior; multi-document streams / loadAll (API surface absent from all designs; harness is single-doc only); %YAML/%TAG directives; 1.1-era int forms js-yaml still accepts (0b…, sexagesimal — nodeca/js-yaml#627). Duplicate keys: js-yaml throws by default (“duplicated mapping key”, node_modules/js-yaml/lib/loader.js:442) — C’s last-wins delegation silently changes error behavior, and “dup detection free via object insertion” is not free if throw-parity is required (needs a per-key in/hasOwnProperty check after all).

6. Anchors/aliases: cycles and reference identity are fatal to Design B and unspecced in A/C

js-yaml supports cyclic aliases (&a [*a]) and returns shared object references for repeated aliases. Design B’s YAML→JSON transcode cannot represent cycles at all (infinite emit) or reference sharing (structural copies) — this is a hard input-class failure beyond the acknowledged “alias bomb” budget guard, and it’s unhandled. A and C say “pay-on-first-use anchor machinery” but never design alias resolution, cycle support, or the anchor-inside-flow / anchor-on-mapping-key edge cases; the dossier’s own evidence (Stripe yarn.lock: 50 ms js-yaml vs 6,900 ms yaml@2) proves alias handling is where competitors die, yet no design specs it.

7. Delegation soundness (Design C) has unenumerated holes in both directions

JSON accepted / YAML rejected: lone-surrogate escapes ("\uD800") are valid JSON.parse input but outside YAML’s c-printable character set — a strict YAML parser must reject what the router silently accepts. js-yaml-rejected / JSON.parse-accepted: js-yaml 4.3.0 defaults maxDepth = 100 and throws above it (loader.js:154, 1421-1422), and throws on duplicate keys — valid deep or dup-keyed JSON therefore succeeds under delegation but fails under js-yaml, a divergence absent from C’s enumeration. No design proposes cross-checking the enumeration against the yaml-test-suite’s JSON-subset cases, which is the cheap mechanical way to close it.

8. Security/robustness parity is unaddressed

(a) Prototype pollution: a fast path doing obj[key] = v is vulnerable to __proto__ keys; js-yaml special-cases it (loader.js:121-133) and JSON.parse is engine-safe; the designs cite skipping js-yaml’s per-pair checks as a speedup without committing to a safe-assignment strategy. (b) Stack overflow: V8’s JSON.parse falls back to an iterative continuation-stack parser near stack exhaustion (V8 dossier T3); a pure-JS recursive-descent parser throws RangeError on ~10³-10⁴ depth — no design specs a depth guard or iterative fallback, and the nested fixture (makeTree depth 6, bench/fixtures/generate.ts:69) never tests it. (c) DoS guards: js-yaml 4.3.0 ships maxDepth/maxTotalMergeKeys (loader.js:154-155); shipping a faster parser with fewer DoS protections is a regression story waiting to be written up.

TIER 3 — Benchmark-validity traps not yet accounted for

9. Fixture blind spots (verified directly): zero escapes, zero non-ASCII, unused shapes

I scanned all six generated fixtures: 0 backslash escapes and 0 non-ASCII bytes in every file (952 B → 10.7 MB). Consequences: the escape-decode slow path — a major differentiator vs eemeli/yaml’s per-char res += ch pathology and a correctness hotspot — is never exercised by the committed benchmark; two-byte-string handling is never exercised. Moreover datasets.ts defines strings and numbers shapes (bench/fixtures/datasets.ts:14) with generators already written (generate.ts:39-52, incl. escape-bearing quoted: "line1\nline2\t…"), but the matrix (datasets.ts:26-33) uses only records/nested. The “beats native on string-heavy input” claim has no fixture behind it, and the numbers shape is the measured worst case for pure JS (2.6× slower than native, bench3) — if the maintainer ever enables it, headline ratios drop materially. The homogeneous 8-key record shape also maximally flatters hidden-class feedback and key interning; no heterogeneous-key or adversarial fixture exists.

10. The committed heap Δ column will expose SlicedString pinning and intern-cache retention — undesigned

The memory worker measures heapUsed after GC while the last parse result is still reachable (bench/memory/worker.ts:39-49), and README calls heap Δ one of “the stable figures” (README.md:106-107). A slice-based parser’s retained tree pins the entire source string via SlicedStrings (≥13 chars), so ours’ heap Δ ≈ JSON’s + full fixture size (+ any module-level key-intern cache) — e.g. ~32 MB vs JSON’s ~17.4 MB on the 10 MB fixture, a visibly worse committed ratio. Designs discuss RSS at length but never this column; the countermeasure (flatten/copy strings above a threshold, bounded intern cache, per-parse cache) trades against the very slice fast path the speed numbers depend on. Decision needed before M1, since bench:self publishes it per commit.

11. Warm-cache benchmarking overstates everyone’s steady state — and hides cold-start

mitata parses the same string object thousands of times; V8 caches FastIterableState on descriptor arrays across parses (V8 dossier T6), so JSON.parse’s benchmark speed partially reflects cross-parse warmup, and ours’ shape/intern caches warm identically. Meanwhile the dominant real-world YAML workload (parse one config file, exit) runs in interpreted/baseline tiers — a pure-JS parser needs thousands of iterations to reach TurboFan, and no design measured first-call or 100-call performance. The 1 KB “2.5-4× js-yaml” claims implicitly assume fully optimized code.

12. Harness mechanics the designs assume but that don’t exist

(a) Design C’s ethically-required “router-off” row has no mechanism: candidates.ts has no per-candidate variants/options; the fix (register two ours candidates) is easy but unspecced and doubles every self-bench row including stringify. (b) Adding block fixtures = “datasets change” → CLAUDE.md mandates a full bench:competition re-run, including yaml@2.9 on a 10 MB block fixture at ~1.5 MB/s (≈7-8 s/parse × mitata sampling + 25 memory iterations at multi-GB RSS on a 16 GB box) — runtime and OOM risk never sized. (c) Design A’s “<5% regression budget enforced by bench:self” has no enforcement substrate: bench:self compares only against the JSON baseline, overwrites the README block in place (report.ts:110-115), keeps no history, and README itself warns timings drift run-to-run; nobody quantified the mitata noise floor per fixture, so a 5% gate is currently unenforceable. (d) Trivial but real: worker default iters is 20 (worker.ts:20) vs orchestrator 25 (run.ts:29) — only manual invocations diverge, but it contradicts “iterations are fixed”.

13. Unresolved internal contradiction: bytes vs charCodeAt scanning

The pure-JS ceiling section claims Uint8Array scanning is ~1.5× faster than charCodeAt (760 vs 513 MB/s, bench1); the local micro-benchmark section claims parity (1253 vs 1255 MB/s, bench b) on the same machine/Node. The “parse from the JS string, never transcode” doctrine — load-bearing for Designs A and C — rests on the second measurement. The discrepancy (likely different string flatness or loop structure) was never reconciled; if bytes scanning really is 1.5× faster, a bytes-prescan hybrid deserves reconsideration.

TIER 4 — Ecosystem / maintenance angles

14. Browser and engine portability claims are evidence-free

“Browser nice-to-have” appears in the goal, yet zero measurements exist on JSC/SpiderMonkey, where the tuned-for constants (SlicedString kMinLength=13, indexOf memchr speed, Smi behavior, hidden-class transition sharing) differ or don’t exist. Two-byte (non-ASCII) input performance is unmeasured on any engine. Node 20 (older V8) — the stated floor per package.json engines (“:8-10”) — was never benchmarked; all constants come from Node 22.22.2/V8 12.4.

15. Product surface undefined for the pure-JS designs

No design covers: ESM/CJS packaging, TypeScript types, options-API compat with js-yaml (schema, json, onWarning, filename, listener), error-class shape (YAMLException with mark/snippet — js-yaml builds these lazily but users catch them), loadAll, or bundle size vs js-yaml’s ~50 KB. Design B covers packaging; A and C skip it entirely.

16. The conformance bar is undefined because js-yaml’s own yaml-test-suite pass rate was never measured

“Match js-yaml’s pass rate, document deviations” (Design A) is unactionable: nobody ran the yaml-test-suite against js-yaml 4.3.0 or 5.2.1 to establish the number. rapidyaml’s “100% with deviations” shows how gameable the scoreboard is. Also missing: a real-world corpus (k8s manifests, GitHub Actions workflows, docker-compose, OpenAPI, the repo’s own /home/user/lightning-yaml/pnpm-lock.yaml) for both conformance and block-mode perf — every speed datapoint in the entire research effort is JSON-shaped.

17. No CI exists

The repo has no .github/ or CI config; designs assume “suite-in-CI from Phase 1” and per-commit bench:self discipline, but both are manual conventions with no enforcement — a process gap given the README-table-per-commit workflow CLAUDE.md prescribes.

KEY FACTS

• All three designs omit stringify, but the harness makes it mandatory and published: Candidate requires a stringify fn (bench/candidates.ts:26-27), bench:self runs stringify.bench.ts (bench/report.ts:57) and the memory matrix runs OPS=[parse,stringify] (bench/memory/run.ts:30) — and stringify is the one axis where eemeli/yaml beats js-yaml (2442 vs 7946 ms, github.com/eemeli/yaml/discussions/358)
• A native N-API addon (napi-rs + ryml/saphyr -> JSON string -> JSON.parse) was never evaluated, even though the dossier’s transcode precedent (swc/oxc, github.com/oxc-project/oxc/issues/2409) is N-API not wasm — it avoids the 1.45-2.5x wasm derate and the never-shrink linear-memory floor, strictly dominating Design B for Node >=20; shape-codegen (new Function per mapping shape) and fork-js-yaml strategies also unexplored
• js-yaml 5.2.1 is the verified npm latest (registry.npmjs.org/js-yaml/latest) and has never been benchmarked; the dossier also self-contradicts on the 4.x baseline — micro-bench section says 4.1.0 installed, actual is 4.3.0 (node_modules/.pnpm/js-yaml@4.3.0, verified)
• Block-mode throughput — the only number that proves ‘clearly beats js-yaml on real YAML’ — is pure extrapolation with invented multipliers (‘<=1.5x flow penalty’); no block-scanner spike was ever built or measured, and Design C admits a 2-3x tax outcome kills the win condition
• Cyclic and shared-reference aliases (&a [*a]) are unrepresentable in Design B’s YAML->JSON transcode (hard failure class beyond ‘alias bombs’) and alias/merge-key resolution is unspecced in Designs A/C despite being where competitors die (Stripe file: 50 ms js-yaml vs 6900 ms yaml@2)
• No design has a compat matrix vs js-yaml defaults: timestamp->Date for bare dates, merge keys, !!binary/omap/set, loadAll multi-doc, non-string keys, dup-keys THROW by default (lib/loader.js:442), and js-yaml 4.3.0’s maxDepth=100 default (loader.js:154,1421) — valid deep/dup-key JSON that JSON.parse accepts but js-yaml rejects punches an unenumerated hole in Design C’s delegation-soundness claim, as do lone-surrogate escapes (valid JSON, outside YAML c-printable)
• Verified fixture blind spots: all 6 generated fixtures contain 0 backslash escapes and 0 non-ASCII bytes, and the strings/numbers shapes exist in datasets.ts:14/generate.ts:39-52 but are absent from the matrix (datasets.ts:26-33) — the escape path, the ‘beats native on strings’ claim, and the measured 2.6x-worse numbers case are all untested by the committed benchmark
• The committed heap-Δ column (README calls it a ‘stable figure’) is measured with the last result retained (bench/memory/worker.ts:39-49), so a slice-based parser’s SlicedStrings pin the whole source (~+10.7 MB vs JSON’s 17.4 MB on xlarge) and module-level intern caches count too — no design addresses this column
• Security/robustness parity unaddressed: proto pollution via naive obj[key]=v fast paths (js-yaml guards at loader.js:121-133), and stack-overflow on deep nesting — V8 JSON.parse falls back to an iterative parser but a recursive pure-JS parser throws RangeError; no design specs a depth guard/iterative fallback, and js-yaml 4.3’s DoS guards (maxDepth/maxTotalMergeKeys, loader.js:154-155) have no counterpart
• Unresolved dossier contradiction: Uint8Array scanning measured 1.5x faster than charCodeAt in one section (760 vs 513 MB/s) and exactly at parity in another (1253 vs 1255 MB/s) — the ‘parse from the JS string’ doctrine load-bearing for Designs A/C rests on the unreconciled second result; also zero measurements exist on Node 20 (the engines floor), Node 24, JSC, or SpiderMonkey despite the browser nice-to-have goal