Wiki · Backlog
Normalize consumer text semantics without rewriting bytes
Shelf
Backlog
Source
docs/backlog/archive/WI-082-consumer-crlf-normalization.md
Status
done

Proposal (rationale)

Rungs reads generated Markdown and TOML back from the consumer repository. On a normal Windows checkout with core.autocrlf=true, Git materializes those tracked files with CRLF even though the package sources use LF. Several engines parse the consumer bytes with LF-only anchored expressions, so present frontmatter and rule metadata read as absent. The upgrade record editor separately splits on LF and writes replacement lines with LF, creating a mixed-newline diff on a semantic no-op.

The WI-064 packed canary reproduced both failures from exact commits. An LF consumer passed 20/20; a fresh CRLF checkout failed two gates, and each same-version upgrade --apply reported no work but dirtied .ai/rungs.toml. Arena cannot become a stable direct consumer while checkout policy changes gate truth or a no-op command changes tracked bytes.

Decision

accepted — 2026-09-05, during the user-authorised Arena Lab bootstrap. Normalize line endings at semantic parsing boundaries rather than changing consumer-owned bytes or scattering optional-CR regexes. Preserve the record’s existing newline convention and do not write it when the computed content is byte-identical.

Plan

Requirements

  • All declared engines must interpret LF and CRLF consumer text identically, including frontmatter, headings, Markdown tables, registers, rules and render-source parsing.
  • Normalization is read-only and semantic: ownership hashes and byte-preservation comparisons must continue to use the actual on-disk bytes.
  • updateRecordAfterUpgrade must parse CRLF, preserve the original newline convention and terminal newline, and skip its write when no bytes change.
  • The packed consumer journey must commit adoption, create a fresh core.autocrlf=true checkout and exercise an origin-only feature branch plus detached HEAD.

Impacts

  • Shared text reads in src/engines*.ts, neutral-rule parsing in src/render.ts, and the surgical install-record update in src/lifecycle.ts.
  • Focused engine/record tests and the isolated packed existing-repository journey.
  • F-034 and the 0.4.0 changelog fragment; no generated consumer API or schema changes.

Approach

Introduce one small semantic-newline helper that maps CRLF and lone CR to LF after a file has been decoded as text. Use it at engine and render parsing boundaries only. Keep install/upgrade hashing, divergence checks and ownership reads byte-faithful. For the record editor, detect the first newline sequence, split semantically, rejoin with that sequence, preserve whether the source ended in a newline, and compare the final bytes before writing.

Extend the real packed candidate journey instead of adding only a synthetic regex test: after the tracked scaffold is committed, make a fresh worktree/checkout under core.autocrlf=true, prove the generated bytes are CRLF, run checks and two no-op applies in both origin-only feature and detached states, then assert raw bytes and Git status never change.

Acceptance criteria / tests

  1. Exact CRLF copies of representative emitted rules, skills, findings and gate tables produce the same engine results and examined counts as LF copies; LF behavior remains unchanged.
  2. A CRLF .ai/rungs.toml receives real version/hash replacements using CRLF throughout, preserves its terminal-newline state, and a semantic no-op performs no write and leaves identical bytes.
  3. The packed consumer is committed and freshly checked out with core.autocrlf=true; all installed gates pass on an origin-only feature branch and detached HEAD.
  4. Preview and two same-version applies report zero updates/divergence and leave the record bytes and complete Git status unchanged in both Git states.
  5. Focused tests, full npm test, package dry-run, git diff --check, all registered gates and the exact six-cell OS/Node plus site matrix pass.

Out of scope

  • Enforcing a consumer-wide .gitattributes policy, rewriting existing consumer files to LF, or normalizing binary/ownership hash inputs.
  • WI-077 ejection independence, WI-078 branch-local exemptions, WI-080 APFS path aliases, WI-081 scratch-worktree verification, release publication or Arena’s final pin commit.

Execution

Started from exact verified green/main 6c3b846bbbc21d6255db84323db54facc08ddef2.

  • Added a semantic-only newline normalizer and routed all three declared-engine families plus neutral rule parsing through it. Ownership hashes, divergence reads and byte comparisons remain raw.
  • Reworked the surgical install-record update to parse normalized lines, rejoin with the source’s first newline convention, retain terminal-newline state, count only real line replacements and avoid writeFileSync when no semantic line changed.
  • Extended the packed existing-repository journey with a true fresh checkout configured core.autocrlf=true. It proves CRLF bytes, an origin-only stable ref and detached HEAD, then runs full checks, preview and two applies in each state while comparing record bytes, every tracked byte and Git status.
  • Added LF/CRLF engine equivalence, CRLF record replacement, terminal-newline and mixed-newline no-write unit regressions. The small text helper is included in the package and ejected source inventory so this change does not introduce another missing local import.

Review

Final acceptance evidence:

  1. Independent review approved exact implementation commit c2ebdce5ca8aa15ab7ea3eb769e7baa4c653ef30 with no findings. Later integrations changed no production semantics; their only manual conflict combined filesystem imports in the test file.
  2. On final integrated tip a5185b4fc6693f552ac222b7433b07bf12562898, the focused semantic/record/packed journey passed 3/3 and full npm test passed 139 tests with 3 expected platform skips and 0 failures.
  3. Registered Rungs gates passed 30/30, git diff --check was clean, and npm pack --dry-run --json contained 114 entries including src/text.ts and src/storage-key.ts at 385,222 packed bytes.
  4. Exact-tip GitHub Actions run 33977653274 passed the site plus both Node versions on Linux, macOS and Windows: 7/7 jobs green.
  5. rungs land verified the merged implementation tree at 2e9bd0611cf0b1ce380353fc3ce50e19116fe561 with 30/30 gates. A direct check of that exact tree then failed only backlog-merged-status, because this item was intentionally still marked in_progress until the implementation landing succeeded.