fix(schema)!: span.bytes are UTF-8 byte offsets, every node and level (#179) - #189
Merged
Conversation
…#179) ts-morph positions are UTF-16 code units; the wire said bytes and the graph sliced a Buffer by them, so :TSCallable.code ran short by the multibyte surplus inside the span. One conversion utility (schema/offsets.ts): producers convert on the way out (builders, dataflow/attach, artifacts/yamlKeys); consumers needing compiler positions convert on the way in (configUse, defuse-linker factory lookup, entrypoint default-export resolution). Values change only for nodes after a non-ASCII char; ASCII files are byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #179. Spec:
docs/design/specs/span-bytes-are-bytes.md(D1–D3, mirrored in.claude/SCHEMA_DECISIONS.md). Analyzer 1.5.0; Neo4j contract stays 2.0.0.What was wrong
span.byteswas produced (and documented inschema.ts) as UTF-16 char offsets — ts-morph'sgetStart()/getEnd()— while the canonical keystone and codeanalyzer-python mean UTF-8 bytes (byte_offsets,_span_code=source.encode()[lo:hi]). The projection'sspanCodealready sliced aBufferby them, so:TSCallable.coderan short by exactly the multibyte surplus inside the span: python-sdk's samplesrc/index.mainholds one em dash (3 bytes) → 2 bytes short → the closing\n}gone.What changes
span.bytes= UTF-8 byte offsets on every node at every level: declarations,call/config_access(L1), statements and@entry/@exit(L3), the module span ([0, byteLength]), yamlConfigKeyspans.Buffer.from(source).subarray(lo, hi)reproduces the text;source.sliceno longer does on a non-ASCII file. Values change only for nodes after a non-ASCII character; ASCII files are byte-identical before and after (-j1/-j4determinism re-checked).src/schema/offsets.ts: ASCII fast path is the identity; otherwise one cumulative table per text, cached per owning object (SourceFile / TSModule). Producers convert on the way out; the three consumers that need compiler positions (configUse.nodeAtSpan, the defuse linker's factory lookup,entrypoints/matchingdefault-export resolution) convert on the way in. The dataflow IR keeps char offsets internally.schema.tscomments, the consumer skill's vocabulary,CLAUDE.md.Tests
test/byte-offsets.test.ts: a fixture with an em dash, CJK and an emoji before and inside declarations — every callable, field, call site,config_access, L3 statement,@entry, the module span, the yaml key span, and the graph'scodeslice byte-exact; the config-use literal tier still resolves past multibyte text;offsetMapOfround-trips incl. surrogate pairs.schema-v2.test.ts's slice assertions now use the byte rule.bun test: 339 pass.Downstream
python-sdk #343 (leg 2.5b): slice
span.bytesas bytes on the TypeScript leg too — its python side already does — and the fourxfail(strict=True)marks ontest_typescript_bulk_parity_live.pyflip when the pin moves to the release carrying this.