Chore: refactoring occurrence indexing

[camdencheek]

This is a bit of refactoring as a followup to #63217. The goal is to unify how we use Occurrences so that we can use the same interface for both syntax highlighting data and occurrence data from the new API. An additional goal is to encode more of the invariants in the type system so it's more difficult to use incorrectly.

The primary entrypoint to reviewing this PR is the new OccurrenceIndex class, which extends an array of occurrences to add a line index, enforce the non-overlapping and sorted invariants, and add helper methods for common operations using the index.

Stacked on #63217 (and thus blocked by that)

Test plan

Existing unit tests (though there are not many 😬), and manually compared the behavior between S2 and local for a variety of files with different indexing states, file types, and highlighting availability. There was no visible change in behavior.

[camdencheek]

Moved to OccurrenceIndex.next

[camdencheek]

Moved onto the IndexedCodeGraphData type which extends the type from GraphQL so CodeGraphData better matches the API shape

[camdencheek]

Added OccurrenceIndex to capture the invariants required by the line index and the various functions that operate on the index. Conveniently, this type can be generic over both syntax highlighting occurrences and occurrences from the code intel API.

[camdencheek]

Mostly moved to OccurrenceIndex, inlined the remainder

[camdencheek]

Moved to OccurrenceIndex

[camdencheek]

Moved to where we filter to interactive syntax occurrences

[camdencheek]

Replaced by OccurrenceIndex.atPosition()

[fkling]

I like this a lot!

[fkling]

Seems to be a clever way to write

let low = 0
let high = this.length

Don't know if engines can optimize this to not allocate an array. But maybe it's not a big deal either.

FWIW, if you want to have a single let statement you could also just write

let low = 0, high = this.length

[camdencheek]

Ha, yes. I shouldn't be clever 🙂

[fkling]

We could go even further to unify this: Have a separate facet for interactive occurrences, have codeGraphData and syntaxHighlight both contribute to that facet, and use CodeMirror's built-in extension priority system to determine the order of the data (which also works for facet (inputs) because they are extensions too). We can decide whether the facet should "return" all provided occurrences or just return the first (high pri) one.

Then this could look like this:

const interactiveOccurrences = Facet.define<OccurrenceIndex, OccurrenceIndex>({
  combine(values) {
    return values[0] ?? new OccurrenceIndex([])
  },
})

Then we can just use it as

const occurrences = state.facet(interactiveOccurrences)

And the input would be provided with something like this:

// Not sure we still need to have the facet map `CodeGraphData` to `IndexCodeGraphData` if we do that
export const codeGraphData = Facet.define<CodeGraphData[], IndexedCodeGraphData[]>({
    static: true,
    combine: values =>
        values[0]?.map(data => ({ ...data, occurrenceIndex: new OccurrenceIndex(data.occurrences) })) ?? [],
    })
    // Maybe we want to only use precise data here, idk
    enable: facet => interactiveOccurrences.computeN([facet], state => state.facet(facet).map(datum => datum.occurrences)),
})
// Similar for syntaxHighlight

Then we just need to ensure that the codeGraphData facet has a higher priority than the syntaxHighlight one.

---

Side note: This is how the whole hovercard system works too. There is a single 'show hovercard' facet and multiple extensions that provide input for it: Actual hover (with mouse), keyboard activated hovercard and pinned hovecards. Those sources are ordered by priority and the facet simply picks the first one.

[camdencheek]

Brilliant! This was the direction I was attempting to move us in, but didn't quite get that far. Will leave this as a follow up, but totally agreed on the vision.