Migrate ziglyph to zg

[mitchellh]

Ziglyph is no longer maintained and the replacement is zg. We should migrate to zg.

I'd be willing to accept a PR to do this at any point, but this is going to become a hard requirement for our 0.14 transition when Zig 0.14 is released.

I don't know if zg has a 0.13-compatible build but it may be easiest to find that and upgrade sooner.

[pluiedev]

d667d18 seems to be the latest commit that claims to support Zig 0.13 and in terms of commit history isn't that far behind. We could try using that. One thing to note is that zg incorporates various performance improvements from Ghostty itself (mainly from https://mitchellh.com/writing/ghostty-devlog-006) and so we could probably remove a fair amount of our own code in favor of zg's implementation (which should be pretty much identical anyway).

[mitchellh]

(mainly from https://mitchellh.com/writing/ghostty-devlog-006) and so we could probably remove a fair amount of our own code in favor of zg's implementation (which should be pretty much identical anyway).

We could, but I'd rather not. Since we don't need every property, we're able to optimize our lookup tables to only include the data we need and therefore make them smaller and more cache friendly. It is trivial to build the LUTs (our unigen binary) and I think there's a benefit to having that machinery for our fast paths.

Additionally, there are optimizations we make (i.e. eliminating impossible unicode combinations) that we can do because the VT parser ensures this to be true, but generalized Unicode does not, so zg is likely slower for that reason.

These lookup tables are only used in one place though (our super hot IO path). To keep things simpler, we kept using ziglyph (and now zg) in the codepaths that aren't performance sensitive. And we should keep doing that.

[pluiedev]

So I was looking at zg's API design and one thing I immediately noticed was that it's not nearly as accessible as ziglyph's is. We need to pass around a central "context" struct that holds all of the information zg uses since now it has to decompress tables at runtime...

[jacobsandlund]

I've been researching this task, and will continue to do so Sunday, but I wanted to report:

• As @pluiedev mentioned, zg is written in such a way that it loads and decompresses lookup tables that they codegen, that are compressed in the binary. Two tables ghostty would use are gbp (grapheme break property) from Graphemes.zig and gencat from GeneralCategories.zig. I'll measure the in-memory footprint Sunday, but the Graphemes "stage3" data is a u8 and GeneralCategories "stage3" is a u5. (I'll also confirm if we need any of the other tables.)
• Unlike ziglyph where the emoji-data.txt is exactly (i think?) represented in autogen/emoj_data.zig, zg only supports an isEmoji method coming from a single bit in the Graphemes stage3 u8, and that is defined by checking for "Extended_Pictographic" alone.
  • This means their graphemeBreak code doesn't handle emoji modifiers like ghostty here.
  • Also, our GraphemeBoundaryClass can't separate out these different emoji classifications with zg as currently written.
• I'm looking at other places in ghostty that use ziglyph:
  • src/font/CodepointResolver.zig calls isEmojiPresentation in getIndex, which would need the "Emoji_Presentation" from emoji-data.txt that is represented in ziglyph but missing in zg.
  • src/font/shaper/web_canvas.zig calls graphemeBreak here, but that could be rewritten to use our unicode_tables.
  • src/renderer/cell.zig will need GeneralCategories, and also a brief search for isDingbats in zg makes me think it's missing as well.

Given all this, I'm not positive switching out ziglyph for zg is the right approach. Perhaps we pull in the unicode data text files ourselves and use that to generate our unicode_tables. For the isEmojiPresentation and isDingbats (and potentially others) we could also generate somehow (perhaps a separate generated table, given the flexibility of src/unicode/lut.zig). Alternatively, we could make several pull requests to get zg in a place where we can use it. Thoughts?

[pluiedev]

I came to a very similar conclusion when I investigated this myself. Add in the fact that Kovid is writing has written a comprehensive spec for grapheme processing in the context of terminals (see #6931, kovidgoyal/kitty#8533), I think we might need to move unicode processing fully in-house. zg's API design just doesn't make sense for our use case, plus hauling around decompressed data just gets tedious with how things are right now.

[mitchellh]

I'm probably going to be bringing in zg for my new keycode branch (w3c-key) because I need to do Unicode case folding. I think the right solution is a mix of our own unicode tables and zg the same we do do that with ziglyph today. Our own Unicode tables are optimized for space usage and logic for highest speed and cache friendliness in the context of a terminal. But there are other parts of the codebase that don't require terminal-specific knowledge (e.g. case folding in my branch) and I don't think it makes sense to maintain our own data there.

zg requiring state is annoying but as far as I can tell its fully thread safe because its all write-once up front and then read-only. So we can set this up in global.zig probably.

[jacobsandlund]

Finally got around to working on this, and I've got a PR up in #7806.

There's still a little bit of a question about adopting zg, as I needed to make 5 PRs to that repo to get it into a state we could switch to it, but I think it's probably okay. Hopefully they'll be reviewed and merged soon.

[mitchellh]

Removing this from the 1.2 milestone. There’s no reality where this gets solved and tested in time. zg has proven to be unsuitable for Ghostty’s use case (see PR linked above) without significant upstream changes. There are community members trying. We’ll have to reassess our path forward with newer Unicode versions in the 1.3 cycle.