Use vector operations in text decoding

[Fuuzetsu]

There's a fair amount of literature about decoding UTF8 as well as conversion to UTF16. It seems like cbits.c could probably benefit massively from a modern update.

See https://github.com/cyb70289/utf8 for example, also u8u16 library/paper &c.

[Fuuzetsu]

Doing some more research https://bitbucket.org/stgatilov/utf8lut seems quite good, it can validate and convert between utf8 <-> utf16/32 which would allow replacing a bunch of decoders/encoders in Text

[ethercrow]

@Fuuzetsu ohai

Just saw this issue. Lately I've been applying SSE2 to obvious places in encoding and decoding:

• Use SSE2 in the x86_64 C version of encodeUtf8 #300
• Use SSE2 in the ASCII fast path of decodeUtf8 #298
• Use SSE2 in the x86_64 C version of decodeLatin1 #297

I haven't yet read the state of the art in SIMD conversions, thanks for the links.

If we're to go into SSE4 and AVX, I wonder if GHC should support a -march=native flag so that users don't have to manually pick individual instruction set flags like -msse4.2 specifically for their machines.

[Fuuzetsu]

@Lysxia can you comment on why the issue for closed? are we happy enough with current state of vectorisation in modern text? If so, great, just nice to have couple of words

[Lysxia]

Sure thing. Given @ethercrow's SSE2 patches in text-1.2.5.0 and @Bodigrim's switch to UTF-8 (PR #365) which bundled the C++ simdutf8 library with text-2.0, it's fair to say that the situation regarding vectorization in text has significantly evolved in the last two years. I'm closing this issue based on that assessment.

I realize I tend to be trigger-happy in closing issues. I won't be opposed if you or @haskell/text maintainers would prefer this issue to remain open to keep track of long-term progress on vectorization in text. But my view is that it's a given that comparisons against the state of the art and further improvements in the area are always welcome. Hence I think issues have more utility when they are driven by more focused discussion and more actionable goals.