Tracking issue: UTF-8 decoder in libcore

[strake]

Update (@SimonSapin): this is now the tracking issue for these items in both core::char and std::char:

• decode_utf8() which takes an iterable of u8 and return DecodeUtf8
• DecodeUtf8 which implements Iterator<Item=Result<char, InvalidSequence>>
• InvalidSequence which is opaque

Original issue:

In libcore we have a facility to encode a character to UTF-8, i.e. char::EncodeUtf8, but no facility to decode a character from potentially-invalid UTF-8, and return 0xFFFD if it reads an invalid sequence, which seems a surprising omission to me as a libcore user, given in libstd we have string::String::from_utf8_lossy.

These options came to mind:

• A function str::next_code_point_lossy or so which behaves as str::next_code_point but checks whether its input is valid and returns 0xFFFD if not
• An iterator DecodeUtf8 which one can make from an arbitrary iterator of bytes, which decodes them

[SimonSapin]

I’ve submitted #35947 to make this emit errors as specified in Unicode, like String::from_utf8_lossy does.

[jethrogb]

I think this is the tracking issue for this functionality?

DecodeUtf8, like DecodeUtf16, should have a way to recover invalid byte sequences encountered.

[bluss]

std::str::next_code_point being public is bad; it assumes valid UTF-8 input and libcore needs the agility to use unsafe code in this function if it turns out to be beneficial.

[SimonSapin]

The initial message of this issue is somewhat misleading now that this is the tracking issue for the std::char::DecodUtf8 iterator adaptor.

It is not anymore about core::str::next_code_point which is marked #[unstable(feature = "str_internals", issue = "0")] and is only public so that std can use it.

[SimonSapin]

I’m inclined to not stabilize this.

Now that str::Utf8Error has an error_len method it is possible to solve the original issue (equivalent of String::from_utf8_lossy in libcore) by calling str::from_utf8 repeatedly. I’ve submitted #49105 to add an example showing how.

from_utf8 can optimize its inner loop for example to look at multiple bytes at a time. This API however is forced to only work on one char at a time.

If you want to build a &str-like (like ArrayString), with DecodeUtf8 you end up converting to chars (effectively UTF-32) only to then convert back to str (UTF-8). str::from_utf8 in comparison only does UTF-8 validation. (The chars method can then actually decode if that’s desired.)

It’s tempting to add new APIs to libcore for something like the example above, but there’s a lot of possible variation: returning an Iterator instead of taking a callback, using Result instead of hard-coding U+FFFD, supporting chunked input (e.g. where a single code point can be made of bytes that end up in separate network packets), …

https://docs.rs/utf-8/ tries to support all of these use cases (still on top of str::from_utf8), but ends up with a non-trivial API surface. Something like this is probably better off in a crate outside of the standard library.

@strake what do you think?