Add tests for common TextDecoder implementation mistakes

[ChALkeR]

This testsuite collects cases where at least one implementation fails

This is an export of https://github.com/ExodusOSS/bytes/blob/9c8c9baa/tests/encoding/mistakes.test.js

It demonstrates bugs in all major implementations - not a single of the existing ones was correct for all encodings.
In total, 10 different implementations were tested + my implementation in js.

Important

All of the three major browser engines got UTF-8 decoding wrong.

---

#56799 and #56844 were tiny parts of this.

---

On no testcase all implementations agree on an incorrect behavior: at least one is consistent with the spec.
Perhaps the most controversial part of this is a test for whatwg/encoding#115, but Firefox, Deno, Servo agree with spec and the spec is clear on what should happen and documents that specific case. Also, WebKit and Chrome now treat concatenated input as invalid too, but perform replacement on it incorrectly.

On some tests,

---

For data, see https://docs.google.com/spreadsheets/d/1pdEefRG6r9fZy61WHGz0TKSt8cO4ISWqlpBN5KntIvQ/edit
Legend:

• FAIL means returning incorrect results on some inputs, i.e. not matching the spec on a simple .decode() call.
• STATE is more serious in some cases - it means internal inconsistency, non-streaming .decode() result depends on previous call, state leaks between calls.
• STREAM also could be more serious in some cases and demonstrates an internal inconsistency
  Decoding result depends on the buffers shape: decode(a, { stream }) + decode(b) !== decode(a + b)
  With Dynamic Record Sizing, this could potentially be controlled with a MitM and cause different decoding results for the same bytes, all without affecting TLS (as it verifies bytes).
  Demo for utf-8 mishandling on WebKit and Chrome: https://tmp-demo.rray.org/utf-8 (small static html over https)

Due to the nature of cross-tests, implementations not behaving correctly per spec on single-shot new TextDecoder(encoding).decode(arg) calls were not fully tested for STATE / STREAM, so FAIL takes priority (even in cases where it's less significant).

The rest is explained in the table and the testcases

---

Bug refs, found with these cross-tests:

• https://issues.chromium.org/issues/468458744 / https://chromium-review.googlesource.com/7260738
• https://issues.chromium.org/issues/468458388 / https://chromium-review.googlesource.com/7263765, https://chromium-review.googlesource.com/7266545
• https://issues.chromium.org/issues/467727340
• https://issues.chromium.org/issues/467624168
• https://bugs.webkit.org/show_bug.cgi?id=304181 / Incorrect handling of invalid UTF-8 in streaming decoder WebKit/WebKit#55452
• https://bugs.webkit.org/show_bug.cgi?id=304238 / Big5 decoder doesn't recover to emit ASCII after an invalid leading byte WebKit/WebKit#55538
• https://bugzilla.mozilla.org/show_bug.cgi?id=2005419
• TextDecoder behavior is wrong since 20.18.3/22.13.0 nodejs/node#60888
• TextDecoder is wrong and very slow nodejs/node#61041
• TextDecoder does not error incorrectly for legacy byte sequences nodejs/node#40091 (comment)
• BOM handling is incorrect in utf-8/utf-16 TextDecoder oven-sh/bun#25495
• Document discrepancies with TextDecoder jsdom/whatwg-encoding#22
• feff should not be stripped for legacy multi-byte encodings and utf-8 jsdom/whatwg-encoding#23
• TextDecoder with stream:true is invalid on big5, shift_jis, euc-kr denoland/deno#31605
• TextDecoder with stream:true is invalid on big5, shift_jis, euc-kr servo/servo#41259
• Deno and Servo fail at big5/shift_jis/euc-kr streaming hsivonen/encoding_rs#126
• Encoding decoder tests don't test codepoints #56748
• Encodings used in iconv-lite mismatch the WHATWG Encoding spec significantly pillarjs/iconv-lite#367
• Almost all non-utf8 encodings are incorrect expressjs/body-parser#680
• ... more incoming, I filed just a portion of the issues, feel free to file more from this testcase

---

For a clean implementation in js, see @exodus/bytes/encoding.js

[annevk]

This is amazing work. Thanks so much!

If have two questions:

• I guess the lack of semicolons was maybe copied from existing tests? It would be preferable to have them, but it's not required.
• Can we perhaps split this file by encoding type? E.g., UTF-8, UTF-16, single-byte encodings, and multi-byte encodings?

[annevk]

This seems very sketchy. I wonder if we should change this somehow in the standard. No need to change the test for now though as it indeed appears correct.

[ChALkeR]

My take is that continuing streaming after fatal has thrown an error is a very bad idea, regardless of the encoding, and should be banned.

I will elaborate more in whatwg/encoding#358

[anonrig]

I recommend merging this and we can iterate over it. None of the reviews seems to be a blocker.