Half-width Katakana should be representable in ISO-2022-JP

[hsivonen]

A query string-based (and, therefore, IE/Edge-incompatible) test shows that Gecko, WebKit, Blink and Presto can encode half-width Katakana as ISO-2022-JP without NCRs.

The spec should be amended to match (both encoder and decoder side).

[annevk]

It seems this is a special feature of the encoder only: "ﾐミ" both encode to 0x25 0x5F. I wonder if all Japanese encoders first convert halfwidth to fullwidth now.

[vyv03354]

I wonder if all Japanese encoders first convert halfwidth to fullwidth now.

No, ISO-2022-JP only.

[annevk]

You'd think someone would have already written and published an algorithm for this conversion. I guess I'll just find the mapping for each code point myself.

[annevk]

Okay, so I guess what we want to do is to apply Unicode Normalization Form KC on any code point in the range U+FF65 to U+FF9F, inclusive.

[annevk]

const start = 0xFF61,
      end = 0xFF9F + 1;
for(let i = start; i < end; i++) {
  const cp = String.fromCodePoint(i),
        fullwidthCP = cp.normalize("NFKC");
  // ...
}

If I write those out and use @hsivonen's demo I get the results I was expecting per the above analysis.

[aphillips]

Correct me if I'm wrong, but wouldn't halfwidth katakana involve a switch to JIS X 0201 (Roman) mode? There's no need to destroy the round-trip by normalizing to fullwidth.

[annevk]

@aphillips that is not what implementations do.

[aphillips]

@annevk Yes, although that seems like a bug in the coders. I saw this thread this morning and was surprised, since I recall having to implement this when I was writing an ISO-2022 coder about 15 years ago. I can't imagine that the encoding's formal definition has changed, so I'm surprised to see implementations doing this.

[annevk]

Sure, but after such a long time bugs become features.

[jungshik]

@aphillips @annevk
I wouldn't call it a bug.

On the Internet/Web, only the original ISO-2022-JP defined in RFC 1468 was "widely" (relative to subsequent versions) used, but subsequent versions of ISO-2022-JP, ISO-2022-JP-[123] never got much traction. Why would anybody use ISO-2022-JP-* to encode Chinese, Korean, Latin beyond ASCII, and Greek? And, JIS X 0212 (supported in ISO-2022-JP-1 or later) is not critical enough to Japanese users (Shift_JIS does not support it, either).

The original ISO-2022-JP does not support Halfwidth Katakana. That's why ICU has a fallback encoding for Halfwidth Katakana for the original ISO-2022-JP.

It's only ISO-2022-JP-3 that supports Halfwidth Katakana. ICU supports ISO-2022-JP-3 as defined and does not have fallback encoding for Halfwidth Katakana in ISO-2022-JP-3.
Note that ISO-2022-JP-2 defined in RFC 1554 does not support them either.

[annevk]

Note that we do support decoding halfwidth Katakana: https://encoding.spec.whatwg.org/#iso-2022-jp-decoder-katakana. Should we remove that then?