ToAscii: Some russian texts in Cyrillic (Azbuka) are wrong rewrited

[janbarasek]

Hi,

I can speak Russian language and I noticed that the transcription of Cyrillic (Azbuka) to Ascii is sometimes inaccurate.

For example string:

dump(Strings::toAscii('для'));

should be dlja, but not current dla, because char я means ja (Czech) or ya (English).

Sample:

If my suggestion makes sense I can implement better translation with support for whole syllables and special cases.

Thanks.

[dg]

Do you have enabled Transliterator?

utils/src/Utils/Strings.php

Line 141 in c133e18

if ($transliterator === null && class_exists('Transliterator', false)) {

And what implementation of iconv are you using?

utils/src/Utils/Strings.php

Line 154 in c133e18

if (ICONV_IMPL === 'glibc') {

[janbarasek]

@dg Yes, Transliterator was instanced and ICONV_IMPL is libiconv.

I detected it by:

/**
 * Converts UTF-8 string to ASCII.
 */
public static function toAscii(string $s): string
{
	static $transliterator = null;
	if ($transliterator === null && class_exists('Transliterator', false)) {
		$transliterator = \Transliterator::create('Any-Latin; Latin-ASCII');
	}
	dump(['transliterator' => $transliterator, 'iconv' => ICONV_IMPL]);

Dumped:

Thanks.

[dg]

Transliterator converts я to â (according to ISO 9 https://cs.wikipedia.org/wiki/ISO_9) and then â to a.

Probably solution is to add strtr for chars Я я Ю ю and maybe some others.

[janbarasek]

I think we can use (for Czech language):

[
	'ё' => 'jo',
	'ъ' => '',
	'ы' => 'y',
 	'ь' => '', 
	'э' => 'eh', 
	'ю' => 'ju',
	'я' => 'ja',
]

But for correct behavior is very important use target language (which can be loaded from second parameter string $language = 'en').

And table for English:

[
	'ё' => 'jo',
	'ъ' => '',
	'ы' => 'y',
 	'ь' => '', 
	'э' => 'e', 
	'ю' => 'yu',
	'я' => 'ya',
]