Casing transforms in snippets only support Latin characters

[dmitrivMS]

Testing #233798

Use the following snippet:

"testSnippet": {
  "scope": "javascript,typescript",
  "prefix": "testSnippet",
  "body": "${1} -> ${1/(.*)/${1:/upcase} ${1:/downcase} ${1:/camelcase} ${1:/pascalcase} ${1:/kebabcase} ${1:/snakecase}/}"
}

Give it input oneTwo you will get oneTwo -> ONETWO onetwo oneTwo OneTwo one-two one_two which is correct.
Give it input одинДва you will get одинДва -> ОДИНДВА одиндва одинДва одинДва одинДва одиндва which is incorrect as non-Latin letters are not handled correctly (in this case Russian alphabet, but code only cares about a-z).

We should probably use Unicode definition of words if possible to perform the case transforms.

[lucas-gomes-santana]

Hello, I saw your issue and have interest to help you with it. I found the part of the project where the snippets on VsCode are generated and processed. Go to this folder path on the repository to found these files: src/vs/editor/contrib/snippet/browser/ .

I think the reason for the error are in these regex here:

// snippetParser.ts, 403 line to 481
private _toKebabCase(value: string): string {
		const match = value.match(/[a-z0-9]+/gi);
		if (!match) {
			return value;
		}

		if (!value.match(/[a-z0-9]/)) {
			return value
				.trim()
				.toLowerCase()
				.replace(/^_+|_+$/g, '')
				.replace(/[\s_]+/g, '-');
		}

		const match2 = value
			.trim()
			.match(/[A-Z]{2,}(?=[A-Z][a-z]+[0-9]*|\b)|[A-Z]?[a-z]+[0-9]*|[A-Z]|[0-9]+/g);

		if (!match2) {
			return value;
		}

		return match2
			.map(x => x.toLowerCase())
			.join('-');
	}

	private _toPascalCase(value: string): string {
		const match = value.match(/[a-z0-9]+/gi);
		if (!match) {
			return value;
		}
		return match.map(word => {
			return word.charAt(0).toUpperCase() + word.substr(1);
		})
			.join('');
	}

	private _toCamelCase(value: string): string {
		const match = value.match(/[a-z0-9]+/gi);
		if (!match) {
			return value;
		}
		return match.map((word, index) => {
			if (index === 0) {
				return word.charAt(0).toLowerCase() + word.substr(1);
			}
			return word.charAt(0).toUpperCase() + word.substr(1);
		})
			.join('');
	}

	private _toSnakeCase(value: string): string {
		return value.replace(/([a-z])([A-Z])/g, '$1_$2')
			.replace(/[\s\-]+/g, '_')
			.toLowerCase();
	}

	toTextmateString(): string {
		let value = '${';
		value += this.index;
		if (this.shorthandName) {
			value += `:/${this.shorthandName}`;

		} else if (this.ifValue && this.elseValue) {
			value += `:?${this.ifValue}:${this.elseValue}`;
		} else if (this.ifValue) {
			value += `:+${this.ifValue}`;
		} else if (this.elseValue) {
			value += `:-${this.elseValue}`;
		}
		value += '}';
		return value;
	}

	clone(): FormatString {
		const ret = new FormatString(this.index, this.shorthandName, this.ifValue, this.elseValue);
		return ret;
	}

Can you explain more about the problem you reported?

[dmitrivMS]

@lucas-gomes-santana Yes, the regex-es only handle Latin alphabet, we should consider using something like \w instead, but it probably needs some testing.

[lucas-gomes-santana]

@dmitrivMS I think now the bug you related is solved:

одинДва -> ОДИНДВА одиндва одинДва ОдинДва один-два один_два
一个测试 -> 一个测试 一个测试 一个测试 一个测试 一个测试 一个测试
έναςΔύο -> ΈΝΑΣΔΎΟ έναςδύο έναςΔύο ΈναςΔύο ένας-δύο ένας_δύο
ひらがなカタカナ -> ひらがなカタカナ ひらがなカタカナ ひらがなカタカナ ひらがなカタカナ ひらがなカタカナ ひらがなカタカナ
واحدإثنين -> واحدإثنين واحدإثنين واحدإثنين واحدإثنين واحدإثنين واحدإثنين
하나둘 -> 하나둘 하나둘 하나둘 하나둘 하나둘 하나둘
одинДва3 -> ОДИНДВА3 одиндва3 одинДва3 ОдинДва3 один-два3 один_два3
ένας_δύο -> ΈΝΑΣ_ΔΎΟ ένας_δύο έναςΔύο ΈναςΔύο ένας-δύο ένας_δύο
こんにちはWorld -> こんにちはWORLD こんにちはworld こんにちはWorld こんにちはWorld world こんにちはworld

I decided to change the regexs on snippetParser.ts and make some tests, using other languages beyond russian and the final inputs are on the top. Some languages such as Chinese, Japanese, Arabic and Hebrew, these changes to the regex don't work in those languages because they don't have uppercase and downcase letters.

// Note: word-based case transforms rely on uppercase/lowercase distinctions.
// For scripts without case, transforms are effectively no-ops.
private _toKebabCase(value: string): string {
	const match = value.match(/[\p{L}0-9]+/gu);
	if (!match) {
		return value;
	}

	if (!value.match(/[\p{L}0-9]/u)) {
		return value
			.trim()
			.toLocaleLowerCase()
			.replace(/^_+|_+$/g, '')
			.replace(/[\s_]+/g, '-');
	}

	const match2 = value
		.trim()
		.match(/\p{Lu}{2,}(?=\p{Lu}\p{Ll}+[0-9]*|\b)|\p{Lu}?\p{Ll}+[0-9]*|\p{Lu}|[0-9]+/gu);

	if (!match2) {
		return value;
	}

	return match2
		.map(x => x.toLocaleLowerCase())
		.join('-');
}

private _toPascalCase(value: string): string {
	const match = value.match(/[\p{L}0-9]+/gu);
	if (!match) {
		return value;
	}
	return match.map(word => {
		return word.charAt(0).toLocaleUpperCase() + word.substr(1);
	})
		.join('');
}

private _toCamelCase(value: string): string {
	const match = value.match(/[\p{L}0-9]+/gu);
	if (!match) {
		return value;
	}
	return match.map((word, index) => {
		if (index === 0) {
			return word.charAt(0).toLocaleLowerCase() + word.substr(1);
		}
		return word.charAt(0).toLocaleUpperCase() + word.substr(1);
	})
		.join('');
}

private _toSnakeCase(value: string): string {
	return value.replace(/(\p{Ll})(\p{Lu})/gu, '$1_$2')
		.replace(/[\s\-]+/g, '_')
		.toLocaleLowerCase();
}

I make a PR here: #287150 with the changes here, waiting for community and project mainteners approvals.

[dmitrivMS]

The languages to test would be those that are derived from Latin or Cyrillic alphabets (European), also Turkish which it's i problem.

[lucas-gomes-santana]

@dmitrivMS Here some tests with latin languages:

Tests with latin languages

oneTwo -> ONETWO onetwo oneTwo OneTwo one-two one_two (English)
umDois -> UMDOIS umdois umDois UmDois um-dois um_dois (Portuguese)

(Portuguese with accents)
caféRestaurante -> CAFÉRESTAURANTE caférestaurante caféRestaurante CaféRestaurante café-restaurante café_restaurante 
açãoReação -> AÇÃOREAÇÃO açãoreação açãoReação AçãoReação ação-reação ação_reação

êtreÀ -> ÊTREÀ êtreà êtreÀ ÊtreÀ être-à être_à (French)
mañanaNiño -> MAÑANANIÑO mañananiño mañanaNiño MañanaNiño mañana-niño mañana_niño (Spanish)
übungÄpfel -> ÜBUNGÄPFEL übungäpfel übungÄpfel ÜbungÄpfel übung-äpfel übung_äpfel (German)
cittàPerché -> CITTÀPERCHÉ cittàperché cittàPerché CittàPerché città-perché città_perché (Italian with accents)