Bring in localization into the API

[TylerLeonhardt]

vscode-nls and it's build time companion vscode-nls-dev has existed for quite a while now... and in its original design it was very filepath driven. We felt this pain when bundling extensions became the norm and introduced a webpack loader and bundler that baked filepaths into source code...

That mitigation of rewriting JS code worked... but this all became more complicated when talking about the web because the web has no "nice" way of loading strings synchronously. And even if there was such a nice thing, we wouldn't want synchronous loading from the internet... so instead we are moving in a direction where Core is responsible for loading the bundle of strings for an extension, asynchronously, before an extension is loaded and then those strings are provided to the extension via an API.

declare module 'vscode' {
	export namespace env {
		export function nls(message: string, ...args: string[]): string;
		export function nls(options: { message: string; comment: string[] }, ...args: string[]): string;
		export interface nls {
			contents: { [key: string ]: string };
			uri: Uri;
		}
	}
}

In addition to this, we will have an additional field in the package.json: nlsBundleLocation which is at the root next to main and browser. This is useful for:

• being explicit in an extension that supports localized strings
• allows VS Code to not have to "guess" where the localized strings are in the extension (this is especially important in web where we don't have a filesystem that we can crawl)

Important definitions:

• message: the message parameter is the English string. This is used for a few things:
  • It's the string used when VS Code is in the default (English) language
  • It's the fallback string when a bundle doesn't contain a translation for a particular string
  • It's the key in the bundle (which is essentially just a big dictionary) who's value will be the translated version of itself in whatever language is chosen
• options: This follows the same format we have when you wanna add a comment in a package.nls.json. The comment array will be an array of comments that will be included in the exported XLF files (the files sent to translators). The message is the same as defined above.
• i18n.contents: This will contain the key-value pairs of the strings that VS Code loaded. This is useful if you want to pass a subprocess, a web worker, or a webview the contents of the bundle so that that environment can have all the translations that are currently being used
• i18n.uri: This will contain the Uri to the bundle that was loaded. This is useful if you want to pass a subprocess, a web worker, or a webview the Uri of the bundle so that that environment can load the translations that are currently being used.

Along side of this work, we will ship a very small helper package called @vscode/nls or something like that that can be given the contents or uri value and load the bundle and handle string interpolation. This library will really only be used for subprocesses/workers/webviews that can't depend on the vscode API directly.

@vscode/nls

Current API proposal:

export function config(config: { bundleUri: string; } | { bundleContents: string | i18nJsonFormat; }): void;

export function nls(message: string, ...args: string[]): string;
export function nls(options: { message: string; comment: string[] }, ...args: string[]): string;

@vscode/nls-dev

This is a CLI & library for managing the usages of vscode.env.nls. It has two main usecases:

• Export: export the English strings out of the code and put them in a familiar JSON or XLF format
• Import: imports XLF files and turns them into the respective package.nls.*.json and bundle.nls.*.json files

It currently has three commands:

• Export

  cli.js export [args] <pattern..>    Export strings from source files

Positionals:
  pattern  Glob pattern of ts files to include  [array] [required] [default: []]

Options:
      --version  Show version number                                   [boolean]
      --help     Show help                                             [boolean]
  -o, --outDir   Output directory                        [string] [default: "."]

• Generate XLF

  cli.js generate-xlf [args]          Generate an XLF file from a JSON i18n file

Options:
      --version             Show version number                        [boolean]
      --help                Show help                                  [boolean]
  -p, --packageNlsJsonPath  JSON i18n file to generate XLF from
                                                             [string] [required]
  -b, --i18nBundleJsonPath  JSON i18n file to generate XLF from
                                                             [string] [required]
  -o, --outFile             Output file                      [string] [required]

• Import XLF

  cli.js import-xlf [args] <xlfPath>  Import an XLF file into a JSON i18n file

Positionals:
  xlfPath  Glob pattern of ts files to include               [string] [required]

Options:
      --version  Show version number                                   [boolean]
      --help     Show help                                             [boolean]
  -o, --outDir   Output directory that will contain the i18n.<locale>.json file
                 and the package.nls.json file           [string] [default: "."]

[wandyezj]

Is it possible to see which language is currently being used via the API?

Does anything need to be changed in an extension based on Left to Right languages (e.g. English) verses Right to Left languages (e.g. Arabic)? The main case I tend to see is layout, for example if there is a custom rendered view.

[TylerLeonhardt]

@wandyezj

Is it possible to see which language is currently being used via the API?

vscode.env.language

Does anything need to be changed in an extension based on Left to Right languages (e.g. English) verses Right to Left languages (e.g. Arabic)? The main case I tend to see is layout, for example if there is a custom rendered view.

VS Code doesn't support RTL: #11770 and I suspect that idea of LTR vs RTL would come in as an API separate to this proposal.

[wandyezj]

I see:

vscode.env.language

Represents the preferred user-language, like de-CH, fr, or en-US.

• Will the vscode.end.language sync up with language translation the vscode.env.i18n API returns?
• How does this work if the language is changed? Does the entire editor and the extension reload?
• Is there a way to account for formatted strings like loads ${value} or Hello ${name}, would you like ${item}? (Including the interpolated piece matters for translation, simply concatenating doesn't translate well).

With vscode.env.language it should be simple enough for a developer to create their own function to map the provided language code to a direction if the feature is needed for the extension.

[TylerLeonhardt]

Will the vscode.end.language sync up with language translation the vscode.env.i18n API returns?

Yes. vscode.env.i18n() will return the string in whatever vscode.env.language is set to so long as there was a translation available for that string in that language. And fallback to English if not.

Does the entire editor and the extension reload?

Yes. This is what happens today and will continue to happen.

Is there a way to account for formatted strings like loads ${value} or Hello ${name}, would you like ${item}?

Not at this time. We are aligning to what vscode-nls already does but in theory something like that could be added in the future.

It should be simple enough for a developer to create their own function to map the provided language code to a direction if the feature is needed for the extension.

Yes but I think there's enough value here:

• String interpolation - the {0}
• Automatic loading of your strings before your extension activates (so that any top level vscode.env.i18n calls still work) - this is especially important in the web where we must load strings via an async fetch call
• Tooling around exporting strings out of your source code into a familiar format (keeping the same XLF format that vscode-nls uses) but additional formats could be added

[TylerLeonhardt]

Updates from UX sync:

• nls isn't better than i18n... maybe l10n?
• use a t function on i18n instead of this half function/half object thing
• either take in (key, ...args) or ({ key: ..., args: ..., comments: ...})

[TylerLeonhardt]

Ok I have been playing with this a lot today… and here’s what I’d like to run:

• vscode.l10n - The API for translating strings in your extension’s code
• @vscode/l10n-dev - The tooling used for extracting l10n strings from vscode extensions and working with XLF files
• @vscode/l10n - The library used for loading the translations into subprocesses of your extension
• package.l10n.json - The file used for translating static contributions in your extension’s package.json

vscode.l10n

This contains the following definition:

declare module 'vscode' {
	export namespace l10n {
		export function t(message: string, ...args: string[]): string;
		export function t(options: { message: string; args: string[], comment: string[] }): string;
		contents: { [key: string ]: string };
		uri: Uri;
	}
}

The new namespace keeps the i18n.t(...) that we will see all over code short and sweet.

@vscode/l10n-dev

Tooling used for extracting l10n strings from vscode extensions and working with XLF files. Here's what the CLI looks like:

❯node ./dist/cli.js export --help
cli.js export [args] <path..>

Export strings from source files. Supports glob patterns.

Positionals:
  path  TypeScript files to extract strings from. Supports folders and glob
        patterns.                               [array] [required] [default: []]

Options:
      --version  Show version number                                   [boolean]
      --help     Show help                                             [boolean]
  -o, --outDir   Output directory                        [string] [default: "."]

❯node ./dist/cli.js generate-xlf --help
cli.js generate-xlf [args] <path..>

Generate an XLF file from a collection of `*.l10n.json` files. Supports glob
patterns.

Positionals:
  path  L10N JSON files to generate an XLF from. Supports folders and glob
        patterns.                               [array] [required] [default: []]

Options:
      --version   Show version number                                  [boolean]
      --help      Show help                                            [boolean]
  -l, --language  The source language that will be written to the XLF file.
                                                        [string] [default: "en"]
  -o, --outFile   Output file                                [string] [required]

❯node ./dist/cli.js import-xlf --help  
cli.js import-xlf [args] <path..>

Import an XLF file into a JSON l10n file

Positionals:
  path  XLF files to turn into `*.l10n.<language>.json` files. Supports folders
        and glob patterns.                      [array] [required] [default: []]

Options:
      --version  Show version number                                   [boolean]
      --help     Show help                                             [boolean]
  -o, --outDir   Output directory that will contain the l10n.<language>.json
                 files                                   [string] [default: "."]

There is also a programatic way to interact with the tooling via JS.

@vscode/l10n

Library used for loading the translations into subprocesses of your extension. It has an API of:

• the same t function
• a config function that looks like this that will receive:

function config(config: { uri: string; } | { contents: string | l10nJsonFormat }): void

interface l10nJsonFormat {
	[key: string]: MessageInfo;
}

package.l10n.json

> NOTE: for backwards compatibility, package.nls.json is also supported.

This file is used for translating static contributions in your extension's package.json.
deferred

Niceness

The api being vscode.l10n.t() pairs nicely with @vscode/l10n because you either do this in your extension code:

import { l10n } from 'vscode';

or in a subprocess you do

import * as l10n from '@vscode/l10n';

and the API becomes the same:

l10n.t('Hello world {0}', blah);