Proposal: Convert all our config files to JSON5

[octogonz]

Problem

The web-build-tools repo has used .json config files for a long time. We've always put // comments in these files, because the proposition of an undocumented config file is preposterous and unprofessional. :-P

However, GitHub's syntax highlighter has the differing opinion that JSON standard (despite ironically being a REQUEST FOR COMMENTS) does not allow comments. And despite the father of JSON himself saying that comments are fine. It's making us look bad, because all our config files get red highlighting everywhere, like this:

/**
 * This is the main configuration file for Rush.
 * For full documentation, please see https://rushjs.io
 */
{
  "$schema": "https://developer.microsoft.com/json-schemas/rush/v5/rush.schema.json",

  /**
   * (Required) This specifies the version of the Rush engine to be used in this repo.
   * Rush's "version selector" feature ensures that the globally installed tool will
   * behave like this release, regardless of which version is installed globally.
   *
   * The common/scripts/install-run-rush.js automation script also uses this version.
   *
   * NOTE: If you upgrade to a new major version of Rush, you should replace the "v5"
   * path segment in the "$schema" field for all your Rush config files.  This will ensure
   * correct error-underlining and tab-completion for editors such as VS Code.
   */
  "rushVersion": "5.5.4",

Investigation

Why doesn't JSON allow comments? It was originally intended as a machine-to-machine data format. Back when JSON was proposed, XML was already the accepted standard for large config files intended to be maintained by humans. Humans authoring JSON is a relatively new usage scenario that arose mainly with NodeJS, due to its lack of a decent XML library.

It seems like the JSON standard isn't going to fix this problem. I did some research, and here's the most obvious alternatives:

• YAML: I don't like YAML because (1) the syntax has many redundant ways to express a given thing, so I've observed people constantly having trouble remembering exactly what the different symbols mean, and (2) parsing all this YAML grammar requires a library that is large and slow, and (3) YAML doesn't have a mature schema facility; they expect you to convert YAML to JSON and then use a JSON schema validator. I will say that I do like YAML as an output format (e.g. shrinkwrap.yaml) because it is concise and easy to diff, and the library allows you to disable the weird notations.

• Hjson: This seems somewhat popular, however the syntax seems pretty counterintuitive and error-prone to me. For example, x: ab ef is translates to "x": "ab ef", whereas x: [ab ef] does not translate to "x": ["ab ef"] or "x": ["ab", "ef"].

• jsonc: Not very popular. Its syntax is different from Hjson, and also seemed counterintuitive and error-prone. For example hey: [this needs no commas] translates to "hey": ["this", "needs", "no", "commas"] and {a : b c : d} translates to {"a": "b", "c": "d"}. (Also the name is confusingly similar to https://github.com/json-c/json-c which is a JSON parser written in C.)

• JSON5: Out of the gate, JSON5 has the advantage that it's not trying to invent anything new. Their idea is to carve out a JSON-like subset of the ECMAScript 5 grammar that is already standardized and familiar to everyone. It's better than ECMAScript because this is a pure data representation (e.g. that can be described by a conventional JSON schema), and it can be read and rewritten with a relatively simple parser. Conveniently, the @microsoft/node-core-library JsonFile API is based on jju which already supports JSON5. (I didn't realize this, but you can already put JSON5 notation in rush.json and Rush doesn't complain. Oops!)

  The one concern I had with JSON5 was that ECMAScript is continually evolving: Their file extension is .json5 -- will we have to come back every year and rename all our file extensions to .json6 or .json7 or .json8 with each new JavaScript standard? Nobody does that with .js files. I opened an issue to ask about this, and they had a reasonable argument that newer ECMAScript versions have relatively little to contribute to the syntaxes relevant to JSON.

Proposal

1. Let's convert all our config files to use JSON5.
2. Let's rename the file extension to .json5 so GitHub highlights it correctly
3. Let's fix the places where we write .json with non-JSON content (e.g. issue Ensure schema JSON documents are valid JSON #996)

What do you think? Comments welcome...

@mike-north @hogmoru @iclanton @daspek @patmill @dzearing @cliffkoh @kenotron

[octogonz]

BTW I noticed that VS Code's JSON language service supports comments using a library called jsonc-parser that accepts "JSON with JavaScript style comments" and no other syntax extensions besides that. (Not to be confused with jsonc which I mentioned above.)

[octogonz]

If you want JSON5 highlighting in VS Code, apparently it's not built-in. You have to install the JSON5 syntax extension. If you want this highlighter to operation on file with the .json file extension, you can add this to your VS Code settings.json:

"files.associations": { "*.json": "json5" }

[daspek]

How big of a problem is the syntax highlighting in github? I would think that changing our file extension would be more confusing than leaving things as they are.

This problem can't be unique to us. I just perused the repos of a handful of popular open source projects. Most had comments in their .json files somewhere, complete with syntax highlighting errors. None had a json5 or jsonc or hjson file.

[octogonz]

How big of a problem is the syntax highlighting in github? I would think that changing our file extension would be more confusing than leaving things as they are.

VS Code is now following GitHub's convention and displaying rush.json with tons of red underlines everywhere. It seems like the community is deciding that "JSON is for machines," and we really should be using a different file format for human-editable config files.

JSON5 has all the right characteristics:

• It preserves the character of JSON, and adds only a minimal set of enhancements
• The enhancements are all 100% standardized by ECMAScript 5
• The "5" in the file extension protects against further feature creep (if they make a JSON6, we'll probably choose not to adopt it, because JSON5 is good enough)
• JSON5 is already fully implemented by our JSON parser and works with our existing schemas
• There's a growing community of JavaScript developers adopting it

[daspek]

I haven't personally seen evidence of the growing community of developers adopting json5. I do believe that most folks agree with the spirit, but i have never seen a "json5" file in the wild. and according to google trends there doesn't seem to be any appreciable movement. also reference this comparison of xml, json, and json5 over the last 5 years.

i still believe that the disruption from changing our file extension far outweighs any of the benefits.

[octogonz]

I think my "stake in the ground" is that my config files are going to have comments in them. And the syntax highlighters are not going to make them appear to be full of errors.

So it seems like the options are:

• change the file extension to a JSON variant that allows comments
• abandon JSON entirely
• change VS Code and GitHub to accept comments in JSON files (both a matter of persuading some people at Microsoft)

I think I'd be okay with any of these options. Which one are you advocating?

[octogonz]

@zkochan just posted a Twitter poll announcing that PNPM will support package.json5. :-)

[chaseholdren]

.gitattributes file contents:

*.json                       linguist-language=JSON-with-Comments

[xiaoxiangmoe]

I'm using api-extractor run --local --verbose --config api-extractor.jsonc now.

jsonc means JSON-with-Comments in VSCode.

[alipi]

Why not just use ESM?

rush.config.mjs

/**
 * This is the main configuration file for Rush.
 * For full documentation, please see https://rushjs.io
 */

export default {

   $schema: "https://developer.microsoft.com/json-schemas/rush/v5/rush.schema.json",

   /**
    * (Required) This specifies the version of the Rush engine to be used in this repo.
    * Rush's "version selector" feature ensures that the globally installed tool will
    * behave like this release, regardless of which version is installed globally.
    *
    * The common/scripts/install-run-rush.js automation script also uses this version.
    *
    * NOTE: If you upgrade to a new major version of Rush, you should replace the "v5"
    * path segment in the "$schema" field for all your Rush config files.  This will ensure
    * correct error-underlining and tab-completion for editors such as VS Code.
    */
   rushVersion: "5.47.0",

   // ...
};

rush.js

(async () => {
   try {
      const { default: config } = await import("./rush.config.mjs");
      console.log(config);
   }
   catch(ex) {
      console.error(ex);
   }
})();

output

{
  '$schema': 'https://developer.microsoft.com/json-schemas/rush/v5/rush.schema.json',
  rushVersion: '5.47.0'
}

[elliot-nelson]

Closing for now, as the issue is resolved.

• Our JSON files look good in GitHub (and so do any new rush monorepos if they use the provided .gitattributes default).
• When pasting snippets of JSON files in GitHub comments or pull requests, you can use the jsonc header to get JSON-with-comments styling.
• Our parser supports JSON5, but 3 years later I still haven't seen much of a push to switch to JSON5 so I think we can stick with JSONC for now.