Numeric Separator: Stage 1

[hzoo]

Advance Numeric separators proposal to Stage 1 slides (Sam Goto) (@samuelgoto)

Info

Proposed at TC39 Meeting: May 2017
Slides at the meeting: https://docs.google.com/presentation/d/1hcajTemZB2Ruo4EePOyFiva1xpyv5ukKk4aQ0B83dUA/edit#slide=id.p
Proposal Repo: https://github.com/samuelgoto/proposal-numeric-separator

Example

var thousands = 10_000; // Instead of 10000.
var credit_card_number = 1234_5678_9012_3456; // Instead of 123456789012345.
var social_security_number = 999_99_9999; // Instead of 999999999.
var pi = 3.14_15; // Instead of 3.1415
var bytes = 0b11010010_01101001_10010100_10010010; // Instead of 0b11010010011010011001010010010010.
var 0xCAFE_F00D; // Instead of 0XCAFEF00D.

Basic Rules

• to use the _ character.
• only one consecutive underscore is allowed.
• only between digits (not allowed at the beginning or end of literals)

ESTree/Parsing

• looks like we need to figure out how to change the tokenizer tt.num
• Start reading a number at getTokenFromCode :

  babylon/src/tokenizer/index.js

  Lines 456 to 464 in 9709c8d

  	case 48: // '0'
  	const next = this.input.charCodeAt(this.state.pos + 1);
  	if (next === 120 || next === 88) return this.readRadixNumber(16); // '0x', '0X' - hex number
  	if (next === 111 || next === 79) return this.readRadixNumber(8); // '0o', '0O' - octal number
  	if (next === 98 || next === 66) return this.readRadixNumber(2); // '0b', '0B' - binary number
  	// Anything else beginning with a digit is an integer, octal
  	// number, or float.
  	case 49: case 50: case 51: case 52: case 53: case 54: case 55: case 56: case 57: // 1-9
  	return this.readNumber(false);

• calls readNumber, or readRadixNumber or readInt
• ultimately need the value to include the _ or to just keep the regular value + track where the _ are?

Transform

• if the AST is simple (just the value/raw) would be trivial to remove (regex replace _ with '')?

ESTree: I guess there's nothing to add to the AST spec since this changes the value of the number itself?

cc @ajklein and @domenic

[rwaldron]

@hzoo I was thinking about writing a plugin for this, as a way to get involved. I will likely need your guidance :)

[rwaldron]

[Deleted all that non-sense. Everything is good now]

[hzoo]

awesome @rwaldron 😄 , already added some thoughts about the parser/transform. I think the hard part is figuring out how to change the parser itself. The transform should be trivial to remove the _

like in others https://github.com/babel/babel/blob/7.0/packages/babel-plugin-transform-es2015-sticky-regex/src/index.js

Also, can you try with yarn instead of npm install? It's possible I did screw something up yesterday though. https://github.com/babel/babylon/blob/master/CONTRIBUTING.md#contributing

I did do a babel alpha.10 release yesterday and we are using ^ so that's probably the issue

Ok we can discuss in slack if you want to join: http://slack.babeljs.io/

EDIT: the issue is that rollup has a dependency on babel v6 (not a peerDep) and we are on v7 with removed some APIs

[rwaldron]

already added some thoughts about the parser/transform

I'll follow your notes in the op—thanks!

[hzoo]

Ok I found a bug after a lot of investigation.. I pinned the deps in master so can pull and it should be ok now.

[rwaldron]

I have an implementation and a little test runner that verifies all of the following:

☉ 68 tests run:
✔ 68 tests passed
✘ 0 tests failed
.............................
✔ 1_1 is 11 (module)
✔ 1_1 is 11 (script)
✔ 1_1.1_1 is 11.11 (module)
✔ 1_1.1_1 is 11.11 (script)
✔ .1_1 is 0.11 (module)
✔ .1_1 is 0.11 (script)
✔ 0x1_1 is 17 (module)
✔ 0x1_1 is 17 (script)
✔ 0xa_1 is 161 (module)
✔ 0xa_1 is 161 (script)
✔ 0xA_1 is 161 (module)
✔ 0xA_1 is 161 (script)
✔ 0b01_1 is 3 (module)
✔ 0b01_1 is 3 (script)
✔ 0b0_1_1 is 3 (module)
✔ 0b0_1_1 is 3 (script)
✔ 0o1_1 is 9 (module)
✔ 0o1_1 is 9 (script)
✔ 0o0_11 is 9 (module)
✔ 0o0_11 is 9 (script)
✔ 1_ is Invalid NumericLiteralSeparator (1:1) (module)
✔ 1_ is Invalid NumericLiteralSeparator (1:1) (script)
✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (module)
✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (script)
✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (module)
✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (script)
✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (module)
✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (script)
✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (module)
✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (script)
✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (module)
✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (script)
✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (module)
✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (script)
✔ ._1_1 is Unexpected token (1:0) (module)
✔ ._1_1 is Unexpected token (1:0) (script)
✔ 0o01_8 is Unexpected token, expected ; (1:5) (module)
✔ 0o01_8 is Unexpected token, expected ; (1:5) (script)
✔ 0b2_1 is Expected number in radix 2 (1:2) (module)
✔ 0b2_1 is Expected number in radix 2 (1:2) (script)
✔ 0xZ_1 is Expected number in radix 16 (1:2) (module)
✔ 0xZ_1 is Expected number in radix 16 (1:2) (script)

I'm now figuring out a way to migrate my fixtures.

[samuelgoto]

Neat!!! Good stuff Rick!! I believe we converged to keeping parseInt intact and changing the Number constructor to take the new syntax (which leads to a backwards incompatible change, but seemed like an acceptable one). Does that make sense? (sent from phone, apologies for the brevity) On May 26, 2017 2:22 PM, "Rick Waldron" <notifications@github.com> wrote: I have an implementation and a little test runner that verifies all of the following: ☉ 68 tests run: ✔ 68 tests passed ✘ 0 tests failed ............................. ✔ 1_1 is 11 (module) ✔ 1_1 is 11 (script) ✔ 1_1.1_1 is 11.11 (module) ✔ 1_1.1_1 is 11.11 (script) ✔ .1_1 is 0.11 (module) ✔ .1_1 is 0.11 (script) ✔ 0x1_1 is 17 (module) ✔ 0x1_1 is 17 (script) ✔ 0xa_1 is 161 (module) ✔ 0xa_1 is 161 (script) ✔ 0xA_1 is 161 (module) ✔ 0xA_1 is 161 (script) ✔ 0b01_1 is 3 (module) ✔ 0b01_1 is 3 (script) ✔ 0b0_1_1 is 3 (module) ✔ 0b0_1_1 is 3 (script) ✔ 0o1_1 is 9 (module) ✔ 0o1_1 is 9 (script) ✔ 0o0_11 is 9 (module) ✔ 0o0_11 is 9 (script) ✔ 1_ is Invalid NumericLiteralSeparator (1:1) (module) ✔ 1_ is Invalid NumericLiteralSeparator (1:1) (script) ✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (module) ✔ 1_1_ is Invalid NumericLiteralSeparator (1:3) (script) ✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (module) ✔ 1_1__ is Invalid NumericLiteralSeparator (1:3) (script) ✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (module) ✔ 1__1 is Invalid NumericLiteralSeparator (1:1) (script) ✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (module) ✔ 1_1_.1_1 is Invalid NumericLiteralSeparator (1:3) (script) ✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (module) ✔ 1_1._1_1 is Invalid NumericLiteralSeparator (1:4) (script) ✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (module) ✔ 0x1_1_ is Invalid NumericLiteralSeparator (1:5) (script) ✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (module) ✔ 0xa_1_ is Invalid NumericLiteralSeparator (1:5) (script) ✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0x_a_1 is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0x__1_1_ is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0x_1__1 is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0x_1_1_ is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0o_1_1 is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0o_11 is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0o_01_1_ is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0b_0_1_1 is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0b_01_1_ is Invalid NumericLiteralSeparator (1:2) (script) ✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (module) ✔ 0b01_1_ is Invalid NumericLiteralSeparator (1:6) (script) ✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (module) ✔ 0o1_1_ is Invalid NumericLiteralSeparator (1:5) (script) ✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (module) ✔ 0o_1_1_ is Invalid NumericLiteralSeparator (1:2) (script) ✔ ._1_1 is Unexpected token (1:0) (module) ✔ ._1_1 is Unexpected token (1:0) (script) ✔ 0o01_8 is Unexpected token, expected ; (1:5) (module) ✔ 0o01_8 is Unexpected token, expected ; (1:5) (script) ✔ 0b2_1 is Expected number in radix 2 (1:2) (module) ✔ 0b2_1 is Expected number in radix 2 (1:2) (script) ✔ 0xZ_1 is Expected number in radix 16 (1:2) (module) ✔ 0xZ_1 is Expected number in radix 16 (1:2) (script) I'm now figuring out a way to migrate my fixtures. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#538 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqV6qj2h_i6A-wgHJ-Np9Tx8tLo9sAMks5r9xhNgaJpZM4NnDpy> .

[rwaldron]

I believe we converged to keeping parseInt intact and changing the Number constructor to take the new syntax (which leads to a backwards incompatible change, but seemed like an acceptable one).

That's correct, but I have to admit that I don't understand where that fits for Babylon? Guidance please :)

[samuelgoto]

Ah fair question. Henry? (sent from phone, apologies for the brevity)

…

On May 26, 2017 3:05 PM, "Rick Waldron" ***@***.***> wrote: I believe we converged to keeping parseInt intact and changing the Number constructor to take the new syntax (which leads to a backwards incompatible change, but seemed like an acceptable one). That's correct, but I have to admit that I don't understand where that fits for Babylon? Guidance please :) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#538 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqV6jzmkbW_UkCcHIXCYGI6KLdGL4j0ks5r9yJbgaJpZM4NnDpy> .

[hzoo]

Yeah it wouldn't be related to the parser. After the parser change, we need to make the transform itself and then determine whether we need a new polyfill for those kinds of methods or not.