PEP 515: Tokenizer: allow underscores for grouping in numeric literals

[birkenfeld]

BPO	26331
Nosy	@brettcannon, @birkenfeld, @rhettinger, @mdickinson, @scoder, @ericvsmith, @ned-deily, @encukou, @skrah, @ethanfurman, @serhiy-storchaka, @1st1, @zhangyangyu
Files	numeric_underscores_strict.patch numeric_underscores_v4_full.diff numeric_underscores_final_v6.diff numeric_underscores_final_v7.diff numeric_underscores_final_v8.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2016-09-09.21:58:37.474>
created_at = <Date 2016-02-10.17:50:24.454>
labels = ['interpreter-core', 'type-feature', 'release-blocker']
title = 'PEP 515: Tokenizer: allow underscores for grouping in numeric literals'
updated_at = <Date 2016-09-11.16:20:07.708>
user = 'https://github.com/birkenfeld'

bugs.python.org fields:

activity = <Date 2016-09-11.16:20:07.708>
actor = 'georg.brandl'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2016-09-09.21:58:37.474>
closer = 'brett.cannon'
components = ['Interpreter Core']
creation = <Date 2016-02-10.17:50:24.454>
creator = 'georg.brandl'
dependencies = []
files = ['41893', '41896', '42854', '42887', '42939']
hgrepos = []
issue_num = 26331
keywords = ['patch']
message_count = 41.0
messages = ['260026', '260029', '260031', '260033', '260036', '260037', '260046', '260057', '260077', '260081', '260093', '260102', '260229', '260230', '260231', '260232', '260233', '260235', '260239', '260240', '260241', '262037', '262043', '262050', '265587', '265589', '265616', '265804', '265821', '265825', '266024', '266058', '272834', '273384', '275148', '275262', '275364', '275461', '275463', '275551', '275804']
nosy_count = 14.0
nosy_names = ['brett.cannon', 'georg.brandl', 'rhettinger', 'mark.dickinson', 'scoder', 'eric.smith', 'ned.deily', 'petr.viktorin', 'skrah', 'ethan.furman', 'python-dev', 'serhiy.storchaka', 'yselivanov', 'xiang.zhang']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26331'
versions = ['Python 3.6']

[birkenfeld]

As discussed on python-ideas: https://mail.python.org/pipermail/python-ideas/2016-February/038354.html

The rules are:
Underscores are allowed anywhere in numeric literals, except:

• at the beginning of a literal (obviously)
• at the end of a literal
• directly after a dot (since the underscore could start an attribute name)
• directly after a sign in exponents (for consistency with leading signs)
• in the middle of the "0x", "0o" or "0b" base specifiers

Currently this only touches literals, not the inputs of int() or float(). Whether they should accept this syntax is debatable (I'd vote no).

Otherwise missing: doc updates.

Review question: is PyMem_RawStrdup/RawFree the right API to use here?

[serhiy-storchaka]

I prefer simpler and more strict rule:

• Underscores are allowed only between digits in numeric literals.

Thus 1__2, 12_, 1_.2, 1_e2, 1e_2, 1_j, 0x_12 are not allowed.

It is easier to make the rule more lenient later if it will be needed.

[birkenfeld]

It sure is more strict, but I don't think it's simpler (and it's definitely not simpler to implement).

(Also 1_j is pretty nice, I wouldn't want to lose that.)

We can also check what other languages do.

• Rust: very much like this, but trailing underscores allowed.

• Perl 5: same as here, but underscores after dot and trailing underscores allowed.

• Ruby: only between digits.

• Swift: the grammar productions say it's basically the same as Rust. The textual description says "between digits".

[serhiy-storchaka]

• Java: only between digits. [1]
• Julia: only between digits. [2] (not well specified)
• C# 7.0 (proposal): only between digits, but adjacent underscores allowed. [3]
• Ada: only between digits. [4] (strong but very simple rules)
• D: very much like proposed patch, but trailing underscores allowed. [5]
• Perl 5: only between digits as documented (23__500 is not legal), but actually more lenient. [6]

[1] https://docs.oracle.com/javase/7/docs/technotes/guides/language/underscores-literals.html
[2] http://docs.julialang.org/en/release-0.4/manual/integers-and-floating-point-numbers/
[3] dotnet/roslyn#216
[4] http://archive.adaic.com/standards/83lrm/html/lrm-02-04.html#2.4
[5] http://dlang.org/spec/lex.html#integerliteral
[6] http://perldoc.perl.org/perldata.html#Scalar-value-constructors

[1st1]

I prefer simpler and more strict rule:

• Underscores are allowed only between digits in numeric literals.

+1. But in any case we need a PEP for this change.

[serhiy-storchaka]

C++14 uses the same strict rule as Ada, but uses apostrphes instead of underscores. [1]

Thus there are two groups of languages, implementing strict or lenient rules:

• Strict: Ada, C++, Java, C#, Ruby, Julia, Perl (as documented), Swift (textual description).
• Lenient: D, Rust, Perl (actually), Swift (grammar productions).

[1] http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3499.html

[birkenfeld]

PEP-515 is written up and posted to python-dev.

[encukou]

Regarding the patch: if trailing underscores are not allowed, 0 if 1_____else 1 should be illegal.

[birkenfeld]

New patch matching revision of PEP.

[serhiy-storchaka]

Proposed patch implements strict underscore rules. The implementation is not more complex.

[birkenfeld]

This patch includes int(), float(), complex() operations, as well as _pydecimal.

[birkenfeld]

New patch with minimal doc updates.

[skrah]

I like the feature for literals, but I'm not sure about conversions from string. It slows down the conversion for (IMO) a very small benefit.

Other languages allow it, but I've never attempted to use the feature:

$ ocaml
        OCaml version 4.02.1

# float_of_string "__1____2____.___e___101_";;

• : float = 1.2e+102