Implement PEP 498: Literal String Formatting

[ericvsmith]

BPO	24965
Nosy	@warsaw, @ericvsmith, @Rosuav, @vadmium, @1st1, @JelleZijlstra
Files	pep-498-3.diff pep-498-4.diff pep-498-5.diff pep-498-6.diff pep-498-7.diff pep-498-8.diff pep-498-9.diff pep-498-10.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2015-09-19.18:57:44.348>
created_at = <Date 2015-08-30.17:47:00.728>
labels = ['interpreter-core', 'type-feature']
title = 'Implement PEP 498: Literal String Formatting'
updated_at = <Date 2015-09-19.18:57:44.346>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2015-09-19.18:57:44.346>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2015-09-19.18:57:44.348>
closer = 'eric.smith'
components = ['Interpreter Core']
creation = <Date 2015-08-30.17:47:00.728>
creator = 'eric.smith'
dependencies = []
files = ['40421', '40430', '40447', '40480', '40484', '40495', '40502', '40515']
hgrepos = []
issue_num = 24965
keywords = ['patch']
message_count = 39.0
messages = ['249362', '249364', '249365', '249475', '249481', '249810', '250343', '250344', '250354', '250420', '250422', '250465', '250467', '250485', '250493', '250525', '250527', '250528', '250529', '250530', '250531', '250532', '250538', '250541', '250544', '250546', '250548', '250557', '250598', '250821', '250849', '250926', '250927', '250929', '250931', '250970', '251073', '251100', '251102']
nosy_count = 8.0
nosy_names = ['barry', 'eric.smith', 'python-dev', 'Rosuav', 'martin.panter', 'yselivanov', 'JelleZijlstra', 'elvis']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24965'
versions = ['Python 3.6']

[ericvsmith]

See PEP-498.

>>> f'New for Python {sys.version.split()[0]}'
'New for Python 3.6.0a0'

[ericvsmith]

One thing I've done in this implementation is to build up a string to pass to str.format(), instead of using the original string. This new string uses positional parameters instead of named parameters.

I had originally proposed to add a string.interpolate() to do the heavy lifting here, which would have meant I could use the original string (as seen in the source code), and not build up a new string and pass it to str.format(). I still might do that, but for now, the approach using str.format() is good enough.

[ericvsmith]

Oops, I didn't really mean to include imporlib.h. Oh, well.

[ericvsmith]

Fixed validate_exprs bug.

[ericvsmith]

Make sure f-strings are identified as literals in error messages.

[python-dev]

New changeset a0194ec4195c by Eric V. Smith in branch 'default':
Removed Implementation Limitations section. While the version of the code on http://bugs.python.org/issue24965 has the 255 expression limitation, I'm going to remove this limit. The i18n section was purely speculative. We can worry about it if/when we add i18n and i-strings.
https://hg.python.org/peps/rev/a0194ec4195c

[ericvsmith]

This implements the accepted PEP-498. The only other real change I plan on making is to do dynamic memory allocation when building the expressions that make up a JoinedStr AST node. The code has all of the places to do that already laid out, it's just a matter of hooking it up.

There's one nit where I accept 'f' and 'F', but the PEP just says 'f'. I'm not sure if we should accept the upper case version. I'd think not, but all of the other ones (b, r, and u) do.

I need to do one more scan for memory leaks. I've rearranged some code since the last time I checked for leaks, and that's always a recipe for some sneaking in.

And I need to write some more tests, mostly for syntax errors, but also for a few edge conditions.

Comments welcome.

[warsaw]

On Sep 09, 2015, at 11:57 PM, Eric V. Smith wrote:

There's one nit where I accept 'f' and 'F', but the PEP just says 'f'. I'm
not sure if we should accept the upper case version. I'd think not, but all
of the other ones (b, r, and u) do.

I think it should be consistent with the other prefixes. Shouldn't be a big
deal to amend the PEP to describe this.

[ericvsmith]

I discussed it with Guido and added 'F' to the PEP.

[ericvsmith]

This version does dynamic allocation for the expression list, and fixes some memory leaks and early decrefs.

I think it's complete, but I'll take some more passes through it checking for leaks.

[ericvsmith]

The good news is that the performance is pretty good, and finally I have a case where I can beat %-formatting:

$ ./python.bat -mtimeit -s 'a=2' "'%s' % a"
1000000 loops, best of 3: 0.883 usec per loop

$ ./python.bat -mtimeit -s 'a=2' '"{}".format(a)'
1000000 loops, best of 3: 1.16 usec per loop

$ ./python.bat -mtimeit -s 'a=2' 'f"{a}"'
1000000 loops, best of 3: 0.792 usec per loop

This example is mildly contrived, and the performance of f-strings is slightly worse than %-formatting once the f-strings contains both expressions and literals.

I could speed it up significantly (I think) by adding opcodes for 2 things: calling __format__ and joining the strings together. Calling __format__ in an opcode could be a win because I could optimize for known types (str, int, float). Having a join opcode would be a win because I could use _PyUnicodeWriter instead of ''.join.

I'm inclined to check this code in as-is, then optimize it later, if we think it's needed and if I get motivated.

For reference, here's the ast and opcodes for f'a={a}':

>>> ast.dump(ast.parse("f'a={a}'"))
"Module(body=[Expr(value=JoinedStr(values=[Str(s='a='), FormattedValue(value=Name(id='a', ctx=Load()), conversion=0, format_spec=None)]))])"

>>> dis.dis("f'a={a}'")
  1           0 LOAD_CONST               0 ('')
              3 LOAD_ATTR                0 (join)
              6 LOAD_CONST               1 ('a=')
              9 LOAD_NAME                1 (a)
             12 LOAD_ATTR                2 (__format__)
             15 LOAD_CONST               0 ('')
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             21 BUILD_LIST               2
             24 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             27 RETURN_VALUE

[vadmium]

Another version of that AST that is better for my digestion:

f'a={a}'

Module(body=[Expr(
value=JoinedStr(values=[
Str(s='a='),
FormattedValue(
value=Name(id='a', ctx=Load()),
conversion=0,
format_spec=None,
),
]),
)])

I have been reading over the test cases, and left a bunch of suggestions for more edge cases etc. Some of them might reflect that I haven’t completely learnt how the inner Python expression syntax, outer string escaping syntax, {{curly bracket}} escaping, automatic concatenation, etc, are all meant to fit together.