ast unparse does not support f-string new debug format.

[Carreau]

BPO	37003
Nosy	@ericvsmith, @Carreau, @pablogsal
PRs	bpo-37003: handle f-debug in ast-unparse, #13475

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2019-05-28.01:39:45.964>
created_at = <Date 2019-05-21.22:03:08.301>
labels = ['3.8', 'library']
title = 'ast unparse does not support f-string new debug format.'
updated_at = <Date 2019-05-28.01:39:45.964>
user = 'https://github.com/Carreau'

bugs.python.org fields:

activity = <Date 2019-05-28.01:39:45.964>
actor = 'mbussonn'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2019-05-28.01:39:45.964>
closer = 'mbussonn'
components = ['Library (Lib)']
creation = <Date 2019-05-21.22:03:08.301>
creator = 'mbussonn'
dependencies = []
files = []
hgrepos = []
issue_num = 37003
keywords = ['patch']
message_count = 11.0
messages = ['343105', '343128', '343131', '343144', '343153', '343212', '343222', '343226', '343227', '343725', '343726']
nosy_count = 3.0
nosy_names = ['eric.smith', 'mbussonn', 'pablogsal']
pr_nums = ['13475']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37003'
versions = ['Python 3.8']

[Carreau]

All in title, f"{x=}" implemented in https://bugs.python.org/issue36817
are not liked by unparse: #13473,

[ericvsmith]

I don’t think this is worth changing. I’m planning on removing the expr_text field before beta 1. At which point I don’t think there will be any work to do here.

See https://mail.python.org/pipermail/python-dev/2019-May/157493.html

Also, can you explain what’s currently broken? I’ll need to see some example code that doesn’t work in order to understand your change and if the ast change will affect it.

[ericvsmith]

Never mind on the failing code. I see your test cases now.

[Carreau]

Just for clarification for future readers;

I landed a PR that used f"{x=}" in some new functionalities, and that broke the buildbots, the buildbots do test that many of the Lib/ files can be round-tripped ast->unparse->ast, and as ast-unparse did not understood f-debug, if roundtripped f"{x=}" to f"{x!r}". The PRs CIs did not fail as apparently they don't test this functionality.

So currently not having support for f-string-debug in ast-unparse kind of prevent using f-string debug format in the stdlib.

I'm happy if expr_text get removed, and having unparse round-trip to x={x!r} sounds sufficient to me; it may make like of formatters like black a bit harder maybe ? Though I believe black is likely not using ast but lib2to3.

[ericvsmith]

Thanks for the recap, Matthias. Once I remove expr_text (hopefully this weekend!), we should still at least add tests to make sure nothing fails, even with the ast-expanded version of the = f-strings.

[pablogsal]

Notice that test_tools will fail if f'{x=}' becomes f'x={x!r}' when unparsed as it compares the text of the file and the text of the roundtrip of the ast of the file

[Carreau]

I thought it was comparing the AST of the file to the AST of the unparsed
AST.

So it's actually checking if parse(unparse(x)) is indempotent and not
wether parse(unparse(x)) is indempotent.

So x={x!r} should be fine.

On Wed, May 22, 2019, 09:22 Pablo Galindo Salgado <report@bugs.python.org>
wrote:

Pablo Galindo Salgado <pablogsal@gmail.com> added the comment:

Notice that test_tools will fail if f'{x=}' becomes f'x={x!r}' when
unparsed as it compares the text of the file and the text of the roundtrip
of the ast of the file

----------
nosy: +pablogsal

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue37003\>

---

[pablogsal]

Actually, it checks if the dump is the same:

 class ASTTestCase(unittest.TestCase):
    def assertASTEqual(self, ast1, ast2):
        self.assertEqual(ast.dump(ast1), ast.dump(ast2))

    def check_roundtrip(self, code1, filename="internal"):
        ast1 = compile(code1, filename, "exec", ast.PyCF_ONLY_AST)
        unparse_buffer = io.StringIO()
        unparse.Unparser(ast1, unparse_buffer)
        code2 = unparse_buffer.getvalue()
        ast2 = compile(code2, filename, "exec", ast.PyCF_ONLY_AST)
        self.assertASTEqual(ast1, ast2)

[pablogsal]

Notice that test_tools will fail if f'{x=}' becomes f'x={x!r}'

I arrived at the wrong conclusion as Matthias points out.

>>> import ast
>>> ast.dump(compile("f'{x=}'","<string>","exec",ast.PyCF_ONLY_AST))
"Module(body=[Expr(value=JoinedStr(values=[FormattedValue(value=Name(id='x', ctx=Load()), conversion=114, format_spec=None, expr_text='x=')]))], type_ignores=[])"
>>> ast.dump(compile("f'{x!r}'","<string>","exec",ast.PyCF_ONLY_AST))
"Module(body=[Expr(value=JoinedStr(values=[FormattedValue(value=Name(id='x', ctx=Load()), conversion=114, format_spec=None, expr_text=None)]))], type_ignores=[])"

if expr_text is removed those strings will be the same, so we will be ok. Sorry for the confusion.

[ericvsmith]

mbussonn: can you re-check this in light of the changes for bpo-37050? I'm not sure if there's still a problem that needs fixing or not.

[Carreau]

I believe all is fine now. Thanks !