Improve exception message in ast.literal_eval

[Rosuav]

BPO	32888
Nosy	@briancurtin, @Rosuav, @serhiy-storchaka
PRs	bpo-32888: Improve exception message in ast.literal_eval #340 bpo-38396: dump(node) instead of repr(node) on ast.literal_eval #16620 bpo-32888: enhance ast.literal_eval error messagess with context information #17662
Dependencies	bpo-32892: Remove specific constant AST types in favor of ast.Constant bpo-32893: ast.literal_eval() shouldn't accept booleans as numbers in AST

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2018-02-20.17:26:21.508>
labels = ['3.8', 'library']
title = 'Improve exception message in ast.literal_eval'
updated_at = <Date 2019-12-19.15:26:59.362>
user = 'https://github.com/Rosuav'

bugs.python.org fields:

activity = <Date 2019-12-19.15:26:59.362>
actor = 'BTaskaya'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2018-02-20.17:26:21.508>
creator = 'Rosuav'
dependencies = ['32892', '32893']
files = []
hgrepos = []
issue_num = 32888
keywords = ['patch']
message_count = 6.0
messages = ['312423', '312424', '312427', '312429', '312442', '320993']
nosy_count = 3.0
nosy_names = ['brian.curtin', 'Rosuav', 'serhiy.storchaka']
pr_nums = ['340', '16620', '17662']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue32888'
versions = ['Python 3.8']

[Rosuav]

When a non-literal is given to literal_eval, attempt to be more
helpful with the message, rather than calling it 'malformed'.

[serhiy-storchaka]

I afraid this makes an error message more confusing and misleading.

>>> ast.literal_eval('~2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/ast.py", line 93, in literal_eval
    return _convert(node_or_string)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 92, in _convert
    return _convert_signed_num(node)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 65, in _convert_signed_num
    return _convert_num(node)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 56, in _convert_num
    raise ValueError('%s not allowed in literal' % type(node).__name__)
ValueError: UnaryOp not allowed in literal

This is not true since + and - are allowed:

>>> ast.literal_eval('-2')
-2

>>> ast.literal_eval('2+3')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/ast.py", line 93, in literal_eval
    return _convert(node_or_string)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 92, in _convert
    return _convert_signed_num(node)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 65, in _convert_signed_num
    return _convert_num(node)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 56, in _convert_num
    raise ValueError('%s not allowed in literal' % type(node).__name__)
ValueError: BinOp not allowed in literal

But:

>>> ast.literal_eval('2+3j')
(2+3j)

>>> ast.literal_eval('"a"+"b"')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/ast.py", line 93, in literal_eval
    return _convert(node_or_string)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 85, in _convert
    left = _convert_signed_num(node.left)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 65, in _convert_signed_num
    return _convert_num(node)
  File "/home/serhiy/py/cpython/Lib/ast.py", line 56, in _convert_num
    raise ValueError('%s not allowed in literal' % type(node).__name__)
ValueError: Str not allowed in literal

But Str is allowed:

>>> ast.literal_eval('"ab"')
'ab'

[Rosuav]

Hmm, I think I see what I broke there. It was part of the merge conflict resolution - I moved the check into the function, which is actually incorrect. It wasn't misleading like that in the original patch. Will fix that.

[Rosuav]

Actually, it's a bit more complicated than I thought. Current proposed solution: Track the context of each conversion, thus allowing different errors to be distinguished.

[Rosuav]

(BTW, by "proposed" I mean that the change that I describe is in the PR.)

[serhiy-storchaka]

Please wait until bpo-32892 and bpo-32893 be committed or rejected before merging this PR.