Python 3.4 gives wrong col_offset for Call nodes returned from ast.parse

[aivarannamaa]

BPO	21295
Nosy	@brettcannon, @birkenfeld, @ncoghlan, @benjaminp, @florentx, @markshannon, @aivarannamaa
Files	py34_ast_call_bug.py: Small demonstration of the bug

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-02-02.15:53:31.829>
created_at = <Date 2014-04-18.10:49:40.566>
labels = ['interpreter-core', 'type-bug']
title = 'Python 3.4 gives wrong col_offset for Call nodes returned from ast.parse'
updated_at = <Date 2015-10-06.10:56:58.976>
user = 'https://github.com/aivarannamaa'

bugs.python.org fields:

activity = <Date 2015-10-06.10:56:58.976>
actor = 'Aivar.Annamaa'
assignee = 'none'
closed = True
closed_date = <Date 2015-02-02.15:53:31.829>
closer = 'python-dev'
components = ['Interpreter Core']
creation = <Date 2014-04-18.10:49:40.566>
creator = 'Aivar.Annamaa'
dependencies = []
files = ['34962']
hgrepos = []
issue_num = 21295
keywords = ['3.4regression']
message_count = 21.0
messages = ['216777', '216778', '216821', '216846', '221360', '235245', '235246', '235261', '235266', '237577', '237581', '237585', '237670', '237671', '237672', '237675', '251522', '252380', '252381', '252384', '252386']
nosy_count = 10.0
nosy_names = ['brett.cannon', 'georg.brandl', 'ncoghlan', 'benjamin.peterson', 'flox', 'Mark.Shannon', 'scummos', 'python-dev', 'Aivar.Annamaa', 'rnovacek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21295'
versions = ['Python 3.4']

[aivarannamaa]

Following program gives correct result in Python versions older than 3.4, but incorrect result in 3.4:

----------------------

import ast
tree = ast.parse("sin(0.5)")
first_stmt = tree.body[0]
call = first_stmt.value
print("col_offset of call expression:", call.col_offset)
print("col_offset of func of the call:", call.func.col_offset)

---

it should print:
col_offset of call expression: 0
col_offset of func of the call: 0

but in 3.4 it prints:
col_offset of call expression: 3
col_offset of func of the call: 0

[aivarannamaa]

... also, lineno is wrong for both Call and call's func, when func and arguments are on different lines:

import ast
tree = ast.parse("(sin\n(0.5))")
first_stmt = tree.body[0]
call = first_stmt.value
print("col_offset of call expression:", call.col_offset)
print("col_offset of func of the call:", call.func.col_offset)
print("lineno of call expression:", call.lineno)
print("lineno of func of the call:", call.lineno)

# lineno-s should be 1 for both call and func

[benjaminp]

I suspect this was an intentional result of bpo-16795.

[aivarannamaa]

Regarding bpo-16795, the documentation says "The lineno is the line number of source text and the col_offset is the UTF-8 byte offset of the first token that generated the node", not that lineno and col_offset indicate a suitable position to mention in the error messages related to this node.

IMO lineno and col_offset should stay as predictable means for finding the (beginning of) source text of the node. In error reporting code one could inspect the situation and compute locations suitable for this.

Alternatively, these attributes could be left for purposes mentioned in bpo-16795 and parser developers could introduce new attributes in ast nodes which indicate both start and end positions of corresponding source. (Hopefully this would resolve also bpo-18374 and bpo-16806)

[aivarannamaa]

Just found out that ast.Attribute in Python 3.4 has similar problem

[markshannon]

This is caused by https://hg.python.org/cpython/rev/7c5c678e4164/
which is a supposed fix for http://bugs.python.org/issue16795
which claims to make "some changes to AST to make it more useful for static language analysis", seemingly by breaking all existing static analysis tools.

Could we just revert https://hg.python.org/cpython/rev/7c5c678e4164/ ?

[markshannon]

It is now very hard to determine accurate locations for an expression such as (x+y).attr as the column offset of leftmost subexpression of the expression is not the same as the column offset of the location.

[markshannon]

This also breaks the col_offset for subscripts like x[y] and, of course any statement with one of these expressions as its leftmost sub-expression.

[python-dev]

New changeset 7d1c32ddc432 by Benjamin Peterson in branch '3.4':
revert lineno and col_offset changes from bpo-16795 (closes bpo-21295)
https://hg.python.org/cpython/rev/7d1c32ddc432

New changeset 8ab6b404248c by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-21295)
https://hg.python.org/cpython/rev/8ab6b404248c

[scummos]

Why did you not CC me in this discussion? It is not very nice to have this behaviour changed back from what I relied upon in a minor version without notice.

Which regression was effectively caused by this patch, except for the documentation being out of date?

[markshannon]

You are on the nosy list. You should have got sent an email.

This bug is the regression.
https://hg.python.org/cpython/rev/7c5c678e4164/ resulted in incorrect column offsets for many compound expressions.

[scummos]

Hmm, strange, I did not receive any emails.

"Incorrect" by what definition of incorrect? The word does not really help to clarify the issue you see with this change, since the behaviour was changed on purpose. What is the (preferably real-world) application which is broken by this change?