Doc strings no longer stored in body of AST

[markshannon]

BPO	32911
Nosy	@brettcannon, @birkenfeld, @rhettinger, @ncoghlan, @benjaminp, @ned-deily, @methane, @ambv, @markshannon, @serhiy-storchaka, @grimreaper, @minrk, @Carreau
PRs	bpo-32911: Add new AST node for docstring #5927 bpo-32911: Remove the docstring attribute of AST types #7121 [3.8] bpo-32911: Revert bpo-29463. (GH-7121) #7197 [3.7] bpo-32911: Update the historical note about the magic number. #7272 [3.8] bpo-32911: Add the historical note about the magic number. #7273

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-30.12:27:57.966>
created_at = <Date 2018-02-22.12:20:07.907>
labels = ['interpreter-core', 'type-bug', '3.7', 'release-blocker']
title = 'Doc strings no longer stored in body of AST'
updated_at = <Date 2018-05-31.06:12:17.993>
user = 'https://github.com/markshannon'

bugs.python.org fields:

activity = <Date 2018-05-31.06:12:17.993>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-30.12:27:57.966>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2018-02-22.12:20:07.907>
creator = 'Mark.Shannon'
dependencies = []
files = []
hgrepos = []
issue_num = 32911
keywords = ['patch', '3.7regression']
message_count = 55.0
messages = ['312557', '312558', '312559', '312561', '312562', '312564', '312565', '312567', '312580', '312588', '312625', '312629', '312630', '312856', '312989', '312999', '313000', '313001', '313002', '313005', '313315', '313698', '313997', '314056', '315238', '315240', '315581', '315582', '317450', '317491', '317492', '317493', '317654', '317655', '317657', '317665', '317671', '317683', '317686', '317696', '317716', '317718', '317777', '317782', '317783', '317784', '317785', '317988', '317992', '318089', '318147', '318150', '318176', '318258', '318259']
nosy_count = 15.0
nosy_names = ['brett.cannon', 'georg.brandl', 'rhettinger', 'ncoghlan', 'benjamin.peterson', 'ned.deily', 'methane', 'lukasz.langa', 'Mark.Shannon', 'serhiy.storchaka', 'eitan.adler', 'minrk', 'mbussonn', 'flherne', 'Benjamin Ragan-Kelley']
pr_nums = ['5927', '7121', '7197', '7272', '7273']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32911'
versions = ['Python 3.7']

[markshannon]

Python 3.7.0b1+ (heads/3.7:dfa1144, Feb 22 2018, 12:10:59)

>>> m = ast.parse("def foo():\n   'help'")
>>> m.body[0].body
[]

Correct behaviour (3.6 or earlier)

>>> m = ast.parse("def foo():\n   'help'")
>>> m.body[0].body
[<_ast.Expr object at 0x7fb9eeb1d4e0>]

[methane]

AST is changed slightly from Python 3.7.
ast.get_docstring() works for both of 3.6 and 3.7.

$ ./python
Python 3.8.0a0 (heads/master-dirty:451d1edaf4, Feb 22 2018, 21:11:54)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> m = ast.parse("def foo():\n   'help'")
>>> ast.get_docstring(m.body[0])
'help'

[markshannon]

This is an unnecessary and breaking change.
Changes like this should not be made unless necessary to fix a bug.

[methane]

ref: https://bugs.python.org/issue29463

[markshannon]

That issue has to do with bytecode generation, not the AST.

The AST should be an accurate representation of the Python source code.
Making a better representation of the source would be fine, but this makes it worse.

Doc-strings may be semantically distinct from other expressions, but they are syntactically the same.

[methane]

We implemented AST-level constant folding (bpo-29469) based on the AST change (bpo-29463).

I feel it's possible to revert AST change without reverting AST constant folding:

• Docstring is exposed from AST before calling AST optimizer.
• AST optimizer removes unused constant.

But I don't remember how difficult it is because since this change is made one year ago (GH-46)...

[serhiy-storchaka]

This isn't a bug, but a feature. You no longer need to check and skip the first statement if it is a literal string. The body attribute now always represents a sequence of statements, and the docstring attribute represents a docstring.

[markshannon]

Stating that "this is a feature not a bug" does not make it so.
This breaks existing code and reduces the capabilities of the ast module.
For example, how does one get the location of the docstring now?

From a syntactic point of view.
def foo():
"help"
and
def foo():
b'"help"
barely differ.
The S in AST stands for Syntax not Semantics.

[ned-deily]

It seems we have a difference of opinion here. Serhiy closed this issue so, Mark, if you feel strongly enough to pursue it, you should reopen it and solicit other opinions. The clock has just about run out to change the now current behavior for 3.7.0.