Add `docstring` field to AST nodes

[methane]

BPO	29463
Nosy	@birkenfeld, @vstinner, @benjaminp, @methane, @serhiy-storchaka, @jeff5, @Carreau
PRs	bpo-29463: Add docstring field to some AST nodes. #46 bpo-29637: clean docstring only if not None #267 [3.8] bpo-32911: Revert bpo-29463. (GH-7121) #7197
Files	ast-docstring.patch ast-docstring-2.patch ast-docstring-3.patch: fix docs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-02-23.03:36:33.764>
created_at = <Date 2017-02-06.14:21:16.297>
labels = ['interpreter-core', 'type-feature', '3.7']
title = 'Add `docstring` field to AST nodes'
updated_at = <Date 2018-05-29.09:04:58.660>
user = 'https://github.com/methane'

bugs.python.org fields:

activity = <Date 2018-05-29.09:04:58.660>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2017-02-23.03:36:33.764>
closer = 'methane'
components = ['Interpreter Core']
creation = <Date 2017-02-06.14:21:16.297>
creator = 'methane'
dependencies = []
files = ['46543', '46567', '46569']
hgrepos = []
issue_num = 29463
keywords = ['patch']
message_count = 23.0
messages = ['287136', '287207', '287215', '287273', '287283', '287287', '287289', '287292', '287293', '287365', '287366', '287378', '287620', '287634', '287636', '288373', '288374', '288399', '288400', '290419', '290429', '317987', '317991']
nosy_count = 7.0
nosy_names = ['georg.brandl', 'vstinner', 'benjamin.peterson', 'methane', 'serhiy.storchaka', 'jeff.allen', 'mbussonn']
pr_nums = ['46', '267', '7197']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue29463'
versions = ['Python 3.7']

[methane]

spin off of bpo-11549.

http://bugs.python.org/issue11549#msg130955

b) Docstring is now an attribute of Module, FunctionDef and ClassDef, > rather than a first statement. Docstring is a special syntactic
construction, it's not an executable code, so it makes sense to separate it. Otherwise, optimizer would have to take extra care not to introduce, change or remove docstring. For example:

def foo():
"doc" + "string"

Without optimizations foo doesn't have a docstring. After folding, however, the first statement in foo is a string literal. This means that docstring depends on the level of optimizations. Making it an attribute avoids the problem.

[vstinner]

I like the change because (IMHO) it makes the code simpler, and becase it also changes the first line of code object. I reviewed the patch: need basic unit tests.

[serhiy-storchaka]

I like this change. Added comments on Rietveld.

Are changes in importlib.h only due to changing first line numbers?

[methane]

lnotab is changed too.

-    0,0,0,115,12,0,0,0,8,4,4,2,8,8,8,12,
-    8,25,8,13,114,18,0,0,0,99,0,0,0,0,0,0,
+    0,0,0,115,10,0,0,0,12,6,8,8,8,12,8,25,
+    8,13,114,18,0,0,0,99,0,0,0,0,0,0,0,0,

115 is header for bytes type.
next 4 bytes is it's length (little endian, 12->10 bytes)
and everything after that is slided by 2 bytes.

[vstinner]

def func(): "doc" + "string"

Currently (Python 2.7-3.6), func.__doc__ is None. I suggest to add an unit test for this corner case, even if the result is going to change in a near future. We need to "specify" the expected behaviour, and make sure that we get the same result if optimizations are enabled or not.

[serhiy-storchaka]

Support adding tests. Tests should cover all cases: module, class, function, coroutine and check also the first line number.

What is the value of co_firstlineno if the function doesn't have any statements?

    def f():
        '''docstring'''

[methane]

Oh, I misunderstood something.
patched Python 3.7 and system's Python 3.5 shows same output for code below.
I'll check what is actually changed.

inada-n@x250 ~/w/p/ast-docstring> cat -n x.py
1 """module docstring"""
2
3 def func():
4 """func docstring"""
5
6 def func2():
7 """func docstring"""
8 1+1
9
10 print(func.__code__.co_firstlineno)
11 print(func.__code__.co_lnotab)
12 print(func2.__code__.co_firstlineno)
13 print(func2.__code__.co_lnotab)
inada-n@x250 ~/w/p/ast-docstring> ./python x.py
3
b''
6
b'\x00\x02'
inada-n@x250 ~/w/p/ast-docstring> /usr/bin/python3 x.py
3
b''
6
b'\x00\x02'

[vstinner]

2017-02-08 10:08 GMT+01:00 INADA Naoki <report@bugs.python.org>:

 6  def func2():
 7      """func docstring"""
 8      1+1

1+1 is replaced with 2 and lone integer literals are removed by the
peephole optimizer. See also the issue bpo-26204.

[vstinner]

Oops, I spoke too fast :-) "1+1" is not removed.

"1+1" is replaced with "2" by the peephole optimizer, whereas the compiler ignoring constants comes before the peephole optimizer.

One more time, it would be better to implement constant folding at the AST level ;-)

$ python3
Python 3.5.2 (default, Sep 14 2016, 11:28:32) 
>>> def func():
...  "docstring"
...  1+1
... 
>>> import dis
>>> dis.dis(func)
  3           0 LOAD_CONST               3 (2)
              3 POP_TOP
              4 LOAD_CONST               2 (None)
              7 RETURN_VALUE

[methane]

This patch affects firstlineno and lnotab of module and class, but not functions.

module:
-<code object <module> at 0x7f053a8f8b70, file "Lib/importlib/_bootstrap.py", line 8>
+<code object <module> at 0x7fdefbf10340, file "Lib/importlib/_bootstrap.py", line 25>
 filename Lib/importlib/_bootstrap.py
-firstlineno: 8
+firstlineno: 25
-lnotab: b'\x04\x11\x04\x02\x08\x08\x08\x07\x04\x02\x04\x03\x10\x04\x0eD\x0e\x15\x0e\x13\x08\x13\x08\x13\x08\x0b\x0e\x08\x08\x0b\x08\x0c\x08\x10\x08$\x0e\x1b\x0ee\x10\x1a\x06\x03\n-\x0e<\x08\x11\x08\x11\x08\x19\x08\x1d\x08\x17\x08\x10\x0eI\x0eM\x0e\r\x08\t\x08\t\n/\x08\x14\x04\x01\x08\x02\x08\x1b\x08\x06\n\x19\x08\x1f\x08\x1b\x12#\x08\x07\x08/'
-  8           0 LOAD_CONST               0 ('Core implementation of import.\n\nThis module is NOT meant to be directly imported! It has been designed such\nthat it can be bootstrapped into Python as the implementation of import. As\nsuch it requires the injection of specific modules and attributes in order to\nwork. One should use importlib as the public-facing version of this module.\n\n')
+lnotab: b'\x04\x00\x04\x02\x08\x08\x08\x07\x04\x02\x04\x03\x10\x04\x0eD\x0e\x15\x0e\x13\x08\x13\x08\x13\x08\x0b\x0e\x08\x08\x0b\x08\x0c\x08\x10\x08$\x0e\x1b\x0ee\x10\x1a\x06\x03\n-\x0e<\x08\x11\x08\x11\x08\x19\x08\x1d\x08\x17\x08\x10\x0eI\x0eM\x0e\r\x08\t\x08\t\n/\x08\x14\x04\x01\x08\x02\x08\x1b\x08\x06\n\x19\x08\x1f\x08\x1b\x12#\x08\x07\x08/'
+ 25           0 LOAD_CONST               0 ('Core implementation of import.\n\nThis module is NOT meant to be directly imported! It has been designed such\nthat it can be bootstrapped into Python as the implementation of import. As\nsuch it requires the injection of specific modules and attributes in order to\nwork. One should use importlib as the public-facing version of this module.\n\n')
               2 STORE_NAME               0 (__doc__)
-
- 25           4 LOAD_CONST               1 (None)
+              4 LOAD_CONST               1 (None)
               6 STORE_GLOBAL             1 (_bootstrap_external)
 

class:
-<code object _ModuleLock at 0x7f053ab61db0, file "Lib/importlib/_bootstrap.py", line 51>
+<code object _ModuleLock at 0x7fdefc61c580, file "Lib/importlib/_bootstrap.py", line 51>
 filename Lib/importlib/_bootstrap.py
 firstlineno: 51
-lnotab: b'\x08\x04\x04\x02\x08\x08\x08\x0c\x08\x19\x08\r'
+lnotab: b'\x0c\x06\x08\x08\x08\x0c\x08\x19\x08\r'
  51           0 LOAD_NAME                0 (__name__)
               2 STORE_NAME               1 (__module__)
               4 LOAD_CONST               0 ('_ModuleLock')
               6 STORE_NAME               2 (__qualname__)
-
- 55           8 LOAD_CONST               1 ('A recursive lock implementation which is able to detect deadlocks\n    (e.g. thread 1 trying to take locks A then B, and thread 2 trying to\n    take locks B then A).\n    ')
+              8 LOAD_CONST               1 ('A recursive lock implementation which is able to detect deadlocks\n    (e.g. thread 1 trying to take locks A then B, and thread 2 trying to\n    take locks B then A).\n    ')
              10 STORE_NAME               3 (__doc__)

[methane]

So what's new entry may be:

+* ``Module``, ``FunctionDef``, ``AsyncFunctionDef``, and
+ ``ClassDef`` AST nodes now have a new ``docstring`` attribute.
+ The first statement in their body is not considered as a docstring anymore.
+ This affects ``co_firstlineno`` and ``co_lnotab`` attribute of code object
+ for module and class.
+ (Contributed by Eugene Toder and INADA Naoki in :issue:`29463`.)
+