PEP 3131 -- Supporting Non-ASCII Identifiers

[holymonson]

Python3 already support non-ASCII identifiers, pep-3131. But yapf fails to parse.

See example below, 中 is a non-ASCII token.

$ echo '中=1' | yapf
Traceback (most recent call last):
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
    ilabel = self.classify(type, value, context)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/yapf_api.py", line 129, in FormatCode
    tree = pytree_utils.ParseCodeToTree(unformatted_source)
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
    tree = parser_driver.parse_string(code, debug=False)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
    return self.parse_tokens(tokens, debug)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
    if p.addtoken(type, value, (prefix, start)):
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
    ilabel = self.classify(type, value, context)
  File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
    raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda3/bin/yapf", line 10, in <module>
    sys.exit(run_main())
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/__init__.py", line 313, in run_main
    sys.exit(main(sys.argv))
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/__init__.py", line 183, in main
    verify=args.verify)
  File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/yapf_api.py", line 131, in FormatCode
    raise parse.ParseError(filename + ': ' + e.message)
AttributeError: 'ParseError' object has no attribute 'message'```

[kamahen]

Strange. Here's some code I wrote for testing lib2to3's encoding utility ... yapf handles it fine. So, maybe the problem is with a one-line program?

import io
from lib2to3.pgen2 import tokenize
s='網目錦蛇=1'
sb = s.encode('utf8')
sb_f = io.BytesIO(sb)
print( tokenize.detect_encoding(sb_f.readline) )

When run, this outputs:

('utf-8', [b'\xe7\xb6\xb2\xe7\x9b\xae\xe9\x8c\xa6\xe8\x9b\x87=1'])

and yapf reformats the code to:

import io
from lib2to3.pgen2 import tokenize
s = '網目錦蛇=1'
sb = s.encode('utf8')
sb_f = io.BytesIO(sb)
print(tokenize.detect_encoding(sb_f.readline))

[holymonson]

maybe the problem is with a one-line program?

No, this bug is found when parsing file, the one-line program is just a sample.

Below example is clearer.

from yapf.yapflib.pytree_utils import ParseCodeToTree

ParseCodeToTree('a = 1\n')
# Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'a'), Leaf(22, '='), Leaf(2, '1')]), Leaf(4, '\n')]), Leaf(0, '')])

ParseCodeToTree('中 = 1\n')
# Traceback (most recent call last):
#   File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 115, in ParseCodeToTree
#     tree = parser_driver.parse_string(code, debug=False)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
#     return self.parse_tokens(tokens, debug)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
#     if p.addtoken(type, value, (prefix, start)):
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
#     ilabel = self.classify(type, value, context)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
#     raise ParseError("bad token", type, value, context)
# lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))
# 
# During handling of the above exception, another exception occurred:
# 
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/usr/local/miniconda3/lib/python3.7/site-packages/yapf/yapflib/pytree_utils.py", line 121, in ParseCodeToTree
#     tree = parser_driver.parse_string(code, debug=False)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 104, in parse_string
#     return self.parse_tokens(tokens, debug)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/driver.py", line 72, in parse_tokens
#     if p.addtoken(type, value, (prefix, start)):
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 116, in addtoken
#     ilabel = self.classify(type, value, context)
#   File "/usr/local/miniconda3/lib/python3.7/lib2to3/pgen2/parse.py", line 172, in classify
#     raise ParseError("bad token", type, value, context)
# lib2to3.pgen2.parse.ParseError: bad token: type=58, value='中', context=('', (1, 0))

[kamahen]

Which version of Python are you using? Your sample program worked for me with Python 3.5, 3.6, 3.7 (I don't have anything older installed).

[holymonson]

Which version of Python are you using?

sys.version
# '3.7.0 (default, Jun 28 2018, 07:39:16) \n[Clang 4.0.1 (tags/RELEASE_401/final)]'
yapf.__version__
# '0.22.0'

[kamahen]

I have the August 6 version of Python, but that shouldn't matter because I got similar results with 3.5 and 3.6.

This is how I ran my test:

$ cd src/yapf
$ git remote show origin
* remote origin
  Fetch URL: https://github.com/google/yapf.git
  Push  URL: https://github.com/google/yapf.git
  HEAD branch: master
  Remote branches:
    master                     tracked
    no-split-before-dict-value tracked
    style-config               tracked
    testrun-discovery          tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)
$ python3.7 -m yapf /tmp/e.py
from yapf.yapflib.pytree_utils import ParseCodeToTree

ParseCodeToTree('a = 1\n')
# Node(file_input, [Node(simple_stmt, [Node(expr_stmt, [Leaf(1, 'a'), Leaf(22, '='), Leaf(2, '1')]), Leaf(4, '\n')]), Leaf(0, '')])

ParseCodeToTree('中 = 1\n')
$ find . -type f | xargs grep __version

./yapftests/main_test.py:      version = 'yapf {}\n'.format(yapf.__version__)
./setup.py:      version=yapf.__version__,
./yapf/__init__.py:__version__ = '0.22.0'
./yapf/__init__.py:    print('yapf {}'.format(__version__))

[holymonson]

tokenize recognizes 'a' and 'a中' as NAME, but '中' as ERRORTOKEN.

$ echo 'a = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	NAME	'a'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

$ echo '中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,1:	ERRORTOKEN	'中'
1,2-1,3:	OP	'='
1,4-1,5:	NUMBER	'1'
1,5-1,6:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

$ echo 'a中 = 1' | python3.7 -m lib2to3.pgen2.tokenize
1,0-1,2:	NAME	'a中'
1,3-1,4:	OP	'='
1,5-1,6:	NUMBER	'1'
1,6-1,7:	NEWLINE	'\n'
2,0-2,0:	ENDMARKER	''

It seems should blame tokenize.py#L504 , only those token initialized with namechars = string.ascii_letters + '_' can be recognized as NAME, while Python 3 Interpreter dosen't restrict that.

                elif initial in namechars:                 # ordinary name

However, lib2to3.pgen2 seems a parser aiming for converting Python 2 code to Python 3, not parsing Python 3 code (I'm not sure). And it may retire when the End-Of-Life date of Python 2 (2020-01-01) comes.

---

@kamahen , I don't know why your python works, is there a parser other than lib2to3.pgen2.tokenize?

[kamahen]

@holymonson - I can reproduce your results.

I agree -- it looks like a bug in lib2to3.pgen2.tokenize.

[holymonson]

xfer python/cpython#8950