Different error messages for same error - invalid string prefixes

@gvanrossum

BPO	40246
Nosy	@gvanrossum, @terryjreedy, @vstinner, @ericvsmith, @encukou, @serhiy-storchaka, @hroncok, @lysnikolaou, @pablogsal, @aeros
PRs	bpo-40246: Report a different error message for invalid string prefixes #19476 bpo-40246: Revert reporting of invalid string prefixes #19888 [3.9] bpo-40246: Fix test_fstring when run with the old parser #20402

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-26.00:12:34.805>
created_at = <Date 2020-04-10.18:05:58.302>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Different error messages for same error - invalid string prefixes'
updated_at = <Date 2020-05-26.00:12:34.804>
user = 'https://github.com/lysnikolaou'

bugs.python.org fields:

activity = <Date 2020-05-26.00:12:34.804>
actor = 'lys.nikolaou'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-26.00:12:34.805>
closer = 'lys.nikolaou'
components = ['Interpreter Core']
creation = <Date 2020-04-10.18:05:58.302>
creator = 'lys.nikolaou'
dependencies = []
files = []
hgrepos = []
issue_num = 40246
keywords = ['patch']
message_count = 41.0
messages = ['366143', '366146', '366164', '366165', '366187', '366197', '366202', '366215', '366260', '367586', '367587', '367588', '367591', '367592', '367597', '367599', '367611', '367628', '367649', '367658', '367660', '367664', '367672', '367689', '367690', '367691', '367693', '367698', '367977', '368038', '368039', '368040', '368045', '368062', '368191', '368919', '368920', '368925', '369077', '369217', '369931']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'terry.reedy', 'vstinner', 'eric.smith', 'petr.viktorin', 'serhiy.storchaka', 'hroncok', 'lys.nikolaou', 'pablogsal', 'aeros']
pr_nums = ['19476', '19888', '20402']
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40246'
versions = ['Python 3.9']

While testing pegen, we found this out:

Python 3.9.0a5+ (heads/pegen:502dfb719e, Apr 10 2020, 20:57:05) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> fur''
  File "<stdin>", line 1
    fur''
       ^
SyntaxError: invalid syntax
>>> eval("fur''")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    fur''
       ^
SyntaxError: unexpected EOF while parsing

After some investigating, I found out that this is happening because when we execute fur'' in the interactive interpreter there is an implicit newline, which means that EOF does not get reached.

OTOH there is no newline when passing it as a string inside an eval call. After the tokenizer encounters the first quote, it calls tok_nextc twice more, in order to differentiate between a single-quoted and a triple-quoted string, thus reaching EOF and proapgating an EOF error up through the perrdetail struct.

So a slightly shorter example uses ru''. This is an error because you can't combine the r prefix and the u prefix (in fact you can't combine anything with the r prefix).

I declare that this is a bug to report EOF here and the code should *first* check for a valid combination (e.g. fr'' or rf'') and only *then* try to distinguish between single and triple quotes.

At the moment, if the prefix is invalid, it gets tokenized as a NAME node and the string following is tokenized as a STRING node with no prefix.

╰─ echo "ur''" | ./python -m tokenize
1,0-1,2: NAME 'ur'
1,2-1,4: STRING "''"
1,4-1,5: NEWLINE '\n'
2,0-2,0: ENDMARKER ''

╰─ echo "f''" | ./python -m tokenize
1,0-1,3: STRING "f''"
1,3-1,4: NEWLINE '\n'
2,0-2,0: ENDMARKER ''

But that is the tokenize.py module. It does not have 100% the same behavior as the builtin tokenizer.c module. In this case it probably doesn't.

In my understanding of the C code that's what the C tokenizer is doing as well.

Here's the relevant snippet of the tokenizer (

cpython/Parser/tokenizer.c

Line 1358 in 4b222c9

if (is_potential_identifier_start(c)) {

): When the tokenizer sees a valid identifier start, it goes into a loop that checks for a valid combination of string prefixes. If the combination is valid and it sees a quote directly after that, it goto's to the STRING-handling code. If not, then it breaks out of the loop and returns a NAME node.

Am I missing something?

I have working code that checks if there is a quotation mark right after an identifier. Here is an example:

╰─ ./python
Python 3.9.0a5+ (heads/pegen-dirty:502dfb719e, Apr 11 2020, 14:43:12) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ur''
  File "<stdin>", line 1
    ur''
     ^
SyntaxError: invalid string prefix

One observation about this is that it has precedence over an EOL error:

>>> ur'
  File "<stdin>", line 1
    ur'
     ^
SyntaxError: invalid string prefix

Would that be acceptable?

yes

On Sat, Apr 11, 2020 at 04:46 Lysandros Nikolaou <report@bugs.python.org>
wrote:

Lysandros Nikolaou <lisandrosnik@gmail.com> added the comment:

I have working code that checks if there is a quotation mark right after
an identifier. Here is an example:

╰─ ./python
Python 3.9.0a5+ (heads/pegen-dirty:502dfb719e, Apr 11 2020, 14:43:12)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ur''
File "<stdin>", line 1
ur''
^
SyntaxError: invalid string prefix

One observation about this is that it has precedence over an EOL error:

>>> ur'
File "<stdin>", line 1
ur'
^
SyntaxError: invalid string prefix

Would that be acceptable?

----------

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue40246\>

--
--Guido (mobile)

New changeset 41d5b94 by Lysandros Nikolaou in branch 'master':
bpo-40246: Report a better error message for invalid string prefixes (GH-19476)
41d5b94

I will soon come back with what Fedora package were affected by the problem. That could give some data about how to handle this.

Not that many:

cpython itself (fixed via PR)
demjson (fixed via PR)
asn1crypto (fixed via PR)
dnf (fixed via PR)
freeipa (fixed via PR)

I gave up sending PRs at this point.

waf
weasyprint
virt-who
thrift
salt
wxpython4
rosdistro
mne
pycairo
libstoragemgmt (possibly via bundled lsm/external/xmltodict)
dput-ng
ddupdate
aubio (via bundled waf)

Lysandros:

The revert is in. Now the question is if we want to take additional action to address the original issue of this.

If someone cares of that, I suggest to open an issue in pylint, pyflakes and similar tools to emit a warning in linters.

Ok, I'm closing this, after consulting with Guido.

The following change broke test_fstring when using the old parser. I reopen the issue.

commit 846d8b2 (refs/bisect/bad)
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date: Mon May 4 14:32:18 2020 +0300

bpo-40246: Revert reporting of invalid string prefixes (GH-19888)

Due to backwards compatibility concerns regarding keywords immediately followed by a string without whitespace between them (like in `bg="#d00" if clear else"#fca"`) will fail to parse,
commit 41d5b94af44e34ac05d4cd57460ed104ccf96628 has to be reverted.

$ ./python -X oldparser -m test -v test_fstring 
(...)

======================================================================
FAIL: test_invalid_string_prefixes (test.test_fstring.TestCase) (str='BF""')
----------------------------------------------------------------------
File "<string>", line 1
BF""
^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vstinner/python/master/Lib/test/test_fstring.py", line 29, in assertAllRaise
    eval(str)
AssertionError: "unexpected EOF while parsing" does not match "invalid syntax (<string>, line 1)"
(...)

The error can be seen on the new cool AMD64 Arch Linux VintageParser 3.x, vintage is the new cool:
https://buildbot.python.org/all/#builders/648/builds/185

We should run the tests with the old parser in at least one build.

The >else"#fca"< code was added 6 years ago (commit 8b95d5e)

That was my typo in turtledemo.__main__ (similar lines before had the space) and I wish that it had been caught then.

Why did compileall in the test suite not catch this SyntaxError before 3.6.0a6 was released? Does it intentionally skip .__main__ files? Should I add a test_turtledemo file (with checks for the presence of tkinter and turtle)?

"make install" ignores compileall errors no? If I recall correctly, only an error is logged into stdlib, "make install" doesn't fail.

New changeset 6cb0ad2 by Lysandros Nikolaou in branch '3.9':
bpo-40246: Fix test_fstring when run with the old parser (GH-20402)
6cb0ad2

Navigation Menu

Repository navigation

Different error messages for same error - invalid string prefixes #84427

26 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Different error messages for same error - invalid string prefixes #84427

Description

Activity

lysnikolaou commented on Apr 10, 2020

lysnikolaou commented on Apr 10, 2020

gvanrossum commented on Apr 10, 2020

lysnikolaou commented on Apr 10, 2020

gvanrossum commented on Apr 11, 2020

lysnikolaou commented on Apr 11, 2020

lysnikolaou commented on Apr 11, 2020

gvanrossum commented on Apr 11, 2020

pablogsal commented on Apr 12, 2020

26 remaining items

hroncok commented on May 4, 2020

hroncok commented on May 4, 2020

vstinner commented on May 4, 2020

lysnikolaou commented on May 5, 2020

vstinner commented on May 15, 2020

vstinner commented on May 15, 2020

gvanrossum commented on May 15, 2020

terryjreedy commented on May 16, 2020

vstinner commented on May 18, 2020

pablogsal commented on May 26, 2020

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions