Different error messages for same error - invalid string prefixes

[lysnikolaou]

BPO	40246
Nosy	@gvanrossum, @terryjreedy, @vstinner, @ericvsmith, @encukou, @serhiy-storchaka, @hroncok, @lysnikolaou, @pablogsal, @aeros
PRs	bpo-40246: Report a different error message for invalid string prefixes #19476 bpo-40246: Revert reporting of invalid string prefixes #19888 [3.9] bpo-40246: Fix test_fstring when run with the old parser #20402

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-05-26.00:12:34.805>
created_at = <Date 2020-04-10.18:05:58.302>
labels = ['interpreter-core', 'type-bug', '3.9']
title = 'Different error messages for same error - invalid string prefixes'
updated_at = <Date 2020-05-26.00:12:34.804>
user = 'https://github.com/lysnikolaou'

bugs.python.org fields:

activity = <Date 2020-05-26.00:12:34.804>
actor = 'lys.nikolaou'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-26.00:12:34.805>
closer = 'lys.nikolaou'
components = ['Interpreter Core']
creation = <Date 2020-04-10.18:05:58.302>
creator = 'lys.nikolaou'
dependencies = []
files = []
hgrepos = []
issue_num = 40246
keywords = ['patch']
message_count = 41.0
messages = ['366143', '366146', '366164', '366165', '366187', '366197', '366202', '366215', '366260', '367586', '367587', '367588', '367591', '367592', '367597', '367599', '367611', '367628', '367649', '367658', '367660', '367664', '367672', '367689', '367690', '367691', '367693', '367698', '367977', '368038', '368039', '368040', '368045', '368062', '368191', '368919', '368920', '368925', '369077', '369217', '369931']
nosy_count = 10.0
nosy_names = ['gvanrossum', 'terry.reedy', 'vstinner', 'eric.smith', 'petr.viktorin', 'serhiy.storchaka', 'hroncok', 'lys.nikolaou', 'pablogsal', 'aeros']
pr_nums = ['19476', '19888', '20402']
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40246'
versions = ['Python 3.9']

[lysnikolaou]

While testing pegen, we found this out:

Python 3.9.0a5+ (heads/pegen:502dfb719e, Apr 10 2020, 20:57:05) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> fur''
  File "<stdin>", line 1
    fur''
       ^
SyntaxError: invalid syntax
>>> eval("fur''")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    fur''
       ^
SyntaxError: unexpected EOF while parsing

[lysnikolaou]

After some investigating, I found out that this is happening because when we execute fur'' in the interactive interpreter there is an implicit newline, which means that EOF does not get reached.

OTOH there is no newline when passing it as a string inside an eval call. After the tokenizer encounters the first quote, it calls tok_nextc twice more, in order to differentiate between a single-quoted and a triple-quoted string, thus reaching EOF and proapgating an EOF error up through the perrdetail struct.

[gvanrossum]

So a slightly shorter example uses ru''. This is an error because you can't combine the r prefix and the u prefix (in fact you can't combine anything with the r prefix).

I declare that this is a bug to report EOF here and the code should *first* check for a valid combination (e.g. fr'' or rf'') and only *then* try to distinguish between single and triple quotes.

[lysnikolaou]

At the moment, if the prefix is invalid, it gets tokenized as a NAME node and the string following is tokenized as a STRING node with no prefix.

╰─ echo "ur''" | ./python -m tokenize
1,0-1,2: NAME 'ur'
1,2-1,4: STRING "''"
1,4-1,5: NEWLINE '\n'
2,0-2,0: ENDMARKER ''

╰─ echo "f''" | ./python -m tokenize
1,0-1,3: STRING "f''"
1,3-1,4: NEWLINE '\n'
2,0-2,0: ENDMARKER ''

[gvanrossum]

But that is the tokenize.py module. It does not have 100% the same behavior as the builtin tokenizer.c module. In this case it probably doesn't.

[lysnikolaou]

In my understanding of the C code that's what the C tokenizer is doing as well.

Here's the relevant snippet of the tokenizer (

cpython/Parser/tokenizer.c

Line 1358 in 4b222c9

if (is_potential_identifier_start(c)) {

): When the tokenizer sees a valid identifier start, it goes into a loop that checks for a valid combination of string prefixes. If the combination is valid and it sees a quote directly after that, it goto's to the STRING-handling code. If not, then it breaks out of the loop and returns a NAME node.

Am I missing something?

[lysnikolaou]

I have working code that checks if there is a quotation mark right after an identifier. Here is an example:

╰─ ./python
Python 3.9.0a5+ (heads/pegen-dirty:502dfb719e, Apr 11 2020, 14:43:12) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ur''
  File "<stdin>", line 1
    ur''
     ^
SyntaxError: invalid string prefix

One observation about this is that it has precedence over an EOL error:

>>> ur'
  File "<stdin>", line 1
    ur'
     ^
SyntaxError: invalid string prefix

Would that be acceptable?

[gvanrossum]

yes

On Sat, Apr 11, 2020 at 04:46 Lysandros Nikolaou <report@bugs.python.org>
wrote:

Lysandros Nikolaou <lisandrosnik@gmail.com> added the comment:

I have working code that checks if there is a quotation mark right after
an identifier. Here is an example:

╰─ ./python
Python 3.9.0a5+ (heads/pegen-dirty:502dfb719e, Apr 11 2020, 14:43:12)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ur''
File "<stdin>", line 1
ur''
^
SyntaxError: invalid string prefix

One observation about this is that it has precedence over an EOL error:

>>> ur'
File "<stdin>", line 1
ur'
^
SyntaxError: invalid string prefix

Would that be acceptable?

----------

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue40246\>

---

--
--Guido (mobile)

[pablogsal]

New changeset 41d5b94 by Lysandros Nikolaou in branch 'master':
bpo-40246: Report a better error message for invalid string prefixes (GH-19476)
41d5b94