Double coding cookie

[serhiy-storchaka]

BPO	26581
Nosy	@malemburg, @gvanrossum, @loewis, @vstinner, @serhiy-storchaka
Files	tokenize_double_coding.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-03-20.21:52:08.611>
created_at = <Date 2016-03-17.12:00:36.043>
labels = ['interpreter-core', 'type-bug', 'library']
title = 'Double coding cookie'
updated_at = <Date 2016-03-20.21:52:08.611>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2016-03-20.21:52:08.611>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-03-20.21:52:08.611>
closer = 'serhiy.storchaka'
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2016-03-17.12:00:36.043>
creator = 'serhiy.storchaka'
dependencies = []
files = ['42185']
hgrepos = []
issue_num = 26581
keywords = ['patch']
message_count = 8.0
messages = ['261909', '262051', '262052', '262053', '262054', '262089', '262090', '262092']
nosy_count = 6.0
nosy_names = ['lemburg', 'gvanrossum', 'loewis', 'vstinner', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26581'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[serhiy-storchaka]

When Python source file contains double coding cookies on different lines, the first wins. When it contains double coding cookies on the same line, the last wins.

PEP-263 was sufficiently vague about this. Now this is clarified (22490711c870). The first coding cookie should always win.

Proposed patch fixes Python tokenizer, the tokenize module, and other places. Tests are taken from bpo-25643.

[serhiy-storchaka]

I just tested with Emacs, and it looks that when specify different codings on two different lines, the first coding wins, but when specify different codings on the same line, the last coding wins.

Therefore current CPython behavior can be correct, and the regular expression in PEP-263 should be changed to use greedy repetition.

[gvanrossum]

Do you have write permission to the PEP? Just update it.

[serhiy-storchaka]

Yes, I have. But I were not sure what behavior should be correct in Python. On one side, always choosing the first declaration (on the same or on different lines) looks more consistent. On other side, current behavior was in CPython from the initial implementing PEP-263 in bpo-526840 and it matches Emacs behavior (if I understand this correctly).

I can update the regular expression, but may be this obscure corner case needs the verbal explanation.

[gvanrossum]

Right. Please go ahead with both. I am fine with defining the current
behavior correct.

--Guido (mobile)
On Mar 19, 2016 9:37 AM, "Serhiy Storchaka" <report@bugs.python.org> wrote:

Serhiy Storchaka added the comment:

Yes, I have. But I were not sure what behavior should be correct in
Python. On one side, always choosing the first declaration (on the same or
on different lines) looks more consistent. On other side, current behavior
was in CPython from the initial implementing PEP-263 in bpo-526840 and it
matches Emacs behavior (if I understand this correctly).

I can update the regular expression, but may be this obscure corner case
needs the verbal explanation.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue26581\>

---

[serhiy-storchaka]

Ah, I made a mistake! In 2.7 the first coding on the same line wins. And that behavior was from start. Regression was unintentionally introduced in bpo-18470.

Thus *there is* a bug in Python 3. PEP-263 doesn't need more changes, but Python tokenizer and related tools do.

Sorry for misleading.

[python-dev]

New changeset 23a7481eafd4 by Serhiy Storchaka in branch 'default':
Issues bpo-25643, bpo-26581: Added new tests for detecting Python source code encoding.
https://hg.python.org/cpython/rev/23a7481eafd4

[python-dev]

New changeset 1c44cea2ea8f by Serhiy Storchaka in branch '3.5':
Issue bpo-26581: Use the first coding cookie on a line, not the last one.
https://hg.python.org/cpython/rev/1c44cea2ea8f

New changeset 8506d127d482 by Serhiy Storchaka in branch '2.7':
Issue bpo-26581: Use the first coding cookie on a line, not the last one.
https://hg.python.org/cpython/rev/8506d127d482

New changeset e86cd4a872b8 by Serhiy Storchaka in branch 'default':
Issue bpo-26581: Use the first coding cookie on a line, not the last one.
https://hg.python.org/cpython/rev/e86cd4a872b8