py_compile: foo.bar.py → __pycache__/foo.cpython-34.pyc

[piotr]

BPO	22966
Nosy	@warsaw, @brettcannon, @doko42, @bitdancer
Files	22966.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/warsaw'
closed_at = <Date 2014-12-02.17:12:20.035>
created_at = <Date 2014-11-29.10:16:42.752>
labels = ['type-bug', 'library']
title = 'py_compile: foo.bar.py \xe2\x86\x92 __pycache__/foo.cpython-34.pyc'
updated_at = <Date 2014-12-02.17:12:20.034>
user = 'https://bugs.python.org/piotr'

bugs.python.org fields:

activity = <Date 2014-12-02.17:12:20.034>
actor = 'barry'
assignee = 'barry'
closed = True
closed_date = <Date 2014-12-02.17:12:20.035>
closer = 'barry'
components = ['Library (Lib)']
creation = <Date 2014-11-29.10:16:42.752>
creator = 'piotr'
dependencies = []
files = ['37339']
hgrepos = []
issue_num = 22966
keywords = ['3.3regression', '3.4regression']
message_count = 10.0
messages = ['231859', '231866', '231905', '231940', '231944', '231946', '231956', '231963', '232010', '232027']
nosy_count = 7.0
nosy_names = ['barry', 'brett.cannon', 'doko', 'Arfrever', 'r.david.murray', 'piotr', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22966'
versions = ['Python 3.4', 'Python 3.5']

[piotr]

when py_compile module is asked to create .pyc file for file with invalid name, it doesn't fail or ignore such request - it creates/overwrites .pyc file for sane file name (i.e. ignores everything after dot)

$ touch foo.bar.py
$ python3.4 -m py_compile foo.bar.py
$ ls __pycache__
foo.cpython-34.pyc

Even though foo.bar.py is not a valid file name, some programmers are using such names for plugins which leads to bugs similar to https://lists.debian.org/debian-python/2014/11/msg00061.html.

I will update my scripts to not feed py_compile module with file names containing dot (and remove already generated .pyc files if such files are detected), but please do not generate them or generate .pyc files with invalid file names as well.

[piotr]

Python 3.2 generates foo.bar.cpython-32.pyc so it's a regression

[bitdancer]

And, therefore, a missing test :(. This probably snuck in when we refactored the CLI.

[doko42]

seen as well with 3.3

[doko42]

proposed patch:

diff -r 8dacb1a21793 Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py	Fri Nov 28 13:15:41 2014 +0100
+++ b/Lib/importlib/_bootstrap.py	Mon Dec 01 17:05:00 2014 +0100
@@ -453,7 +453,7 @@
     else:
         suffixes = OPTIMIZED_BYTECODE_SUFFIXES
     head, tail = _path_split(path)
-    base_filename, sep, _ = tail.partition('.')
+    base_filename, sep, _ = tail.rpartition('.')
     tag = sys.implementation.cache_tag
     if tag is None:
         raise NotImplementedError('sys.implementation.cache_tag is None')

[warsaw]

I'll take this one. I think it should be easy to add a test case, which I'll do.

[warsaw]

Not counting importlib.h, here's the diff I'm going to apply to 3.4. It passes all the existing tests and includes a new test for this behavior.

[python-dev]

New changeset 269bf37a57a1 by Barry Warsaw in branch '3.4':

• Issue bpo-22966: Fix __pycache__ pyc file name clobber when pyc_compile is
  https://hg.python.org/cpython/rev/269bf37a57a1

New changeset 3b3ba38d503a by Barry Warsaw in branch '3.4':

• Issue bpo-22966: Fix __pycache__ pyc file name clobber when pyc_compile is
  https://hg.python.org/cpython/rev/3b3ba38d503a

New changeset 25113281d543 by Barry Warsaw in branch 'default':

• Issue bpo-22966: Fix __pycache__ pyc file name clobber when pyc_compile is
  https://hg.python.org/cpython/rev/25113281d543

[brettcannon]

Apparently this broke under Windows: http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/8999/steps/test/logs/stdio

[warsaw]

I already pushed a fix.

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.4/builds/702/steps/test/logs/stdio

(although asyncio is still failing there but that should be unrelated)