ModuleFinder.load_module skips incorrect number of bytes in pyc files

[bkabrda]

BPO	20778
Nosy	@brettcannon, @birkenfeld, @bitdancer, @ericsnowcurrently, @takluyver
Files	fix-bytes-skipped-in-load_module.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2014-02-28.15:56:19.335>
created_at = <Date 2014-02-26.09:49:14.733>
labels = ['library']
title = 'ModuleFinder.load_module skips incorrect number of bytes in pyc files'
updated_at = <Date 2014-03-28.02:11:54.737>
user = 'https://bugs.python.org/bkabrda'

bugs.python.org fields:

activity = <Date 2014-03-28.02:11:54.737>
actor = 'takluyver'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2014-02-28.15:56:19.335>
closer = 'bkabrda'
components = ['Library (Lib)']
creation = <Date 2014-02-26.09:49:14.733>
creator = 'bkabrda'
dependencies = []
files = ['34226']
hgrepos = []
issue_num = 20778
keywords = ['patch', '3.3regression']
message_count = 14.0
messages = ['212244', '212256', '212259', '212260', '212262', '212265', '212268', '212274', '212276', '212447', '212448', '212449', '212450', '215013']
nosy_count = 8.0
nosy_names = ['brett.cannon', 'georg.brandl', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow', 'takluyver', 'bkabrda']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue20778'
versions = ['Python 3.3', 'Python 3.4']

[bkabrda]

ModuleFinder.load_module currently only skips 8 bytes before trying to marshal.load the rest of the file, but it should skip 12 since 3.3 (magic, date, file size). I'm attaching a patch with test case.

BTW this was very painful to find out, since I couldn't find any reference to written pyc files - am I searching wrong or is this really not documented anywhere?

(Note, that this was originally reported at https://bugzilla.redhat.com/show_bug.cgi?id=1060338).

[brettcannon]

It's not documented because the format of .pyc files is considered an internal implementation detail.

[brettcannon]

Probably want to make sure that modulefinder uses importlib._bootstrap._validate_bytecode_header() to do the parsing.

[bitdancer]

Since modulefinder is used by freeze tools (notably cx_Freeze, which seems to be the most popular currently), should this be considered a release blocker?

[brettcannon]

It's been broken since Python 3.3 so this is not a 3.3 regression.

[bitdancer]

Right. I'm asking if it should be a release blocker for the next 3.3, too :)

I'm not saying it should be, just raising the question. It's in my mind because I'm currently using cx_Freeze in a project for a client. It could have affected me, since I was going to release using 3.3, but as it turns out I have to use 2.7 because of a non-ported dependency that I don't have time to port myself.

[bitdancer]

Oops, didn't mean to remove the keyword.

[brettcannon]

Don't know why this is any more special of a bug because it influences cx_freeze compared to any other bug that influences a popular project. I mean I'm not going to stop you from making it a blocker but I'm also not going to rush to fix it myself either.

[bitdancer]

Well, because of the fact that freeze tools are used to distribute programs on the Windows platform. But, given that it hasn't been reported before and has been a problem since 3.3, it seems like there is no rush.

[python-dev]

New changeset 432cb56db05d by Brett Cannon in branch '3.3':
Issue bpo-20778: Fix modulefinder to work with bytecode-only modules.
http://hg.python.org/cpython/rev/432cb56db05d

New changeset b6e999c8907c by Brett Cannon in branch 'default':
merge for issue bpo-20778
http://hg.python.org/cpython/rev/b6e999c8907c

[brettcannon]

Fixed in Python 3.3.6 and 3.4.1.

Bohuslav, could you sign the contributor agreement at http://www.python.org/psf/contrib/contrib-form/ ? While I didn't directly use your patch this time I noticed you are already in Misc/ACKS so it would be helpful if you could sign the document.