C extension naming doesn't take bitness into account

[pitrou]

BPO	22980
Nosy	@malemburg, @warsaw, @doko42, @ncoghlan, @pitrou, @larryhastings, @tjguk, @ned-deily, @ericsnowcurrently, @berkerpeksag, @zware, @1st1, @zooba, @dstufft
Files	22980_windows.patch 22980_2.patch: Patch with version in tag abi_bitness.patch ma.diff: alternate patch ma2.diff larry.whatsnew35.ext.module.suffix.diff.1.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-09-12.16:13:19.053>
created_at = <Date 2014-12-02.14:35:25.510>
labels = ['deferred-blocker', 'type-feature']
title = "C extension naming doesn't take bitness into account"
updated_at = <Date 2017-01-14.07:25:16.112>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2017-01-14.07:25:16.112>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2015-09-12.16:13:19.053>
closer = 'larry'
components = []
creation = <Date 2014-12-02.14:35:25.510>
creator = 'pitrou'
dependencies = []
files = ['37379', '37384', '38155', '38964', '39027', '40420']
hgrepos = []
issue_num = 22980
keywords = ['patch']
message_count = 96.0
messages = ['232000', '232001', '232002', '232003', '232008', '232014', '232016', '232026', '232030', '232031', '232033', '232034', '232036', '232037', '232038', '232039', '232040', '232041', '232042', '232043', '232044', '232045', '232047', '232052', '232063', '232065', '232251', '232256', '232266', '232268', '232270', '232272', '232277', '232297', '232710', '232711', '232729', '232740', '232759', '232760', '232764', '232765', '232807', '233822', '236109', '236111', '237558', '237559', '240516', '240548', '240746', '240748', '240750', '240761', '240762', '240769', '240775', '241065', '241127', '241139', '241141', '241159', '241167', '241183', '241211', '241223', '241228', '241230', '241231', '241236', '241242', '241248', '241249', '241250', '241251', '241252', '241253', '241254', '241283', '241285', '241287', '241291', '241512', '241513', '246247', '249725', '250045', '250299', '250302', '250308', '250341', '250424', '250425', '250431', '250524', '285464']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'doko', 'ncoghlan', 'pitrou', 'larry', 'tim.golden', 'ned.deily', 'Arfrever', 'python-dev', 'eric.snow', 'berker.peksag', 'zach.ware', 'yselivanov', 'steve.dower', 'dstufft']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22980'
versions = ['Python 3.5']

[pitrou]

Currently, C extensions are named something like "_helperlib.cpython-34dm.so". This doesn't take into account the bitness of the interpreter (32- vs. 64-bit), which makes it awkward to use the same working copy with two different interpreters (you have to rebuild everything each time you switch bitnesses).

Worse, under Windows it seems ABI tags aren't even used, giving generic names such as "_helperlib.pyd". Why is that?

[vstinner]

See also the PEP-3149.

[pitrou]

PEP-3149 says """It is not currently clear that the facilities in this PEP are even useful for Windows""". Well, it seems I have found a use for it :-)

[pitrou]

Ideally, we would use distutils.util.get_platform(). However, there are two cases where it relies on other modules:

• the re module under CygWin
• the sysconfig and _osx_support under OS X

Of course, ideally we should be able to hardcode this into the compiled CPython executable...

[pitrou]

As a side-note, it is interesting to note that Python currently wrongly identifies 32-bit builds under 64-bit Linux:

Python 3.5.0a0 (default:64a54f0c87d7, Nov  2 2014, 17:18:13) 
[GCC 4.9.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys, os, sysconfig
>>> sys.maxsize
2147483647
>>> os.uname()
posix.uname_result(sysname='Linux', nodename='fsol', release='3.16.0-25-generic', version='#33-Ubuntu SMP Tue Nov 4 12:06:54 UTC 2014', machine='x86_64')
>>> sysconfig.get_platform()
'linux-x86_64'

AFAIU, sysconfig.get_platform() (or its sibling distutils.util.get_platform()) is used for the naming of binary distributions...

[pitrou]

The MULTIARCH variable can help at least under Linux:

>>> import sysconfig
>>> sysconfig.get_platform()
'linux-x86_64'
>>> sysconfig.get_config_var('MULTIARCH')
'i386-linux-gnu'

[vstinner]

There is also platform.architecture(). I don't like its implementation, it relies on the external file program :-(

[zooba]

I'm very much in favor of adding this for .pyds on Windows.

I assume the hard part will be getting the details for Linux (doesn't bitness have to be compiled in there? For Windows it can be determined at compile-time...), but preferring an extension with the ABI tag and falling back on one without seems easy enough.

(Would/could this also work for .py files? So a 2.7/3.x or Jython/CPython/IronPython package could include tags in pure-Python code files?)

[malemburg]

Note that there's a difference between the platform's architecture (which is what get_platform() returns) and the pointer size of the currently running Python executable.

On 64-bit Linux, it's rather rare to have an application built as 32-bit executable. On 64-bit Windows, it's rather common to have 32-bit applications running.

The best way to determine 32-bit vs. 64-bit is by using the struct module:

    # Determine bitsize used by Python (not necessarily the same as
    # the one used by the platform)
    import struct
    bits = struct.calcsize('P') * 8

This should be portable across all platforms and will always refer to the pointer size of the currently running Python executable.

[pitrou]

Note that there's a difference between the platform's architecture

Yes, that's pointed out above.

[pitrou]

Nothing new should be necessary for pyc files under Windows:

Python 3.4.2 |Continuum Analytics, Inc.| (default, Oct 22 2014, 11:51:45) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.implementation.cache_tag
'cpython-34'

The problem is with C extensions:

>>> import _imp
>>> _imp.extension_suffixes()
['.pyd']

Compare with Linux:

>>> import _imp
>>> _imp.extension_suffixes()
['.cpython-35dm.so', '.abi3.so', '.so']

[pitrou]

I assume the hard part will be getting the details for Linux (doesn't bitness
have to be compiled in there? For Windows it can be determined at compile-
time...), but preferring an extension with the ABI tag and falling back on
one without seems easy enough.

Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.

[zooba]

I was more interested in source file resolution than bytecode caching. If Python 3.5 would prefer "spam.cpython-35.py" or "spam.cpython-3.py" over "spam.py" and Python 2 preferred "spam.py", then I can more easily separate the code that won't parse in the alternative.

Happy to be told it's unrelated and I should raise it separately, but from my POV resolving .pyd filenames looks very similar to resolving .py files.

[malemburg]

On 02.12.2014 19:02, Antoine Pitrou wrote:

Sticking to bitness should be easy (although I wonder if it would be desirable for platforms with fat binaries - Ned?). If we can go the extra mile and include platform identification all the better, of course.

I hear the "can of worms" alarm ringing :-)

Seriously, I think that putting platform infos into the file name
is bound to cause more trouble than it tries to solve. Fat builds
leave the decision to the linker, which is a good method and avoids
the file name clashes.

I think we should only focus on platforms where fat builds are
uncommon, while at the same time you do have to support multiple
architectures, like e.g. Windows:

http://en.wikipedia.org/wiki/Fat_binary

Note that on Linux, 32-bit and 64-bit versions are typically placed
into different directory trees:

http://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

so I'm not sure whether it's a real problem on Linux.