Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE

[jafo]

BPO	1294959
Nosy	@malemburg, @warsaw, @doko42, @jcea, @abalkin, @vstinner, @matejcik, @tarekziade, @mcepl, @merwok, @akitada, @agbuckley, @encukou, @mgorny, @grimreaper, @stratakis, @hroncok, @miss-islington, @pillarsdotnet
PRs	bpo-1294959 - better support for systems with /usr/lib64 #3698 [WIP] bpo-1294959: Better support for systems with /usr/lib64 #11755 bpo-1294959: Add --with-platlibdir to configure #18381 bpo-21016: pydoc and trace use sysconfig #18476 [3.8] bpo-21016: pydoc and trace use sysconfig (GH-18476) #18482 [3.7] bpo-21016: pydoc and trace use sysconfig (GH-18476) #18483 [WIP] bpo-1294959: setup.py uses sys.platlibdir #18917 bpo-1294959: Fix typo for new attribute platlibdir. #18960 bpo-1294959: Try to clarify the meaning of platlibdir #20332 [3.9] bpo-1294959: Try to clarify the meaning of platlibdir (GH-20332) #20497 [3.9] bpo-40854: Allow overriding sys.platlibdir via PYTHONPLATLIBDIR env-var (GH-20605) #20725
Files	python-2.3.4-lib64-regex.patch: One of the Fedora patches. python-2.4.1-lib64.patch: Another of the Fedora patches. japanese-codecs-lib64.patch: The third obviously lib64 Fedora patch. Python-2.6.2-multilib.patch: SUSE patch for the lib64 issue Python-3.3.0b2-multilib.patch: SUSE patch in python3 python-3.6.0-multilib-new.patch lib64_tests.py lib64_tests-2.py lib64_tests-3.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-13.16:04:36.444>
created_at = <Date 2005-09-19.03:05:28.000>
labels = ['type-bug', '3.9', 'build']
title = 'Add sys.platlibdir and configure --with-platlibdir to use /usr/lib64 on Fedora and SuSE'
updated_at = <Date 2021-08-10.16:25:34.973>
user = 'https://bugs.python.org/jafo'

bugs.python.org fields:

activity = <Date 2021-08-10.16:25:34.973>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-13.16:04:36.444>
closer = 'vstinner'
components = ['Build']
creation = <Date 2005-09-19.03:05:28.000>
creator = 'jafo'
dependencies = []
files = ['1777', '1778', '1779', '14726', '32529', '46301', '48892', '48962', '48966']
hgrepos = ['12']
issue_num = 1294959
keywords = ['patch']
message_count = 96.0
messages = ['26314', '59310', '64592', '64593', '81099', '82929', '91559', '91769', '91781', '92541', '94941', '94982', '94986', '94989', '132424', '132503', '132505', '132506', '132511', '132535', '132537', '132538', '132540', '132542', '132544', '132546', '160482', '160671', '160691', '202341', '202343', '202354', '202359', '202365', '202366', '202367', '202368', '202378', '222643', '242608', '285126', '285222', '285556', '285737', '285738', '307231', '308555', '361844', '361847', '361849', '361850', '361851', '361853', '361854', '361882', '361888', '361890', '361893', '361894', '361897', '361899', '361900', '361901', '361902', '361903', '361907', '361909', '362324', '363288', '363441', '363728', '363811', '363816', '363894', '363896', '363900', '364024', '364104', '364105', '369323', '369326', '369327', '369352', '369354', '369360', '369365', '369368', '369467', '369744', '370238', '370242', '370248', '370704', '371008', '382964', '399339']
nosy_count = 26.0
nosy_names = ['lemburg', 'barry', 'doko', 'jafo', 'jcea', 'belopolsky', 'vstinner', 'matejcik', 'tarek', 'ivazquez', 'mcepl', 'eric.araujo', 'Arfrever', 'akitada', 'andybuckley', 'petr.viktorin', 'catalin.iacob', 'mgorny', 'python-dev', 'piotr.dobrogost', 'eitan.adler', 'cstratak', 'hroncok', 'miss-islington', 'pillarsdotnet', 'carlos.velasco']
pr_nums = ['3698', '11755', '18381', '18476', '18482', '18483', '18917', '18960', '20332', '20497', '20725']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1294959'
versions = ['Python 3.9']

[jafo]

This is something that's becoming more and more of a
problem as more people get systems that run both 32-bit
and 64-bit code. There are patches (926209 858809) in
the tracker to resolve this issue, I don't know exactly
what state they are in. They seem fairly old, one is
closed and one is still open.

The Fedora RPMs include the following patches for lib64
(attached).

Any thoughts on what we need to do to allow building
lib64 as a part of the standard release? Or do we just
want to wait for these terrible transition days to end
and rely on 32+64 packagers to deal with it?

Here's a short run-down of the situation: Some systems
can run both 32 and 64 bit software. On these systems
you *CAN* install a native 64-bit tool-chain, but some
people set up both 32 and 64 bit software on these
systems for compatibility. You can do 32+64 in a
couple of ways, one is to set up a "chroot" environment
for the secondary (typically 32-bit) set of libraries
and programs. The other is to run the system with
64-bit libraries in /usr/lib64 and 32-bit in /usr/lib
(because most software that needs compatibility would
be running against /usr/lib, not /usr/lib32).

It's a kludge to be sure, but if you need any legacy
software that you do not have source for, this is a way
of getting 64-bits while still retaining the ability to
run 32-bit This is not a problem for systems which
only run 64-bit code, nor is it a problem for these
32+64 systems which are running only 64-bit
distributions on them.

Thoughts?
Sean

[tiran]

The problem with 64 bit builds should be discussed before we release
2.6. Another bug day topic.

[abalkin]

Can someone update the priority so that this is looked at before the 2.6
release?

[abalkin]

Placing the entire library tree in /usr/lib64 is wasteful on dual
32/64bit installation, but placing just the C modules there is contrary
to python import logic and may cause problems to relative imports.

I have suggested what I believed was a workable solution: have 64-bit
python search lib64-dynload subdirectories instead of lib-dynload.

See http://mail.python.org/pipermail/python-dev/2007-April/072653.html

Currently $(prefix)/pythonX.Y/lib-dynload is inserted in the sys.path,
but I think it would be better to handle this inside the importer in a
way similar to how the importer looks for both foo.so and foomodule.so
when importing foo. This would allow submodules and user modules treated
the same way.

[akitada]

Similar problem report: http://bugs.python.org/issue1019715

[akitada]

3rd party C modules are put in site-packages,
so just having importer of 64-bit python look at lib64-dynload is not
enough for solving this.

To work around this problem, I did some hacks on my local Python to look
at lib and lib64. It worked, but just as belopolsky said, this is
wasteful and ugly.

[matejcik]

for completenes, here's a patch that's in use in SUSE. it's advantage
over Fedora's is that it works on both 32bit and 64bit installs

[doko42]

both patches assume that everybody uses lib64 for 64bit libs, which is
not true for Debian/Ubuntu. Even the FHS doesn't mandate the use of lib64.

[matejcik]

well in our patch, at least, the directory is governed by sys.lib which
is defined through configure.
i don't understand the configure language well enough, but i'd assume
that making it parametrized isn't too hard?

[akitada]

I think this is duplicate of bpo-858809.

[merwok]

If I understand correctly, using lib32 or lib64 is a kludge. Debian
and Ubuntu want to come up with a better way to do this:
http://wiki.debian.org/ReleaseGoals/MultiArch
https://wiki.ubuntu.com/MultiarchSpec

Kind regards.

[matejcik]

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

[malemburg]

jan matejek wrote:

jan matejek <jmatejek@suse.cz> added the comment:

thanks for the info; however, it still means that python needs to learn
to live in places other than "/usr/lib"

The main problem is that Python's configuration system is not
geared up to having the lib directories for platform dependent
and platform independent parts use different names.

It currently only supports using different path *prefixes* for
such setups (--prefix and --exec-prefix), e.g. /usr and /usr64
would work just fine. It doesn't follow --libdir.

Note that Tarek is currently working on a cleanup of the
installation schemes (see distutils/commands/install.py,
distutils/sysconfig.py and site.py) which will then also
be used by site.py to setup sys.path.

A new modules will unite all these settings, so this should
be the target of any patches regarding installation and
path lookup schemes.

BTW: The "sys.lib" setting mentioned on the tracker is not standard.
The naming also doesn't look right, since the name of the "lib"
directory path component is an OS feature, not a system one that
you configure. os.lib_dir would be more appropriate.