AIX shared library extension modules installation broken

[DavidEdelsohn]

BPO	25825
Nosy	@pitrou, @ericvw, @vadmium, @aixtools
Files	Issue25825-patch1.txt: Patch for Makefile.pre.in Issue25825-patch2.txt: Patch to configure.ac for AIX LDSHARED

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-05.01:19:38.834>
created_at = <Date 2015-12-08.19:29:34.469>
labels = ['type-bug', 'expert-installation']
title = 'AIX shared library extension modules installation broken'
updated_at = <Date 2018-09-18.15:08:46.629>
user = 'https://bugs.python.org/DavidEdelsohn'

bugs.python.org fields:

activity = <Date 2018-09-18.15:08:46.629>
actor = 'Michael.Felt'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-05.01:19:38.834>
closer = 'martin.panter'
components = ['Installation']
creation = <Date 2015-12-08.19:29:34.469>
creator = 'David.Edelsohn'
dependencies = []
files = ['44068', '44096']
hgrepos = []
issue_num = 25825
keywords = []
message_count = 20.0
messages = ['256123', '256136', '256155', '261127', '271417', '272336', '272490', '272538', '272539', '272602', '273389', '273416', '273417', '273425', '273448', '273756', '273766', '274375', '325654', '325656']
nosy_count = 6.0
nosy_names = ['pitrou', 'ericvw', 'python-dev', 'martin.panter', 'David.Edelsohn', 'Michael.Felt']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25825'
versions = ['Python 3.5', 'Python 3.6']

[DavidEdelsohn]

AIX requires helper scripts to build Python shared extension modules. The definitions and Makefile installation rules have bitrotted.

Makefile.pre.in:

  @if [ -s [Programs/python.exp](https://github.com/python/cpython/blob/main/Programs/python.exp) -a \

except python.exp is created in Modules/python.exp, not Programs.exp

           $(INSTALL_DATA) [Programs/python.exp](https://github.com/python/cpython/blob/main/Programs/python.exp)             \
                           $(DESTDIR)$(LIBPL)/python.exp;

Post-substitution Makefile.pre:

BINLIBDEST= $(LIBDIR)/python$(VERSION)
LIBDEST= $(SCRIPTDIR)/python$(VERSION)
LIBPL= $(LIBDEST)/config-$(LDVERSION)

configure.ac:

    AIX*)
            BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
            LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"

which is forever enshrined in _sysconfigdata.py

In other words, configure sets LDSHARED to $(BINLIBDEST)/config, which does not exist in current installations. And Makefile installs the files in LIBPL, which is based on LIBDEST (prefix and exec_prefix could be different).

And the files are not installed, because the installation tests Programs/python.exp instead of Modules/python.exp.

Changing Makefile.pre.in to test Modules/python.exp is easy enough as a partial fix.

What is the preferred location to install the files so that configure.ac and Makefile.pre.in can be adjusted to match each other?

[vadmium]

It looks like the Programs/python.exp change was done accidentally in revision 88a532a31eb3 (bpo-18093), so we should be safe reverting the relevant lines back to Modules/python.exp.

The configure script currently sets

LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
AC_SUBST(LIBPL)

However there is a contradictory comment in the makefile:

# This goes into $(exec_prefix)
LIBPL= @LIBPL@

It looks like the comment came first. $(LIBPL) was changed from $(LIBDIR) [presumably using $(exec_prefix)] to use $(prefix) in r86731 (bpo-9807).

This isn’t my area of expertise, but my understanding is $(exec_prefix) is for architecture-specific files only. If python.exp is not architecture-specific, perhaps it should go in $(prefix), and the configure script (and makefile comment) needs fixing.

[DavidEdelsohn]

$(prefix) and $(exec_prefix) result in the same path on AIX, so it does not matter in practice, although the semantics are different.

# Install prefix for architecture-dependent files
exec_prefix= ${prefix}

python.exp is not architecture dependent, although it only is useful on AIX target. It is essentially equivalent to a list of symbols with ELF global, non-hidden visibility. It is less confusing if the list is co-located with the scripts that use it.

LIBPL is fine with me. configure.ac and Makefile.pre.in must match.

[aixtools]

FYI: when build and src are the same directory, there is no error message. However, when src and build are in seperate directories (e.g.,

build = '.',
src = '../src/python-2.7.11'

the following message occurs MANY times:

unable to execute '../src/python-2.7.11/Modules/ld_so_aix': No such file or directory

during configure the file
./Modules/ld_so_aix is 'created' in the build area.

root@x064:[/data/prj/aixtools/python-2.7.11]ls -l Modules/ld*aix*
-rwxr-xr-x 1 root system 6205 Mar 02 16:24 Modules/ld_so_aix

Just guessing, but a line such as:
BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"

using $(srcdir) is not correct.

Further, a find in $(srcdir) for ld_so_aix returns:
root@x064:[/data/prj/aixtools/python-2.7.11]ls -dl ../src/py*
drwxr-xr-x 17 199 1954 4096 May 23 2015 ../src/python-2.7.10
drwxr-xr-x 17 199 1954 4096 Dec 05 19:47 ../src/python-2.7.11

So, $(srcdir) seems to be wrong, by definition.

For now, I am copying $(srcdir) to $(builddir) to work-around this.

[aixtools]

hmm. needs patch. For what?

One comment says a change will be undone.

If a patch is expected for src/builddir issue:

a) I sinned - that is a new issue, and should be posted seperately.
b) i do not know autotools well enough to solve the srcdir and builddir problem.

rather than a patch i can on that is see if the same issue occurs with linux and post a new issue.

So, my question would be - was a rollback done - if yes, perhaps a comment to verify and then close this issue.

[DavidEdelsohn]

Let's start with this patch to revert the change mentioned in msg256136 to look for python.exp in Modules -- the test, the source for the file, and the location to delete the file.

A follow-up patch will fix the data in _sysconfigdata.py.

[vadmium]

Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?

[DavidEdelsohn]

Yes, please apply Patch 1 that reverts the mistaken change of revision 88a532a31eb3 . I want to work through this incrementally so that it's clear to reviewers.

[python-dev]

New changeset 58c8cae6c61a by Martin Panter in branch '3.5':
Issue bpo-25825: Fix references to Modules/python.exp
https://hg.python.org/cpython/rev/58c8cae6c61a

New changeset 4d4b5b978b7e by Martin Panter in branch 'default':
Issue bpo-25825: Merge AIX fix from 3.5
https://hg.python.org/cpython/rev/4d4b5b978b7e

[DavidEdelsohn]

The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1. The definition from configure propagates into _sysconfigdata.py.

This change will (further) break test_distutils, but the testcase is wrong on AIX. And, for AIX, a correct test result clearly does not correspond to correct behavior.

The patch affects both configure.ac and configure because the corresponding change to configure is obvious. The person who installs the patch may regenerate configure, as appropriate.

[ericvw]

Patch1 looks fine to me, though I will have to rely on you people to verify that it does what it’s supposed to. Do you want me to commit it straight away, or wait for your follow-up patch?

Independently, I have created a similar patch as well and can also verify that it does what it's suppose to do.

The second of two patches. This patch changes the definition of LDSHARED for AIX in configure to reference the matching installed location as defined in Makefile.pre.in by Patch1. [...]

I can also confirm that Patch2 is necessary as well and does what it's suppose to do, having creating this patch independently as well.

[vadmium]

It looks like the change from plain config to config-<ABI> was made in bpo-9807 (r86731, 3.2). So the patch seems reasonable to me; it is just catching up with that change.

This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.

[DavidEdelsohn]

This bug was marked for 2.7 as well. Is there anything that needs to be done for 2.7?

It would be great if both patches were applied to 2.7 also.

How does patch 2 make the test_distutils situation worse? Is there anything that should be done to improve things first, or should that just be handled independently? I see that the 2.7 and 3.5 buildbots are currently failing with errors like

distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

while 3.6 is failing with

distutils.errors.LinkError: command '/usr/local/lib/python3.6/config/ld_so_aix' failed with exit status 1

Presumably the 3.6 failure may be helped by Patch 2, since it is complaining about the plain “config” path.

Patch2 makes test_distutils worse because LDSHARED refers to a file that only exists when Python is installed. If one tests in tree, the files don't exist. This may be related to the existence of BLDSHARED, used to build modules in the source tree, which can be overridden separately -- otherwise it defaults to LDSHARED. If the rest of the testsuite ever is cleaned up for AIX, one either could skip the parts of test_distutils that require the LDSHARED files or add special hooks in the test for AIX. As you point out, the change doesn't make the current testsuite results situation any worse.