"Out of tree" build fails on AIX

[pedz]

BPO	10656
Nosy	@merwok, @haubi, @vadmium
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	aix-fix-out-of-source.diff: Patch to apply on 2.7 branch issue10656-out-of-source-build-on-aix.patch: patch for 3.4

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-11-21.03:31:23.763>
created_at = <Date 2010-12-09.03:35:03.276>
labels = ['build', '3.7']
title = '"Out of tree" build fails on AIX'
updated_at = <Date 2018-01-02.15:40:35.893>
user = 'https://bugs.python.org/pedz'

bugs.python.org fields:

activity = <Date 2018-01-02.15:40:35.893>
actor = 'David.Edelsohn'
assignee = 'none'
closed = True
closed_date = <Date 2016-11-21.03:31:23.763>
closer = 'martin.panter'
components = ['Build']
creation = <Date 2010-12-09.03:35:03.276>
creator = 'pedz'
dependencies = []
files = ['21489', '35476']
hgrepos = ['252']
issue_num = 10656
keywords = ['patch']
message_count = 10.0
messages = ['123668', '128275', '128315', '132649', '219740', '279474', '279475', '279526', '281196', '281307']
nosy_count = 8.0
nosy_names = ['sable', 'eric.araujo', 'haubi', 'pedz', 'python-dev', 'Tristan.Carel', 'martin.panter', 'David.Edelsohn']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue10656'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[pedz]

I do "out of tree" builds (I think that is the right term). I make an
empty directory, cd into it, and then do:

<path to src directory>/configure

followed by make. This works on most open source pages. It appears to
be very close to working with Python 2.7.1 but the problem is that it assumes ld_so_aix is in the source tree but it is in the build tree.

I did an "in tree" build using basically just ./configure && make and so far (I'm still testing) that has worked fine.

[sable]

There are some unit tests concerning distutils which are currently failing on AIX and may be an illustration of the same problem:

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 85, in test_build_ext
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 343, in test_get_outputs
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

I will take a look.

[sable]

I tried with trunk (python 3.2) on AIX:
./build/configure --with-pydebug --without-computed-gotos
make
and that worked correctly.

Not sure with python 2.7.

[TristanCarel]

I have experienced the same problem under ppc64 aix 6.1.2.0 while compiling Python 2.7.1

$ cd Python-2.7.1
$ mkdir __build
$ cd __build
$ ../configure --with-gcc=xlc_r --with-cxx-main=xlC_r --with-threads --disable-ipv6
$ /usr/linux/bin/make
[...]
../Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/ngbuilder/python/2.7.1/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
unable to execute ../Modules/ld_so_aix: No such file or directory

There are references to $(srcdir)/Modules/ld_so_aix in both configure.in and Makefile.pre.in whereas ld_so_aix is created in build tree.

==================================================

diff -r f4e6e18cae4b Makefile.pre.in
--- a/Makefile.pre.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Makefile.pre.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1028,7 +1028,7 @@
 		$(INSTALL_SCRIPT) $(srcdir)/Modules/makexp_aix	\
 				$(DESTDIR)$(LIBPL)/makexp_aix;		\
 		echo "$(LIBPL)/makexp_aix";			\
-		$(INSTALL_SCRIPT) $(srcdir)/Modules/ld_so_aix	\
+		$(INSTALL_SCRIPT) Modules/ld_so_aix	\
 				$(DESTDIR)$(LIBPL)/ld_so_aix;		\
 		echo "$(LIBPL)/ld_so_aix";			\
 		echo; echo "See Misc/AIX-NOTES for details.";	\
diff -r f4e6e18cae4b configure.in
--- a/configure.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/configure.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1778,7 +1778,7 @@
 then
 	case $ac_sys_system/$ac_sys_release in
 	AIX*)
-		BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
+		BLDSHARED="Modules/ld_so_aix \$(CC) -bI:Modules/python.exp -L\$(srcdir)"
 		LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"
 		;;
 	BeOS*)

==================================================

After fixing these 2 files, Modules/ld_so_aix is found and properly executed by compilation process, but then there is a new issue:

Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
Modules/ld_so_aix[183]: Modules/makexp_aix: not found.

This time, makexp_aix is expected to be is the same directory than ld_so_aix whereas makeexp_aix remains in the source tree.

Here is a fix:

==================================================

diff -r f4e6e18cae4b Modules/ld_so_aix.in
--- a/Modules/ld_so_aix.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Modules/ld_so_aix.in	Thu Mar 31 13:01:57 2011 +0200
@@ -69,7 +69,10 @@
   echo $usage; exit 2
 fi
 
-makexp=`dirname $0`/makexp_aix
+makexp=`dirname $0`/@srcdir@/makexp_aix
+if test ! -e "$makexp" ; then
+  makeexp=makexp_aix
+fi
 
 # Check for existence of compiler.
 CC=$1; shift

==================================================

The first value fix the issue while building Python. The second will use PATH to find makeexp_aix as explained in section `SHARED LIBRARY SUPPORT' in AIX-NOTES (for building shared extension modules).

You can find a patch for 2.7 branch in attachment.

Regards

Tristan Carel
Exalead

[haubi]

Basically the same as Tristan's patch, with a little improvement to not rely on PATH to find makexp_aix within ld_so_aix.

Thanks!

[vadmium]

The idea of the patches makes sense to me.

Embedding the name of whatever directory Python was built in into the installed version of Modules/ld_so_aix.in does not seem very clean to me. Is there another way of doing this? Perhaps a file copy or symlink?

[vadmium]

Actually, why do you remove $(srcdir) for Modules/python.exp? This code was added in r88426; see <https://bugs.python.org/issue941346#msg128586\>.

[haubi]

Modules/python.exp is generated at build-time, thus does not belong to $(srcdir).

[vadmium]

Okay I will assume that $(srcdir) reference was just due to a mistake or bad cargo-culting then.

Will commit this soon unless anyone has a good idea to avoid embedding @abs_srcdir@.

[python-dev]

New changeset 48526666321a by Martin Panter in branch '3.5':
Issue bpo-10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/48526666321a

New changeset 76d1f8001e27 by Martin Panter in branch '3.6':
Issue bpo-10656: Merge AIX build fix from 3.5
https://hg.python.org/cpython/rev/76d1f8001e27

New changeset 180f046b597e by Martin Panter in branch 'default':
Issue bpo-10656: Merge AIX build fix from 3.6
https://hg.python.org/cpython/rev/180f046b597e

New changeset ca46883fc5cf by Martin Panter in branch '2.7':
Issue bpo-10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/ca46883fc5cf