AIX shared library fix

[dickdunbar]

BPO	941346
Nosy	@warsaw, @birkenfeld, @pitrou, @benjaminp, @tardigradus, @haubi
Files	patch_AIX_shared.diff Python-2.6.6_shared_AIX.diff: Patch to correct compilation of Python 2.6.6 as a shared library on AIX python266_AIX61_static_gcc_nopatch.txt.gz: Build and tests log for pristine Python compiled statically on AIX 6.1 with gcc python266_AIX61_static_gcc_patchshared.txt.gz: Build and tests log for patched Python compiled statically on AIX 6.1 with gcc python266_AIX61_static_xlc_nopatch.txt.gz: Build and tests log for pristine Python compiled statically on AIX 6.1 with xlc python266_AIX61_static_xlc_patchshared.txt.gz: Build and tests log for patched Python compiled statically on AIX 6.1 with xlc python266_AIX61_shared_gcc_nopatch.txt.gz: Build and tests log for pristine Python compiled dynamically on AIX 6.1 with gcc python266_AIX61_shared_gcc_patchshared.txt.gz: Build and tests log for patched Python compiled dynamically on AIX 6.1 with gcc python266_AIX61_shared_xlc_nopatch.txt.gz: Build and tests log for pristine Python compiled dynamically on AIX 6.1 with xlc python266_AIX61_shared_xlc_patchshared.txt.gz: Build and tests log for patched Python compiled dynamically on AIX 6.1 with xlc Python-3.1.2_shared_AIX.diff: Patch to correct compilation of Python 3.1.2 as a shared library on AIX python312_AIX61_static_nopatch.txt.gz: Build and tests log for pristine Python 3.1.2 compiled statically on AIX 6.1 with xlc python312_AIX61_static_patch_shared.txt.gz: Build and tests log for patched Python 3.1.2 compiled statically on AIX 6.1 with xlc python312_AIX61_shared_patch_shared.txt.gz: Build and tests log for patched Python 3.1.2 compiled dynamically on AIX 6.1 with xlc aix32.diff Python-2.7_shared_AIX.diff: Patch to correct compilation of Python 2.7 as a shared library on AIX python312_AIX61_static_gcc_patchshared.txt.gz: Build and tests log for patched Python 3.1.2 compiled statically on AIX 6.1 with gcc python312_AIX61_shared_gcc_patchshared.txt.gz: Build and tests log for patched Python 3.1.2 compiled dynamically on AIX 6.1 with gcc py3k_aix61_static_xlc_nopatch.txt.gz py3k_aix61_static_xlc_patchshared.txt.gz py3k_aix61_static_gcc_patchshared.txt.gz py3k_aix61_shared_gcc_patchshared.txt.gz py3k_aix61_shared_xlc_pathshared.txt.gz patch_aix_shared_32.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2011-02-25.11:20:59.353>
created_at = <Date 2004-04-24.15:21:59.000>
labels = ['extension-modules', 'type-feature', 'build']
title = 'AIX shared library fix'
updated_at = <Date 2011-02-25.11:20:59.351>
user = 'https://bugs.python.org/dickdunbar'

bugs.python.org fields:

activity = <Date 2011-02-25.11:20:59.351>
actor = 'georg.brandl'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2011-02-25.11:20:59.353>
closer = 'georg.brandl'
components = ['Build', 'Extension Modules']
creation = <Date 2004-04-24.15:21:59.000>
creator = 'dickdunbar'
dependencies = []
files = ['11081', '18687', '18688', '18689', '18690', '18691', '18692', '18693', '18694', '18695', '18762', '18763', '18764', '18765', '18767', '18768', '18796', '18797', '18818', '18819', '18821', '18822', '18832', '20764']
hgrepos = []
issue_num = 941346
keywords = ['patch']
message_count = 38.0
messages = ['60482', '60483', '70894', '70896', '70902', '88115', '88116', '115262', '115279', '115375', '115703', '115707', '115708', '115709', '115729', '115731', '115734', '115865', '115876', '115987', '116040', '116042', '128214', '128281', '128586', '128591', '128592', '128593', '128594', '128596', '128597', '128602', '128634', '128638', '128639', '128640', '128652', '129355']
nosy_count = 9.0
nosy_names = ['barry', 'nnorwitz', 'georg.brandl', 'pitrou', 'sable', 'benjamin.peterson', 'dickdunbar', 'loris', 'haubi']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue941346'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7']

[dickdunbar]

By default, the ld command only looks for *.a archives
to resolve library references.

To recognize dynamic shared libaries (*.so), two flags
need to be added to the Modules/ld_so_aix script: -brtl
-bnortllib

Line 170 in the script should read:

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -brtl
-bnortllib -lm -o $objfile"

-brtl ... says you want to include libtk8.4.so
when searching for the library -ltk8.4

-bnortllib ... says you do not want the semantics of
the AIX "run time linking" services ... just the search
for the library.

This resolves a problem when Python is built with tk
support.

[nnorwitz]

Logged In: YES
user_id=33168

I added a comment based on the above information. I don't
want to make a real change, since I can't test it. If we
get access to some AIX boxes, we could really fix. I'm
hestitant to break stuff without testing.

Checked in as:

Modules/ld_so_aix 2.6

[sable]

Hi,

I have modified a bit this patch to make it work with Python 2.5.2 (see
attached patched).

It is a great enhancement for Python users on AIX to be able to use a
shared library instead of a static one, so I would appreciate if this
feature could be integrated in trunk.

We have various AIX 5.2 and 5.3 servers in the Sungard company where I
work, and we could run some tests according to your instructions. It may
also be possible to provide some limited access to one of those servers
if needed but it would require some time and paper work.

What kind of access would you need in order to validate those patchs?

thanks in advance

[pitrou]

Sébastien, what does your patch change exactly? Does it build a shared
lib for Python by default? If that's the case, I'm not sure it's a good
idea due to AIX's particular way of treating shared libraries; it might
break e.g. when trying to embed Python in another program.
(I say this while being an AIX novice, but I did have to compile Python
under AIX for embedded use in another third-party software...
unfortunately right now I don't have access to an AIX machine with a
compiler)

[sable]

Hum, I tried to recompile Python without the --enable-shared option on
AIX and it crashed; so this patch may not be compatible with a static
version of Python (though I have other patches in my version - cf issue
3526 - which may be incompatible).

I will do a bit of cleanup in my environment and try to make a version
of the patch that works also when compiling a static Python.

[tardigradus]

Shouldn't line 170 be

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -Wl,-brtl
-Wl,-bnortllib -lm -o $objfile"

instead of

CCOPT="$CCOPT -Wl,-bM:SRE -Wl,-T512 -Wl,-H512 -brtl -bnortllib -lm -o
$objfile"

Otherwise the -brtl and -bnortllib flags are interpreted as compiler
flags and errors like the following will occur:

gcc: unrecognized option '-brtl'

[tardigradus]

Addition to last message:

More importantly, without "-Wl", the flags are not passed to the linker
and the tcl/tk libs will not be found.

[sable]

Hi,

As reported in this issue and bpo-1756343 and bpo-1542544, Python does not produce a shared python library on AIX even with the --enable-shared flag.

I had provided a patch to correct that, but it was breaking static compilation of Python on AIX.

We are upgrading our AIX build environment to AIX 6.1 and Python 2.6.6, so I took the time to review this patch again and to clean it so that it works well when compiling Python statically and dynamically, with either gcc and xlc_r.

I attach the resulting patch, and the build and unit test logs. You can notice that not only this patch allows to compile Python dynamically on AIX, but also that it improves the unit tests results when compiling Python statically.

For example with the xlc_r compiler and Python compiled statically, the results without the patch are:
274 tests OK.
22 tests failed
70 tests skipped
The results with the patch are:
286 tests OK.
21 tests failed
59 tests skipped

A minor issue with this patch is that "libpython2.6.so" is hardcoded in the ld_so_aix file. That should be modified by moving ld_so_aix to ld_so_aix.in and using $LDLIBRARY, or by using an explicit flag when calling ld_so_aix to compile libpythonx.y.so.

If you agree to the principle of this patch, I can make the extra work to clean ld_so_aix and test this patch on Python 2.7 and 3.1.

regards

[pitrou]

In dynamic builds, there seem to be lots of messages such as:

ld: 0711-224 WARNING: Duplicate symbol: .PyLong_FromString
ld: 0711-224 WARNING: Duplicate symbol: PyLong_FromString

Are you sure this is normal?

In any case, if you find a way to cleanup ld_so_aix, this should probably be checked in anyway. I doubt anybody else tests under AIX, so it's reasonable to trust you on this.

Even better if you manage to diagnose and/or fix some of the test failures, by the way. (if not, you can still open separate issues for each of them)