ctypes find_library should search LD_LIBRARY_PATH on Linux

[jniehof]

BPO	9998
Nosy	@vsajip, @amauryfa, @abalkin, @meadori, @ambv, @vadmium, @aixtools
Files	ctypes_ld_lib_path.patch: Patch: ctypes.find_library() search LD_LIBRARY_PATH on linux new-patch-using-ld-and-test.diff: Patch using ld rather than gcc and with a test refresh-2016.diff: Patch updated to apply against 3.6. refresh-2016-07-29.diff: New patch addressing vadmium's comments. refresh-2016-08-07.diff: Responded to review comments. refresh-2016-08-08.diff: Added exception handler for case where ld is unavailable

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vsajip'
closed_at = <Date 2016-08-17.15:20:39.174>
created_at = <Date 2010-09-30.14:45:07.917>
labels = ['ctypes', 'type-feature']
title = 'ctypes find_library should search LD_LIBRARY_PATH on Linux'
updated_at = <Date 2016-08-17.15:20:39.172>
user = 'https://bugs.python.org/jniehof'

bugs.python.org fields:

activity = <Date 2016-08-17.15:20:39.172>
actor = 'python-dev'
assignee = 'vinay.sajip'
closed = True
closed_date = <Date 2016-08-17.15:20:39.174>
closer = 'python-dev'
components = ['ctypes']
creation = <Date 2010-09-30.14:45:07.917>
creator = 'jniehof'
dependencies = []
files = ['19068', '25822', '43929', '43937', '44036', '44052']
hgrepos = []
issue_num = 9998
keywords = ['patch']
message_count = 28.0
messages = ['117738', '138738', '141079', '141093', '141115', '143255', '162296', '165794', '165806', '165807', '205069', '205097', '261412', '261628', '261731', '264146', '264164', '264191', '264247', '265281', '271512', '271579', '271583', '271586', '271587', '271621', '272184', '272959']
nosy_count = 12.0
nosy_names = ['vinay.sajip', 'amaury.forgeotdarc', 'belopolsky', 'meador.inge', 'lukasz.langa', 'jniehof', 'python-dev', 'Brian.Larsen', 'yaroslavvb', 'martin.panter', 'Michael.Felt', 'Pau Tallada']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9998'
versions = ['Python 3.6']

[jniehof]

It's come up on occasion (bpo-2936, http://osdir.com/ml/python.ctypes/2008-05/msg00046.html) that ctypes find_library doesn't search LD_LIBRARY_PATH for libraries, which is different behaviour from the runtime linker. The attached patch adds this search.

Unfortunately I can't conceive of a reasonable unit test for this (compiling a shared library in setUp seems a bit overkill.)

[BrianLarsen]

Hey I have this problem too. I would love to see this fixed.

[yaroslavvb]

This causes problem for Freetype Python bindings on Linux

[jniehof]

Yaroslav: does the patch cause problems, or the original issue? If the former, could you be specific so I can try and fix it?

[yaroslavvb]

Sorry for confusion, I meant the original problem causes problems. I haven't tested the patch because my target machines don't have gcc

[vsajip]

There is a problem in this area of the code, not especially with the patch but with how gcc works (or doesn't). To illustrate:

-----------------------
Version info
-----------------------
vinay@eta-hardy:/tmp$ uname -a
Linux eta-hardy 2.6.24-29-generic #1 SMP Wed Aug 10 16:34:32 UTC 2011 i686 GNU/Linux
vinay@eta-hardy:/tmp$ gcc --version
gcc (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
-----------------------
Create an empty file:
-----------------------
vinay@eta-hardy:/tmp$ touch dummy.c
-----------------------
Compile a shared library from it:
-----------------------
vinay@eta-hardy:/tmp$ gcc -shared -o libdummy.so dummy.c
-----------------------
Invoke gcc with patched command line:
-----------------------
vinay@eta-hardy:/tmp$ gcc -L /tmp -Wl,t -o /dev/null -ldummy
/usr/bin/ld: t: No such file: No such file or directory
collect2: ld returned 1 exit status
-----------------------
It's not patch related, here's what happens with the unpatched line
-----------------------
vinay@eta-hardy:/tmp$ gcc -Wl,t -o /dev/null -ldummy
/usr/bin/ld: t: No such file: No such file or directory
collect2: ld returned 1 exit status
-----------------------
Calling ld directly works as expected. With no search path:
-----------------------
vinay@eta-hardy:/tmp$ ld -t -o /dev/null -ldummy
ld: mode elf_i386
ld: cannot find -ldummy
-----------------------
With the search path supplied:
-----------------------
vinay@eta-hardy:/tmp$ ld -t -L /tmp -o /dev/null -ldummy
ld: mode elf_i386
-ldummy (/tmp/libdummy.so)
ld: warning: cannot find entry symbol _start; not setting start address
vinay@eta-hardy:/tmp$

So, ISTM that the find_library code needs changing to use ld, else it will not work on all platforms.

[vsajip]

I added an updated patch which uses ld, and added a test (Linux/Unix only).

[ambv]

What changed since http://bugs.python.org/issue2936?

[vsajip]

What changed since http://bugs.python.org/issue2936?

Well for one thing, bpo-2936 was a bug report ("behavior") but this is an enhancement request. It seems that if you don't know exactly what a library is called, you can't use find_library to locate it even if it happens to be on LD_LIBRARY_PATH - and if you don't have the name, you can't load it using cdll.LoadLibrary.

ISTM LoadLibrary wants the full library name (including extension, on Linux); find_library helps determine that in a cross-platform way (since you can just give it the stem of the library rather than the whole filename).

[ambv]

Fair enough. From what I've gathered though, it looks like the window for inclusion in 3.3 is already closed, right?

[dan-blanchard]

Any chance of this making it into 3.4? This is a rather annoying deficiency of find_library().