ssl/hashlib: Add configure option to set or auto-detect rpath to OpenSSL libs

[tiran]

BPO	43466
Nosy	@warsaw, @gpshead, @vstinner, @tiran, @pablogsal, @miss-islington
PRs	bpo-43466: Add --with-openssl-rpath configure option (GH-24820) #24820 bpo-43466: Add -Wl,--exclude-libs to ssl #24989 bpo-43466: Unsupported static build hack #25002 bpo-43466: Use pkg-config with --static in the PY_UNSUPPORTED_OPENSSL_BUILD path in setup.py #25475 bpo-43466: Link with libz in PY_UNSUPPORTED_OPENSSL_BUILD path (GH-25587) #25587

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-03-29.17:46:19.796>
created_at = <Date 2021-03-10.23:14:03.328>
labels = ['expert-SSL', 'type-feature', 'expert-installation', '3.10', 'docs']
title = 'ssl/hashlib: Add configure option to set or auto-detect rpath to OpenSSL libs'
updated_at = <Date 2021-10-05.09:12:18.455>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2021-10-05.09:12:18.455>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-03-29.17:46:19.796>
closer = 'christian.heimes'
components = ['Documentation', 'Installation', 'SSL']
creation = <Date 2021-03-10.23:14:03.328>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 43466
keywords = ['patch']
message_count = 25.0
messages = ['388463', '388464', '388465', '388466', '388467', '388468', '388470', '388471', '388487', '388501', '388823', '388826', '388831', '388832', '388834', '389058', '389059', '389379', '389610', '389715', '391921', '403199', '403200', '403202', '403207']
nosy_count = 7.0
nosy_names = ['barry', 'gregory.p.smith', 'vstinner', 'christian.heimes', 'chaz6', 'pablogsal', 'miss-islington']
pr_nums = ['24820', '24989', '25002', '25475', '25587']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue43466'
versions = ['Python 3.10']

[tiran]

Python's configure script has the option --with-openssl. It sets a path to a custom OpenSSL installation. Internally it provides OPENSSL_INCLUDES, OPENSSL_LIBS, and OPENSSL_LDFLAGS. The setup.py script turns the variables into include_dirs, library_dirs, and libraries arguments for _ssl and _hashlib extension modules.

However neither --with-openssl nor setup.py sets a custom runtime library path (rpath). This makes it confusing and hard for users to use a custom OpenSSL installation. They need to know that a) they have to take care of rpath on the first place, and b) how to set an rpath at compile or runtime. Without an rpath, the dynamic linker either fails to locate libssl/libcrypto or load system-provided shared libraries. Ticket bpo-34028 contains examples of user issues.

I propose to include a new option to make it easier for users to use a custom build of OpenSSL:

--with-openssl-rpath=<DIR|auto|no>

no (default): don't set an rpath
auto: auto-detect rpath from OPENSSL_LDFLAGS (--with-openssl or pkg-config)
DIR: set a custom rpath

The option will only affect the rpath of _ssl and _hashlib modules. The default value "no" is fully backwards compatible with 3.9 and earlier.

[pablogsal]

I have a suggestion (I am ok with also doing both). We should add also a flag that allows to statically compile openssl into the extension modules (if the .a are available) so there will be no shared object dependency. This allows us to have artifacts that are more self contained (regarding OpenSSL), partially eliminating the problem.

We could also go a step ahead and also hide the OpenSSL symbols from the dynamic table, so they don't collide if another shared object is already loaded in the future.

[tiran]

$ tar -xzf openssl-1.1.1j.tar.gz
$ pushd openssl-1.1.1j
$ ./config \
    --prefix=/home/heimes/dev/python/custom-openssl \
    --openssldir=\
        $(find /etc/ -name openssl.cnf -quit -printf "%h" 2>/dev/null)
$ make
$ make install_sw
$ popd

$ pushd cpython
$ ./configure \
    --with-openssl=/home/heimes/dev/python/custom-openssl \
    --with-openssl-rpath=auto
$ make
$ ldd build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffde07bc000)
        libssl.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libssl.so.1.1 (0x00007f937493a000)
        libcrypto.so.1.1 => /home/heimes/dev/python/custom-openssl/lib/libcrypto.so.1.1 (0x00007f937465a000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9374608000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f937443d000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f9374436000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f93749ff000)
$ ./python
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.1.1j  16 Feb 2021'
>>> import urllib.request
>>> r = urllib.request.urlopen("https://www.python.org")
                                   ^^^ No cert validation error!

[tiran]

I would rather not support static linking.

OpenSSL uses dynamic linking by default. Static linking is problematic for dynamic engine support. This is going to become an even bigger issue with OSSL providers in OpenSSL 3.0.0. I don't know yet how well OpenSSL 3.0.0 will support static linking.

Our macOS and Windows builds are now dynamically linked, too. At least Windows builds used to be statically linked.

Dynamic linked OpenSSL has a major benefit: It allows users to update OpenSSL without re-compiling Python.

[pablogsal]

OpenSSL uses dynamic linking by default.

As long as OpenSSL provide a .a file (they do) you can always static link. That is a decision of how you integrate the library.

Static linking is problematic for dynamic engine support.

Not sure I follow. What's the problem here? The advantage of static linking here will be to not have a dependency on the shared object, which can be quite beneficial.

Dynamic linked OpenSSL has a major benefit: It allows users to update OpenSSL without re-compiling Python.

Yeah, I agree. That's what should always be the default. But there are still advantages regarding static linking.