Update Modules/Setup

[brettcannon]

BPO	45548
Nosy	@malemburg, @brettcannon, @tiran, @ned-deily, @zware, @asottile, @miss-islington
PRs	bpo-45548: add some missing entries to Modules/Setup #29115 bpo-45548: Make Modules/Setup easier to read #29143 bpo-45548: Have Modules/Setup only try to compile _math once #29177 bpo-45548: Remove _math.c workarounds for pre-C99 libm #29179 bpo-45548: Inline _math.c into _math.h #29181 bpo-45548: Put Modules/Setup extensions into builddir #29188 bpo-45548: Add missing extensions to Modules/Setup #29199 bpo-45548: Remove checks for finite and gamma (GH-29206) #29206 bpo-45548: FreeBSD doesn't like auto vars in makesetup (GH-29216) #29216 bpo-45548: makesetup improvements (GH-29225) #29225 bpo-45548: Fix out-of-tree and Debian builds (GH-29263) #29263 bpo-45548: Some test modules must be built as shared libs (GH-29268) #29268

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2021-11-23.10:58:45.780>
created_at = <Date 2021-10-20.23:58:48.199>
labels = ['build', '3.11']
title = 'Update Modules/Setup'
updated_at = <Date 2021-11-23.10:58:45.779>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2021-11-23.10:58:45.779>
actor = 'christian.heimes'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2021-11-23.10:58:45.780>
closer = 'christian.heimes'
components = ['Build']
creation = <Date 2021-10-20.23:58:48.199>
creator = 'brett.cannon'
dependencies = []
files = []
hgrepos = []
issue_num = 45548
keywords = ['patch']
message_count = 31.0
messages = ['404548', '404559', '404560', '404627', '404710', '404790', '404864', '404897', '404898', '404899', '404900', '404906', '404924', '404953', '404980', '404987', '404992', '405032', '405045', '405046', '405047', '405053', '405055', '405056', '405092', '405123', '405172', '405227', '405314', '406812', '406834']
nosy_count = 7.0
nosy_names = ['lemburg', 'brett.cannon', 'christian.heimes', 'ned.deily', 'zach.ware', 'Anthony Sottile', 'miss-islington']
pr_nums = ['29115', '29143', '29177', '29179', '29181', '29188', '29199', '29206', '29216', '29225', '29263', '29268']
priority = 'low'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue45548'
versions = ['Python 3.11']

[brettcannon]

[Modules/Setup](https://github.com/python/cpython/blob/main/Modules/Setup) is currently incomplete/broken. Some things are completely missing from it, while others won't work if you uncomment them in the file.

When trying to compile a completely static CPython interpreter I ran into the following issues:

• _weakref is listed twice
• _testcapi can't be statically compiled
• math/cmath conflict with a build rule in Makefile.pre.in
• _dbm needs more logic/comments (at least on macOS)
• nis needs more logic/comments (at least on macOS)

And the following modules are completely missing from the file:

• _ctypes
• _decimal
• _lsprof
• _lzma
• _multiprocessing
• _opcode
• _posixshmem
• _queue
• _scproxy
• _sqlite3
• _testbuffer
• _testipmortmultiple
• _testmultiphase
• _uuid
• xxsubinterpreters
• xxtestfuzz
• ossaudiodev
• xxlimited
• xxlimited_35

[miss-islington]

New changeset dd86f63 by Brett Cannon in branch 'main':
bpo-45548: add some missing entries to [Modules/Setup](https://github.com/python/cpython/blob/main/Modules/Setup) (GH-29115)
dd86f63

[brettcannon]

Status is now:

• math/cmath conflict with a build rule in Makefile.pre.in
• _dbm needs more logic/comments (at least on macOS)
• nis needs more logic/comments (at least on macOS)

And the following modules are completely missing from the file:

• _bz2
• _ctypes
• _decimal
• _multiprocessing
• _posixshmem
• _scproxy
• _sqlite3
• _uuid

[tiran]

I started https://github.com/python/cpython/compare/main...tiran:configure_pkgconfig?expand=1 to integrate pkg-config with configure. The patchset also contains examples how to pass flags from configure into Makefile and Modules/Setup.

[brettcannon]

New changeset 01cf4fb by Brett Cannon in branch 'main':
bpo-45548: Make [Modules/Setup](https://github.com/python/cpython/blob/main/Modules/Setup) easier to read (GH-29143)
01cf4fb

[tiran]

I added several missing module definitions in #29164

math/cmath conflict when _math.c is included twice.

[tiran]

I ran into another problem. In shared mode, Modules/Setup places the shared modules in ./Modules/ instead of the build directory ./build/lib.something/. The ./Modules/ directory is not on sys.path. getpath.c only adds the build dir to sys.path. It takes the value from ./pybuilddir.txt. The file is generated by sysconfig.py.

$ make
...
*** WARNING: renaming "_asyncio" since importing it failed: No module named 'math'

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc cmath math
pwd time

$ find -name 'math*.so'
./Modules/math.cpython-311-x86_64-linux-gnu.so

[malemburg]

FYI: I've been working with a fixed Setup file in PyRun for a long while. There are indeed a number of modules missing from Setup, since the whole logic was left behind a bit after things moved to setup.py.

The issue with _math.o is actually in the main Makefile.pre.in. The version listed there does not match the Makefile lines added via Setup. For PyRun, I had to comment out the one in Makefile.pre.in and then only add one instance to the math module and not the cmath one. This avoids a (harmless) warning during the build.

I'm not sure what the _math.o entry exists in Makefile.pre.in. It's only needed by those two modules, AFAIK.

Here's the list of modules I had to add in the past (taken from the 3.10 port):

"""
### Built-in extensions for which there are no entries in Setup.dist/Setup:

# _decimal needs more complex setup, punting on this for now
#DECIMAL_DEFS=-DCONFIG_64=1 -DASM=1
#_decimal \
# _decimal/_decimal.c \
# _decimal/libmpdec/basearith.c \
# _decimal/libmpdec/constants.c \
# _decimal/libmpdec/context.c \
# _decimal/libmpdec/convolute.c \
# _decimal/libmpdec/crt.c \
# _decimal/libmpdec/difradix2.c \
# _decimal/libmpdec/fnt.c \
# _decimal/libmpdec/fourstep.c \
# _decimal/libmpdec/io.c \
# _decimal/libmpdec/memory.c \
# _decimal/libmpdec/mpdecimal.c \
# _decimal/libmpdec/numbertheory.c \
# _decimal/libmpdec/sixstep.c \
# _decimal/libmpdec/transpose.c \
# $(DECIMAL_DEFS) \
# -I$(srcdir)/Modules/_decimal \
# -I$(srcdir)/Modules/_decimal/libmpdec \
# -I$(prefix)/include -L$(exec_prefix)/lib

# _opcode
_opcode _opcode.c

# _ctypes needs to build libffi first - punting on this

# _lsprof
_lsprof _lsprof.c rotatingtree.c

# _sqlite3
SQLITE_DEFS=-DMODULE_NAME='"sqlite3"' -DSQLITE_OMIT_LOAD_EXTENSION
# @if freebsd: SQLITE_LIBS=-I/usr/local/include -L/usr/local/lib
# @if not freebsd: SQLITE_LIBS=
_sqlite3 \
_sqlite/module.c \
_sqlite/cache.c \
_sqlite/connection.c \
_sqlite/cursor.c \
_sqlite/microprotocols.c \
_sqlite/prepare_protocol.c \
_sqlite/row.c \
_sqlite/statement.c \
_sqlite/util.c \
$(SQLITE_DEFS) -I$(srcdir)/Modules/_sqlite \
$(SQLITE_LIBS) \
-I$(prefix)/include -L$(exec_prefix)/lib \
-lsqlite3

# bz2
_bz2 _bz2module.c -lbz2

# lzma

# Note: Adding this can cause serious issues, since the needed lib isn't
# universally installed everywhere. See bpo-1793 and bpo-1794.

#_lzma _lzmamodule.c -llzma

# multiprocessing
_multiprocessing \
_multiprocessing/semaphore.c \
_multiprocessing/multiprocessing.c \
-I$(srcdir)/Modules/_multiprocessing

# Optional add-on for multiprocessing to use shared memory
#POSIXSHMEM_LIBS=rt
POSIXSHMEM_LIBS=
_posixshmem \
        _multiprocessing/posixshmem.c \
        -I$(srcdir)/Modules/_multiprocessing \
        $(POSIXSHMEM_LIBS)

# queue
_queue _queuemodule.c
"""

Not all modules are included, since I did not need all missing ones.

[tiran]

PR #73365 or #73367 address the issue with _math.o

PR #73350 adds the missing modules and also introduces pkg-config lookups for dependencies.

Brett and I discussed that we have to introduce conditionals to Modules/Setup to make use of the AM_CONDITIONALs from PR #73350. We also have to figure out how to deal with system-libmpdec and system-cffi.