Race condition compiling Modules/_math.c

[vadmium]

BPO	24421
Nosy	@Yhg1s, @taleinat, @merwok, @vadmium, @floppym, @dstufft
Files	slow-cc.py include-math.c.patch math-once.patch: Compile from makefile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-02-26.21:46:18.055>
created_at = <Date 2015-06-10.08:06:40.639>
labels = ['type-bug', 'library', 'build']
title = 'Race condition compiling Modules/_math.c'
updated_at = <Date 2016-02-26.21:46:18.054>
user = 'https://github.com/vadmium'

bugs.python.org fields:

activity = <Date 2016-02-26.21:46:18.054>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-02-26.21:46:18.055>
closer = 'martin.panter'
components = ['Build', 'Distutils']
creation = <Date 2015-06-10.08:06:40.639>
creator = 'martin.panter'
dependencies = []
files = ['39672', '39724', '41270']
hgrepos = []
issue_num = 24421
keywords = ['patch']
message_count = 11.0
messages = ['245121', '245298', '245429', '245453', '245459', '254560', '256137', '258349', '259362', '260886', '260888']
nosy_count = 8.0
nosy_names = ['twouters', 'taleinat', 'eric.araujo', 'Arfrever', 'python-dev', 'martin.panter', 'floppymaster', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24421'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[vadmium]

Once or twice I have had the math or cmath modules fail to build. There seems to be a race condition, because the _math.c file seems to be compiled twice, once for each module. Attached is a GCC wrapper that should help reproduce this fairly consistently. First, build Python normally:

$ ./configure
$ make -j2

Then force Modules/_math.c to be recompiled using my wrapper script to amplify the race condition:

$ touch Modules/_math.c
$ rm build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
$ make -j2 CC=./slow-cc.py
running build
running build_ext
building 'cmath' extension
compiling for  cmath
./slow-cc.py [. . .]/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py [. . .]/Modules/mathmodule.o
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
INITIAL COMPILE
./slow-cc.py -fPIC -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/media/disk/home/proj/python/cpython/Include -I/media/disk/home/proj/python/cpython -c /media/disk/home/proj/python/cpython/Modules/_math.c -o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/mathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/math.cpython-36m-x86_64-linux-gnu.so
build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o: file not recognized: File truncated
collect2: error: ld returned 1 exit status
RECOMPILE FINISHED
./slow-cc.py -shared build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/cmathmodule.o build/temp.linux-x86_64-3.6/media/disk/home/proj/python/cpython/Modules/_math.o -L/usr/local/lib -lm -o build/lib.linux-x86_64-3.6/cmath.cpython-36m-x86_64-linux-gnu.so

Failed to build these modules:
math

[taleinat]

Confirmed on OSX 10.10. Here's my output:

running build
running build_ext
building 'cmath' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/cmathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o
building 'math' extension
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/mathmodule.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
./slow-cc.py -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/local/include -I/Users/taleinat/dev/cpython/Include -I/Users/taleinat/dev/cpython -c /Users/taleinat/dev/cpython/Modules/_math.c -o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o
INITIAL COMPILE
RECOMPILE
RECOMPIILE TRUNCATING
INITIAL COMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/mathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/math.cpython-36m-darwin.so
ld: file too small (length=0) file 'build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
RECOMPILE FINISHED
./slow-cc.py -bundle -undefined dynamic_lookup build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/cmathmodule.o build/temp.macosx-10.10-x86_64-3.6/Users/taleinat/dev/cpython/Modules/_math.o -L/usr/local/lib -o build/lib.macosx-10.10-x86_64-3.6/cmath.cpython-36m-darwin.so

[vadmium]

I think this may have been introduced when _math.c was added as a source file of the “cmath” module in r76978 (bpo-7518). In /setup.py:585 there are two distutils.core.Extension objects, both mentioning _math.c. In the build_ext._build_extensions_parallel() method at /Lib/distutils/command/build_ext.py:449, it appears to be invoking build_extension() in two concurrent threads, once for each module, including math and cmath.

I don’t know how this code is meant to work, but it looks like either the “build_ext” class should not be creating the same object file in two different threads, or /setup.py should not be giving it the same _math.c file for two different modules.

[vadmium]

Here is a hacky patch that adds #include "_math.c" so that it is not compiled as a separate object file. This was suggested by Mark in bpo-7518. It is far from a perfect solution, but I cannot suggest anything better without knowing more about Python’s build system.

[taleinat]

Adding Thomas Wouters to the nosy list, since he's listed on the Experts Index under "autoconf/makefiles" as an interest area. Hopefully he can help move this forward.

[vadmium]

Ideally, I think the solution would be to compile _math.c once, as a dependency shared between both extension modules (perhaps like a static library if necessary). Does anyone know if the build system supports this, or how to do it?

Another semi-favourable solution might be to compile _math.c twice, but send the output to separate directories, e.g. . . ./math-build/_math.o and . . ./cmath-build/_math.o. Again, I don’t know if this is practical.

[vadmium]

Here is an alternative patch to compile _math.c once from the main makefile, before the extension modules are built. I have only tested this on Linux, building in the source directory, and from a separate directory.

[vadmium]

I am inclined to try committing my second patch to 3.6, as the race condition keeps bugging me every now and again.

My biggest worry is that it may not work on Windows. I assume Windows does not use the Makefile, but does use setup.py. In the worst case, perhaps I could add conditional code to restore the original setup.py parameters for Windows.

[python-dev]

New changeset 76624d47ee99 by Martin Panter in branch 'default':
Issue bpo-24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/76624d47ee99

[python-dev]

New changeset a99580157968 by Martin Panter in branch '3.5':
Issue bpo-24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/a99580157968

[python-dev]

New changeset d4190ed586a4 by Martin Panter in branch '2.7':
Issue bpo-24421: Compile _math.c separately to avoid race condition
https://hg.python.org/cpython/rev/d4190ed586a4