Remove incorrect uses of recursive make

[Sjlver]

BPO	22359
Nosy	@loewis, @brettcannon, @doko42, @pitrou, @xdegaye, @vadmium, @koobs
Files	makefile_parallel.patch: Patch: Remove incorrect uses of recursive make koobs-freebsd9.python-2.7-build.912.txt cross-chgeset-c2a53aa27cad.patch separate-regen.patch: Separate "make graminit" etc targets cross-chgeset-c2a53aa27cad_2.patch crossbuild-sources-readonly.patch: Do not modify build source files crossbuild-sources-readonly_2.patch: _freeze_importlib and pgen are cross-built

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-24.03:41:09.132>
created_at = <Date 2014-09-08.08:44:29.657>
labels = ['type-bug', 'build']
title = 'Remove incorrect uses of recursive make'
updated_at = <Date 2016-04-24.03:41:09.131>
user = 'https://bugs.python.org/Sjlver'

bugs.python.org fields:

activity = <Date 2016-04-24.03:41:09.131>
actor = 'martin.panter'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-24.03:41:09.132>
closer = 'martin.panter'
components = ['Build']
creation = <Date 2014-09-08.08:44:29.657>
creator = 'Sjlver'
dependencies = []
files = ['36570', '40063', '42164', '42169', '42517', '42530', '42554']
hgrepos = []
issue_num = 22359
keywords = ['patch']
message_count = 29.0
messages = ['226563', '226797', '227186', '227187', '230766', '238199', '245110', '245112', '245117', '245119', '245122', '245365', '245366', '247650', '247651', '261775', '261786', '261799', '261824', '263721', '263793', '263815', '263875', '263891', '263898', '263902', '263915', '264035', '264089']
nosy_count = 11.0
nosy_names = ['loewis', 'brett.cannon', 'doko', 'pitrou', 'Arfrever', 'xdegaye', 'python-dev', 'martin.panter', 'koobs', 'Sjlver', 'WanderingLogic']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22359'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[Sjlver]

The attached patch fixes issues with Python's Makefile, which manifest when doing parallel builds. The Makefile invoked "make" recursively for some targets. This caused some files (which were depended upon by multiple targets) to be built by both the original "make" and the sub-"make".

Besides duplicate work, this caused failed builds with non-threadsafe compilers.

The proposed patch removes recursive calls to "make", and instead builds all targets in the same "make" process.

[pitrou]

Martin, Matthias, do you think this would break any legitimate use?

[python-dev]

New changeset c2a53aa27cad by Antoine Pitrou in branch 'default':
Issue bpo-22359: Remove incorrect uses of recursive make. Patch by Jonas Wagner.
https://hg.python.org/cpython/rev/c2a53aa27cad

[pitrou]

Ok, I've pushed the patch. Let's see if anyone complains.

[WanderingLogic]

Sorry, I'm complaining. Cross builds broke. Please see bpo-22809.

[doko42]

reopening, breaks cross builds. I'll have a look

[koobs]

We've (FreeBSD) seen and received several random and intermittent reports of failures during this stage of the build for all versions of Python, resulting in errors such as, among others:

Parser/pgen.o: file not recognized: File truncated
*** [Parser/pgen] Error code 1

pgenmain.c:(.text+0x258): undefined reference to `_Py_pgen'
*** [Parser/pgen] Error code 1
1 error
*** [Include/graminit.h] Error code 2

See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=200622

Given the existing patch breaks cross-builds, is there an alternative or safer way to make this section of the build more robust in order to make progress?

[vadmium]

FWIW two times recently I saw strange intermittent build errors. I use “make -j2” on Linux. The one time that I investigated was a Python module (math or cmath perhaps?) failing to build because some dependency was zero bytes, and looking through the build log, I saw duplicate compiler commands lines printed out one after the other to build that dependency.