no PGO for built-in modules with `make profile-opt`

[tzot]

BPO	26307
Nosy	@gpshead, @stratakis
Files	pgofix-cpython2.patch pgofix-cpython3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2016-09-08.05:33:30.788>
created_at = <Date 2016-02-08.13:28:26.701>
labels = ['build', 'performance']
title = 'no PGO for built-in modules with `make profile-opt`'
updated_at = <Date 2016-09-08.05:33:30.786>
user = 'https://bugs.python.org/tzot'

bugs.python.org fields:

activity = <Date 2016-09-08.05:33:30.786>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2016-09-08.05:33:30.788>
closer = 'gregory.p.smith'
components = ['Build']
creation = <Date 2016-02-08.13:28:26.701>
creator = 'tzot'
dependencies = []
files = ['41999', '42000']
hgrepos = []
issue_num = 26307
keywords = ['patch']
message_count = 8.0
messages = ['259842', '259884', '260270', '260659', '274039', '274676', '274694', '274966']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'tzot', 'python-dev', 'alecsandru.patrascu', 'cstratak']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue26307'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[tzot]

(related to issue bpo-24915)
I discovered that make profile-opt does not use the profile information for the builtin-modules (e.g. arraymodule or _pickle) because in the profile-opt target there is the following sequence of actions:

…
$(MAKE) build_all_merge_profile
@echo "Rebuilding with profile guided optimizations:"
$(MAKE) clean
$(MAKE) build_all_use_profile
…

The action $(MAKE) clean performs an rm -rf build, destroying among other things all the *.gcda files generated for the built-in modules.
On my Linux system with gcc, a kludge to Makefile.pre.in that works is:

…
@echo "Rebuilding with profile guided optimizations:"
find build -name \*.gcda -print | cpio -o >_modules.gcda.cpio # XXX
$(MAKE) clean
cpio -id <_modules.gcda.cpio # XXX
$(MAKE) build_all_use_profile
…

but, like I said, it's a kludge and it's POSIX-only (-print0 can be avoided since I believe it's guaranteed that there will be no whitespace-containing-filenames).

Now, if this road is to be taken, I believe the most cross-platform method available to save the profile-generated files is to use a custom python script (at this point, the local profile-generating python executable is functional) and make a tar file, which will be untared back. However, I don't like it much.

I'm willing to provide any necessary patches if someone gives me a "proper" roadmap as to how to handle this issue.

[alecsandrupatrascu]

Thank you for noticing and signaling this issue. Since I proposed the PGO patches, I will fix it in another patch, as I don't want it to break any builds.

[alecsandrupatrascu]

I've added a fix for the PGO builds after the issue pointed out by you. Can you please test it also?

[alecsandrupatrascu]

I've added the patch here also.

[stratakis]

Pinging here. Christos could you test the patch?

[python-dev]

New changeset 7d9cd4a0d488 by Gregory P. Smith in branch '3.5':
Fixes bpo-26307: The profile-opt build now applys PGO to the built-in
https://hg.python.org/cpython/rev/7d9cd4a0d488

New changeset bdc7292cf87e by Gregory P. Smith in branch 'default':
Fixes bpo-26307: The profile-opt build now applys PGO to the built-in modules.
https://hg.python.org/cpython/rev/bdc7292cf87e

[python-dev]

New changeset 75dae0b2ccb3 by Gregory P. Smith in branch '2.7':
Fixes bpo-26307: The profile-opt build now applys PGO to the built-in modules.
https://hg.python.org/cpython/rev/75dae0b2ccb3

[gpshead]

My change means that the build/ directory tree exists but won't have any files other than *.gc?? files in it after a make clean. I doubt this will bother anyone. That felt better for clean than just avoiding the removal of build/ entirely.