Profile Guided Optimization improvements (better training, llvm support, etc)

[alecsandrupatrascu]

BPO	24915
Nosy	@smontanaro, @brettcannon, @gpshead, @pitrou, @scoder, @vstinner, @larryhastings, @skrah
Dependencies	bpo-25188: regrtest.py improvement for Profile Guided Optimization builds
Files	python2.7-pgo.patch: PGO patch for Python2.7 python3.6-pgo.patch: PGO patch for Python3.6 python2.7-pgo-v02.patch python3.6-pgo-v02.patch python2.7-pgo-v02-mac.patch python3.6-pgo-v02-mac.patch python2.7-pgo-v03.patch python3.6-pgo-v03.patch python2.7-pgo-v04.patch python3.6-pgo-v04.patch python2.7-pgo-v05.patch python3.6-pgo-v05.patch python2.7-pgo-v06.patch python3.6-pgo-v06.patch README.pgo README2.7-pgo-v01.patch README3.6-pgo-v01.patch README2.7-pgo-v02.patch README3.6-pgo-v02.patch python2.7-pgo-v07.patch python3.6-pgo-v07.patch issue24915-python2.7.diff: Unified patch for Python 2.7 pgo.py pgofix-cpython2.patch pgofix-cpython3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2016-02-20.20:28:06.655>
created_at = <Date 2015-08-22.14:41:42.207>
labels = ['build', 'performance']
title = 'Profile Guided Optimization improvements (better training, llvm support, etc)'
updated_at = <Date 2016-02-20.20:28:06.653>
user = 'https://bugs.python.org/alecsandrupatrascu'

bugs.python.org fields:

activity = <Date 2016-02-20.20:28:06.653>
actor = 'brett.cannon'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2016-02-20.20:28:06.655>
closer = 'brett.cannon'
components = ['Build']
creation = <Date 2015-08-22.14:41:42.207>
creator = 'alecsandru.patrascu'
dependencies = ['25188']
files = ['40231', '40232', '40236', '40237', '40238', '40239', '40254', '40255', '40267', '40268', '40273', '40274', '40288', '40289', '40297', '40299', '40300', '40389', '40390', '40391', '40392', '40441', '40522', '41916', '41917']
hgrepos = []
issue_num = 24915
keywords = ['patch']
message_count = 63.0
messages = ['248988', '248992', '249002', '249004', '249006', '249008', '249013', '249014', '249052', '249053', '249055', '249061', '249071', '249128', '249131', '249143', '249146', '249155', '249200', '249227', '249246', '249256', '249286', '249315', '249333', '249336', '249355', '250008', '250011', '250093', '250095', '250096', '250099', '250100', '250102', '250105', '250489', '251033', '251034', '251035', '251036', '251065', '251090', '251091', '251096', '251110', '251112', '251125', '251126', '251127', '251128', '251129', '252179', '252182', '252184', '259840', '259868', '259870', '259878', '259881', '259883', '260269', '260573']
nosy_count = 11.0
nosy_names = ['skip.montanaro', 'brett.cannon', 'gregory.p.smith', 'tzot', 'pitrou', 'scoder', 'vstinner', 'larry', 'skrah', 'python-dev', 'alecsandru.patrascu']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue24915'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

[alecsandrupatrascu]

Hi All,

This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation.

I would like to submit a request to turn-on Profile Guided Optimization or PGO as the default build option for Python (both 2.7 and 3.6), given its performance benefits on a wide variety of workloads and hardware. For instance, as shown from attached sample performance results from the Grand Unified Python Benchmark, >20% speed up was observed. In addition, we are seeing 2-9% performance boost from OpenStack/Swift where more than 60% of the codes are in Python 2.7. Our analysis indicates the performance gain was mainly due to reduction of icache misses and CPU front-end stalls.

Attached is the Makefile patches that modify the all build target and adds a new one called "disable-profile-opt". We built and tested this patch for Python 2.7 and 3.6 on our Linux machines (CentOS 7/Ubuntu Server 14.04, Intel Xeon Haswell/Broadwell with 18/8 cores). We use "regrtest" suite for training as it provides the best performance improvement. Some of the test programs in the suite may fail which leads to build fail. One solution is to disable the specific failed test using the "-x " flag (as shown in the patch)

Steps to apply the patch:

1. hg clone https://hg.python.org/cpython cpython
2. cd cpython
3. hg update 2.7 (needed for 2.7 only)
4. Copy *.patch to the current directory
5. patch < python2.7-pgo.patch (or patch < python3.6-pgo.patch)
6. ./configure
7. make

To disable PGO
7b. make disable-profile-opt

In the following, please find our sample performance results from latest XEON machine, XEON Broadwell EP.
Hardware (HW): Intel XEON (Broadwell) 8 Cores

BIOS settings: Intel Turbo Boost Technology: false
Hyper-Threading: false

Operating System: Ubuntu 14.04.3 LTS trusty

OS configuration: CPU freq set at fixed: 2.6GHz by
echo 2600000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
echo 2600000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
Address Space Layout Randomization (ASLR) disabled (to reduce run to run variation) by
echo 0 > /proc/sys/kernel/randomize_va_space

GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)

Benchmark: Grand Unified Python Benchmark (GUPB)
GUPB Source: https://hg.python.org/benchmarks/

Python2.7 results:
Python source: hg clone https://hg.python.org/cpython cpython
Python Source: hg update 2.7
hg id: 0511b1165bb6 (2.7)
hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10
hg --debug id -i: 0511b1165bb6cf40ada0768a7efc7ba89316f6a5

    Benchmarks          Speedup(%)
    simple_logging      20
    raytrace            20
    silent_logging      19
    richards            19
    chaos               16
    formatted_logging   16
    json_dump           15
    hexiom2             13
    pidigits            12
    slowunpickle        12
    django_v2           12
    unpack_sequence     11
    float               11
    mako                11
    slowpickle          11
    fastpickle          11
    django              11
    go                  10
    json_dump_v2        10
    pathlib             10
    regex_compile       10
    pybench             9.9
    etree_process       9
    regex_v8            8
    bzr_startup         8
    2to3                8
    slowspitfire        8
    telco               8
    pickle_list         8
    fannkuch            8
    etree_iterparse     8
    nqueens             8
    mako_v2             8
    etree_generate      8
    call_method_slots   7
    html5lib_warmup     7
    html5lib            7
    nbody               7
    spectral_norm       7
    spambayes           7
    fastunpickle        6
    meteor_contest      6
    chameleon           6
    rietveld            6
    tornado_http        5
    unpickle_list       5
    pickle_dict         4
    regex_effbot        3
    normal_startup      3
    startup_nosite      3
    etree_parse         2
    call_method_unknown 2
    call_simple         1
    json_load           1
    call_method         1

Python3.6 results
Python source: hg clone https://hg.python.org/cpython cpython
hg id: 96d016f78726 tip
hg id -r 'ancestors(.) and tag()': 1a58b1227501 (3.5) v3.5.0rc1
hg --debug id -i: 96d016f78726afbf66d396f084b291ea43792af1

    Benchmark           Speedup(%)
    fastunpickle        22.94
    fastpickle          21.67
    json_load           17.64
    simple_logging      17.49
    meteor_contest      16.67
    formatted_logging   15.33
    etree_process       14.61
    raytrace            13.57
    etree_generate      13.56
    chaos               12.09
    hexiom2             12
    nbody               11.88
    json_dump_v2        11.24
    richards            11.02
    nqueens             10.96
    fannkuch            10.79
    go                  10.77
    float               10.26
    regex_compile       9.8
    silent_logging      9.63
    pidigits            9.58
    etree_iterparse     9.48
    2to3                8.44
    regex_v8            8.09
    regex_effbot        7.88
    call_simple         7.63
    tornado_http        7.38
    etree_parse         4.92
    spectral_norm       4.72
    normal_startup      4.39
    telco               3.88
    startup_nosite      3.7
    call_method         3.63
    unpack_sequence     3.6
    call_method_slots   2.91
    call_method_unknown 2.59
    iterative_count     0.45
    threaded_count      -2.79

Thank you,
Alecsandru

[scoder]

Please upload your patches as separate, uncompressed files for review.

PGO was already proposed here, but nothing came out of it:

https://bugs.python.org/issue17781

I suggest rejecting that old ticket and sticking with this one since it has an actual patch.

[alecsandrupatrascu]

I added the patches as individual files and removed the zip file.

[smontanaro]

Is this supposed to work on Macs using Apple's version of gcc? I've got the latest version of Yosemite and XCode, and am getting these warnings when trying to build 2.7:

clang: warning: argument unused during compilation: '-fprofile-generate'

Should this be enabled using a configure check? Perhaps gcc/clang supports this but spells the feature differently. gcc --help tells me:

% gcc --help | egrep profile
-fprofile-instr-generate
-fprofile-instr-use=<value>
Use instrumentation data for profile-guided optimization
-fprofile-sample-use=<value>
Enable sample-based profile guided optimizations

[alecsandrupatrascu]

The patches are tested on Linux machines, with GNU GCC >4.8.3. From your output I see that you are using the CLANG compiler. CLANG uses a different set of flags for PGO that are not compatible with GCC's, therefore the compilation will fail. Can you please use the GNU GCC compiler to test the patches?

[smontanaro]

It is executed using the gcc command:

% gcc -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fprofile-generate -I. -IInclude -I./Include -DPy_BUILD_CORE -o Modules/gcmodule.o Modules/gcmodule.c
clang: warning: argument unused during compilation: '-fprofile-generate'
% type gcc
gcc is /usr/bin/gcc

I have no idea if you can even use something other than clang on Macs now. In any case, the default compiler should work to build Python out of the box, if necessary, by checking things during configure.

[brettcannon]

I did an initial code review on the 3.6 patch.

What would it take to add clang support for PGO? Is it simply using different flags that configure can set in the generated Makefile? Or is it more involved and would require maintaining two separate compile lines in the Makefile?

[alecsandrupatrascu]

I received the review and will post new patch versions as soon as I update them.

Regarding PGO on clang, I will need a bit more time to edit the Makefile and will post it just for clang, to be easier for us to see the differences.

[alecsandrupatrascu]

I modified the patches after the review made by Brett (python2.7-pgo-v02.patch and python3.6-pgo-v02.patch):

• removed the call to pybench
• left the PGO steps as optional. To use it we run "make profile-opt"
• in the initial patches, I left out test_hashlib because in our benchmarks we did not see any gain by applying PGO to the hash functions. It is not harmful and we can let it there or skip it. Nevertheless, in order not to create confusions about it, I removed that parameter from the patch.

I also added the patches for Mac exclusively (python2.7-pgo-v02-mac.patch and python3.6-pgo-v02-mac.patch). You must have llvm-profdata installed and in your path (in /Library/Developer/CommandLineTools/usr/bin/) to use it. I tested on Yosemite and it compiles fine with clang. I am working on a generic version (configure and Makefile patches) to merge all these platforms and will post them as soon as it is done.

[skrah]

My initial reaction is that the default should be optimized for
short build times. I would not want to type "disable-profile-opt"
every time I'm running the tests.

[skrah]

I see that your latest patch leaves PGO as an option, so
please ignore my previous comment.

[brettcannon]

the v02 patches LGTM. I'm fine with seeing those committed as-is knowing Alecsandru is actively working towards Clang support.

[gpshead]

i'm updating the title to be more accurate.

turning it on by default is likely not desirable as the makefile is primarily used by developers who are iterating on changes.

but having it use a good workload (regrtest) and work with llvm and os x are good. :)