./configure --enable-optimizations && make && make install compiles Python twice

[vstinner]

BPO	29641
Nosy	@vstinner
Superseder	bpo-29243: --enable-optimizations makes common build commands always need to compile from scratch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-02.13:34:12.209>
created_at = <Date 2017-02-24.12:46:45.551>
labels = ['build', '3.7']
title = './configure --enable-optimizations && make && make install compiles Python twice'
updated_at = <Date 2017-05-02.13:34:12.208>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-05-02.13:34:12.208>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-02.13:34:12.209>
closer = 'vstinner'
components = ['Build']
creation = <Date 2017-02-24.12:46:45.551>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 29641
keywords = []
message_count = 6.0
messages = ['288523', '288524', '288525', '288527', '292753', '292754']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '29243'
type = None
url = 'https://bugs.python.org/issue29641'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[vstinner]

Hi,

I compiled Python 3.7 using PGO+LTO with "make && make install", but it seems like it lacks the _sysconfigdata module.

Moreover, I don't see any "Makefile" file in the installed directory, so the "~/installed/3.7/bin/python3 -S -m sysconfig --generate-posix-vars" command fails on trying to import _sysconfigdata_m_linux_x86_64-linux-gnu ...

$ find /home/haypo/installed/3.7 -name "Makefile"
<nothing>

Is it correct to use "make && make install" using PGO?

cd ~/cpython
make distclean ||:
./configure --enable-optimizations --with-lto --prefix=/home/haypo/installed/3.7
make clean
make
make install

haypo@speed-python$ ~/installed/3.7/bin/python3 
Failed to import the site module
Traceback (most recent call last):
  File "/home/haypo/installed/3.7/lib/python3.7/site.py", line 544, in <module>
    main()
  File "/home/haypo/installed/3.7/lib/python3.7/site.py", line 530, in main
    known_paths = addusersitepackages(known_paths)
  File "/home/haypo/installed/3.7/lib/python3.7/site.py", line 282, in addusersitepackages
    user_site = getusersitepackages()
  File "/home/haypo/installed/3.7/lib/python3.7/site.py", line 258, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/home/haypo/installed/3.7/lib/python3.7/site.py", line 248, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/home/haypo/installed/3.7/lib/python3.7/sysconfig.py", line 601, in get_config_var
    return get_config_vars().get(name)
  File "/home/haypo/installed/3.7/lib/python3.7/sysconfig.py", line 550, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/home/haypo/installed/3.7/lib/python3.7/sysconfig.py", line 421, in _init_posix
    _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
ModuleNotFoundError: No module named '_sysconfigdata_m_linux_x86_64-linux-gnu'

[vstinner]

PGO compilation uses "make profile-removal", maybe my issue is linked to that?

I'm sure that PGO compilation + "make install" worked two months, but using: make profile-opt && make install.

[vstinner]

The --enable-optimizations option of ./configure was added by the issue bpo-26359 (first named --with-optimizations, it was recalled to --enable-optimizations later).

See also:

• issue bpo-28032: "never imply --with-lto as part of --with-optimizations"
• issue bpo-28605: Fix the help and What's New entry for --with-optimizations

[vstinner]

Ok, I confirm that it's possible to compile and install Python using PGO+LTO without --enable-optimizatioins:

cd ~/cpython
make distclean ||:
./configure --with-lto --prefix=/home/haypo/installed/3.7
make clean
make profile-opt
make install

[vstinner]

Ok, I tested again: in fact, "it works" but "make install" recompiles Python from scratch (1st build with -fprofile-generate, profile task, 2nd build with -fprofile-use) whereas "make" just succeeded to build Python!?

[vstinner]

Oh, it's just a duplicate of issue bpo-29243.