gh-141808: Don't remove the JIT stencils when building with PGO

[hroncok]

See https://discuss.python.org/t/building-the-jit-with-pre-built-stencils/91838/12

• Issue: Build with PGO needlesly generates JIT stencils twice (and prevents usage of prebuilt stencils) #141808

[hroncok]

I opened this as a draft, because keeping the files from being deleted when make clean is run might be surprising for somebody. An alternative solution for Fedora would be to add something like --with-jit-stencils PATH to configure, which would replace JIT building with copying. However, I still think that building the JIT files multiple times during a PGO build is wasting resources.

[emmatyping]

My understanding of clean-retain-profile is that it is meant to remove files that depend on PYTHON_FOR_REGEN, since that will change between the PGO run and the final build. If that is correct, then I would say the stencils should be rm'd in make clean and make distclean, but should not change with make clean-retain-profile, since, as I understand it, the JIT stencils do not depend on the Python binary used other than platform information (which should not change across executables for the same host and build).

So I would suggest making a new target clean-jit-stencils which will be a dep of distclean and clean but not clean-retain-profile

[hroncok]

However, when the stencils are removed in make clean, they are apparently also removed during the pgo task that calls it. I have not yet figured out why and where make clean is called.

[hroncok]

Ah:

cpython/Makefile.pre.in

Lines 838 to 841 in bc9e63d

	# Profile generation build must start from a clean tree.
	profile-clean-stamp:
	$(MAKE) clean
	touch $@

So the problem with my approach is that by dropping in the prebuilt stencils before this runs, they are deleted here. That's probably only problematic for my use case :/

[hroncok]

So, I modified the behavior to keep removing the stencils with make clean but not run make clean during the PTO build. I think this is cleaner.

[efimov-mikhail]

LGTM.
I can confirm that issue with double generation of JIT stencils for PGO build is fixed.
And Makefile.pre.in changes looks clean for me.

[vstinner]

I can confirm that issue with double generation of JIT stencils for PGO build is fixed.

I also confirm that the change works as expected.

I built Python with the JIT and PGO using commands:

./configure --enable-experimental-jit=yes --enable-optimizations
make clean
make

Currently, I see the command twice in make logs:

python3.15 ./Tools/jit/build.py x86_64-pc-linux-gnu --output-dir . --pyconfig-dir . --cflags="" --llvm-version=""

With this PR, I only see the command once.

[vstinner]

LGTM

[vstinner]

Merged, thanks!

The question is now if this change should be backported to 3.13 and 3.14 branches. Other branches are also affected by the issue, no? What do you think?

[merwok]

I would say yes to backport, as this is a build change and more a fix for specific needs than a new feature.

Pinging RM @hugovk

[hroncok]

Merged, thanks!

The question is now if this change should be backported to 3.13 and 3.14 branches. Other branches are also affected by the issue, no? What do you think?

The other branches have the rm without the asterisk:

cpython/Makefile.pre.in

Line 3225 in 04ecff5

-rm -f jit_stencils.h

So technically, I am not even sure this is needed. I do not need a backport of this for Fedora.

[vstinner]

@hroncok:

So technically, I am not even sure this is needed.

At least, I can say that make runs ./Tools/jit/build.py twice when building with PGO.

[hroncok]

It runs twice but should only actually build the stencils once, due to the digest-match check.

[hugovk]

I would say yes to backport, as this is a build change and more a fix for specific needs than a new feature.

Pinging RM @hugovk

Yep, looks like a backportable bug fix to me.

[miss-islington-app]

Thanks @hroncok for the PR, and @vstinner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @hroncok and @vstinner, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 8914148151c957aebfaab1f3c890144d1b33968d 3.14