[C API] Add tests on Py_FrozenMain()

[vstinner]

BPO	44131
Nosy	@vstinner, @encukou, @corona10, @pablogsal, @miss-islington
PRs	bpo-44131: Test Py_FrozenMain() #26126 bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() #26201 bpo-44131: Fix Makefile for test_frozenmain #26203 bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI  #26241 [3.10] bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) #26353

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-25.12:52:42.813>
created_at = <Date 2021-05-14.12:12:49.610>
labels = ['expert-C-API', '3.11']
title = '[C API] Add tests on Py_FrozenMain()'
updated_at = <Date 2021-05-26.22:38:25.804>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-26.22:38:25.804>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-25.12:52:42.813>
closer = 'pablogsal'
components = ['C API']
creation = <Date 2021-05-14.12:12:49.610>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44131
keywords = ['patch']
message_count = 8.0
messages = ['393643', '393836', '393841', '393859', '393893', '394334', '394376', '394483']
nosy_count = 5.0
nosy_names = ['vstinner', 'petr.viktorin', 'corona10', 'pablogsal', 'miss-islington']
pr_nums = ['26126', '26201', '26203', '26241', '26353']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44131'
versions = ['Python 3.11']

[vstinner]

I would like to change Py_FrozenMain() using the new PyConfig C API, but I would prefer to first write tests of its current behavior. Currently, Py_FrozenMain() has no test.

Py_FrozenMain() is used by Tools/freeze/freeze.py. Old freeze.py issues:

• bpo-42613: freeze.py doesn't support sys.platlibdir different than lib nor multiarch
• bpo-40954: freeze.py aborts on macOS
• bpo-32217: freeze.py fails to work.
• bpo-27727: Update Tools/freeze to use .vcxproj files
• bpo-27566: Tools/freeze/winmakemakefile.py clean target should use 'del' instead of 'rm'
• bpo-26271: freeze.py makefile uses the wrong flags variables
• bpo-25504: undefined name 'modules' in Tools/freeze/checkextensions_win32.py
• bpo-24871: freeze.py doesn't work on x86_64 Linux out of the box
• bpo-23405: Tools/freeze "make" gets missing file error with unix shared builds
• bpo-21502: freeze.py not working properly with OS X framework builds
• bpo-16047: Tools/freeze no longer works in Python 3
• bpo-11824: freeze.py broken due to ABI flags
• bpo-7517: freeze.py not ported to python3
• bpo-6078: freeze.py doesn't work
• bpo-1985: Bug/Patch: Problem with xml/init.py when using freeze.py
• bpo-588452: $DEBUG_PYTHON -O freeze.py broken

I never used freeze.py, but since we get bug reports, it seems like it's used by users :-)

Note: freeze.py has no test neither.

Py_FrozenMain() has been modified recently in bpo-44113:

commit 7565586
Author: Dong-hee Na <donghee.na@python.org>
Date: Thu May 13 10:19:46 2021 +0900

bpo-44113: Update fromzenmain not to use Py_SetProgramName (GH-26085)

See also: [capi-sig] Py_FrozenMain and the stable ABI
https://mail.python.org/archives/list/capi-sig@python.org/thread/5QLI3NUP3OSGLCCIBAQOTX4GEJQBWJ6F/

See also: [Python-Dev] What's the story on Py_FrozenMain?
https://mail.python.org/pipermail/python-dev/2013-November/130280.html

[vstinner]

commit eaede0d
Author: Victor Stinner <vstinner@python.org>
Date: Mon May 17 23:48:35 2021 +0200

bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

[vstinner]

Compilation failed on s390x RHEL7 LTO + PGO 3.x buildbot:
https://buildbot.python.org/all/#/builders/244/builds/208

(...)
rm -f libpython3.11.a
ar rcs libpython3.11.a Modules/getbuildinfo.o (...) Python/frozen.o
lto1: error: libpython3.11.a: file too short
(null):0: confused by earlier errors, bailing out
lto-wrapper: gcc returned 1 exit status
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

make[3]: *** [python] Error 1
make[3]: *** Waiting for unfinished jobs....

gcc -pthread -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.11.a -lcrypt -lpthread -ldl -lutil -lm -lm
# Regenerate Programs/test_frozenmain.h
# from Programs/test_frozenmain.py
# using Programs/freeze_test_frozenmain.py
./python Programs/freeze_test_frozenmain.py Programs/test_frozenmain.h
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Programs/test_frozenmain.h written

make[4]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[3]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
make[2]: *** [build_all_generate_profile] Error 2
make[2]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[1]: *** [profile-gen-stamp] Error 2 make[1]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
make: *** [profile-run-stamp] Error 2
program finished with exit code 2
elapsedTime=62.919052

[encukou]

The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

IMO, Py_FrozenMain is quite specific: it's OK if it's exported, but also OK if it's missing. (Unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain.)

[vstinner]

The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

On Unix, _testembed gets the symbol since it's used. Currently, hasattr(ctypes.pythonapi, 'Py_FrozenMain') returns false if Python is built without --enable-shared, even if _testembed is able to get Py_FrozenMain().

I hope that bpo-44133 will be solved soon.

[pablogsal]

New changeset 68b9c20 by Miss Islington (bot) in branch '3.10':
bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) (GH-26353)
68b9c20

[vstinner]

commit eaede0d
Author: Victor Stinner <vstinner@python.org>
Date: Mon May 17 23:48:35 2021 +0200

bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

commit 58a9939
Author: Victor Stinner <vstinner@python.org>
Date: Thu May 20 12:08:05 2021 +0200

bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() (GH-26201)

Moreover, Py_FrozenMain() relies on Py_InitializeFromConfig() to
handle the PYTHONUNBUFFERED environment variable and configure C
stdio streams like stdout (make the stream unbuffered).

[vstinner]

I marked bpo-36940 as a duplicate of this issue.