[C API] Add tests on Py_FrozenMain()

@vstinner

BPO	44131
Nosy	@vstinner, @encukou, @corona10, @pablogsal, @miss-islington
PRs	bpo-44131: Test Py_FrozenMain() #26126 bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() #26201 bpo-44131: Fix Makefile for test_frozenmain #26203 bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI #26241 [3.10] bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) #26353

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-25.12:52:42.813>
created_at = <Date 2021-05-14.12:12:49.610>
labels = ['expert-C-API', '3.11']
title = '[C API] Add tests on Py_FrozenMain()'
updated_at = <Date 2021-05-26.22:38:25.804>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-26.22:38:25.804>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-25.12:52:42.813>
closer = 'pablogsal'
components = ['C API']
creation = <Date 2021-05-14.12:12:49.610>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 44131
keywords = ['patch']
message_count = 8.0
messages = ['393643', '393836', '393841', '393859', '393893', '394334', '394376', '394483']
nosy_count = 5.0
nosy_names = ['vstinner', 'petr.viktorin', 'corona10', 'pablogsal', 'miss-islington']
pr_nums = ['26126', '26201', '26203', '26241', '26353']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44131'
versions = ['Python 3.11']

I would like to change Py_FrozenMain() using the new PyConfig C API, but I would prefer to first write tests of its current behavior. Currently, Py_FrozenMain() has no test.

Py_FrozenMain() is used by Tools/freeze/freeze.py. Old freeze.py issues:

bpo-42613: freeze.py doesn't support sys.platlibdir different than lib nor multiarch
bpo-40954: freeze.py aborts on macOS
bpo-32217: freeze.py fails to work.
bpo-27727: Update Tools/freeze to use .vcxproj files
bpo-27566: Tools/freeze/winmakemakefile.py clean target should use 'del' instead of 'rm'
bpo-26271: freeze.py makefile uses the wrong flags variables
bpo-25504: undefined name 'modules' in Tools/freeze/checkextensions_win32.py
bpo-24871: freeze.py doesn't work on x86_64 Linux out of the box
bpo-23405: Tools/freeze "make" gets missing file error with unix shared builds
bpo-21502: freeze.py not working properly with OS X framework builds
bpo-16047: Tools/freeze no longer works in Python 3
bpo-11824: freeze.py broken due to ABI flags
bpo-7517: freeze.py not ported to python3
bpo-6078: freeze.py doesn't work
bpo-1985: Bug/Patch: Problem with xml/init.py when using freeze.py
bpo-588452: $DEBUG_PYTHON -O freeze.py broken

I never used freeze.py, but since we get bug reports, it seems like it's used by users :-)

Note: freeze.py has no test neither.

Py_FrozenMain() has been modified recently in bpo-44113:

commit 7565586
Author: Dong-hee Na <donghee.na@python.org>
Date: Thu May 13 10:19:46 2021 +0900

bpo-44113: Update fromzenmain not to use Py_SetProgramName (GH-26085)

See also: [capi-sig] Py_FrozenMain and the stable ABI
https://mail.python.org/archives/list/capi-sig@python.org/thread/5QLI3NUP3OSGLCCIBAQOTX4GEJQBWJ6F/

See also: [Python-Dev] What's the story on Py_FrozenMain?
https://mail.python.org/pipermail/python-dev/2013-November/130280.html

commit eaede0d
Author: Victor Stinner <vstinner@python.org>
Date: Mon May 17 23:48:35 2021 +0200

bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

Compilation failed on s390x RHEL7 LTO + PGO 3.x buildbot:
https://buildbot.python.org/all/#/builders/244/builds/208

(...)
rm -f libpython3.11.a
ar rcs libpython3.11.a Modules/getbuildinfo.o (...) Python/frozen.o
lto1: error: libpython3.11.a: file too short
(null):0: confused by earlier errors, bailing out
lto-wrapper: gcc returned 1 exit status
/usr/bin/ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

make[3]: *** [python] Error 1
make[3]: *** Waiting for unfinished jobs....

gcc -pthread -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.11.a -lcrypt -lpthread -ldl -lutil -lm -lm
# Regenerate Programs/test_frozenmain.h
# from Programs/test_frozenmain.py
# using Programs/freeze_test_frozenmain.py
./python Programs/freeze_test_frozenmain.py Programs/test_frozenmain.h
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Programs/test_frozenmain.h written

make[4]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[3]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
make[2]: *** [build_all_generate_profile] Error 2
make[2]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build' make[1]: *** [profile-gen-stamp] Error 2 make[1]: Leaving directory /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z.lto-pgo/build'
make: *** [profile-run-stamp] Error 2
program finished with exit code 2
elapsedTime=62.919052

The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

IMO, Py_FrozenMain is quite specific: it's OK if it's exported, but also OK if it's missing. (Unless you're compiling a special build with frozen modules, which won't work at all without Py_FrozenMain.)

The tests are now skipped on Windows. It might be better to use a feature check: hasattr(ctypes.pythonapi, 'Py_FrozenMain')

On Unix, _testembed gets the symbol since it's used. Currently, hasattr(ctypes.pythonapi, 'Py_FrozenMain') returns false if Python is built without --enable-shared, even if _testembed is able to get Py_FrozenMain().

I hope that bpo-44133 will be solved soon.

New changeset 68b9c20 by Miss Islington (bot) in branch '3.10':
bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI (GH-26241) (GH-26353)
68b9c20

commit eaede0d
Author: Victor Stinner <vstinner@python.org>
Date: Mon May 17 23:48:35 2021 +0200

bpo-44131: Test Py_FrozenMain() (GH-26126)

* Add test_frozenmain to test_embed
* Add [Programs/test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.py)
* Add [Programs/freeze_test_frozenmain.py](https://github.com/python/cpython/blob/main/Programs/freeze_test_frozenmain.py)
* Add [Programs/test_frozenmain.h](https://github.com/python/cpython/blob/main/Programs/test_frozenmain.h)
* Add make regen-test-frozenmain
* Add test_frozenmain command to [Programs/_testembed](https://github.com/python/cpython/blob/main/Programs/_testembed)
* _testembed.c: add error(msg) function

commit 58a9939
Author: Victor Stinner <vstinner@python.org>
Date: Thu May 20 12:08:05 2021 +0200

bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() (GH-26201)

Moreover, Py_FrozenMain() relies on Py_InitializeFromConfig() to
handle the PYTHONUNBUFFERED environment variable and configure C
stdio streams like stdout (make the stream unbuffered).

I marked bpo-36940 as a duplicate of this issue.

Navigation Menu

Repository navigation

[C API] Add tests on Py_FrozenMain() #88297

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[C API] Add tests on Py_FrozenMain() #88297

Description

Activity

vstinner commented on May 14, 2021

vstinner commented on May 17, 2021

vstinner commented on May 17, 2021

encukou commented on May 18, 2021

vstinner commented on May 18, 2021

pablogsal commented on May 25, 2021

vstinner commented on May 25, 2021

vstinner commented on May 26, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions