PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher

[ronaldoussoren]

BPO	34247
Nosy	@brettcannon, @ronaldoussoren, @ncoghlan, @vstinner, @ned-deily, @ericsnowcurrently, @miss-islington
PRs	bpo-34247: actually use environment variables in Py_Initialize #8521 [3.7] bpo-34247: Fix Python 3.7 initialization #8659 bpo-34247: add porting note to 3.7 What's New #9223 [3.7] bpo-34247: add porting note to 3.7 What's New (GH-9223) #9275 bpo-36142: PYTHONMALLOC overrides PYTHONDEV #12191
Files	t.c foo.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vstinner'
closed_at = <Date 2018-09-20.12:32:35.054>
created_at = <Date 2018-07-27.12:07:29.981>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'PYTHONOPTIMIZE ignored in 3.7.0 when using custom launcher'
updated_at = <Date 2019-03-06.11:51:55.385>
user = 'https://github.com/ronaldoussoren'

bugs.python.org fields:

activity = <Date 2019-03-06.11:51:55.385>
actor = 'vstinner'
assignee = 'vstinner'
closed = True
closed_date = <Date 2018-09-20.12:32:35.054>
closer = 'ncoghlan'
components = ['Interpreter Core']
creation = <Date 2018-07-27.12:07:29.981>
creator = 'ronaldoussoren'
dependencies = []
files = ['47716', '47717']
hgrepos = []
issue_num = 34247
keywords = ['patch', '3.7regression']
message_count = 27.0
messages = ['322487', '322519', '322520', '322521', '322522', '322523', '322524', '322529', '322532', '322533', '322550', '322562', '323068', '323070', '323135', '323144', '323145', '323156', '325172', '325281', '325283', '325288', '325289', '325862', '325874', '325877', '337294']
nosy_count = 7.0
nosy_names = ['brett.cannon', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'eric.snow', 'miss-islington']
pr_nums = ['8521', '8659', '9223', '9275', '12191']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34247'
versions = ['Python 3.7']

[ronaldoussoren]

Attached is a very basic launcher for python scripts. This script works as expected in Python 3.6, but not in 3.7.0.

In particular, the launcher sets the environment variable PYTHONOPTIMIZE before calling Py_Initialize and that setting is ignored.

I haven't tried testing with the tip of the tree yet.

[ncoghlan]

This feels like a duplicate of https://bugs.python.org/issue31845 (see d7ac061 )

Is this definitely with the 3.7.0 release, or is it potentially with one of the earlier alphas before that fix was applied?

[ncoghlan]

Ah, wait - I see a relevant difference: the test case for bpo-31845 uses Py_Main, *not* Py_Initialize (since it's a command line test, not an embedding test).

That means it's currently possible for the sequence of operations in

cpython/Python/pylifecycle.c

Line 913 in 7cbde0e

_Py_InitializeEx_Private(int install_sigs, int install_importlib)

to be wrong (or missing steps) without causing the test suite to fail.

[ncoghlan]

Comparing to Python 3.6, we can see that the code to update the C global flags from the environment variables used to live directly in _PyInitializeEx_Private:

cpython/Python/pylifecycle.c

Line 305 in 9d85856

_Py_InitializeEx_Private(int install_sigs, int install_importlib)

In Python 3.7 that section (https://github.com/python/cpython/blob/3.7/Python/pylifecycle.c#L913 ) currently calls _PyCoreConfig_Read instead, which *only* updates the config struct - it doesn't update the C level global variables.

By contrast, Py_Main eventually calls

cpython/Modules/main.c

Line 1394 in 7cbde0e

pymain_set_global_config(_PyMain *pymain, _Py_CommandLineDetails *cmdline)

which handles writing the state from the config struct back to the global variables.

[ncoghlan]

Next steps:

• add a test case that runs the same sys.flags checks as in d7ac061, but as a test_embed embedding test using Py_Initialize rather than Py_Main
• create a shared internal API that both Py_Main and Py_Initialize can use to write the environment variable settings back to the global variables

[ncoghlan]

Ned, I think we should consider this a release blocker for 3.7.1

[ncoghlan]

This seems closely related to the work Victor is already pursuing to resolve bpo-34170 for Python 3.8 (e.g. 56b29b6 ).

The bpo-34170 changes are more invasive than we'd like for a maintenance release, but it would be preferable to keep at least any new Python 3.7 test cases somewhat aligned with their Python 3.8 counterparts.

[ronaldoussoren]

I'm at the EuroPython sprints and currently working on a testcase for this. Should be a nice way to get back into actively contributing to CPython :-)

[ronaldoussoren]

FYI: I've filed bpo-34255 while working on this, test_embed assumes that you're building in the source directory (which I usually don't do).

[ronaldoussoren]

Interesting... pylifecycle.c uses code in main.c, I hadn't expected that.

If I've read the code correctly Py_Initialize won't look at PYTHONOPTIMZE at all (and other environment variables that have a command-line equivalent). Those are read in main.c:cmdline_get_env_flags, which is only called (indirectly) from Py_Main and _Py_UnixMain.

I'm note sure what's the correct solution for this.

1. Reading PYTHONOPTIMZE (and related variables) could be done
   in _PyCoreConfig_Read, but that would then need to grow a flag to ensure
   that this won't replace values calculated earlier in the implementation
   of Py_Main

2. Writing back the global variables is probably best done using a new
   function (lets _PyCoreConfig_SetGlobals), mirroring
   pymain_set_global_config but using a _PyCoreConfig config structure
   instead of the command-line structure used by the latter function.