Make it easier to use a venv with an embedded Python interpreter

[uellue]

BPO	35706
Nosy	@pfmoore, @ncoghlan, @tds333, @pitrou, @vstinner, @tjguk, @zware, @zooba, @uellue
Superseder	bpo-22213: Make pyvenv style virtual environments easier to configure when embedding Python
Files	Source.cpp

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-02-13.17:44:49.338>
created_at = <Date 2019-01-10.14:04:59.605>
labels = ['type-feature', 'OS-windows']
title = 'Make it easier to use a venv with an embedded Python interpreter'
updated_at = <Date 2019-02-13.17:44:49.337>
user = 'https://github.com/uellue'

bugs.python.org fields:

activity = <Date 2019-02-13.17:44:49.337>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-13.17:44:49.338>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2019-01-10.14:04:59.605>
creator = 'Dieter Weber'
dependencies = []
files = ['48040']
hgrepos = []
issue_num = 35706
keywords = []
message_count = 11.0
messages = ['333378', '334745', '334790', '334792', '334815', '334831', '334832', '334834', '335466', '335467', '335469']
nosy_count = 10.0
nosy_names = ['paul.moore', 'ncoghlan', 'tds333', 'pitrou', 'vstinner', 'tim.golden', 'pyscripter', 'zach.ware', 'steve.dower', 'Dieter Weber']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '22213'
type = 'enhancement'
url = 'https://bugs.python.org/issue35706'
versions = ['Python 3.6']

[uellue]

Python virtual environments are awesome! Using venvs with an embedded Python interpreter has proven difficult, unfortunately. With conda environments it works. See appended a sample file to reproduce the behavior.

The core of the problem seems to be that a venv doesn't contain a full Python installation, and Py_Initialize() apparently doesn't support setting up the combination of venv directories and base installation correctly, i.e. setting sys.prefix and sys.base_prefix and potentially other values.

Observed behavior when trying to use a venv:

"""
Initializing... Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00001e90 (most recent call first):
"""

Expected behavior:
Setting Py_SetPythonHome() to a venv works and sets up all paths and prefixes correctly to use the venv, just like it does for a conda environment.

[pyscripter]

Similar experience. I have found no way to get to use a venv as embedded Python in PyScripter. Tried Py_SetPythonhome, Py_SetPythonName, combinations... Nothing worked.

[tds333]

I think this pep is related to the issue and could be a solution:
https://www.python.org/dev/peps/pep-0582/

[pyscripter]

I don't see how pep-0582 is related to embedded python.

The solution is to check for the presence of pyvenv.cfg in PYTHONHOME and set up the system.path accordingly.

[zooba]

Py_Initialize() is not supposed to set it up. If you are embedding Python, you have to be responsible for configuring the paths that it uses - pyvenv.cfg is configuration for Python interactive, not the shared lib (using Windows terminology, it's configuration for python.exe but not python36.dll).

Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from. That said, I'm also in favor of making it easy for embedders to imitate the interactive shell's configuration.

Victor (nosied) and others are in the long process of updating embedded interpreter configuration to be easier to use, so adding him here as feedback, but I don't believe there is going to be a specific fix for this. Consider calling Py_SetPath before Py_Initialize to configure the search paths you want to use.

[pyscripter]

"Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from."

Why is that? When the embedding application is using Py_SetPythonHome there is clear intent to use a specific python setup. This is no different to using any other Python setup. Why venvs should be treated differently that standard installations?

As to manually setting the path, why pass the burden to the embedding application. The idea is that python, in the embedding application, should work as in the python executable. Trying to match the complex and evolving algorithms that python uses to setup the path, is a huge and unnecessary burden on the embedding application.

[zooba]

The idea is that python, in the embedding application, should work as in the python executable.

I disagree that this is the idea. Python in the embedding application should work as in the embedding application, not the standalone one. The embedding application should be able to easily opt-in to "standalone-like configuration", which is also what I said above, but right now that doesn't exist (it's under development).

Now, I'd be very much in favour of being able to embed Python without having to load any encodings. That ought to get us to the point where you can call "site.main()" in your embedding app, which does most of the work of what you consider the "default" behavior. But right now you just have to provide enough search path to initialize before doing that.

[pyscripter]

This issue is related (duplicate) to https://bugs.python.org/issue22213.
It appears that there is a workaround but only for Linux using Py_SetProgramName. I wish that there was some similar way for Windows.

[vstinner]

PEP-432 will allow to give with fine control on parameters used to initialize Python. Sadly, I failed to agree with Nick Coghlan and Eric Snow on the API. The current implementation (_PyCoreConfig and _PyMainInterpreterConfig) has some flaw (don't separate clearly the early initialization and Unicode-ready state, the interpreter contains main and core config whereas some options are duplicated in both configs, etc.).

[vstinner]

See also bpo-22213.

[zooba]

That's not a "see also" - it's just a duplicate :) I'll close this one as the other one has been around longer.