Repository navigation
Make it easier to use a venv with an embedded Python interpreter #79887
Description
| BPO | 35706 |
|---|---|
| Nosy | @pfmoore, @ncoghlan, @tds333, @pitrou, @vstinner, @tjguk, @zware, @zooba, @uellue |
| Superseder | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2019-02-13.17:44:49.338>
created_at = <Date 2019-01-10.14:04:59.605>
labels = ['type-feature', 'OS-windows']
title = 'Make it easier to use a venv with an embedded Python interpreter'
updated_at = <Date 2019-02-13.17:44:49.337>
user = 'https://github.com/uellue'bugs.python.org fields:
activity = <Date 2019-02-13.17:44:49.337>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-13.17:44:49.338>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2019-01-10.14:04:59.605>
creator = 'Dieter Weber'
dependencies = []
files = ['48040']
hgrepos = []
issue_num = 35706
keywords = []
message_count = 11.0
messages = ['333378', '334745', '334790', '334792', '334815', '334831', '334832', '334834', '335466', '335467', '335469']
nosy_count = 10.0
nosy_names = ['paul.moore', 'ncoghlan', 'tds333', 'pitrou', 'vstinner', 'tim.golden', 'pyscripter', 'zach.ware', 'steve.dower', 'Dieter Weber']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '22213'
type = 'enhancement'
url = 'https://bugs.python.org/issue35706'
versions = ['Python 3.6']
Activity
uellue commentedon Jan 10, 2019
Python virtual environments are awesome! Using venvs with an embedded Python interpreter has proven difficult, unfortunately. With conda environments it works. See appended a sample file to reproduce the behavior.
The core of the problem seems to be that a venv doesn't contain a full Python installation, and Py_Initialize() apparently doesn't support setting up the combination of venv directories and base installation correctly, i.e. setting sys.prefix and sys.base_prefix and potentially other values.
Observed behavior when trying to use a venv:
"""
Initializing... Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'
Current thread 0x00001e90 (most recent call first):
"""
Expected behavior:
Setting Py_SetPythonHome() to a venv works and sets up all paths and prefixes correctly to use the venv, just like it does for a conda environment.
[-]Making an embedded Python interpreter use a venv is difficult[/-][+]Make it easier to use a venv with an embedded Python interpreter[/+]pyscripter commentedon Feb 2, 2019
Similar experience. I have found no way to get to use a venv as embedded Python in PyScripter. Tried Py_SetPythonhome, Py_SetPythonName, combinations... Nothing worked.
tds333 commentedon Feb 3, 2019
I think this pep is related to the issue and could be a solution:
https://www.python.org/dev/peps/pep-0582/
pyscripter commentedon Feb 3, 2019
I don't see how pep-0582 is related to embedded python.
The solution is to check for the presence of pyvenv.cfg in PYTHONHOME and set up the system.path accordingly.
zooba commentedon Feb 4, 2019
Py_Initialize() is not supposed to set it up. If you are embedding Python, you have to be responsible for configuring the paths that it uses - pyvenv.cfg is configuration for Python interactive, not the shared lib (using Windows terminology, it's configuration for python.exe but not python36.dll).
Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from. That said, I'm also in favor of making it easy for embedders to imitate the interactive shell's configuration.
Victor (nosied) and others are in the long process of updating embedded interpreter configuration to be easier to use, so adding him here as feedback, but I don't believe there is going to be a specific fix for this. Consider calling Py_SetPath before Py_Initialize to configure the search paths you want to use.
pyscripter commentedon Feb 4, 2019
"Personally, I am 100% against letting an embedded runtime automatically pick up any settings from the environment. You don't know where they came from."
Why is that? When the embedding application is using Py_SetPythonHome there is clear intent to use a specific python setup. This is no different to using any other Python setup. Why venvs should be treated differently that standard installations?
As to manually setting the path, why pass the burden to the embedding application. The idea is that python, in the embedding application, should work as in the python executable. Trying to match the complex and evolving algorithms that python uses to setup the path, is a huge and unnecessary burden on the embedding application.
zooba commentedon Feb 4, 2019
I disagree that this is the idea. Python in the embedding application should work as in the embedding application, not the standalone one. The embedding application should be able to easily opt-in to "standalone-like configuration", which is also what I said above, but right now that doesn't exist (it's under development).
Now, I'd be very much in favour of being able to embed Python without having to load any encodings. That ought to get us to the point where you can call "site.main()" in your embedding app, which does most of the work of what you consider the "default" behavior. But right now you just have to provide enough search path to initialize before doing that.
pyscripter commentedon Feb 4, 2019
This issue is related (duplicate) to https://bugs.python.org/issue22213.
It appears that there is a workaround but only for Linux using Py_SetProgramName. I wish that there was some similar way for Windows.
vstinner commentedon Feb 13, 2019
PEP-432 will allow to give with fine control on parameters used to initialize Python. Sadly, I failed to agree with Nick Coghlan and Eric Snow on the API. The current implementation (_PyCoreConfig and _PyMainInterpreterConfig) has some flaw (don't separate clearly the early initialization and Unicode-ready state, the interpreter contains main and core config whereas some options are duplicated in both configs, etc.).
vstinner commentedon Feb 13, 2019
See also bpo-22213.
zooba commentedon Feb 13, 2019
That's not a "see also" - it's just a duplicate :) I'll close this one as the other one has been around longer.