Implement PEP 486 - Make the Python Launcher aware of virtual environments

[pfmoore]

BPO	23465
Nosy	@pfmoore, @vsajip, @tjguk, @zware, @zooba, @wm75
Files	pep486.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-03-25.06:36:23.474>
created_at = <Date 2015-02-14.14:59:15.354>
labels = ['type-feature', 'OS-windows']
title = 'Implement PEP 486 - Make the Python Launcher aware of virtual environments'
updated_at = <Date 2015-03-25.06:36:23.474>
user = 'https://github.com/pfmoore'

bugs.python.org fields:

activity = <Date 2015-03-25.06:36:23.474>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-03-25.06:36:23.474>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2015-02-14.14:59:15.354>
creator = 'paul.moore'
dependencies = []
files = ['38139']
hgrepos = []
issue_num = 23465
keywords = ['patch']
message_count = 15.0
messages = ['235974', '235977', '236008', '236290', '236293', '236299', '236300', '236305', '236310', '236391', '236394', '236708', '236709', '236710', '236711']
nosy_count = 7.0
nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'python-dev', 'zach.ware', 'steve.dower', 'wolma']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23465'
versions = ['Python 3.5']

[pfmoore]

Implementation of PEP-486 (Make the Python Launcher aware of virtual environments). Tested manually on my local PC - there aren't currently any tests for the launcher that I can see (and I'm not entirely sure how I'd write such a test) so the patch includes code and docs but no tests.

[zooba]

I don't think there are any tests for the launcher at all, though it would be fairly simple to write a Python script that runs py -c "import sys; print(sys.prefix)" and checks the output.

The patch looks fine to me, once I noticed that venv_python is a static local :). When the PEP is accepted I'll apply the patch (I don't think I've been appointed BDFL-delegate for this yet as Nick suggested, but if I am then I don't see any reason not to accept it).

[vsajip]

The patch looks good to me, too.

The standalone launcher has basic tests:

https://bitbucket.org/pypa/pylauncher/src/4613e10e26a8ca98d4fa4609c6659ef6b623baef/tests.py?at=default

To do a proper job, you need to have multiple Pythons installed, ideally 32- and 64-bit - so it's not really a job for the standard Python test suite.

[wm75]

am I correct that when a script contains a shebang line like:

#! python3

or

#! python3.4

i.e., one indicating just a version of, but not a full path to the interpreter, the current patch would not use an active virtualenv even if it has a suitable version ?

If so, is that a desirable behaviour?
If I've misread the patch and the PEP, then sorry for the noise.

[pfmoore]

That's correct. The problem here is that it's not possible to know what version of Python a virtualenv has (at least, not without running it, which isn't appropriate in the launcher). So the only case it's possible to support is #!python.

[wm75]

isn't the pyvenv.cfg file specifying the version ?

[pfmoore]

Hmm, I didn't know that (although virtualenv-based environments don't have an equivalent to pyvenv.cfg).

But there's some confusion here. This patch only affects command line usage (running "py.exe" to start Python). I don't really see a use case for making "py -3" mean "the virtualenv if it's Python 3 otherwise ignore the virtualenv and use the system Python 3".

For scripts, shebang processing isn't altered. Nothing uses a virtualenv except "#!/usr/bin/env python". And that does so only because it searches PATH before looking at the registry. If you use "#!/usr/bin/env python3" it won't see a virtualenv because there is no python3.exe in a virtualenv...

The scope of this PEP is just to make the "py" command (with no explicit version) use an active virtualenv before falling back to the default Python. This is specifically to allow people who don't put Python on their PATH but use virtualenvs to use "py" consistently, rather than having to switch to "python" when they are in a virtualenv. See the PEP (specifically the rationale section) for details.

[wm75]

Hmm, I didn't know that (although virtualenv-based environments don't have an equivalent to pyvenv.cfg).

Well, that complicates things then :(

But there's some confusion here. This patch only affects command line usage (running "py.exe" to start Python). I don't really see a use case for making "py -3" mean "the virtualenv if it's Python 3 otherwise ignore the virtualenv and use the system Python 3".

For scripts, shebang processing isn't altered. Nothing uses a virtualenv except "#!/usr/bin/env python". And that does so only because it searches PATH before looking at the registry. If you use "#!/usr/bin/env python3" it won't see a virtualenv because there is no python3.exe in a virtualenv...

Well, and that's exactly what I think is a mistake ...

The scope of this PEP is just to make the "py" command (with no explicit version) use an active virtualenv before falling back to the default Python. This is specifically to allow people who don't put Python on their PATH but use virtualenvs to use "py" consistently, rather than having to switch to "python" when they are in a virtualenv. See the PEP (specifically the rationale section) for details.

Right, just that

#!/usr/bin/env python3

is a very plausible shebang line for a Python3 script for use under UNIX where #!/usr/bin/env python typically means python2.

So, with the current patch users could still not use the py launcher from a virtual environment with scripts that are supposed to work under UNIX :(

[pfmoore]

On 20 February 2015 at 16:31, Wolfgang Maier <report@bugs.python.org> wrote:

> The scope of this PEP is just to make the "py" command (with no explicit version) use an active virtualenv before falling back to the default Python. This is specifically to allow people who don't put Python on their PATH but use virtualenvs to use "py" consistently, rather than having to switch to "python" when they are in a virtualenv. See the PEP (specifically the rationale section) for details.
>

Right, just that

#!/usr/bin/env python3

is a very plausible shebang line for a Python3 script for use under UNIX where #!/usr/bin/env python typically means python2.

That seems completely reasonable. Presumably this works for Unix
because virtualenvs have a "python3" executable installed, not just a
"python" executable?

So, with the current patch users could still not use the py launcher from a virtual environment with scripts that are supposed to work under UNIX :(

Correct. That's not the problem this PEP is intended to solve. Another
PEP could be written to look at this, but I suspect it could be quite
hard to balance the various issues involved. As a start, maybe the
Python Windows installer should be writing a "python3.exe" as well as
"python.exe" and venv should put that in the virtualenv. (That's
certainly something that has been suggested in the past). That may be
a simpler solution than adding yet more complexity and special cases
to the launcher.

Personally, I don't have a need for this functionality, so I'm happy
to leave that PEP to someone else to write.

[wm75]

> So, with the current patch users could still not use the py launcher from a virtual environment with scripts that are supposed to work under UNIX :(

Correct. That's not the problem this PEP is intended to solve.

Granted :)

Another PEP could be written to look at this, but I suspect it could be quite
hard to balance the various issues involved.

Well, changes to the launcher will not typically break any existing code, but constant changes in behaviour may still be a cause of confusion and be hard to explain, à la:

"since Python 3.5 the launcher supports virtual environments, but the details of this support differ between 3.5 and 3.6, e.g. in 3.5, scripts with a python3 shebang line will be executed by the system python3 independent of the virtual environment, but in 3.6 a virtual environment python3 interpreter will be used."

IMO it would be preferable to have all virtualenv related changes happening with one release.

As a start, maybe the
Python Windows installer should be writing a "python3.exe" as well as
"python.exe" and venv should put that in the virtualenv. (That's
certainly something that has been suggested in the past). That may be
a simpler solution than adding yet more complexity and special cases
to the launcher.

As an alternative, virtualenv could be changed to create a pyvenv.cfg file with the interpreter version like pyvenv does. Seems pretty simple and unproblematic to me and it might actually be useful to know the interpreter version without running it in other places besides the launcher. What do the pypa devs think about this?

[pfmoore]

As an alternative, virtualenv could be changed to create a pyvenv.cfg file with the interpreter version like pyvenv does. Seems pretty simple and unproblematic to me and it might actually be useful to know the interpreter version without running it in other places besides the launcher. What do the pypa devs think about this?

As one of the pypa devs, I'm in favour of this. I'll probably
implement it in due course, time permitting. But of course, a PR from
someone else would be a help :-)

[pfmoore]

The PEP has now been accepted, and as far as I am aware this patch is complete. If anyone has any review comments, please speak up, otherwise I believe this is ready to be committed, if some kind soul is willing :-)