Issue with pip in venv on Powershell in Windows

[crookedstorm]

BPO	37369
Nosy	@pfmoore, @vsajip, @tjguk, @ned-deily, @zware, @eryksun, @zooba, @gaborbernat, @crookedstorm, @miikama
PRs	bpo-37369: Fix initialization of sys members when launched via an app container #14428 [3.7] bpo-37369: Fixes path for sys.executable when running from the Microsoft Store #14450 [3.7] bpo-37369: Fix venv and test symlinking #14456 [3.7] bpo-37369: Fix path handling when python.exe is used as a symlink #14461 [3.8] bpo-37369: Fix initialization of sys members when launched via an app container (GH-14428) #14467 bpo-37369: Fixes crash when reporting fatal error #14468

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-06-30.04:49:04.140>
created_at = <Date 2019-06-21.23:54:16.077>
labels = ['deferred-blocker', 'type-bug', '3.8', '3.9', 'OS-windows']
title = 'Issue with pip in venv on Powershell in Windows'
updated_at = <Date 2020-08-26.08:13:49.241>
user = 'https://github.com/crookedstorm'

bugs.python.org fields:

activity = <Date 2020-08-26.08:13:49.241>
actor = 'eryksun'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-06-30.04:49:04.140>
closer = 'steve.dower'
components = ['Windows']
creation = <Date 2019-06-21.23:54:16.077>
creator = 'bstorm'
dependencies = []
files = []
hgrepos = []
issue_num = 37369
keywords = ['patch']
message_count = 27.0
messages = ['346264', '346265', '346658', '346683', '346685', '346694', '346761', '346783', '346786', '346837', '346849', '346877', '346890', '346899', '346900', '346915', '347160', '347374', '347376', '347384', '347499', '371655', '371759', '372436', '375874', '375893', '375920']
nosy_count = 11.0
nosy_names = ['paul.moore', 'vinay.sajip', 'tim.golden', 'ned.deily', 'zach.ware', 'eryksun', 'steve.dower', 'gaborjbernat', 'bstorm', 'miikama', 'federico2']
pr_nums = ['14428', '14450', '14456', '14461', '14467', '14468']
priority = 'deferred blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37369'
versions = ['Python 3.8', 'Python 3.9']

[crookedstorm]

I am finding that, using Powershell on Windows 10 and the current version of Python 3.7.3 installed from the Microsoft Store, when I create a virtualenv via "python -m venv <name>" and activate it in Powershell with the Activate.ps1 script that is generated, pip fails with:

pip freeze
Fatal error in launcher: Unable to create process using '"c:\users\<home>\<path-to-venv>\scripts\python.exe" "C:\Users\<home>\<path-to-venv>\Scripts\pip.exe" freeze'

There are no spaces in my python path, and I cannot find any "simple workaround" that actually works online. I am using pip 19.1.1 (current as of now). This only happens with Powershell. cmd.exe is able to use the virtualenv and pip just fine after using the activate.bat script.

If I activate in Powershell and run "python -m pip" or similar commands, including importing pip, I get a message that pip is not installed, which is interesting.

My Powershell version as installed is:
Major Minor Build Revision
----- ----- ----- --------
5 1 18917 1000

I also tried Powershell Core 6 with the same result. This seems to be a consistent behavior of the resultant Activate.ps1 script, which is why I'm creating the issue here. I didn't find quite a duplicate when I searched, but I could be wrong.

[crookedstorm]

I should add that, after testing a bit, it isn't actually working in cmd.exe. That is simply using the overarching python install. It's not using the virtualenv at all. The virtualenv that was created by the venv module appears to be non-functional at least as far as pip goes (which makes it hard to use at all).

[zooba]

Okay, this definitely used to work, but now it's broken for me too.

What version of Windows are you on (copy-paste from cmd.exe's ver command)?

[zooba]

This seems to be a change in Windows at some point, as it still works on one of my other PCs. I've pinged some colleagues to find out what might have happened, but it definitely looks like the lack of Read+Execute permission is working correctly now :(

One potential fix is to return the path under C:\Users\<username>\AppData\Local\Microsoft\WindowsApp as sys.executable (and sys.exec_prefix?), but it only contains the executable launchers and none of the actual Python install. Making this work for venv might require new values in pyvenv.cfg (however, it also seems that the current sys.executable changes with each update, so if we can change it universally to point at the user's WindowsApp directory then that will stop).

The good news is that if you're trying to launch it from within Python it still works, so multiprocessing is not affected. It's only things that write sys.executable out and try to use it later, like venv.

Adding Ned FYI, as if I can come up with a fix for this before we ship 3.7.4 I'd like to include it. No idea yet how possible that will be or what our fix will be though.

[ned-deily]

OK, I"m bumping it to "deferred blocker" for now so it remains visible.

[crookedstorm]

To answer Steve's question, the ver command gives me:
Microsoft Windows [Version 10.0.18917.1000]

Thank you for looking into this.

[zooba]

So I have a fix for 3.9 for this, I believe (involves making sys.base_executable configurable via PyConfig and overriding it and config->home when we know we're inside an app container).

For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

For 3.7, I'm going to have to come up with a different fix, and possibly expose some new private API to be able to pass through the correct executable path.

Fundamentally, the problem is that Windows is going to start enforcing some ACLs that were previously not being enforced. In theory, when an application is deployed in an app package (as required through the Store), you are not supposed to be able to launch the executable directly. Instead, you should go through your current user's apps directory (C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>, which contains only symlinks to the actual executables).

This has a very good advantage that the package family name does not change with new versions, while the full install path does. Currently, a behind-the-scenes update of Python will actually break environments when their "home" path is no longer correct.

However, it does mean that we need to start reporting a sys.executable that is not inside sys.prefix, and I'm not sure whether that is a good idea.

Looking at venv (+Vinay for confirmation here), it seems to treat "home" as "the directory containing the original Python binary" rather than "the directory containing the Python install". So in theory (and in my simple testing), using the sys.executable-not-in-prefix here is fine. But is that intentional?

[eryksun]

you should go through your current user's apps directory
(C:\Users\name\AppData\Local\Microsoft\WindowsApps\<package family>,
which contains only symlinks to the actual executables).

"%LocalAppData%\Microsoft\WindowsApps" contains IO_REPARSE_TAG_APPEXECLINK reparse points. An application-execution link is not like a regular symlink, i.e. it does not have the name-surrogate bit set in the tag value. In addition to the real application path, an app-exec link also contains the package information that the system uses to create the special access token that's required in order to execute the application.

I stepped through CreateProcessW in 10.0.18362.175 (slow ring), which has the old behavior. The first NtCreateUserProcess system call (the call that actually creates the Process object) fails with STATUS_IO_REPARSE_TAG_NOT_HANDLED. CreateProcessW handles this by getting the package information from the reparse point and creating a new token that contains 3 additional security attributes (WIN://SYSAPPID, WIN://PKG, and WIN://PKGHOSTID). For the subsequent NtCreateUserProcess system call, it uses the real path under "%ProgramFiles%\WindowsApps" and impersonates the new access token.

The real executable has two ACEs for standard users. There's a basic access-allowed ACE that grants read access, but does not grant execute access. There's also an access-allowed callback ACE that grants read and execute access. Normally the kernel security routines ignore callback ACEs in an access check, since they're intended for an application-defined callback in the user-mode AuthZ API. Apparently something is special-casing the access check in this case, to conditionally check the security attributes in the token in order to apply this callback ACE. If I remove the callback ACE, users can no longer execute python.exe, so I know this is the ACE that's granting execute access.

Back to CreateProcessW. If I try to directly execute the real executable in "%ProgramFiles%\WindowsApps", the NtCreateUserProcess system call fails with STATUS_ACCESS_DENIED, as expected. But then CreateProcessW does something that I didn't expect. It parses out the base filename "python.exe" and joins it to "%LocalAppData%\Microsoft\WindowsApps". If it finds the app-exec link, then it reads it to create the special package access token, as if we had run the app-exec link directly. If I had to guess, I would assume this behavior has been disabled in newer versions of Windows 10. I'm using the slow ring for now, so I don't have a way to test this guess.

[vstinner]

For 3.8 I can apply the same fix and make sys.base_executable public, or keep it as sys._base_executable. Either way, it'll show up in PyConfig. (Victor - thoughts here? Should be a non-breaking change, right?)

There is _PyConfig._config_version which is designed to support PyConfig change without losing ABI compatibility... but I didn't implement code for that yet :-)

If you have to modify PyConfig in 3.8, I suggest to directly use "base_executable" "public" name there (don't change it from _base_executable to base_executable from 3.8 to 3.9, it would be painful).

I'm not sure of the purpose of sys._base_executable. Maybe open a separated issue to make the attribute public? I'm not sure if multiprocessing (for example) should use it, or if sys.executble should be used.