ensurepip/venv broken on Windows if path includes unicode

[dimaqq]

BPO	25758
Nosy	@brettcannon, @pfmoore, @ncoghlan, @vstinner, @tjguk, @ezio-melotti, @ericsnowcurrently, @dimaqq, @zware, @eryksun, @zooba, @dstufft, @jayvdb
Files	issue25758_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-09-10.00:35:46.571>
created_at = <Date 2015-11-28.09:56:52.995>
labels = ['expert-unicode', 'OS-windows']
title = 'ensurepip/venv broken on Windows if path includes unicode'
updated_at = <Date 2016-09-10.00:35:46.570>
user = 'https://github.com/dimaqq'

bugs.python.org fields:

activity = <Date 2016-09-10.00:35:46.570>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-09-10.00:35:46.571>
closer = 'steve.dower'
components = ['Unicode', 'Windows']
creation = <Date 2015-11-28.09:56:52.995>
creator = 'Dima.Tisnek'
dependencies = []
files = ['41185']
hgrepos = []
issue_num = 25758
keywords = ['patch']
message_count = 6.0
messages = ['255534', '255544', '274095', '274382', '275511', '275512']
nosy_count = 15.0
nosy_names = ['brett.cannon', 'paul.moore', 'ncoghlan', 'vstinner', 'tim.golden', 'ezio.melotti', 'python-dev', 'eric.snow', 'Dima.Tisnek', 'zach.ware', 'eryksun', 'steve.dower', 'dstufft', 'Marcus.Smith', 'jayvdb']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue25758'
versions = ['Python 3.5', 'Python 3.6']

[dimaqq]

One of my students installed Python 3.5 on Windows 10 to default location where user name "Łukasz" contains unicode.

Now "-m venv" and "-m ensurepip" do not work:

C:\Users\Łukasz>C:\Users\Łukasz\AppData\Local\Programs\Python\Python35-32\python.exe -m venv workshops
Error: Command '['C:\\Users\\\u0141ukasz\\workshops\\Scripts\\python.exe', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1

C:\Users\Łukasz>C:\Users\Łukasz\AppData\Local\Programs\Python\Python35-32\python.exe -m ensurepip
Traceback (most recent call last):
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__main__.py", line 4, in <module>
    ensurepip._main()
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 209, in _main
    default_pip=args.default_pip,
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 116, in bootstrap
    _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "C:\Users\\u0141ukasz\AppData\Local\Programs\Python\Python35-32\lib\ensurepip\__init__.py", line 40, in _run_pip
    import pip
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1136, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1093, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 444, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 530, in spec_from_file_location
UnicodeEncodeError: 'mbcs' codec can't encode characters in position 0--1: invalid character

[eryksun]

The problem is that the compile_source function in Modules/zipimport.c calls PyUnicode_EncodeFSDefault to get an encoded string to pass as the filename argument of Py_CompileString. On Windows this uses the ANSI codepage (i.e. 'mbcs'). Apparently your system's ANSI codepage doesn't map the "Ł" character.

I reproduced the problem more simply by copying pip-7.1.2-py2.py3-none-any.whl to a subdirectory named "Łukasz"; adding the wheel path to sys.path; and attempting to execute "import pip".

One solution is to replace Py_CompileString with Py_CompileStringObject. This way compile_source doesn't have to worry about encoding its pathname argument. A minimal patch is attached, but it needs a test.

[ncoghlan]

As a test case for handling non-ASCII characters in the name of the zipfile itself, I believe it should be sufficient to add 'Ł' to TEMP_DIR and TEMP_ZIP in https://hg.python.org/cpython/file/default/Lib/test/test_zipimport.py

[zooba]

Given a quick read, it looks like bpo-27781 (PEP-529) will resolve this?

Not encoding the path at all is obviously better, but maybe I'll add this as supporting evidence to the PEP...

[python-dev]

New changeset 663a62bcf9c9 by Steve Dower in branch '3.5':
Issue bpo-25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun)
https://hg.python.org/cpython/rev/663a62bcf9c9

New changeset ead30e7262d5 by Steve Dower in branch 'default':
Issue bpo-25758: Prevents zipimport from unnecessarily encoding a filename (patch by Eryk Sun)
https://hg.python.org/cpython/rev/ead30e7262d5

[zooba]

Applied Eryk's patch and updated the test to repro it (though it wouldn't repro on 3.6 with PEP-529 applied, but it definitely did on 3.5).