Cannot install package with unicode module names on Windows

[julienmalard]

BPO	34421
Nosy	@jkloth, @merwok, @serhiy-storchaka, @dstufft, @miss-islington, @julienmalard
PRs	bpo-34421 Fix unicode error on Windows #8799 [3.6] bpo-34421 avoid unicode error in distutils logging (GH-8799) #9117 [3.7] bpo-34421 avoid unicode error in distutils logging (GH-8799) #9118 bpo-34421: Improve distutils logging for non-ASCII strings. #9126 [3.7] bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) #9503 [3.6] bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) #9504 [3.7] bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) #9506 [3.6] bpo-34421: Improve distutils logging for non-ASCII strings. (GH-9126) (GH-9506) #9510

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2018-09-23.11:11:42.033>
created_at = <Date 2018-08-17.17:06:14.932>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'Cannot install package with unicode module names on Windows'
updated_at = <Date 2018-09-23.11:11:42.032>
user = 'https://github.com/julienmalard'

bugs.python.org fields:

activity = <Date 2018-09-23.11:11:42.032>
actor = 'serhiy.storchaka'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2018-09-23.11:11:42.033>
closer = 'serhiy.storchaka'
components = ['Distutils']
creation = <Date 2018-08-17.17:06:14.932>
creator = 'julien.malard'
dependencies = []
files = []
hgrepos = []
issue_num = 34421
keywords = ['patch']
message_count = 14.0
messages = ['323699', '323714', '324861', '324862', '324863', '324874', '324878', '324888', '324889', '324907', '324921', '326137', '326143', '326145']
nosy_count = 6.0
nosy_names = ['jkloth', 'eric.araujo', 'serhiy.storchaka', 'dstufft', 'miss-islington', 'julien.malard']
pr_nums = ['8799', '9117', '9118', '9126', '9503', '9504', '9506', '9510']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34421'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

Please provide more details. How to reproduce your issue? What you got, and what you expect to get?

Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work?

[julienmalard]

Hello,

Yes, it does seem odd that that code does not work. On my Windows machine (WIndows 7, 64 bits, running 32-bit Python) I checked and it seems that the code in the if block immediately preceding my PR does not run at all, whereby the error.

For a reproducible example, my Taqdir package, mostly consisting of unicode packages and modules, runs into this issue (and installs successfully after my proposed fix here combined with a separate PR in pip). Perhaps the most easily accessible example would be the Appveyor build (https://ci.appveyor.com/project/julienmalard/Tinamit) for my TInamit project, which has Taqdir as a dependency.

Thanks!

-Julien Malard

---

દ્વારા: Serhiy Storchaka <report@bugs.python.org>
મોકલ્યું: 18 ઑગસ્ટ 2018 06:32
પ્રતિ: Julien Malard
વિષય: [bpo-34421] Cannot install package with unicode module names on Windows

New submission from Serhiy Storchaka <storchaka+cpython@gmail.com>:

Please provide more details. How to reproduce your issue? What you got, and what you expect to get?

Seems the code just before lines modified by your PR are purposed to solve this issue. Why it doesn't work?

----------
nosy: +serhiy.storchaka

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue34421\>

---

[merwok]

New changeset 0afada1 by Éric Araujo (Julien Malard) in branch 'master':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
0afada1

[miss-islington]

New changeset 3b36642 by Miss Islington (bot) in branch '3.6':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
3b36642

[miss-islington]

New changeset 77b92b1 by Miss Islington (bot) in branch '3.7':
bpo-34421 avoid unicode error in distutils logging (GH-8799)
77b92b1

[serhiy-storchaka]

I would prefer to use the backslashreplace error handler rather of the unicode-escape codec. Just as few lines above, but with ASCII encoding.

    msg = msg.encode('ascii', 'backslashreplace').decode('ascii')

It is still not clear to me why the current code purposed to handle this problem doesn't work in this case. We need to find the cause and fix the existing solution.

[jkloth]

The existing re-code solution is being triggered, as the errors in this case is 'surrogateescape' with an encoding of 'cp1252'.

Here, pip is using subprocess.Popen() to have Python run setup.py. During execution, a filename, 'taqdir\\\u0634\u0645\u0627\u0631.py', which has characters not encodable in cp1252.

I think that here, Python is not configuring its stdin/stdout/stderr streams correctly when run as a subprocess connected to pipes. Or, at least, subprocess.Popen() isn't passing the right (or enough) information to Python to get itself configured.

There should ultimately be a way to have Python (in a subprocess, on Windows) pass through Unicode untouched to its calling process. I suppose it would mean setting the PYTHONIOENCODING envvar when using subprocess.

After all that, it seems that:

1. pip needs to be changed to support calling Python subprocesses to enable lossless unicode transmission,
2. change the errors check in distutils.log to include 'surrogateescape'? (the heart of this issue)

[serhiy-storchaka]

PR 9126 makes distutils.log using "backslashreplace" instead of "unicode-escape" and simplifies the code (it is more efficient now, although the performance of logging is not critical).

"unicode-escape" escapes all non-ASCII characters, even encodable. It also escapes control characters like \t, \b, \r or \x1a (which starts control sequences for ANSI compatible terminals), this can be not desirable.