venv activate.bat is UTF-8 encoded but uses current console codepage

[Jac0]

BPO	32409
Nosy	@pfmoore, @vsajip, @vstinner, @tjguk, @ezio-melotti, @zware, @eryksun, @zooba, @pablogsal, @miss-islington, @samstagern
PRs	bpo-32409: Ensures activate.bat can handle Unicode contents #5757 [3.7] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5765 [3.7] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5765 [3.6] bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757) #5766 bpo-32409 Ensures activate.bat can handle Unicode contents #10295 [3.7] bpo-32409: Fix regression in activate.bat on international Windows (GH-10295) #10377 bpo-32409: Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" #10403 [3.7] Revert "bpo-32409: Fix regression in activate.bat on international Windows (GH-10295)" (GH-10403) #10405 bpo-34144: Fix of venv acvtivate.bat for win 10 #8321

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2019-07-26.16:39:28.243>
created_at = <Date 2017-12-22.10:36:15.658>
labels = ['type-bug', '3.8', 'OS-windows', '3.7', 'library', 'expert-unicode']
title = 'venv activate.bat is UTF-8 encoded but uses current console codepage'
updated_at = <Date 2019-07-26.16:39:28.243>
user = 'https://bugs.python.org/Jac0'

bugs.python.org fields:

activity = <Date 2019-07-26.16:39:28.243>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2019-07-26.16:39:28.243>
closer = 'steve.dower'
components = ['Library (Lib)', 'Unicode', 'Windows']
creation = <Date 2017-12-22.10:36:15.658>
creator = 'Jac0'
dependencies = []
files = []
hgrepos = []
issue_num = 32409
keywords = ['patch']
message_count = 18.0
messages = ['308931', '308934', '312356', '312389', '312390', '312393', '329141', '329142', '329425', '329431', '329440', '329447', '329450', '329451', '329464', '329481', '332324', '348505']
nosy_count = 12.0
nosy_names = ['paul.moore', 'vinay.sajip', 'vstinner', 'tim.golden', 'ezio.melotti', 'zach.ware', 'eryksun', 'steve.dower', 'pablogsal', 'Jac0', 'miss-islington', 'Martin Bijl-Schwab']
pr_nums = ['5757', '5765', '5765', '5766', '10295', '10377', '10403', '10405', '8321']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32409'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[Jac0]

Let's say I have a folder c:\test-ä in Windows

Now if I run: py -m venv env
and activate: env\scripts\activate
and check: where python

the result is incorrectly just: C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

If I run: path
the result is: PATH=C:\test-ä\env\Scripts;...

So clearly the encoding is broken for the folder name.

I can fix this by changing activate.bat character encoding to OEM-US and then replacing "test-├ż" by "test-ä".

If I now activate and run: where python
the result is (as should be):
C:\test-ä\env\Scripts\python.exe
C:\Users\Username\AppData\Local\Programs\Python\Python36\python.exe

By running: path
I get: PATH=C:\test-ä\env\Scripts;...

So looks good here as well.

I suggest that what ever is creating activate.bat file, is using incorrect character encoding for the creation of the file. If this is somehow platform specific, there could be a guide in the venv documentation about how to fix this.

[eryksun]

The CMD shell decodes batch scripts using the attached console's output codepage, which defaults to OEM. OTOH, venv writes the replacement values for the template activate.bat as UTF-8 (codepage 65001), which is correct and should not be downgraded to OEM.

Instead, the batch script could temporarily switch the console to codepage 65001. Then restore the previous codepage at the end. For example:

@echo off
for /f "tokens=2 delims=:" %%a in ('"%SystemRoot%\System32\chcp.com"') do (
    set "CODEPAGE=%%a"
)
"%SystemRoot%\System32\chcp.com" 65001 > nul

[rest of script]

"%SystemRoot%\System32\chcp.com" %CODEPAGE% > nul
set "CODEPAGE="
:END

[zooba]

Eryk's solution seems to be best, so I'll add that.

[zooba]

New changeset 6240917 by Steve Dower in branch 'master':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5757)
6240917

[zooba]

New changeset a3d6c1b by Steve Dower (Miss Islington (bot)) in branch '3.7':
bpo-32409: Ensures activate.bat can handle Unicode contents (GH-5765)
a3d6c1b