Wrong ANSI encoding used by subprocess for some locales

[segevfiner]

BPO	32370
Nosy	@pfmoore, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @segevfiner, @miss-islington
PRs	bpo-32370: Use the correct encoding for ipconfig output in the uuid module #5608 [3.7] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608) #5652 [3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). #5654

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-02-13.09:18:06.475>
created_at = <Date 2017-12-19.00:24:51.164>
labels = ['3.8', 'type-bug', '3.7', 'OS-windows']
title = 'Wrong ANSI encoding used by subprocess for some locales'
updated_at = <Date 2018-02-13.09:18:06.474>
user = 'https://github.com/segevfiner'

bugs.python.org fields:

activity = <Date 2018-02-13.09:18:06.474>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-02-13.09:18:06.475>
closer = 'serhiy.storchaka'
components = ['Windows']
creation = <Date 2017-12-19.00:24:51.164>
creator = 'Segev Finer'
dependencies = []
files = []
hgrepos = []
issue_num = 32370
keywords = ['patch']
message_count = 6.0
messages = ['308597', '308614', '311952', '312104', '312110', '312111']
nosy_count = 8.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'Segev Finer', 'miss-islington']
pr_nums = ['5608', '5652', '5654']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32370'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[segevfiner]

The following test is failing randomly for me (python -m test test_uuid -v):
======================================================================
ERROR: test_ipconfig_getnode (test.test_uuid.TestInternalsWithoutExtModule)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "cpython\lib\test\test_uuid.py", line 551, in test_ipconfig_getnode
    node = self.uuid._ipconfig_getnode()
  File "cpython\lib\uuid.py", line 487, in _ipconfig_getnode
    for line in pipe:
  File "cpython\lib\encodings\cp1255.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 682: character maps to <undefined>

This is caused by trying to decode the output of "ipconfig" using cp1255, while the output really uses cp862 and annoyingly it started to print times using the current locale (Which displays broken in the console anyhow, question mark boxes... *sigh*) in some Windows version (Using Windows 10.0.16299 ATM).

[eryksun]

ipconfig uses (or defaults to) OEM encoded output when writing to a pipe or file. On the other hand, Python's TextIOWrapper defaults to ANSI (i.e. 'mbcs'). In 3.6+, uuid._ipconfig_getnode could be rewritten to call subprocess.Popen with the new 'oem' encoding.

In Windows 10, the networking command-line utilities (e.g. ipconfig, netstat, nbtstat, ping, tracert, hostname, finger, and ftp) support an "OutputEncoding" environment variable. Its value can be set to "Unicode" (UTF-16), "UTF-8", or "Ansi". I don't think this is supported in Windows 7, however.

[zooba]

The patch looks good, but the rest of the code for this looks horrible(and potentially a security risk)... maybe we ought to take this fix for now but prioritise replacing this whole function with a C implementation or at least a better search for the ipconfig tool.

[serhiy-storchaka]

New changeset da6c3da by Serhiy Storchaka (Segev Finer) in branch 'master':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
da6c3da

[miss-islington]

New changeset 9b5a90b by Miss Islington (bot) in branch '3.7':
bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608)
9b5a90b

[serhiy-storchaka]

New changeset c3f9d7e by Serhiy Storchaka in branch '3.6':
[3.6] bpo-32370: Use the correct encoding for ipconfig output in the uuid module. (GH-5608). (bpo-5654)
c3f9d7e