regrtest: win_utils decodes typeperf output from the wrong encoding (test suite broken due to cpu usage feature on win 10/ german)

[LorenzMende]

BPO	36670
Nosy	@pfmoore, @db3l, @vstinner, @tjguk, @zware, @eryksun, @zooba, @ammaraskar, @miss-islington, @LorenzMende, @skoslowski, @heckad
PRs	bpo-36670: fixed encoding issue with libregrtest on win systems #15488 bpo-36670: fixed encoding issue with libregrtest on win systems #16456 bpo-36670: Multiple regrtest bugfixes #16511 [3.8] bpo-36670: Multiple regrtest bugfixes (GH-16511) #16513 [3.7] bpo-36670: Multiple regrtest bugfixes (GH-16511) #16514 bpo-36670, regrtest: Fix TestWorkerProcess.stop() #16530 bpo-36670, regrtest: Fix repr(TestWorkerProcess) #16537 [3.8] bpo-36670: regrtest bug fixes (GH-16537) #16538 bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line #16550 bpo-36670: Enhance regrtest WindowsLoadTracker #16553 [3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) #16554 bpo-36670, regrtest: Fix WindowsLoadTracker sampling rate #16555 bpo-36670: regrtest WindowsLoadTracker computes LOAD_FACTOR_1 #16556 [3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) #16559 [3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) #16560 [3.7] [3.8] bpo-36670, regrtest: Fix WindowsLoadTracker() for partial line (GH-16550) (GH-16560) #16562
Files	win_utils.py: patched win_utils.py with registry lookup for counter name and oemdecoding

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-10-03.15:31:37.450>
created_at = <Date 2019-04-19.15:52:37.352>
labels = ['easy', 'type-bug', '3.8', '3.9', '3.7', 'tests', 'OS-windows']
title = 'regrtest: win_utils decodes typeperf output from the wrong encoding (test suite broken due to cpu usage feature on win 10/ german)'
updated_at = <Date 2020-04-14.23:05:24.337>
user = 'https://github.com/LorenzMende'

bugs.python.org fields:

activity = <Date 2020-04-14.23:05:24.337>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-10-03.15:31:37.450>
closer = 'vstinner'
components = ['Tests', 'Windows']
creation = <Date 2019-04-19.15:52:37.352>
creator = 'LorenzMende'
dependencies = []
files = ['48406']
hgrepos = []
issue_num = 36670
keywords = ['patch', 'easy', 'newcomer friendly']
message_count = 35.0
messages = ['340547', '340548', '340550', '340551', '340562', '340575', '340576', '340632', '340797', '340806', '344293', '345076', '348719', '353554', '353555', '353556', '353660', '353664', '353667', '353669', '353670', '353731', '353732', '353768', '353770', '353772', '353779', '353783', '353786', '353832', '353850', '353853', '353857', '353860', '366462']
nosy_count = 12.0
nosy_names = ['paul.moore', 'db3l', 'vstinner', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'ammar2', 'miss-islington', 'LorenzMende', 'skoslowski', 'heckad']
pr_nums = ['15488', '16456', '16511', '16513', '16514', '16530', '16537', '16538', '16550', '16553', '16554', '16555', '16556', '16559', '16560', '16562']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36670'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[LorenzMende]

The test suite fails with the first tests (I assume 1st call of getloadavg of WindowsLoadTracker).
Traceback (most recent call last):
  File "P:\Repos\CPython\cpython\lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "P:\Repos\CPython\cpython\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "P:\Repos\CPython\cpython\lib\test\__main__.py", line 2, in <module>
    main()
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 653, in main
    Regrtest().main(tests=tests, **kwargs)
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 586, in main
    self._main(tests, kwargs)
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 632, in _main
    self.run_tests()
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 515, in run_tests
    self.run_tests_sequential()
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 396, in run_tests_sequential
    self.display_progress(test_index, text)
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\main.py", line 150, in display_progress
    load_avg_1min = self.getloadavg()
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\win_utils.py", line 81, in getloadavg
    typeperf_output = self.read_output()
  File "P:\Repos\CPython\cpython\lib\test\libregrtest\win_utils.py", line 78, in read_output
    return response.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 67: invalid start byte
##########################################################
The windows 'typeperf "\System\Processor Queue Length" -si 1' command unluckily returns an string with an umlaut which leads to the Decode-Error. This comes up because the <counter> for the typeperf is location dependend. (In german the counter would read \System\Prozessor-Warteschlangenlänge)

I see two possible solutions to this issue.

1. Raising an exception earlier on creation of WindowsLoadTracker resulting in the same behaviour as if there is no typeperf available (german pythoneers would have a drawback with this)
2. Getting the typeperf counter correctly from registry (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\CurrentLanguage, described here https://social.technet.microsoft.com/Forums/de-DE/25bc6907-cf2c-4dc8-8687-974b799ba754/powershell-ausgabesprache-umstellen?forum=powershell_de)

environment:
Windows 10 x64, 1809, german
cpython @e16467af0bfcc9f399df251495ff2d2ad20a1669
commit of assumed root cause of https://bugs.python.org/issue34060

[ammaraskar]

What does typeperf "\System\Processor Queue Length" -si 1 actually return on your non-English system?

Does it just return an error with the counter's name or is the umalet just in the first header line, i.e this one for me:

"(PDH-CSV 4.0)","\\MSI\System\Processor Queue Length"

If it's the latter then I think the correct fix would be to figure out what encoding typeperf is outputting in and then just decode with that.

[LorenzMende]

Hi Ammar, you are correct.
typeperf returns:
P:\Repos\CPython\cpython>typeperf "\System\Prozessor-Warteschlangenlänge" -si 1

"(PDH-CSV 4.0)","\\ZERO\System\Prozessor-Warteschlangenlänge"
"04/19/2019 19:09:14.510","0.000000"
"04/19/2019 19:09:15.514","0.000000"

So even with correct counter name the outpu needs to be decoded correctly.
I already got a solution to get the location specific counter name from registry - if it helps I'll commit it.

[ammaraskar]

Thank you, could you also share the output if you just give it the English name of the counter?

[terryjreedy]

'crash' mean *nix coredump or Windows equivelent, rather than traceback.

[zooba]

It'll probably be mbcs or oem encoding. Certainly not UTF-8

[eryksun]

It'll probably be mbcs or oem encoding. Certainly not UTF-8

Unless the system locale's codepage is set to UTF-8 in Windows 10.

I stepped through this in the debugger. When writing to a pipe, typeperf.exe calls WideCharToMultiByte(CP_OEMCP, ...), so it's "oem".

[zooba]

If the code page is set to UTF-8 then mbcs will still be the right encoding :)

[LorenzMende]

Sorry, was off some days. I tried to decode the output with mbcs, solves the issue partly - the counter name is still wrong.
Was able to pick the localization specific counter name from registry and use it for the typeperf.

But the tests fail after several seconds with Broken Pipe Error of the command_stdout pipe.

@ammar - why is the handle closed in start()? If I uncomment it, the load tracker workks fine.

Added the fixed file, may someone evaluate the solution on another windows localization?