Test failures with non-ascii character in hostname on Windows

[Vgr255]

BPO	26226
Nosy	@pfmoore, @vstinner, @tjguk, @ezio-melotti, @bitdancer, @vadmium, @zware, @eryksun, @zooba, @Vgr255
Files	failed_tests_output.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-15.10:27:40.770>
created_at = <Date 2016-01-28.00:25:51.625>
labels = ['type-bug', 'OS-windows']
title = 'Test failures with non-ascii character in hostname on Windows'
updated_at = <Date 2019-01-15.10:27:40.769>
user = 'https://github.com/Vgr255'

bugs.python.org fields:

activity = <Date 2019-01-15.10:27:40.769>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-15.10:27:40.770>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2016-01-28.00:25:51.625>
creator = 'abarry'
dependencies = []
files = ['41733']
hgrepos = []
issue_num = 26226
keywords = []
message_count = 19.0
messages = ['259075', '259076', '259077', '259080', '259081', '261655', '269420', '269423', '269431', '269436', '269437', '269455', '269462', '269463', '269609', '271391', '271397', '333659', '333666']
nosy_count = 11.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'ezio.melotti', 'r.david.murray', 'martin.panter', 'zach.ware', 'eryksun', 'steve.dower', 'abarry', 'Jack01']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26226'
versions = ['Python 3.6']

[Vgr255]

Compiled latest master and ran test suite (Py_DEBUG build). A few failures here and there, and some tests skipped. I haven't yet looked into getting the proper libraries to make some of the skipped tests execute, I'm trying to make the whole test suite pass first.

Attached file includes verbose output for all failed tests. The end of the file and the traceback at the end of the message here happen, then Python hangs and never resumes (tested for ~20 minutes).

Python 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)] on w
in32
Type "help", "copyright", "credits" or "license" for more information.
>>> import test.regrtest
>>> test.regrtest.main_in_temp_cwd() # same as 'py -m test'
== CPython 3.6.0a0 (default, Jan 27 2016, 10:49:09) [MSC v.1900 32 bit (Intel)]
==   Windows-7-6.1.7601-SP1 little-endian
==   hash algorithm: siphash24 32bit
==   E:\GitHub\cpython\build\test_python_13304
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0,
bytes_warning=0, quiet=0, hash_randomization=1, isolated=0)
[  1/400] test_grammar
[  2/400] test_opcodes
[  3/400] test_dict
[  4/400] test_builtin
[  5/400] test_exceptions
[  6/400] test_types
[  7/400] test_unittest
[  8/400] test_doctest
[  9/400] test_doctest2
[ 10/400] test_support
[ 11/400] test___all__
[ 12/400] test___future__
[ 13/400] test__locale
[ 14/400] test__opcode
[ 15/400] test__osx_support
[ 16/400] test_abc
[ 17/400] test_abstract_numbers
[ 18/400] test_aifc
[ 19/400] test_argparse
[ 20/400] test_array
[ 21/400] test_asdl_parser
[ 22/400] test_ast
[ 23/400] test_asynchat
[ 24/400] test_asyncio
E:\GitHub\cpython\lib\asyncio\sslproto.py:327: ResourceWarning: unclosed transport
<asyncio.sslproto._SSLProtocolTransport object at 0x05EA8BD0>
  warnings.warn("unclosed transport %r" % self, ResourceWarning)
test test_asyncio failed -- multiple errors occurred; run in verbose mode for details
[ 25/400/1] test_asyncore
[ 26/400/1] test_atexit
[ 27/400/1] test_audioop
[ 28/400/1] test_augassign
[ 29/400/1] test_base64
[ 30/400/1] test_bigaddrspace
[ 31/400/1] test_bigmem
[ 32/400/1] test_binascii
[ 33/400/1] test_binhex
[ 34/400/1] test_binop
[ 35/400/1] test_bisect
[ 36/400/1] test_bool
[ 37/400/1] test_buffer
[ 38/400/1] test_bufio
[ 39/400/1] test_bytes
[ 40/400/1] test_bz2
[ 41/400/1] test_calendar
[ 42/400/1] test_call
[ 43/400/1] test_capi
[ 44/400/1] test_cgi
[ 45/400/1] test_cgitb
[ 46/400/1] test_charmapcodec
[ 47/400/1] test_class
[ 48/400/1] test_cmath
[ 49/400/1] test_cmd
[ 50/400/1] test_cmd_line
[ 51/400/1] test_cmd_line_script
[ 52/400/1] test_code
[ 53/400/1] test_code_module
test test_code_module failed -- multiple errors occurred; run in verbose mode for details
[ 54/400/2] test_codeccallbacks
[ 55/400/2] test_codecencodings_cn
[ 56/400/2] test_codecencodings_hk
[ 57/400/2] test_codecencodings_iso2022
test test_codecencodings_iso2022 failed -- multiple errors occurred; run in verbose mode for details
[ 58/400/3] test_codecencodings_jp
[ 59/400/3] test_codecencodings_kr
[ 60/400/3] test_codecencodings_tw
[ 61/400/3] test_codecmaps_cn
[ 62/400/3] test_codecmaps_hk
[ 63/400/3] test_codecmaps_jp
[ 64/400/3] test_codecmaps_kr
[ 65/400/3] test_codecmaps_tw
[ 66/400/3] test_codecs
[ 67/400/3] test_codeop
[ 68/400/3] test_collections
[ 69/400/3] test_colorsys
[ 70/400/3] test_compare
[ 71/400/3] test_compile
[ 72/400/3] test_compileall
[ 73/400/3] test_complex
[ 74/400/3] test_concurrent_futures
[ 75/400/3] test_configparser
[ 76/400/3] test_contains
[ 77/400/3] test_contextlib
[ 78/400/3] test_copy
[ 79/400/3] test_copyreg
[ 80/400/3] test_coroutines
[ 81/400/3] test_cprofile
[ 82/400/3] test_crashers
[ 83/400/3] test_crypt
test_crypt skipped -- No module named '_crypt'
[ 84/400/3] test_csv
[ 85/400/3] test_ctypes
[ 86/400/3] test_curses
test_curses skipped -- Use of the 'curses' resource not enabled
[ 87/400/3] test_datetime
[ 88/400/3] test_dbm
[ 89/400/3] test_dbm_dumb
[ 90/400/3] test_dbm_gnu
test_dbm_gnu skipped -- No module named '_gdbm'
[ 91/400/3] test_dbm_ndbm
test_dbm_ndbm skipped -- No module named '_dbm'
[ 92/400/3] test_decimal
[ 93/400/3] test_decorators
[ 94/400/3] test_defaultdict
[ 95/400/3] test_deque
[ 96/400/3] test_descr
[ 97/400/3] test_descrtut
[ 98/400/3] test_devpoll
test_devpoll skipped -- test works only on Solaris OS family
[ 99/400/3] test_dictcomps
[100/400/3] test_dictviews
[101/400/3] test_difflib
[102/400/3] test_dis
[103/400/3] test_distutils

E:\GitHub\cpython\build\test_python_13304>exit 1

E:\GitHub\cpython\build\test_python_13304>exit 0
Warning -- files was modified by test_distutils
test test_distutils failed -- multiple errors occurred; run in verbose mode for details
[104/400/4] test_docxmlrpc
[105/400/4] test_dummy_thread
[106/400/4] test_dummy_threading
[107/400/4] test_dynamic
[108/400/4] test_dynamicclassattribute
[109/400/4] test_eintr
[110/400/4] test_email
[111/400/4] test_ensurepip
[112/400/4] test_enum
[113/400/4] test_enumerate
[114/400/4] test_eof
[115/400/4] test_epoll
test_epoll skipped -- test works only on Linux 2.6
[116/400/4] test_errno
[117/400/4] test_exception_variations
[118/400/4] test_extcall
[119/400/4] test_faulthandler
[120/400/4] test_fcntl
test_fcntl skipped -- No module named 'fcntl'
[121/400/4] test_file
[122/400/4] test_file_eintr
[123/400/4] test_filecmp
[124/400/4] test_fileinput
[125/400/4] test_fileio
[126/400/4] test_finalization
[127/400/4] test_float
[128/400/4] test_flufl
[129/400/4] test_fnmatch
[130/400/4] test_fork1
test_fork1 skipped -- object <module 'os' from 'E:\\GitHub\\cpython\\lib\\os.py'> has no attribute 'fork'
[131/400/4] test_format
[132/400/4] test_fractions
[133/400/4] test_frame
[134/400/4] test_fstring
[135/400/4] test_ftplib
[136/400/4] test_funcattrs
[137/400/4] test_functools
[138/400/4] test_future
[139/400/4] test_future3
[140/400/4] test_future4
[141/400/4] test_future5
[142/400/4] test_gc
[143/400/4] test_gdb
test_gdb skipped -- Couldn't find gdb on the path
[144/400/4] test_generators
[145/400/4] test_genericpath
[146/400/4] test_genexps
[147/400/4] test_getargs2
[148/400/4] test_getopt
[149/400/4] test_getpass
[150/400/4] test_gettext
[151/400/4] test_glob
[152/400/4] test_global
[153/400/4] test_grp
test_grp skipped -- No module named 'grp'
[154/400/4] test_gzip
[155/400/4] test_hash
[156/400/4] test_hashlib
[157/400/4] test_heapq
[158/400/4] test_hmac
[159/400/4] test_html
[160/400/4] test_htmlparser
[161/400/4] test_http_cookiejar
[162/400/4] test_http_cookies
[163/400/4] test_httplib
test test_httplib failed -- multiple errors occurred; run in verbose mode for details
[164/400/5] test_httpservers
Exception in thread Thread-857:
Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "E:\GitHub\cpython\lib\test\test_httpservers.py", line 42, in run
    self.server = HTTPServer(('localhost', 0), self.request_handler)
  File "E:\GitHub\cpython\lib\socketserver.py", line 443, in __init__
    self.server_bind()
  File "E:\GitHub\cpython\lib\http\server.py", line 140, in server_bind
    self.server_name = socket.getfqdn(host)
  File "E:\GitHub\cpython\lib\socket.py", line 662, in getfqdn
    hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 0: invalid
continuation byte

[vstinner]

For the test_httpservers failure, can you please try the following commands on your PC?

>>> socket.gethostname()
'selma'
>>> socket.gethostbyaddr(socket.gethostname())
('selma', [], ['2a01:e34:ec8d:4c70:3ea9:f4ff:fe65:c0c', 'fe80::3ea9:f4ff:fe65:c0c'])

[Vgr255]

Well, it has a non-ASCII character in it, so I wouldn't be surprised if this was the issue :)

Latest master (3.6):

>>> import socket
>>> socket.gethostname()
'Émanuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno 11004] getaddrinfo failed

Python 2.7.11:

>>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'
>>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104'])

[vstinner]

I'm surprised by the test_codecencodings_iso2022 failures. Example:

======================================================================
FAIL: test_incrementaldecoder (test.test_codecencodings_iso2022.Test_ISO2022_KR)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in test_incrementaldecoder
    self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'

It looks like Python uses the wrong encoding to decode stdout of child processes:

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\distutils\tests\test_build_ext.py", line 61, in test_build_ext
    cmd.run()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 338, in run
    self.build_extensions()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 447, in build_extensions
    self._build_extensions_serial()
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 472, in _build_extensions_serial
    self.build_extension(ext)
  File "E:\GitHub\cpython\lib\distutils\command\build_ext.py", line 532, in build_extension
    depends=ext.depends)
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 306, in compile
    self.initialize()
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 199, in initialize
    vc_env = _get_vc_env(plat_spec)
  File "E:\GitHub\cpython\lib\distutils\_msvccompiler.py", line 92, in _get_vc_env
    universal_newlines=True,
  File "E:\GitHub\cpython\lib\subprocess.py", line 636, in check_output
    **kwargs).stdout
  File "E:\GitHub\cpython\lib\subprocess.py", line 705, in run
    stdout, stderr = process.communicate(input, timeout=timeout)
  File "E:\GitHub\cpython\lib\subprocess.py", line 1062, in communicate
    stdout = self.stdout.read()
  File "E:\GitHub\cpython\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 49: character maps to <undefined>

It would help to add a print() to lib\distutils\_msvccompiler.py:92 to get the executed command, then run manually the command, and see non-ASCII characters in the output.

[vstinner]

Python 2.7.11:

>>> import socket
>>> socket.gethostname()
'\xc9manuel-PC'

This one works on Python 3 because the Python function is implemented with a call to the Windows native API.

>>> socket.gethostbyaddr(socket.gethostname())
('\xc9manuel-PC.home', [], ['fe80::c9b7:5117:eea4:a104'])

This one fails on Python 3 because it uses the gethostbyaddr() C function and then decodes the hostname from UTF-8, whereas the hostname looks more to be encoded to ISO 8859-1 or something like that. IMHO it should be decoded from the ANSI code page.

I opened the issue bpo-26227 to track this bug.

[ezio-melotti]

I'm surprised by the test_codecencodings_iso2022 failures. Example:

======================================================================
FAIL: test_incrementaldecoder (test.test_codecencodings_iso2022.Test_ISO2022_KR)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "E:\GitHub\cpython\lib\test\multibytecodec_support.py", line 208, in test_incrementaldecoder
    self.assertEqual(ostream.getvalue(), self.tstring[1])
AssertionError: b'\xe[334 chars]\x80\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\xec[1668 chars]4.\n' != b'\xe[334 chars]\x80\r\n\xed\x9a\xa8\xec\x9c\xa8\xec\xa0\x81\x[1682 chars]\r\n'

I run into this problem during a sprint, with a machine that was also running Windows and cloned from the git repo. Those bytes are read from txt files in the test/cjkencodings dir, and they are marked as binary files in .hgeol. Since git ignores .hgeol and treats them as text files, it also changes the newline to \r\n causing the failure.
The tests pass without problems after cloning with Mercurial.

[vadmium]

Regarding the files getting messed up in Lib/test/cjkencodings, IMO the solution is to not configure Git to mess with newlines in files. ;) But perhaps a reasonable workaround might be to rename the relevant files to something like *.txt.bin. Perhaps that will bypass whatever logic Git uses.

[vstinner]

It looks like all issues are now fixed in the default branch, no? Can we close this issue?

[bitdancer]

It looks like this issue can be closed, but a new one should be opened for dealing with the newline issue under git on windows. However, I'm surprised there weren't failures in the email tests. Although I've been moving the email tests away from depending on data on the disk being in DOS format, I think there are some compat32 tests that still do so.

[Vgr255]

Some of the tests that used to fail are now passing, however some are still failing (and new ones are also failing now).

Tests that are still failing:

test_code_module (TestInteractiveConsole; test_ps1 and test_ps2 failed)
test_codecencodings_iso2022 (blame git for changing line endings; will open a new issue about this)

New failures:

test_capi (_testcapi doesn't have test_buildvalue_N)
test_codecencodings_jp (passes when I run it by itself... ?)
test_ctypes (AssertionError: "phi" does not match "duplicate values for field '???'")
test_decimal (will open a new issue for that one, there are over 300 lines of failures/errors)
test_distutils (env changed, also no tests run when I try to run it by itself)
test_getargs2 (fails on import with 'from _testcapi import getargs_positional_only_and_keywords as getargs'; ImportError: cannot import name 'getargs_positional_only_and_keywords')

Tests previously hung up after that point. The tests continue now, so here are more failures:

test_logging (socket timeout)
test_os (many failed C++ assertions! [Visual C++ Runtime library] Will open a new issue about that)
test_random (line endings mismatch)
test_sax (again line endings mismatch)
test_sqlite (sqlite3.test.dbapi.CursorTests.CheckLastRowIDOnReplace fails with self.cu.lastrowid =! 1 [it's None])
test_strptime (that's a DST issue: AssertionError: False is not true : timezone est (heure d?été) not found in (frozenset({'gmt', 'est', 'utc'}), frozenset({'est (heure d\x92été)'})))
test_uuid (sounds like it's related to my hostname having non-ASCII characters)
test_warnings ('__main__' != '' [I'm running tests via the interactive prompt, this might be why])
test_xml_etree_c ('Windows fatal error: stack overflow' in the test that makes sure recursive repr doesn't overflow [but it doesn't fail when I run it individually, apparently])

I'll spawn new issues for the more serious/specific ones, but unless you'd rather have one issue per failure, I'd like to keep this one open (for now at least).

@r. David: 4 skips, but no failures in the email tests, so at least there's that!

[Vgr255]

About the strptime failure - it seems time.strftime returns "Est (heure d?été)" while _strptime.LocaleTime().timezone has "est (heure d\x92été)". I don't know why it's like that, but neither are correct - the character that goes there is a single quote ('). I'm also puzzled by the fact that 'é' appears properly despite not being ASCII.

[vadmium]

test_distutils: Is the failure the same as before, or changed? Are you testing with new code that includes the bpo-27048 fix? It looks like test_distutils doesn’t support running directly via “unittest”. Perhaps try “python -m test -W test_distutils” or similar if you were previously only trying unittest.

Some of your new failures look like symptoms of running the test suite on out-of-date compiled modules. For test_ctypes, see bpo-27343; test_getargs2: bpo-26282; test_capi: bpo-26168.

strptime: Maybe the test suite is trying to report a string that contains both U+0092 and “é”, but cannot encode U+0092, and uses backslashreplace error handler.

[eryksun]

time.strftime calls the CRT's strftime function, which the Windows universal CRT implements by calling wcsftime and encoding the result. The timezone name is actually stored as a char string (tzname), so wcsftime has to decode it via mbstowcs.

The problem is that in the C locale tzname is an ANSI (1252) string while mbstowcs simply casts to wchar_t, which is the same as decoding as Latin-1. This works fine for "é" (U+00E9). But the right single quote character (U+2019) is "\x92" in 1252, and a simple cast maps it to the non-character U+0092.

When the CRT's strftime encodes this back as an ANSI string, it maps U+0092 to the replacement character for 1252, a question mark. Similarly, time.tzname decodes the tzname ANSI strings using mbstowcs, with the same mismatch between LC_CTYPE and LC_TIME, resulting in the string "Est (heure d\x92été)"

In summary, the problem is that LC_TIME uses ANSI in the C locale, while LC_CTYPE uses Latin-1. A workaround (in most cases) is to delay importing the time module until after setting LC_CTYPE (also setting LC_TIME should cover all cases). For example:

    >>> import sys, locale
    >>> 'time' in sys.modules
    False
    >>> locale.setlocale(locale.LC_CTYPE, '')
    'French_France.1252'
    >>> import time
    >>> time.tzname
    ('Est', 'Est (heure d’été)')
    >>> time.strftime('%Z')
    'Est (heure d’été)'

Note that Unix Python 3 sets LC_CTYPE at startup, so doing the same on Windows would actually improve cross-platform consistency.