test_site.StartupImportTests.test_startup_imports fails if default code page is cp65001

[paulmon]

BPO	36778
Nosy	@pfmoore, @vstinner, @tjguk, @methane, @zware, @serhiy-storchaka, @eryksun, @zooba, @paulmon
PRs	bpo-36778: fix test_startup_imports if default codepage is UTF-8 #13069 bpo-36778: fix test_startup_imports if default codepage is cp65001 #13072 bpo-36778: Avoid functools in encodings.cp65001 #13110 bpo-36778: Avoid functools in encodings.cp65001 #13211 bpo-36778: cp65001 encoding becomes an alias to utf_8 #13230 bpo-36778: Update cp65001 codec documentation #13240 bpo-36778: Remove outdated comment #13807

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-10.03:30:36.027>
created_at = <Date 2019-05-02.20:46:25.306>
labels = ['3.8', 'type-bug', 'tests', 'OS-windows']
title = 'test_site.StartupImportTests.test_startup_imports fails if default code page is cp65001'
updated_at = <Date 2019-06-04.15:09:15.116>
user = 'https://github.com/paulmon'

bugs.python.org fields:

activity = <Date 2019-06-04.15:09:15.116>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-10.03:30:36.027>
closer = 'vstinner'
components = ['Tests', 'Windows']
creation = <Date 2019-05-02.20:46:25.306>
creator = 'Paul Monson'
dependencies = []
files = []
hgrepos = []
issue_num = 36778
keywords = ['patch']
message_count = 31.0
messages = ['341316', '341323', '341324', '341326', '341372', '341377', '341383', '341401', '341520', '341531', '341570', '341671', '341924', '341926', '341947', '341955', '341968', '342004', '342006', '342008', '342010', '342019', '342020', '342025', '342027', '342029', '342032', '342047', '342053', '342290', '344588']
nosy_count = 9.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'methane', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'Paul Monson']
pr_nums = ['13069', '13072', '13110', '13211', '13230', '13240', '13807']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36778'
versions = ['Python 3.8']

[paulmon]

Windows desktop skus have a default ANSI codepage (returned by GetACP()) of 1252 (Western European). Windows IoT Core and Windows Nano Server have a default codepage of 65001 (UTF-8).

This causes test_site.StartupImportTests.test_startup_imports to fail on Windows IoT Core and Windows Nano Server because cp65001.py is loaded instead of the frozen cp1252.py at startup.

I tried changing the default codepage to 65001 on my dev machine and rebuilding Python and it had no effect that I could tell on the generated frozen importlibs.

The simplest solutions would be for the test_startup_imports test to be skipped or changed to pass when the locale.getpreferredencoding() returns 'cp65001'

[methane]

Could you paste how the test fails?

[paulmon]

======================================================================
FAIL: test_startup_imports (test.test_site.StartupImportTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "c:\docker\pythond\lib\test\test_site.py", line 542, in test_startup_imports
    self.assertFalse(modules.intersection(collection_mods), stderr)
AssertionError: {'operator', 'keyword', 'functools', 'heapq', 'collections', 'reprlib'} is not false : import _frozen_importlib # frozen
import _imp # builtin
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import '_frozen_importlib_external' # <class '_frozen_importlib.FrozenImporter'>
import '_io' # <class '_frozen_importlib.BuiltinImporter'>
import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
import 'nt' # <class '_frozen_importlib.BuiltinImporter'>
import _thread # previously loaded ('_thread')
import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
import _weakref # previously loaded ('_weakref')
import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
import 'winreg' # <class '_frozen_importlib.BuiltinImporter'>
# installing zipimport hook
import 'time' # <class '_frozen_importlib.BuiltinImporter'>
import 'zipimport' # <class '_frozen_importlib.FrozenImporter'>
# installed zipimport hook
# c:\docker\pythond\lib\encodings\__pycache__\__init__.cpython-38.pyc matches c:\docker\pythond\lib\encodings\__init__.py
# code object from 'c:\\docker\\pythond\\lib\\encodings\\__pycache__\\__init__.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\codecs.cpython-38.pyc matches c:\docker\pythond\lib\codecs.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\codecs.cpython-38.pyc'
import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at 0x01D9DBD0>
# c:\docker\pythond\lib\encodings\__pycache__\aliases.cpython-38.pyc matches c:\docker\pythond\lib\encodings\aliases.py
# code object from 'c:\\docker\\pythond\\lib\\encodings\\__pycache__\\aliases.cpython-38.pyc'
import 'encodings.aliases' # <_frozen_importlib_external.SourceFileLoader object at 0x01EFF900>
import 'encodings' # <_frozen_importlib_external.SourceFileLoader object at 0x01D9DA50>
# c:\docker\pythond\lib\encodings\__pycache__\utf_8.cpython-38.pyc matches c:\docker\pythond\lib\encodings\utf_8.py
# code object from 'c:\\docker\\pythond\\lib\\encodings\\__pycache__\\utf_8.cpython-38.pyc'
import 'encodings.utf_8' # <_frozen_importlib_external.SourceFileLoader object at 0x01D9DCC0>
import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
# c:\docker\pythond\lib\encodings\__pycache__\cp65001.cpython-38.pyc matches c:\docker\pythond\lib\encodings\cp65001.py
# code object from 'c:\\docker\\pythond\\lib\\encodings\\__pycache__\\cp65001.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\functools.cpython-38.pyc matches c:\docker\pythond\lib\functools.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\functools.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\abc.cpython-38.pyc matches c:\docker\pythond\lib\abc.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\abc.cpython-38.pyc'
import '_abc' # <class '_frozen_importlib.BuiltinImporter'>
import 'abc' # <_frozen_importlib_external.SourceFileLoader object at 0x01F16FC0>
# c:\docker\pythond\lib\collections\__pycache__\__init__.cpython-38.pyc matches c:\docker\pythond\lib\collections\__init__.py
# code object from 'c:\\docker\\pythond\\lib\\collections\\__pycache__\\__init__.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\_collections_abc.cpython-38.pyc matches c:\docker\pythond\lib\_collections_abc.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\_collections_abc.cpython-38.pyc'
import '_collections_abc' # <_frozen_importlib_external.SourceFileLoader object at 0x01F423C0>
# c:\docker\pythond\lib\__pycache__\operator.cpython-38.pyc matches c:\docker\pythond\lib\operator.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\operator.cpython-38.pyc'
import '_operator' # <class '_frozen_importlib.BuiltinImporter'>
import 'operator' # <_frozen_importlib_external.SourceFileLoader object at 0x01F4D630>
# c:\docker\pythond\lib\__pycache__\keyword.cpython-38.pyc matches c:\docker\pythond\lib\keyword.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\keyword.cpython-38.pyc'
import 'keyword' # <_frozen_importlib_external.SourceFileLoader object at 0x01F58810>
# c:\docker\pythond\lib\__pycache__\heapq.cpython-38.pyc matches c:\docker\pythond\lib\heapq.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\heapq.cpython-38.pyc'
import '_heapq' # <class '_frozen_importlib.BuiltinImporter'>
import 'heapq' # <_frozen_importlib_external.SourceFileLoader object at 0x01F588D0>
import 'itertools' # <class '_frozen_importlib.BuiltinImporter'>
# c:\docker\pythond\lib\__pycache__\reprlib.cpython-38.pyc matches c:\docker\pythond\lib\reprlib.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\reprlib.cpython-38.pyc'
import 'reprlib' # <_frozen_importlib_external.SourceFileLoader object at 0x01F59900>
import '_collections' # <class '_frozen_importlib.BuiltinImporter'>
import 'collections' # <_frozen_importlib_external.SourceFileLoader object at 0x01F25810>
import '_functools' # <class '_frozen_importlib.BuiltinImporter'>
import 'functools' # <_frozen_importlib_external.SourceFileLoader object at 0x01EFFCC0>
import 'encodings.cp65001' # <_frozen_importlib_external.SourceFileLoader object at 0x01EFF9F0>
# c:\docker\pythond\lib\encodings\__pycache__\latin_1.cpython-38.pyc matches c:\docker\pythond\lib\encodings\latin_1.py
# code object from 'c:\\docker\\pythond\\lib\\encodings\\__pycache__\\latin_1.cpython-38.pyc'
import 'encodings.latin_1' # <_frozen_importlib_external.SourceFileLoader object at 0x01EFF810>
# c:\docker\pythond\lib\__pycache__\io.cpython-38.pyc matches c:\docker\pythond\lib\io.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\io.cpython-38.pyc'
import 'io' # <_frozen_importlib_external.SourceFileLoader object at 0x01D88DB0>
Python 3.8.0a3+ (heads/iot-merged-dirty:88716a51a3, Apr  5 2019, 11:11:18) [MSC v.1916 32 bit (ARM)] on win32
Type "help", "copyright", "credits" or "license" for more information.
# c:\docker\pythond\lib\__pycache__\site.cpython-38.pyc matches c:\docker\pythond\lib\site.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\site.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\os.cpython-38.pyc matches c:\docker\pythond\lib\os.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\os.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\stat.cpython-38.pyc matches c:\docker\pythond\lib\stat.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\stat.cpython-38.pyc'
import '_stat' # <class '_frozen_importlib.BuiltinImporter'>
import 'stat' # <_frozen_importlib_external.SourceFileLoader object at 0x01F25990>
# c:\docker\pythond\lib\__pycache__\ntpath.cpython-38.pyc matches c:\docker\pythond\lib\ntpath.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\ntpath.cpython-38.pyc'
# c:\docker\pythond\lib\__pycache__\genericpath.cpython-38.pyc matches c:\docker\pythond\lib\genericpath.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\genericpath.cpython-38.pyc'
import 'genericpath' # <_frozen_importlib_external.SourceFileLoader object at 0x01F9CDE0>
import 'ntpath' # <_frozen_importlib_external.SourceFileLoader object at 0x01F9C5D0>
import 'os' # <_frozen_importlib_external.SourceFileLoader object at 0x01F873F0>
# c:\docker\pythond\lib\__pycache__\_sitebuiltins.cpython-38.pyc matches c:\docker\pythond\lib\_sitebuiltins.py
# code object from 'c:\\docker\\pythond\\lib\\__pycache__\\_sitebuiltins.cpython-38.pyc'
import '_sitebuiltins' # <_frozen_importlib_external.SourceFileLoader object at 0x01F87FC0>
import 'site' # <_frozen_importlib_external.SourceFileLoader object at 0x01F16C60>
# cleanup[3] wiping _functools
# cleanup[3] wiping _collections
# cleanup[3] wiping heapq
# cleanup[3] wiping _heapq
# destroy _heapq
# cleanup[3] wiping _operator
# cleanup[3] wiping _collections_abc
# cleanup[3] wiping _abc
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping codecs
# cleanup[3] wiping _codecs
# cleanup[3] wiping winreg
# cleanup[3] wiping _weakref
# cleanup[3] wiping _thread
# cleanup[3] wiping nt
# cleanup[3] wiping marshal
# cleanup[3] wiping _io
# cleanup[3] wiping _frozen_importlib_external
# destroy io
# destroy nt
# destroy winreg
# destroy marshal
# cleanup[3] wiping _warnings
# cleanup[3] wiping _imp
# cleanup[3] wiping _frozen_importlib
# destroy _frozen_importlib_external
# destroy _imp
# destroy _warnings
# cleanup[3] wiping sys
# clear builtins._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.__interactivehook__
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing sys
# cleanup[2] removing builtins
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _imp
# cleanup[2] removing _warnings
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing nt
# cleanup[2] removing _thread
# cleanup[2] removing _weakref
# cleanup[2] removing winreg
# cleanup[2] removing time
# cleanup[2] removing zipimport
# destroy zipimport
# cleanup[2] removing _codecs
# cleanup[2] removing codecs
# cleanup[2] removing encodings.aliases
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _signal
# cleanup[2] removing __main__
# destroy __main__
# cleanup[2] removing _abc
# cleanup[2] removing abc
# cleanup[2] removing _collections_abc
# cleanup[2] removing _operator
# cleanup[2] removing operator
# destroy operator
# cleanup[2] removing keyword
# destroy keyword
# cleanup[2] removing _heapq
# cleanup[2] removing heapq
# cleanup[2] removing itertools
# cleanup[2] removing reprlib
# destroy reprlib
# cleanup[2] removing _collections
# cleanup[2] removing collections
# destroy collections
# cleanup[2] removing _functools
# cleanup[2] removing functools
# cleanup[2] removing encodings.cp65001
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing io
# destroy io
# cleanup[2] removing _stat
# cleanup[2] removing stat
# cleanup[2] removing genericpath
# cleanup[2] removing ntpath
# cleanup[2] removing os.path
# cleanup[2] removing os
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing site
# destroy site
# destroy time
# destroy _signal
# destroy itertools
# destroy _sitebuiltins
# destroy abc
# destroy ntpath
# destroy _stat
# destroy os
# destroy stat
# destroy genericpath
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping encodings.cp65001
# destroy functools
# cleanup[3] wiping builtins
# destroy _functools
# destroy _collections_abc
# destroy _operator
# destroy heapq
# destroy _weakref
# destroy _collections
# destroy _thread
# destroy _abc
# destroy _frozen_importlib

[methane]

@victor It seems you added cp65001 as Windows-only encoding in bpo-13216.

How do you think about removing cp65001 encoding, and add 'cp65001' -> 'utf_8' alias which is available on all platforms?

[vstinner]

cp65001 is *not* utf-8: Microsoft decided to handle surrogates differently
for some reasons.

[eryksun]

cp65001 is *not* utf-8: Microsoft decided to handle surrogates
differently for some reasons.

Do you mean valid UTF-16 surrogate pairs? For example:

    >>> codecs.code_page_encode(65001, '\ud800\udc00')
    (b'\xf0\x90\x80\x80', 2)

PyUnicode_AsUnicodeAndSize is neutral about storing surrogate codes in a 16-bit wchar_t string. In particular, the Python string in this case contains two surrogate codes, but they're passed to WideCharToMultiByte as a UTF-16 surrogate pair for the single character U+10000.

Anyway, it seems to me this issue will be resolved if cp65001.py is rewritten without functools.partial.

[serhiy-storchaka]

I think it is better to just make the check in the test conditional. It already contains some macOs specific conditions.

[eryksun]

I think it is better to just make the check in the test conditional.

Okay. The test verifies work done to minimize interpreter startup time, but probably the relative cost of importing functools (and thus collections et al) isn't significant compared to the overall cost of spawning a process in a Windows desktop environment. That may not be the case for Nano Server and IoT Core.

[vstinner]

Paul Monson: I'm unable to reproduce exactly your issue, but I tried to reproduce it partially using PYTHONIOENCODING=cp65001.

My PR 13110 avoids "import functools" at startup. Can you please try it and check if it fix test_site?

[vstinner]

Victor:

cp65001 is *not* utf-8: Microsoft decided to handle surrogates differently for some reasons.

Eryk:

Do you mean valid UTF-16 surrogate pairs? (...)

Code page 65001 handles lone surrogate differently on Windows XP and older. It changed in Windows Vista:
https://unicodebook.readthedocs.io/operating_systems.html#encode-and-decode-functions

Steve Dower removed support for Vista from test_codecs.py 3 years ago:

commit f5aba58
Author: Steve Dower <steve.dower@microsoft.com>
Date: Tue Sep 6 19:42:27 2016 -0700

Issue bpo-27959: Adds oem encoding, alias ansi to mbcs, move aliasmbcs to codec lookup

Maybe it's time to remove Lib/encodings/cp65001.py and add an alias cp65001 => utf_8 in Lib/encodings/aliases.py? See bpo-32592.