Reencode filenames when setting the filesystem encoding

[vstinner]

BPO	9630
Nosy	@malemburg, @loewis, @brettcannon, @amauryfa, @pitrou, @vstinner
Files	redecode_modules_path-4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-29.16:37:07.731>
created_at = <Date 2010-08-17.23:47:04.786>
labels = ['interpreter-core', 'expert-unicode']
title = 'Reencode filenames when setting the filesystem encoding'
updated_at = <Date 2010-09-29.17:39:03.881>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-09-29.17:39:03.881>
actor = 'lemburg'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-29.16:37:07.731>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-08-17.23:47:04.786>
creator = 'vstinner'
dependencies = []
files = ['19057']
hgrepos = []
issue_num = 9630
keywords = ['patch']
message_count = 24.0
messages = ['114191', '114193', '114348', '114349', '114352', '115546', '115610', '115668', '117268', '117269', '117270', '117271', '117272', '117273', '117274', '117277', '117278', '117281', '117593', '117595', '117605', '117607', '117629', '117634']
nosy_count = 7.0
nosy_names = ['lemburg', 'loewis', 'brett.cannon', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9630'
versions = ['Python 3.2']

[vstinner]

Python 3 has a very important variable: the filesystem encoding, sys.getfilesystemencoding(). It is used to encode and decode filenames to access to the filesystem, to encode program arguments in subprocess, etc.

The encoding is hardcoded to "mbcs" on Windows and "utf-8" on Mac OS X. On other OSes, Python gets the encoding from the locale. The problem is that the code getting the locale encoding loads Python modules (eg. locale) and Python uses a default encoding before the locale encoding is known. As a result, modules and code objects created before Python sets the locale encoding are encoded with the old encoding.

The default encoding is "utf-8". If the locale encoding is also "utf-8", there is no problem because the filename are correctly encoded. If the locale encoding is different, we keep filenames encoded in the wrong encoding.

It becomes worse when the locale encoding is unable to encode the filenames, eg. ASCII encoding.

--

A solution would be to avoid loading any Python module, but I don't think that it is possible. The locale encoding can be something different than ascii, latin-1, utf-8 or mbcs. The locale encoding can be an alias like 'utf8' (instead of 'utf-8'), 'iso-8859-1' (Python uses 'latin_1') or 'ANSI_x3.4_1968' (for 'ascii') and encoding aliases are implemented as Lib/encodings/aliases.py which is... a Python module.

--

I wrote a patch to reencode filenames of all module and code objects in initfsencoding() when the locale encoding is known.

I tested my patch on my import_unicode branch (branch to fix bpo-8611, see also bpo-9425: issue to merge the branch to py3k). I would like one or more reviews of the patch because it is long and complex. Please check for refleaks :-)

--

About the patch.

I don't know how to list *all* code objects and so I created a list to store weak references to all code objects, list filled by the code object constructor. The list is destroyed at initfsencoding() exit (early in Python initialization).

There is a FIXME: I don't know if sys.path_importer_cache keys should also be reencoded.

I tried to apply all remarks made on the first patch (posted on Rietveld for bpo-9425). The patch now stores weak references instead of strong references to code objects in the code object list.

(r84168 creates PyModule_GetFilenameObject, function needed by this patch)

[vstinner]

While working on bpo-8622, I realized that it's not enough: sys.path and sys.executable (and sys.meta_path) should also be reencoded. New patch does that.

[amauryfa]

I would rename the feature to something like "redecode-modules": the filenames were decoded with the wrong encoding, and must be decoded again.

[amauryfa]

Some thoughts: since the modules were successfully imported, surely it means that their filenames where correctly computed and encoded? So why is the __filename__ attribute wrong?

[vstinner]

since the modules were successfully imported, surely it means that
their filenames where correctly computed and encoded? So why is the
__filename__ attribute wrong?

Python starts with 'utf-8' encoding. If the new encoding is "smaller" (unable to encode as much characters as utf-8), PyUnicode_EncodeFS() and os.fsencode() will raise UnicodeEncodeError.

Eg. your Python setup is installed in a directory called b'py3k\xc3\xa9' and your locale is C (ascii encoding). At startup, the directory name is decoded to 'py3ké' (using the defautlt encoding, utf-8). initfsencoding() sets the encoding to ascii: 'py3ké' cannot be encoded to the filesystem encoding (ascii) anymore.

--

If we set the default filesystem encoding to ascii (bpo-8725), it will work but the filenames will be full of surrogates characters. Eg. you Python setup is installed in b'py3k\xc3\xa9' and your locale encoding is utf-8: b'py3k\xc3\xa9' will be decoded to 'py3k\udcc3\udca9' and leaved unchanged by initfsencoding(). Surrogates characters are not pratical: you have to escape them to display them. Print a filename with surrogates in a terminal raise a UnicodeEncodeError (even with utf-8 encoding).

[vstinner]

Another possibility is to use _Py_char2wchar() + PyUnicode_FromWideChar() / _Py_wchar2char() + PyUnicode_AsWideChar() to decode / encode filenames. These functions use the locale encoding. This solution was possible in Python 3.1, but no more in Python 3.2 because of the PYTHONFSENCODING environment variable.

Even if I don't like my own solution, I don't see better solution.

[amauryfa]

Python is installed in a directory called b'py3k\xc3\xa9'
and your locale is C
Do we really want to support this kind of configuration?

[vstinner]

Do we really want to support this kind of configuration?

There is also a problem is the directory name is b'py3k\xe9': at startup (utf-8 encoding), the name is decoded to 'py3k\udce9'. When the locale encoding is set to iso-8859-1: the name should be reencoded to 'py3k\xe9' to avoid the surrogate character.

[vstinner]

New version of the patch:

• reencode sys.path_importer_cache (and remove the last FIXME)
• fix different reference leaks
• catch PyIter_Next() failures
• create a subfunction to reencode sys.modules: it's easier to review and manager errors in shorter functions
• add some comments

[vstinner]

I would rename the feature to something like "redecode-modules"

Yes, right. I will rename the functions before commiting the patch.

[malemburg]

STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

New version of the patch:

• reencode sys.path_importer_cache (and remove the last FIXME)
• fix different reference leaks
• catch PyIter_Next() failures
• create a subfunction to reencode sys.modules: it's easier to review and manager errors in shorter functions
• add some comments

Why is this needed ?

With PYTHONFSENCODING there should be no need to change the FS
encoding after startup.

[vstinner]

Why is this needed ?

Short answer: to support filesystem encoding different than utf-8. See bpo-8611 for a longer explanation.

Example:

$ pwd
/home/SHARE/SVN/py3ké
$ PYTHONFSENCODING=ascii ./python test_fs_encoding.py 
Fatal Python error: Py_Initialize: can't initialize sys standard streams
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 20: ordinal not in range(128)
Abandon

My patch fixes this specific case and prepare the work for the complete fix (support different *locale* encodings, see bpo-8611 and bpo-9425).

--

Longer answer: Py_FilesystemDefaultEncoding is changed too late. Some modules are already loaded, sys.executable is already set, etc. Py_FilesystemDefaultEncoding is changed but modules filenames are decoded with utf-8 and should be "redecoded".

It is not possible to set Py_FilesystemDefaultEncoding before loading the first module. initfsencoding() loads codecs and encodings modules to check the codec name. sys.executable is also set before initfsencoding().

Read my other messages of this issue to get other reasons why the patch is needed. I explained other possibilities (but they don't work).