sys.path[0] is incorrect if PYTHONFSENCODING is used

[vstinner]

BPO	10014
Nosy	@vstinner
Dependencies	bpo-10039: python é.py fails with UnicodeEncodeError if PYTHONFSENCODING is used
Files	realpath_fs_encoding-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-10-13.22:25:10.615>
created_at = <Date 2010-10-02.12:14:21.087>
labels = ['expert-unicode']
title = 'sys.path[0] is incorrect if PYTHONFSENCODING is used'
updated_at = <Date 2010-10-13.22:25:10.614>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-10-13.22:25:10.614>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-13.22:25:10.615>
closer = 'vstinner'
components = ['Unicode']
creation = <Date 2010-10-02.12:14:21.087>
creator = 'vstinner'
dependencies = ['10039']
files = ['19153']
hgrepos = []
issue_num = 10014
keywords = ['patch']
message_count = 7.0
messages = ['117870', '118088', '118090', '118101', '118102', '118151', '118595']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10014'
versions = ['Python 3.2']

[vstinner]

In the following example, sys.path[0] should be '/home/SHARE/SVN/py3k\udcc3\udca9' (my locale and filesystem encodings are utf-8):

$ cd /home/SHARE/SVN/py3ké
$ echo "import sys; print(sys.path[0])" > x.py
$ ./python x.py
/home/SHARE/SVN/py3ké
$ PYTHONFSENCODING=ascii ./python x.py
/home/SHARE/SVN/py3ké

The problem is that PySys_SetArgvEx() inserts argv[0] at sys.path[0], but argv[0] is decoded using the locale encoding (by _Py_char2wchar() in main()), whereas paths of sys.path are supposed to be encodable (and decoded) by sys.getfilesystemencoding().

argv array should be decoded using the filesystem encoding (see issue bpo-9992) or argv[0] should be redecoded (encode to the locale encoding, and decode from the filesystem encoding, see issue bpo-9630).

[vstinner]

The problem is that PySys_SetArgvEx() ...

Not only PySys_SetArgvEx(). There is another issue with RunMainFromImporter() which do: sys.path[0] = filename

[vstinner]

See also issue bpo-10039.

[vstinner]

This issue depends on issue bpo-10039.

[vstinner]

r85302: _wrealpath() and _Py_wreadlink() support surrogates in the input path.

--

realpath_fs_encoding.patch: patch _wrealpath() to encode the resulting path with the filesystem encoding (with surrogateescape) instead of the locale encoding. This patch is incomplete: it doesn't fix the issue for non-Windows platforms without the realpath() function.

redecode_filename.patch (from issue bpo-10039) + realpath_fs_encoding.patch fix this issue on my Linux (Debian Sid) box.

[vstinner]

I just created Python/fileutils.c: update the patch for this new file.

[vstinner]

Fixed by r85430 (remove PYTHONFSENCODING), see bpo-9992.