python é.py fails with UnicodeEncodeError if PYTHONFSENCODING is used

[vstinner]

BPO	10039
Nosy	@vstinner, @merwok, @bitdancer
Files	redecode_filename.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-10-13.22:20:20.436>
created_at = <Date 2010-10-07.01:25:31.304>
labels = ['interpreter-core', 'expert-unicode']
title = 'python \xc3\xa9.py fails with UnicodeEncodeError if PYTHONFSENCODING is used'
updated_at = <Date 2010-10-18.17:03:42.916>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-10-18.17:03:42.916>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2010-10-13.22:20:20.436>
closer = 'vstinner'
components = ['Interpreter Core', 'Unicode']
creation = <Date 2010-10-07.01:25:31.304>
creator = 'vstinner'
dependencies = []
files = ['19143']
hgrepos = []
issue_num = 10039
keywords = ['patch']
message_count = 7.0
messages = ['118089', '118436', '118444', '118445', '118492', '118593', '119039']
nosy_count = 3.0
nosy_names = ['vstinner', 'eric.araujo', 'r.david.murray']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10039'
versions = ['Python 3.2']

[vstinner]

If a program name contains a non-ascii character in its name and/or full path and PYTHONFSENCODING is set to an encoding different than the locale encoding, Python fails to open the program.

Example in the utf-8 locale:

$ PYTHONFSENCODING=ascii ./python é.py
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 0: ordinal not in range(128)

This issue is similar to bpo-9992 and bpo-10014.

Solutions: remove PYTHONFSENCODING environment variable or redecode the filename from the locale encoding to the filesystem encoding.

Attached patch implements the latter.

--

We may also redecode Py_GetProgramName().

[merwok]

I don’t understand why reading a filename would not respect the envvar stating the filesystem encoding.

[bitdancer]

Éric, if you are saying, "the user asked for it, it *should* fail", then that is indeed one of the arguments put forward in bpo-9992 where this was discussed. But I think the emerging consensus is that it is better to just avoid the problem by always using the locale on Unix, and solve the problem that PYTHONFSENCODING was supposed to solve in a different way (by always using utf-8 on OSX and unicode on Windows).

[merwok]

if you are saying, "the user asked for it, it *should* fail", then
that is indeed one of the arguments put forward in bpo-9992 where
this was discussed.
You could put it that way, thanks for phrasing my thoughts :)

But I think the emerging consensus is that it is better to just avoid
the problem by always using the locale on Unix,
*displays his lack of knowledge* Is it always correct to decode a filename with the locale encoding on Unix? Can’t each filesystem have its own encoding?

and solve the problem that PYTHONFSENCODING was supposed to solve in a
different way (by always using utf-8 on OSX and unicode on Windows).
If there is a better alternate way, let’s go for it, and maybe remove PYTHONFSENCODING altogether, since it’s new in 3.2.

Thanks for explaining! I’ll repay your time by reviewing the doc patches.

[vstinner]

Is it always correct to decode a filename with the locale encoding
on Unix?

Do you know something better than the locale encoding? I don't.

Can’t each filesystem have its own encoding?

Yes, but how do you get the encoding of each filesystem? I think that few or no application support such case without mojibake. Backup programs can use the "raw" (bytes) API of Python 3 to avoid all encoding issues.

--

As wrote R. David Murray, read issue bpo-9992 if you would like to know more about this problem and the different proposed solutions. I voted for removal of PYTHONFSENCODING which fix most issues.

[vstinner]

Fixed by r85430 (remove PYTHONFSENCODING), see bpo-9992.

[merwok]

Do you know something better than the locale encoding? I don't.
Neither do I, sorry.

> Can’t each filesystem have its own encoding?
Yes, but how do you get the encoding of each filesystem?
If I really had to, on linux I could parse the output of the mount command, but this could get messy quickly, and of course is not okay for official Python.

Backup programs can use the "raw" (bytes) API of Python 3 to avoid
all encoding issues.
Neat!

As wrote R. David Murray, read issue bpo-9992 if you would like to know
more about this problem and the different proposed solutions.
I did so, thanks for the pointer and all the explanations.