'ascii' is a bad filesystem default encoding

[bz2]

BPO	13643
Nosy	@terryjreedy, @ncoghlan, @pitrou, @vstinner, @benjaminp, @bitdancer, @bz2, @4kir4
Files	/tmp/filesystem_encoding_utf8.patch: Patch for using utf-8 instead of ascii if given as codeset

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-12-24.03:01:38.178>
created_at = <Date 2011-12-20.19:02:21.313>
labels = ['interpreter-core', 'type-feature']
title = "'ascii' is a bad filesystem default encoding"
updated_at = <Date 2017-12-19.01:01:26.958>
user = 'https://github.com/bz2'

bugs.python.org fields:

activity = <Date 2017-12-19.01:01:26.958>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2011-12-24.03:01:38.178>
closer = 'terry.reedy'
components = ['Interpreter Core']
creation = <Date 2011-12-20.19:02:21.313>
creator = 'gz'
dependencies = []
files = ['24064']
hgrepos = []
issue_num = 13643
keywords = ['patch']
message_count = 36.0
messages = ['149924', '149925', '149926', '149927', '149928', '149929', '149938', '149939', '149941', '149942', '149943', '149944', '149947', '149948', '149949', '149950', '149951', '149952', '150031', '150039', '150040', '150050', '150052', '150053', '150056', '150058', '150061', '150062', '150066', '150067', '150068', '150069', '150204', '150215', '283718', '308601']
nosy_count = 10.0
nosy_names = ['terry.reedy', 'ncoghlan', 'pitrou', 'vstinner', 'vila', 'benjamin.peterson', 'r.david.murray', 'gz', 'akira', 'poolie']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'test needed'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13643'
versions = ['Python 3.3']

[bz2]

Currently when running Python on a non-OSX posix environment under either the C locale, or with an invalid or missing locale, it's not possible to operate using unicode filenames outside the ascii range. Using bytes works, as does reading expecting unicode, using the surrogates hack.

This makes robustly working with non-ascii filenames on different platforms needlessly annoying, given no modern nix should have problems just using UTF-8 in these cases.

See the downstream bzr bug for more:
<https://bugs.launchpad.net/bzr/+bug/794353\>

One option is to just use UTF-8 for encoding and decoding filenames when otherwise ascii would be used. As a strict superset, this shouldn't break too many existing assumptions, and it's unlikely that non-UTF-8 filenames will accidentally be mangled due to a locale setting blip. See the attached patch for this behaviour change. It does not include a test currently, but it's possible to write one using subprocess and overriden LANG and LC_ALL vars.

[bitdancer]

I'm not sure why having a locale set to C or something invalid should be considered a Python bug. You have to handle un-decodable filenames no matter what you do, since things aren't always encoded in utf-8 on non-OSX unix even when that is the system locale. It's just something you have to live with.

[vstinner]

Currently when running Python on a non-OSX posix environment
under either the C locale, or with an invalid or missing locale,
it's not possible to operate using unicode filenames outside
the ascii range.

It was already discussed: using a different encoding for filenames and for other things is really not a good idea. The main problem is the interaction with other programs.

Read discussion of issues bpo-8622, bpo-8775 and bpo-9992.

[vstinner]

under either the C locale, or with an invalid or missing locale

The right fix is to fix your locale, not Python.

[bz2]

I'm not sure why having a locale set to C or something invalid should be
considered a Python bug. You have to handle un-decodable filenames no
matter what you do, since things aren't always encoded in utf-8 on non-OSX
unix even when that is the system locale. It's just something you have to
live with.

This is more about un-encodable filenames. At the moment work with non-ascii filenames in Python robustly requires two branches, one using unicode and one that encodes to bytestrings and deals with the case where the name can't be represented in the declared filesystem encoding. That may be something that just had to be lived with, but it's a little annoying when even without a UTF-8 locale for a particular process, that's what most systems will want on disk.

[bz2]

It was already discussed: using a different encoding for filenames and for
other things is really not a good idea. The main problem is the interaction
with other programs.

Yes, for many programs, a change like this will mean they create the file, but then throw a traceback anyway when trying to print its name to stdout or something.

Read discussion of issues bpo-8622, bpo-8775 and bpo-9992.

Thanks. I agree that spreading different values to things like subprocess arguments and the environment is asking for trouble. Just changing how unicode filename are encoded by default seems safer, though it certainly won't help all code.

The right fix is to fix your locale, not Python.

I've found that hard to stick to in the face of bug reports where "your locale" turns out to be "the locale used by some cronjob". Fixing my library to work under LANG=C is easier than bugging every downstream project.

[poolie]

I'm not sure why having a locale set to C or something invalid should be considered a Python bug.

Programs like bzr that hit these problems can tell their users, either in the docs or an error message, "change your locale to a UTF-8 one".

There are two problems with this: one is just the practical one that it scales poorly to have to tell every user to do this and to take them through working out how to set this in a way that covers cron jobs, daemons, things run over ssh, etc.

The other problem is that the locale variables primarily describe the locale for input/output, and that can very reasonably be different from the filesystem encoding. As a specific common example people may have UTF-8 filenames but want a C locale terminal. If there was a separate LC_FILENAMES then Python could respect that and insist people set it, but there isn't.

[vstinner]

If there was a separate LC_FILENAMES then Python could respect
that and insist people set it, but there isn't.

During 1 month, we had PYTHONFSENCODING environment variable. It was not a good idea. Again: please read the discussion (in closed issues) explaing why we removed it (and which problems it introduced).

[vstinner]

There are two problems with this: one is just the practical
one that it scales poorly to have to tell every user to do this
and to take them through working out how to set this in a way
that covers cron jobs, daemons, things run over ssh, etc.

I never checked which locale is used by default for programs called by cron. So I checked: on Fedora 16, programs start with a very few environment variables, and LANG and LC_ALL are not set. You can add "LANG=fr_FR.UTF-8" (for example) to /etc/environment to set the default language for the whole system (for all programs). I checked, it works with cron. Or if you don't want to affect all programs, it is maybe safer to only set the locale for one specific program in your crontab by adding "LANG=fr_FR.UTF-8 " before you command. Example:

• • • • • LANG=fr_FR.UTF-8 /home/haypo/test.sh

--

If you want to handle any filename without having to care of the locale, the simplest solution is to use the bytes type to store filenames.

[poolie]

On 21 December 2011 11:01, STINNER Victor <report@bugs.python.org> wrote:

Again: please read the discussion (in closed issues) explaing why we removed it (and which problems it introduced).

There's a lot of history, so I'm not sure exactly which problems
you're referring to. The main problem I see being discussed is that
changing the encoding after Python starts would be dangerous, which I
agree with, but we're not proposing to do that.

[poolie]

On 21 December 2011 11:26, STINNER Victor <report@bugs.python.org> wrote:

I never checked which locale is used by default for programs called by cron. So I checked: on Fedora 16, programs start with a very few environment variables, and LANG and LC_ALL are not set. You can add "LANG=fr_FR.UTF-8" (for example) to /etc/environment to set the default language for the whole system (for all programs). I checked, it works with cron. Or if you don't want to affect all programs, it is maybe safer to only set the locale for one specific program in your crontab by adding "LANG=fr_FR.UTF-8 " before you command. Example:

• •  *  *  * LANG=fr_FR.UTF-8 /home/haypo/test.sh

That is the correct kind of configuration. When I say it scales
poorly I mean that every user running a Python program on a unicode
system needs to insert this configuration in every relevant place, and
they need to work this out from what is typically a fairly cryptic
message. (bzr just added a workaround for this, but for other
programs it still exists.)

Also, my other point, is that people may very well want their cron
scripts to send ascii output but cope with unicode filenames.

[vstinner]

The main problem I see being discussed is that
changing the encoding after Python starts would
be dangerous, which I agree with, but we're not
proposing to do that.

Not after Python start. Using two encodings at the same would just adds new problems. On UNIX (at least on Linux?), it is mandatory to use the same encoding for:

• command line arguments
• environment variables
• filenames
• and more generally, all data exchanged with the system and other programs

Let's take an example: you use UTF-8 for filenames and ISO-8859-1 for all other data. You want to check if a specific filename is present in your home directory: encode the filename to UTF-8 and read the home directory from the HOME environment variable. But environment variables are decoded from ISO-8859-1, so you have to encode them back to ISO-8859-1 to avoid mojibake (and real bugs, like file not found).

Ok, let say that filenames and environment variables are UTF-8 and that other data are ISO-8859-1. You would like to play a MP3 using mplayer: you pass the filename encoded to UTF-8 as an argument of mplayer command line. But mplayer uses ISO-8859-1 to decode its command line (it's not exactly like that, but image that it's the case): mplayer will be unable to find your MP3.

etc.

That's why on UNIX there is one unique encoding, the locale encoding, and that Python uses the same encoding (called "the filesystem encoding", I don't like this name, sys.getfilesystemencoding()).

--

It is no more possible to change the Python filesystem encoding at runtime (I remove sys.setfilesystemencoding()) because I would like to inconsistency. If you decoded a filename before changing the encoding, and then you decode the same filename after changing the encoding: you will get two different names and encode the filenames back will give you two different byte sequences (and more likely, a Unicode encode error).

It was possible to override the filesystem encoding using a PYTHONFSENCODING environment variable, but it introduced all the inconsistencies listed before (especially with external programs).

Now the only right way to change the Python (filesystem) encoding is the UNIX way of doing that: set LC_ALL, LC_CTYPE or LANG environment variable (configure your locale).

[vstinner]

I should not write comments so late :-p

Not after Python start. Using two encodings at the same would just ...

at the same time

> ... because I would like to inconsistency.

because it would lead to inconsistencies