Remove sys.setfilesystemencoding()

[vstinner]

BPO	9632
Nosy	@malemburg, @pitrou, @vstinner, @merwok
Files	remove_sys_setfilesystemencoding-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-09-10.21:59:24.638>
created_at = <Date 2010-08-18.11:56:02.939>
labels = ['type-feature', 'library', 'expert-unicode']
title = 'Remove sys.setfilesystemencoding()'
updated_at = <Date 2010-09-11.10:50:09.728>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-09-11.10:50:09.728>
actor = 'lemburg'
assignee = 'none'
closed = True
closed_date = <Date 2010-09-10.21:59:24.638>
closer = 'vstinner'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2010-08-18.11:56:02.939>
creator = 'vstinner'
dependencies = []
files = ['18576']
hgrepos = []
issue_num = 9632
keywords = ['patch']
message_count = 14.0
messages = ['114211', '114342', '114409', '114855', '114856', '115024', '115105', '115127', '115547', '115821', '115822', '115854', '116047', '116089']
nosy_count = 5.0
nosy_names = ['lemburg', 'pitrou', 'vstinner', 'eric.araujo', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9632'
versions = ['Python 3.2']

[vstinner]

sys.setfilesystemencoding() function is dangerous because it introduces a lot of inconsistencies: this function is unable to reencode all filenames in all objects (eg. Python is unable to find filenames in user objects or 3rd party libraries). Eg. if you change the filesystem from utf8 to ascii, it will not be possible to use existing non-ascii (unicode) filenames: they will raise UnicodeEncodeError.

As sys.setdefaultencoding() in Python2, I think that sys.setfilesystemencoding() is the root of evil :-) PYTHONFSENCODING (issue bpo-8622) is the right solution to set the filesysteme encoding.

Attached patch removes sys.setfilesystemencoding().

[vstinner]

New version of the patch: remove also _Py_SetFileSystemEncoding().

[malemburg]

While you're right that adjusting the FS encoding long after Python has already started is probably not such a good idea, I do think that we need to provide a way to set the FS encoding from within Python without having to rely on external settings.

Think of e.g. embedded Python interpreters or py2exe-style applications running on Linux or other systems that don't use Unicode APIs for FS-interaction or have fixed FS-encodings.

[vstinner]

Think of e.g. embedded Python interpreters or py2exe-style applications
running on Linux or other systems that don't use Unicode APIs
for FS-interaction or have fixed FS-encodings.

What is the problem here? Python does guess the filesystem encoding. If the encoding is "wrong" (not the value expected by the user), filenames are not displayed correctly (mojibake) but it does just work. Anyway, why is it not possible to use PYTHONFSENCODING here? Are you talking to Python modules loaded from a non-ascii path?

Sorry, but I do not understand.

[vstinner]

About the patch: it should patch "Filenames and unicode" section of Doc/whatsnew/3.2.rst (to explain that sys.setfilesystemencoding() is replaced by the PYTHONFSENCODING env var).

[malemburg]

STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

> Think of e.g. embedded Python interpreters or py2exe-style applications
> running on Linux or other systems that don't use Unicode APIs
> for FS-interaction or have fixed FS-encodings.

What is the problem here? Python does guess the filesystem encoding. If the encoding is "wrong" (not the value expected by the user), filenames are not displayed correctly (mojibake) but it does just work. Anyway, why is it not possible to use PYTHONFSENCODING here? Are you talking to Python modules loaded from a non-ascii path?

Sorry, but I do not understand.

In such environments you cannot expect the user to configure the
system properly (i.e. set an environment variable). Instead, the
application has to provide an educated guess to the Python
interpreter in some way, hence the idea to use a configuration
file or perhaps provide a C API that can be used to set the
variable before initializing the interpreter.

[pitrou]

>> Think of e.g. embedded Python interpreters or py2exe-style applications
>> running on Linux or other systems that don't use Unicode APIs
>> for FS-interaction or have fixed FS-encodings.
>
> What is the problem here? Python does guess the filesystem encoding. If the encoding is "wrong" (not the value expected by the user), filenames are not displayed correctly (mojibake) but it does just work. Anyway, why is it not possible to use PYTHONFSENCODING here? Are you talking to Python modules loaded from a non-ascii path?
>
> Sorry, but I do not understand.

In such environments you cannot expect the user to configure the
system properly (i.e. set an environment variable). Instead, the
application has to provide an educated guess to the Python
interpreter in some way, hence the idea to use a configuration
file or perhaps provide a C API that can be used to set the
variable before initializing the interpreter.

Why wouldn't the embedding application just set the environment var
before initializing the Python interpreter?

[malemburg]

Antoine Pitrou wrote:
> 
> Antoine Pitrou <pitrou@free.fr> added the comment:
> 
>>>> Think of e.g. embedded Python interpreters or py2exe-style applications
>>>> running on Linux or other systems that don't use Unicode APIs 
>>>> for FS-interaction or have fixed FS-encodings.
>>>
>>> What is the problem here? Python does guess the filesystem encoding. If the encoding is "wrong" (not the value expected by the user), filenames are not displayed correctly (mojibake) but it does just work. Anyway, why is it not possible to use PYTHONFSENCODING here? Are you talking to Python modules loaded from a non-ascii path?
>>>
>>> Sorry, but I do not understand.
>>
>> In such environments you cannot expect the user to configure the
>> system properly (i.e. set an environment variable). Instead, the
>> application has to provide an educated guess to the Python
>> interpreter in some way, hence the idea to use a configuration
>> file or perhaps provide a C API that can be used to set the
>> variable before initializing the interpreter.
> 
> Why wouldn't the embedding application just set the environment var
> before initializing the Python interpreter?

Because that's not easy to do in a platform independent way.
OTOH, it's very easy to do via a C API function in Python and
since this env var is essential for the operation of Python,
adding such an API is warranted.

[vstinner]

In such environments you cannot expect the user to configure the
system properly (i.e. set an environment variable).

Why would it be different for embeded python?

Instead, the application has to provide an educated guess
to the Python interpreter in some way, ...

How can the application guess the encoding better than Python? If the user doesn't configure correctly its environment, I don't see how the application can get the real (correct) environment config?!

If Python is unable to start because of the filesystem encoding, it is a bug (see bpo-8611). If Python starts but displays incorrectly filenames, it is the user fault: the user have to setup its environment.

[vstinner]

About "embedded Python interpreters or py2exe-style applications": do you mean that the application calls a C function to set the encoding before starting the interpreter? Or you mean the Python function, sys.setfilesystemencoding()?

I would like to remove the Python function just because it doesn't work (it doesn't reencode filenames from all Python objects). But we might keep the C function if you really want to :-)

[vstinner]

"keep the C function"

Hum, currently, Python3 only has a *private* function called _Py_SetFileSystemEncoding() which can only be called after _Py_InitializeEx() (because it relies on the codecs API). If you consider that there is a real use case, we should create a function to set the filesystem encoding, function that should (have to?) be called before Py_InitializeEx().

I still think that Python knows better than the application how to set the encoding (when, how to choose it, etc.).

[malemburg]

STINNER Victor wrote:

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

"keep the C function"

Hum, currently, Python3 only has a *private* function called _Py_SetFileSystemEncoding() which can only be called after _Py_InitializeEx() (because it relies on the codecs API). If you consider that there is a real use case, we should create a function to set the filesystem encoding, function that should (have to?) be called before Py_InitializeEx().

I still think that Python knows better than the application how to set the encoding (when, how to choose it, etc.).

If you embed Python into another application, say as scripting
language for that application, that other application may have
completely different requirements for the user setup than Python
expects, e.g. for a Windows GUI application it's not feasible to
ask the user to change the environment variables via the registry
in order for Python to pick up the right encoding information.

What we'd need is a way for the embedding application to provide this
information in a way that doesn't require setting up the environment
in some special way. The application will likely have its own way
of configuring things like file system or I/O stream encodings. Think
of e.g. GTK or Qt applications as example.

The Py_InitializeEx() function sounds like a good idea to pass the
information about such important extra parameters to Python. This
could take arguments for setting the file system encoding as
well as the I/O encoding. The arguments would then override the env var
settings.

So you can remove the function, but have to keep a backdoor open
for use cases like the one I described above.

The Py_InitializeEx()
function approach would also avoid all the issues that you have
with calling _Py_SetFileSystemEncoding() after the interpreter
has been initialized.