Change sys.getfilesystemencoding() on Windows to UTF-8

[zooba]

BPO	27781
Nosy	@brettcannon, @pfmoore, @mdickinson, @ncoghlan, @vstinner, @tjguk, @jkloth, @ned-deily, @zware, @zooba, @yan12125, @AraHaan
Files	fsencoding.diff fsencoding.diff test_cmd_line_unicode.py: Possible new test case for command line Unicode handling 27781_1.patch osx_failed_compile.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2016-11-07.03:36:05.023>
created_at = <Date 2016-08-17.03:49:40.936>
labels = ['type-bug', '3.7', 'OS-windows', 'release-blocker']
title = 'Change sys.getfilesystemencoding() on Windows to UTF-8'
updated_at = <Date 2016-11-07.03:36:05.021>
user = 'https://github.com/zooba'

bugs.python.org fields:

activity = <Date 2016-11-07.03:36:05.021>
actor = 'python-dev'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2016-11-07.03:36:05.023>
closer = 'python-dev'
components = ['Windows']
creation = <Date 2016-08-17.03:49:40.936>
creator = 'steve.dower'
dependencies = []
files = ['44130', '44131', '44369', '44414', '44491']
hgrepos = []
issue_num = 27781
keywords = ['patch']
message_count = 24.0
messages = ['272899', '272900', '272916', '272917', '272935', '272949', '272950', '272961', '272962', '272963', '274392', '274691', '274887', '274910', '275063', '275075', '275078', '275095', '275289', '275290', '275326', '275331', '279847', '280186']
nosy_count = 13.0
nosy_names = ['brett.cannon', 'paul.moore', 'mark.dickinson', 'ncoghlan', 'vstinner', 'tim.golden', 'jkloth', 'ned.deily', 'python-dev', 'zach.ware', 'steve.dower', 'yan12125', 'Decorater']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27781'
versions = ['Python 3.6', 'Python 3.7']

[zooba]

I've attached my first pass at a patch to change the file system encoding on Windows to UTF-8 and remove use of the *A APIs.

It would be trivial to change the encoding from UTF-8 back to CP_ACP and change the error mode if that's what we decide is better, but my vote is strongly for an encoding that never drops characters when converted from UTF-16.

Discussion is still ongoing on python-ideas, so let's argue about yes/no and utf-8/mbcs there and just discuss the patch here.

[AraHaan]

I personally hate ansi myself so +1 to UTF-8/UTF-16.

[vstinner]

Would it be acceptable for you to add a new option to switch to UTF-8 in Python 3.6, and discuss later if it's ok to enable it by default?

In the python-ideas threed, you wrote that Windows allow surrogate characters in filenames, but not the UTF-8/strict Python codec. Would it make sense to use UTF-8/surrogatepass codec to avoid any unicode error?

[vstinner]

Steve Dower: Please don't use git format for diff, or the bug tracker is unable to create reviews. I regenerated the patch.

By the way, you introduced a bug in posix_do_stat(): you added a new "else" in the !MS_WINDOWS path which leads to a compilation error. I fixed it.

[zooba]

Thanks for the regen. I don't think git format is the problem as most of my patches are fine, it's probably because it was in a patch queue and so the parent isn't actually a known commit. I haven't tested whether this works without my other console patches but I think it should.

Is there a surrogatepass option? If so, I'll definitely use that, as that'll fix the one remaining edge case.

I suspect we'll have to go to Guido to get a ruling on the default, but I'll add an environment variable to switch.

[vstinner]

Is there a surrogatepass option?

I'm talking about error handlers of Python codecs: text.encode('utf8',
'surrogatepass')

[vstinner]

I suspect we'll have to go to Guido to get a ruling on the default, but I'll add an environment variable to switch.

If you go in this direction, I would like to follow you for the
UNIX/BSD side to make the switch portable. I was thinking about "-X
utf8" which avoids to change the command line parser.

If we agree on a plan, I would like to write it down as a PEP since I
expect a lot of complains and questions which I would prefer to only
answer once (see for example the length of your thread on python-ideas
where each people repeated the same things multiple times ;-))

[zooba]

By portable, do you mean not using an environment variable?

Command line parsing is potentially affected by this on Windows - I'd have to look deeper - as command lines are provided as UTF-16. But we may not ever expose them as bytes.

I don't even know that this matters on the UNIX/BSD side as the file system encoding provided there is correct, no? It's just Windows where the file system encoding used for bytes doesn't match what the file system actually uses.

I was afraid a PEP would be necessary out of this, but I want to see how the python-dev discussion goes first.

[vstinner]

Steve Dower added the comment:

By portable, do you mean not using an environment variable?

I mean that "python3 -X utf8" should force sys.getfilesystemencoding()
to UTF-8 on UNIX/BSD, it would ignore the current locale setting.

[zooba]

Ah I see, if we end up sticking with MBCS and offering a switch to enable UTF-8. In that case, we'll definitely ensure the flag is the same (but I'm hopeful we will just make the reliable behavior on Windows the default, so it won't matter).

[ncoghlan]

I belatedly remembered I've had this new test case hanging around for a while, and never got around to getting it into shape for inclusion in the standard library.

With the prospect of reasonable cross-platform consistency in this area, it could be a good thing to add as part of this PEP.

[zooba]

Also see PEP-529 for the latest updates there.

This is likely to be accepted as experimental for 3.6.0b1-3, and we'll commit to either the new default or a compatible default for b4.