os.device_encoding() doesn't respect the UTF-8 Mode

[vstinner]

BPO	42236
Nosy	@vstinner, @methane
PRs	bpo-42236: Enhance _locale._get_locale_encoding() #23083 bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails #23086 bpo-42236: Enhance init and encoding documentation #23109 bpo-42236: os.device_encoding() respects UTF-8 Mode #23119

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-04.10:26:56.126>
created_at = <Date 2020-11-01.18:12:56.742>
labels = ['library', '3.10']
title = "os.device_encoding() doesn't respect the UTF-8 Mode"
updated_at = <Date 2020-11-04.10:26:56.125>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-11-04.10:26:56.125>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-04.10:26:56.126>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-11-01.18:12:56.742>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 42236
keywords = ['patch']
message_count = 8.0
messages = ['380156', '380158', '380159', '380166', '380234', '380261', '380313', '380314']
nosy_count = 2.0
nosy_names = ['vstinner', 'methane']
pr_nums = ['23083', '23086', '23109', '23119']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42236'
versions = ['Python 3.10']

[vstinner]

When the UTF-8 Mode is enabled, sys.stdout.encoding is always UTF-8, whereas os.devide_encoding(sys.stdout.fileno()) returns the locale encoding. os.devide_encoding() must return UTF-8 when the UTF-8 Mode is used.

[vstinner]

In bpo-42208, I added C _Py_GetLocaleEncoding() function and Python _locale._get_locale_encoding() function.

[vstinner]

New changeset 82458b6 by Victor Stinner in branch 'master':
bpo-42236: Enhance _locale._get_locale_encoding() (GH-23083)
82458b6

[vstinner]

New changeset e662c39 by Victor Stinner in branch 'master':
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
e662c39

[vstinner]

New changeset 4b9aad4 by Victor Stinner in branch 'master':
bpo-42236: Enhance init and encoding documentation (GH-23109)
4b9aad4

[methane]

I don't think UTF-8 mode should override os.device_encoding() on Windows.

UTF-8 mode can be used to ignore legacy locale encoding, and os.device_encoding() uses the locale encoding on Unix. So overriding it make sense.

But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.

[vstinner]

New changeset 3529718 by Victor Stinner in branch 'master':
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
3529718

[vstinner]

But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.

You're right. My final change leaves Windows unchanged.