On FreeBSD, Python 3 doesn't support support the POSIX locale

[vstinner]

BPO	34527
Nosy	@ncoghlan, @vstinner
PRs	bpo-34527: POSIX locale enables the UTF-8 Mode #8972 [3.7] bpo-34527: POSIX locale enables the UTF-8 Mode (GH-8972) #8974 [3.6] bpo-34527: Fix FreeBSD with POSIX locale #8975

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-28.13:06:41.173>
created_at = <Date 2018-08-28.09:35:24.881>
labels = ['interpreter-core', '3.7', '3.8']
title = "On FreeBSD, Python 3 doesn't support support the POSIX locale"
updated_at = <Date 2018-08-29.09:02:23.922>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-08-29.09:02:23.922>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-28.13:06:41.173>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-08-28.09:35:24.881>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 34527
keywords = ['patch']
message_count = 7.0
messages = ['324230', '324234', '324240', '324241', '324247', '324250', '324310']
nosy_count = 2.0
nosy_names = ['ncoghlan', 'vstinner']
pr_nums = ['8972', '8974', '8975']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34527'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[vstinner]

vstinner@freebsd$ env -i LC_ALL=POSIX ./python -m test -v test_cmd_line_script -m test_non_ascii
== CPython 3.8.0a0 (heads/master:096329f0b2, Aug 28 2018, 09:30:55) [Clang 4.0.0 (tags/RELEASE_400/final 297347)]
== FreeBSD-11.1-RELEASE-amd64-64bit-ELF little-endian
== cwd: /usr/home/vstinner/prog/python/master/build/test_python_13073
== CPU count: 2
== encodings: locale=US-ASCII, FS=ascii
Run tests sequentially
0:00:00 load avg: 0.48 [1/1] test_cmd_line_script
test_non_ascii (test.test_cmd_line_script.CmdLineTest) ... FAIL

======================================================================
FAIL: test_non_ascii (test.test_cmd_line_script.CmdLineTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/vstinner/prog/python/master/Lib/test/test_cmd_line_script.py", line 512, in test_non_ascii
    'stdout=%r stderr=%r' % (stdout, stderr))
AssertionError: "'./@test_13073_tmp\\udce7w\\udcf0.py'" != "'<filename encoding error>'"
- './@test_13073_tmp\udce7w\udcf0.py'
+ '<filename encoding error>'
 : stdout=b"'<filename encoding error>'\n" stderr=b"Failed checking if argv[0] is an import path entry\nUnicodeEncodeError: 'ascii' codec can't encode character '\\xe7' in position 17: ordinal not in range(128)"

---

Ran 1 test in 0.093s

FAILED (failures=1)
test test_cmd_line_script failed
test_cmd_line_script failed

== Tests result: FAILURE ==

1 test failed:
test_cmd_line_script

Total duration: 170 ms
Tests result: FAILURE

By the way, the UTF-8 Mode is not enabled by the POSIX locale:

vstinner@freebsd$ env -i LC_ALL=POSIX ./python -c 'import sys; print(sys.flags.utf8_mode)'
0

It contradicts the PEP-540 which says the the UTF-8 Mode is enabled by the POSIX locale:

https://www.python.org/dev/peps/pep-0540/#abstract

This mode is off by default, but is automatically activated when using the "POSIX" locale.

[vstinner]

New changeset 5cb2589 by Victor Stinner in branch 'master':
bpo-34527: POSIX locale enables the UTF-8 Mode (GH-8972)
5cb2589

[vstinner]

New changeset 65ef742 by Victor Stinner in branch '3.7':
bpo-34527: POSIX locale enables the UTF-8 Mode (GH-8972) (GH-8974)
65ef742

[vstinner]

Note: Python 2.7 is not affected: it doesn't have "force_ascii" and "LC_ALL=POSIX ./python -m test test_cmd_line_script" pass successfully.

[vstinner]

New changeset 4ff3887 by Victor Stinner in branch '3.6':
bpo-34527: Fix FreeBSD with POSIX locale (GH-8975)
4ff3887

[vstinner]

I wanted to quickly fix this issue since I'm working on 3 different issues where I got conflicts with these changes, and the other issues were impacted by this bug.

I sent an email to python-dev about this issue:
https://mail.python.org/pipermail/python-dev/2018-August/155052.html

[vstinner]

See also bpo-34485. Example:

New changeset 0b9ea4b by Victor Stinner in branch '3.7':
[3.7] bpo-34485: stdout uses surrogateescape on POSIX locale (GH-8986) (GH-8987)
0b9ea4b