[FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT

[vstinner]

BPO	35290
Nosy	@ncoghlan, @vstinner, @koobs
PRs	bpo-35290: Add debug info to test_c_locale_coercion #10631 bpo-34523: Fix C locale coercion on FreeBSD CURRENT #10672 [3.7] bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672) #10673

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-23.12:38:21.849>
created_at = <Date 2018-11-21.10:49:52.346>
labels = ['interpreter-core', '3.7', '3.8']
title = "[FreeBSD] test_c_locale_coercion doesn't support new C.UTF-8 locale of FreeBSD CURRENT"
updated_at = <Date 2018-11-23.12:38:21.848>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-11-23.12:38:21.848>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-23.12:38:21.849>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-21.10:49:52.346>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35290
keywords = ['patch']
message_count = 13.0
messages = ['330189', '330190', '330238', '330241', '330243', '330254', '330258', '330303', '330306', '330308', '330312', '330318', '330319']
nosy_count = 3.0
nosy_names = ['ncoghlan', 'vstinner', 'koobs']
pr_nums = ['10631', '10672', '10673']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35290'
versions = ['Python 3.7', 'Python 3.8']

[vstinner]

It seems like the commit 02e6bf7 (bpo-28604) broke test_c_locale_coercion on FreeBSD CURRENT Shared 3.x:

https://buildbot.python.org/all/#/builders/168/builds/247

0:19:33 load avg: 7.69 [401/419/2] test_c_locale_coercion failed -- running: test_decimal (40 sec 626 ms), test_lib2to3 (2 min 3 sec)
test_external_target_locale_configuration (test.test_c_locale_coercion.LocaleConfigurationTests) ... ok
test_LC_ALL_set_to_C (test.test_c_locale_coercion.LocaleCoercionTests) ... ok
test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) ... test_PYTHONCOERCECLOCALE_set_to_zero (test.test_c_locale_coercion.LocaleCoercionTests) ... ok

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_set (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE=None)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='1')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='1')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='1')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='1')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='1')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='true')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='true')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='true')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='true')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='true')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='false')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='false')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='false')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='false')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_not_zero (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='false')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (default_locale=True, PYTHONCOERCECLOCALE='warn')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 341, in _check_c_locale_coercion
    self._check_child_encoding_details(base_var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='C', PYTHONCOERCECLOCALE='warn')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[131 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'c',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='C', PYTHONCOERCECLOCALE='warn')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LANG', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='warn')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[143 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': 'invalid.ascii',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

======================================================================
FAIL: test_PYTHONCOERCECLOCALE_set_to_warn (test.test_c_locale_coercion.LocaleCoercionTests) (env_var='LC_CTYPE', nominal_locale='invalid.ascii', PYTHONCOERCECLOCALE='warn')
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 356, in _check_c_locale_coercion
    self._check_child_encoding_details(var_dict,
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-current/build/Lib/test/test_c_locale_coercion.py", line 231, in _check_child_encoding_details
    self.assertEqual(encoding_details, expected_details)
AssertionError: {'fsencoding': 'ascii', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''} != {'fsencoding': 'utf-8', 'stdin_info': 'utf-8:surrogateesc[130 chars]: ''}
- {'fsencoding': 'ascii',
?                 ^^^^^

+ {'fsencoding': 'utf-8',
? ^^^^^

'lang': '',
'lc_all': '',
'lc_ctype': 'c.utf-8',
'stderr_info': 'utf-8:backslashreplace',
'stdin_info': 'utf-8:surrogateescape',
'stdout_info': 'utf-8:surrogateescape'}

----------------------------------------------------------------------

Ran 6 tests in 6.205s

FAILED (failures=30)
test test_c_locale_coercion failed

[vstinner]

New changeset 7c2d570 by Victor Stinner in branch 'master':
bpo-35290: Add debug info to test_c_locale_coercion (GH-10631)
7c2d570

[vstinner]

Result of the additional debug info, on the FreeBSD CURRENT buildbot:

AVAILABLE_TARGETS = ['C.UTF-8']
EXPECTED_C_LOCALE_EQUIVALENTS = ['C', 'invalid.ascii']
EXPECTED_C_LOCALE_STREAM_ENCODING = 'ascii'
EXPECTED_C_LOCALE_FS_ENCODING = 'ascii'
EXPECT_COERCION_IN_DEFAULT_LOCALE = True
_C_UTF8_LOCALES = ('C.UTF-8', 'C.utf8', 'UTF-8')
_check_nl_langinfo_CODESET = True

[vstinner]

AVAILABLE_TARGETS = ['C.UTF-8']

Aha, that's something new in FreeBSD. In FreeBSD 11.2, this locale is invalid. The locale has been added to FreeBSD CURRENT (future FreeBSD 12) at November 4:
https://reviews.freebsd.org/D17833

See discussion on freebsd-current:

http://freebsd.1045724.x6.nabble.com/C-UTF-8-as-default-locale-td6292359.html

So it's unrelated to my change, but it's a bug in test_c_locale_coercion on FreeBSD 12.

[vstinner]

More FreeBSD commits, "Teach man(1) about C.UTF-8":
https://svnweb.freebsd.org/base?view=revision&revision=340128

---

EXPECTED_C_LOCALE_STREAM_ENCODING = 'ascii'
EXPECTED_C_LOCALE_FS_ENCODING = 'ascii'

I guess that the problem comes from that. But I'm not sure of what is the FS encoding for LC_ALL=C or LC_ALL=C.UTF-8.

[ncoghlan]

Ah, interesting. https://bugs.python.org/issue30672 covered the fact that this test was already weird on FreeBSD, but the current status is that it was expecting BSD variants to act somewhat like Mac OS X, not like Linux.

I'm wondering if we may need to make test_c_locale_coercion less opinionated, where for most platforms it's just a "don't segfault" test, and it's only for ones where we're certain about how we expect it to behave (i.e. Linux, Mac OS X, AIX, maybe FreeBSD) that we try to check the specifics.

[vstinner]

I'm wondering if we may need to make test_c_locale_coercion less opinionated, where for most platforms it's just a "don't segfault" test, and it's only for ones where we're certain about how we expect it to behave (i.e. Linux, Mac OS X, AIX, maybe FreeBSD) that we try to check the specifics.

For test_embed, I tried to be smart to guess the encodings (ex: hardcode some encodings depending on the platform) and... I gave up. It's very hard when you consider Linux, Windows, macOS, FreeBSD, but also HP-UX and AIX. Each platform has weird cases about encodings... Now the test spawns a Python subprocess to get the encoding. See get_stdio_encoding() and get_filesystem_encoding() in Lib/test/test_embed.py.

[vstinner]

Ok, I found the bug: it is related to the commit 905f1ac of bpo-34523. Python now uses ASCII for the filesystem encoding if the internal "force ASCII" mode is enabled in Py_DecodeLocale/Py_EncodeLocale.

FreeBSD CURRENT introduced a new C.UTF-8 locale which is used by C locale coercion (PEP-538).

_PyCoreConfig_Read() now uses ASCII for the filesystem encoding if the "force ASCII" mode is enable (to ensure that encodings are consistent everywhere Python).

Problem: The LC_CTYPE now uses UTF-8 (thanks to C locale coercion) whereas Python uses ASCII for the filesystem encoding since the "force ASCII" is still enabled.

I wrote PR 10672 to reset the "force ASCII" mode. I tested manually my fix on the FreeBSD CURRENT buildbot.

[vstinner]

Oh, I also modified Python 3.7 to respect the "force ASCII" mode in bpo-34403, commit 21220bb. Hopefully, this change is not part of the latest Python 3.7.1 bugfix release.

By the way, the bug only impacts FreeBSD CURRENT users (after Nov 4, after C.UTF-8 has been added) who have no locale set (LC_CTYPE is "C", ex: python3 running in an empty environment) and explicitly disable the UTF-8 mode... Well, I guess that very few users (I mean no one) are impacted :-)

[vstinner]

The bug is that sys.getfilesystemencoding() should be the locale encoding. On FreeBSD CURRENT, Python uses ASCII for the filesystem encoding, whereas the locale encoding is UTF-8.

CURRENT-amd64% env -i ./python -X utf8=0 -c 'import locale, sys; print(sys.getfilesystemencoding(), locale.getpreferredencoding())'
ascii UTF-8

[vstinner]

New changeset 353933e by Victor Stinner in branch 'master':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672)
353933e

[vstinner]

New changeset f6e323c by Victor Stinner in branch '3.7':
bpo-34523: Fix C locale coercion on FreeBSD CURRENT (GH-10672) (GH-10673)
f6e323c