android: test_sys fails

[xdegaye]

BPO	26920
Nosy	@malemburg, @loewis, @vstinner, @xdegaye, @serhiy-storchaka, @moreati
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	test_ioencoding_nonascii.patch test_c_locale_surrogateescape.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/xdegaye'
closed_at = <Date 2016-11-16.07:19:52.513>
created_at = <Date 2016-05-03.09:26:22.543>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'android: test_sys fails'
updated_at = <Date 2017-03-31.16:36:16.821>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:16.821>
actor = 'dstufft'
assignee = 'xdegaye'
closed = True
closed_date = <Date 2016-11-16.07:19:52.513>
closer = 'xdegaye'
components = ['Interpreter Core']
creation = <Date 2016-05-03.09:26:22.543>
creator = 'xdegaye'
dependencies = []
files = ['42844', '42845']
hgrepos = []
issue_num = 26920
keywords = ['patch']
message_count = 12.0
messages = ['264705', '265463', '265465', '265480', '265509', '265511', '265512', '265518', '265522', '265548', '279822', '280917']
nosy_count = 7.0
nosy_names = ['lemburg', 'loewis', 'vstinner', 'xdegaye', 'python-dev', 'serhiy.storchaka', 'Alex.Willmer']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26920'
versions = ['Python 3.6', 'Python 3.7']

[xdegaye]

test_sys fails on an android emulator running an x86 system image at API level 21.

...
test_implementation (test.test_sys.SysModuleTest) ... ok
test_intern (test.test_sys.SysModuleTest) ... ok
test_ioencoding (test.test_sys.SysModuleTest) ... ok
test_ioencoding_nonascii (test.test_sys.SysModuleTest) ... Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe6' in position 0: ordinal not in range(
128)
FAIL
test_is_finalizing (test.test_sys.SysModuleTest) ... ok
test_lost_displayhook (test.test_sys.SysModuleTest) ... ok
...

======================================================================
FAIL: test_c_locale_surrogateescape (test.test_sys.SysModuleTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_sys.py", line 709, in test_c_locale_surr
ogateescape
    'stdin: surrogateescape\n'
AssertionError: 'stdin: strict\nstdout: strict\nstderr: backslashreplace\n' != 'stdin: surrogateesca
pe\nstdout: surrogateescape\nstderr: backslashreplace\n'
- stdin: strict
- stdout: strict
+ stdin: surrogateescape
+ stdout: surrogateescape
  stderr: backslashreplace

======================================================================
FAIL: test_ioencoding_nonascii (test.test_sys.SysModuleTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_sys.py", line 638, in test_ioencoding_no
nascii
    self.assertEqual(out, os.fsencode(test.support.FS_NONASCII))
AssertionError: b'' != b'\xc3\xa6'

---

Ran 43 tests in 0.913s

FAILED (failures=2, skipped=2)
test test_sys failed
1 test failed:
test_sys
Total duration: 0:00:01

[xdegaye]

test_ioencoding_nonascii does not fail when LANG is set to en_GB.UTF-8 in the environment.

[xdegaye]

test_ioencoding_nonascii does not fail when LANG is set to en_GB.UTF-8 in the environment.

When LANG is not set, we have on an android emulator:

>>> from test.support import FS_NONASCII
>>> print(FS_NONASCII)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\xe6' in position 0: ordinal not in range(128)

But on linux when LANG is not set:

$ LANG= ./python
Python 3.6.0a0 (default:eee959fee5f5+, May 13 2016, 11:32:27) 
[GCC 6.1.1 20160501] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from test.support import FS_NONASCII
>>> print(FS_NONASCII)
None
>>> 

And test_ioencoding_nonascii is skipped accordingly on linux for the following printed reason: 'requires OS support of non-ASCII encodings'.

On Android, os.fsdecode(os.fsencode('\xe6')) does not raise UnicodeError in the test.support module. And locale.getpreferredencoding() returns 'ascii'. So it seems that this criterion (FS_NONASCII is not None) is not sufficient to decide when this test should be run and not skipped.

[xdegaye]

On Android, os.fsdecode(os.fsencode('\xe6')) does not raise UnicodeError in the test.support module.

Because of changeset ad6be34ce8c9.

[xdegaye]

This patch fixes test_ioencoding_nonascii.

[xdegaye]

About the failures in test_c_locale_surrogateescape, it seems that on Android the locale is not set correctly on startup when LC_ALL is set to C:

root@generic_x86:/data/local/tmp # LC_ALL=C python
Python 3.6.0a0 (default:eee959fee5f5+, May 14 2016, 10:19:09) 
[GCC 4.9 20150123 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale; locale.getlocale()
('en_US', 'UTF-8')
>>> locale.setlocale(locale.LC_ALL, '')
'C'
>>> locale.getlocale()
(None, None)
>>>

[xdegaye]

This patch fixes the locale setting on startup when the LC_ALL environment variable is set to C, and as a consequence test_c_locale_surrogateescape does not fail anymore. Note that on Android HAVE_LANGINFO_H is undefined, see issue bpo-22747.

[serhiy-storchaka]

See also bpo-19058.

[xdegaye]

With Serhiy's latest patch named sys_test_ioencoding.patch in bpo-19058, test_sys fails only in test_c_locale_surrogateescape on the android-21-x86 emulator.

[vstinner]

test_c_locale_surrogateescape.patch LGTM.

[xdegaye]

For the record, on the Android emulator we have now (not sure where this change has been made):
>>> sys.getfilesystemencoding()
'utf-8'
>>> locale.getpreferredencoding(False)
'ascii'

So test_ioencoding_nonascii succeeds now. Anyway the problem with this test is being fixed at bpo-19058.