on Android python fails to decode/encode command line arguments

[xdegaye]

BPO	26919
Nosy	@vstinner, @xdegaye, @moreati
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Dependencies	bpo-27027: add the 'is_android' attribute to test.support
Files	retrofit_osx.patch retrofit_osx_2.patch: test description changed retrofit_osx_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/xdegaye'
closed_at = <Date 2016-12-15.20:37:29.112>
created_at = <Date 2016-05-03.09:18:35.370>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'on Android python fails to decode/encode command line arguments'
updated_at = <Date 2017-03-31.16:36:20.029>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:20.029>
actor = 'dstufft'
assignee = 'xdegaye'
closed = True
closed_date = <Date 2016-12-15.20:37:29.112>
closer = 'xdegaye'
components = ['Interpreter Core']
creation = <Date 2016-05-03.09:18:35.370>
creator = 'xdegaye'
dependencies = ['27027']
files = ['42838', '42839', '42911']
hgrepos = []
issue_num = 26919
keywords = ['patch']
message_count = 10.0
messages = ['264703', '264800', '265458', '265462', '265550', '265916', '279803', '283340', '283341', '283342']
nosy_count = 4.0
nosy_names = ['vstinner', 'xdegaye', 'python-dev', 'Alex.Willmer']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26919'
versions = ['Python 3.6', 'Python 3.7']

[xdegaye]

test_cmd_line fails on an android emulator running an x86 system image at API level 21.

======================================================================
FAIL: test_non_ascii (test.test_cmd_line.CmdLineTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/test_cmd_line.py", line 141, in test_non_ascii
    assert_python_ok('-c', command)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 135, in assert_python_ok
    return _assert_python(True, *args, **env_vars)
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/test/support/script_helper.py", line 121, in _assert_python
    err))
AssertionError: Process return code is 1
command line: ['/data/local/tmp/python/bin/python', '-X', 'faulthandler', '-I', '-c', "assert(ord('\xe6') == 230)"]

stdout:
---

---

stderr:
---
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
---

----------------------------------------------------------------------
Ran 33 tests in 1.943s

FAILED (failures=1, skipped=1)
test test_cmd_line failed
1 test failed:
test_cmd_line
Total duration: 0:00:03

[xdegaye]

There is a related issue bpo-4388 although different, and issue bpo-22747 may also be related.

[xdegaye]

This patch follows Victor suggestion in msg230407 and brings the changes made in issue bpo-4388 and issue bpo-16416 to the Android platform. As a consequence, it adds a new test for Android in test_cmd_line named test_osx_android_utf8 (was previously test_osx_utf8). The platform.android_ver() function is defined by issue bpo-26855: "add platform.android_ver() for android".

With this patch test_cmd_line does not have any failure.

[xdegaye]

Patch updated after vadmium review.

[vstinner]

I commented retrofit_osx_2.patch on the review.

[xdegaye]

This new patch adds a dependency to issue bpo-27027.

[xdegaye]

An interactive session confirms that the problem is indeed with the command line arguments of python invoked by subprocess (and the problem is fixed by the patch):

>>> from test.support import FS_NONASCII
>>> cmd = "assert(ord(%r) == %s)" % (FS_NONASCII, ord(FS_NONASCII))
>>> exec(cmd)
>>> import subprocess, sys
>>> subprocess.run([sys.executable, '-c', cmd])
Unable to decode the command from the command line:
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 12-13: surrogates not allowed
CompletedProcess(args=['/data/data/org.bitbucket.pyona/python/bin/python', '-c', "assert(ord('\xe6') == 230)"], returncode=1)