test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions

[michael-o]

BPO	34403
Nosy	@terryjreedy, @vstinner, @aixtools, @michael-o
PRs	bpo-34403: Skip test_utf8_mode.test_cmd_line() on HP-UX #8966 bpo-34403: Fix test_utf8_mode.test_cmd_line() on HP-UX #8967 bpo-34403: On HP-UX, force ASCII for C locale #8969 bpo-34523: Fix config_init_fs_encoding() for ASCII #10232 [3.7] bpo-34403: Fix initfsencoding() for ASCII #10233 [3.7] bpo-34403: Always implement _Py_GetForceASCII() #10235
Dependencies	bpo-34207: test_cmd_line test_utf8_mode test_warnings fail in all FreeBSD 3.x (3.8) buildbots
Files	c_locale.c py3.8-LC-C.text py3.8-default.text

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-08-29.13:49:04.403>
created_at = <Date 2018-08-14.12:57:12.659>
labels = ['3.7', 'tests', '3.8', 'type-bug', 'library']
title = 'test_utf8_mode.test_cmd_line() fails on HP-UX due to false assumptions'
updated_at = <Date 2018-10-30.15:43:10.318>
user = 'https://github.com/michael-o'

bugs.python.org fields:

activity = <Date 2018-10-30.15:43:10.318>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-08-29.13:49:04.403>
closer = 'vstinner'
components = ['Library (Lib)', 'Tests']
creation = <Date 2018-08-14.12:57:12.659>
creator = 'michael-o'
dependencies = ['34207']
files = ['47767', '47770', '47771']
hgrepos = []
issue_num = 34403
keywords = ['patch', '3.7regression']
message_count = 39.0
messages = ['323516', '323682', '323729', '323797', '324070', '324085', '324100', '324172', '324175', '324176', '324196', '324211', '324213', '324214', '324215', '324216', '324217', '324219', '324221', '324225', '324236', '324238', '324242', '324253', '324260', '324261', '324273', '324274', '324289', '324322', '324323', '324463', '324502', '328901', '328902', '328903', '328916', '328924', '328932']
nosy_count = 4.0
nosy_names = ['terry.reedy', 'vstinner', 'Michael.Felt', 'michael-o']
pr_nums = ['8966', '8967', '8969', '10232', '10233', '10235']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue34403'
versions = ['Python 3.7', 'Python 3.8']

[michael-o]

Running from 3.7 branch on HP-UX 11.31 ia64, 32 bit, big endian.
The test output is:

Re-running failed tests in verbose mode
Re-running test 'test_utf8_mode' in verbose mode
test_cmd_line (test.test_utf8_mode.UTF8ModeTests) ... FAIL
test_env_var (test.test_utf8_mode.UTF8ModeTests) ... ok
test_filesystemencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io (test.test_utf8_mode.UTF8ModeTests) ... ok
test_io_encoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_locale_getpreferredencoding (test.test_utf8_mode.UTF8ModeTests) ... ok
test_optim_level (test.test_utf8_mode.UTF8ModeTests) ... ok
test_posix_locale (test.test_utf8_mode.UTF8ModeTests) ... ok
test_stdio (test.test_utf8_mode.UTF8ModeTests) ... ok
test_xoption (test.test_utf8_mode.UTF8ModeTests) ... ok

======================================================================
FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 230, in test_cmd_line
>     check('utf8=0', [c_arg], LC_ALL='C')
>   File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
>     self.assertEqual(args, ascii(expected), out)
> AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
> - ['h\xc3\xa9\xe2\x82\xac']
> + ['h\udcc3\udca9\udce2\udc82\udcac']
>  : roman8:['h\xc3\xa9\xe2\x82\xac']
> 
> 

---

Ran 10 tests in 2.595s

FAILED (failures=1)
test test_utf8_mode failed
1 test failed again:
test_utf8_mode

== Tests result: FAILURE then FAILURE ==

1 test failed:
test_utf8_mode

1 re-run test:
test_utf8_mode

Total duration: 7 sec 265 ms
Tests result: FAILURE then FAILURE
Makefile:1066: recipe for target 'test' failed
gmake: *** [test] Error 2

I tried to understand the issue, but my Python knowledge is too low, especially I do not understand by a byte array "arg = 'h\xe9\u20ac'.encode('utf-8')" is passed as one arg to the forked process.

I highly assume that this is related to the non-standard, default character encoding on HP-UX: https://en.wikipedia.org/wiki/HP_Roman#HP_Roman-8 (roman8).

A stupid 8 bit encoding. The very same snippet on FreeBSD says:

$ LC_ALL=C python3.6 test_utf8.py
US-ASCII:[]

Willing to test and modify if someone tells what to do.

[terryjreedy]

You might get more information asking questions on python-list.

[michael-o]

Thanks, I'll do that. Hopefully I can provide a patch for. Though, I am convinced that I have to write a custom codec for roman8 to make all at stuff work flawlessly.

[aixtools]

Although the default is different (i.e., roman8 versus latin1 (iso8859-1)) both HP-UX and AIX (like Windows, cp1252) this issue and bpo-33347 are related.

As I mentioned in https://bugs.python.org/issue34347#msg323319 the string seen by self.get_output() is not the same string as "expected".

If I recall, there may be a way to almost get the two be the same - excect "expected" is a bytes object and the value returned as CLI output is a regular string.

I am thinking, maybe the "easy" way will be to add AIX, HP-UX, and others to skip this test. Rather than hard-code, do a query to see what the default is, and it it is not UTF-8 - skip the test.

In any case, it seems to be broken for any system that does not have UTF-8 as default.

[aixtools]

It might be as simple as what I saw for AIX:

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..3e918fd54c 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
         check('utf8', [arg_utf8])
         if sys.platform == 'darwin' or support.is_android:
             c_arg = arg_utf8
+        elif sys.platform.startswith("aix"):
+            c_arg = arg.decode('iso-8859-1')
         else:
             c_arg = arg_ascii
         check('utf8=0', [c_arg], LC_ALL='C')

so, adding below might be all that is needed:
+ elif sys.platform == "hpux":
+ c_arg = arg.decode('roman8')

[aixtools]

As the AIX complaint is (was once the PR merges):
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"

• ['h\xc3\xa9\xe2\x82\xac']
  + ['h\udcc3\udca9\udce2\udc82\udcac']

And the HP-UX complaint is:
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 223, in check
    self.assertEqual(args, ascii(expected), out)
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\udcc3\\udca9\\udce2\\udc82\\udcac']"
- ['h\xc3\xa9\xe2\x82\xac']
+ ['h\udcc3\udca9\udce2\udc82\udcac']

Maybe a change such as:

--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -219,6 +219,8 @@ class UTF8ModeTests(unittest.TestCase):
         check('utf8', [arg_utf8])
         if sys.platform == 'darwin' or support.is_android:
             c_arg = arg_utf8
+        elif (platform.system == "AIX") or
+              sys.platform.startswith("hp-ux"):
+            c_arg = arg.decode('iso-8859-1')
         else:
             c_arg = arg_ascii
         check('utf8=0', [c_arg], LC_ALL='C')

I mention this because it seems neither roman8 nor roman9 have 'official' iso names or alias (correct me if I am wrong).

[michael-o]

I think you are absoltely right.

In any case, it seems to be broken for any system that does not have UTF-8 as default.

You likely mean ASCII. Python assumes that LANG=C is ASCII which is not the case for AIX and HP-UX.

Your patch looks reasonable, I will try this on Monday. The problem is that there is no roman8 codec in Python. Maybe ISO-8859-1 will do it for the test, but I am still eager to add one.

I mention this because it seems neither roman8 nor roman9 have 'official' iso names or alias (correct me if I am wrong).

There are no ISO names because this is not an ISO encoding. This is an HP invention aka hp-roman8 (roman8, ibm-1051, r8, Cp1051).

Edit: there is roman8 support: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Lib/encodings/hp_roman8.py as well as aliases.

There are a few aliases missing: cp1051, ibm1051 and hp-roman8. This needs an additonal PR.

[michael-o]

So I changed the test code to:

diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..d9f8a3ed8b 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -208,7 +208,7 @@ class UTF8ModeTests(unittest.TestCase):
     def test_cmd_line(self):
         arg = 'h\xe9\u20ac'.encode('utf-8')
         arg_utf8 = arg.decode('utf-8')
-        arg_ascii = arg.decode('ascii', 'surrogateescape')
+        arg_ascii = arg.decode('roman8', 'surrogateescape')
         code = 'import locale, sys; print("%s:%s" % (locale.getpreferredencoding(), ascii(sys.argv[1:])))'

         def check(utf8_opt, expected, **kw):

and the output is:
======================================================================
FAIL: test_cmd_line (test.test_utf8_mode.UTF8ModeTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 224, in test_cmd_line
    check('utf8=0', [c_arg], LC_ALL='C')
  File "/var/osipovmi/cpython/Lib/test/test_utf8_mode.py", line 217, in check
    self.assertEqual(args, ascii(expected), out)
AssertionError: "['h\\xc3\\xa9\\xe2\\x82\\xac']" != "['h\\xfb\\u02cb\\xe3\\x82\\u02dc']"
- ['h\xc3\xa9\xe2\x82\xac']
+ ['h\xfb\u02cb\xe3\x82\u02dc']
 : roman8:['h\xc3\xa9\xe2\x82\xac']

I still don't understand that.

I believe that surrogate escape only works for ASCII and nothing else. If so, this test must be skipped on HP-UX and AIX.

[michael-o]

Maybe skipping the test is the best thing:
diff --git a/Lib/test/test_utf8_mode.py b/Lib/test/test_utf8_mode.py
index 26e2e13ec5..d6c4b321be 100644
--- a/Lib/test/test_utf8_mode.py
+++ b/Lib/test/test_utf8_mode.py
@@ -12,7 +12,7 @@ from test.support.script_helper import assert_python_ok, assert_python_failure

 MS_WINDOWS = (sys.platform == 'win32')
-
+HPUX = (sys.platform.startswith('hp-ux'))

 class UTF8ModeTests(unittest.TestCase):
     DEFAULT_ENV = {
@@ -205,6 +205,7 @@ class UTF8ModeTests(unittest.TestCase):
         self.assertEqual(out, 'UTF-8 UTF-8')

 @unittest.skipIf(MS_WINDOWS, 'test specific to Unix')

+ @unittest.skipIf(HPUX, 'test specific to Unix with ASCII default locale')
def test_cmd_line(self):
arg = 'h\xe9\u20ac'.encode('utf-8')
arg_utf8 = arg.decode('utf-8')

[michael-o]

Maybe Victor Stinner has some insights here.