h2py.py can fail with UnicodeDecodeError

[Arfrever]

BPO	13032
Nosy	@birkenfeld, @vstinner, @floppym
Files	h2py.py.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-07-30.15:45:43.888>
created_at = <Date 2011-09-23.08:37:40.682>
labels = []
title = 'h2py.py can fail with UnicodeDecodeError'
updated_at = <Date 2019-07-30.15:45:43.887>
user = 'https://bugs.python.org/Arfrever'

bugs.python.org fields:

activity = <Date 2019-07-30.15:45:43.887>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-07-30.15:45:43.888>
closer = 'vstinner'
components = ['Demos and Tools']
creation = <Date 2011-09-23.08:37:40.682>
creator = 'Arfrever'
dependencies = []
files = ['23230']
hgrepos = []
issue_num = 13032
keywords = ['patch', 'needs review']
message_count = 5.0
messages = ['144438', '145651', '159814', '348644', '348755']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'vstinner', 'Arfrever', 'floppymaster']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue13032'
versions = ['Python 3.2', 'Python 3.3']

[Arfrever]

Tools/scripts/h2py.py fails with UnicodeDecodeError when a header file contains characters undecodable in current locale. I suggest to use binary mode. I'm attaching a patch.

[vstinner]

Using your patch, h2py.py skips all statements that cannot be decoded from UTF-8, whereas unpatched h2py.py accepts all statements that can be decoded from the locale encoding. I don't know if it is intentional to accept non-ASCII statements. It is maybe safer to ensure that a statement is encodable to ASCII using:

try:
stmt.encode('ASCII')
exec(stmt, env)
except:
...

Anyway, I would prefer to just drop this script with all Lib/plat-*/ directoes. I reopened this topic on python-dev (I already asked when I was working on the sys.platform=="linux3" issue: bpo-12326).

[Arfrever]

UTF-8 is default encoding in Python 3, so statements with UTF-8 characters could be accepted.

Any strings are very rare in these statements. On my system, only generated TYPES.py contains 2 strings:
# Included from bits/select.h
__FD_ZERO_STOS = "stosq"
__FD_ZERO_STOS = "stosl"

/usr/include/bits/select.h contains:
# if __WORDSIZE == 64
# define __FD_ZERO_STOS "stosq"
# else
# define __FD_ZERO_STOS "stosl"
# endif

[vstinner]

I created bpo-37704 to propose to remove Tools/scripts/h2py.py.

[vstinner]

I just removed Tools/scripts/h2py.py in bpo-37704.