Make _Py_char2wchar() and _Py_wchar2char() public

[vstinner]

BPO	18395
Nosy	@warsaw, @vstinner, @takluyver, @MojoVampire

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-08-01.10:36:33.655>
created_at = <Date 2013-07-07.14:00:36.528>
labels = []
title = 'Make _Py_char2wchar() and _Py_wchar2char() public'
updated_at = <Date 2014-08-01.10:36:33.654>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2014-08-01.10:36:33.654>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-08-01.10:36:33.655>
closer = 'vstinner'
components = []
creation = <Date 2013-07-07.14:00:36.528>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 18395
keywords = []
message_count = 5.0
messages = ['192557', '223393', '223396', '223430', '224483']
nosy_count = 5.0
nosy_names = ['barry', 'vstinner', 'python-dev', 'takluyver', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue18395'
versions = ['Python 3.4']

[vstinner]

The Python C API has two very useful functions: _Py_char2wchar() and _Py_wchar2char(). They must be used to handle correctly undecodable byte sequences. _Py_char2wchar() and _Py_wchar2char() use the surrogateescape error handler (PEP-383). _Py_char2wchar() forces also the ASCII encoding on FreeBSD and Solaris when the LC_CTYPE locale is C.

Py_Main() expects an array of wide character strings (wchar_t*) for the command line argument, whereas main() gets an array or byte strings (char*). _Py_char2wchar() must be used to be able to call Py_Main().

I propose the following names:

wchar_t* Py_DecodeLocale(const char* arg, size_t *size);
char* Py_EncodeLocale(const wchar_t *text, size_t *error_pos);

See Python/fileutils.c for more information about these functions.

Python 3.3 has already higher level functions (calling _Py_char2_wchar() and _Py_wchar2char()):

PyObject* PyUnicode_DecodeLocale(const char *str, const char *errors);
PyObject* PyUnicode_EncodeLocale(PyObject *unicode, const char *errors);

But these functions cannot be used before Python is initialized.

[MojoVampire]

How often do people need to convert to do platform independent locale encoding before Python is initialized? Encouraging use of platform dependent wchar_t's seems like a bad idea when PyUnicode abstracts away the difference ever since 3.3 released.

[takluyver]

You seem to need wchar_t to call Py_Main and Py_SetProgramName.

I think there's an example in the docs which is wrong, because it appears to pass a char* to Py_SetProgramName:
https://docs.python.org/3.4/extending/embedding.html#very-high-level-embedding

[vstinner]

You seem to need wchar_t to call Py_Main and Py_SetProgramName.

Yes, exactly.

[python-dev]

New changeset 93a798c7f270 by Victor Stinner in branch 'default':
Issue bpo-18395: Rename ``_Py_char2wchar()`` to :c:func:`Py_DecodeLocale`, rename
http://hg.python.org/cpython/rev/93a798c7f270

New changeset 94d0e842b9ea by Victor Stinner in branch 'default':
Issue bpo-18395, bpo-22108: Update embedded Python examples to decode correctly
http://hg.python.org/cpython/rev/94d0e842b9ea