os.execvpe() doesn't support surrogates in env

[vstinner]

BPO	8391
Nosy	@vstinner
Files	os_execvpe_surrogates-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-04-23.21:45:07.683>
created_at = <Date 2010-04-14.00:01:36.067>
labels = ['library', 'expert-unicode']
title = "os.execvpe() doesn't support surrogates in env"
updated_at = <Date 2010-04-25.22:40:52.782>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2010-04-25.22:40:52.782>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2010-04-23.21:45:07.683>
closer = 'vstinner'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2010-04-14.00:01:36.067>
creator = 'vstinner'
dependencies = []
files = ['16940']
hgrepos = []
issue_num = 8391
keywords = ['patch']
message_count = 9.0
messages = ['103100', '103101', '103107', '103277', '103278', '103279', '103459', '104055', '104175']
nosy_count = 2.0
nosy_names = ['vstinner', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue8391'
versions = ['Python 3.1', 'Python 3.2']

[vstinner]

It would be nice to support the PEP-383 (surrogateescape) for environment variables in os.execvpe(). Attached patch uses PyUnicode_AsEncodedString(val, Py_FileSystemDefaultEncoding, "surrogateescape") to encode an environment variable value.

I'm not sure that PyUnicode_AsEncodedString(val, Py_FileSystemDefaultEncoding, "surrogateescape") does always return a PyBytes object.

I not patched environment keys, but it might be useful.

[vstinner]

See also issue bpo-4036.

[vstinner]

See also bpo-8393.

[vstinner]

My patch doesn't work for types bytes and bytearray.

I noticed that py3k uses surrogateescape to encode environment variable values ;-)

[vstinner]

Other notes: Environment variable *names* use also surrogateescape "encoding". os.spawnve() and os.spawnvpe() should also be patched (the code should also be factorized).

[vstinner]

New version of the patch:

• factorize code between execve(), spawnve() and spawnvpe()
• support also surrogates in environment variable names
• support bytes and bytearray (bytearray cannot be used as a dictionary key, but my patch supports it)
• remove unrelated fix (my first patch contains a fix for os.system(), also about surrogates)

Because of the factorization, the error messages doesn't contain the function name anymore. spawnve() and spawnvpe() omit BEGINLIBPATH and ENDLIBPATH, as execve(): "that Would Confuse Programs if Passed On". I suppose that if execve() ignore them, spawn*e() should also ignore them.

I don't have an OS/2, so I'm unable to test my patch on this OS :-/

Note: The patch fixes also subprocess to support bytes and bytearray in the environment dictionary.

[vstinner]

Current code of execve() has a bug: it uses the length of the environment variable value in *characters* and not in *bytes* to allocate the "p" buffer. I remember that someone wrote a comment somewhere about that... The result is that the environment variable value is truncated by 1 byte.

Example (copy of http://dpaste.com/184803/):
-----------

$ cat test.py
#!/usr/bin/python
# -*- coding: utf-8 -*-

import os

env = {"VAR": "ćd"}
os.execve("test.sh", [], env)
$ cat test.sh
#!/bin/bash

declare -p VAR
$ python2.6 test.py
declare -x VAR="ćd"
$ python3.1 test.py
declare -x VAR="ć"
-----------

[vstinner]

Commited: r80421 (py3k), blocked in 3.1 (80422). The commit fixes also os.getenv() to support bytes environment name.

[vstinner]

I blocked the fix in Python 3.1 because it's non trivial and I prefer to avoid complex changes in Python 3.1. But then I realized that Python 3.1 has two bugs about environment variables.

It uses sys.getfilesystemencoding()+surrogateecape to decode variables and sys.getdefaultencoding()+strict to encode variables: the encoding is different!

It counts the number of *characters* to allocate the *byte* string buffer and so non-ASCII values are truncated.

So I decided to backport the fix: r80494.