Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony))

[palmkevin]

BPO	11320
Nosy	@warsaw, @ncoghlan, @pitrou, @vstinner, @tarekziade, @ericsnowcurrently, @zooba, @melvyn-sopacua

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-27.15:33:59.760>
created_at = <Date 2011-02-25.14:32:02.620>
labels = ['interpreter-core', 'type-bug']
title = 'Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony))'
updated_at = <Date 2019-05-27.15:33:59.759>
user = 'https://bugs.python.org/palmkevin'

bugs.python.org fields:

activity = <Date 2019-05-27.15:33:59.759>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-27.15:33:59.760>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2011-02-25.14:32:02.620>
creator = 'palm.kevin'
dependencies = []
files = []
hgrepos = []
issue_num = 11320
keywords = []
message_count = 22.0
messages = ['129372', '129395', '129396', '131101', '131131', '131137', '131292', '131293', '131297', '131301', '131316', '131317', '131318', '131324', '144244', '249276', '249311', '249321', '249342', '249383', '276050', '343647']
nosy_count = 11.0
nosy_names = ['barry', 'ncoghlan', 'pitrou', 'vstinner', 'tarek', 'srid', 'palm.kevin', 'eric.snow', 'dbzhang800', 'steve.dower', 'msopacua']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11320'
versions = ['Python 3.2', 'Python 3.3', 'Python 3.4', 'Python 3.5']

[palmkevin]

The new API method Py_SetPath seems bugged. When executing the following code, then the application crashes on Py_Initialize()-call:
#include "Python.h"

main(int argc, char **argv)
{
  Py_SetPath(Py_GetPath());
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

The raised exception is the following:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 332, in _init_posix
    _parse_makefile(makefile, vars)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 220, in _parse_makefile
    with open(filename, errors="surrogateescape") as f:
IOError: [Errno 2] No such file or directory: 'lib/python3.2/config-3.2m/Makefile'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 529, in <module>
    main()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 517, in main
    known_paths = addusersitepackages(known_paths)
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 263, in addusersitepackages
    user_site = getusersitepackages()
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 238, in getusersitepackages
    user_base = getuserbase() # this will also set USER_BASE
  File "/usr/labsolution/python32/lib/python3.2/site.py", line 228, in getuserbase
    USER_BASE = get_config_var('userbase')
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 590, in get_config_var
    return get_config_vars().get(name)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 487, in get_config_vars
    _init_posix(_CONFIG_VARS)
  File "/usr/labsolution/python32/lib/python3.2/sysconfig.py", line 337, in _init_posix
    raise IOError(msg)
IOError: invalid Python installation: unable to open lib/python3.2/config-3.2m/Makefile (No such file or directory)

(perhaps linked to issue bpo-10743 ?!)

[srid]

This issue is potentially breaking virtualenv5,
http://code.google.com/p/virtualenv5/issues/detail?id=6

[pitrou]

Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be provided by virtualenv (by copying it, I guess)?

[palmkevin]

@sridhar: Could you please provide input for the question asked by Antoine?
I'd love to have this issue fixed for next Python release.

[ncoghlan]

Note that this will be fixed for 3.3, as siteconfig will be updated to use static data (generated during the build process) rather than relying on build artifacts being available at runtime.

However, virtualenv (et al) will still need to provide the relevant static config file in the isolated environments.

[srid]

[pitrou]

Can you explain why this is a problem in Python?
Can't lib/python3.2/config-3.2m/Makefile simply be
provided by virtualenv (by copying it, I guess)?

Yes, I believe virtualenv already does that (or symlinks to it). Python 3.2 changed the path to config and include directories for some reason, viz.

$ ls -d /opt/ActivePython-3.*/lib/python3.?/*config*/
/opt/ActivePython-3.1/lib/python3.1/config/
/opt/ActivePython-3.2/lib/python3.2/config-3.2m/
$ 

and:

$ ls -d /opt/ActivePython-3.*/include/python3.?*
/opt/ActivePython-3.1/include/python3.1  
/opt/ActivePython-3.2/include/python3.2m
$

It is possible that virtualenv is hardcoding the relative path to 'config' (and 'include') directories and thus failing to find the new 'config-3.2m' dir. If that is the case, this is not a problem with Python.

Although msg129372 does point to a Python bug, it may or may not be related to the virtualenv issue noted earlier. Given that virtualenv doesn't officially Python 3 and virtualenv5 is more of a hack, I haven't investigated into this much.

Does that answer your question?

[pitrou]

Sridhar: I'm sorry, but at this point you should investigate a bit more into the actual causes of the problem. I'm not going to dig in virtualenv myself. Also, since there's already bpo-10743 for the virtualenv issue, perhaps further virtualenv-related comments should be posted there.

(Palm, I take it your issue doesn't have anything to do with virtualenv?)

[pitrou]

Ok, Palm's example even segfaults here:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff72452bc in free () from /lib64/libc.so.6
(gdb) bt
#0 0x00007ffff72452bc in free () from /lib64/libc.so.6
#1 0x000000000042a591 in Py_SetPath (
path=0x88c330 L"/home/antoine/cpython/default/usr/lib/python33.zip:/home/antoine/cpython/default/usr/lib/python3.3/:/home/antoine/cpython/default/usr/lib/python3.3/plat-linux2:/home/antoine/cpython/default/usr/lib/py"...)
at ./Modules/getpath.c:729
#2 0x0000000000416600 in main ()

The problem is calling free() on a pointer to statically allocated memory.

[pitrou]

Crash fixed in a791dd7d51f3 (3.2) and b4104ffd5127 (3.3), but the original problem remains, or a variant of it. I now get:

Init
Fatal Python error: Py_Initialize: Unable to get the locale encoding
LookupError: no codec search functions registered: can't find encoding
Abandon

[pitrou]

Ok, the issue here is that you can't call Py_SetPath() on the point returned by Py_GetPath(), since the memory refered to that pointer is free()d at the beginning of Py_SetPath(). The following works, though:

main(int argc, char **argv)
{
  wchar_t *path, *newpath;
  path = Py_GetPath();
  newpath = malloc((wcslen(path) + 1) * sizeof(wchar_t));
  wcscpy(newpath, path);
  Py_SetPath(newpath);
  free(newpath);
  printf("Init\n");
  Py_Initialize();
  printf("-- END\n");
}

Perhaps we could modify Py_SetPath() so that it copies the new path first before deallocating the old one, but I'm not sure I see the point of calling Py_SetPath() with the pointer returned by Py_GetPath().