The Modules/ directory should not be added to sys.path

[vstinner]

BPO	22285
Nosy	@loewis, @Yhg1s, @pitrou, @vstinner, @ned-deily, @skrah
Superseder	bpo-17095: Shared modules built with Modules/Setup are not found when run from build directory
Files	getpath_no_modules.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-09-08.00:25:40.263>
created_at = <Date 2014-08-27.10:07:47.847>
labels = []
title = 'The Modules/ directory should not be added to sys.path'
updated_at = <Date 2014-09-08.00:25:40.242>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2014-09-08.00:25:40.242>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2014-09-08.00:25:40.263>
closer = 'ned.deily'
components = []
creation = <Date 2014-08-27.10:07:47.847>
creator = 'vstinner'
dependencies = []
files = ['36484']
hgrepos = []
issue_num = 22285
keywords = ['patch']
message_count = 9.0
messages = ['225955', '225957', '225959', '225961', '225974', '225975', '225982', '225991', '226549']
nosy_count = 7.0
nosy_names = ['loewis', 'twouters', 'pitrou', 'vstinner', 'ned.deily', 'Arfrever', 'skrah']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '17095'
type = None
url = 'https://bugs.python.org/issue22285'
versions = ['Python 3.4', 'Python 3.5']

[vstinner]

When Python is built from source, the "Modules/" subdirectory is added to sys.path on UNIX. I don't understand why: it does not contain .py files nor .so dynamic modules. Dynamic modules are built in "build/lib.linux-x86_64-3.5-pydebug".

A side effect of adding Modules/ subdirectory to sys.path is that Modules subdirectories (ex: _sqlite, _ctypes, _io, _decimal, etc.) can be used as packages.

For example, when the _decimal module cannot be compiled, Lib/decimal.py uses Modules/_decimal as a Python package which is wrong. The decimal becomes an empty module (no symbol, because Modules/_decimal does not contain an __init__.py nor any .so file) because decimal.py uses "import _decimal" at the end (see decimal.py).

Attached patch removes Modules/ from sys.path on UNIX.

Maybe adding Modules/ to sys.path was useful before the introduction of pybuildir.txt? See issue bpo-9589 and the changeset 4742e7aea2f5 (and the issue bpo-586680).

[vstinner]

A side effect of this issue is that when the _decimal cannot be build (ex: bpo-22283), the Python implementation of the decimal cannot be used. Extract of buildbot test logs related to bpo-22283:
---
Failed to build these modules:
_decimal

(...)
File "/usr/home/buildbot/buildarea/3.x.krah-freebsd/build/Lib/test/test_statistics.py", line 15, in <module>
from decimal import Decimal
ImportError: cannot import name 'Decimal'
---
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/7213/steps/test/logs/stdio

[vstinner]

See also issue bpo-22280 for the case of the wrong _decimal package.

[loewis]

The motivation for this feature is that modules built as shared libraries through Modules/Setup end up in Modules, so Modules is added so that they are found.

I'd like to preserve support for building dynamic extension modules through Modules/Setup, but they don't have to live in Modules; putting them into build (say) would be fine as well.

[vstinner]

I'd like to preserve support for building dynamic extension modules through Modules/Setup, but they don't have to live in Modules; putting them into build (say) would be fine as well.

Does it mean that you are ok to commit my patch?

[skrah]

I think we have this behavior since 6c468df214dc and 227ce85bdbe0 (bpo-17095).

[vstinner]

Another bug linked to the bad Modules/_ctypes package:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8549/steps/test/logs/stdio

test test_ctypes crashed -- Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/regrtest.py", line 1271, in runtest_inner
    the_module = importlib.import_module(abstest)
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2203, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2186, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2175, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1149, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1420, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_ctypes.py", line 3, in <module>
    from ctypes.test import load_tests
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/ctypes/__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
ImportError: cannot import name 'Union'

[loewis]

Does it mean that you are ok to commit my patch?

No, because it will break bpo-17095 again. A proper patch would do what you do, plus find some other solution to bpo-17095. Just reverting 6c468df214dc is not ok.

[ned-deily]

Closing as a duplicate of the re-opened bpo-17095 (since the changes there haven't been released yet)