site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set

[anjos]

BPO	30167
Nosy	@ned-deily, @methane, @anjos, @steverweber
PRs	bpo-30167: site: Ignore TypeError in abs_paths() #6731 bpo-30167: Remove __cached__ when removing __file__ #7415 [3.7] bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) #7606 [3.6] bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) #7607 bpo-30167: Add test for module.__cached__ is None #7617

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-30.11:32:45.168>
created_at = <Date 2017-04-26.08:53:33.616>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = 'site.main() does not work on Python 3.6 and superior if PYTHONSTARTUP is set'
updated_at = <Date 2018-11-30.11:32:45.167>
user = 'https://github.com/anjos'

bugs.python.org fields:

activity = <Date 2018-11-30.11:32:45.167>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-30.11:32:45.168>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2017-04-26.08:53:33.616>
creator = 'anjos'
dependencies = []
files = []
hgrepos = []
issue_num = 30167
keywords = ['patch']
message_count = 11.0
messages = ['292325', '292327', '292329', '316295', '318725', '319258', '319260', '319261', '319262', '326797', '330677']
nosy_count = 4.0
nosy_names = ['ned.deily', 'methane', 'anjos', 'steverweber']
pr_nums = ['6731', '7415', '7606', '7607', '7617']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30167'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[anjos]

Apparently, "import site; site.main()" does not seem to work anymore on Python 3.6 and superior.

The reason is a change on the behavior of "os.path.abspath(None)". Before Python 3.6, it used to report an AttributeError which is properly caught inside "site.abs_paths" (see: https://github.com/python/cpython/blob/master/Lib/site.py#L99), making it ignore __main__, one of sys.modules, which has __file__ and __cached__ set to None.

With Python 3.6 and superior, os.path.abspath(None) reports a TypeError, which makes calling "site.main()" raise an exception and stop.

How to reproduce: On python 3.6 or superior, do "import site; site.main()".

Expected behavior: Exception is properly caught and treated inside "site.abs_paths", ignoring modules in which __file__ and/or __cached__ are set to None.

[anjos]

More information: to reproduce the problem, don't use Python's "-c" command-line option. In this case "__main__" won't be inside sys.modules which mitigates the issue.

[anjos]

After further investigation, the issue can only be reproduced iff the user sets PYTHONSTARTUP which triggers "__main__" to appear in sys.modules and the problem to occur.

[steverweber]

ref to other projects that hit this issue.

https://gitlab.idiap.ch/bob/bob.buildout/issues/15
saltstack/salt#45080
https://groups.google.com/forum/#!msg/comp.lang.python/tpfiHAJhl9Y/hZj1f50vBQAJ

[methane]

cpython/Python/pythonrun.c

Lines 391 to 399 in f822549

	if (PyDict_SetItemString(d, "__file__", f) < 0) {
	Py_DECREF(f);
	goto done;
	}
	if (PyDict_SetItemString(d, "__cached__", Py_None) < 0) {
	Py_DECREF(f);
	goto done;
	}
	set_file_name = 1;

__file__ and __cached__ are added here. And,

cpython/Python/pythonrun.c

Lines 441 to 442 in f822549

	if (set_file_name && PyDict_DelItemString(d, "__file__"))
	PyErr_Clear();

Only __file__ is removed here. I feel __cached__ should be removed too.

On the other hand, if None is valid value for __cached__, site.abs_paths() should ignore it.

https://www.python.org/dev/peps/pep-3147/#file

It is recommended that when nothing sensible can be calculated, implementations should set the __cached__ attribute to None.

It seems it's valid, and recommended for some cases. So PR-6731 looks OK.

[ned-deily]

New changeset 2487f30 by Ned Deily (Steve Weber) in branch 'master':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731)
2487f30

[ned-deily]

New changeset ec4343c by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7606)
ec4343c

[ned-deily]

New changeset 3e12158 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-30167: Prevent site.main() exception if PYTHONSTARTUP is set. (GH-6731) (GH-7607)
3e12158

[ned-deily]

As I noted in the discussion on PR 6731, I think there should be a test for this so we don't break it again. But, since it seems that the problem has affected a number of users and projects and since the fix is easy and easily testable manually, I decided to merge the PR for 3.7.0rc1 and 3.6.6rc1. I am leaving the issue open for someone to supply a PR with a test case and for discussion of PR 7415. Thank you all!