Repository navigation
Add -P command line option and PYTHONSAFEENV environment variable #57684
Description
| BPO | 13475 |
|---|---|
| Nosy | @gpshead, @ncoghlan, @pitrou, @kristjanvalur, @vstinner, @mattheww, @jwilk, @merwok, @ericsnowcurrently |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = None
created_at = <Date 2011-11-25.00:46:33.649>
labels = ['type-feature', '3.9']
title = "Add '--mainpath'/'--nomainpath' command line options to override sys.path[0] initialisation"
updated_at = <Date 2022-02-24.02:55:12.913>
user = 'https://github.com/ncoghlan'bugs.python.org fields:
activity = <Date 2022-02-24.02:55:12.913>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = []
creation = <Date 2011-11-25.00:46:33.649>
creator = 'ncoghlan'
dependencies = []
files = ['25723', '25730', '25759']
hgrepos = []
issue_num = 13475
keywords = ['patch']
message_count = 37.0
messages = ['148295', '148305', '148310', '148336', '148350', '148378', '148379', '148394', '148433', '148458', '148552', '161676', '161677', '161685', '161686', '161689', '161691', '161695', '161696', '161749', '161926', '162677', '162708', '165164', '168428', '176160', '176522', '176526', '176527', '176529', '216789', '314041', '314193', '339624', '413868', '413871', '413877']
nosy_count = 11.0
nosy_names = ['gregory.p.smith', 'ncoghlan', 'pitrou', 'kristjan.jonsson', 'vstinner', 'mattheww', 'jwilk', 'eric.araujo', 'Arfrever', 'eric.snow', 'nisaac']
pr_nums = ['31542']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue13475'
versions = ['Python 3.9']
Activity
ncoghlan commentedon Nov 25, 2011
PEP-395 spends a lot of time discussing ways that the current automatic initialisation of sys.path[0] can go wrong, and even the proposed improvements in that PEP don't claim to fix the default behaviour for every possible scenario (just many of the most common ones).
The unittest module gets around similar problems with test autodiscovery by providing an explicit "-t" (for 'toplevel') command line option.
While '-t' is not available for the main interpreter executable (and nor is '-d' for directory), '-p' for "path0" is a possibility.
Directory execution (for example) would then be equivalent to:
python -p dirname dirname/main.py
A separate '--nopath0' option could also be provided to explicitly disable path initialisation based on the script being executed (see http://lists.debian.org/debian-python/2011/11/msg00058.html)
ericsnowcurrently commentedon Nov 25, 2011
+1 Both the -p and --nopath0 would be great additions and a good match for PEP-395.
So "-p ." would be equivalent to the current implicit sys.path[0] initialization, right? Would any other effects happen with "-p" than sys.path[0] initialization? I'll follow up with my one concern (the implicit "-p .") in a follow-up message.
ericsnowcurrently commentedon Nov 25, 2011
The current behavior is an implicit "-p .", which causes all sorts of hard-to-figure-out problems, most of which PEP-395 is rightly trying to fix. I'm suggesting that the next step would be to make "--nopath0" the default (rendering the flag unnecessary). Since this changes the status quo, I'll try to make the case here that it's worth it.
---
First of all, the current implicit sys.path[0] initialization for scripts has been around since forever. Why would we change it now? The inspiration is three-fold: we _said_ we got rid of implicit relative imports already, PEP-395 already has made a case for changing the status quo (though not to this degree), and a "-p" flag provides a simple mechanism to get the current behavior explicitly.
Like Nick says in PEP-395, "PEP-328 eliminated implicit relative imports from imported modules. This PEP proposes that the de facto implicit relative imports from main modules that are provided by the current initialisation behaviour for sys.path[0] also be eliminated."
I'm just saying we should go all the way, and that the "-p" flag would allow that. As far as I can tell, there are two meaningful use cases for the currently implicit sys.path[0]: When you are first _learning_ Python (not actually using it)... When you are working on a new project... Let's look at both, first relative to the currently implicit "-p .", then to the explicit site.
When a beginner is first learning Python, they learn at the interactive prompt or by running a single script. Pretty quickly they learn about writing their own modules and using import to load them.
Where do the modules live? In the CWD from which they are calling the Python executable. The current implicit "-p ." means they can import their modules right there, and not have to learn yet about sys.path. That's nice since they can concentrate their focus.
But later they _will_ run into hard-to-figure-out problems when they start relying on what are effectively implicit relative imports. Or they will run into one of the problems that PEP-395 explains, which are likewise head-scratchers. All of this can be pretty frustrating when you still don't know about things like the import machinery.
Consider how that beginner uses the implicit sys.path[0] behavior on a project, once they have some experience (hopefully I don't misrepresent something Nick explained to me). The src/ directory of the project won't be under sys.path yet. So to quickly test out their script, they change to src/ and run their script from there (something PEP-395 rightfully proposes fine-tuning). They conveniently don't have to manually fiddle with sys.path. That's a pretty neat trick.
The problem I have with this is that there's no difference between the sys.path workaround for the script and it's expected normal usage. If you aren't aware of what's going on behind-the-scenes, you may end up with some confusing import exceptions later on.
With a -p flag, it's really easy to say "-p ." and get the previous implicit behavior. Consider the impact on our beginner. The try to run their script that imports modules in their CWD, but end up with an ImportError because we no longer have an implicit "-p .".
How is this an improvement? No more mysterious default behavior. There's only one way it will fail and it will fail the same way every time. They'll learn pretty quickly (if not taught already) you have to use the -p flag if you want to import modules that aren't in the "standard" places. The ImportError message, special-cased for imports in __main__, could help in that regard.
<aside>
As an aside, implicit or explicit "--nopath0", the path to the file for the failed import can really help in related situations. Including it as an attribute on the ImportError, and especially in the error message, would rock. See bpo-1559549.
</aside>
For the "src/" use-case, the developer won't be able to rely on the implicit behavior anymore. A simple, explicit "-p ." fills the gap and makes it clear that they are doing something unusual. On top of that, they can use -p with the path to the "src/" directory from anywhere and not have to change their directory. All the better if -p takes advantage of the proposals in PEP-395.
In the end, this boils down to opt-in vs. opt-out. Either you opt-in to the sys.path[0] initialization by using -p, or you opt-out of the default implicit "-p ." by passing "--nopath0". It's a case of "explicit is better than implicit" without a clear reason, to me, that the status quo is worth the exception anymore in the face of the alternative.
My only concern would be backwards compatibility. How much do scripts out there rely on the implicit sys.path[0] initialization that aren't already broken by the removal of implicit relative imports? I simply don't know.
However, I do know that having the -p flag that Nick has recommended would be awesome.
merwok commentedon Nov 25, 2011
I don’t understand all the issues and it’s too late for me to read all the thread, but I hope these comments are helpful:
If our goal is to help naïve users, then one or two new options wouldn’t help (people want to run “python path/to/thing.py” and want it to Just Work™)
Getting a very helpful error message is second-best to Just Working™
This probably needs a wider audience, python-ideas or python-dev, and maybe a PEP to explain all the issues and all the considered solutions.
ericsnowcurrently commentedon Nov 25, 2011
First of all, I don't want Nick's proposal in this issue (the -p and --nopath0 flags) to be misunderstood because of me. His is a great idea that will make a really useful shortcut available and will _not_ change any current behavior.
My (overly) long message is an attempt to justify going with a more drastic variation, to which your response applies exclusively. I stand by what I said, but I would be quite happy with the original proposal, to which your response does not apply. :)
Just wanted to make all that clear so I don't get in the way of a good thing. :)
Your comments are spot on. Your concerns are exactly the ones I would need to assuage before my alternative would be acceptable. When I get a chance I'll take this to python-ideas.
ncoghlan commentedon Nov 26, 2011
Yeah, sorry Eric (Snow), you're trying to bite off *way* more than is reasonable by proposing to removing sys.path[0] auto-initialisation. While it has its problems, it's also far too entrenched in people's expections of Python's behaviour to consider removing at this late date. The earliest such a radical change could likely be considered is Python 4.
Éric (Araujo): the "just do the right thing" aspect is covered by PEP-395. This is an orthogonal proposal that allows power users the ability to override the automatic initialisation to handle those cases where it goes wrong.
That said, I may still make this suggestion part of PEP-395 rather than a distinct proposal, even though there's no direct dependency (in either direction) between this suggestion and the other ideas in that PEP.
ericsnowcurrently commentedon Nov 26, 2011
Yeah, the more I think about it, the more I agree it's Python 4 fodder.
merwok commentedon Nov 26, 2011
Eric: Actually I posted my comments after reading only Nick’s message, not yours, so no worry :)
Nick:
ncoghlan commentedon Nov 27, 2011
Zbigniew posted a nice summary of some of the issues sys.path[0] autoinitialisation can cause to python-dev: http://mail.python.org/pipermail/python-dev/2011-November/114668.html
mattheww commentedon Nov 27, 2011
The proposed --nopath0 option is something I've wished I had in the past.
If this is added, it would be good if it could be given a single-letter form too, because it's an option that would be useful in #! lines (they don't reliably support using more than one command-line argument, and single-letter switches can be combined while long-form ones can't).
ncoghlan commentedon Nov 29, 2011
I realised "-P" is available for use as the short form of the "suppress sys.path[0] initialisation" option.
So "-p" would override the calculation of sys.path[0], while "-P" would switch it off completely (similar to the way "-S" and "-E" switch off other aspects of the normal initialisation process).
Whether or not to provide "--path0" and "--nopath0" as long form alternatives would then be a separate question.
ericsnowcurrently commentedon May 26, 2012
Here's a stab at implementing -p/-P. There are a couple warnings that I'm not sure about and I've undoubtedly missed some detail, but it should be pretty close.
pitrou commentedon May 26, 2012
Before this is assigned a short option form, I would like to ask whether anybody but experts will be able to make a proper use of this option.
(I also don't understand what it adds over PYTHONPATH)
As for the patch, it lacks error checking when calling C API functions.
ericsnowcurrently commentedon May 26, 2012
Do you mean relative to a long form? And what would constitute improper use for the option?
As Nick noted earlier in this issue, the implicit setting of sys.path[0] is a convenience for some cases, and I can appreciate that. My complaint is that it is not an obvious behavior. Furthermore, in an import system that has eliminated implicit relative imports it can lead to confusing behavior. I consider Nick's proposal the best solution given the constraints we have.
If people do not use it, the status quo is unaffected. If they do use it, they will be explicitly affecting sys.path[0]. If they use it without knowing what it's for, it will at worst cause imports to happen from unexpected sources (or not happen at all). To me this doesn't seem that different from the way things are now when someone doesn't understand about sys.path[0] initialization.
It provides an explicit alternative to the default implicit insertion to sys.path[0]. If the default behavior were no implicit initialization, then I'd agree that PYTHONPATH is sufficient.
Ah, yes. Spaced it. Patch updated (and warnings fixed).
This brings me to ask what the behavior should be when we have errors come back from those C API functions? In the patch I just have it fall back to the default sys.path[0] behavior. However, wouldn't an error indicate a deeper problem? If so, shouldn't Py_FatalError() be called?
pitrou commentedon May 26, 2012
Yes.
As a replacement of PYTHONPATH or "setup.py install" or "setup.py
develop", I guess.
Agreed. I'm not arguing against the principle.
Ah, indeed, I'd missed that.
Well, no, errors should not pass silently. The error should be
propagated. Here, it probably means print the error and abort (or
whatever strategy the rest of the function adopts).
Py_FatalError() is a low-level exit; it dumps core. PyErr_Print()
followed by a proper exit may be better.
36 remaining items
vstinner commentedon May 5, 2022
If #31542 is merged, it will become possible to deprecate PySys_SetArgv() and PySys_SetArgvEx() in favor of the new PyConfig API: PySys_SetArgvEx(argc, argv, 0) can then be implemented with safe_path=1.
gh-57684: Add -P cmdline option and PYTHONSAFEPATH env var (#31542)
gh-57684: Update tests for PYTHONSAFEPATH=1 (#92358)
gh-57684: Document safe path in What's New in Python 3.11 (#92362)
[-]Add '--mainpath'/'--nomainpath' command line options to override sys.path[0] initialisation[/-][+]Add -P command line option and PYTHONSAFEENV environment variable[/+]AA-Turner commentedon Jun 7, 2022
@vstinner is there anything left to do on this issue?
A
vstinner commentedon Jun 7, 2022
Yes: merge or reject #92361 (-p option).
kumaraditya303 commentedon Jun 18, 2022
Closing as #92361 was rejected.
ericsnowcurrently commentedon Jul 5, 2022
I don't think the fate of this issue is directly tied to #92361. @vstinner, is there some reason it should be?
merwok commentedon Jul 6, 2022
This issue was about adding
-P, which was done, then AT asked what was left to be done and VS replied with that other PR to add-p, so that’s why KA closed this.vstinner commentedon Jul 6, 2022
Since 2011, the issue title and the proposed solution changed multiple times. I added -P command line option to Python 3.11: https://docs.python.org/dev/using/cmdline.html#cmdoption-P
It seems like adding -p is not needed only the default sys.path behavior changes. I wanted to add -p for PYTHONSAFEPATH, but I didn't convince anyone that it's required, so for now, there is no -p option.
IMO the initial issue has been fixed, so I close the issue.
ericsnowcurrently commentedon Jul 6, 2022
Thanks for getting this done, @vstinner!