No way to recover original argv with python -m

[bdarnell]

BPO	14208
Nosy	@warsaw, @brettcannon, @birkenfeld, @rhettinger, @ncoghlan, @pitrou, @jwilk, @merwok, @florentx, @bdarnell

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-03-20.06:34:44.534>
created_at = <Date 2012-03-06.06:53:22.908>
labels = ['type-feature']
title = 'No way to recover original argv with python -m'
updated_at = <Date 2017-03-20.06:34:44.505>
user = 'https://github.com/bdarnell'

bugs.python.org fields:

activity = <Date 2017-03-20.06:34:44.505>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2017-03-20.06:34:44.534>
closer = 'ncoghlan'
components = []
creation = <Date 2012-03-06.06:53:22.908>
creator = 'Ben.Darnell'
dependencies = []
files = []
hgrepos = []
issue_num = 14208
keywords = []
message_count = 13.0
messages = ['155002', '155003', '155044', '155053', '155073', '155102', '179845', '179852', '179855', '179856', '179882', '180455', '289874']
nosy_count = 12.0
nosy_names = ['barry', 'brett.cannon', 'georg.brandl', 'rhettinger', 'ncoghlan', 'pitrou', 'jwilk', 'eric.araujo', 'Arfrever', 'flox', 'Ben.Darnell', 'danielsh']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14208'
versions = ['Python 3.4']

[bdarnell]

I have a script which attempts to re-invoke itself using sys.argv, but it fails when run with "python -m package.module". The problem is that the handling of -m (via the runpy module) rewrites sys.argv as if it were run as "python package/module.py", but the two command lines are not equivalent: With -m the current directory is inserted at the head of sys.path, but without -m it's the directory containing module.py. The net effect is that the initial run of "python -m package.module" works as expected, but when it re-runs itself as "python package/module.py" the imports from module.py are effectively relative instead of absolute.

One possible solution would be to provide an immutable sys.__argv__ (by analogy with sys.__stdout__ and friends).

[birkenfeld]

I agree this would be useful. It would be even more useful to have an __argv__ that includes all command-line flags given to Python, such as -Wi.

[ncoghlan]

This is closely related to PEP-395, since multiprocessing currently hits the same issue in trying to figure out the correct setting for sys.argv0.

I quite like the sys.__argv__ idea for preserving the *exact* underlying command line (Alex Gaynor was recently asking for something similar to this).

In the meantime, it should be possible to work around the problem by running the affected subprocess invocations (i.e. when __main__.__package__ exists and is not empty) with something like:

    launch_template = """
    import runpy, sys
    sys.argv[:] = {argv}
    sys.path[:] = {path}
    runpy._run_module_as_main({main})
    """
    import sys, subprocess, os.path, __main__
    main_base = os.path.basename(__main.__file__).splitext()[0]
    main_ref = __main__.__package__ + "." + main_base
    launch = launch_template.format(argv=sys.argv, path=sys.path, main=main_ref)
    subprocess.call(launch, shell=True, executable=sys.executable)
   
Note: the above isn't tested code, since it's an approach that only occurred to me recently and I haven't had a chance to fully explore it myself. However, if it works, we could make use of it in 2.7 and 3.2 to fix multiprocessing's current misbehaviour on Windows.

[rhettinger]

Instead of sys.__argv__, may I suggest sys.argv_original or somesuch.

[ncoghlan]

In framing a question for Raymond regarding his preference for avoiding the __argv__ name, I realised I agreed with him. My reasoning is that, when a Python process starts, sys.stdin is sys.__stdin__, sys.stdout is sys.__stdout__ and sys.stderr is sys.__stderr__. The dunder versions capture the original values as created by the interpreter initialisation, not the raw OS level file descriptors.

The new attribute proposed here is different - it's not an immutable copy of the original value of sys.argv, it's a *different* sequence altogether. The analogy with the standard stream initial value capture created by the use of sys.__argv__ would actually be misleading rather than helpful.

For the same reason, Raymond's specific argv_original suggestion doesn't really work for me. Alas, I can think of several other possible colours for that particular bikeshed (such as argv_os, argv_main, argv_raw, argv_platform, argv_executable) without having any particular good way of choosing between them.

[birkenfeld]

I agree. Maybe I may throw "full_argv" or "executable_argv" (i.e. to be used with exec([sys.executable] + sys.executable_arg)) in the air?

[ncoghlan]

For PEP-432, I'm proposing to expose this as sys._configuration.raw_argv

[pitrou]

For PEP-432, I'm proposing to expose this as sys._configuration.raw_argv

I think sys.raw_argv would be as good. It's not really (not only) a
configuration item.

[ncoghlan]

I'm not seeing a good justification for doing anything more, though:

• needing to access this information is an exceedingly niche use case
• I don't see any way for raw_argv to be useful in a cross-implementation manner.
• the exposure as sys._configuration.raw_argv is a natural consequence of the PEP-432 implementation (it's there to allow an embedding application to override the command line arguments seen by CPython)

Elevating it to part of the language standard doesn't make sense, while exposing it as an implementation feature of the CPython initialisation process seems more reasonable.

[pitrou]

I'm not seeing a good justification for doing anything more, though:

• needing to access this information is an exceedingly niche use case
• I don't see any way for raw_argv to be useful in a cross-implementation manner.

I expect it to be useful in the "launch (almost) the same command in the
same way" situation.
Not that it happens often :-)

[danielsh]

Antoine Pitrou wrote on Sun, Jan 13, 2013 at 10:19:20 +0000:

Antoine Pitrou added the comment:

> I'm not seeing a good justification for doing anything more, though:
>
> - needing to access this information is an exceedingly niche use case
> - I don't see any way for raw_argv to be useful in a cross-implementation manner.

I expect it to be useful in the "launch (almost) the same command in the
same way" situation.
Not that it happens often :-)

What about the "launch a different command with the same interpreter
flags" use-case? For example, having 'python -E foo.py --foooptions'
want to execute 'python -E bar.py'.

Perhaps it'll be cleaner to expose in state_argv ['python', '-E']; in
sys.argv ['foo.py', '--foooptions']; and have scripts that want to exec
themselves use an idiomatic os.execv(sys.executable, sys.state_argv + sys.argv).

[tjguk]

My use case is the reloader or restarter. I've initially fallen foul of this when using the cherrypy reloader (which does an execv by building from sys.executable + sys.argv) but I also have web services running which I'd like to restart remotely by forcing them to execv themselves.

It's trivial to retrieve the original command line from the Windows API, shlex.split it and pass it along to execv so this is hardly a showstopper. But it would be useful to have something equivalent built-in.

[ncoghlan]

A few updates here:

• For the specific case of python -m, the original argument has been available as __main__.__spec__.name since Python 3.4

• Also since Python 3.4, the multiprocessing module has correctly handled the -m switch. For more details, see https://docs.python.org/3/whatsnew/3.4.html#multiprocessing and the linked issues.

• However, there are still some cases where it is potentially useful to have access to the full details of how the host Python runtime was invoked, rather than just the arguments that were left after CPython's runtime processing was completed. I filed bpo-29857 as a new RFE specifically suggesting a sys._raw_argv attribute addressing that question.