argparse set_defaults on subcommands should override top level set_defaults

[bethard]

BPO	9351
Nosy	@ezio-melotti, @bitdancer, @akheron, @remram44, @shihai1991
Files	issue9351.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2010-07-23.14:06:50.789>
labels = ['type-bug', 'library']
title = 'argparse set_defaults on subcommands should override top level set_defaults'
updated_at = <Date 2021-10-27.21:44:47.104>
user = 'https://bugs.python.org/bethard'

bugs.python.org fields:

activity = <Date 2021-10-27.21:44:47.104>
actor = 'paul.j3'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2010-07-23.14:06:50.789>
creator = 'bethard'
dependencies = []
files = ['27795']
hgrepos = []
issue_num = 9351
keywords = ['patch']
message_count = 21.0
messages = ['111324', '173602', '174147', '174206', '174208', '206993', '219332', '229619', '229620', '229968', '229983', '230056', '230084', '233041', '244768', '244786', '274369', '342154', '350638', '373665', '405130']
nosy_count = 12.0
nosy_names = ['bethard', 'ezio.melotti', 'r.david.murray', 'nailor', 'python-dev', 'petri.lehtinen', 'paul.j3', 'mikn', 'remram', 'Changaco', 'smparkes', 'shihai1991']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9351'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[bethard]

If you use set_defaults on a subparser, but a default exists on the top level parser, the subparser defaults are ignored:

>>> parser = argparse.ArgumentParser()
>>> xparser = parser.add_subparsers().add_parser('X')
>>> parser.set_defaults(foo=1)
>>> xparser.set_defaults(foo=2)
>>> parser.parse_args(['X'])
Namespace(foo=1)

This is counter to what people probably expect, that the subparser, when selected, would override the top level parser.

The behavior is because of the following code in parse_known_args:

        for dest in self._defaults:
            if not hasattr(namespace, dest):
                setattr(namespace, dest, self._defaults[dest])

This happens before the subparser sees the namespace object, and so the subparser sees that no defaults need to be filled in.

[nailor]

I've attached a proposed fix where the subparser parses to an empty namespace and that namespace is then merged with the parent namespace

[akheron]

+        for key in vars(subnamespace):
+            setattr(namespace, key, getattr(subnamespace, key))

There might be even more clever ways to achieve this, but what about at least saying "for key, value in vars(subnamespace).items()", and then using the value accordingly, to avoid the getattr() call?

[nailor]

Yeah, I tried figuring out something more clever, as this, in the current form, has a bit too hackish feeling in it, but I couldn't find a proper tool for the job.

Anyway, attached a patch with the getattr removed.

[akheron]

LGTM. Steven?

[mikn]

Just wanted to drop in here to let you know that I hit this behaviour recently when writing a small tool using both configparser and argparse and the workaround proves rather annoying (custom namespace object or similar).

Would be awesome if this moved forward with the proposed patch!

[paulj3]

This is not getting much attention for several reasons:

• there's quite a backlog of argparse patches and issues

• 'set_defaults' is not commonly used. Setting default in 'add_argument' seems more common.

• defining the same argument for both the parser and subparser can create other difficulties.

• in Bethard's example, 'set_default' sets an attribute that has no connection to any argument. It's a cleaver trick that few users will think to use, and probably won't be of much value.

The idea proposed here of using a subnamespace for the subparser is interesting. It reminds me of a StackOverflow question about implementing nested namespaces.

http://stackoverflow.com/questions/18668227

[python-dev]

New changeset e9cb45ccf42b by R David Murray in branch '3.4':
bpo-9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/e9cb45ccf42b

New changeset b35a811d4420 by R David Murray in branch 'default':
Merge: bpo-9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/b35a811d4420

New changeset 1a3143752db2 by R David Murray in branch '2.7':
bpo-9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/1a3143752db2

[bitdancer]

Thanks, Jyrki.

[paulj3]

A possible problem with this patch is that it overrides any Namespace given by the user. In the test example:

    parser.parse_args(['X'], namespace=argparse.Namespace(foo=3))

the result is 'foo=2', the setdefault from the subparser. Previously, and with a single level parser, the result would be 'foo=3'. This is also true for an ordinary argument default in the subparser.

It could be argued that a user is unlikely to use both the 'namespace=...' parameter, and 'setdefault' for the same variable, especially one that isn't an argument 'dest'. But the fact that the custom namespace does not override regular subparser argument defaults bothers me.

Also, should an untested change like this be applied to 3.4 and 2.7? This is not a backward compatible bug fix.

The test case also touches on a real bug in the recent releases - subparsers are now optional. 'parser.parse_args([])' runs without complaint about the missing 'X'. Actually it is convenient for testing this case. But it is still an incompatibility with earlier behavior

http://bugs.python.org/issue9253

[bitdancer]

Yeah, I hesitated a bit about the backports, but didn't visualize the scenario you are suggesting and thought it would be safe. Perhaps it should be backed out of 2.7 and 3.4.

For 3.5, do you have any thoughts about how to make namespace= play nicely with this problem?