Flatten nested functools.partial

[AlexanderBelopolsky]

BPO	7830
Nosy	@rhettinger, @abalkin, @pitrou, @vstinner, @jackdied, @MojoVampire
Files	no-nested-partial.diff no-nested-partial-exact.diff issue7830.diff issue7830-2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2015-03-01.20:09:11.372>
created_at = <Date 2010-02-01.19:07:29.413>
labels = ['extension-modules', 'library', 'performance']
title = 'Flatten nested functools.partial'
updated_at = <Date 2015-03-01.20:55:12.337>
user = 'https://bugs.python.org/AlexanderBelopolsky'

bugs.python.org fields:

activity = <Date 2015-03-01.20:55:12.337>
actor = 'vstinner'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2015-03-01.20:09:11.372>
closer = 'belopolsky'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2010-02-01.19:07:29.413>
creator = 'Alexander.Belopolsky'
dependencies = []
files = ['16083', '16091', '17813', '36837']
hgrepos = []
issue_num = 7830
keywords = ['patch']
message_count = 18.0
messages = ['98674', '98698', '98706', '108980', '111287', '111289', '111319', '111401', '111508', '122932', '122934', '228735', '228737', '228743', '228747', '228796', '236975', '236979']
nosy_count = 9.0
nosy_names = ['rhettinger', 'belopolsky', 'pitrou', 'vstinner', 'jackdied', 'vdupras', 'Christophe Simonis', 'python-dev', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue7830'
versions = ['Python 3.5']

[AlexanderBelopolsky]

Currently applying functools.partial to a callable that is already functools.partial object results in a nested object:

>>> from functools import partial
>>> def f(a,b,c): pass
... 
>>> p = partial(partial(f, 1), 2)
>>> p.func, p.args
(<functools.partial object at 0x100431d60>, (2,))


Proposed patch makes partial(partial(f, 1), 2) return partial(f, 1, 2) instead:
>>> p.func, p.args
(<function f at 0x10055d3a8>, (1, 2))

This patch is partially (no pun intended) motivated by a patch submitted by Christophe Simonis for bpo-4331. Christophe's patch flattens nested partials for a specific case of using partials as bound methods.

As proposed, the patch will enable flattening for subclasses of functools.partial, but will return a baseclass instance. Flattening will also discard any state attached to the nested partial such as __name__, __doc__, etc or any subclass data. I believe this is the right behavior, but this caveat is the reason I classify this patch as a "feature request" rather than "performance" or "resource usage".

[pitrou]

Flattening should only happen for instances of the exact type. When people create subclasses, there's usually a reason for it.

[AlexanderBelopolsky]

Antoine> Flattening should only happen for instances of the exact type.

I am attaching a variant of the patch that will only flatten if both nested and wrapping partial is of the exact type. Other possibilities would include flattening partial_subtype(f, ...) if type(f) == partial and if type(f) == partial_subtype, but I'll present only the least and most conservative variants.

Antoine> When people create subclasses [of partial type], there's usually a reason for it.

It is hard not to agree with this thesis, but I don't see how it follows that subclass writers will not benefit from flattening their instances. Can you suggest a use case where flattening in functools.partial subtype would be undesirable?

[abalkin]

I am attaching a patch, bpo-7830.diff, that takes an ultra-concervative approach: partials are only flattened if both outer and inner are of exact functools.partial type and the inner partial does not have __dict__.

[vdupras]

Applies cleanly on the py3k branch at r83069, the tests work correctly (fail before applying the patch, success afterwards), and, to the best of my C-API knowledge, the C code is alright.

Oh, and it behaves as described...

Python 3.2a0 (py3k:83069M, Jul 23 2010, 12:40:49) 
[GCC 4.2.1 (Apple Inc. build 5659)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def foo(a, b, c): pass
... 
>>> from functools import partial
>>> p1 = partial(foo, 1)
>>> p1.func, p1.args
(<function foo at 0x1004531e8>, (1,))
>>> p2 = partial(foo, 2)
>>> p2.func, p2.args
(<function foo at 0x1004531e8>, (2,))

[vdupras]

Oops, used it wrong (but it still works correctly).

>>> p2 = partial(p1, 2)
>>> p2.func, p2.args
(<function foo at 0x10051da68>, (1, 2))

[abalkin]

Since I am the OP of this patch, I would like a +1 from another developer before checking this in. (Or a couple of -1s before rejecting:-)

Antoine,

Does the latest patch address your concerns?

Virgil,

If you care enough about this feature, please post a summary on python-dev and ask for comments.

[rhettinger]

Antoine> Flattening should only happen for instances of the exact type.

FWIW, I agree with Antoine. You cannot know in advance whether a partial-subclass has semantics that need to be preserved when flattening.

[abalkin]

FWIW, I agree with Antoine. You cannot know in advance whether a
partial-subclass has semantics that need to be preserved when
flattening.

Raymond,

I have actually conceded this point to Antoine. See msg108980 above. Not only the latest patch preserves partial-subclasses, it also foregoes the optimization if there is anything in __dict__ of the inner partial.

It looks like we have a consensus on the features, the remaining question is whether this is enough of an optimization to justify adding extra code.

[rhettinger]

Alexander, I don't see anything wrong with patch, nor anything compelling about it either. It's your choice whether or not to apply.

[abalkin]

The original motivation for the patch was that if partial() objects are guaranteed to be flat, it would simplify code that deals with them. See bpo-4331 for one example.

With a "conservative" patch, however, it will still be possible to create nested partials and the code consuming partials should be ready for that.