Make Generators Pickle-able

[jaysonvantuyl]

BPO	1092962
Nosy	@loewis, @birkenfeld, @josiahcarlson, @tiran, @avassalotti

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-07-20.11:25:10.336>
created_at = <Date 2004-12-29.21:52:55.000>
labels = ['type-feature', 'library']
title = 'Make Generators Pickle-able'
updated_at = <Date 2008-07-20.11:25:10.335>
user = 'https://bugs.python.org/jaysonvantuyl'

bugs.python.org fields:

activity = <Date 2008-07-20.11:25:10.335>
actor = 'georg.brandl'
assignee = 'none'
closed = True
closed_date = <Date 2008-07-20.11:25:10.336>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2004-12-29.21:52:55.000>
creator = 'jaysonvantuyl'
dependencies = []
files = []
hgrepos = []
issue_num = 1092962
keywords = []
message_count = 7.0
messages = ['54345', '54346', '54347', '54348', '59303', '67893', '70080']
nosy_count = 6.0
nosy_names = ['loewis', 'georg.brandl', 'josiahcarlson', 'christian.heimes', 'alexandre.vassalotti', 'jaysonvantuyl']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1092962'
versions = ['Python 2.6']

[jaysonvantuyl]

Would it be possible to make generators pickle-able? I
mean, currently the internal state is saved in some
way. Would it be possible to make pickle handle them?

Put another way, if generators had a __getnewargs__
function that returned some data (say a tuple of module
name, function name, locals/globals dicts and some code
dependent location data) and then allow:

  generator.__new__(statedata)

to reconstruct it (or something more elegant).

[loewis]

Logged In: YES
user_id=21627

Moving into the feature requests tracker.

I don't think this is possible (or should be done if it were
possible). Pickle has traditionally abstained from pickling
functions, so IMO it should not picke generators, either. If
this was enabled by default, it would open up yet another
security hole.

[jaysonvantuyl]

Logged In: YES
user_id=626872

I'm not talking about pickling functions. Which, by the
way, is not entirely accurate. See the following:

>>> from pickle import dumps,loads
>>> loads(dumps(dumps))
<function dumps at 0x40376dbc>
>>>

As you see, we don't pickle the function code, just a
reference to it. I propose doing the same with generators,
but including the function namespace and frame info in the
pickle to allow it to continue execution after unpickling.
Furthermore, this is nothing more than what the interpreter
already does internally. When a generator yields, all of
its state is neatly stashed away. I just would like pickle
to be able to get at it, store it, and then later recover
it--without dealing with any actual code objects, just the
state.

As for security, what I am talking about is nothing more or
less secure than pickling classes.

When we pickle a class, we don't pickle the methods on it.
Rather, we pickle the information to reconstruct the class
(getstate, __getnewargs__, __reduce__). There is a
security concern in that modified pickles could be used to
put bogus data into the unpickled classes (i.e. a password
stored as an attribute on a class could be replaced).

What I'm asking for is nothing more than pickling a form of
the frozen frame object (or something akin to it) for the
generator. Put another way, when the generator isn't
running, something stores the entire state of its execution.
I'm not sure what it is, but I'd be willing to be it
consists of little more than a few dicts (namespaces), some
scoping info, and some sort of instruction pointer.

By pickling the generator, I propose nothing more radical
than pickling a class. The generator is still instantiated
from the some code as before (just as a class or function
reference is) and it still can/will act on that information
(just as a class does). No actual code is pickled. What
this does allow is the use of generators for efficent
handling of a class.

Specifically, I'm writing an application that uses a seeded
pseudo-random number generator. The idea is to transmit the
state of the generator over the network so that the
client/server can deterministically make the same random
choices without communicating the complex state that results
from that.

I have a choice of the following:

def randfunc(seed,num,otherstate):
  # Costly Setup
  # Iteration to appropriate number

otherside.sendPickle( (seed,num,otherstate,) )

l = [ randfunc(123,i,...) for i in range(5) ]

Versus

def randGen(seed):
  # Costly Setup
  # yield in a simple loop

r = randGen(123)

otherside.sendPickle(r)

l = [ r() for i in range(5) ]

I think you can see which one is more efficient in terms of
both simplicity of expression and ease of coding.

Of course, the standard answer to this is to implement a
randgen class that contains all of the state.

This complicates the code a great deal, since a generator
can't be used.

What is problematic is that generators are like black holes.
Once information goes into them, it won't come back out.
There's not a good way to use them for anything more than
runtime. This rules out things like process migration,
pickling of any structure involving generator data, and
generally using them like any other language component.
Using them for counters, prngs, prime number generators, OGR
sieves, and anything that needs to be used to durably
generate a sequence is impossible if that data needs to be
persisted.

No one would think about implementing a new pickle without
the ability to represent classes and function references,
generators shouldn't be second class control structures.

FYI, Stackless Python already does this, but it may be
easier for them due to the way they've modified frame handling.

[josiahcarlson]

Logged In: YES
user_id=341410

In a practical sense I believe this kind of thing is
possible (doing a little spelunking in some generators
reveals gen.gi_frame, which looks to be the stackframe for
the generator). Why it isn't done could have something to
do with executing on arbitrary stack frames.

In your random-number-generator case, using old-style
iterators via classes can do what you want them to do now
(that is, won't have to wait until Python 2.5 to make it in,
which is at least 1.5 years away).

class g:
    def __init__(self, state=None):
        if state is None:
            #generate some default state
        else:
            #process the state to validate and internalize it.
    def __iter__(self):
        return self
    def next(self):
        state = self.state
        ret = #some function on the current internal state
        self.state = #some function on the current internal
state
        return ret

Heck, you don't even need pickle hooks, because pickle will
pickle your iterator's __dict__ attribute.

Alternatively, you can use Christian Tismer's Stackless
Python, which has generator pickling.

[tiran]

Discuss this for 2.6

I thin it's neither possible to find a generic solution nor a good idea
to make generators picklable.

[avassalotti]

I think is a bad idea too. Unless I see patch that implements this
feature cleanly, I will have to reject this feature request.

[birkenfeld]

A patch can open a new issue, then.