Pickling of NoneType raises PicklingError

[tifv]

BPO	6477
Nosy	@birkenfeld, @jaraco, @ncoghlan, @abalkin, @pitrou, @avassalotti, @tifv
Files	backport_pickle_singleton_types_fix.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/avassalotti'
closed_at = <Date 2013-12-01.19:44:48.660>
created_at = <Date 2009-07-13.17:20:39.843>
labels = ['easy', 'type-bug', 'docs']
title = 'Pickling of NoneType raises PicklingError'
updated_at = <Date 2013-12-01.19:44:48.659>
user = 'https://github.com/tifv'

bugs.python.org fields:

activity = <Date 2013-12-01.19:44:48.659>
actor = 'alexandre.vassalotti'
assignee = 'alexandre.vassalotti'
closed = True
closed_date = <Date 2013-12-01.19:44:48.660>
closer = 'alexandre.vassalotti'
components = ['Documentation']
creation = <Date 2009-07-13.17:20:39.843>
creator = 'july'
dependencies = []
files = ['32915']
hgrepos = []
issue_num = 6477
keywords = ['patch', 'easy']
message_count = 17.0
messages = ['90496', '90538', '90540', '90555', '90641', '90648', '108957', '141106', '204862', '204866', '204867', '204869', '204870', '204871', '204874', '204877', '204951']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'jaraco', 'ncoghlan', 'belopolsky', 'pitrou', 'alexandre.vassalotti', 'hagen', 'july', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6477'
versions = ['Python 2.7']

[tifv]

Python 3.2a0 (py3k:73749M, Jul  1 2009, 23:17:59)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
[40072 refs]
>>> pickle.dumps(type(None))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'NoneType'>: attribute 
lookup builtins.NoneType failed
[40137 refs]
>>>

[avassalotti]

I don't see why you want to pickle NoneType. Do you have a proper
use-case for this, or are you just playing around with pickle?

[tifv]

No, my program failed on this. It was not a big problem to manage this,
but I think it is a bug. And it isn't documented (or I can't find it).

Other built-in types have no such problem. Is there something special
with NoneType?

[hagen]

but I think it is a bug

I think it is either a feature request (make NoneType picklable) or a
documentation issue (document that it's not).

[ncoghlan]

Given that dumps(type(Ellipsis)) and dumps(type(NotImplemented)) don't
work either, I am reclassifying this as a documentation bug.

The thing about the types of these three objects (None, Ellipsis,
NotImplemented) is that they are all designed to be singletons and they
all disallow creation of new instances of them.

>>> type(None)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NoneType' instances
>>> type(NotImplemented)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NotImplementedType' instances
>>> type(Ellipsis)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'ellipsis' instances

This does mean None has to be special cased if pickling "(object,
type(object))" pairs, but passing types through pickle strikes me as a
dubious enough exercise that it would take a fairly convincing use case
to accept an RFE to make NoneType pickleable (note that any such patch
would have to make sure that doing loads() on such a pickle didn't
manage to create a second copy of NoneType or None).

[avassalotti]

I agree with Nick.

And if you really want to, you could hack a Pickler subclass to support
NoneType:

import io
import pickle

class XPickler(pickle.Pickler):
  def persistent_id(self, obj):
    if obj is type(None):
      return "NoneType"
    return None

class XUnpickler(pickle.Unpickler):
  def persistent_load(self, persistent_id):
    if persistent_id == "NoneType":
      return type(None)

def dumps(obj):
    f = io.BytesIO()
    XPickler(f).dump(obj)
    return f.getvalue()

def loads(s):
    return XUnpickler(io.BytesIO(s)).load()

Not super elegant, but it works:

>>> loads(dumps([type(None), None]))
[<type 'NoneType'>, None]

[abalkin]

Ellipsis and NotImplemented are different from None because they are not pickleable themselves. The None situation is more similar to that of say module level functions:

>>> def f(): pass
... 
>>> dumps(f)  # works
b'\x80\x03c__main__\nf\nq\x00.'
>>> dumps(type(f))  # does not work
Traceback (most recent call last):
  ..
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed

I think documentation at http://docs.python.org/dev/py3k/library/pickle.html#what-can-be-pickled-and-unpickled

Can be improved in two ways:

1. s/types/objects/ in "The following types can be pickled"

2. Add a note that "type(x) can be pickled" does not follow from "x can be pickled." The object type(x) must itself fall into one on the listed categories.

3. "built-in functions defined at the top level of a module" should probably be "built-in classes or functions defined at the top level of the builtins module"

4. "instances of such classes" should be "instances of picklable classes"

5. "__setstate__() is picklable" should be "output of __getstate__() is picklable"

[jaraco]

I've encountered a use-case where the need to pickle NoneType is more relevant (and difficult to work around).

We have a strongly-type Limited Python language, LimPy, which is based on Python (https://bitbucket.org/yougov/limpy). This parser takes, as part of its initialization arguments, a type specification (indicating which types are allowed and not allowed). In some cases, the return value may be None, in which case the specification says the type must be NoneType.

We're attempting to run this parser in a separate process, using the multiprocessing module, which requires that the arguments passed to and from the parser be pickleable. Unfortunately, because NoneType is in the type specification, it cannot be passed to the parser.

Here's an example of one such type specification (from the test suite):

    class SomeFunctionNamespace:
        @signature([IListType], [], None, ListOfInt)
        def listcount(self, l):
            return range(len(l))

    @signature([IListType], [], None, int)
    def listlen(self, l):
        return len(l)

    # ...

    @signature([IIntType], [IStringType], IStringType, NoneType)
    def givespec(self, *args):
        for a in args:
            print a

Note that we can pickle str and int just fine. Only type(None) fails.

It would be possible to re-write the entire LimPy system (and its child projects) to use a different object where currently NoneType is used, though NoneType is precisely the right thing to be used here except that it can't be pickled.

Since type(None) is a fundamental Python type, it strikes me as a bug that it's not pickleable, though I concede that it's also reasonable to interpret this issue as a feature request (as it's never been pickleable).

Nick makes some good comments that pickling of NoneType should be done right, but other than that, there haven't been any reasons why in principle NoneType should not be pickleable.

NoneType is more akin to str and int which are pickleable than it is to a function or type(NotImplemented), and this is evident by the way that LimPy uses it (before there was any consideration for pickleability).

Therefore, I propose we reconsider that NoneType should be made pickleable.