pickle not 64-bit ready

[nyevik]

BPO	11564
Nosy	@jcea, @amauryfa, @abalkin, @pitrou, @avassalotti
Files	pickle64.patch pickle64-3.3.patch pickle64-4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-08-29.21:43:22.661>
created_at = <Date 2011-03-15.22:58:27.914>
labels = ['interpreter-core', 'type-crash']
title = 'pickle not 64-bit ready'
updated_at = <Date 2011-12-11.01:22:28.894>
user = 'https://bugs.python.org/nyevik'

bugs.python.org fields:

activity = <Date 2011-12-11.01:22:28.894>
actor = 'jcea'
assignee = 'none'
closed = True
closed_date = <Date 2011-08-29.21:43:22.661>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2011-03-15.22:58:27.914>
creator = 'nyevik'
dependencies = []
files = ['22891', '22914', '23052']
hgrepos = []
issue_num = 11564
keywords = ['patch']
message_count = 28.0
messages = ['131060', '131062', '131064', '131118', '131192', '131193', '131196', '131197', '141284', '141981', '142216', '142415', '142426', '142434', '142436', '142476', '142477', '142481', '142756', '142759', '142760', '142854', '142855', '142857', '143064', '143166', '143180', '143183']
nosy_count = 10.0
nosy_names = ['jcea', 'amaury.forgeotdarc', 'belopolsky', 'pitrou', 'alexandre.vassalotti', 'nadeem.vawda', 'santoso.wijaya', 'python-dev', 'nyevik', 'jorgsk']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue11564'
versions = ['Python 3.2', 'Python 3.3']

[nyevik]

Python 3.2 on linux (RHEL 5.3) x86_64 build from source code.
Configure options:
./configure --prefix=/scratch/Python-3.2 --enable-big-digits=30 --with-universal-archs=all --with-fpectl --enable-shared
Built with GCC 4.3.3 with major options
-g3 -O3 -m64 -fPIC.

Testcase that shows the issue:

#import numpy

import pickle
print("begin")
#a = numpy.zeros((2.5e9 / 8,), dtype = numpy.float64)
a = ('a' * (2 ** 31))
print("allocated")
#print(a);
pickle.dumps(a, pickle.DEFAULT_PROTOCOL)
print("end")

The problem as I can see it is that in pickle.py most types defined either as 2 bytes, or 4 bytes.
For example it is peppered with lines like:
self.write(SOMETYPE + pack("<i", n) + obj)
while pickling,
when unpickling:
len = mloads('i' + self.read(4))

Which limits the range and the size of the datatype that can be pickled, if I understand correctly.

replacing in pickle.py above lines with something like
self.write(SOMETYPE + pack("<Q", n) + obj)
and
len = mloads('Q' + self.read(8))

lets above testcase run to completion.
Othervise it crashes (on Python 2.7.1 with SIGSEGV) on Python 3.2 strace shows:

.......

 open("/scratch/Python-3.2/lib/python3.2/lib-dynload/_pickle.cpython-32m.so", O_RDONLY) = 4
fstat(4, {st_mode=S_IFREG|0755, st_size=412939, ...}) = 0
open("/scratch/hpl005/UIT_test/apps_exc/Python-3.2/lib/python3.2/lib-dynload/_pickle.cpython-32m.so", O_RDONLY) = 5
read(5, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300>\0\0\0\0\0\0"..., 832) = 832
fstat(5, {st_mode=S_IFREG|0755, st_size=412939, ...}) = 0
mmap(NULL, 2185384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x2b05b5f68000
mprotect(0x2b05b5f7b000, 2093056, PROT_NONE) = 0
mmap(0x2b05b617a000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x12000) = 0x2b05b617a000
close(5)                                = 0
close(4)                                = 0
close(3)                                = 0
write(1, "begin\n", 6begin
)                  = 6
mmap(NULL, 4294971392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b05b617e000
write(1, "allocated\n", 10allocated
)             = 10
mmap(NULL, 8589938688, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b06b617f000
mremap(0x2b06b617f000, 8589938688, 2147487744, MREMAP_MAYMOVE) = 0x2b06b617f000
mmap(NULL, 4294971392, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b0736180000
munmap(0x2b06b617f000, 2147487744)      = 0
munmap(0x2b0736180000, 4294971392)      = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35
write(2, "  File \"pickle_long.py\", line 9,"..., 45  File "pickle_long.py", line 9, in <module>
) = 45
open("pickle_long.py", O_RDONLY)        = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=251, ...}) = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7ffff9f7c9e0) = -1 ENOTTY (Inappropriate ioctl for device)
fstat(3, {st_mode=S_IFREG|0644, st_size=251, ...}) = 0
lseek(3, 0, SEEK_CUR)                   = 0
dup(3)                                  = 4
fcntl(4, F_GETFL)                       = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fstat(4, {st_mode=S_IFREG|0644, st_size=251, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b06b617f000
lseek(4, 0, SEEK_CUR)                   = 0
read(4, "#import numpy\n\nimport pickle\npri"..., 4096) = 251
close(4)                                = 0
munmap(0x2b06b617f000, 4096)            = 0
lseek(3, 0, SEEK_SET)                   = 0
lseek(3, 0, SEEK_CUR)                   = 0
read(3, "#import numpy\n\nimport pickle\npri"..., 4096) = 251
close(3)                                = 0
write(2, "    pickle.dumps(a, pickle.DEFAU"..., 45    pickle.dumps(a, pickle.DEFAULT_PROTOCOL)
) = 45
write(2, "SystemError: error return withou"..., 48SystemError: error return without exception set
) = 48
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x2b05b118e4c0}, {0x2b05b0e7a570, [], SA_RESTORER, 0x2b05b118e4c0}, 8) = 0
munmap(0x2b05b617e000, 4294971392)      = 0
exit_group(1)                           = ?

Why is this limitation?
Please advise.

[pitrou]

Indeed:

>>> s = b'a' * (2**31)
>>> d = pickle.dumps(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set

There are two aspects to this:

• (bugfix) raise a proper exception when an object too large for handling by pickle is given
• (feature) improve the pickle protocol to handle objects larger than (2**31-1) elements

The improvement to the pickle protocol should probably be considered along other improvements, because we don't want to create a new protocol too often.

See also bpo-9614.

[avassalotti]

We could resort to the text-based protocol which doesn't have these limitations with respect to object lengths (IIRC). Performance won't be amazing, but we won't have to modify the current pickle protocol.

[abalkin]

On Tue, Mar 15, 2011 at 7:05 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..

• (bugfix) raise a proper exception when an object too large for handling by pickle is given

What would be the "proper exception" here? With _pickle acceleration
disabled, I get a struct.error:

$ cat p.py
import sys
sys.modules['_pickle'] = None
import pickle
s = b'a' * (2**31)
d = pickle.dumps(s)

$ ./python.exe p.py
Traceback (most recent call last):
  ..
  File "Lib/pickle.py", line 496, in save_bytes
    self.write(BINBYTES + pack("<i", n) + bytes(obj))
struct.error: 'i' format requires -2147483648 <= number <= 2147483647

I would say "proper exception" would be ValueError, but that means
that we should change python implementation in an incompatible way.

[pitrou]

On Tue, Mar 15, 2011 at 7:05 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..
> - (bugfix) raise a proper exception when an object too large for handling by pickle is given

What would be the "proper exception" here?

OverflowError. This is the exception that gets raised when some
user-supplied value exceeds some internal limit.

[abalkin]

On Wed, Mar 16, 2011 at 6:59 PM, Antoine Pitrou <report@bugs.python.org> wrote:
..

>
> What would be the "proper exception" here?

OverflowError. This is the exception that gets raised when some
user-supplied value exceeds some internal limit.

I don't think so. OverflowError is a subclass of ArithmeticError and
is raised when result of an arithmetic operation cannot be represented
by the python type. For example,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: (34, 'Result too large')

I don't think failing pickle dump should raise an ArithmeticError.

[pitrou]

I don't think so. OverflowError is a subclass of ArithmeticError and
is raised when result of an arithmetic operation cannot be represented
by the python type. For example,

If you grep for OverflowError in the C code base, you'll see that it is
in practice used for the present kind of error. Examples:

• "signed short integer is less than minimum"
• "%c arg not in range(0x110000)"
• "size does not fit in an int"
• "module name is too long"
• "modification time overflows a 4 byte field"
• "range too large to represent as a range_iterator"
• "Python int too large to convert to C size_t"
  (at this point I am bored of pasting examples but you get the point)

[nyevik]

Thank you all for your responses. While getting the meaningful error message in this case is very important, the main thing for us is to somehow fix this problem to allow larger objects serialization which is not at all uncommon on a 64-bit machines with large amounts of memory.
This issue affects cPickle as well, I believe, as well cStringIO that uses pickle too, I believe.

So, what are your plans/thoughts - would there be any action on fixing this problem in the near future? I think I grasp the extent of changes that need to be made to Python code, but I think this issue will have to bee addressed soonoer or later anyhow.

[jorgsk]

I recently ran into this problem when I tried to multiprocess jobs with large objects (3-4 GB). I have plenty of memory for this, but multiprocessing hangs without error, presumably because pickle hangs without error when multiprocessing tries to pickle the object. I can't offer a solution, but I can verify that the limitation in pickle is affecting Python usage.

[pitrou]

This patch contains assorted improvements for 64-bit compatibility of the pickle module. The protocol still doesn't support >4GB bytes or str objects, but at least its behaviour shouldn't be misleading anymore.

[nadeemvawda]

pickle64.patch applies cleanly to 3.2, but not 3.3. I've attached an
adapted version that applies cleanly to 3.3.

[nadeemvawda]

I have tried running the tests on a machine with 12GB of RAM, but when I do so,
the new tests get skipped saying "not enough memory", even when I specify "-M 11G"
on the command-line. The problem seems to be the change to the precisionbigmemtest
decorator in test.support. I don't understand what the purpose of the "dryrun"
flag is, but the modified condition for skipping doesn't look right to me.

(Now that I think about it, I should be able to get the tests to run by undoing
that one part of the change. I'll get back to you about the results later today.)