bpo-16865: Support arrays >=2GB in ctypes

[segevfiner]

https://bugs.python.org/issue16865

[orenmn]

using PyLong_AsSsize_t() (instead of PyLong_AsLongAndOverflow()) makes it harder to figure out whether _length_ is too big or negative.
Also, ignoring the wrong error messages for negative _length_ (which i hope to fix in bpo-29843), this patch would cause an inconsistency in error messages. for example, on 32-bit Python on Windows, you get:

class MyArray(Array):
    _type_ = c_longlong
    _length_ = (1 << 31) - 1
OverflowError: array too large

class MyArray(Array):
    _type_ = c_longlong
    _length_ = 1 << 31
OverflowError: Python int too large to convert to C ssize_t

Maybe using PyLong_AsLongLongAndOverflow() is an option?

[segevfiner]

@orenmn Changed it.

[orenmn]

Unless i am missing something, this #if is potentially problematic. In case sizeof(size_t) < sizeof(long), there could be a silent truncation here.
Maybe change to #if SIZEOF_SIZE_T == SIZEOF_LONG, and then later #elif SIZEOF_SIZE_T == SIZEOF_LONG_LONG?

[segevfiner]

This will of course make the code fail to build in cases where sizeof(size_t) != sizeof(long) && sizeof(size_t) != sizeof(long long). Is that desirable @serhiy-storchaka ? It used to previously truncate if sizeof(size_t) < sizeof(long). (Does Python even support such systems...?)

[orenmn]

You could raise an OverflowError in cases where sizeof(size_t) != sizeof(long) && sizeof(size_t) != sizeof(long long) and _length_ is too large.
IIUC, without your patch, there is no silent truncation, but an OverflowError.
(Of course, this is only my opinion, so take it with a grain of salt :) )

[segevfiner]

It was previously just: Modules/_ctypes/_ctypes.c:1416, so it used to silently truncate if sizeof(size_t) < sizeof(long). When sizeof(size_t) > sizeof(long), it raised OverflowError when _length_ was too big. With the patch, the behavior when sizeof(size_t) < sizeof(long) is preserved, while when it's bigger, it will use PyLong_AsLongLongAndOverflow which allows to pass larger lengths.

I'm reluctant to add code to handle sizeof(size_t) < sizeof(long). Especially if CPython itself doesn't support such a platform, since it will never be tested. Even more so since it requires to write a custom variation of PyLong_As{Type}AndOverflow to handle the smaller types, and which types should that be...

[orenmn]

Ah, my bad with regard to the silent truncation: it seems that you are right, as later on we have stgdict->length = length;, and stgdict->length is Py_ssize_t.

[serhiy-storchaka]

In this case I think that it is better to restore the initial variant of this PR, with using PyLong_AsSsize_t().

[segevfiner]

@serhiy-storchaka @orenmn Maybe if I use PyLong_AsSsize_t and do something like:

if (PyErr_ExceptionMatches(PyExc_OverflowError))
    PyErr_SetString(PyExc_OverflowError,
                    "The '_length_' attribute is too large");

[serhiy-storchaka]

LGTM.

[segevfiner]

BTW, it seems we will only reach the code here when declaring an array via inheritance, it will fail sooner in Objects/abstract.c:953-967 due to the call to PyNumber_AsSsize_t, when declaring an array via the * operator (Obviously with a different error). This happens because the * operator in ctypes is implemented via PySequenceMethods.sq_repeat.

[miss-islington]

Thanks @segevfiner for the PR, and @serhiy-storchaka for merging it 🌮🎉.. I'm working now to backport this PR to: 2.7, 3.6, 3.7.
🐍🍒⛏🤖

[bedevere-bot]

GH-6842 is a backport of this pull request to the 3.7 branch.

[miss-islington]

Sorry, @segevfiner and @serhiy-storchaka, I could not cleanly backport this to 2.7 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 735abadd5bd91db4a9e6f4311969b0afacca0a1a 2.7

[bedevere-bot]

GH-6843 is a backport of this pull request to the 3.6 branch.

[bedevere-bot]

GH-7441 is a backport of this pull request to the 2.7 branch.