Add *Calloc functions to CPython memory allocation API

[njsmith]

BPO	21233
Nosy	@pitrou, @vstinner, @njsmith, @skrah, @MojoVampire
Files	calloc.patch calloc-2.patch calloc-3.patch bench_alloc.py test.c calloc-4.patch use_calloc.patch bench_alloc2.py calloc-5.patch calloc-6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-02.20:29:45.268>
created_at = <Date 2014-04-15.08:56:01.630>
labels = ['interpreter-core', 'type-feature']
title = 'Add *Calloc functions to CPython memory allocation API'
updated_at = <Date 2014-06-02.20:29:45.267>
user = 'https://github.com/njsmith'

bugs.python.org fields:

activity = <Date 2014-06-02.20:29:45.267>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-02.20:29:45.268>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2014-04-15.08:56:01.630>
creator = 'njs'
dependencies = []
files = ['34897', '34903', '34924', '35052', '35059', '35063', '35064', '35065', '35072', '35097']
hgrepos = []
issue_num = 21233
keywords = ['patch']
message_count = 95.0
messages = ['216281', '216390', '216394', '216399', '216403', '216404', '216422', '216425', '216431', '216433', '216444', '216451', '216452', '216455', '216515', '216567', '216668', '216671', '216681', '216682', '216686', '217228', '217242', '217246', '217251', '217252', '217253', '217254', '217255', '217256', '217257', '217262', '217274', '217276', '217282', '217283', '217284', '217291', '217294', '217295', '217297', '217298', '217302', '217303', '217304', '217305', '217306', '217307', '217308', '217309', '217310', '217323', '217324', '217325', '217326', '217330', '217331', '217333', '217346', '217348', '217349', '217351', '217357', '217375', '217380', '217382', '217423', '217436', '217445', '217549', '217553', '217594', '217617', '217619', '217785', '217786', '217794', '217797', '217826', '217829', '217831', '217832', '217838', '217839', '217840', '217841', '217844', '217866', '217972', '219627', '219628', '219630', '219631', '219634', '219635']
nosy_count = 8.0
nosy_names = ['pitrou', 'vstinner', 'njs', 'skrah', 'neologix', 'python-dev', 'jtaylor', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue21233'
versions = ['Python 3.5']

[njsmith]

Numpy would like to switch to using the CPython allocator interface in order to take advantage of the new tracemalloc infrastructure in 3.4. But, numpy relies on the availability of calloc(), and the CPython allocator API does not expose calloc().
https://docs.python.org/3.5/c-api/memory.html#c.PyMemAllocator

So, we should add *Calloc variants. This met general approval on python-dev. Thread here:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

This would involve adding a new .calloc field to the PyMemAllocator struct, exposed through new API functions PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc. [It's not clear that all 3 would really be used, but since we have only one PyMemAllocator struct that they all share, it'd be hard to add support to only one or two of these domains and not the rest. And the higher-level calloc variants might well be used. Numpy array buffers are often small (e.g., holding only a single value), and these small buffers benefit from small-alloc optimizations; meanwhile, large buffers benefit from calloc optimizations. So it might be optimal to use a single allocator that has both.]

We might also have to rename the PyMemAllocator struct to ensure that compiling old code with the new headers doesn't silently leave garbage in the .calloc field and lead to crashes.

[vstinner]

Here is a first patch adding the following functions:

  void* PyMem_RawCalloc(size_t n);
  void* PyMem_Calloc(size_t n);
  void* PyObject_Calloc(size_t n);
  PyObject* _PyObject_GC_Calloc(size_t);

It adds the following field after malloc field to PyMemAllocator structure:

  void* (*calloc) (void *ctx, size_t size);

It changes the tracemalloc module to trace "calloc" allocations, add new tests and document new functions.

The patch also contains an important change: PyType_GenericAlloc() uses calloc instead of malloc+memset(0). It may be faster, I didn't check.

[pitrou]

So what is the point of _PyObject_GC_Calloc ?

[MojoVampire]

General comment on patch: For the flag value that toggles zero-ing, perhaps use a different name, e.g. setzero, clearmem, initzero or somesuch instead of calloc? calloc already gets used to refer to both the C standard function and the function pointer structure member; it's mildly confusing to have it *also* refer to a boolean flag as well.

[MojoVampire]

Additional comment on clarity: Might it make sense to make the calloc structure member take both the num and size arguments that the underlying calloc takes? That is, instead of:

void* (*calloc) (void *ctx, size_t size);

Declare it as:

void* (*calloc) (void *ctx, size_t num, size_t size);

Beyond potentially allowing more detailed tracing info at some later point (and much like the original calloc, potentially allowing us to verify that the components do not overflow on multiply, instead of assuming every caller must multiply and check for themselves), it also seems like it's a bit more friendly to have the prototype for the structure calloc to follow the same pattern as the other members for consistency (Principle of Least Surprise): A context pointer, plus the arguments expected by the equivalent C function.

[MojoVampire]

Sorry for breaking it up, but the same comment on consistent prototypes mirroring the C standard lib calloc would apply to all the API functions as well, e.g. PyMem_RawCalloc, PyMem_Calloc, PyObject_Calloc and _PyObject_GC_Calloc, not just the structure function pointer.

[vstinner]

So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

_PyObject_GC_Calloc() is used by PyType_GenericAlloc(). If I understand directly, it is the default allocator to allocate Python objects.

[vstinner]

In numpy, I found the two following functions:

/*NUMPY_API
 * Allocates memory for array data.
 */
void* PyDataMem_NEW(size_t size);

/*NUMPY_API
 * Allocates zeroed memory for array data.
 */
void* PyDataMem_NEW_ZEROED(size_t size, size_t elsize);

So it looks like it needs two size_t parameters. Prototype of the C function calloc():

void *calloc(size_t nmemb, size_t size);

I agree that it's better to provide the same prototype than calloc().

[vstinner]

New patch:

• replace "size_t size" with "size_t nelem, size_t elsize" in the prototype of calloc functions (the parameter names come from the POSIX standard)
• replace "int calloc" with "int zero" in helper functions

[pitrou]

Le 16/04/2014 04:40, STINNER Victor a écrit :

STINNER Victor added the comment:

> So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size)

No, the question is why you didn't simply change _PyObject_GC_Malloc
(which is a private function).

[neologix]

> So what is the point of _PyObject_GC_Calloc ?

It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
https://mail.python.org/pipermail/python-dev/2014-April/133985.html

It will only make a difference if the allocated region is large enough
to be allocated by mmap (so not for 90% of objects).

[vstinner]

>> So what is the point of _PyObject_GC_Calloc ?
>
> It calls calloc(size) instead of malloc(size)

No, the question is why you didn't simply change _PyObject_GC_Malloc
(which is a private function).

Oh ok, I didn't understand. I don't like changing the behaviour of
functions, but it's maybe fine if the function is private.

[vstinner]

2014-04-16 3:18 GMT-04:00 Charles-François Natali <report@bugs.python.org>:

> It calls calloc(size) instead of malloc(size), calloc() which can be faster than malloc()+memset(), see:
> https://mail.python.org/pipermail/python-dev/2014-April/133985.html

It will only make a difference if the allocated region is large enough
to be allocated by mmap (so not for 90% of objects).

Even if there are only 10% of cases where it may be faster, I think
that it's interesting to use calloc() to allocate Python objects. You
may create large Python objects ;-)

I didn't check which objects use (indirectly) _PyObject_GC_Calloc().