tracemalloc: add C API to manually track/untrack memory allocations

[vstinner]

BPO	26530
Nosy	@pitrou, @vstinner, @njsmith
Files	tracemalloc_track.patch tracemalloc_track-2.patch tracemalloc_track-3.patch tracemalloc_track_fd.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-11-22.09:32:17.386>
created_at = <Date 2016-03-10.14:17:10.641>
labels = ['type-feature']
title = 'tracemalloc: add C API to manually track/untrack memory allocations'
updated_at = <Date 2017-04-12.13:42:47.846>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-04-12.13:42:47.846>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-11-22.09:32:17.386>
closer = 'vstinner'
components = []
creation = <Date 2016-03-10.14:17:10.641>
creator = 'vstinner'
dependencies = []
files = ['42115', '42116', '42118', '42289']
hgrepos = []
issue_num = 26530
keywords = ['patch']
message_count = 37.0
messages = ['261501', '261502', '261505', '261507', '261510', '261580', '261586', '261630', '261905', '261945', '261948', '261959', '261960', '261961', '261965', '261966', '261967', '261977', '261999', '262179', '262180', '262434', '263865', '265932', '274125', '274127', '274128', '274129', '274130', '274133', '274136', '274137', '281456', '291551', '291552', '291553', '291555']
nosy_count = 5.0
nosy_names = ['pitrou', 'vstinner', 'njs', 'python-dev', 'jtaylor']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26530'
versions = ['Python 3.6']

[vstinner]

The API of Python memory allocators was extended for numpy: calloc() was added (issue bpo-21233). It looks like it's not enough because allocations with an alignment is also required for vector operations (SIMD). For Python, there is the issue bpo-18835 but the benefit for Python code is unclear or negligible.

Instead of extending the API each time, it would be simpler to add an API to allow third party memory allocators to track/untrack memory in tracemalloc. I guess that the API should make the assumption that the GIL is not hold and so try to acquire the GIL (if it's not already hold).

Related numpy issue: numpy/numpy#4663

See also: https://mail.scipy.org/pipermail/numpy-discussion/2015-January/072068.html

[vstinner]

Patch without test.

I chose to require the GIL to be held for efficiency. IMHO the common case is more than the GIL is held.

Example to call the function if the GIL is not held:

    int res;
    PyGILState_STATE gil_state;
    gil_state = PyGILState_Ensure();
    res = _PyTraceMalloc_Track(ptr, size);
    PyGILState_Release(gil_state);
    return res;

See also my numpy pull request on PyMem_Malloc() called without the GIL being held:
numpy/numpy#7404

[pitrou]

This may indeed be useful but:

• those functions should be no-ops when tracing isn't enabled (so as to be as fast as possible)
• you should take the GIL if necessary (but only if tracing is enabled, of course)

[vstinner]

Antoine Pitrou:

This may indeed be useful but:

• those functions should be no-ops when tracing isn't enabled (so as to be as fast as possible)

Done

• you should take the GIL if necessary (but only if tracing is enabled, of course)

Ok, done.

The new patch has a safer API:

• _PyTraceMalloc_Track() can be called twice with the same pointer: the old trace is removed, a new trace is added
• _PyTraceMalloc_Track() ensures that the GIL is hold
• _PyTraceMalloc_Track() & _PyTraceMalloc_Untrack() do nothing if tracemalloc is disabled

I also added unit tests.

[vstinner]

Hum, there is an issue in the patch version 2: tracemalloc_add_trace() rely on the assumption that the pointer is not already tracked. Since we give control to the traces to the user, this assumption may become wrong (if the API is badly used).

This issue is now fixed in the patch version 3: tracemalloc_add_trace() now works if the pointer is already tracked. It allows to implement a micro-optimization on realloc() if the new pointer is equal to the old pointer (memory block didn't move).

By the way, to track realloc(): _PyTraceMalloc_Untrack() must be called with the old pointer and _PyTraceMalloc_Track() must be called with the new pointer. I don't think that it's worth to add an helper function just for two calls. (The optimization old_ptr==new_ptr is really a micro-optimization, don't use tracemalloc if you care of performances, tracemalloc simply kills the performance ;-))

[njsmith]

Hi Victor,

This is really great, thanks for working on it!

What do you think about the richer api I proposed here?
http://bugs.python.org/issue18835#msg232221

[vstinner]

"""
What do you think about the richer api I proposed here?
http://bugs.python.org/issue18835#msg232221
"""

msg232221:
"""
For numpy's purposes, I think the best approach would be to add a tracemalloc "escape valve", with an interface like:

PyMem_RecordAlloc(const char* domain, void* tag, size_t quantity, 
PyMem_RecordRealloc(const char* domain, void* old_tag, void* new_tag, size_t new_quantity)
PyMem_RecordFree(const char* domain, void* tag)
"""

I don't plan to store new information in tracemalloc. tracemalloc design is "simple": it stores a trace for each memory block. A memory block is identified by its address (void* pointer) and a trace is a tuple (size, traceback).

Extending a trace would increase the memory footprint overhead of tracemalloc.

Note: I don't understand what are domain, tag (address of the memory block?) or quantity (size of the memory block?).

Storing the C filename and C line number was proposed but rejected in the design of the PEP-445:
https://www.python.org/dev/peps/pep-0445/#pass-the-c-filename-and-line-number

I'm open to extending tracemalloc, but it requires to discuss the API and how changes will impact the code (tracemalloc, but also users of the tracemalloc API).

[njsmith]

Note: I don't understand what are domain, tag (address of the memory block?) or quantity (size of the memory block?).

Yes, the idea is basically the same as now, except with an extra const char * specifying the "domain", so that one could keep separate track of different kinds of allocations. So PyMem_Malloc would just call PyMem_RecordAlloc("heap", ptr, size) (or act equivalently to something that called that, etc.), but something like PyCuda might do PyMem_RecordAlloc("gpu", ptr, size) to track allocations in GPU memory. All the tracing stuff in tracemalloc would be awesome to have for GPU allocations, and it would hardly require any changes to enable it. Ditto for other leakable resources like file descriptors or shmem segments.

I don't plan to store new information in tracemalloc. tracemalloc design is "simple": it stores a trace for each memory block. A memory block is identified by its address (void* pointer) and a trace is a tuple (size, traceback).
Extending a trace would increase the memory footprint overhead of tracemalloc.

I think the extra footprint would be tiny? Logically, you'd index traces by (domain, pointer) instead of (pointer), but since we expect that the number of distinct domains is small (like... 1, in the common case) then the natural implementation strategy would be to replace the current {pointer: trace} mapping with a mapping {domain: {pointer: trace}}. So the total overhead would be basically the overhead of allocating a single additional dict.

That said, I think having the domain argument *in the public hookable API* is useful even if the tracemalloc hooks themselves decide to ignore it or just ignore all calls that don't refer to the "heap" domain -- because it's extremely cheap to put this in the API now while we're adding it, and it leaves the door open for someone else to come along later and make an awesome gpu heap tracker or whatever without having to first argue with python-dev and wait for the cpython release cycle.

[vstinner]

Nathaniel Smith:

So PyMem_Malloc would just call PyMem_RecordAlloc("heap", ptr, size) (or act equivalently to something that called that, etc.), but something like PyCuda might do PyMem_RecordAlloc("gpu", ptr, size) to track allocations in GPU memory.

If I change tracemalloc, it's not to fullfit numpy requirements, it must remain very generic. *If* we add something, I see 3 choices:

• add a C int to trace_t
• add a C char* to trace_t
• add a C void* to trace_t

int uses 2x less memory than char* or void* on 64-bit systems.

The problem of void* is to find a way to expose it in Python. An option is not ignore it in the Python API, and only provide a C API to retrieve traces with the extra info.

void* allows to implement the rejected option of also storing the C filename an C line number:
https://www.python.org/dev/peps/pep-0445/#pass-the-c-filename-and-line-number

When I designed the PEP-445 (malloc API) an PEP-454 (tracemalloc), I recall that it was proposed to add "colors" (red, blue, etc.) to memory allocations. It sounds similar do you "heap" and "gpu" use case. It's just that you use an integer rather than a string.

Anyway, extending tracemalloc is non-trivial, we have to investigate use cases to design the new API. I would prefer to move step by step, an begin with exposing existing API. What do you think?

All the tracing stuff in tracemalloc would be awesome to have for GPU allocations, and it would hardly require any changes to enable it. Ditto for other leakable resources like file descriptors or shmem segments.

FYI I opened an issue to use tracemalloc when logging ResourceWarning:
http://bugs.python.org/issue26567

I think the extra footprint would be tiny?

In my experience, tracemalloc footprint is large. Basically, it doubles the total memory footprint. So adding 4 or 8 bytes to a trace which currently takes 16 bytes is not negligible!

Maybe we can be smart and use compact trace when extra info is not stored (current trace_t) and switch to "extended" trace (trace_t + int/char*/void*) when the extended API is used? It requires to convert all existing traces from the compact to the extende format. It doesn't look too complex to support two formats, expecially if the extended format is based on the compact format (trace_t structure used in extended_trace_t structure).

Logically, you'd index traces by (domain, pointer) instead of (pointer)

It's not how tracemalloc is designed. _tracemalloc has a simple design. It's a simple hashtable: pointer => trace. The Python module tracemalloc.py is responsible to group traces:
https://docs.python.org/dev/library/tracemalloc.html#tracemalloc.Snapshot.statistics

The design is to have a simple and efficient _tracemalloc module, an off-load statistics later. It allows to capture traces on a small and slow device, and then analyze data on a fast compuer with more memory (ex: transfer data by network). The idea is also to limit the overhead of using _tracemalloc.

Moreover, your proposed structure looks specific to your use case. I'm not sure that you always want to group by the domain. If domain is a C traceback (filename, line number), you probably don't want to group by traceback, but group by C filename for example.

[njsmith]

I think we're talking past each other :-).

If I change tracemalloc, it's not to fullfit numpy requirements, it must remain very generic

Nothing about what I'm saying is relevant to numpy -- the patch attached to this bug report is already plenty for what numpy needs. (Well, it still needs a public API like PyMem_Track/Untrack or something, but never mind for now.)

The only reason I'm speaking up now is that if you're adding a manual track/untrack API, then a relatively trivial addition now makes tracemalloc vastly more powerful, so I don't want to miss this opportunity.

void* allows to implement the rejected option of also storing the C filename an C line number:

And if you want to attach some extra metadata to traces, then that's an interesting idea that I can potentially imagine various use cases for. But it's not the idea I'm talking about :-). (FWIW, I think the biggest challenge for your idea will be how the allocation sites -- which might be in arbitrary user code -- are supposed to figure out what kind of metadata they should be attaching. And if it's information that tracemalloc can compute itself -- like C backtraces -- then there's no reason for it to be in the public API, which is the thing I'm concerned about here.)

What I'm talking about is different: I think it should be possible to re-use the tracemalloc infrastructure to track other resources besides "heap allocations". So for my use case, it's crucial that we index by (domain, pointer), because the address 0xdeadbeef on the heap is different from the address 0xdeadbeef on the GPU. We'll never want to group by pointer alone without the domain, because that would cause us to actually misinterpret the data (if I do PyMem_Track("gpu", 0xdeadbeef); PyMem_Untrack("heap", 0xdeadbeef), then this should not cause tracemalloc to forget about the gpu allocation! I think this is very different than your C backtrace example). And, it's always obvious to callers what kind of thing to pass here, because they know perfectly well whether they just allocated memory on the heap or on the GPU, so the public API is an appropriate place for this information. And, it's immediately obvious that for this use case, there will only be a few different domains in use at one time, so it's very inefficient to literally store (domain, pointer) pairs -- replacing the current pointer => trace design with a domain => (pointer => trace) design would indeed require changing tracemalloc's design a bit, but not, I think, in any fundamental way?

[vstinner]

Instead of having to change the API for tracking GPU memory, I suggest to
try a hack. Memory allocations use an alignement. For pymalloc, it's 8
bytes for example. On a GPU I expect at least 2 bytes or more likey
something much bigger like 16 bytes. For malloc, I also expect at least 2
bytes.

If it's the minimum is 2 bytes, good. Use the lowest bit as a "GPU" flag!
pointer|1.

Tracemalloc doesn't give access to pointers in its API, so it shouldn't
matter.

[pitrou]

Le 18/03/2016 04:57, Nathaniel Smith a écrit :

What I'm talking about is different: I think it should be possible
to
re-use the tracemalloc infrastructure to track other resources besides
"heap allocations". So for my use case, it's crucial that we index by
(domain, pointer), because the address 0xdeadbeef on the heap is
different from the address 0xdeadbeef on the GPU. We'll never want to
group by pointer alone without the domain, because that would cause us
to actually misinterpret the data (if I do PyMem_Track("gpu",
0xdeadbeef); PyMem_Untrack("heap", 0xdeadbeef), then this should not
cause tracemalloc to forget about the gpu allocation! I think this is
very different than your C backtrace example).

FWIW, LLVM calls the "domain" "address space", and simply uses an
integer for it (0 being the default CPU-addressable address space). You
can probably restrict yourself to 8-bit address spaces...

[pitrou]

Le 18/03/2016 08:09, STINNER Victor a écrit :

If it's the minimum is 2 bytes, good. Use the lowest bit as a "GPU" flag!
pointer|1.

So you guarantee tracemalloc itself won't use such a hack for other
purposes? ;-)

[vstinner]

So you guarantee tracemalloc itself won't use such a hack for other
purposes? ;-)

As the defacto maintainer of the module, yes, I can guarantee that.

FWIW, LLVM calls the "domain" "address space", and simply uses an
integer for it (0 being the default CPU-addressable address space). You
can probably restrict yourself to 8-bit address spaces...

I understood that we have the CPU and GPU address spaces. Do you know other address spaces?

I only care of address spaces where two equal pointers point to two different memory blocks.