String allocations waste 3 bytes of memory on average.

[mdickinson]

BPO	4445
Nosy	@rhettinger, @mdickinson, @tiran
Files	string_alloc.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mdickinson'
closed_at = <Date 2008-12-06.15:34:04.137>
created_at = <Date 2008-11-27.12:14:44.506>
labels = ['interpreter-core', 'performance']
title = 'String allocations waste 3 bytes of memory on average.'
updated_at = <Date 2008-12-06.15:34:04.135>
user = 'https://github.com/mdickinson'

bugs.python.org fields:

activity = <Date 2008-12-06.15:34:04.135>
actor = 'mark.dickinson'
assignee = 'mark.dickinson'
closed = True
closed_date = <Date 2008-12-06.15:34:04.137>
closer = 'mark.dickinson'
components = ['Interpreter Core']
creation = <Date 2008-11-27.12:14:44.506>
creator = 'mark.dickinson'
dependencies = []
files = ['12149']
hgrepos = []
issue_num = 4445
keywords = ['patch']
message_count = 12.0
messages = ['76495', '76498', '76503', '76512', '76552', '76553', '76554', '76555', '76558', '76577', '77078', '77137']
nosy_count = 3.0
nosy_names = ['rhettinger', 'mark.dickinson', 'christian.heimes']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue4445'
versions = ['Python 2.6', 'Python 3.0', 'Python 2.7']

[mdickinson]

There are a number of places in Objects/stringobject.c where memory is
allocated for a string of length n using:

PyObject_MALLOC(sizeof(PyStringObject) + n)

On my computer (OS X 10.5.5/Intel), and, I suspect, on most common
platforms, the PyStringObject struct is going to contain some number of
bytes (probably 3) of trailing padding; the result is that the
PyObject_MALLOC call above asks for 3 more bytes than are necessary, and
on average the Python interpreter will waste 3 bytes of memory per string
allocation. Is there any reason not to replace these calls with:

PyObject_MALLOC(offsetof(PyStringObject, ob_sval) + n + 1)

instead?

Patch attached.

[mdickinson]

Updated patch: fix overflow checks to use offsetof instead of sizeof as
well.

[rhettinger]

+1 on the idea. -1 on the current patch. Can you encapsulate this in a
simpler macro? I find the proposed replacement to be cryptic and
error-prone (i.e. hard to mentally verify its correctness and somewhat
likely to be screwed-up by a future maintainer).

[rhettinger]

Suggestion:

#define PyStringObject_SIZE (offsetof(PyStringObject, ob_sval) + 1)

 	/* Inline PyObject_NewVar */
-	op = (PyStringObject *)PyObject_MALLOC(sizeof(PyStringObject) + size);
 	/* Inline PyObject_NewVar */
+	op = (PyStringObject *)PyObject_MALLOC(PyStringObject_SIZE + size);

[mdickinson]

Yep. That works nicely.

Here's a revised patch.

[tiran]

Why is +1 required here? If I understand offsetof() correctly than it
returns the position of the ob_sval element. Shouldn't PyStringObject +
offsetof(PyStringObject, ob_sval) point to the first element of the
ob_svall array?

[mdickinson]

The +1 is there for the trailing null byte on the string: if s is a
Python string with len(s) == n, then the ob_sval array needs space for n+1
characters.

[tiran]

Ah! I forgot the trailing \0 byte ... Thanks Mark!

[mdickinson]

Hmmm. test_sys fails on 64-bit build.
Patch updated to fix this.

All tests now pass on 32-bit and 64-bit, debug and non-debug builds.

[rhettinger]

Looks much cleaner.

[mdickinson]

Applied to the trunk in r67601. Will merge to other branches if the
buildbots look okay.

[mdickinson]

Applied to py3k in r67610.