memoryviews and ctypes

[dabeaz]

BPO	15944
Nosy	@pitrou, @skrah, @vadmium, @eryksun, @MojoVampire
Files	cast-bytes.patch: Universal cast to bytes

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/skrah'
closed_at = <Date 2015-08-08.11:45:15.291>
created_at = <Date 2012-09-14.15:05:29.975>
labels = ['interpreter-core', 'type-bug']
title = 'memoryviews and ctypes'
updated_at = <Date 2015-08-08.11:45:15.195>
user = 'https://bugs.python.org/dabeaz'

bugs.python.org fields:

activity = <Date 2015-08-08.11:45:15.195>
actor = 'skrah'
assignee = 'skrah'
closed = True
closed_date = <Date 2015-08-08.11:45:15.291>
closer = 'skrah'
components = ['Interpreter Core']
creation = <Date 2012-09-14.15:05:29.975>
creator = 'dabeaz'
dependencies = []
files = ['40139']
hgrepos = []
issue_num = 15944
keywords = ['patch']
message_count = 31.0
messages = ['170477', '170480', '170481', '170482', '170483', '170484', '170487', '170488', '170489', '170490', '170492', '170494', '170496', '170795', '170818', '170819', '229560', '248037', '248089', '248111', '248127', '248134', '248135', '248164', '248182', '248186', '248189', '248190', '248191', '248261', '248262']
nosy_count = 8.0
nosy_names = ['pitrou', 'skrah', 'dabeaz', 'python-dev', 'martin.panter', 'cblp', 'eryksun', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue15944'
versions = ['Python 3.5', 'Python 3.6']

[dabeaz]

I've been playing with the interaction of ctypes and memoryviews and am curious about intended behavior. Consider the following:

>>> import ctypes
>>> d = ctypes.c_double()
>>> m = memoryview(d)
>>> m.ndim
0
>>> m.shape
()
>>> m.readonly
False
>>> m.itemsize
8
>>>

As you can see, you have a memory view for the ctypes double object. However, the fact that it has a 0-dimension and no shape seems to cause all sorts of weird behavior. For instance, indexing and slicing don't work:

>>> m[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid indexing of 0-dim memory
>>> m[:]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: invalid indexing of 0-dim memory
>>> 

As such, you can't really seem to do anything interesting with the resulting memory view. For example, you can't pull data out of it. Nor can you overwrite the contents (i.e., replacing the contents with an 8-byte byte string).

Attempting to cast the memory view to something else doesn't work either.

>>> d = ctypes.c_double()
>>> m = memoryview(d)
>>> m2 = m.cast('c')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: memoryview: source format must be a native single character format prefixed with an optional '@'
>>> 

I must be missing something really obvious here. Is there no way to get access to the memory behind a ctypes object?

[pitrou]

You can still read the underlying representation:

>>> d = ctypes.c_double(0.6)
>>> m = memoryview(d)
>>> bytes(m)
b'333333\xe3?'
>>> d.value = 0.7
>>> bytes(m)
b'ffffff\xe6?'

[dabeaz]

I don't want to read the representation by copying it into a bytes object. I want direct access to the underlying memory--including the ability to modify it. As it stands now, it's completely useless.

[skrah]

0-dim memory is indexed by x[()]. The ctypes example has an additional
problem, because format="<d" is not yet implemented in memoryview.

Only native single character formats in struct module syntax are
implemented, and "<d" in struct module syntax means "standard size,
little endian".

To demonstrate 0-dim indexing, here's an example using _testbuffer:

>>> x = ndarray(3.14, shape=[], format='d', flags=ND_WRITABLE)
>>> x[()]
3.14
>>> tau = 6.28
>>> x[()] = tau
>>> x[()]
6.28
>>> m = memoryview(x)
>>> m[()]
6.28
>>> m[()] = 100.111
>>> m[()]
100.111

[skrah]

BTW, if c_double means "native machine double", then ctypes should
fill in Py_buffer.format with "d" and not "<d" in order to be PEP-3118
compatible.

[dabeaz]

Even with the <d format, I'm not sure why it can't be cast to simple byte-view. None of that seems to work at all.

[skrah]

The decision was made in order to be able to cast back and forth between
known formats. Otherwise one would be able to cast from '<d' to 'B'
but not from 'B' to '<d'.

Python 3.4 will have support for all formats in struct module syntax,
but all non-native formats will be *far* slower than the native ones.

You can still pack/unpack directly using the struct module:

>>> import ctypes, struct
>>> d = ctypes.c_double()
>>> m = memoryview(d)
>>> struct.pack_into(m.format, m, 0, 22.7)
>>> struct.unpack_from(m.format, m, 0)[0]
22.7

[dabeaz]

I don't think memoryviews should be imposing any casting restrictions at all. It's low level. Get out of the way.

[skrah]

So you want to be able to segfault the core interpreter using the
builtins?

[dabeaz]

No, I want to be able to access the raw bytes sitting behind a memoryview as bytes without all of this casting and reinterpretation. Just show me the raw bytes. Not doubles, not ints, not structure packing, not copying into byte strings, or whatever. Is this really impossible? It sure seems so.

[dabeaz]

Just to be specific, why is something like this not possible?

>>> d = ctypes.c_double()
>>> m = memoryview(d)
>>> m[0:8] = b'abcdefgh'
>>> d.value
8.540883223036124e+194
>>>

(Doesn't have to be exactly like this, but what's wrong with overwriting bytes with bytes of a compatible size?).

[dabeaz]

I should add that 0-dim indexing doesn't work as described either:

>>> import ctypes
>>> d = ctypes.c_double()
>>> m = memoryview(d)
>>> m[()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NotImplementedError: memoryview: unsupported format <d
>>>

[skrah]

Please read msg170482. It even contains a copy and paste example!

[skrah]

As I understand it, you prefer memoryviews where the format is
purely informational, whereas we now have typed memoryviews.

Typed memoryviews are certainly useful, in fact they are
present in Cython, see here for examples:

http://docs.cython.org/src/userguide/memoryviews.html

I can see only one obvious benefit of ignoring the format: All possible
formats are accepted. What I don't understand is why this ...

m[0] = b'\x00\x00\x00\x01'

... should be preferable to:

m[0] = 1

If you think that typed memoryviews are a mistake, I suggest raising
the issue on python-dev as soon as possible (3.3 is due soon). All
memoryview operations are now based on values instead of bit patterns,
see for example bpo-15573.