Python and C implementations of io are out of sync

[pitrou]

BPO	9858
Nosy	@gpshead, @pitrou, @rbtcollins, @benjaminp
Files	missing_methods.py issue9858.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-20.19:52:57.042>
created_at = <Date 2010-09-15.08:53:09.266>
labels = ['easy', 'type-bug', 'expert-IO']
title = 'Python and C implementations of io are out of sync'
updated_at = <Date 2015-05-20.19:52:57.040>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2015-05-20.19:52:57.040>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-20.19:52:57.042>
closer = 'pitrou'
components = ['IO']
creation = <Date 2010-09-15.08:53:09.266>
creator = 'pitrou'
dependencies = []
files = ['18889', '39212']
hgrepos = []
issue_num = 9858
keywords = ['patch', 'easy']
message_count = 21.0
messages = ['116440', '116442', '116443', '116444', '116446', '116455', '116460', '116462', '116463', '116464', '116468', '116501', '116502', '221727', '241007', '242101', '243686', '243688', '243689', '243691', '243692']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'pitrou', 'rbcollins', 'benjamin.peterson', 'stutzbach', 'BreamoreBoy', 'python-dev', 'jcon', 'laura']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9858'
versions = ['Python 3.5']

[pitrou]

RawIOBase is defined as having the readinto() method but it doesn't. It should at least have a failing one (raising io.UnsupportedOperation like the BufferedIOBase.{read,readinto} methods do).

This is, of course, mainly for auto-documenting purposes (with help() and friends).

[benjaminp]

+1 for a failing one. (Does the base implementation not raise?)

Is it even possible to implement a real one without buffering?

[pitrou]

+1 for a failing one. (Does the base implementation not raise?)

The base implementation doesn't exist :)

Is it even possible to implement a real one without buffering?

FileIO does. It's the same as read(), except that you read into an existing buffer rather than allocating a new bytes object.

[stutzbach]

Attached is a script to find all of the mismatches between the C and Python implementations. There are several. Below is the output:

RawIOBase C is missing: ['readinto', 'write']
StringIO C is missing: ['name']
StringIO python is missing: ['__getstate__', '__setstate__']
BufferedRandom python is missing: ['raw']
BlockingIOError python is missing: ['characters_written']
TextIOWrapper python is missing: ['buffer']
BufferedReader python is missing: ['raw']
BufferedWriter python is missing: ['raw']
BytesIO python is missing: ['__setstate__']

Since the Python version was the original reference implementation, adding the attributes missing from the C side seems to be a straightforward decision (and it should simply match whatever the Python version does). It looks like a number of new attributes have creeped into the C side, which will require more thoughtful decision making.

We should probably double-check that each of these is documented, while we're at it.

[stutzbach]

FWIW, I just opened bpo-9859: Add tests to verify API match of modules with 2 implementations.

[benjaminp]

2010/9/15 Daniel Stutzbach <report@bugs.python.org>:

Daniel Stutzbach <daniel@stutzbachenterprises.com> added the comment:

Attached is a script to find all of the mismatches between the C and Python implementations.  There are several.  Below is the output:

BufferedRandom python is missing: ['raw']
TextIOWrapper python is missing: ['buffer']
BufferedReader python is missing: ['raw']
BufferedWriter python is missing: ['raw']

These attributes exist; they're just not properties.

[stutzbach]

These attributes exist; they're just not properties.

Yes, I see. They're added to the instance in the constructor, so they don't exist as attributes of the class. Also in that category:

BlockingIOError python is missing: ['characters_written']

That leaves:

RawIOBase C is missing: ['readinto', 'write']
StringIO C is missing: ['name']
StringIO python is missing: ['__getstate__', '__setstate__']
BytesIO python is missing: ['__setstate__']

The Python version of StringIO throws an AttributeException on the .name attribute. It's a property inherited from the TextIOWrapper. Effectively, TextIOWrapper provides the .name attribute if the object that it's wrapping provides the .name attribute. This behavior is undocumented.

Is that reasonable behavior? Or should TextIOWrapper define .name always and return some suitable value if the wrapped object doesn't define .name? (e.g., None)

In any case, it needs documentation.

[benjaminp]

2010/9/15 Daniel Stutzbach <report@bugs.python.org>:

Daniel Stutzbach <daniel@stutzbachenterprises.com> added the comment:

> These attributes exist; they're just not properties.

Yes, I see. They're added to the instance in the constructor, so they don't exist as attributes of the class. Also in that category:

BlockingIOError python is missing: ['characters_written']

That leaves:

RawIOBase C is missing: ['readinto', 'write']
StringIO C is missing: ['name']
StringIO python is missing: ['__getstate__', '__setstate__']
BytesIO python is missing: ['__setstate__']

I'm not sure if this pickling stuff matters as long as they both pickle.

The Python version of StringIO throws an AttributeException on the .name attribute. It's a property inherited from the TextIOWrapper. Effectively, TextIOWrapper provides the .name attribute if the object that it's wrapping provides the .name attribute. This behavior is undocumented.

Is that reasonable behavior? Or should TextIOWrapper define .name always and return some suitable value if the wrapped object doesn't define .name? (e.g., None)

Yes, all the code expects an AttributeError on name is it's not
present. For example, FileIO only sets its name attribute if it is
passed a filename and not an fd.

[stutzbach]

On Wed, Sep 15, 2010 at 10:51 AM, Benjamin Peterson
<report@bugs.python.org>wrote:

I'm not sure if this pickling stuff matters as long as they both pickle.

I'm not sure either. Do other Python implementations try to maintain a
compatible pickle format with CPython? If so, I think CPython's C and
Python versions should also pickle to the same format.

Yes, all the code expects an AttributeError on name is it's not
present. For example, FileIO only sets its name attribute if it is
passed a filename and not an fd.

Actually, FileIO sets the name attribute to the file descriptor if it isn't
passed a name:

1

[stutzbach]

Roundup does not play well with Gmail when Gmail is in Rich Format mode ;-)

That example should have read:

>>> f = io.FileIO(1)
>>> f.name
1

[benjaminp]

2010/9/15 Daniel Stutzbach <report@bugs.python.org>:
>
> Daniel Stutzbach <daniel@stutzbachenterprises.com> added the comment:
>
> Roundup does not play well with Gmail when Gmail is in Rich Format mode ;-)
>
> That example should have read:
>
>>>> f = io.FileIO(1)
>>>> f.name
> 1

Hmm, none the less other code expects it.