Implement PEP 529 for io.FileIO

[eryksun]

BPO	29409
Nosy	@pfmoore, @tjguk, @zware, @eryksun, @zooba
Files	issue_29409_01.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2018-01-07.01:37:04.102>
created_at = <Date 2017-02-01.03:06:20.594>
labels = ['type-bug', '3.7', 'expert-IO', 'OS-windows']
title = 'Implement PEP 529 for io.FileIO'
updated_at = <Date 2018-01-07.01:37:04.102>
user = 'https://github.com/eryksun'

bugs.python.org fields:

activity = <Date 2018-01-07.01:37:04.102>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2018-01-07.01:37:04.102>
closer = 'steve.dower'
components = ['Windows', 'IO']
creation = <Date 2017-02-01.03:06:20.594>
creator = 'eryksun'
dependencies = []
files = ['46473']
hgrepos = []
issue_num = 29409
keywords = ['patch']
message_count = 6.0
messages = ['286585', '286998', '287004', '287006', '287015', '287020']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29409'
versions = ['Python 3.6', 'Python 3.7']

[eryksun]

PEP-529 isn't implemented for io.FileIO, and I think it should be. If a UTF-8 path is passed to open(), it ends up calling C _open instead of decoding the path and calling C _wopen. Also, if a pathlike object is passed to io.FileIO, it calls PyUnicode_FSConverter on it, converting it to UTF-8 and then passing it to _open. For example:

    >>> p = r'C:\Temp\αβψδ'
    >>> os.path.exists(p)
    True

    >>> open(p.encode())
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: b'C:\\Temp\\\xce\xb1\xce\xb2\xcf\x88\xce\xb4'

    >>> io.FileIO(pathlib.Path(p))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    FileNotFoundError: [Errno 2] No such file or directory: WindowsPath('C:/Temp/αβψδ')

The Windows implementation should mirror the POSIX implementation via PyUnicode_FSDecoder.

[zooba]

Merging this in now, along with a few other patches. This is the test I'm adding, in case anyone spots any problems with it before I push (copied from the test for pure-ASCII bytes):

@unittest.skipIf(sys.getfilesystemencoding() != 'utf-8',
                 "test only works for utf-8 filesystems")
def testUtf8BytesOpen(self):
    # Opening a UTF-8 bytes filename
    try:
        fn = TESTFN_UNICODE.encode("utf-8")
    except UnicodeEncodeError:
        self.skipTest('could not encode %r to utf-8' % TESTFN_UNICODE)
    f = self.FileIO(fn, "w")
    try:
        f.write(b"abc")
        f.close()
        with open(TESTFN_UNICODE, "rb") as f:
            self.assertEqual(f.read(), b"abc")
    finally:
        os.unlink(TESTFN_UNICODE)

[python-dev]

New changeset 0bf72810f8ea by Steve Dower in branch '3.6':
Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
https://hg.python.org/cpython/rev/0bf72810f8ea

New changeset a5538734cc87 by Steve Dower in branch 'default':
Merge issue bpo-28164 and issue bpo-29409
https://hg.python.org/cpython/rev/a5538734cc87

[zooba]

Committed, but I'll leave this open for a bit in case anyone wants to comment on the commit.

[python-dev]

New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch 'master':
Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
fe7e868

New changeset 965f8d68a8dcc2ebb2480fe7e9121ac7efdee54e by Steve Dower in branch 'master':
Merge issue bpo-28164 and issue bpo-29409
965f8d6

[python-dev]

New changeset fe7e868bad5bffaf863c80fa2bcc4e3bf87a413a by Steve Dower in branch '3.6':
Issue bpo-29409: Implement PEP-529 for io.FileIO (Patch by Eryk Sun)
fe7e868