mock_open does not support iteration around text files.

[TonyFlury]

BPO	32933
Nosy	@rbtcollins, @ned-deily, @ezio-melotti, @merwok, @voidspace, @berkerpeksag, @TonyFlury, @bbayles
PRs	bpo-32933: Implement dunder iter method on mock_open #5974 3.7 Implement dunder iter method on mock_open bpo-32933 #5975 [3.7] Implement dunder iter method on mock_open bpo-32933 #5976 [3.6] : Implement dunder iter method on mock_open bpo-32933 #5977 [3.7] bpo-32933: Implement __iter__ method on mock_open() (GH-5974) #9311

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-12.22:24:40.785>
created_at = <Date 2018-02-24.08:58:38.826>
labels = ['3.7', '3.8', 'type-feature', 'library']
title = 'mock_open does not support iteration around text files.'
updated_at = <Date 2018-09-14.21:30:07.650>
user = 'https://github.com/TonyFlury'

bugs.python.org fields:

activity = <Date 2018-09-14.21:30:07.650>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-12.22:24:40.785>
closer = 'berker.peksag'
components = ['Library (Lib)']
creation = <Date 2018-02-24.08:58:38.826>
creator = 'anthony-flury'
dependencies = []
files = []
hgrepos = []
issue_num = 32933
keywords = ['patch']
message_count = 16.0
messages = ['312706', '315025', '315026', '320278', '321334', '321336', '325197', '325198', '325201', '325300', '325309', '325335', '325349', '325378', '325379', '325402']
nosy_count = 8.0
nosy_names = ['rbcollins', 'ned.deily', 'ezio.melotti', 'eric.araujo', 'michael.foord', 'berker.peksag', 'anthony-flury', 'bbayles']
pr_nums = ['5974', '5975', '5976', '5977', '9311']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32933'
versions = ['Python 3.7', 'Python 3.8']

[TonyFlury]

Using the unittest.mock helper mock_open with multi-line read data, although readlines method will work on the mocked open data, the commonly used iterator idiom on an open file returns the equivalent of an empty file.

    from unittest.mock import mock_open

    read_data = 'line 1\nline 2\nline 3\nline 4\n'
    with patch('builtins.open', mock_open) as mocked:
       with open('a.txt', 'r') as fp:
           assert [l for l in StringIO(read_data)] ==
                  [l for l in fp] 

will fail although it will work on a normal file with the same data, and using [l for l in fp.readlines()] will also work.

There is a relatively simple fix which I have a working local version - but I don't know how to provide that back to the library - or even if i should.

[merwok]

Is this related to bpo-33236 ?

[TonyFlury]

No - it isn't related.

In the case of mock_open; it isn't intended to be a simple MagicMock - it is meant to be a mocked version of open, and so to be useful as a testing tool, it should emulate a file as much as possible.

When a mock_open is created, you can provide an argument 'read_data' which is meant to be the data from your mocked file, so it is key that the dunder iter method actually returns an iterator. The mock_open implementation already provides special versions of read, readline and readlines methods which use the 'read_data' initial value as the content.
Currently though the dunder iter method isn't set at all - so the returned value would currently be an empty iterator (which makes mock_open unable to be used to test idiomatic python :

    def display(file_name):
        with open('a.txt', 'r') as fp:
            for line in fp:
                print(line)  

As a trivial example the above code when mock_open is used will be equivalent of opening an empty file, but this code :

    def display(file_name):
        with open('a.txt', 'r') as fp:
            while True:
                line = readline(fp)
                if line == '':
                    break
                print(line)

Will work correctly with the data provided to mock_open.

Regardless of how and when bpo-33236 is solved - a fix would still be needed for mock_open to make it provide an iterator for the mocked file.

[ned-deily]

Anthony's PR is awaiting merge. Although Yury has reviewed it, as the core developers mocktest experts, it would be good if Michael and/or Robert could also take a look.

[berkerpeksag]

This is basically a duplicate of bpo-21258, but I haven't closely look at the patches in both issues yet.

We should probably consider adding support for __next__ as well.

[TonyFlury]

But the __next__ is a method on the iterator;

So long as __iter__ returns a valid iterator (which it does in my pull request), it will by definition support __next___

Although it is entirely possible that I have misunderstood what you are saying.

[berkerpeksag]

New changeset 2087023 by Berker Peksag (Tony Flury) in branch 'master':
bpo-32933: Implement __iter__ method on mock_open() (GH-5974)
2087023

[berkerpeksag]

Thanks for the patch, Anthony.I consider this a new feature, so I removed 3.6 and 3.7 from the versions field. We can backport to 3.7 if other core developers think that it's worth to fix in the latest maintenance branch.