linecache and MemoryError

[serhiy-storchaka]

BPO	23838
Nosy	@pitrou, @vstinner, @ezio-melotti, @voidspace, @serhiy-storchaka
Files	linecache_memoryerror.patch linecache_memoryerror_2.patch linecache_memoryerror_3.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-04-01.14:01:34.267>
created_at = <Date 2015-04-01.11:24:24.066>
labels = ['library', 'performance']
title = 'linecache and MemoryError'
updated_at = <Date 2015-04-01.14:01:34.266>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-04-01.14:01:34.266>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-04-01.14:01:34.267>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2015-04-01.11:24:24.066>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38779', '38782', '38783']
hgrepos = []
issue_num = 23838
keywords = ['patch']
message_count = 13.0
messages = ['239781', '239785', '239788', '239790', '239792', '239798', '239800', '239801', '239802', '239804', '239805', '239812', '239814']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'ezio.melotti', 'michael.foord', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue23838'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[serhiy-storchaka]

When format a traceback of MemoryError, linecache tries to read all lines of source file and likely raises a MemoryError. For example:

http://buildbot.python.org/all/builders/x86%20OpenIndiana%203.x/builds/9712/steps/test/logs/stdio

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1280, in runtest_inner
    test_runner()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/test_socket.py", line 5340, in test_main
    support.run_unittest(*tests)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1808, in run_unittest
    _run_suite(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/support/__init__.py", line 1774, in _run_suite
    result = runner.run(suite)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 176, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/suite.py", line 122, in run
    test(result)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 625, in __call__
    return self.run(*args, **kwds)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 585, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/case.py", line 518, in _feedErrorsToResult
    result.addError(test, exc_info)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/runner.py", line 67, in addError
    super(TextTestResult, self).addError(test, err)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 17, in inner
    return method(self, *args, **kw)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 115, in addError
    self.errors.append((test, self._exc_info_to_string(err, test)))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/unittest/result.py", line 186, in _exc_info_to_string
    exctype, value, tb, limit=length, capture_locals=self.tb_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 532, in main
    result = runtest(*args, **kwargs)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 967, in runtest
    display_failure=False)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1304, in runtest_inner
    msg = traceback.format_exc()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 161, in format_exc
    return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 115, in format_exception
    type(value), value, tb, limit=limit).format(chain=chain))
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 456, in __init__
    capture_locals=capture_locals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 341, in extract
    f.line
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/traceback.py", line 270, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 44, in getlines
    return updatecache(filename, module_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/linecache.py", line 134, in updatecache
    lines = fp.readlines()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
MemoryError
Traceback (most recent call last):
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/__main__.py", line 3, in <module>
    regrtest.main_in_temp_cwd()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 1564, in main_in_temp_cwd
    main()
  File "/export/home/buildbot/32bits/3.x.cea-indiana-x86/build/Lib/test/regrtest.py", line 738, in main
    raise Exception("Child error on {}: {}".format(test, result[1]))
Exception: Child error on test_socket: 
make: *** [buildbottest] Error 1
program finished with exit code 2

Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read).

I hesitate how to classify this issue. On one hand, it looks as new feature, on other hand it is related to testing on all versions.

[vstinner]

Proposed patch clears the cache and repeats an attempt to read source file, and returns empty result if the second attempt fails (as when the source file can't be read).

IMO it's a bad idea. You should not retry an operation on MemoryError but only do you best to limit errors.

I don't think that it's correct to return an empty list for getlines() on MemoryError. So you maybe chose the wrong module: you should handle MemoryError in the traceback module, not in the linecache module.

It's just fine to return an empty string for FrameSummary.line. It's common to not being able to display a line: because the .py was removed (only the .pyc is available), invalid line number, invalid filename ("<string>" or "<stdin>"), etc.

If you are motivated to make the traceback module more reliable, you should maybe also enhance _Py_DisplaySourceLine() in Python/traceback.c. You can probably ignore almost all errors in _Py_DisplaySourceLine() ("except (Exception, MemoryError): pass").

[vstinner]

Serhiy: you look to be interested in fixing Python to not fail badly under low memory condition. You may take a look at bpo-19817 which proposes a function to arbitrary limit the memory allocated to Python. I fixed a lot of issues found by failmalloc, but they are still serious issues. For example, unit tests crash on subinterpreter tests with failmalloc, and there is also the memory allocation failure when raising MemoryError: see bpo-19835.

[serhiy-storchaka]

getlines() already returns an empty list on OSError. Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance the repeated reading will be successful. In any case it wouldn't make worse.

If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place.

[vstinner]

2015-04-01 14:02 GMT+02:00 Serhiy Storchaka <report@bugs.python.org>:

getlines() already returns an empty list on OSError.

Oh, I misread getlines(), you're right.

Clearing the cache on MemoryError will help to avoid other MemoryErrors. And after this there is a large chance

I agree, but please avoid retrying on MemoryError. IMO clearing the
cache and return [] is enough.

If don't change linecache, then we should patch all places where linecache is used: traceback, idlelib, warnings, trace, debugger, etc. I intentionally made the fix at the low level, so it fixes all issues in one place.

I agree.

[serhiy-storchaka]

OK, here is a patch that just clear the cache and returns an empty result.

It is hard to write reliable test, but I think we will see the result on OpenIndiana buildbot.

The question is left: should the patch be applied only on 3.5 or all versions?

[vstinner]

It is hard to write reliable test

Ah? Why not using something like mock.patch('mock.updatecache', side_effect=MemoryError)?

[vstinner]

The question is left: should the patch be applied only on 3.5 or all versions?

linecache_memoryerror_2.patch looks good to me, but it would be better with an unit test.

IMO Python 2.7 and 3.4 can be fixed too.

[serhiy-storchaka]

This test depends on implementation details. I prefer to avoid adding such tests for trivial changes.

[serhiy-storchaka]

On other side, linecache tests already use updatecache(). Here is a patch with a test.

[vstinner]

linecache_memoryerror_3.patch reviewed (comments on Rietveld).

[python-dev]

New changeset d444496e714a by Serhiy Storchaka in branch '2.7':
Issue bpo-23838: linecache now clears the cache and returns an empty result on
https://hg.python.org/cpython/rev/d444496e714a

New changeset 88a0e6cd93c3 by Serhiy Storchaka in branch '3.4':
Issue bpo-23838: linecache now clears the cache and returns an empty result on
https://hg.python.org/cpython/rev/88a0e6cd93c3

New changeset ab2326c801c2 by Serhiy Storchaka in branch 'default':
Issue bpo-23838: linecache now clears the cache and returns an empty result on
https://hg.python.org/cpython/rev/ab2326c801c2

[serhiy-storchaka]

Thank you for your review Victor. Will look on OpenIndiana buildbot.