test_multiprocessing_spawn ResourceWarning with -Werror

[vadmium]

BPO	25654
Nosy	@pitrou, @vstinner, @vadmium, @serhiy-storchaka, @applio
Files	cleanup_std_streams.patch multiprocessing_no_close_fd.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-15.04:50:25.230>
created_at = <Date 2015-11-18.07:13:48.195>
labels = ['interpreter-core', 'tests', 'type-bug', 'library']
title = 'test_multiprocessing_spawn ResourceWarning with -Werror'
updated_at = <Date 2016-04-15.08:31:42.752>
user = 'https://github.com/vadmium'

bugs.python.org fields:

activity = <Date 2016-04-15.08:31:42.752>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-15.04:50:25.230>
closer = 'martin.panter'
components = ['Interpreter Core', 'Library (Lib)', 'Tests']
creation = <Date 2015-11-18.07:13:48.195>
creator = 'martin.panter'
dependencies = []
files = ['41068', '41075']
hgrepos = []
issue_num = 25654
keywords = ['patch']
message_count = 27.0
messages = ['254834', '254843', '254864', '254885', '254888', '254893', '254912', '254915', '262358', '262391', '262406', '262407', '262411', '262413', '262414', '262415', '262416', '262417', '262418', '262419', '262421', '262437', '262458', '262480', '263406', '263447', '263459']
nosy_count = 8.0
nosy_names = ['pitrou', 'vstinner', 'jnoller', 'python-dev', 'sbt', 'martin.panter', 'serhiy.storchaka', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25654'
versions = ['Python 3.5', 'Python 3.6']

[vadmium]

Running the test suite with -Werror enabled causes test_multiprocessing_spawn to print out lots of unhandled errors from the garbage collector, and one of these causes a test failure. It looks like there are lots of files that should be explicitly closed rather than leaving them for the garbage collector.

Also, one error occurred _after_ the test command had exiting. That can’t be a good thing, though I have never used the multiprocessing module so maybe this is unavoidable.

$ ./python -Werror -m unittest -v test.test_multiprocessing_spawn
. . .

======================================================================
FAIL: test_sys_exit (test.test_multiprocessing_spawn.WithProcessesTestSubclassingProcess)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/media/disk/home/proj/python/cpython/Lib/test/_test_multiprocessing.py", line 483, in test_sys_exit
    self.assertEqual(f.read().rstrip(), str(reason))
AssertionError: "[1, 2, 3]\nException ignored in: <_io.Fi[136 chars]-8'>" != '[1, 2, 3]'
- [1, 2, 3]
?          -
+ [1, 2, 3]- Exception ignored in: <_io.FileIO name='/dev/null' mode='rb' closefd=True>
- ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

---

Ran 265 tests in 151.904s

FAILED (failures=1, skipped=15)
[Exit 1]
$ Exception ignored in: <_io.FileIO name='/dev/null' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

[serhiy-storchaka]

sys.stdin is set to open(os.devnull) in multiprocessing.forkserer and multiprocessing.process (see bpo-5313). At shutdown stage sys.std* streams are restored to sys.__std*__. Here the resource warning is emitted.

An example that demonstrates similar issue:

$ ./python -We -c 'import sys; sys.stdin = open("/dev/null")'
Exception ignored in: <_io.FileIO name='/dev/null' mode='rb' closefd=True>
ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

A workaround is to set sys.__stdin__ as well as sys.stdin. Now a resource warning is no longer emitted at the stage of restoring standard streams, but it can be randomly emitted at the stage of clearing sys dict (depending on the order of deallocating stdin and stderr). The question is why a resource warning is not emitted in normal case, with standard stdin.

Proposed patch closes sys.__stdin__ and set it to sys.stdin in multiprocessing, makes standard streams cleanup at shutdown more reliable, and fixes two tests that set sys.stderr.

[vadmium]

Perhaps we don’t normally see a warning because stdin normally has closefd=False set?

[serhiy-storchaka]

Good point!

The question is whether we want to see resource warnings when override standard streams with open files (usually without closefd=False)?

[serhiy-storchaka]

You can commit your patch right now (it shouldn't make things worse), and continue to work on additional tests.

[serhiy-storchaka]

Here is an alternative patch that uses closefd=False.

[applio]

Note bpo-21779 has been around for quite some time with its own patch specifically for multiprocessing, making its tests more robust.

I am hesitant to embrace the patch suggested here.

[serhiy-storchaka]

Oh, I forgot about bpo-21779. And there was at least yet one duplicate bpo-23827. Thanks for the reminder Davin.

The patch in bpo-21779 fixes only symptoms, makes failed test more lenient. I prefer to got rid of resource warnings at all.

[vstinner]

I marked the issue bpo-10305 as a duplicate of this issue.

[vstinner]

Oh, with fixes for issues bpo-21925 and bpo-26637, test_multiprocessing_spawn started to fail:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.5/builds/727/steps/test/logs/stdio

======================================================================
FAIL: test_sys_exit (test.test_multiprocessing_spawn.WithProcessesTestSubclassingProcess)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/buildarea/3.5.angelico-debian-amd64/build/Lib/test/_test_multiprocessing.py", line 483, in test_sys_exit
    self.assertEqual(f.read().rstrip(), str(reason))
AssertionError: "[1, 2, 3]\nsys:1: ResourceWarning: unclo[67 chars]-8'>" != '[1, 2, 3]'
- [1, 2, 3]
?          -
+ [1, 2, 3]- sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>

[python-dev]

New changeset 9faa03e9cea8 by Victor Stinner in branch 'default':
Issue bpo-25654:
https://hg.python.org/cpython/rev/9faa03e9cea8