Emit ResourceWarning in multiprocessing Queue destructor

[vstinner]

BPO	30171
Nosy	@pitrou, @vstinner, @serhiy-storchaka, @applio
Files	queue_leak.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/applio'
closed_at = <Date 2017-07-22.10:21:01.344>
created_at = <Date 2017-04-26.13:52:50.417>
labels = ['invalid', '3.7', 'library', 'performance']
title = 'Emit ResourceWarning in multiprocessing Queue destructor'
updated_at = <Date 2017-07-22.10:21:01.343>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2017-07-22.10:21:01.343>
actor = 'pitrou'
assignee = 'davin'
closed = True
closed_date = <Date 2017-07-22.10:21:01.344>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2017-04-26.13:52:50.417>
creator = 'vstinner'
dependencies = []
files = ['46830']
hgrepos = []
issue_num = 30171
keywords = []
message_count = 12.0
messages = ['292346', '292347', '292348', '292850', '293001', '298015', '298034', '298035', '298036', '298039', '298045', '298662']
nosy_count = 4.0
nosy_names = ['pitrou', 'vstinner', 'serhiy.storchaka', 'davin']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue30171'
versions = ['Python 3.7']

[vstinner]

A multiprocessing Queue object managers multiple resources:

• a multiprocessing Pipe
• a thread
• (a lock and a semaphore)

If a Queue is not cleaned up properly, your application may leak many resources.

Try attached queue_leak.py to see an example "leaking a thread".

I suggest to emit a ResourceWarning warning in Queue destrutor. I don't know what should be the test to decide if a warning must be emitted?

• if the queue wasn't closed yet?
• if the thread is alive?
• if the queue wasn't closed yet and/or the thread is alive? (my favorite choice)

Other examples of objects emitting ResourceWarning:

• io files: io.FileIO, io.TextIOWrapper, etc.
• socket.socket
• subprocess.Popen: I recently added a ResourceWarning on that one
• asyncio transports and event loops

[vstinner]

Example:

haypo@selma$ ./python queue_leak.py
number of thread diff: +1
dangling threads!
before: [<_MainThread(MainThread, started 139814961067072)>]
after: [<_MainThread(MainThread, started 139814961067072)>]

Note: queue_leak.py resource check is based on test.support.reap_threads.

[vstinner]

Oh, I forgot that I hitted this issue while analyzing issue bpo-30131: test_logging leaks a "dangling" thread. It took me a while to find multiprocessing queues in the big test_logging file!

[vstinner]

See also issue bpo-30244: Emit a ResourceWarning in concurrent.futures executor destructors.

[pitrou]

The thread seems to be stopped when the Queue object is finalized:

        # Send sentinel to the thread queue object when garbage collected
        self._close = Finalize(
            self, Queue._finalize_close,
            [self._buffer, self._notempty],
            exitpriority=10
            )

I don't think the other resources (pipe, lock, semaphore) need explicit cleaning.

[vstinner]

Another example:
http://bugs.python.org/issue30886#msg298014

"The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess."

[pitrou]

The problem is that multiprocessing.Queue.join_thread() does nothing since the thread wasn't started by a subprocess.

I don't understand what this means. Can you clarify a bit?

[pitrou]

Specifically "the thread wasn't started by a subprocess"...

[vstinner]

Specifically "the thread wasn't started by a subprocess"...

I'm talking about this check in Queue._start_thread() of multiprocessing.queues:

        created_by_this_process = (self._opid == os.getpid())
        if not self._joincancelled and not created_by_this_process:
            self._jointhread = Finalize(
                self._thread, Queue._finalize_join,
                [weakref.ref(self._thread)],
                exitpriority=-5
                )

[vstinner]

Let's discuss created_by_this_process in bpo-30886.

This issue is more about adding or not a ResourceWarning.

[pitrou]

I don't think a ResourceWarning should be emitted. There is no risk of data loss or resource leak if you don't close a multiprocessing Queue explicitly. Actually, in the real world, I don't think I've ever seen code that closes queues explicitly.