pydoc: ServerThread.stop() leaves a dangling thread

[vstinner]

BPO	31238
Nosy	@pitrou, @vstinner
PRs	bpo-31238: pydoc ServerThread.stop() now joins itself #3151 [3.6] bpo-31238: pydoc ServerThread.stop() clears ref cycle #3177 [3.6] bpo-31238: pydoc ServerThread.stop() now joins itself (GH-3151) #7324

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-08-22.14:55:52.744>
created_at = <Date 2017-08-18.23:50:45.663>
labels = ['3.7', 'library', 'performance']
title = 'pydoc: ServerThread.stop() leaves a dangling thread'
updated_at = <Date 2018-06-01.17:39:19.372>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-06-01.17:39:19.372>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-08-22.14:55:52.744>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-08-18.23:50:45.663>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31238
keywords = []
message_count = 5.0
messages = ['300556', '300557', '300652', '300701', '318439']
nosy_count = 2.0
nosy_names = ['pitrou', 'vstinner']
pr_nums = ['3151', '3177', '7324']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue31238'
versions = ['Python 3.7']

[vstinner]

When using #3138 I see that test_pydoc leaves a dangling thread. The bug comes from test_server() which uses pydoc ServerThread.

ServerThread.stop() and test_pydoc don't join the thread. Moreover, ServerThread.docserver has a reference cycle through the DocServer.callback attribute.

Attached PR modifies ServerThread.stop() to join itself (the thread), to wait until the HTTP server completes, and then explicitly break the reference cycle.

With the PR, pydoc may hang if a bad HTTP client. So another option is to only modify test_pydoc to join() + break the ref cycle.

[vstinner]

This issue is similar to bpo-31151: socketserver ForkingMixIn.server_close() leaks zombie processes.

[vstinner]

New changeset 4cab2cd by Victor Stinner in branch 'master':
bpo-31238: pydoc ServerThread.stop() now joins itself (bpo-3151)
4cab2cd

[vstinner]

I left Python 2.7 and 3.6 unchanged to focus on fixing dangling threads and other similar bugs in the master branch first. I will see later if we really need to backport the fix to stable branches. In the meanwhile, I close the issue.

[vstinner]

New changeset f329743 by Victor Stinner in branch '3.6':
bpo-31238: pydoc ServerThread.stop() now joins itself (GH-3151) (GH-7324)
f329743