Rework Python finalization

[vstinner]

BPO	21788
Nosy	@vstinner, @benjaminp
Files	test_4_daemon_threads.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-03-18.13:30:34.039>
created_at = <Date 2014-06-17.07:40:27.147>
labels = []
title = 'Rework Python finalization'
updated_at = <Date 2015-03-18.13:30:34.038>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2015-03-18.13:30:34.038>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-03-18.13:30:34.039>
closer = 'vstinner'
components = []
creation = <Date 2014-06-17.07:40:27.147>
creator = 'vstinner'
dependencies = []
files = ['35669']
hgrepos = []
issue_num = 21788
keywords = []
message_count = 4.0
messages = ['220804', '220805', '220807', '238436']
nosy_count = 2.0
nosy_names = ['vstinner', 'benjamin.peterson']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue21788'
versions = ['Python 3.5']

[vstinner]

Hi,

During the development of Python 3.4, I tried to emit warnings if a file is destroyed without being explicitly closed. The warnings were not emited in threads during Python finalization. Here are related changes:

• Issue bpo-19421: "FileIO destructor imports indirectly the io module at exit"
• Issue bpo-19424: "_warnings: patch to avoid conversions from/to UTF-8"
• Issue bpo-19442: "Python crashes when a warning is emitted during shutdown"
• Change in Python shutdown: issue bpo-19466 "Clear state of threads earlier in Python shutdown"
• Regression => issue bpo-20526

The change bpo-19466 had to be reverted a few days before the release of Python 3.4.0 because it caused the regression bpo-20526.

I'm still not convinced that bpo-20526 was a new bug. IMO the bug still exists, but it is just less likely without the change bpo-19466.

There is still something wrong in Python finalization, so I open this issue to rework it.

The goal is to get warnings in test_4_daemon_threads() of test_threading. I attached the test as a Python script.

[vstinner]

Benjamin changed ThreadState_DeleteCurrent() to fix an issue in this part of the code, but then reverted his change:
"revert tstate_delete_common, since it's pretty much wrong"
http://hg.python.org/cpython/rev/3ce746735b99

(Yeah this part of the code is really tricky, likely to deadlock.)

[vstinner]

Charles-François Natali wrote a general comment about daemon threads:
http://bugs.python.org/issue19466#msg206028

[vstinner]

This issue was more a reminder for myself (TODO). I'm no more interesting to work on it, so I close it.

The Python finalization is very fragile. I prefer to not touch it again :-p