Tkinter hangs if using multiple threads and event handlers

[native-api]

BPO	33412
Nosy	@terryjreedy, @serhiy-storchaka, @native-api
Files	TkinterHanders3.py trace.zip: trace log trace.py: modified Lib/trace.py to show thread IDs TkinterHanders32.py: fixed script

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-05-03.03:30:05.101>
created_at = <Date 2018-05-02.17:51:49.449>
labels = ['3.8', 'invalid', '3.7', 'expert-tkinter', 'type-crash']
title = 'Tkinter hangs if using multiple threads and event handlers'
updated_at = <Date 2018-05-06.21:20:10.696>
user = 'https://github.com/native-api'

bugs.python.org fields:

activity = <Date 2018-05-06.21:20:10.696>
actor = 'Ivan.Pozdeev'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-03.03:30:05.101>
closer = 'gvanrossum'
components = ['Tkinter']
creation = <Date 2018-05-02.17:51:49.449>
creator = 'Ivan.Pozdeev'
dependencies = []
files = ['47562', '47563', '47564', '47570']
hgrepos = []
issue_num = 33412
keywords = []
message_count = 11.0
messages = ['316082', '316083', '316090', '316102', '316104', '316132', '316151', '316152', '316225', '316242', '316243']
nosy_count = 3.0
nosy_names = ['terry.reedy', 'serhiy.storchaka', 'Ivan.Pozdeev']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue33412'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[native-api]

With threaded Tkinter, TkinterHanders3.py from https://bugs.python.org/issue33257 (attached) hangs.

Tracing with thread_debug and a modified trace.py (to show TIDs, attached) shows that worker threads are waiting for the Tcl lock while the main thread that holds it keeps waiting for some other lock with a strange timeout:

19000: PyThread_acquire_lock_timed(00000000001B0F80, 0) called
19000: PyThread_acquire_lock(00000000001B0F80, 0) -> 0
19000: PyThread_acquire_lock_timed(00000000001B0F80, -1000000) called

Tested on 3.6 head, win7 x64, debug build.

[native-api]

worker threads are waiting for the Tcl lock

Pardon. They are waiting for Tkapp_ThreadSend()s into the main thread to return. The effect is still the same.

[gvanrossum]

So it seems threads and Tkinter events don't mix. This doesn't surprise me much. (Similar issues can occur when mixing threads and asyncio if you don't follow the documentation's advice about how to send events across threads.)

Perhaps event_generate() needs to be more careful with locking?

Do you have a suggestion for what to do short of dropping Tkinter support?

[native-api]

Do you have a suggestion for what to do short of dropping Tkinter support?

Didn't really look into this.
At first glance, from the trace log, the main thread seems to grab a lock at some initial point, and then tries to grab it again when running an event handler. So making the lock reentrant may help.

[gvanrossum]

I guess nobody gives a damn.

[terryjreedy]

From 1994 to 2017, _tkinter.c has received a steady flow of multiple revisions each year, for 333 total, by maybe 20 people including Guido. This makes it one of the more actively maintained files and indicates the opposite of indifference and not caring.

I tested event generation from threads on my Win 10 machine. With installed 2.7, with non-thread tcl 8.5, the process hangs as described, after 1 key event is sent and received. So we should document "don't do that".

With installed 64-bit 3.6 with thread-compiled tcl 8.6, I see something completely different. The process runs for 5 seconds until the stop call. The two threads alternate sending key events, which are all received in the main thread. Ditto for built 32-big debug 3.6 and 3.8.

The only problem is that the first t.join() hangs because of a thread deadlock bug. t.join() blocks until t.run exits. t.run does not exit until the last event_generate, with running=False, returns. But that blocks until dummy_handler runs. Dummy_handler does not run when the main thread is blocked by t.join.

I fixed this by not generating events when running is False. The revised program exits properly.

    def run(self):
        tid = self.ident
        while True:
            time.sleep(0.02)
            c = random.choice(string.ascii_letters)
            print(running, "%d: sending '%s'"%(tid,c),file=sys.stderr)
            if running:
                self.target.event_generate(c)
            else:
                break

I suppose there is a teeny possibility that 'running' could be flipped between the test and the call. Can that be prevented with a lock?

Another possibility is for stop() to change conditions so that 'self.target.event_generate(c)' fails with an exception, and change if/else to for/except. My first try sort of works in IDLE but not the console.

[native-api]

Another possibility is for stop() to change conditions so that 'self.target.event_generate(c)' fails with an exception

Could you elaborate? Since there're no docs on event_generate(), I can't look up how to make it "fail with an exception" without actually posting an event.

The only problem is that the first t.join() hangs because of a thread deadlock bug. t.join() blocks until t.run exits. t.run does not exit until the last event_generate, with running=False, returns. But that blocks until dummy_handler runs.
I suppose there is a teeny possibility that 'running' could be flipped between the test and the call. Can that be prevented with a lock?

The idea is to let the worker threads finish their work, not terminate them forcibly.
So the real problem is that stop() blocks the event loop.
It should rather run asynchronously, wait for threads, then trigger self.root.destroy() in the main thread... somehow.

[native-api]

Attached a fixed script.

Tk.after() works from a worker thread, while Tk.destroy() doesn't.

That's because Tkinter implements Tcl calls (_tkinter.c:Tkapp_Call) from another thread by posting an event to the interpreter's queue (Tcl_ThreadQueueEvent) and waiting for result. So a call normally works, but would hang if the interpreter's event loop is not running.

destroy()'s Python part (Lib\tkinter_init_.py:2055) stops the event loop, then makes more Tcl calls -- which hang for the aforementioned reason if made from another thread.

[terryjreedy]

When I closed the main thread by clicking [x], thus destroying root, both event threads raised instead of hanging. So my experiment involved calling root.destroy instead of setting running to False. The better result when running under IDLE might be due to IDLE's run process executes tcl.call('update') about 20 times per second. Even if the exception idea could be made to work, it seems like a kludge.

Waiting on the event threads from a separate non-gui thread and leaving the main thread fully functional and responsive until the gui threads die seems much cleaner. Perhaps this should be recommended as a standard way to shut down the main thread when there might be active gui threads. Thank you for following through with this to get a solution we both like.