Skip to content

Conversation

@asvetlov
Copy link
Contributor

@asvetlov asvetlov commented Jun 3, 2019

To remove test warnings about dangling threads

https://bugs.python.org/issue37137

@asvetlov asvetlov requested a review from 1st1 as a code owner June 3, 2019 23:33
@asvetlov asvetlov changed the title Close pid waiters on watcher.close() bpo-37137: Close pid waiters on watcher.close() Jun 3, 2019
@njsmith
Copy link
Contributor

njsmith commented Jun 4, 2019

I'm not sure this is correct... isn't it allowed to start a process through asyncio, and then for the parent process to exit and leave the child running?

In trio we use the threaded strategy, and we make it a daemon thread and if the trio loop exits, oh well, the thread won't outlive either the host process or the watched process, so it's OK to leak it.

To solve the test failures, maybe it's enough to say: the child process should be exited before the test finishes, and if the test waits to see that the child process has exited, then it's guaranteed that the thread has exited too. (If you want to be extra-certain, then the code that handles the child process notification in the main thread could call thread.join(). This is safe because it knows that the thread will exit within the next few milliseconds at worst, and makes sure that the Python threading infrastructure has also noticed that the thread has exited.)

@asvetlov
Copy link
Contributor Author

asvetlov commented Jun 4, 2019

There are two things here:

  1. Dangling threads are super hard to catch and debug. I never saw "dangling thread" reports locally while I constantly use --fail-env-changed option when executing tests. Now I understand how to emulate this problem (add time.sleep(1) at the end of threaded code) but there is no guarantee that a new test doesn't add a mess again.
  2. Surprisingly, watcher.close() is never called on the program execution. It's used by set_child_watcher() call only to close the previous watcher if exists. I suspect people change watcher even much rarely than change event loop policy, and always do it before starting asyncio code.

@njsmith
Copy link
Contributor

njsmith commented Jun 4, 2019

...but how does this solve the test issues if no-one calls close?

@asvetlov
Copy link
Contributor Author

asvetlov commented Jun 4, 2019

Well, you are right.
The fix should move pid-water threads closing into the test suite and add ResourceWarnings on __del__.
I'll do it in another PR

@asvetlov asvetlov closed this Jun 4, 2019
@asvetlov asvetlov deleted the dangling-threads-in-watcher branch June 4, 2019 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants