Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor

Question

I've a piece of code that submits a task to a ThreadPoolExecutor which starts a Process. I've realised that in Python 3.8 that Process finished with exit code 0. But I've updated Python to the 3.9 version and this started to finishing with exit code 1! Even when the Process executes an empty task.

Here's a minimal example:

from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor


def some_task():
    pass


def execute_error():
    p = Process(target=some_task)
    p.start()
    p.join()
    print(p.exitcode)  # This is always 1 on a ThreadPoolExecutor!!!


executor = ThreadPoolExecutor(max_workers=4)
executor.submit(execute_error)
# execute_error()  # IMPORTANT: this works correctly (exit 0)

My versions:

Ubuntu 21.04
Python 3.9.4

Note that if __execute_error is called outside the ThreadPoolExecutor it works correctly. Is there I am missing? Is this a Python 3.9 bug? Is there any workaround? Any kind of help would be really appreciated

Interesting find. I can confirm the 3.8/3.9 discrepancy. Also, seems like ProcessPoolExecutor always returns 0 as expected, only ThreadPoolExecutor is affected. — xjcl
– xjcl, Commented Apr 26, 2021 at 22:07
I've submitted an issue with the same title to the Python issue tracker — Genarito
– Genarito, Commented Apr 26, 2021 at 23:17
That's awkward! Could you leave a link to the issue you posted for posterity? — xjcl
– xjcl, Commented Apr 26, 2021 at 23:20
Yes sorry! I commented from the cellphone where I didn't have the link. Here it is: bugs.python.org/issue43944 — Genarito
– Genarito, Commented Apr 26, 2021 at 23:22

xjcl · Accepted Answer · 2021-04-26 23:19:21Z

1

The multiprocessing module provides 3 methods of starting a process: spawn, fork, and forkserver. My guess is that you are using a Unix-based system which defaults to the fork method. The documentation for fork mentions:

Note that safely forking a multithreaded process is problematic.

fork copies the entire process at the call site which runs into above issue.
spawn creates a fresh Python process from scratch and passes needed resources to it.
forkserver creates a single-threaded server process which will be used as a template to fork future processes from. As it is single-threaded this fork is safe.

Based on my testing, and the descriptions, choosing either spawn or forkserver should work for you and lead to exitcode 0.

from multiprocessing import Process, set_start_method
set_start_method('spawn')

The regression from Python 3.8 to 3.9 for the fork method is worrying to me however, so I left a comment on the Python bug tracker. Maybe someone will fix this issue but as I said, as far as the documentation is concerned, forking a multithreaded process is just not supported.

answered Apr 26, 2021 at 23:19

xjcl

15.8k8 gold badges90 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Genarito Over a year ago

Cool! Thanks for your answer. I've submitted an issue too: bugs.python.org/issue43944. I'll wait to get an answer from the official community before I close this question

Collectives™ on Stack Overflow

Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related