Python 3.9 compatibility

[jcrist]

Opening this issue to track Python 3.9 compatibility issues.

[jcrist]

Currently distributed fails to start worker processes on Python 3.9 with a local cluster (at least on osx). I see

In [3]: client = Client()
Task exception was never retrieved
future: <Task finished name='Task-396' coro=<_wrap_awaitable() done, defined at /opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py:678> exception=ImportError("cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)")>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/asyncio/tasks.py", line 685, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/core.py", line 305, in _
    await self.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 295, in start
    response = await self.instantiate()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 378, in instantiate
    result = await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/nanny.py", line 575, in start
    await self.process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 34, in _call_and_set_future
    res = func(*args, **kwargs)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/site-packages/distributed/process.py", line 202, in _start
    process.start()
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/context.py", line 283, in _Popen
    from .popen_spawn_posix import Popen
ImportError: cannot import name 'Popen' from partially initialized module 'multiprocessing.popen_spawn_posix' (most likely due to a circular import) (/opt/miniconda3/envs/prefect39/lib/python3.9/multiprocessing/popen_spawn_posix.py)

[zanieb]

I get the same error in the python3.9 docker image (Debain 10 - Linux version 4.19.76) so I presume this is not limited to OSX

[mrocklin]

cc @itamarst this seems like something that might be up your alley

[itamarst]

Always impressed at your nerd sniping skills 😁

[mrocklin]

I think that my main value to society today is indexing people by technical interest

[mrocklin]

Also, thank you for taking a look, I appreciate it

(I'm interpretting your 😁 as enthusiasm)

[itamarst]

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

No idea what's going on yet, but I suspect it's a bug in Python.

[mrocklin]

Thank you for investigating this @itamarst

[mrocklin]

Short term, would a solution be to add that import to the appropriate file within distributed?

[itamarst]

I would guess so, yes (on POSIX platforms, anyway).

[lingfeiwang]

And a similar command line workaround:

dask-worker --preload-nanny multiprocessing.popen_spawn_posix

A workaround:

>>> import multiprocessing.popen_spawn_posix
>>> from distributed import Client
>>> Client()
<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB>

[mrocklin]

Would you be interested in opening a PR Itamar? Nice solution Lingfei Wang!

…

On Wed, Nov 4, 2020 at 4:30 PM Lingfei Wang ***@***.***> wrote: And a similar command line workaround: dask-worker --preload-nanny multiprocessing.popen_spawn_posix A workaround: >>> import multiprocessing.popen_spawn_posix>>> from distributed import Client>>> Client()<Client: 'tcp://127.0.0.1:41565' processes=4 threads=4, memory=8.26 GB> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTBDSD6NGWPZZ7G44SDSOHW2TANCNFSM4SRDXDTQ> .