Global variables are not accessible from child processes (multiprocessing.Pool)

[agmt]

BPO	39931
Nosy	@ronaldoussoren, @pitrou, @vstinner, @ned-deily, @graingert, @applio, @agmt
Files	test.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-11.06:27:20.206>
created_at = <Date 2020-03-11.09:20:52.512>
labels = ['OS-mac', '3.8', 'type-bug', 'library', 'invalid']
title = 'Global variables are not accessible from child processes (multiprocessing.Pool)'
updated_at = <Date 2020-11-11.06:27:20.205>
user = 'https://github.com/agmt'

bugs.python.org fields:

activity = <Date 2020-11-11.06:27:20.205>
actor = 'josh.r'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-11.06:27:20.206>
closer = 'josh.r'
components = ['Library (Lib)', 'macOS']
creation = <Date 2020-03-11.09:20:52.512>
creator = 'agmt'
dependencies = []
files = ['48968']
hgrepos = []
issue_num = 39931
keywords = []
message_count = 5.0
messages = ['363892', '363901', '363902', '363903', '380595']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'pitrou', 'vstinner', 'ned.deily', 'graingert', 'davin', 'agmt']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39931'
versions = ['Python 3.8']

[agmt]

Attached test works correctly on linux (3.7, 3.8) and mac (only 3.7).
Mac python3.8 falls with exception:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "test.py", line 8, in work
    print(F"Work={arg} args={args}")
NameError: name 'args' is not defined
"""

[graingert]

yeah this is normal. on Python 3.8 mac multiprocessing switched to spawn

you have to use https://docs.python.org/3/library/multiprocessing.shared_memory.html to share content between processes or pass it to be pickled in the args of imap_unordered

https://bugs.python.org/issue33725

[agmt]

Shared memory/pickle cannot send file descriptors

[vstinner]

Shared memory/pickle cannot send file descriptors

FYI Python 3.9 got new socket.send_fds() and socket.recv_fds() functions :-)
https://docs.python.org/dev/library/socket.html#socket.socket.send_fds

[ronaldoussoren]

Can this issue be closed? Multiprocessing seems to work as designed. There is a behaviour change between 3.7 and 3.8, but that's documented in What's New.