`_scproxy` calls SystemConfiguration functions in a way that can cause deadlocks

[MaximeBelanger]

BPO	31903
Nosy	@ronaldoussoren, @benjaminp, @ned-deily, @miss-islington
PRs	bpo-31903: Release the GIL when calling into SystemConfiguration #4178 [3.6] closes bpo-31903: Release the GIL when calling into SystemConfiguration (GH-4178) #9215 [3.7] closes bpo-31903: Release the GIL when calling into SystemConfiguration (GH-4178) #9216

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-09-11.23:14:03.347>
created_at = <Date 2017-10-30.19:38:53.508>
labels = ['OS-mac', '3.7', 'type-crash']
title = '`_scproxy` calls SystemConfiguration functions in a way that can cause deadlocks'
updated_at = <Date 2018-09-12.15:38:39.919>
user = 'https://bugs.python.org/MaximeBelanger'

bugs.python.org fields:

activity = <Date 2018-09-12.15:38:39.919>
actor = 'miss-islington'
assignee = 'none'
closed = True
closed_date = <Date 2018-09-11.23:14:03.347>
closer = 'benjamin.peterson'
components = ['macOS']
creation = <Date 2017-10-30.19:38:53.508>
creator = 'Maxime Belanger'
dependencies = []
files = []
hgrepos = []
issue_num = 31903
keywords = ['patch']
message_count = 6.0
messages = ['305247', '325090', '325104', '325135', '325140', '325141']
nosy_count = 5.0
nosy_names = ['ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'Maxime Belanger', 'miss-islington']
pr_nums = ['4178', '9215', '9216']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue31903'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[MaximeBelanger]

Through the use of various Python packages (such as pyobjc), it is possible for a deadlock to occur due to how _scproxy.c calls SCDynamicStoreCopyProxies.

In more recent versions of macOS (10.7+), this function can block on acquiring a lock deep inside NSUserPreferences. As pyobjc allows Python-wrapped NSStrings to be stored in CFPreferences, it is thus possible for one thread to hold the CFPreferences lock and block on the GIL while another thread holds the GIL and blocks on the CFPreferences lock.

We've observed this on a significant number of macOS devices before fixing ourselves by wrapping the calls to SCDynamicStoreCopyProxies with Py_BEGIN/END_ALLOW_THREADS.

[benjaminp]

New changeset 4859ba0 by Benjamin Peterson (Max Bélanger) in branch 'master':
closes bpo-31903: Release the GIL when calling into SystemConfiguration (GH-4178)
4859ba0

[ned-deily]

Thanks, Benjamin!

Ronald, do you think this should be backported?

[ronaldoussoren]

Yes it should, the same problem has been present from the start.

[miss-islington]

New changeset 72c34cf by Miss Islington (bot) in branch '3.7':
closes bpo-31903: Release the GIL when calling into SystemConfiguration (GH-4178)
72c34cf

[miss-islington]

New changeset aa12534 by Miss Islington (bot) in branch '3.6':
closes bpo-31903: Release the GIL when calling into SystemConfiguration (GH-4178)
aa12534