Windows hangs in call to CRT setlocale()

[jkloth]

BPO	26624
Nosy	@terryjreedy, @pfmoore, @vstinner, @tjguk, @jkloth, @zware, @eryksun, @zooba, @AraHaan
Files	dd_vs_community_20160702172149.log

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-13.03:12:48.408>
created_at = <Date 2016-03-23.15:44:31.063>
labels = ['interpreter-core', 'OS-windows']
title = 'Windows hangs in call to CRT setlocale()'
updated_at = <Date 2016-08-07.21:03:51.265>
user = 'https://github.com/jkloth'

bugs.python.org fields:

activity = <Date 2016-08-07.21:03:51.265>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-13.03:12:48.408>
closer = 'python-dev'
components = ['Interpreter Core', 'Windows']
creation = <Date 2016-03-23.15:44:31.063>
creator = 'jkloth'
dependencies = []
files = ['43615']
hgrepos = []
issue_num = 26624
keywords = []
message_count = 34.0
messages = ['262276', '262277', '262278', '262279', '262285', '262287', '262360', '262361', '262364', '262381', '262462', '262463', '262484', '262603', '262611', '262630', '262665', '262669', '262672', '262674', '263286', '263290', '263293', '263299', '263300', '269741', '269742', '269743', '270694', '270695', '270699', '270701', '270749', '272133']
nosy_count = 10.0
nosy_names = ['terry.reedy', 'paul.moore', 'vstinner', 'tim.golden', 'jkloth', 'python-dev', 'zach.ware', 'eryksun', 'steve.dower', 'Decorater']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue26624'
versions = ['Python 3.3', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[jkloth]

My Windows BuildBot (http://buildbot.python.org/all/buildslaves/kloth-win64) is hanging in calls to the CRT function setlocale() as determined by attaching to the hung test process in Visual Studio.

This has been happening occasionally (every tenth+ build) for quite some time, but not to the frequency as of late (every fourth build or so).

I would debug further, however my Visual Studio debugging-fu is not up to this challenge it seems.

Pertinent details of the buildbot:

• Windows 7 SP1 64-bit on a quad-core processor
• Running buildbot as a service
• Tests run using '-j4'
• SSD used for storage

Order of the tests do not seem to matter as re-running the test suite using the random seed that hung will pass when run from a console on the buildbot.

Locale-using functions in the CRT have internally locking which is what I believe is the cause of the deadlock, but cannot determine exactly as this is where my debugging knowledge runs out.

Help to diagnose further is greatly appreciated!

[zooba]

Have you actually seen this occurring back to 3.3? I'd expect something like this to either be pre-3.5 or post-3.5, but not both.

[jkloth]

Oh, yes. It even occasionally happens on 2.7. The oldest occurrence I can dig up is May/June of 2013 (for 2.7).

[vstinner]

The bug occurs in test_strptime and test__locale according to Jeremy.

• Tests run using '-j4'

With this option, each test file is run in a new fresh process, so it's more likely a race condition which depends on the system load (on exact timing) than the interaction with other tests running in parallel.

[vstinner]

Random links from Google:

• http://stackoverflow.com/questions/35572792/setlocale-stuck-on-windows "Sometimes it works but sometimes it never returns from it. I can not identify the reason. I use Visual Studio 2015 and Windows 7."

• https://bugs.chromium.org/p/webrtc/issues/detail?id=5507 " Summary: Dr Memory errors in MSVCP140D.dll when switching to VS2015"

[jkloth]

From the UCRT sources:

// Deadlock Avoidance:  When a new thread is created in the process, we
// create a new PTD for the thread.  The PTD initialization function is
// called under the loader lock.  This initialization function will also
// acquire the locale lock in order to acquire a reference to the current
// global locale for the new thread.
//
// Some of the locale APIs are not available on all supported target OSes.
// We dynamically obtain these libraries via LoadLibrary/GetProcAddress.
// We must ensure that no call to LoadLibrary is made while we hold the
// locale lock, lest we deadlock due to lock order inversion between the
// loader lock and the locale lock.

[zooba]

If the comment is in the sources, presumably it's also handled correctly. But it does seem like there's potential here for simultaneous thread creation and first-time calls into OS locale functions to cause a deadlock.

I'll email the CRT team and see what they know about this.

[vstinner]

I'll email the CRT team and see what they know about this.

Thanks. It helps to have a Microsoft employee working on CPython :-)

[zooba]

It's been fixed, just figuring out when the update will be available, or how to get it if it already is.

Jeremy - how up to date is your build machine? It's likely this fix will come through Windows Update for most users, but since you're running debug builds I think you'll need the latest Visual Studio updates to get it.

[jkloth]

The lastest set of updates were installed on 1/28. Visual Studio does have Update 1. I'm doing another round of updates now, although nothing related to VC 2015 or Visual Studio, however.

[vstinner]

The issue bpo-25289 has marked as a duplicate of this issue.

[vstinner]

It's been fixed, just figuring out when the update will be available, or how to get it if it already is.

Can you elaborate? What was the bug? How was it fixed? Is it a bug in Windows, in Visual Studio or in the C library?

[zooba]

Locking inversion I think, I don't have the bug details handy.

The fix went out over Windows Update recently, but that's only for the release version of the runtime. The fixed debug version should have been installed with VS 2015 Update 1.

However, since it's still occurring on Jeremy's machine, it may not actually be the same bug. Checking the file version of ucrtbased.dll will help, but I can't verify the exact certain until Monday.