mono_threads_state_poll_with_info crashes with error: Cannot transition thread from STATE_BLOCKING with STATE_POLL

[praeclarum]

I ran into this error today while running on the latest macOS (10.15.2). I continually run into this error and wonder if anyone know what causes it and whether there is a way to recover. This error makes it impossible to debug since it just keeps crashing with these native exceptions.

2020-01-22 15:17:08.511 iCircuit2[1064:17218] error: Cannot transition thread 0x70000baae000 from STATE_BLOCKING with STATE_POLL

=================================================================
	Native Crash Reporting
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

=================================================================
	Native stacktrace:
=================================================================
	0x10e8c88f8 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_dump_native_crash_info
	0x10e8bc115 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_handle_native_crash
	0x10e8c7e6b - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : sigabrt_signal_handler
	0x7fff7074942d - /usr/lib/system/libsystem_platform.dylib : _sigtramp
	0x70000baab008 - Unknown
	0x7fff7061ea1c - /usr/lib/system/libsystem_c.dylib : abort
	0x10e7dbc60 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : _ZL12log_callbackPKcS0_S0_iPv
	0x10eb06e35 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : monoeg_g_logv_nofree
	0x10eb06ef2 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : monoeg_g_log
	0x10eaf7546 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_threads_transition_state_poll
	0x10eaf526a - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_threads_state_poll_with_info
	0x160efd623 - Unknown
	0x11c323db3 - Unknown
	0x15287470d - Unknown
	0x147c7eda7 - Unknown
	0x147c7e0aa - Unknown
	0x10e8d3167 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_jit_runtime_invoke
	0x10ea0ec47 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_runtime_invoke_checked
	0x10ea1358e - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : mono_runtime_invoke
	0x10e7e5a62 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : xamarin_invoke_trampoline
	0x10e7e6f5b - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : xamarin_arch_trampoline
	0x10e7e8162 - /Users/fak/Dropbox/Projects/Circuit2/CircuitMac/bin/Debug/iCircuit2.app/Contents/MacOS/iCircuit2 : xamarin_x86_64_common_trampoline
	0x7fff450e3a78 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNRenderer _update:]
	0x7fff450e6482 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNRenderer _drawSceneWithNewRenderer:]
	0x7fff450e6a41 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNRenderer _drawScene:]
	0x7fff450e6f46 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNRenderer _drawAtTime:]
	0x7fff450d3fc3 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNView _drawAtTime:WithContext:]
	0x7fff450d38ee - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : -[SCNView SCN_displayLinkCallback:]
	0x7fff4506e25c - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : __69-[NSObject(SCN_DisplayLinkExtensions) SCN_setupDisplayLinkWithQueue:]_block_invoke
	0x7fff45160ad8 - /System/Library/Frameworks/SceneKit.framework/Versions/A/SceneKit : __36-[SCNDisplayLink _callbackWithTime:]_block_invoke.13

Steps to Reproduce

1. Use SceneKit
2. Implement code in the scene renderer update function

Expected Behavior

No crash

Actual Behavior

Above mono error.

Environment

=== Visual Studio Community 2019 for Mac ===

Version 8.4.2 (build 59)
Installation UUID: 4ba352e1-73e8-4dab-a9db-28012ca05af8
	GTK+ 2.24.23 (Raleigh theme)
	Xamarin.Mac 5.16.1.25 (issue-7441-d16-3-vsmac / 881172e73)

	Package version: 606000155

=== Mono Framework MDK ===

Runtime:
	Mono 6.6.0.155 (2019-08/296a9afdb24) (64-bit)
	Package version: 606000155

=== Roslyn (Language Service) ===

3.4.0-beta4-19562-05+ff930dec4565e2bc424ad3bf3e22ecb20542c87d

=== NuGet ===

Version: 5.3.0.6192

=== .NET Core SDK ===

SDK: /usr/local/share/dotnet/sdk/3.1.101/Sdks
SDK Versions:
	3.1.101
	3.0.100
	3.0.100-preview5-011568
	3.0.100-preview-009812
	2.2.203
	2.2.103
	2.1.302
	2.1.300-preview2-008530
	2.0.3
MSBuild SDKs: /Library/Frameworks/Mono.framework/Versions/6.6.0/lib/mono/msbuild/Current/bin/Sdks

=== .NET Core Runtime ===

Runtime: /usr/local/share/dotnet/dotnet
Runtime Versions:
	3.1.1
	3.0.0
	3.0.0-preview5-27626-15
	3.0.0-preview-27122-01
	2.2.4
	2.2.1
	2.1.15
	2.1.13
	2.1.2
	2.1.0-preview2-26406-04
	2.0.3

=== Xamarin.Profiler ===

Version: 1.6.12.26
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Updater ===

Version: 11

=== Apple Developer Tools ===

Xcode 11.3.1 (15715)
Build 11C504

=== Xamarin Designer ===

Version: 16.4.0.478
Hash: 95f0ab363
Branch: remotes/origin/d16-4
Build date: 2020-01-08 23:59:46 UTC

=== Xamarin.Mac ===

Version: 6.10.0.17 (Visual Studio Community)
Hash: 5f802ef53
Branch: xcode11.3
Build date: 2020-01-07 11:53:06-0500

=== Xamarin.iOS ===

Version: 13.10.0.17 (Visual Studio Community)
Hash: 5f802ef53
Branch: xcode11.3
Build date: 2020-01-07 11:53:06-0500

=== Xamarin.Android ===

Version: 10.1.3.7 (Visual Studio Community)
Commit: xamarin-android/d16-4/d66aed0
Android SDK: /Users/fak/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.4 (API level 19)
		7.0 (API level 24)
		7.1 (API level 25)
		8.1 (API level 27)

SDK Tools Version: 26.1.1
SDK Platform Tools Version: 26.0.2
SDK Build Tools Version: 26.0.3

Build Information: 
Mono: fd9f379
Java.Interop: xamarin/java.interop/d16-4@c4e569f
ProGuard: xamarin/proguard/master@905836d
SQLite: xamarin/sqlite/3.28.0@46204c4
Xamarin.Android Tools: xamarin/xamarin-android-tools/master@9f4ed4b

=== Microsoft Mobile OpenJDK ===

Java SDK: /Users/fak/Library/Developer/Xamarin/jdk/microsoft_dist_openjdk_8.0.25
1.8.0-25
Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Android SDK Manager ===

Version: 16.4.0.9
Hash: 3f7256f
Branch: remotes/origin/d16-4
Build date: 2020-01-14 22:19:04 UTC

=== Android Device Manager ===

Version: 16.4.0.30
Hash: f9172e2
Branch: remotes/origin/d16-4
Build date: 2020-01-14 22:19:24 UTC

=== Xamarin Inspector ===

Version: 1.4.3
Hash: db27525
Branch: 1.4-release
Build date: Mon, 09 Jul 2018 21:20:18 GMT
Client compatibility: 1

=== Build Information ===

Release ID: 804020059
Git revision: eb6fcdd83a227678e487aa733df3c8745f54fafc
Build date: 2020-01-17 12:12:02+00
Build branch: release-8.4
Xamarin extensions: ec32c90978c94f03d72f507b01f5aee70952ce87

=== Operating System ===

Mac OS X 10.15.2
Darwin 19.2.0 Darwin Kernel Version 19.2.0
    Sat Nov  9 03:47:04 PST 2019
    root:xnu-6153.61.1~20/RELEASE_X86_64 x86_64

=== Enabled user installed extensions ===

MSBuild Editor 2.3.1

Build Logs

Build Log.txt

[chamons]

Possibly related to one of these?

mono/mono#10800
mono/mono#7981
mono/mono#6921

@praeclarum - Can you reproduce this at all?

@marek-safar - Thoughts?

[praeclarum]

@chamons It's happened 3 times to me now. It's always in that callback from SceneKit - so it's at least somewhat reproducible. I don't have any strict steps yet though, will keep working on it.

I should also note that I didn't start getting this until upgrading macOS to 10.15.2, Xcode to 10.3 and whatever Xamarin Stable is right now - it has been stable for over a year prior. Unfortunately, I made all those changes at once so am not sure which one is the culprit.

[praeclarum]

@chamons It's becoming more and more common - still no repro steps other than use the app for awhile :-(

[marek-safar]

@lambdageek do we have any better way to debug this?

[praeclarum]

@chamons is there any way to get symbols for those mono methods? I tried to make an AOT build but still didn't get symbols.

Also, this may anecdotal at the moment, the crash has only happened with the debugger attached. I had 1 instance run all night without a crash.

[praeclarum]

Ah, it turns out that it's an assert triggering:

socket_transport_send Cannot transition thread 0x700009b66000 from STATE_BLOCKING with DO_BLOCKING
* Assertion at /Users/builder/jenkins/workspace/archive-mono/2019-08/mac/release/mono/metadata/threads.c:6455, condition `this_thread_controls == thread_given_control' not met

https://github.com/mono/mono/blob/fd9f379d294e8208a63dd3f63d104057180c9789/mono/metadata/threads.c#L6446-L6456

static gboolean
mono_threads_summarize_execute_internal (MonoContext *ctx, gchar **out, MonoStackHash *hashes, gboolean silent, gchar *working_mem, size_t provided_size, gboolean this_thread_controls)
{
	static SummarizerGlobalState state;

	int current_idx;
	MonoNativeThreadId current = mono_native_thread_id_get ();
	gboolean thread_given_control = summarizer_state_init (&state, current, &current_idx);

	g_assert (this_thread_controls == thread_given_control);

[lambdageek]

Not a super great way to debug, but you can try:

1. Get monobt.py; save it in ~/monobt/monobt.py and edit ~/.lldbinit to add

      command script import ~/monobt/monobt.py
   

2. Pass MONO_DEBUG=suspend-on-sigsegv to your app. For the SceneKit sample in VS, I do this from the command line

   MONO_DEBUG=suspend-on-sigsegv MONOMAC_LOGDIR=~/Projects/helloworld/ open helloworld.app

3. Wait for the app to crash. It will print Received SIGABRT, suspending... after it crashes. to stdout.log (in the log dir)
4. start lldb and do attach nnnnn where nnnnn is the process id of the app
5. run the lldb command monobt to get a stack trace that includes managed frames.

When you're done, kill from lldb to close the app. (detaching from it is not useful, it's stuck in a busy loop in a signal handler)

---

I guess what I expect to see is that we're in some kind of wrapper calling from managed code to native code. Specifically we switched to GC Safe mode and we're about to call the actual native function, but now we called the GC polling function for some reason.

---

You can probably get the crash to happen more often if you start a background thread that just does something like this:

  while (true) {
    GC.Collect ();
    Thread.Sleep (200);
  }

[praeclarum]

@lambdageek Some good and bad news:

Your GC.Collect certainly does speed up the bug. It's basically 100% repro now (still can take a few seconds of playing though).

It seems to only repro with the debugger attached. :-( Without the debugger, it seems to run fine even with your GC.Collect.

So it's good that it doesn't repro outside the dev environment, but bad because I can't use the debugger.

[chamons]

We have a private sample and @lambdageek is going to take a look, so no more need-info.