Race condition in RubyGems/Bundler causes NullPointerException

[eregon]

Noticed by @fniephaus in https://github.com/graalvm/setup-graalvm/actions/runs/5289231919/jobs/9571920888?pr=46#step:6:363
Stacktrace:

Caused by:
<no message> (java.lang.NullPointerException)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary.getHashed(PackedHashStoreLibrary.java:75)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary$LookupPackedEntryNode.getPackedArray(PackedHashStoreLibrary.java:459)
	from org.truffleruby.core.hash.library.PackedHashStoreLibraryFactory$LookupPackedEntryNodeGen.execute(PackedHashStoreLibraryFactory.java:102)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary.lookupOrDefault(PackedHashStoreLibrary.java:165)
	from org.truffleruby.core.hash.library.PackedHashStoreLibraryGen$HashStoreLibraryExports$Cached.lookupOrDefault(PackedHashStoreLibraryGen.java:458)
	from org.truffleruby.core.hash.HashNodes$GetIndexNode.get(HashNodes.java:205)
	from org.truffleruby.core.hash.HashNodesFactory$GetIndexNodeFactory$GetIndexNodeGen.execute(HashNodesFactory.java:708)
	from org.truffleruby.language.RubyCoreMethodRootNode.execute(RubyCoreMethodRootNode.java:58)
/opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/rubygems/specification.rb:845:in `[]'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/rubygems/specification.rb:845:in `stubs_for'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/rubygems_gem_installer.rb:51:in `generate_plugins'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/rubygems_gem_installer.rb:33:in `install'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/source/rubygems.rb:207:in `install'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/installer/gem_installer.rb:54:in `install'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/installer/gem_installer.rb:16:in `install_from_spec'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/installer/parallel_installer.rb:186:in `do_install'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/worker.rb:62:in `apply_func'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/worker.rb:57:in `block in process_queue'
	from <internal:core> core/kernel.rb:412:in `loop'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/worker.rb:54:in `process_queue'
	from /opt/hostedtoolcache/graalvm-jdk-17_linux-x64_bin/17.0.0/x64/graalvm-jdk-17.0.7+8.1/languages/ruby/lib/mri/bundler/worker.rb:91:in `block (2 levels) in create_threads'

Also #3273 where it manifests as:

Index 9 out of bounds for length 9 (java.lang.ArrayIndexOutOfBoundsException)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary.getHashed(PackedHashStoreLibrary.java:77)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary.promoteToBuckets(PackedHashStoreLibrary.java:136)
	from org.truffleruby.core.hash.library.PackedHashStoreLibrary$Set.set(PackedHashStoreLibrary.java:225)
	from org.truffleruby.core.hash.library.PackedHashStoreLibraryGen$HashStoreLibraryExports$Cached.set(PackedHashStoreLibraryGen.java:316)
	from org.truffleruby.core.hash.HashNodes$SetIndexNode.set(HashNodes.java:243)
	from org.truffleruby.core.hash.HashNodesFactory$SetIndexNodeFactory$SetIndexNodeGen.execute(HashNodesFactory.java:1172)
	from org.truffleruby.language.RubyCoreMethodRootNode.execute(RubyCoreMethodRootNode.java:58)
/Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/specification.rb:845:in `[]='
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/specification.rb:845:in `stubs_for'
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/dependency.rb:276:in `matching_specs'
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/dependency.rb:306:in `to_specs'
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/dependency.rb:326:in `to_spec'
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/core_ext/kernel_gem.rb:57:in `gem'
	from <internal:/Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/core_ext/kernel_require.rb>:73:in `require'
	from /Users/runner/.rubies/truffleruby-23.1.0/lib/mri/tempfile.rb:8:in `<top (required)>'
	from <internal:core> core/kernel.rb:234:in `gem_original_require'
	from <internal:/Users/runner/.rubies/truffleruby-23.1.0/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
...

(https://gist.github.com/froydnj/2e5c50134f0e88a3699c992f612a426d)

[eregon]

https://github.com/oracle/truffleruby/blob/bd96fa823c214a04a5586c230e07671fd41bb0de/lib/mri/rubygems/specification.rb#L845
and @@stubs_by_name is a Hash.
Can be solved by thread-safe Hash, and/or upstream.

[notEthan]

I was about to open an issue but I think the error I encountered is the same as this one.

I encountered this on github actions installing a gem yesterday, hasn't happened before or since:
https://github.com/notEthan/jsi/actions/runs/5617018639/job/15220399594
log copied to gist, in case the action's logs expire:
https://gist.github.com/notEthan/8620a2b082980dc310ddceb725eb4800

[flavorjones]

Just saw this twice in a row in the sqlite3-ruby test suite running truffleruby 24.0.0-dev-260d07b7:

• https://github.com/sparklemotion/sqlite3-ruby/actions/runs/6094514047/job/16536443930#step:5:718
• https://github.com/sparklemotion/sqlite3-ruby/actions/runs/6094514047/job/16593664161

but haven't seen it in previous runs (ever).

[rwstauner]

I was attempting to create a small case for reproducing a similar error (that seems to come from CompactHashStoreGen)
and instead recreated this NullPointerException (from PackedHashStoreLibrary) with a small ruby script:

https://gist.github.com/rwstauner/5dd7e3e723a518083fb04ba2fc3c8c99

With the the GFTC dev build at 1f80340c I don't get this every time but I will get it with enough tries.
With the jvm ce build at 7f6bbb30 I get it every time.

[eregon]

@rwstauner Yes it's expected that Hash is not thread-safe yet (and it wasn't before either) on TruffleRuby (and that it throws such errors in that case).
For now it would be best to find the places in RubyGems/Bundler/etc which do cause these concurrent accesses and either use Concurrent::Map instead or a Mutex around all accesses to that Hash.
I plan to make Hash thread-safe but most likely not for this release.

[eregon]

and @@stubs_by_name is a Hash.

For now I worked around this one by using TruffleRuby::ConcurrentMap in 4061462 / #3316
We can't use Concurrent::Map there because it's code in RubyGems, and TruffleRuby::ConcurrentMap seemed easier than a Mutex and anyway this is a temporary solution.