iHP DRC causes segfaults randomly

[smunaut]

When running DRC using the iHP deck, I'm getting random segfault. This is using 0.29.7 build from source in a nix environment.

I'm attached a test case. However this is random. One run might work 100% fine. The next could crash. And then sometime it reports non-existing DRC error.

The package includes both:

• DRC deck used and the input file and the command line used to start it
• An example of report with non existent DRC errors
• The crash log from a place where it segfaulted.

drc_tst.zip

[klayoutmatthias]

Hmm ... thanks for this report.

However, I can't see the crashes on Ubuntu 24 with the official package from the download page. I also don't see anything suspicious in valgrind.

From the trace, the issue can be anything. A rough guess is a problem with the Ruby integration. Is it safe to assume the build is correctly linked against the same Ruby library that is used at runtime?

I am not familiar with nix. I am plain old distro user. I have not tried builds with nix and honestly that is not on my top priority list.

Is anyone else able to reproduce the issue?

Matthias

[smunaut]

TBH I'm not all that familiar with nix either :/ But it's what's used for all OpenLane2 runs so it's everywhere ...

The issue showed up both in the github actions and on my test VM. They both use the same "nix derivation" (sort of "source package") for klayout but they were built independently on different hardware and showed the same issue.

I also can't reproduce it on my laptop (using the exact same git hash) that runs klayout natively.

[stefanottili]

at the first attempt, it coredumped several times out of 10 times on release build of klayout M1 MacOS Sequoia using homebrew
The next 3x10 run's finished without errors ... but on the 5th attempt I got coredumps again.
Very random ...

ERROR: Signal number: 11
Address: 0x8a87eb3bd698
Program Version: KLayout 0.29.8 (2024-10-20 r22f1778a4)

Backtrace:
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_lay.0.29.8.dylib +0x1e6874 _ZN3lay25enable_signal_handler_guiEb
/usr/lib/system/libsystem_platform.dylib +0x4184 _sigtramp
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x141954 _ZN2db9Instances9do_insertERKNS_8InstanceERN2tl18func_delegate_baseIjEERNS5_ImEE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x141954 _ZN2db9Instances9do_insertERKNS_8InstanceERN2tl18func_delegate_baseIjEERNS5_ImEE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x132c5c _ZN2db17instance_iteratorINS_30TouchingInstanceIteratorTraitsEEppEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x1328c4 _ZN2db17instance_iteratorINS_30TouchingInstanceIteratorTraitsEEppEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x266f54 _ZNK2db22RecursiveShapeIterator10next_shapeEPNS_22RecursiveShapeReceiverE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_db.0.29.8.dylib +0x10a596c _ZNK2db13DeepEdgePairs23insert_into_as_polygonsEPNS_6LayoutEjji
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_rba.0.29.8.dylib +0x99a8 _ZNK3rba22RubyStackTraceProvider11stack_depthEv
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1d429c vm_call_cfunc_with_frame_
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1be354 vm_exec_core
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1bbbd4 rb_vm_exec
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1c8f10 specific_eval
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1d429c vm_call_cfunc_with_frame_
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1be5c8 vm_exec_core
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1bbb24 rb_vm_exec
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1c6fb8 rb_funcallv_scope
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x84ce0 rb_protect
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_rba.0.29.8.dylib +0x26df4 _ZN3rba15RubyInterpreter19remove_exec_handlerEPN3gsi16ExecutionHandlerE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_rba.0.29.8.dylib +0x28458 _ZN3rba15RubyInterpreter19remove_exec_handlerEPN3gsi16ExecutionHandlerE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_gsi.0.29.8.dylib +0x492d4 _ZNK2tl6Recipe11descriptionEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_tl.0.29.8.dylib +0x75220 _ZN2tl10Executable10do_executeEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_lym.0.29.8.dylib +0x29284 _ZN3lym16MacroInterpreter13execute_macroEPKNS_5MacroE
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_lym.0.29.8.dylib +0x36a34 _ZNK3lym5Macro3runEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_lay.0.29.8.dylib +0x6b03c _ZN3lay15ApplicationBase3runEv
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/klayout.app/Contents/MacOS/klayout +0xd904
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_rba.0.29.8.dylib +0x17e4c _ZN3rba15RubyInterpreter10initializeERiPPcPFiS1_S3_E
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1d429c vm_call_cfunc_with_frame_
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1be5c8 vm_exec_core
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x1bbb24 rb_vm_exec
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x841bc rb_ec_exec_node
/opt/homebrew/Cellar/ruby/3.3.5/lib/libruby.3.3.dylib +0x84098 ruby_run_node
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/libklayout_rba.0.29.8.dylib +0x17dd8 _ZN3rba15RubyInterpreter10initializeERiPPcPFiS1_S3_E
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/klayout.app/Contents/MacOS/klayout +0xd2d0
klayout/qt5Brew.bin.macos-Sequoia-release-Rhb33Phbauto/klayout.app/Contents/MacOS/klayout +0xcc34
/usr/lib/dyld +0x6274 start

[stefanottili]

when it occurs, the coredump is triggered by "Rule pSD.d"

[smunaut]

Yes, here when it doesn't crash but reports wrong DRC errors, they are also always in the pSD.d rule.

[klayoutmatthias]

I will try to reproduce the issue with nix. Do you have some basic build instructions for me?

Thanks, Matthias

[Kazzz-S]

Hello folks,

The table below shows another test result set on an Intel Mac using different DMGs (#1871).
Everything seems to be okay in these cases.

Srl.No.	Target OS	DMG file name	Qt5 or Qt6	Ruby	Python	Number of Runs	Number of Crashes
10	Sonoma (14.7)	ST-klayout-0.29.7-macOS-Sonoma-1-qt5MP-RsysPsys.dmg	MacPorts	OS-bundled	OS-bundled	10	0
11	Sonoma (14.7) with MacPorts dev. env.	LW-klayout-0.29.7-macOS-Sonoma-1-qt5MP-Rmp33Pmp312.dmg	MacPorts	MacPorts	MacPorts	10	0
12	Sonoma (14.7) with Homebrew dev. env.	LW-klayout-0.29.7-macOS-Sonoma-1-qt6Brew-Rhb33Phb312.dmg	Homebrew	Homebrew	Homebrew	10	0
13	Sonoma (14.7) with Anaconda3 dev. env.	LW-klayout-0.29.7-macOS-Sonoma-1-qt5Ana3-Rana3Pana3.dmg	Anaconda3	Anaconda3	Anaconda3	10	0
14	Sonoma (14.7)	HW-klayout-0.29.7-macOS-Sonoma-1-qt5MP-RsysPhb311.dmg	MacPorts	OS-bundled	Homebrew	10	0

Kazzz-S

[smunaut]

@klayoutmatthias

So the easiest is to follow the instruction to install OpenLane 2 :
https://openlane2.readthedocs.io/en/latest/getting_started/common/nix_installation/index.html

except instead of using the official repo, use the tt-ihp branch of our fork : https://github.com/TinyTapeout/openlane2/tree/tt-ihp

Then when nix is installed and you're in the OL2 directory just typing nix-shell should get you into the nix environment with the klayout that's exhibiting issues.

[stefanottili]

@Kazzz-S 10 runs might not be enough, I had 2 fails in the first 10, then none in the next 30 before a couple in the next 10.

[smunaut]

Also note that ATM in my latest build I can't get it to segfault anymore ... but it reports random non-existing DRC errors half the time. So this is really weird.

I also tried removing the pSD.d rule and then it works fine.

[smunaut]

As another data point, we're also observing erroneous DRC errors of the pSD.d rule in a github action that doesn't use nix and just uses the ubuntu 24.04 package from the website :

https://github.com/TinyTapeout/tt-gds-action/blob/main/orfs/action.yml#L72

No crashes, just wrong DRC results.

[smunaut]

Valgrind log that shows access to a free'd block when doing db::DeepEdgePairsIterator. Which correlates with the crash log above too.

val.log

[klayoutmatthias]

Thanks for the valgrind log. Most likely it is a modify-while-iterating issue. But it is hard to pinpoint as the code is somewhat complex.

Modification is not direct and the problem is a recomputation of the instance quad tree.. Basically the iterators involved should lock the layout to prevent this effect.

[smunaut]

I haven't confirmed yet, but I'm starting to wonder if this is not a clang vs gcc related.
All the places where is works without issues are built with gcc and both nix and osx are built with clang.

[stefanottili]

Thanks to kazzz build scripts, running on M1 Mac with asan is as easy as adding -d to the build command.
It shows heap-use-after-free errors.

Apple clang version 16.0.0 (clang-1600.0.26.3)
Target: arm64-apple-darwin24.0.0

klayout_crash.log