Proposal

Summary

The current state of the debuginfo testsuite is best summarized by pnkfelix's comment in 2021:

The on-going breakage of the debuginfo tests (and subsequent failure to catch such breakage in CI) is pretty worrisome, and frustrating as a contributor when all you want to do is run x.py test and let it go, without having to think too hard about "well which test suites are going to run that may be broken and that I have to skip myself by hand."

I would like to propose several changes to stabilize the state of this test suite and make it less of a maintenance and contribution workflow headache. In essence:

• Get rid of implicit debugger discoveries, and require opt-in to debugger discovery.
• Pin debugger versions and environments for debuginfo test suite in CI (as much as feasible).
• Downgrade debuginfo test step to be conditionally-default instead of unconditionally-default for ./x test (no selector form) for local workflows.

Scope

• This proposal focuses only on changes to stabilize the state of the debuginfo test suite.
• Improving the debuginfo test suite UX/DX is not within scope of this proposal. That is covered by Creating a new maintainable debuginfo test suite #936 (where @Walnut356 is doing good work as part of a GSoC project). However, this proposal enables the improvements to be maintainable.

Background

What do debuginfo tests, test?

I'm not sure, actually. My best guess is that the debuginfo test suite is intended to exercise some cases of end-user debugging experience:

• accounting for the quality of debuginfo we produce,
• how well the codegen backend makes use of that,
• if the debuginfo is actually embedded or otherwise emitted in/around the final binary,
• if custom lldb formatters / gdb pretty printers / cdb NatVis files we provide for specific standard library types actually work with the respective debuggers.
• if the debugger can actually handle the commands in the tests (e.g. runnning, stepping, printing variables, jumping, breakpoints, etc.).

To put it succintly, if the end-to-end debuginfo<->debugger pipeline actually works to some extent™ and detect some changes to the final debugger output.

What influences user-facing debugger outputs

The main environment landscape for debuginfo tests:

For Tier 1 targets, currently:

• OS x toolchain: {Linux, Windows MSVC, Windows GNU, macOS}
• Target architecture: {x86_64, i686, aarch64}
• Target pointer width: {32, 64}
• Debuggers: {GDB, Vanilla LLDB, Apple/Swift LLDB, CDB}
• Python for GDB/LLDB ¹

Example combinations when running debuginfo tests on x86_64-pc-windows-msvc host:

Of course, not all combinations are valid. Currently we only test a very small subset of these combinations in rust-lang/rust CI.

Plan to stabilize the debuginfo test suite

The test suite will be run in CI against a pinned environment

• For each of {lldb,cdb,gdb}, the debuginfo test suite will only be run against one exact debugger version each.
• We will try to acquire or build the exact pinned debugger version (viability to be investigated).

Debugger discovery

We will no longer support any kind of implicit debugger discovery, and require users to:

• Either opt-in to debugger discovery explicitly via bootstrap build.{lldb,cdb,gdb} = "discover", or
• Provide paths to the debuggers.

Debuginfo test suite will now be a conditionally-default test step

For non-CI environments:

• tests/debuginfo will be conditionally-default for ./x test only if the user explicitly provided build.{lldb,cdb,gdb} configuration:
  • Either opt-in via explicit build.{lldb,cdb,gdb} = "discover", or
  • Explicitly provide path to the debugger.

For instance, for aarch64-apple this means we would have to either set build.lldb = "discover" or (better) provide explicit lldb path.

We may additionally track what debugger versions we test the debuginfo tests against in CI, and issue warnings if the local discovered/provided debugger reports a version we suspect can cause failures due to debugger version differences.

When no suitable debugger is discovered when requested

We will hard error (and likely print searched locations).

Rationale

• Make ./x test ("just give a reasonably comprehensive set of tests to run") more likely to Actually Work:tm:.
• Reduce the maintenance and contribution headaches stemming from implicit debugger discoveries and having to figure out what debugger versions the test suite is supposed to pass under.
• Make it easier (more often than not, possible) to try to reproduce failures and rebless locally with known exact environment and debugger configurations.
• Actually try to keep up test coverage, instead of maintainers having to //@ ignore-* tests or adding inconsistently a bunch of //@ min-{debugger}-version: xxxs.
• Stop rust-lang/rust CI from randomly breaking because of implicit debugger discovery finding different debugger versions as runner image versions change, and requiring maintainers to investigate how to unblock the tree.

Drawbacks

• We need to explicitly build or acquire specific debuggers and versions in CI.
  • However I contest that we should've beeen doing this in the first place, floating debugger versions is basically going to be continuous issues.
  • This may cause additional CI time (we'll evaluate the impact, generally can't make slowest jobs even slower).
• We will need to semi-regularly update pinned version and triage failing tests on update.
  • Yes, but at least we would know which exact version(s) the test suite worked on and which tests fail upon updating the debugger versions (and other environmental components).
    • And at least runner image updates can't introduce tree breakages from floating debugger versions.
  • It's always an option to ignore failing tests if cause is unclear, which is no worse than the status quo (we already do so).
  • I also contend that if a given pinned debugger is not updated for a long time, maybe we should consider dropping testing for that.

Alternatives

rm -rf tests/debuginfo

• I have considered this unironically. There is a strong argument for this too, this test suite has no seen healthy maintenance since 2019 (more like ever since its inception honestly). Only recently have we started to try to savlage a bit of it because it just keeps giving maintenance/contribution headaches. We have very little expertise on debuginfo and especially the visualizers too.
• However, it does feel bad to not exercise debuginfo at all, because DX (and debugging experience) is a very important property of a mature programming language toolchain. Especially for Tier 1 targets.

Do nothing

• Not really viable, the current state of the debuginfo test suite is annoying and frustrating for both CI maintenance and for local dev.

Relevant discussions and issues

• Tracking Issue for debuginfo testsuite test infra woes rust#134682
• Reconsider debugger discovery for debuginfo test suite rust#148361
• debuginfo test suite needs more documentation rust#148288
• Debuginfo tests don't control for ambient tools rust#126092
• Debuginfo tests are highly sensitive to debugger versions and thus very fragile rust#133111
• rustc failed to pass debuginfo testsuite when using GDB 15 rust#122751
• Compiler selects invalid --lldb-python path rust#69869
• Running ./x test tests/debuginfo triggers recompiles for debugger visualizer script changes rust#133692
• Is anyone running our gdb-based Android debuginfo tests? rust#129139
• lldb tests only run on macos rust#81813
• Debuginfo tests sometimes run with the wrong Python version on MacOS rust#123621
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/debuginfo.20tests.20are.20causing.20problems.20.28again.29/with/459935379
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Plan.20for.20debuginfo.20test.20suites/with/608083288
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Salvaging.20.60tests.2Fdebuginfo.60.20one.20step.20at.20a.20time/with/538732948
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Making.20the.20debuginfo.20test.20suite.20more.20enjoyable/with/490503355
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/.E2.9C.94.20.60tests.2Fdebuginfo.2Fstrings-and-strs.2Ers.60.20failure/with/439497422
• https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/How.20do.20I.20skip.20debuginfo.20tests.3F/with/436300705
• https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/dev-desktop.20debuginfo-lldb.20tests.20failing/with/607320592
• https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/msvc.20update.20causing.20debuginfo.20failures/with/482844112

Fission the debuginfo testing into debuginfo quality vs debugger interop

• Basically split from "quantity of produced debuginfo" versus "how well the end-to-end debug experience is likely going to be for the end user". The former we should strive to always keep passing, the latter is more of a nice-to-work.
• There were some earlier ideas by eddyb (Investigate replacing most of the debugger pretty-printing scripts with traits. rust#65564, who debugs the debuginfo #186) to try to use traits instead of debugger pretty-printing scripts.
• That effort never materialized presumably due to both bandwidth and very non-trivial technical/maintenance difficulties.

Prior arts

How does lldb test itself?

Ref: https://lldb.llvm.org/resources/test.html

• It has both shell tests and scripting API Python tests.
• Shell tests: uses lldb-test (which gets to use "priviledged" way to (de-)serialize internal LLDB data structures); uses FileCheck to help with matching expectations.
• LLDB has a dottest.py Python testing framework

Cool, however that is a lot of Python.²

How does gdb test itself?

Ref: https://www.sourceware.org/gdb/5.1/onlinedocs/gdbint_15.html#SEC116

The GDB testsuite uses the DejaGNU testing framework. DejaGNU is built using Tcl and expect. The tests themselves are calls to various Tcl procs; the framework runs all the procs and summarizes the passes and fails.

So, Tcl scripts. And they also do FileCheck-like matching, example.

• It is interesting how they have specific test infra support for handling/identifying flaky/racy tests.

How does cdb test itself?

(I would love to hear about it.)

Unresolved questions

• Can we try to support ranges of debugger versions?
  • IMHO, not currently. Not without knowledgable maintainers with bandwidth to improve the health of this debuginfo test suite. I'd like us to first focus on making even supporting one debugger version maintainable and not a PITA on work with.
  • I would like us to be explicitly clear on:
    • What debugger(s) and debugger version(s) we actually test in CI, versus
    • What debugger(s) and debugger versions we (and target maintainers) reasonably think will work (sufficiently) well, for the given target, but that we don't test in CI
  • I do not want to claim support for ranges of debugger versions unless we actually test the range. {begin, end} is also not enough, because debugger output can change/break even in minor/point versions.
• Building our own debugger artifacts vs acquiring pre-built artifacts?
  • Not sure yet, we'll have to try either approaches.
  • For pre-built artifacts we can mirror them in our own mirror.
  • If we try to build debuggers in CI, infra team will assess build time impact. If they cause our longest jobs to become even slower that's not really going to work.
  • IINM we should not need to codesign gdb/lldb if we build them in macOS CI, because in the macOS GHA jobs SIP is disabled.
• download-ci-{lldb,gdb}?
  • Not in scope of this MCP, but maybe possible.
  • For macOS, codesigning would be needed for e.g. lldb we build ourselves to run on local dev (unless you disable SIP which is a bad idea).
• UX for ./x test tests/debuginfo but host system can run more than one of {gdb,cdb,lldb}?
  • Unsure yet.
• This does not solve "I don't have access to the test environment needed to reproduce the test failure or rebless locally".
  • ... But at least you would know exactly what you don't have access to...
• How to make these tests less fragile.
  • Out of scope of this proposal, and I don't yet have a good answer for it.

Mentors or Reviewers

Compiler folks.

Process

The main points of the Major Change Process are as follows:

• File an issue describing the proposal.
• A compiler team member who is knowledgeable in the area can second by writing @rustbot second or kickoff a team FCP with @rfcbot fcp $RESOLUTION.
  • Refer to Proposals, Approvals and Stabilization docs for when a second is sufficient, or when a full team FCP is required.
• Once an MCP is seconded, the Final Comment Period begins.
  • Final Comment Period lasts for 10 days after all outstanding concerns are solved.
  • Outstanding concerns will block the Final Comment Period from finishing. Once all concerns are resolved, the 10 day countdown is restarted.
  • If no concerns are raised after 10 days since the resolution of the last outstanding concern, the MCP is considered approved.

You can read more about Major Change Proposals on forge.