who tests the tester? #47606

Activity

1. nikomatsakis

   added

   A-testsuiteArea: The testsuite used to check the correctness of rustcArea: The testsuite used to check the correctness of rustc

   T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.Relevant to the compiler team, which will review and decide on the PR/issue.

   on Jan 20, 2018
2. oli-obk commented on Jan 20, 2018

   

   oli-obk

   on Jan 20, 2018

   Contributor

   More actions

   I have some beef with compiletest in general. Instead of having this unstable (as in nightly) library that contains a somewhat random list of test kinds that all behave subtly different, we should probably write a stable extensible library that is distributed via crates.io, developed like any other library (so it has tests) and uses cargo instead of manually calling rustc and hoping for the best.

   So, the list of issues off the top of my head:

   • nightly only
   • rustc submodules need to use the crates.io version, which breaks with rustc changes
   • using dependencies in tests frequently causes "multiple matching crates" because of old compilation artifacts in the target direcory
   • not extensible with new test kinds (cc @killercup for suggestion tests)

   👍React with 👍2Reacted by Andy Russell and Dylan DPC
3. spastorino commented on Jan 20, 2018

   

   spastorino

   on Jan 20, 2018

   Member

   More actions

   I'd be happy to work on a library like this if we can have a conversation about what is exactly needed :).
4. killercup commented on Jan 20, 2018

   

   killercup

   on Jan 20, 2018

   Contributor

   More actions

   @spastorino I started working on a very simple version of this to test clippy's diagnostics output as well as that its suggestions work as intended (when applied by rustfix). I'll have something up in later today, and will ping you in the clippy PR :)

   In the meantime, feel free to say hello in the #clippy channel on irc.mozilla.org :)
5. nikomatsakis commented on Jan 22, 2018

   

   nikomatsakis

   on Jan 22, 2018

   ContributorAuthor

   More actions

   I have some beef with compiletest in general.

   While I don't disagree, I feel a bit wary here. I'm fine with making a new testing setup, but it feels like the kind of thing that will derail progress for a long time. And I'm not convinced that the compiler's needs (e.g., things like mir-opt tests) will be satisfied by a more generic setup.

   Unless you have a clear candidate of what test runner we should adopt, I'd rather see us move towards simplifying and centralizing things gradually and then at some later point perhaps switch. (Centralizing run-pass, compile-fail, and ui seems like an example of something we can do to simplify things now.)

   👍React with 👍2Reacted by Isaac Woods and Mark Rousskov
6. TimNN

   added

   C-enhancementCategory: An issue proposing an enhancement or a PR with one.Category: An issue proposing an enhancement or a PR with one.

   on Jan 23, 2018
7. emilyalbini

   added 2 commits that reference this issue on Dec 15, 2018

   Rollup merge of rust-lang#56792 - phansch:add_compiletest_testsuite, …

   ...

   1f0a730

   Rollup merge of rust-lang#56841 - phansch:add_various_compiletest_uni…

   ...

   3566812
8. steveklabnik commented on Sep 22, 2019

   

   steveklabnik

   on Sep 22, 2019

   Contributor

   More actions

   Triage; Looks like Pietro has added a bit of testing, but it's "just a start"
9. jyn514 commented on Nov 9, 2022

   

   jyn514

   on Nov 9, 2022

   Member

   More actions

   One way to test compiletest is to add more UI tests that test compiletests' behavior, instead of rustc's. See for example https://github.com/rust-lang/rust/pull/103298/files#diff-6542cbbb7ad12df6cb2daadaf279589c5c1712906cd14222c03132ad5a49d590. I think using that instead of a new framework has several advantages:

   • No need to set up a new testing framework
   • Existing contributors are already familiar with UI tests
   • These double as "integration tests" since they test exactly the property we're interested in, how compiletest runs a UI test from beginning to end

   I think it should be fairly simple to keep adding tests to src/test/ui/compiletest-selftest as necessary; I don't have a set of initial tests in mind though.
10. jyn514

    added

    E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.

    E-mediumCall for participation: Medium difficulty. Experience needed to fix: Intermediate.Call for participation: Medium difficulty. Experience needed to fix: Intermediate.

    T-infraRelevant to the infrastructure team, which will review and decide on the PR/issue.Relevant to the infrastructure team, which will review and decide on the PR/issue.

    on Nov 9, 2022
11. oli-obk commented on Nov 9, 2022

    

    oli-obk

    on Nov 9, 2022

    Contributor

    More actions

    I wrote a new ui test framework that miri uses and I'm working on getting ready for clippy to use. It uses itself to test its own output.
12. jyn514 commented on Nov 9, 2022

    

    jyn514

    on Nov 9, 2022

    Member

    More actions

    Sounds like there is duplicate work between that and #103266
13. Alexendoo commented on Nov 15, 2022

    

    Alexendoo

    on Nov 15, 2022

    Last edited by Alexendoo

    Member

    More actions

    It could well be, mainly I went forward with #103266 to avoid the other duplicate work of adding things to https://github.com/Manishearth/compiletest-rs that the in tree one already has. For example the JSON output after every test failure is pretty rough
14. jieyouxu

    added

    A-compiletestArea: The compiletest test runnerArea: The compiletest test runner

    on May 31, 2024
15. jieyouxu

    added this to compiletest maintenance and improvementson Jun 16, 2024
16. github-project-automation

    moved this to Backlog in compiletest maintenance and improvementson Jun 16, 2024
17. jieyouxu

    added

    E-hardCall for participation: Hard difficulty. Experience needed to fix: A lot.Call for participation: Hard difficulty. Experience needed to fix: A lot.

    E-needs-designThis issue needs exploration and design to see how and if we can fix/implement itThis issue needs exploration and design to see how and if we can fix/implement it

    T-bootstrapRelevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap)

    and removed

    E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.Call for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.

    E-mediumCall for participation: Medium difficulty. Experience needed to fix: Intermediate.Call for participation: Medium difficulty. Experience needed to fix: Intermediate.

    on Oct 17, 2024
18. jieyouxu

    self-assigned this

    on Nov 7, 2024
19. jieyouxu commented on Dec 12, 2025

    

    jieyouxu

    on Dec 12, 2025

    Member

    More actions

    I'm going to close this issue because it's quite dated, though the "who tests the tester" challenge very much is still a problem -- and will continue to be so due to the nature of the things compiletest has to support. Some remarks:

    • We're no longer planning to move compiletest out-of-tree. compiletest's core mission is to support testing of rust-lang/rust things, specifically the compiler, standard library, and rustdoc, and a bit of integration testing with cargo. The upside is because compiletest gets to make simplifying / reasonable assumptions that make sense in the context of rust-lang/rust. The tradeoff is compiletest is no longer a general-purpose tool -- and I would strongly prefer to keep it that way to avoid having to handle additional maintenance burden with dealing with out-of-tree consumers.
    • We do have some light self-test coverage now, for better or worse. Directive parsing and such have in-tool integration tests, some compiletest behaviors have integration tests through ui tests. Though overall, a lot more self-test coverage is what I want compiletest to have. However, compiletest is currently structured in a way that makes more comprehensive self-testing quite difficult, and so my plan is to gradually refactor compiletest to make it more testable.

    While I don't disagree, I feel a bit wary here. I'm fine with making a new testing setup, but it feels like the kind of thing that will derail progress for a long time. And I'm not convinced that the compiler's needs (e.g., things like mir-opt tests) will be satisfied by a more generic setup.

    I am very much in agreement with this assessment, there's a lot of complexity in compiletest, especially in cases where compiletest and the $thing_under_test needs to collaborate to be able to effectively test something.
20. jieyouxu

    closed this as completedon Dec 12, 2025