Skip to content

Tracking issue for moving ui tests to subdirectories #73494

New issue
New issue
Closed
Closed
Tracking issue for moving ui tests to subdirectories#73494
Labels
A-testsuiteArea: The testsuite used to check the correctness of rustcC-cleanupCategory: PRs that clean code up or issues documenting cleanup.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCE-easyCall for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue.E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.E-tediousCall for participation: An issue involves lots of work and is better handled as many small tasks.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

Description

@pnkfelix
pnkfelix
opened on Jun 19, 2020

EDIT(@jieyouxu): I personally find just moving the tests to be not super useful. Consider also improving these tests in the process -- see #133895.

NOTE: see #73494 (comment) for mentoring instructions.

Looking at https://github.com/rust-lang/rust/tree/master/src/test/ui, github truncates the list to just the first 1000 entries (of 1564 total entries).

This is especially frustrating since github also sorts by name, not by kind, and so its truncation is cutting off whole subdirectories from the view.

TODO:

Activity

  1. Imagepnkfelix
    added
    A-testsuiteArea: The testsuite used to check the correctness of rustc
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    C-cleanupCategory: PRs that clean code up or issues documenting cleanup.
    on Jun 19, 2020

  2. lcnr commented on Jun 19, 2020

    @lcnr
    lcnr
    on Jun 19, 2020
    Contributor

    The same problem exists for https://github.com/rust-lang/rust/tree/master/src/test/ui/issues with 1,839 omitted entries.

    That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

  3. workingjubilee commented on Oct 19, 2020

    @workingjubilee
    workingjubilee
    on Oct 19, 2020
    Member

    Also see #76268: many of these are run pass tests that do not particularly need to be UI tests.

  4. Havvy commented on Nov 22, 2020

    @Havvy
    Havvy
    on Nov 22, 2020
    Contributor

    Is the compiler team okay with PRs that move tests into subdirectories?

  5. Nadrieril commented on Nov 25, 2020

    @Nadrieril
    Nadrieril
    on Nov 25, 2020
    Member

    That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

    That would technically work, but I'd much rather see some kind of thematic folders. I've been doing that for the bit of the compiler I'm hacking on, and it's been tremendously helpful to have most relevant tests be caught by --test-args pattern. For that same reason I like having the corresponding run-pass/check-pass tests in the same folders too. I don't find the ui/non-ui distinction all that helpful when I'm trying to check if my latest changes broke anything.

  6. lcnr commented on Nov 25, 2020

    @lcnr
    lcnr
    on Nov 25, 2020
    Contributor

    yeah, i agree that it would be better to group tests by what they are testing, but that's a lot of work and probably won't be done in the near future so imo it would be great to start by grouping them by issue id which is fairly easy and then continue sorting them afterwards

  7. Imagebors
    added a commit that references this issue on Dec 7, 2020

    Auto merge of rust-lang#79776 - petrochenkov:mvtest, r=Mark-Simulacrum

    f00ace9

  8. petrochenkov commented on Dec 7, 2020

    @petrochenkov
    petrochenkov
    on Dec 7, 2020
    Contributor

    A simple machine learning model could assist this task well.
    Putting a test into one of N subdirectories is a typical classification task.

    My knowledge in this area is outdated by about 10 years, but here's one suggested solution

    • Each test (including stderr/stdout) is represented as a bag of tokens + their bigrams and trigrams.
    • A classifier is trained on existing subdirectories using e.g. https://www.csie.ntu.edu.tw/~cjlin/liblinear/ (or substitute some modern analogue).
    • Tests in src/test/ui root and src/test/ui/issues are classified, high confidence tests are moved, low confidence tests are left for manual classification.

    (We just need someone who will be able to implement this quickly, that is probably any ML student.)

  9. petrochenkov commented on Dec 7, 2020

    @petrochenkov
    petrochenkov
    on Dec 7, 2020
    Contributor

    cc @Eh2406 on the previous comment.

  10. Eh2406 commented on Dec 7, 2020

    @Eh2406
    Eh2406
    on Dec 7, 2020
    Contributor

    Not an ML student, nor familiar with the compiler. Was I who you meant to ping?
    Code coverage results for each test would probably make for great features, if we can get it to work. (Did this test run this line of code.)
    I'd guess that a simple pca decomposition of that can bring it to a workable number of dimensions, and a k-means can make some clusters. (Keep it low teck until we prove we need more.) But someone with NLP experience may know more about what the best tools are these days.

  11. petrochenkov commented on Dec 7, 2020

    @petrochenkov
    petrochenkov
    on Dec 7, 2020
    Contributor

    (@Eh2406 I just remember you've been talking about being familiar with statistics, not suggesting you to actually implement this.)

  12. matklad commented on Dec 7, 2020

    @matklad
    matklad
    on Dec 7, 2020
    Contributor

    Seems like there's a bunch of discussion here and on #79718. I feel rather strongly about this topic, but I also don't have a large stake here, as I don't work with rust-lang/rust test suite on a daily basis.

    So I've tried creating a summary document here:

    https://hackmd.io/@matklad/Hypu-hosv/edit

  13. ImageDylan-DPC-zz
    added a commit that references this issue on Dec 14, 2020

    Rollup merge of rust-lang#79959 - JohnTitor:tidy-entries, r=petrochenkov

    c3cf83b

  14. 49 remaining items

  15. petrochenkov commented on Jan 30, 2024

    @petrochenkov
    petrochenkov
    on Jan 30, 2024
    Contributor

    and to reduce the number of files in the directory

    I don't think simply reducing this number is a goal.

    The classifui tool is working pretty well, and @c410-f3r has already moved hundreds of tests to "semantically appropriate" subdirectories using it.
    This is just some work that needs to happen in the background, not even intensively, just consistently.

  16. Imagerust-timer
    added a commit that references this issue on Jan 30, 2024

    Unrolled build for rust-lang#120439

    9c695b5

  17. c410-f3r commented on Jan 31, 2024

    @c410-f3r
    c410-f3r
    on Jan 31, 2024
    Contributor

    Time to get back to this topic. Hope it ends this year 🤞

  18. Imagejieyouxu
    added
    E-tediousCall for participation: An issue involves lots of work and is better handled as many small tasks.
    on Oct 10, 2024

  19. jieyouxu commented on Mar 12, 2025

    @jieyouxu
    jieyouxu
    on Mar 12, 2025
    Member

    Triage: I personally find just moving the tests to be not super useful. Consider also improving these tests in the process -- see #133895.

  20. Imagejieyouxu
    added a commit that references this issue on Mar 16, 2025

    Rollup merge of rust-lang#138471 - spencer3035:move-ui-test-1ofn, r=j…

    7c1555c
  21. Imagerust-timer
    added a commit that references this issue on Mar 16, 2025

    Unrolled build for rust-lang#138471

    cd54e1c
  22. Imagematthiaskrgr
    added a commit that references this issue on Mar 22, 2025

    Rollup merge of rust-lang#138639 - spencer3035:clean-ui-tests-2-of-n,…

    f01d096
  23. Imagerust-timer
    added a commit that references this issue on Mar 22, 2025

    Unrolled build for rust-lang#138639

    a7d49a0
  24. Imagematthiaskrgr
    added a commit that references this issue on Apr 16, 2025

    Rollup merge of rust-lang#139889 - spencer3035:clean-ui-tests-3-of-n,…

    afb60d3
  25. Imagerust-timer
    added a commit that references this issue on Apr 17, 2025

    Unrolled build for rust-lang#139889

    3ee2246
  26. ImageChrisDenton
    added a commit that references this issue on Apr 19, 2025

    Rollup merge of rust-lang#139995 - spencer3035:clean-ui-tests-4-of-n,…

    f0a0efd
  27. Imagerust-timer
    added a commit that references this issue on Apr 20, 2025

    Unrolled build for rust-lang#139995

    394c542

  28. jieyouxu commented on Jul 19, 2025

    @jieyouxu
    jieyouxu
    on Jul 19, 2025
    Member

    Triage: I'm going to close this issue re. moving tests out of tests/ui/ (top-level) as completed as of #143908. Now, the top-level tests/ui/ folder should only have sub-directories and not individual tests for organization.

    We'll track further subdirectory adjustments, and breaking up tests/ui/issues/ in #133895 instead.

  29. Imagejieyouxu
    closed this as completedon Jul 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-testsuiteArea: The testsuite used to check the correctness of rustcC-cleanupCategory: PRs that clean code up or issues documenting cleanup.C-tracking-issueCategory: An issue tracking the progress of sth. like the implementation of an RFCE-easyCall for participation: Easy difficulty. Experience needed to fix: Not much. Good first issue.E-mentorCall for participation: This issue has a mentor. Use #t-compiler/help on Zulip for discussion.E-tediousCall for participation: An issue involves lots of work and is better handled as many small tasks.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions