Tracking issue for moving ui tests to subdirectories

[pnkfelix]

EDIT(@jieyouxu): I personally find just moving the tests to be not super useful. Consider also improving these tests in the process -- see #133895.

---

NOTE: see #73494 (comment) for mentoring instructions.

Looking at https://github.com/rust-lang/rust/tree/master/src/test/ui, github truncates the list to just the first 1000 entries (of 1564 total entries).

This is especially frustrating since github also sorts by name, not by kind, and so its truncation is cutting off whole subdirectories from the view.

TODO:

• We should restructure the test suite so that no one directory has more than 1000 entries
• We should add code to tidy to ensure that future changes do not break the above invariant. (done by Check the number of entries in UI test on tidy #79959)

[lcnr]

The same problem exists for https://github.com/rust-lang/rust/tree/master/src/test/ui/issues with 1,839 omitted entries.

That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

[workingjubilee]

Also see #76268: many of these are run pass tests that do not particularly need to be UI tests.

[Havvy]

Is the compiler team okay with PRs that move tests into subdirectories?

[Nadrieril]

That could be solved by adding using a directory every 10000 issue ids, i.e. ui/issues/10000-19999, ui/issues/20000-29999 and so on

That would technically work, but I'd much rather see some kind of thematic folders. I've been doing that for the bit of the compiler I'm hacking on, and it's been tremendously helpful to have most relevant tests be caught by --test-args pattern. For that same reason I like having the corresponding run-pass/check-pass tests in the same folders too. I don't find the ui/non-ui distinction all that helpful when I'm trying to check if my latest changes broke anything.

[lcnr]

yeah, i agree that it would be better to group tests by what they are testing, but that's a lot of work and probably won't be done in the near future so imo it would be great to start by grouping them by issue id which is fairly easy and then continue sorting them afterwards

[petrochenkov]

A simple machine learning model could assist this task well.
Putting a test into one of N subdirectories is a typical classification task.

My knowledge in this area is outdated by about 10 years, but here's one suggested solution

• Each test (including stderr/stdout) is represented as a bag of tokens + their bigrams and trigrams.
• A classifier is trained on existing subdirectories using e.g. https://www.csie.ntu.edu.tw/~cjlin/liblinear/ (or substitute some modern analogue).
• Tests in src/test/ui root and src/test/ui/issues are classified, high confidence tests are moved, low confidence tests are left for manual classification.

(We just need someone who will be able to implement this quickly, that is probably any ML student.)

[petrochenkov]

cc @Eh2406 on the previous comment.

[Eh2406]

Not an ML student, nor familiar with the compiler. Was I who you meant to ping?
Code coverage results for each test would probably make for great features, if we can get it to work. (Did this test run this line of code.)
I'd guess that a simple pca decomposition of that can bring it to a workable number of dimensions, and a k-means can make some clusters. (Keep it low teck until we prove we need more.) But someone with NLP experience may know more about what the best tools are these days.

[petrochenkov]

(@Eh2406 I just remember you've been talking about being familiar with statistics, not suggesting you to actually implement this.)

[matklad]

Seems like there's a bunch of discussion here and on #79718. I feel rather strongly about this topic, but I also don't have a large stake here, as I don't work with rust-lang/rust test suite on a daily basis.

So I've tried creating a summary document here:

https://hackmd.io/@matklad/Hypu-hosv/edit