make revisions in the test harness show up as independent tests

[nikomatsakis]

So @spastorino recently hacked up a test to include two revisions, one of which enables NLL, by adding:

// revisions: lexical nll
#![cfg_attr(nll, feature(nll))]

This generates the following output:

running 1 test
test [run-pass] run-pass/generator/yield-subtype.rs ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2883 filtered out

at which point the question is -- did both of those revisions really run successfully?

Currently, if a test has N revisions, we run them all as part of its "main line". It'd be more reassuring if we made the revision part of the test name, so that you get output like:

running 2 tests
test [run-pass] run-pass/generator/yield-subtype.rs#nonlexical ... ok
test [run-pass] run-pass/generator/yield-subtype.rs#nll ... ok

The "early properties" of a test (which are gathered while we are assembling the list of tests) already contains the list of revisions:

rust/src/tools/compiletest/src/header.rs

Line 29 in 5965b79

pub revisions: Vec<String>,

Therefore, it should be a relatively simple thing to extend the make_test function to return not one test, but potentially many tests (one for each revision):

rust/src/tools/compiletest/src/main.rs

Line 600 in 5965b79

pub fn make_test(config: &Config, testpaths: &TestPaths) -> test::TestDescAndFn {

We would of course have to modify the test description to include which revision to run, and modify the run function, since that is the one that currently iterates over the list of revisions:

rust/src/tools/compiletest/src/runtest.rs

Lines 165 to 178 in 5965b79

	if base_props.revisions.is_empty() {
	base_cx.run_revision()
	} else {
	for revision in &base_props.revisions {
	let revision_props = TestProps::from_file(&testpaths.file, Some(revision), &config);
	let rev_cx = TestCx {
	config: &config,
	props: &revision_props,
	testpaths,
	revision: Some(revision),
	};
	rev_cx.run_revision();
	}
	}

But this shouldn't be a big change, I wouldn't think.

[nikomatsakis]

Tagging as E-mentor. I gave a few pointers in the description above, but of course happy to go into more detail.

[totorigolo]

Can I take this one?

[spastorino]

@totorigolo I'd say yes, go ahead. In any case check with @nikomatsakis. This #47605 is a very close attempt to implement the thing. Some tests were failing and I gave up because I didn't have any time. It's probably very close, I couldn't even check what the error is about, hehe.

[totorigolo]

Can you explain which tests were failing? I applied your changes and everything seems to be working.

[spastorino]

Run RUST_BACKTRACE=full ./x.py test --incremental --stage 1 src/test/{ui,run-pass,compile-fail}

[totorigolo]

I ran RUST_BACKTRACE=full ./x.py test --incremental --stage 1 src/test/{ui,run-pass} two times: with my branch up to date with 'origin/master' (6c15dff) and with your changes. The same tests failed in both cases:

• [ui] ui/explain.rs
• [ui] ui/lint/unused_parens_json_suggestion.rs
• [ui] ui/lint/use_suggestion_json.rs
• [run-pass] run-pass/out-of-stack.rs
• [run-pass] run-pass/stack-probes-lto.rs
• [run-pass] run-pass/stack-probes.rs

I don't understand why they fail, but they all seem unrelated to this issue.

Here is the result of rustc --version --verbose:

rustc 1.25.0-dev
binary: rustc
commit-hash: unknown
commit-date: unknown
host: x86_64-unknown-linux-gnu
release: 1.25.0-dev
LLVM version: 4.0

[spastorino]

I remember seeing failures in CI, hmmm. Could it be that this PR is actually ok?.
@nikomatsakis should I open a new PR with the same changes against master and wait for the CI to run?.

[nikomatsakis]

@spastorino worth a try =)