Support for "related code" in tests

Connor Peet (@connor4312)

Connor Peet (@connor4312) Is there any consideration to how I would present one or more tests as being related to a code item like a function?

For instance, in Pester you can tag tests with anything. If I allowed a special tag that let you say a test was assigned to a certain function named Invoke-MyFunction, I'd want to expose a "play" button right next to the Invoke-MyFunction definition to run all tests "tagged" to that function. In this case though I don't want it to be tracked with a separate ID, I want it to invoke those individual tests as part of the tag "rollup" which might be scattered across different files. Also tricky is how you would present a failure to the user in this case.

Ideally this would work with AutoRun as well whenever I change the function.

At any rate this is definitely a "nice to have" as opposed to a "must have" and could come at a later iteration, just a thought that would make testing really simple for people and not have to lose the context of where they are directly working on the application. I was inspired by your markdown test adapter example.

With the existing implementation it could just be offered under a "Function Test Rollup" header or something and aggregate the results together somehow, but when run it wouldn't update the "referenced" tests necessarily (though I guess I could do this within the context of the extension) so its fragmented and not ideal.

Originally posted by Justin Grote (@JustinGrote) in #107467 (comment)

This is applicable to Go. go test recognizes three types of test functions: TestXxx, BenchmarkXxx, and ExampleXxx. Go has an established naming convention for associating examples with what they are an example of. If you follow this naming convention, pkg.go.dev/go doc/etc will associate each example with the thing it is an example of. This naming convention could be used to attach examples (which are a kind of test) to functions/etc.

As far as I know, that naming convention isn't used for test and benchmark functions, but that's probably because those aren't included in documentation. I think it would be reasonable to use the same convention to attach tests and benchmarks to functions/etc.

I am the author of the Go Test Explorer extension, and I may be submitting PRs to vscode-go to implement the testing API.

Jason Rudolph (@jasonrudolph)

Jason Rudolph (@jasonrudolph) recently did some work on a signficant-other extension that implements something like this. Our friends on the Java team (Sheng Chen (@jdneo)) released similar functionality in November, and Brett Cannon (@brettcannon) wants this in Python too.

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

I don't see why it would need to be more complicated than this, since the extension can also keep updating the location if the code/tests move.

An open question is how to handle "one to many" situations in both directions.

An open question is how to handle "one to many" situations in both directions.

In Java, we leverage the reference-view to deal with the "one to many" case.

Connor Peet (@connor4312)

How would the API for this look like? Would it be something along the lines of related?: vscode.Location[] on the TestItem? Something more or less advanced?

Connor Peet (@connor4312) I think Location[] is a good first step. I do think it would be valuable to have something like relatedSymbols?: vscode.DocumentSymbol[]. The most useful features that come to mind are:

Auto-run a test whenever the user changes the referenced location. This can be done with Location.
Allow the user to see what symbols are linked to a test, similar to (or incorporated into) find references. This would be much more straight forward if the test item can explicitly link to DocumentSymbols.

(2) is interesting, how would you envision the UI/UX for that?

Linking to symbols would be useful, since you wouldn't have to constantly update the location for add-on commands, but it could always be a dropdown menu on right-click for the test (maybe even a tree expansion)

The other simple alternative is a pair of functions such as provideImplementationRanges(testItem: vscode.TestItem): ProviderResult<vscode.Range[]> / provideAssociatedTests(range: vscode.Range): ProviderResult<vscode.TestItem>, which VS Code can call on demand when the user asks to go between tests and implementation. This is likely more efficient for implementors.

Kim-Adeline Miguel (@kimadeline)

/cc Kim-Adeline Miguel (@kimadeline)

Hello, it's been a minute!

I've taken this up again and implemented a proposed API in #222252. This is a simpler provider-based approach, where the provider is called on-demand. There's no passive gutters as in my previous implementation (perhaps an API point we can add in the future) but it should be a lot easier to implement.

In my initial PR this is surfaced in a few ways:

"Run tests at cursor", when called outside the range of a test, will automatically try to find tests related to the code and run those. I feel like this is a pretty nice and natural entrypoint.
Reusing the references infrastructure, there are commands for going to and peeking related tests (if you're in implementation) or related code (if you're in a test)

The PR implements this in the selfhost provider we use to run tests in the VS Code repo, using a simple file-level import graph.

Connor Peet (@connor4312)

Connor Peet (@connor4312) What kind of time frame are you looking for on feedback? Coincidentally I'm currently working on the Go test controller again, but I'm part way through a full rewrite so I'm currently focused on feature parity with the existing implementation. Side note, I implemented a TestItemResolver that allows me to expose Go tests via a TreeDataProvider-style interface, which feels much more natural to me than my previous resolver. Also, this new implementation is using the Go language server (gopls) to discover tests, so my previous statements about document symbols are not applicable to the new implementation.

For anyone else looking for the type definitions, here they are (from the PR):

export interface TestController {
	/**
	 * A provider used for associating code location with tests.
	 */
	relatedCodeProvider?: TestRelatedCodeProvider;
}

export interface TestRelatedCodeProvider {
	/**
	 * Returns the tests related to the given code location. This may be called
	 * by the user either explicitly via a "go to test" action, or implicitly
	 * when running tests at a cursor position.
	 *
	 * @param document The document in which the code location is located.
	 * @param position The position in the document.
	 * @param token A cancellation token.
	 * @returns A list of tests related to the position in the code.
	 */
	provideRelatedTests?(document: TextDocument, position: Position, token: CancellationToken): ProviderResult<TestItem[]>;

	/**
	 * Returns the code related to the given test case.
	 *
	 * @param test The test for which to provide related code.
	 * @param token A cancellation token.
	 * @returns A list of locations related to the test.
	 */
	provideRelatedCode?(test: TestItem, token: CancellationToken): ProviderResult<Location[]>;
}

I would expect that this would not be finalized sooner than September, as we're at the end of the July iteration and will probably want to let it bake in proposed for at least one iteration.

Connor Peet (@connor4312)

Connor Peet (@connor4312) Here's a bug for your brain that's kind of related. Go doesn't have explicit support for table-driven tests, but there are common patterns such as:

func TestMath(t *testing.T) {
	cases := []struct {
		Name   string
		Expr   string
		Result string
	}{
		{"add", "1+1", "2"},
		{"sub", "1-1", "0"},
		{"mul", "1*1", "1"},
		{"div", "1/1", "1"},
	}

	for _, c := range cases {
		// This creates a sub-test with the given name
		t.Run(c.Name, func(t *testing.T) {
			// Example sub-test implementation
			result, err := script.Eval(c.Expr)
			if err != nil {
				t.Fatal(err)
			}
			if result != c.Result {
				t.Errorf("want %s, got %s", c.Result, result)
			}
		})
	}
}

I plan to add static analysis to detect cases like this, but this presents a dilemma: what do I report as the range for the test item? It seems to me that the "Go to test" action should jump to body (func(t *testing.T)) since if I'm jumping to the test, I probably want to see the logic of the test. On the other hand, there are two issues (golang/vscode-go#1602, golang/vscode-go#2445) that request a way to run a specific case, which would be much easier if I reported the entry in the 'table' as the test item's range.

The best solution I can think of (that doesn't involve adding explicit table-drive test support to test items) is some kind of secondaryRanges: Range[] test item property that VSCode uses to add additional ▷ buttons. I could use provideRelatedCode but A) it sounds like that doesn't add ▷ buttons and B) it doesn't really fit the intent of related code as I understand it.

Hm, yea, we don't have a very good solution to that right now. I feel like related code might solve that approach but we'd want some more explicit "executed related tests" binding, such that a user could place their cursor in the case's struct and then run that. But it's also not very discoverable without its own 'play' buttons.

I think the way to go might be to actually declare the subtest's range as its case in the struct, rather than t.Run. That would mean no 'play' button in t.Run(c.Name, but if a user wanted to run all subtests at that point they could just run the parent test.

Tim Pope (@tpope)

👋 I work on Ruby LSP. We're looking forward to having this available for navigating between tests and implementation code.

But there's a more general case that should be considered: There can be other kinds of 'related' files in an codebase, which developers often need to switch between. For example in Rails, for each controller file, there is a typically a corresponding helper file, so there is a mapping app/controllers/foo_controller.rb -> app/helpers/foo_helper.rb.

(apologies if this was discussed already, I haven't read every comment).

(In vim, the projectionist plugin by Tim Pope (@tpope) provides a flexible way to configure these kinds of mappings, which may provide some inspiration).

Hi, and yes, that's actually why I deferred working on this for a bit -- I think a more general "go to related..." action is more useful. For example webstorm's Angular implementation can go between controllers and views, and I think that's something a "related" API could handle.

Assuming that happens, it would be helpful for my case if the extension could provide a "related code type" when vscode asks the extension for related entries. I assume this would be provider-based along the lines of registerRelatedCodeProvider? Or a language server method? I am working with the vscode-go team but I am an outside contributor; I'm working on overhauling vscode-go's test controller/provider but I'm doing it in a separate extension to make it easier to iterate. Besides code related to a test, this could be helpful in handling subtests in Go as I commented about before, so it would be helpful if I could provide results from the extension. Ultimately I'll be doing that analysis with gopls so a language server method might be workable, but on the other hand assuming there's some way to associate related code entries with test items I think that would need to happen in the extension not the language server, at least for Go. I use gopls to detect tests, but it gives me a generic response that's not specifically tailored to vscode (and thus not tailored to the test explorer).

Navigation Menu

Repository navigation

Support for "related code" in tests #126932

18 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support for "related code" in tests #126932

Description

Activity

firelizzard18 commented on Jun 22, 2021

connor4312 commented on Feb 17, 2022

JustinGrote commented on Feb 17, 2022

jdneo commented on Feb 18, 2022

firelizzard18 commented on Feb 20, 2022

connor4312 commented on Feb 22, 2022

JustinGrote commented on Feb 22, 2022

connor4312 commented on Feb 22, 2022

brettcannon commented on Feb 22, 2022

18 remaining items

connor4312 commented on Jul 22, 2024

firelizzard18 commented on Jul 22, 2024

connor4312 commented on Jul 22, 2024

firelizzard18 commented on Jul 25, 2024

connor4312 commented on Jul 31, 2024

andyw8 commented on Dec 10, 2024

connor4312 commented on Dec 10, 2024

firelizzard18 commented on Dec 10, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions