Is it possible for `Run Test` and `Run Tests` in VS Code to invoke nextest?

[DavisVaughan]

See also nextest-rs/nextest#2253 where I cross posted to nextest many months ago.

rust-analyzer has this nice little Run Tests button above a mod of tests:

It also has Run Test above a single test. When switching a project to nextest, most of the time Run Test still works, but often Run Tests will start to fail if you start taking advantage of nextest's one process per test feature, i.e. Run Tests will invoke some call to cargo test rather than nextest itself and your tests won't run in separate processes.

Have you all considered making the hook for Run Tests configurable in some way so that nextest could be invoked instead?

CC @sunshowers, I hope you don't mind me mentioning you here, feel free to ignore, but I thought you might want to be involved if there is movement here

[ShoyuVanilla]

Sounds like a good feature to have! But I think making a hook would be a bit too much. Wouldn't making an alternative command for running test similar to the one for run like https://rust-analyzer.github.io/book/configuration.html#runnables.command suffice?

[psibi]

Thanks for opening this issue, this is something that I have wanted for quite some time.

Wouldn't making an alternative command for running test similar to the one for run like https://rust-analyzer.github.io/book/configuration.html#runnables.command suffice?

Yeah, I think that would be simple. In one of my projects, when I click the "Run test" from my editor - this is what rust analyzer invokes currently:

cargo test --package my-pkg-name --lib -- p2p_large::large_sync --exact --show-output

But just switching test to nextest run won't work and fails with this:

$ cargo nextest run --package my-pkg-name --lib -- p2p_large::large_sync --exact --show-output
error: failed to parse test binary arguments `--show-output`: arguments are unsupported

But removing --show-output makes it work. So we would probably need more than one option: One for the subcommand that is invoked and the additional options that is appeneded to the test runner.

[dfireBird]

From my initial analysis, we'd need 3 new configs like:

rust-analyzer.runnables.test.overrideCommand
rust-analyzer.runnables.bench.overrideCommand
rust-analyzer.runnables.docTest.overrideCommand

The rust-analyzer.runnables.extraTestBinaryArgs config covers the additional options and is used on all 3 cases i.e test, bench and doctest.

Probably we can add just only one config for test and add the remaining later but if someone wants to change for test I assume they will want to change for others as well (even though nextest doesn't support doc tests yet)

I will work on this if nobody is working and the team is okay with config naming.

[DavisVaughan]

Would it be possible to reopen this issue?

I've been trying this out, and unfortunately I think a single "rust-analyzer.runnables.test.overrideCommand" is not flexible enough for all of the different ways that tests can be invoked.

I'll try and explain with some examples from ark, where we have a large variety of different types of tests, and use nextest as our runner.

The main issue is that I've identified 4 distinct ways that tests can be run using rust-analyzer's Runnables, and each requires slightly different command line invocations to work properly with nextest. rust-analyzer's default support for cargo test is able to make these slight adjustments, but I am not able to make them in "rust-analyzer.runnables.test.overrideCommand". The 4 ways are:

• Single library test (i.e. Run Test for --lib)
• Multiple library tests (i.e. Run Tests for --lib)
• Single integration test (i.e. Run Test for --test {target})
• Multiple integration tests (i.e. Run Tests for --test {target})

I go into each in more detail below, showing an example of each, the rust-analyzer cargo test command, the required nextest command to match that, and the required json config to make that command. That way you all can have a (hopefully complete) set of examples to target support for.

It's possible you'd have to expose all of these to really support this:

• rust-analyzer.runnables.libraryTest.overrideCommand
• rust-analyzer.runnables.libraryTests.overrideCommand
• rust-analyzer.runnables.integrationTest.overrideCommand
• rust-analyzer.runnables.integrationTests.overrideCommand

Single library test (i.e. Run Test for a library)

Example in ark, clicking Run Test when it pops up here:
https://github.com/posit-dev/ark/blob/bc032863711f1b5951955b3f0495bc6a8a4bf10b/crates/ark/src/lsp/definitions.rs#L96

• Can't use ${target} (it's a --lib test, so ${target} should not follow --lib)
• Needs --exact

# rust-analyzer cargo command
cargo test --package ark --lib -- lsp::definitions::tests::test_goto_definition --exact --nocapture 

# matching nextest command
cargo nextest run --package ark --lib -- lsp::definitions::tests::test_goto_definition --exact 

Required config to create nextest command

"rust-analyzer.runnables.test.overrideCommand": [
    "cargo",
    "nextest",
    "run",
    "--package",
    "${package}",
    "${target_arg}",
    "--",
    "${test_name}",
    "--exact"
],

Multiple library tests (i.e. Run Tests for a library)

Example in ark, clicking Run Tests when it pops up here:
https://github.com/posit-dev/ark/blob/bc032863711f1b5951955b3f0495bc6a8a4bf10b/crates/ark/src/lsp/definitions.rs#L86

• Can't use ${target} (it's a --lib test, so ${target} should not follow --lib)
• Can't use --exact (need to match multiple)

# rust-analyzer cargo command
cargo test --package ark --lib -- lsp::definitions::tests --nocapture

# matching nextest command
cargo nextest run --package ark --lib -- lsp::definitions::tests 

Required config to create nextest command

"rust-analyzer.runnables.test.overrideCommand": [
    "cargo",
    "nextest",
    "run",
    "--package",
    "${package}",
    "${target_arg}",
    "--",
    "${test_name}"
],

Single integration test (i.e. Run Test for a tests/ folder integration test)

Example in ark, clicking Run Test when it pops up here:
https://github.com/posit-dev/ark/blob/bc032863711f1b5951955b3f0495bc6a8a4bf10b/crates/ark/tests/kernel-r-profile.rs#L11

• Needs ${target} (since we now have --test, rather than --lib, as the ${target_arg})
• Needs --exact

# rust-analyzer cargo command
cargo test --package ark --test kernel-r-profile -- test_r_profile_can_cat --exact --nocapture 

# matching nextest command
cargo nextest run --package ark --test kernel-r-profile -- test_r_profile_can_cat --exact 

Required config to create nextest command

"rust-analyzer.runnables.test.overrideCommand": [
    "cargo",
    "nextest",
    "run",
    "--package",
    "${package}",
    "${target_arg}",
    "${target}",
    "--",
    "${test_name}",
    "--exact"
],

Multiple integration tests (i.e. Run Tests for a tests/ folder integration test file)

Example in ark, clicking Run Tests when it pops up here:
https://github.com/posit-dev/ark/blob/bc032863711f1b5951955b3f0495bc6a8a4bf10b/crates/ark/tests/kernel-r-profile.rs#L1

• Needs ${target}
• Can't use --exact (need to match multiple)
• Also doesn't use ${test_name}, it doesn't seem to even exist in this scenario

# rust-analyzer cargo command
cargo test --package ark --test kernel-r-profile --  --nocapture

# matching nextest command
cargo nextest run --package ark --test kernel-r-profile 

Required config to create nextest command

"rust-analyzer.runnables.test.overrideCommand": [
    "cargo",
    "nextest",
    "run",
    "--package",
    "${package}",
    "${target_arg}",
    "${target}"
],

[dfireBird]

Thanks for the feedback but I do think we don't need that many config, but rather make changes to existing config so that it fits better for all needs.

Can't use ${target} (it's a --lib test, so ${target} should not follow --lib)

This is a mistake done by me where I replaced the ${target} in all cases including --lib test, while the existing behavior (with cargo test) does not do that. I'll fix this with a new PR.

Needs --exact
Can't use --exact (need to match multiple)

I am thinking of solving this by removing the current ${test_name} placeholder with ${executable_args} and replace all needed executable arguments in that place i.e the test name, adding --exact if in case of single test, adding --ignored in case of #[ignore] attribute and value of the config rust-analyzer.runnables.extraTestBinaryArgs.

cc: @ChayimFriedman2

[sircelj]

Just adding to this conversation, since a project I'm working on also relies on nextest instead of test because of the separate processes.

From looking at the cargo test ... vs. cargo nextest run ... examples @DavisVaughan provided, I fail to see any difference other than test -> nextest run.

Wouldn't the simplest solution be to allow multiple commands in the rust-analyzer.runnables.test.command setting? That way, we could set it to "nextest run". Focusing on overrideCommand feels like a shame since we're so close to the solution.

If I set:

"rust-analyzer.runnables.test.command": "nextest run"

Then, when running the codelens buttons, this is the error I get:

 *  Executing task: cargo nextest run --package my-package --test integrationtests -- 

error: no such command: `nextest run`

help: view all installed commands with `cargo --list`
help: find a package to install `nextest run` with `cargo search cargo-nextest run`

 *  The terminal process "cargo 'nextest run', '--package', 'my-package', '--test', 'integrationtests', '--', ''" terminated with exit code: 101.

Notice the cargo 'nextest run'. If I run the above command in my terminal, it executes correctly, for all cases: single unit tests and their modules, single integration tests and their modules.

Would it be possible to allow for multiple commands/spaces in rust-analyzer.runnables.test.command?

[DavisVaughan]

No strong feelings from me. That basically just requires that the command part of rust-analyzer.runnables.test.command should support all (or at least the subset rust-analyzer users) options of cargo test, with cargo nextest run seems to do

With overrideCommand you get to "build your own" command so its slightly more flexible, but in the end your backend basically still needs to support the exact commands that cargo test does, because we have to define overrideCommand in simple json and can't do any fancy remapping between the cargo test specification and whatever your test backend actually uses instead.

[DavisVaughan]

I would once again like to request that this issue be reopened 😢

@dfireBird I don't think your solution in #21395 will actually work. This is what I tried:

    "rust-analyzer.runnables.test.overrideCommand": [
        "cargo",
        "nextest",
        "run",
        "--package",
        "${package}",
        "${target_arg}",
        "${target}",
        "--",
        "${executable_args}"
    ],

nextest doesn't seem to allow -- --nocapture, instead it has --no-capture (note, before the --). So this errors with

Executing task: cargo nextest run --package ark --lib -- lsp::definitions::tests::test_goto_definition --exact --nocapture 

error: failed to parse test binary arguments `--nocapture`: arguments are unsupported

 *  The terminal process "cargo 'nextest', 'run', '--package', 'ark', '--lib', '--', 'lsp::definitions::tests::test_goto_definition', '--exact', '--nocapture'" terminated with exit code: 96. 

And I don't think I can replace "${executable_args}" with my own preferred args because it varies per test type as shown in
#21137 (comment)

So I'm back to wondering if we really do need configuration options for all flavors of test runnables that rust-analyzer may call, i.e. what I mentioned here:

• rust-analyzer.runnables.libraryTest.overrideCommand
• rust-analyzer.runnables.libraryTests.overrideCommand
• rust-analyzer.runnables.integrationTest.overrideCommand
• rust-analyzer.runnables.integrationTests.overrideCommand

At the end of the day this really does provide maximal configuration, which might be a necessary evil to support arbitrary test runners.

Update: Hmm, although I guess with this approach we still wouldn't be able to inject "dynamic" flags like --ignore when there is an ignore attribute. rust-analyzer would have to expose that for us as ${ignore} or something.

---

Note that nextest not supporting -- --nocapture like this also means that the solution from @sircelj doesn't work, as simply replacing cargo test with cargo nextest run and keeping all the rest of the arguments would still fail in the same way.