Skip to content

When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking the abilty to run a single test method. #2454

Closed
Closed
When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking the abilty to run a single test method.#2454
Labels
area-testingbugIssue identified by VS Code Team member as probable bug

Description

@jakegt1
Jake 'Donkey" Torrance (jakegt1)
opened on Aug 28, 2018

When using a workspace with multiple python projects, you cannot configure pytest in a way that allows you to run tests with only one of the projects without breaking something else about the testing process.

The directory structure is like so:

  • projectIWantToTest
    • codeIWant
  • myOtherProject
    • codeIDontWant

Environment data

  • VS Code version: 1.26.1
  • Extension version (available under the Extensions sidebar): 2018.7.1
  • OS and version: Windows 10 1803
  • Python version (& distribution if applicable, e.g. Anaconda): 3.5.3
  • Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
  • Relevant/affected Python packages and their versions: pytest (using 3.5.0)

Actual behavior

  • Option 1 - pytest arguments configured as [".", "--ignore=myOtherProject"]
    Run/debug all tests - works fine
    Run/debug a specific test - Causes all tests to be run.

  • Option 2 - pytest arguments configured as ["projectIWantToTest"]
    Run/debug all tests - Works fine
    Run/debug a specific test - Does not work. Returns an error saying that the test cannot be found (This seems to be due to pytest changing the path of the tests from projectIWantToTest/codeIWant/tests.. to codeIWant/tests..)
    I have tried this with the python.unitTest.cwd config variable and it seems to do nothing, or at least does nothing with test discovery.

Expected behavior

  • Option 1 - Both running all tests and running a specific test works.
  • Option 2 - Either configuring the python.unitTest.cwd variable works with discovery and fixes this problem OR deal with pytest removing the absolute paths OR don't break discovery when --ignore is one of the arguments for pytest (or suggest valid workaround with pytest arguments)

I could also suggest another option that simply allows running all tests in a directory rather than all tests, a specific test file or a specific test method, which is a function i feel this extension has been missing for a long time (especially for people who came from PyCharm.)

Steps to reproduce:

Have two python projects with tests in the same workspace. Attempt to configure pytest to run and discover the tests of only one project. For all options i have tried this breaks the ability to run or debug a single test.

Note: Sorry about the super long title, it's quite a specific bug.

Activity

  1. ImageJake 'Donkey" Torrance (jakegt1)
    changed the title [-]When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking[/-] [+]When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking the abilty to run a single test.[/+] on Aug 28, 2018
  2. ImageJake 'Donkey" Torrance (jakegt1)
    changed the title [-]When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking the abilty to run a single test.[/-] [+]When using pytest with multiple python projects inside the workspaceFolder, you cannot configure pytest to only run one project's test suite without breaking the abilty to run a single test method.[/+] on Aug 28, 2018
  3. ImageDerek Keeler (d3r3kk)
    added
    bugIssue identified by VS Code Team member as probable bug
    needs verification
    area-testing
    on Aug 28, 2018
  4. ImageBrett Cannon (brettcannon)
    added
    info-neededIssue requires more information from poster
    and removed
    needs verification
    on Aug 29, 2018

  5. brettcannon commented on Aug 29, 2018

    @brettcannon
    Brett Cannon (brettcannon)
    on Aug 29, 2018
    Member

    I can't reproduce. Can you try something like pytest 3.6 (we are working on a code lens bug with pytest 3.7 or else I would suggest that)?

  6. jakegt1 commented on Aug 30, 2018

    @jakegt1
    Jake 'Donkey" Torrance (jakegt1)
    on Aug 30, 2018
    Author

    I'll go make a git repo quickly to reproduce it, sorry about the poor explanation!

  7. jakegt1 commented on Aug 30, 2018

    @jakegt1
    Jake 'Donkey" Torrance (jakegt1)
    on Aug 30, 2018
    Author

    Brett Cannon (@brettcannon)
    https://github.com/jakegt1/vscode-python-pytest-bug

  8. jakegt1 commented on Aug 30, 2018

    @jakegt1
    Jake 'Donkey" Torrance (jakegt1)
    on Aug 30, 2018
    Author

    Work around is to simply use multi-root workspaces (which probably makes more sense), but i'll leave this up to let you decide what to do with it.

  9. ImageBrett Cannon (brettcannon)
    added
    needs verification
    and removed
    info-neededIssue requires more information from poster
    on Aug 30, 2018
  10. ImageBrett Cannon (brettcannon)
    self-assigned this
    on Sep 7, 2018

  11. brettcannon commented on Sep 7, 2018

    @brettcannon
    Brett Cannon (brettcannon)
    on Sep 7, 2018
    Member

    I'll verify this.

  12. ImageBrett Cannon (brettcannon)
    added
    needs PR
    and removed
    needs verification
    on Sep 24, 2018
  13. ImageBrett Cannon (brettcannon)
    removed their assignment
    on Sep 24, 2018

  14. brettcannon commented on Sep 24, 2018

    @brettcannon
    Brett Cannon (brettcannon)
    on Sep 24, 2018
    Member

    I can verify that with issue2454.zip running all tests works but running a single test actually runs all tests plus the individual test (i.e. says 3 tests ran instead of 2).

  15. southwood commented on Oct 11, 2018

    @southwood
    Scott Southwood (southwood)
    on Oct 11, 2018

    I am also seeing this but with a single python project. Attempting to run a single test runs all tests. I'm using pytest 3.6.1, python 2.7, and VSCode 1.28.

    It looks like the problem had to do with using addopts in setup.cfg. Overriding the config file in .vscode/settings.json like this fixed the issue:

    "python.unitTest.pyTestArgs": [
    "-c", "/dev/null",
    "--no-cov"
]

  16. luabud commented on Sep 11, 2019

    @luabud
    Luciana Abud (luabud)
    on Sep 11, 2019
    Member

    Original issue is fixed by using multi-root workspaces, so closing it.
    If anyone finds a problem related to this with single projects, please open a new issue.

  17. ImageLuciana Abud (luabud)
    closed this as completedon Sep 11, 2019
  18. ghost removed
    needs PR
    on Sep 11, 2019
  19. Imagelock
    locked as resolved and limited conversation to collaborators on Sep 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-testingbugIssue identified by VS Code Team member as probable bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions