Test run results in status bar reporting incorrect count

Environment data

VSCode Version: May 2018 (1.24)
Extension version (available under the Extensions sidebar): 2018.6.0
OS and version: Windows 10
Python version (& distribution if applicable, e.g. Anaconda): 2.7.14
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): NA
Relevant/affected Python packages and their versions: NA

Actual behavior

I use VS code for my Python projects and we have unit tests written using Python's unittest module. I am facing a weird issue with debugging unit tests.

Let's say I have 20 unit tests in a particular project.

I run the tests by right clicking on a unit test file and click 'Run all unit tests' After the run is complete, the results bar displays how many tests are passed and how many are failed. (e.g. 15 passed, 5 failed).
And I can run/debug individual test because there is a small link on every unit test function for that. If I re-run the tests from same file, then the results bar displays the twice number of tests. (e.g. 30 passed, 10 failed)
Also the links against individual test functions disappear. So I cannot run individual tests.
The only way to be able to run/debug individual tests after this is by re-launching the VS code.

Expected behavior

Re-run of unit tests should not double the number.
Individual test functions debugging should be possible.

Steps to reproduce:

Run the python tests by right clicking on a unit test file and click 'Run all unit tests'.
After the run is complete, the results bar displays how many tests are passed and how many are failed. (e.g. 15 passed, 5 failed). And you can run/debug individual test because there is a small link on every unit test function for that.
Now re-run the tests from same file, then the results bar displays the twice number of tests.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

Starting the classic analysis engine.
##########Linting Output - pylint##########
No config file found, using default configuration

Your code has been rated at 9.95/10 (previous run: 9.72/10, +0.23)

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help)

[Extension Host] Python Extension: Error: read ECONNRESET
t.log @ workbench.main.js:sourcemap:270

I believe I have encountered the test-count bug, but I've not seen the test 'Code Lens' disappear during testing.

Derek Keeler (@d3r3kk)

I believe I have encountered the test-count bug,

Derek Keeler (@d3r3kk) please could you update the issue with instructions to replicate this.

I still face this issue! For me, every time I re-run the unit tests from a file, the count doubles and 'Code Lens' (didn't know this term, nice one though!) disappears.

Derek Keeler (@d3r3kk)

Derek Keeler (@d3r3kk) MandarJKulkarni
Please could you provide instructions along with sample code (repo) to replicate this issue.

ElaineDi

A reproducer thanks to ElaineDi:

import unittest

class PassingTest(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42,42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")

class FailingTest(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42,-13)

    def test_failure_still(self):
        self.assertEqual("I'm right","No I am")

This reports 2 passes and 2 failures the first time you run all tests, 4/4 the second time.

Ensure unittest test results are only recorded once per test run.

I think this may have broken re-running failing tests. With the following code:

import unittest

class PassingTests(unittest.TestCase):

    def test_passing(self):
        self.assertEqual(42, 42)

    def test_passing_still(self):
        self.assertEqual("silly walk", "silly walk")


class FailingTests(unittest.TestCase):

    def test_failure(self):
        self.assertEqual(42, -13)

    def test_failure_still(self):
        self.assertEqual("I'm right!", "no, I am!")

I get the following:

And if you run "All Tests", run just the succeeding tests through the code lens, and then run "All Tests" again, it is reported not tests ran.

In both failure cases the proper tests are actually run, the results are just reported incorrectly.

Notes for tomorrow-d3r3kk:

I have found the source of the trouble. In my latest "fix", when we run tests I register all the event listeners on the server object here.

In the socket.disconnected callback I unregister everything.

This works when we are running a single file/suite/test, but it fails when we run multiple files, multiple suites (which basically amounts to files), or multiple tests, because socket.disconnect is encountered after the first file/suite/test is run and no subsequent tests will be captured!

I will rethink this design tonight and see if I cannot come up with a more useful solution (such as queue-ing up test jobs, or rethinking how we register and unregister tests).

*Addendum to* fixes for microsoft#2143

Navigation Menu

Repository navigation

Test run results in status bar reporting incorrect count #2143

Environment data

Actual behavior

Expected behavior

Steps to reproduce:

Logs

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test run results in status bar reporting incorrect count #2143

Description

Environment data

Actual behavior

Expected behavior

Steps to reproduce:

Logs

Activity

d3r3kk commented on Jul 12, 2018

DonJayamanne commented on Jul 14, 2018

MandarJKulkarni commented on Jul 28, 2018

DonJayamanne commented on Jul 28, 2018

brettcannon commented on Jul 30, 2018

5 remaining items

brettcannon commented on Aug 21, 2018

brettcannon commented on Aug 21, 2018

d3r3kk commented on Aug 22, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions