3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses

[wkschwartz]

BPO	21112
Nosy	@pitrou, @larryhastings, @rbtcollins, @ned-deily, @ezio-melotti, @voidspace, @berkerpeksag, @wkschwartz
Files	test_expectedFailure.py: Test case for unittest.expecteFailure regression issue21112.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/berkerpeksag'
closed_at = <Date 2015-08-27.22:38:35.450>
created_at = <Date 2014-03-31.14:20:26.695>
labels = ['tests', 'type-bug', 'library']
title = '3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses'
updated_at = <Date 2015-08-27.22:38:35.449>
user = 'https://github.com/wkschwartz'

bugs.python.org fields:

activity = <Date 2015-08-27.22:38:35.449>
actor = 'rbcollins'
assignee = 'berker.peksag'
closed = True
closed_date = <Date 2015-08-27.22:38:35.450>
closer = 'rbcollins'
components = ['Library (Lib)', 'Tests']
creation = <Date 2014-03-31.14:20:26.695>
creator = 'William.Schwartz'
dependencies = []
files = ['34680', '34681']
hgrepos = []
issue_num = 21112
keywords = ['patch', '3.4regression']
message_count = 16.0
messages = ['215240', '218491', '218492', '218683', '218727', '218731', '226552', '226553', '226562', '229543', '229545', '229589', '238091', '238168', '249266', '249267']
nosy_count = 9.0
nosy_names = ['pitrou', 'larry', 'rbcollins', 'ned.deily', 'ezio.melotti', 'michael.foord', 'python-dev', 'berker.peksag', 'William.Schwartz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21112'
versions = ['Python 3.4', 'Python 3.5']

[wkschwartz]

In Python 2.7 and 3.3, decorating a unittest.TestCase subclass with unittest.expectedFailure caused test discover to skip the decorated test case. Python 3.4 apparently ignores the decorator when applied to classes.

The attached file when run with Python 2.7.6 on Mac OS X produces the following output

$ python -m unittest discover
F.

======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

---

Ran 2 tests in 0.000s

FAILED (failures=1)

When run with Python 3.4.0 produces the following output.

~/Documents/testtest $ python3 -m unittest discover
F.F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

======================================================================
FAIL: test_fails (test.TestTreatment)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

---

Ran 4 tests in 0.001s

FAILED (failures=2)

The expectedFailure decorator when applied to a class should either skip the class or run all of its tests and check that they failed for consistency with how expectedFailure applies to test methods.

[berkerpeksag]

Is there a chance to get this into 3.4.1?

[ned-deily]

Berker, if you think it should go into 3.4.1, you need to set the priority to "release blocker" to bring it to the attention of the release manager.

[larryhastings]

Berker: do you consider your diff ready to go in, or is it an "early" diff (like a work-in-progress)?

[berkerpeksag]

Berker: do you consider your diff ready to go in, or is it an "early"
diff (like a work-in-progress)?

I tested my patch with test_expectedFailure.py again. The patch is not really fixes the problem described in msg215240. So, I consider it a WIP patch for now. Sorry for the noise.

Here is the output on Python 3.3:

$ python3.3 -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

---

Ran 2 tests in 0.001s

FAILED (failures=1)

And here is the output on Python 3.5 (with bpo-21112.diff patch):

$ ./../python -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok
test_fails (test_expectedFailure.TestTreatment) ... expected failure
test_passes (test_expectedFailure.TestTreatment) ... unexpected success

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

---

Ran 4 tests in 0.002s

FAILED (failures=1, expected failures=1, unexpected successes=1)

[larryhastings]

Considering that I'm tagging 3.4.1 within an hour or two, and we don't have a patch yet, I'd say that this is too late to go into 3.4.1. But I'm happy to consider it for a future 3.4.x revision.

[ned-deily]

Should this be fixed for 3.4.2?

[larryhastings]

Note: current plan for 3.4.2 is to release at the end of the month. RC1 will be in about a week.

[voidspace]

The patch looks good to me.

[wkschwartz]

3.4.2 has been released, it seems, without this getting fixed.

3.4.3 then? Are we still happy with the patch?

[berkerpeksag]

3.4.3 then? Are we still happy with the patch?

Please see msg218727. The patch is not ready yet.

[pitrou]

Berker, it looks like the 3.3 behaviour was buggy: TestTreatment isn't run at all! Instead, it should be run and consider failures as success.

So your patch appears more correct than what 3.3 did.