Test fail (instead of skip) on empty parameter set

[robert-cody]

I need to fail (instead of skip) tests that got empty parameters set.
My test looks like this:

def parameters_provider():
    # HTTP request to some external provider service.
    return service_request().result or []

@pytest.mark.parametrize("parameter", parameters_provider())
def test_something(parameter):
    assert parameter is not None

So if provider fails to return results, test gets empty parameters set and pytest skips it, but for me this is actually error - kind of no tests run and nothing actually checked, so test must fail.

After some investigation i've found pretty nasty way to achieve my goal, in conftest.py:

def pytest_runtest_teardown(item, nextitem):
    skip = item.get_marker("skip")
    if skip:
        reason = skip.kwargs.get("reason")
        if reason.startswith("got empty parameter set"):
            raise ValueError(reason)

I don't like it because it depends on reason text which can change in any next pytest release. Is there any better way to do it?

[RonnyPfannschmidt]

There isn't a better way yet

[nicoddemus]

Erroring out by default could be problematic as well I suppose

[robert-cody]

I see this code in pytest sources:

if not argvalues:
    argvalues = [(NOTSET,) * len(argnames)]
    # we passed a empty list to parameterize, skip that test
    #
    fs, lineno = getfslineno(self.function)
    newmark = pytest.mark.skip(
        reason="got empty parameter set %r, function %s at %s:%d" % (
            argnames, self.function.__name__, fs, lineno))
    newkeywords = [{newmark.markname: newmark}]

There are all parameters set to NOTSET sentinel value, maybe there is some way to check if all parameters is NOTSET, then raise? I didn't find any way to get these argvalues from item passed into pytest_runtest_teardown.

[robert-cody]

Or maybe there is something else i can return from provider (not empty list) so test will fail, but collection will succeed and all other tests will run also?

[RonnyPfannschmidt]

return a senseless parameterset and put a mark onto it you can detect

[robert-cody]

@RonnyPfannschmidt thanks for suggestion, implemented this way:

PARAMETER_NOT_SET = object()

def parameters_provider():
    # HTTP request to some external provider service.
    return service_request().result or [PARAMETER_NOT_SET]

@pytest.mark.parametrize("parameter", parameters_provider())
def test_something(parameter):
    assert parameter is not None

def pytest_pyfunc_call(pyfuncitem):
    arg_values = pyfuncitem.funcargs.values()
    if arg_values and all(av is PARAMETER_NOT_SET for av in arg_values):
        pytest.fail("Test function got empty parameter set!")

[nicoddemus]

Should we close this then?

[robert-cody]

If you don't mind, i'd prefer to make it optional - fail on empty set or skip. I'll make pull request for this a bit later if you think it can be useful.

[nicoddemus]

Thanks for the offer to open a PR, my personal opinion is that it should fail by default, adding another option will only complicate things. @RonnyPfannschmidt what's your take on this?

[RonnyPfannschmidt]

If prefer to avoid complication in that code

[robert-cody]

I'm also agree that it should fail by default, but for some reason it was implemented to skip such tests, that's why i proposed make it an option. If you think that nobody need skip and it won't break compatibility, let it be so. Personally i think there might be many cases when users don't know that some of their tests even not get executed and currently it's a problem.

[RonnyPfannschmidt]

I implemented the skip as people demonstrated valid use cases for empty parameter sets a while back

[robert-cody]

@RonnyPfannschmidt so your choice is option for skip/fail or only fail?
Usually it's better when test marked for skip explicitly - helps to avoid false positives.

[RonnyPfannschmidt]

Would xfail without run fit your use case?

If an option is introduced it should enable skip (default), xfail, fail and ignore, optionally via marker to choose per test