How and why I use pytest's xfail

xfail vs skip

Despite the pytest documentation containing a very clear explanation of when to use pytest.mark.skip and when to use pytest.mark.xfail, I still see a lot of confusion on this point (it's almost as if people don't sit down and read the entirety of a library's documentation before they start using it…). It's understandable — both of them are broadly used to mark tests that won't succeed. The main difference is that with skip you don't run the test at all, whereas with xfail, you run the test and make sure that it failed (if it doesn't fail, it's marked as XPASS). So the way I use these is:

Use skip for tests that aren't supposed to work — if you have a test for Windows-specific functionality, or tests that only work on Python 3.8+, conditionally skip them when running the test suites where they are not supported. This means that you should essentially always be using skip with a condition[2][3], because otherwise it makes no sense to include the test at all: you don't intend to change the software to make the test pass, and you never actually run the test, so it's just dead code.

Use xfail for tests that should, but don't currently pass. A high quality bug report has basically the same features as a good test: it has a self-contained, minimal reproduction of a particular failure mode in a desired behavior of the software. If you've gone through all the trouble of crafting a minimal bug report, you can often times simply take your example reproducer and add it to the test suite, marked as pytest.mark.xfail. Similarly, you could write the tests for a feature you'd like before writing the code to implement it in TDD style, marking the failing tests as xfail until the feature is implemented.

Making failing to fail a failure

By default, when you mark a test as xfail, the test will be run as part of the test suite and the failure or success will be recorded as either XFAIL or XPASS, but the test suite itself will pass no matter which of the two outcomes came to pass. In my opinion, this makes xfail a lot closer to a more resource-hungry version of skip, because when running hundreds or thousands of tests, most people won't notice that a few of them were XPASS and track them down. What gives xfail its super-powers, in my opinion, is the fact that you can make it strict, meaning that any XPASSing tests will cause the test suite to fail. You can either do this per-test with pytest.mark.xfail(strict=True), or you can set it globally in setup.cfg or one of the other global configuration locations.

[tool:pytest]
xfail_strict = true

This immediately makes xfail more useful, because it is enforcing that you've written a test that fails in the current state of the world. If you've written the test correctly, it should start passing when the bug is fixed, and you can be much more confident that your test is actually testing the behavior you're intending to test.

That said, this is not perfect — xfail only measures that a test fails, not necessarily why. For example, imagine we have the following broken function and test:

def is_perfect_square(n: int) -> bool:
    """Determine if any int i exists such that i × i = n."""
    s = math.sqrt(n)
    return s == int(s)

@pytest.mark.xfail(
    reason="Bug #11493: Negative values not supported",
    strict=True
)
def test_negative():
    # When called with a negative value for n, this test raises ValueError!
    v = -4
    assert not is_perfect_square(w)

This test will XFAIL just fine before you fix the underlying code, but if you change the implementation to:

def is_perfect_square(n: int) -> bool:
    """Determine if any int i exists such that i × i = n."""
    if n < 0:
        return False  # No negative numbers are perfect squares

    s = math.sqrt(n)
    return s == int(s)

Your test suite should start failing, because you haven't removed the xfail decorator, but if you were to run the tests at this point, you'd find that they pass! Eagle-eyed readers will notice that the last line of test_negative() is making assertions about unbound local variable w, rather than v! Turns out that our test was failing for the wrong reasons all along! You can mitigate this somewhat with the raises parameter, by specifying that the test must fail by raising ValueError:

@pytest.mark.xfail(strict=True, raises=ValueError)
def test_negative():
    # When called with a negative value for n, this test raises ValueError!
    # See bug #11493
    v = -4
    assert not is_perfect_square(w)

This would have solved our specific issue, but it's a very blunt instrument. It's very easy to imagine a situation where an assertion fails for the wrong reasons but still raises the expected error class (particularly if that error class is AssertionError). Still, strict xfailing tests is better than the alternative, which gives you no insight into whether or not your test was failing.

Reasons to use xfail

As promised, we now come back to look at my three purported benefits of xfail in more detail.

Parting thoughts

Perhaps it is idealistic to imagine a future where every bug report comes with a well-crafted xfailing test suite that the maintainer immediately accepts, and every time you try to fix one bug you accidentally close out 3 other tickets, but I do feel that there is a strong case for using xfail in this way.

I often think of these things through the lens of open source development (you may have seen this bias in my assumptions earlier in the post), so I will also note that "I would be happy to accept an xfailing test for this" is a great way to signal to your users that you actually intend (or at least hope) to fix their bug.

Footnotes