Subtests in Python

Introduction

unittest.TestCase.subTest was originally introduced in Python 3.4 as a lightweight mechanism for test parameterization [1]; it allows you to mark a section of your test as a separate test in its own right using a context manager. The canonical example is testing something in a loop:

def test_loop(self):
    for i in range(5):
        with self.subTest("Message for this subtest", i=i):
            self.assertEqual(i % 2, 0)

Without the self.subTest context manager, this test would fail immediately when i=1 and execution would end, reporting that test_loop has failed. With the context manager, though, the failures in each subTest's context don't cause the test to exit and execution continues. The result of running this test is that you'll see successes reported for 0, 2 and 4 with failures reported for 1 and 3. You can pass any arbitrary keywords to subTest and they will be reported as part of the test failure, e.g.:

______ Test.test_loop [Message for this subtest] (i=1) ___

self = <test.Test testMethod=test_loop>

    def test_loop(self):
        for i in range(1, 5):
            with self.subTest("Message for this subtest", i=i):
>               self.assertEqual(i % 2, 0)
E               AssertionError: 1 != 0

test.py:7: AssertionError

______ Test.test_loop [Message for this subtest] (i=3) ___

...

Why not pytest.mark.parametrize or equivalent?

For users of pytest, the parameterization use case of subtests is not terribly compelling, since we already have several mechanisms to do this including pytest.mark.parameterize, parameterized fixtures, and the much more obscure pytest_generate_tests hook in conftest.py. Even Google's absltest framework (which is mainly minor extensions to unittest) has a parameterization decorator.

I would tend to agree that in general I do not use subtests for test parameterization — I mostly do it when I must use only unittest, such as when developing for the standard library. There are, however, occasionally situations where the subTest form factor offers some advantages even in parameterization. For example, if you have a number of tests you'd like to perform that have an expensive set-up function that builds or acquires an immutable resource that is used in common by all the subtests:

def test_expensive_setup(self):
    resource = self.get_expensive_resource()

    for query, expected_result in self.get_test_cases():
        with self.subTest(query=query):
            self.assertEqual(resource.make_query(query), expected_result)

I'm sure it's possible that you can write a pytest fixture that is scoped such that the resource is acquired only before this particular set of parameterized tests runs and is torn down immediately afterwards, but even if it is possible without any significant contortions, I doubt that the lifetime of the resource will be quite as straightforward to understand for a casual reader or reviewer as the subtest-based approach.

It is also possible for the two approaches to work together harmoniously — using a decorator-based approach to provide canonical "test cases", and then subtests to explore variations on that theme. For example, you could parameterize a test function by the value of its inputs and add in subtests to check multiple properties of the result. For example, if you want to make sure that both the utcoffset() and tzname() functions of a tzinfo object are right at several datetime s: [2]

from datetime import *
from zoneinfo import ZoneInfo

import pytest

def datetime_test_cases():
    GMT = ("GMT", timedelta(0))
    BST = ("BST", timedelta(1))
    zi = ZoneInfo("Europe/London")

    return [
        (datetime(2020, 3, 28, 12, tzinfo=zi), GMT),
        (datetime(2020, 3, 29, 12, tzinfo=zi), BST),
        (datetime(2020, 10, 24, 12, tzinfo=zi), BST),
        (datetime(2020, 10, 25, 12, tzinfo=zi), GMT),
    ]

@pytest.mark.parameterize("dt, offset", datetime_test_cases())
def test_europe_london(subtests, dt, offset):
    exp_tzname, exp_utcoffset = offset
    with subtests.test(msg="tzname", dt=dt):
        assert dt.tzname() == exp_tzname

    with subtests.test(msg="utcoffset", dt=dt):
        assert dt.utcoffset() == exp_utcoffset

Here the test is parameterized by value, but it tests each value in two different ways.

Beyond parameterization

Although there are a few cases where you want a lightweight parameterization mechanism, the times that I've felt that subtests were a missing feature in pytest have little to do with parameterization. One thing that subtests do very well is to let you keep to the spirit of "one assertion per test" when you'd like to explore multiple properties of the state of a system.

For example, let's take a look at this test I wrote for the reference implementation of PEP 615. The PEP describes the creation of a new zoneinfo.ZoneInfo object, which (to simplify the situation a bit) generates singleton objects — to a first approximation, any time you call zoneinfo.ZoneInfo(key) with the same value for key, you should get the same object you got in earlier keys. This also applies to ZoneInfo objects constructed from a pickle, so this is a test to ensure that if we pickle and unpickle a ZoneInfo object, we get the same object back. Starts out simple enough, and I could have written it this way:

def test_cache_hit(self):
    zi_in = ZoneInfo("Europe/Dublin")
    pkl = pickle.dumps(zi_in)
    zi_out = pickle.loads(pkl)

    self.assertIs(zi_in, zi_out)

However, this test inherently is about global state. First I've populated the ZoneInfo cache via the primary constructor, then I've hit it via whatever mechanism pickle.loads uses — what if doing so puts our repository in a weird state? What if only the first cache miss works, and subsequent cache misses do something else? To test for that, I could write a second test:

def test_cache_hit_twice(self):
    zi_in = ZoneInfo("Europe/Dublin")
    pkl = pickle.dumps(zi_in)
    zi_rt = pickle.loads(pkl)
    zi_rt_2 = pickle.loads(pkl)

    self.assertIs(zi_rt, zi_rt2)

But you'll note that up until the second pickle.loads call, this is the same test: I'm establishing the same state! If I were to add a self.assertIs(zi_in, zi_rt) into the second test, I'd be able to do both tests at once, but that would violate the "one assertion per test" guideline — I am testing two different things, they should be two different tests. Subtests solve this conundrum by allowing you to mark off sections of your multi-assertion test as being logically separate tests:

def test_cache_hit(self):
    zi_in = ZoneInfo("Europe/Dublin")
    pkl = pickle.dumps(zi_in)

    zi_rt = pickle.loads(pkl)
    with self.subTest("Round-tripped is non-pickled zoneinfo"):
        self.assertIs(zi_in, zi_rt)

    zi_rt2 = pickle.loads(pkl)
    with self.subTest("Round-trip is second round-tripped zoneinfo"):
        self.assertIs(zi_rt, zi_rt2)

Note also that I've excluded the pickle.loads calls from my subtest contexts. This is because if something in a subtest fails, the rest of the test is executed; if zi_in and zi_rt are not identical objects, that doesn't preclude zi_rt and zi_rt2 from being identical objects, so it makes sense to run both tests, but if either zi_rt or zi_rt2 fails to be constructed, tests involving them will necessarily fail.

Downsides of subtests

I do not want to paint an overly rosy picture of the current subtest landscape; I think that there's great potential in the concepts, but the devil is in the details. As I've been using subtests more as part of implementing PEP 615, and more generally in researching this post, I've come across a few fairly reasonable objections to the use of subtests.

def test_a_loop(subtests):
    for i in range(0, 6, 2):
        with subtests.test(i=i):
            assert i % 2 == 0

def test_a_loop(subtests):
    for i in range(3):
        with subtests.test(i=i):
            assert i % 2 == 0

def test_a_loop(subtests):
    for i in range(3):
        with subtests.test(i=i):
            if i == 2:
                pytest.skip()

            assert i % 2 == 0

def test_a_loop(subtests):
    for i in range(3):
        if i == 1:
            pytest.fail()

        with subtests.test(i=i):
            assert i % 2 == 0

test.py::test_a_bunch_of_stuff PASSED
test.py::test_a_bunch_of_stuff FAILED

def test_loop(self):
    for i in range(5):
        with self.subTest(i=i):
            if i % 2:
                self.skipTest("Skipping odd")
            self.assertEqual(i % 2, 0)

test.py::Test::test_loop SKIPPED    [100%]
test.py::Test::test_loop SKIPPED    [100%]

============ 2 skipped in 0.02s ==========

Summary

This is my longest and most rambling blog post to date, and it covers a lot of ground, so I thought that I should sum up the things I've covered in the post.

Some key use cases I've identified:

1. Allows test parametrization if you cannot use pytest, or if you want to generate some test cases in a way that is scoped to the test you are currently running.
2. When it is expensive to acquire a resource but you want to probe more than one of its properties, subtests provide an easy to use and easy to understand mechanism to logically separate out the tests.
3. If you would like to probe the state of a system as it evolves, subtests allow you to make non-fatal assertions, so you can get the benefits of multiple tests without the boilerplate of setting up a lot of redundant state.

The biggest downsides I've identified:

1. The concept of counting "a test" or "a failure" gets a lot fuzzier.
2. Easy to get spammed with a million results if the failures are correlated.
3. Subtests tend to present edge cases to a lot of other testing tools and features, and as a result there are still a decent number of bugs in the implementations.

I will say that I am optimistic about the future for subtests, whether it's the standard library implementation, pytest-subtests or other related non-fatal failure testing libraries I haven't even talked about like pytest-check.

There are real upsides to using subtests and none of the downsides seem terribly fundamental; with greater adoption of the concepts and some iteration on the tools providing these capabilities, I could see subtests becoming a normal part of one's testing arsenal.

It's worth noting that as of this writing pytest-subtests is still in early development (it's version 0.3.0 on PyPI), and with some additional work, a lot of the rough edges could be smoothed out. If this blog post or the corresponding podcast episode has gotten you interested in subtests, perhaps contributing to the plugin is a good way to start getting involved!

Footnotes