pytest-check - allow multiple failures per test
09:56 Share

Today we're talking about PyTest Check, a plugin that allows multiple failures per test. Welcome to Test & Code. This episode is sponsored by Porkbun.com, named number one domain registrar by USA Today from 2023 to 2025. Right now, Porkbun has a special low price of less than $6 for the first year of your next .app or .dev domain name, ideal for developers, web designers, engineers, or anyone in tech.

.app and .dev domains are perfect for your latest projects. Showcase your world-changing application on .app and use .dev for your blog, documentation, and more. As always, every Pokemon domain comes with lots of freebies like SSL certificates, Whois privacy, URL forwarding, and hosting trials.

With deals like this, backed by personalized support from real people 365 days a year, it's no wonder so many tech people and enthusiasts are making the switch from places like GoDaddy and choosing Porkbun instead. To get the first year of your next .app or .dev domain for under $6, visit porkbun.com slash test and code 25 or click the link in your show notes.

PyTestCheck is a plugin that's been around since 2017. It's one that I wrote, and it is currently number 35 on the top PyTest plugin list. PyTestCheck is a plugin that allows multiple failures per test. Normally, a test function will fail and stop running on the first failed assert.

That's totally fine for tons of kinds of software tests. However, there are times when you'd like to check more than one thing, and you'd really like to know the results of each check, even if one of them fails. iTestCheck allows multiple failed checks per test function so you can see the whole picture of what's going wrong. There are two main ways to use this plugin.

And you can mix and match and use both in the same test file. The first way is to use the context manager. Use the check object as a context manager. So you can say with check, and then within the with check, you can have a whole bunch of code. And it'll run that, and any asserts that fail will stop the with block, but they will continue on in the test code.

So let's say you have three asserts. If you want all three asserts to be evaluated, even if some of them fail, then you can wrap each of them with a with check block.

The other way to do this is to use validation functions. So there's a whole bunch of validation functions provided, like equal, not equal, is, is not, is true, is false. And those are, they act kind of like assert methods, but you pass in the parameters. So like, for instance, equal, the equal method, you pass in two things and those get compared for equality. This is actually the original method

thing that I had in place in 2017, and I got some help from the community to do the context manager version. But you can mix and match. You can use both the validation functions and the with check block within a test. The PyTest check plugin will collect all of the check failures, and for a test, you can have multiple check failures. So at the end of the report, like normally, if there was an assert failure, you could see the traceback.

there's a pseudo-traceback that gets printed for each failed check, and that's controllable.

So for each failed check, there is a traceback and then it gets reported as a failed check also. So it has like a pseudo traceback and then reports of the check failures. But if you have a lot of these, like let's say, let's say you've got three, that's not a big deal. But if you're like, say, comparing each element of a 10,000 element vector, you don't want all of them printed. So you can limit those. There are flags for limiting the number of tracebacks that are listed, limiting the number of reported failures.

And then also limiting the number of failed checks at all. So there's one option where after a certain amount of time, or let's say you have a 10,000 element array, after like 10, if 10 are wrong, stop checking. This helps save time if you want to speed things up.

However, I've tried to make the defaults pretty zippy anyway, so it isn't unlimited, but it's still pretty helpful. But I didn't think that you would want like 10,000 tracebacks. It's annoying to see something like that. So I can't remember what the defaults are, but you can change all of those, of course, with flags and with settings, changing it in your .ini file.

One thing I forgot to mention was that these helper functions, they're pretty easy to make yourself. So if you want to use the helper function, the validation functions, but there's something there that you need that isn't there, there's some examples in the readme of how to make your own. There's also a write-up for how to speed things up. So if you're using this, it's already pretty zippy, but if you really want it to fly, you can look at this speed up section.

Finally, I want to talk about why I wrote this in the first place. Isn't just one assert enough per test? Aren't you supposed to only check one thing per test? That's something that comes up a lot when people are familiar with a lot of the unit test philosophy of testing one thing only. And I get in certain contexts that might be fine, but also I think it's crazily limited.

Also, PyTest is used for a lot more than just unit tests. I'm using it for unit tests for things, but also large system tests. Come up with just a...

A simple example. This is just a dumb example. I've never done financial accounting software, but let's make something up. So let's say I've got something that has multiple accounts. I've got two money accounts and I have a, I want to test the function that transfers money between accounts. So let's say I've got account A and account B. Both of them have $10 in each account. I'm setting it up with given when then for my test. Given two accounts.

A and B, $10 in each account. When I transfer $10 from A to B, then A should have $0 in it and B should have $20 in it. The then statement is A should have 0 and B should have 20. That's two checks. So how do I represent that with one assert statement?

I could grab the balance of A and the balance of B and stick both of them into variables and then put those two variables like bal A and bal B into a tuple and compare that to another tuple that's got 0 and 20 in it. I could do that. That's one assert for checking two values. And I do that occasionally. However, I don't think that's naturally how we think. So I think...

I think sticking those two checks of checking the balance of A, that should be zero. Checking the balance of B, that should be 20. I think it reads well if you have them in order. All right, so that's a silly little example. For a real world example, I often use it for testing waveforms with RF test equipment. So like,

going to simplify this as well but let's say the behavior is correct if a signal is found at a certain frequency at a certain level with characteristics x y z being certain values so that's like five different checks frequency level and like three characteristics

And I want to, if something's off, I want to see what all the values are. I want to know what the frequency was. I want to know what the level was. I want to know the XYZ characteristics. Even if one of them is off, I'd like to know all of them. That's really my reason for putting this together is to have a plugin to check multiple things and report multiple results.

Thanks to everyone who supported the show via Patreon or by learning PyTest from a course at courses.pythontest.com. I've made a change recently that I'm pretty excited about. The Complete PyTest Course is now the Complete PyTest Course Bundle. It was one big course and is now three courses.

since you really need them at different times in your PyTest journey. Part 1, PyTest Primary Power, provides a gentle introduction up through covering the superpowers of PyTest, including fixtures, parameterization, markers, and more. Grab primary power and get to work. Part 2, using PyTest with projects, has strategic topics like debugging, mocking, continuous integration. Part 2 is perfect for when you're applying PyTest to a work or open source project. Part

Part 3: PyTest Booster Rockets, Explore's Plugins, both third-party and building your own,

and advanced parameterization. Although part three can be taken right after the other two, it makes more sense to live with pytest for a while and then explore what more you can do with it. I've also added new intro videos at the beginning of each course, congratulations videos at the end, and printable certificates. Anyway, I think the new structure makes a lot more sense. These are all at courses.pythontest.com. That's all for today. Thanks for listening. Now go ahead and test something.