Ways to Improve Your Development Process - Mutation Testing
This is one of those areas that a lot of research is being done but it hasn't necessarily made its way to the average developer yet. But there is an easy way to test our tests without too much effort on our part: Mutation testing.
Mutation Testing #
Mutation testing is pretty simple conceptually speaking. It's a process where faults are automatically added to your code. The addition of these faults makes the resulting code into something known as a mutation and is accomplished by doing things like changing an if statement to be greater than instead of less than or addition instead of subtraction. After we make our change, we run the tests. If your tests fail then the mutation is killed and if your tests pass then the mutation lived.
OK, so how does this help us? Well with this we can measure the quality of our tests by figuring out the percentage of mutations that were killed. The more mutations killed, the better our tests are at finding issues in our code base.
As an example, let's assume we have a code base with traditional unit tests. We can even have 100% branch code coverage in this example. We then make a change in our code and everything passes on the tests BUT as soon as we try to run this thing in production, it crashes and burns. This could be due to issues with libraries that we are linking to or a configuration issue. Similarly it could also be due to us making assumptions in our test suite that pass but are bad assumptions.
In the above code we have no real knowledge of how SomeProperty gets set to 1. Maybe it's simply taking the 1 we sent in and assigning it or maybe it's a more complex formula. Perhaps it's not even based on the value that we sent in. In any case our one test may pass but it's probably not that great of a test. Mutation testing would help point this out to us. So what's the pros/cons of mutation testing?
It has been used with success to find bugs in code and is generally better than alternative methods for finding issues with unit tests.
- It's slow. Really slow sometimes, with much of the research being aimed at speeding up the process.
- There is potential for getting fatigue from duplicate issues being reported to the developer. A real issue and area of research is finding effective ways to detect equivalent and redundant mutants. Not doing this leads to developers being inundated with the same info over and over again.
So considering the aim of this type of testing and the speed issues, this isn't something that you would run every build. This is more of a check up on your code base that you run late at night or over a weekend to see if it kicks up anything. That said, it's usually fire and forget until it finishes so the trade off is usually worth it.
Mutation Testing Tools #
In the case of C#, it installs as a dotnet tool and from there you just run this from your unit test library:
And that's it. It will then go and try to find issues with your code. With that you have mutation testing now in your pipeline and another tool to help reduce issues in your code.
Items in the Series #
Sign up for More Posts
I have no idea why you'd put yourself through that but if you'd like more posts from me then how about using my RSS feed? Take a look.