Mutation Testing - 9 Lessons Learned

Mutation testing is amazing. It still shocks me that it has not caught on despite being around for decades. This post is in regards to the value add that comes from doing mutation testing.

Don't know what mutation testing is? Check out these posts:

• Can your tests survive the coming mutant apocalypse?
• Java Mutation Testing with PIT

Mutation Testing 101

A mutation is a single syntactic change to the code. Each mutant is different from the original by a single mutation. These small changes can have big results; such as changing an operator from addition to subtraction. Then your unit tests execute against the mutant code and in theory our tests should fail because the code is wrong.

public int additionOrSubtraction(boolean isAddition, int x, int y) {
    if (isAddition) { //mutant: replace isAddition with !isAddition
        return x + y; //mutants: replace + with -,*,/,%
    } else {
        return x - y; //mutants: replace - with +,*,/,%
    }
}

Lesson #1: Mutation Testing Improves Code Reviews

Lesson #2: Not All Mutations Add Value

Lesson #3: Start Small

> mvn org.pitest:pitest-maven:mutationCoverage -DmutationUnitSize=2

Lesson #4: Laziness Creates Surviving Mutants

This lesson was learned from the same mutator that caused the example in lesson #2. Not verifying the results of void method calls causes more mutations to fail than almost anything else. In many cases, this is just due to pure laziness. I just found a project where all builder method calls to populate POJOs only had an assert to make sure the result was not null. So the tests did not care what was populated in those beans and just focused on them not being null. This is a perfect example of having 100% code coverage but the tests themselves being worse than if they did not exist at all. Do yourself a favor and verify every element in the objects returned. I know it takes time and many asserts but that extra effort may save you from a production bug born out of laziness.

Lesson #5: People Won't See the Value of Mutation Testing at First and That is Okay

Mutation testing sounds interesting to people as a theoretical exercise but the benefits of it are often hard to see. It takes a mature organization that is already dedicated to a strong unit test base to even consider adopting it. To get it adopted you, yes you, need to be the zealous advocate for mutation testing. Just do it. Update your builds and put a mutation testing job in your continuous integration tool like Jenkins. If your team hopes to do continuous delivery, you need a rock solid test base. Start pointing out mutations that result from recent commits just like you would if the commits result in bad code coverage. Do not try to convince people of the value of mutation testing but instead show them the lack of defects that could result from weak tests.

Lesson #6: Mutation Testing is Not Just for Java

Although the mutation testing tool I am outspoken about is for Java, that doesn't mean that there are no frameworks for other languages.

(I have not used any of the following tools and am therefore unable to attest to their value.)

• Javascript - Grunt Mutation Testing
• PHP - Humbug
• Ruby - Mutant
• PHP - Mutagenesis
• .Net - NinjaTurles
• Java - The Major mutation framework (not as good as PIT but competition is good)
• C# - Nester
• Javascript - Mutandis

Lesson #7: Be Relentless in Advocating for Mutation Testing

Mutation testing has been around since 1971. The days have long passed when the computer power is not available to make mutation testing a viable tool. If you don't push for it to be adopted, it likely won't because we haven't reached that critical mass to give it a life of its own. So do it and email out the reports. Do a lightning talk or brown bag session on mutation testing to get the word out. Honestly, I don't care if you even need to borrow heavily from or steal the Mutation Testing Slideshow that I recently created. Get the word out.

Lesson #8: 100% Mutation Kill Score is Possible But Expensive

As previously stated, some mutations are silly, not really useful but there are times when there is no margin for error and your code needs to work. Say for example, you are working on code for a medical device. There is no room for error when someone's life is on the line. You need mutation testing to do everything you can to be defect free in cases such as that. However some mutants just aren't worth the time needed to kill them. It is similar to writing unit tests to test all getters and setters in POJOs. Is it really worth your time to write tests for basic getter and setter methods. I don't know that is a personal decision but squashing some mutants might add even less value than those unit tests.

Lesson #9: Mutation Testing Makes You A Better Programmer

Knowing the ways your code could break and how good your tests need to be to protect against that is not a skill that is developed overnight. Mutation testing ensures that you don't cut corners and take the easy route. Write solid code and bombproof unit tests.

Links to Other Posts Related to Unit Testing and Mutation Testing

• 13 steps to make TDD actually work
• Can your tests survive the coming mutant apocalypse?