Unit Test Frameworks

If you ask a group of 10 software engineers to develop unit tests for the same object, you will end up with 10 unique approaches to testing that object. Now imagine each engineer was given a different object. This was my experience with unit testing before I discovered how useful and how much more valuable your unit tests become when they are developed with a test framework. A unit test framework provides a consistent foundation to build all of your tests upon. The method to setup and run each and every test will be the same. Most importantly, the output provided by each test will be the same, no matter who developed the test.

In a previously posted an entry called, The Purpose of a Unit Test, I clarified the benefits of writing and maintaining the tests, who does the work, and why you should seriously consider developing unit tests as well. I mentioned unit test frameworks and test harnesses, but I did not provide any details. In this entry I give a high-level overview of unit test frameworks and a quick start sample with the test framework that I use.

Unit Test Frameworks

Similar to most any tool in our industry, unit test frameworks can incite heated debates about why "My tool is better than yours." For the first half of my career I used simple driver programs to exercise functionality of a new object I was developing. As development progressed, I depended on my test driver less and less, and eventually the driver failed to compile successfully. I had always considered writing a driver framework to make things easier, however, testing was not something I was extremely interested in up to that point in my career.

Once I discovered a few great books written about unit testing, I quickly learned of better approaches to testing software. Some of the new knowledge I had gained taught me that writing and maintaining unit tests does not have to be that much extra work. When I knew what to look for, I found that a plethora of high quality test frameworks already existed. xUnit is the style of test framework that I use. To my knowledge SUnit is the first framework of this type, which was written for Smalltalk by Kent Beck. JUnit seems to be the most well known framework that tests JAVA.

A very detailed list of test frameworks can be found at Wikipedia. The number of frameworks listed is enormous, and it appears there are for frameworks to choose from for C++ than any other language. I assume this is due to the variety of different C++ compilers, platforms and environments C/C++ is used in. Some of the frameworks are commercial products, most are released as open source tools. Here are a few of the frameworks that I am familiar with or I have at least inspected to see if I like them or not:

• Boost Test Library
• CppUnit / CppUnitLite
• CxxTest
• GoogleTest
• Visual Studio

Qualities

There are many factors could play into what will make a unit test policy successful with your development team; some of these factors are the development platform, the type of project and even the culture of the company. These factors affect which qualities will be valuable when selecting a software unit test frameworks. I am sure there are many more than I list below. However, this is the short list of qualities that I considered when I selected a test framework, and it has served me well.

xUnit Format

There is more literature written on the xUnit test frameworks than any other style that exists. In fact, it appears that frameworks are either classified as xUnit or not. This is important because this test paradigm is propagated across many languages. Therefore, if you program in more than one language, or would like to choose a tool that new developers to your company will be able to come up to speed the quickest you should select an xUnit-style framework.

Portability

Even though Visual Studio is my editor of choice and some really nice test tools have been integrated into it with the latest versions, I have chosen a framework that is not dependent on any one platform. I prefer to write portable code for the majority of my work. Moreover, C++ itself is a portable language. There is no reason I should bound all of my tests to a Windows platform that requires Visual Studio.

Simplicity

Any tool that you try to introduce into a development process should be relatively easy to use. Otherwise, developers will get fed up, and create a new process of their own. Since unit tests are thought of by many as extra work, setting up a new test harness and writing tests should not feel like work. Ease of use will lower the resistance for your team to at least try the tool, and increase the chances of successful adoption.

Flexibility

Any type of framework is supposed to give you a basis to build upon. Many frameworks can meet your needs as long as you develop the way the authors of the framework want you to. Microsoft Foundation Classes (MFC) is a good example of this. If you stick with their document/view model you should have no problems. The further your application moves away from that model, the more painful it becomes to use MFC. When selecting a unit test framework you will want flexibility to be able to write tests however you want. Figuring out how to test some pieces of software is very challenging. A framework that requires tests to be written a certain way could make certain types of tests damn near impossible.

I suggest that you do a little bit of research of your own, and weigh the pros and cons for each framework before deciding upon one. They all have strengths and weaknesses, therefore I don't believe that any one of these is better than the others for all situations.

CxxTest

I have built my test environment and process around the CxxTest framework. When I chose this one, I worked with a wide variety of compilers, which many had poor support for some of C++'s features. CxxTest had the least compiler requirements and the framework is only a handful of header files so there are no external libraries that need to be linked or distributed with the tests. Many of the test harnesses developed for C++ early on required RTTI; this feature definitely wasn't supported on the compiler that I started using CxxTest. The syntax for creating unit tests is natural looking C++, which I think makes the tests much easier to read and maintain. Finally, it is one of the frameworks that does not require any special registration lists to be created to run tests that are written; this eliminates some of the work that would make testing more cumbersome.

The one thing that I consider a drawback is that Python is required to use CxxTest. Adding tools to a build environment makes it more complex, and one of my goals has always been to keep things as simple as possible. Fortunately, Python is easily accessible for a variety of platforms, so this has never been a problem. Python is used to generate the test runner with the registered tests, which eliminates the extra work to register the tests manually. When you work on large projects and have thousands of tests, this feature is priceless.

CxxTest is actively maintained at SourceForge.net/CxxTest

Quick start guide

I want to give you a quick summary for how to get started using CxxTest. You can access the full documentation online at http://cxxtest.com/guide.html. The documentation is also available in PDF format. There are four basic things to know when starting with CxxTest.

1. Include the CxxTest header

All of the features from CxxTest can be accessed with the header file:

C/C++

2. Create a test fixture class

A test fixture, or harness, is a class that contains all of the logic to pragmatically run your unit tests. All of the test framework functionality is in the class CxxTest::TestSuite. You must derive your test fixture publicly from the TestSuite class. You are free to name your class however you prefer.

C/C++

3. Test functions

All functions in your fixture that are to be a unit test must start with the word test. This is a case-insensitive name, therefore you can use CamelCase, snake_case, or even _1337_Ca53 as long as the function name starts with "Test" and it is a valid symbol name in C++. Also, test functions must take no parameters and return void. Finally, all test functions must be publicly accessible:

C/C++

4. (Optional) Setup / Teardown

As with the xUnit test philosophy, there exists a pair of virtual functions that you can override in your test suite, setUp and tearDown. Setup is called before each and every test to perform environment initialization that must occur for every test in your test suite. Its counterpart, Teardown, is called after every test to cleanup before the next test. I highly encourage you to factor code out from your tests into utility functions when you see duplication. When all of your tests require the same initialization/shutdown sequences, move the code into these two member functions. It may only be two or three lines per test, however, when you reach 20, 50, 100 and more a change in the initialization of each test becomes significant.

C/C++

Short Example

I have written a short example to demonstrate what a complete test would look like. Reference the implementation for this function that indicates if a positive integer is a leap year or not:

C/C++

Each individual unit test should aim to verify a single code path or piece of logic. I have marked each of the code paths with a numbered comment in the IsLeapYear function. I have identified four unique paths through this function, and implemented four functions to test each of the paths.

C/C++

Before this test can be compiled, the python script must be run against the test suite header file to generate the test runner:

python.exe --runner=ParenPrinter 
           -o IsLeapYearTestRunner.cpp 
           IsLeapYearTestSuite.h

Now you can compile the output cpp file that you specify, along with any other dependency files that your test may require. For this example simply put the IsLeapYear function in the test suite header file. This is the output you should expect to see when all of the tests pass:

Running 4 tests....OK!

If the second test were to fail, the output would appear like this:

Running 4 tests.
In IsLeapYearTestSuite::TestIsLeapYear_True:
IsLeapYearTestSuite.h(19):
        Error: Expected (true == IsLeapYear(2016)), 
               found (true != false)
..
Failed 1 of 4 tests
Success rate: 75%

When I execute the tests within Visual Studio, if any errors occur I can double click on the error and I am taken to the file and line the error occurred, much like is done for compiler errors. The output printers can be customized to display the output however you prefer. One of the output printers that comes with CxxTest is an XmlPrinter that is compatible with the Continuous Integration tools that are common, such as Jenkins.

Summary

I believe that unit tests are almost as misunderstood as how to effectively apply object-oriented design principles, which I will tackle another time. It seems that most everyone you ask will agree that testing is important, a smaller group will agree that unit tests are important, and even fewer put this concept to practice. Select a unit test framework that meets the needs of your organization's development practices to maximize the value of any tests that are developed for your code.