Unit Tests for Code that is Supposed to Not Compile

Introduction

Unit testing exercises all the parts of the code. It is useful to make sure you really instantiated every template you wrote, the compiler accepts usage as intended, and that the functions do what they are supposed to.

However, sometimes you want to make sure something does not compile. This can be handy static_assert statements made to check template classes for suitable parameters. But of particular importance is to make sure that your template functions are not too greedy in overloading, which can create ambiguity.

If your converting constructor is only supposed to appear if U is convertable to T, then test that.

You should especially watch out for things that are commonly overloaded by different types, such as relational operators, to_string, etc.

Technique

The Detection Idiom is a general way to detect whether something would compile, and acting on that result rather than actually compiling it. That is normally used to tell whether some template function should be enabled, but the concept is perfect for the testing problem!

In this example, I have templates that allow various types to be compared using operator==. But, I have specifically defined cases where they may not be compared, and I want to make sure the template is not over-eager in what it accepts, which will cause ambiguities when other operator== templates are present in the program. So, as well as testing the cases that can be compared (and that they compare as expected), I will test both cases that should not compile.

Setting up the Test

To use the detection idiom, you supply a fake expression containing the operations of interest. It needs to be a template, so the actual legality cannot be determined until the template is instantiated. Here I want to see if == can be used between an object of type R1 and an object of type R2.

template<typename R1, typename R2>
using can_eq = decltype( std::declval<R1>() == std::declval<R2>() );

The enclosing decltype is what makes it fake. This opens up the magic of unevaluated contexts. It says “tell me what type the result of this would be”, but does not generate code to actually do that.

The std::declval template is a fake function that returns the specified type. That way, I can get a value of type R1 (and R2) to use for the expression, without needing any real values. Before declval was standardized, you would see people doing ad-hoc things like (*(R1*)(0)).

Performing the Tests

There are two general approaches. The first is to use the detection idiom to produce a boolean value, and then supply that result to the testing framework. Here are a couple of normal tests and then the does-not-compile test:

REQUIRE (b1 == r4);
REQUIRE_FALSE (b1 == r5);

constexpr int A10[] = { 0,1,2,3,4,5,6,7,8,9 };
// nothing to do with my types
// bool z1 { A10 == r1};  // compile-time error
constexpr bool zz1 = D3::is_detected_v<can_eq, decltype(A10),decltype(r1)>;
REQUIRE_FALSE (zz1);

Notice that I have to issue the request in a round-about way:  Instead of A10==r1, I have to say “do the == check with values that are the type of A10 and the type of r1.”

This probably won’t work with the testing framework’s macros, so do it on a normal code line and then CHECK the result. The advantage of putting the tests here is that failures will show up in the report along with all the other (normal) tests.

However, this is compile-time testing, so we can report the error at compile time. The second way is to use static_assert.

static_assert (!(D3::is_detected_v<can_relate, decltype(Begin(A2)), 
                 decltype(buf_view)>), "wrong types");

Also, note that one detection idiom predicate can gang together multiple tests. The above line uses:

template<typename R1, typename R2>
using can_relate = decltype( (std::declval<R1>() 
< std::declval<R2>()) || (std::declval<R1>() > std::declval<R2>())
    || (std::declval<R1>() <= std::declval<R2>())  || 
    (std::declval<R1>() >= std::declval<R2>()) );

This checks whether all the relational operators are legal between the two types. Never mind that the expression itself does not make sense; we are just checking whether it would compile.

Do You Have Detection Idiom Support?

As I write this, some compilers have is_detected et al. in a header named <experimental/type_info>. Some do not. Over time, the templates will move into the regular <type_info> header. So, depending on your specific compiler and version, and when you read this, you may have std::is_detected, std::experimental::is_detected, boost::is_detected, or your own copy added to your code.

To cope with this mess and moving target, I have made a header file that uses either <type_info>, or <experimental/type_info>, or supplies the templates directly. In all cases, it aliases them to its own namespace, so you can refer to it there regardless of which namespace they actually were defined in.

As the compiler is updated, I just need to change a line in the header and all code that uses it will switch over.

The headers (and the unit tests that these examples were drawn from) can be found in GitHub (direct link to file).