Organizing Unit Tests inline with code.

Introduction

In my personal life, I'm something of an amateur outdoor griller.  About a year ago a friend of mine gave me his old kamado smoker and I was instantly hooked.  One of the things I love most about it (aside from cooking fabulous food) is how much of an art there is to outdoor cooking.  For any given recipe there's never a single "right" way to cook it; experimentation is key.  I look at software engineering very much from the same lens.  There are often many different ways to solve any given problem; and although some are certaintly "wrong", there are often more than one "right" way.

As with most things in life, I like to ask "why?" and "what if?", often in that order.  A few days ago I was refactoring some code for work and had one of those moments.  "Why?", I asked, do we always create separate projects for our unit tests, and "What if we didn't?  What if we wrote our tests inline with our code?".

Somewhere out in Internet land there's an angry developer who just banged their fist against their desk and shouted "blasphemy!", but hear me out here.  I'm not advocating that "Test" projects are "wrong", but suggesting that perhaps there's more than one "right" way to solve this problem.

The Concern with Test Projects

While test projects are standard-fare in almost all projects, they aren't free.  For starters we're creating an additional library that now needs to be built along-side our solution.  The layout of this library typically should follow the layout of the library it's testing (e.g. namespaces and class names should match).  The library also needs to stay in sync with the library being tested framework-wise (we typically want to match .net framework), and if the test library houses tests for multiple libraries, then we're responsible for ensuring that it references every dependent library that it's targeting library references.  If we're not using a single test project then we essentially double the number of projects in our solution by having to create a new test project for every library/app in our solution.

From a developers perspective (especially one trying to adhere to TDD), we now have to create two classes in two separate libraries whenever they want to write new code, and should subsequent refactoring cause the class to move within a library (or to another library all together) care must be taken to make sure that the test project reflects this.

This is not to say that having test projects is a bad idea.  Many people use them every single day with very little drama.  The point here is to highlight the fact that there is a cost to having them.  One that most developers seldom think about.

What if we Got Rid of Test Projects?

Let's say we eliminated the test project from our solution.  What would that even look like?  To explore this idea I created a simple library that contained a single class: Calculator (trite, I know).  The code looks like this:

namespace CalculatorLib
{
    using Xunit;

    public class Calculator 
    {
        public int Add(int a, int b)
        {
            throw new NotImplementedException();
        }
   
        public class Tests
        {
            // Create a sub-class for each method we want to test for organization.
            public class Add : Tests
            {
                [Fact]
                public void OnePlusOneEqualsTwo()
                {
                    // Arrange.
                    var calculator = new Calculator();

                    // Act.
                    int result = calculator.Add(1, 1);

                    // Assert.
                    Assert.Equal(2, result);
               }
            }
        }
    }
}

Quote:

Note: I'm using xunit for this example.  However one can just as easily have substituted nunit instead.

As we can see -- the unit test is now inline with the class itself.  There's no need for a separate class for them.  Also note the name of the class itself -- 'Tests'.  Since this is an inner class we don't need to qualify what kind of test class this is.  Looking at this in reflector you'll notice that the class itself is named "Calculator+Tests".  In fact, the same idea carries through all the way to the individual tests themselves.  So our single unit test would actually have the qualified name "CalculatorLib.Calculator+Tests+Add.OnePlusOneEqualsTwo()".  In other words, the tests are self-organizing.

But I Don't Want to Ship Test Code!

It should be pretty obvious from the previous code sample that when the library as built the tests will also be included with them, as will the referenced xunit dlls (more on this later).  How do we get rid of this?  Probably the simplest way to do this would be to use conditional compliation.  Since unit tests are typically run in debug code, and since it's rare for a company to ship debug code out with an official product, we can simply demarcate our unit tests with a few well-placed #if/#endif blocks.  Let's update the previous code example to demonstrate this:

namespace CalculatorLib
{
#if DEBUG
    using Xunit;
#endif

    public class Calculator 
    {
        public int Add(int a, int b)
        {
            throw new NotImplementedException();
        }

#if DEBUG
        public class Tests
        {
            // Create a sub-class for each method we want to test for organization.
            public class Add : Tests
            {
                [Fact]
                public void OnePlusOneEqualsTwo()
                {
                    // Arrange.
                    var calculator = new Calculator();

                    // Act.
                    int result = calculator.Add(1, 1);

                    // Assert.
                    Assert.Equal(2, result);
               }
            }
        }
#endif
    }
}

That's it.  When we compile our code in debug mode the tests will be present.  However when compiled in release mode (or any mode where DEBUG is not defined) they will automatically be stripped out.

What About Those Referenced Dlls?

If you look at the 'Release' folder after building the solution, you should notice that even though we've cut out all of our tests, the xunit libraries are still being copied to the output folder.  Furthermore, if you inspect the dll itself with a tool such as JetBrains dotPeek or Redgate's Reflector, it should become clear that even though those libraries aren't actually being used, they're still being referenced.  What we would like to do is somehow configure our project to link against those libraries only in debug mode, and ignore them for any other build configuration.

Unfortunately we can't do this within Visual Studio.  Fortunately this is fairly trivial to do in MSBuild.

Quote:

Note: this section of the article assumes the reader has a basic understanding of msbuild.  The reader does not need to be an expert, but some familiarity with the tool is very valuable.

The first thing we need to do is unload the project and then edit it's csproj in Visual Studio (VS is ideal since it gives intellisense automatically, although any text editor will work).

Scroll down a little and you should see a tag called <ItemGroup />.  Within it you'll see several <Reference /> tags.  If you've followed along it should look somewhat similar to this:

  <ItemGroup>
    <Reference Include="System" />
    <Reference Include="System.Core" />
    <Reference Include="System.Xml.Linq" />
    <Reference Include="System.Data.DataSetExtensions" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Net.Http" />
    <Reference Include="System.Xml" />
    <Reference Include="xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.abstractions.2.0.0\lib\net35\xunit.abstractions.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.assert, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.assert.2.1.0\lib\dotnet\xunit.assert.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.extensibility.core.2.1.0\lib\dotnet\xunit.core.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.execution.desktop, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.extensibility.execution.2.1.0\lib\net45\xunit.execution.desktop.dll</HintPath>
      <Private>True</Private>
    </Reference>
  </ItemGroup>

What we want to do is exclude the xunit references if the configuration is *not* debug.  The simplest way to do this is to put them in their own <ItemGroup /> element and add a condition to that.  For example:

  <ItemGroup>
    <Reference Include="System" />
    <Reference Include="System.Core" />
    <Reference Include="System.Xml.Linq" />
    <Reference Include="System.Data.DataSetExtensions" />
    <Reference Include="Microsoft.CSharp" />
    <Reference Include="System.Data" />
    <Reference Include="System.Net.Http" />
    <Reference Include="System.Xml" />
  </ItemGroup>

  <!-- Instruct MSBuild to include these only in the 'DEBUG' configuration. -->
  <ItemGroup Condition=" '$(Configuration)' == 'DEBUG' ">
    <Reference Include="xunit.abstractions, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.abstractions.2.0.0\lib\net35\xunit.abstractions.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.assert, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.assert.2.1.0\lib\dotnet\xunit.assert.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.core, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.extensibility.core.2.1.0\lib\dotnet\xunit.core.dll</HintPath>
      <Private>True</Private>
    </Reference>
    <Reference Include="xunit.execution.desktop, Version=2.1.0.3179, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c, processorArchitecture=MSIL">
      <HintPath>..\packages\xunit.extensibility.execution.2.1.0\lib\net45\xunit.execution.desktop.dll</HintPath>
      <Private>True</Private>
    </Reference>
  </ItemGroup>

Go ahead and reload the solution and try re-building the project with both 'Debug' and 'Release' configurations.  You'll notice that all of the xunit libraries are copied to the debug folder, whereas they aren't copied to the release folder.

Quote:

Note: If you installed the xunit.visualstudio.runner nuget package you may notice a few xunit dlls in the release folder.  These are not referenced by the project, but rather copied as an artifact of the package itself.  This can easily be confirmed by inspecting the dll in dotPeek or Reflector.

What Did Any of This Actually Get Us?

A couple of things.  To recap:

• We've eliminated the need for a separate library for tests.
• We no longer have to syncronize dependencies between our library and test library.
• It's no longer possible for the tests library to target a different version of .net framework than the library itself.
• Tests are now self-organzing since they're built on top of the class being tested.
• TDD becomes simpler since code and tests are written together in the same place.
• Refactoring codes is now much simpler.  If a class is ever moved the tests automatically move with them.

Can We Take This Further?

Absolutely!  One obvious way would be to take these csproj file changes and put them in their own msbuild file.  That way we can simply <Import /> them into our csproj file and be done with it.  Another idea would be to extract these changes into a powershell script and create a nuget package from them.  That way with a single package you can ensure that all of your libraries use the same testing framework and that the project files can be automatically be modified to support this without any developer intervention.

Final Thoughts

This article is not intended to be a criticism on the standard test library practice that most developers adhere to.  Many companies have used this pattern to great effect with very little drama.  That said, I think it's important to constantly ask "why" we do things a certain way and "what if" we tried a different way.  Just like outdoor grilling, experimentation is key.  Not every experiment will work, but it's important to at least try new things.  Afterall, AAA unit testing was just an "idea" at one point.  I'd love to hear if anyone is interested in trying this.  So far I've only experimented with this on small projects so I'm curious to know how well it scales.