Building and configuring boost in Visual Studio (MSBuild).
4.94/5 (21 votes)
Integrating third party tools and libraries into Visual Studio (MSBuild) configuration environment.
Introduction
Boost is a very popular open source C++ library. Recently I had to integrate it into Visual Studio projects as part of the build. It proved to be very difficult to properly configure build with just a Makefile project and Command Line tool. It also took me forever to find all the relevant information and learn how to use it. The documentation is scattered across multiple locations (here, here, and here) and not always apparent. I am writing this in hope that others will not need to spend days and weeks going through the same issues as I have.
This is a third article on the subject. In firsth two articles I've covered basics of the Property Pages and it's schema. You could read them here and here. In the fourth article I'll explain how to integrate with Visual Studio project reference system so Boost could be used as dependency by any other MSBuild project. Fifth artice covers adding dependency references to the Boost library. I'll cover how to add reference to zlib library via project reference system.
Purpose
The purpose of this article is to demonstrate how to use MSBuild to create projects for custom libraries and tools and integrate it into Visual Studio configuration environment. The later aspect is very important. These tools and libraries sometimes very tricky to build. I will give you a small example:
Let's say we want to build Boost in minimal configuration, put all the intermediate files into "c:\Temp\" directory and stage the binaries. Very simple command:
b2 --stagedir="c:\Temp\" --build-type=minimal stage
Executing the command will fail with this error:
The value of the --build-type option should be either 'complete' or 'minimal'
What the #$%# ?!! We check the command, spelling and such, and try again ... and again ... and again with complete instead of minimal. ...and it still fails!
The reason it fails is because --stagedir has "\" character at the end of its path! Executing it like this "c:\Temp" successfully builds the library:
b2 --stagedir="c:\Temp" --build-type=minimal stage
Visual Studio requires back slash at the end of a folder path and complains if these are not properly terminated. So for Windows developer it would be absolutely normal to put it there and it will get them in trouble every single time. I am pretty sure it is mentioned in documentation somewhere that it should not contain trailing slash but seriously, who reads documentation from cover to cover?
So integration with development environment and providing simple UI which could take care of all this nonsense should eliminate most of these idiosyncrasies.
Background
Boost is a set of libraries for the C++ programming language. Most of the libraries implemented as header files and could be used without building binaries. Including proper headers into your project will be sufficient to add these modules. But some of the modules require building and linking with the application in order to integrate.
Boost uses its own implementation of Jam build tool for building of library's binaries. The tool comes as set of C++ source files and requires build before any of the modules could be processed.
Getting the library
Boost library is available for download from official distribution or it could be cloned or forked from GitHub.
Library downloaded as one of the official releases comes as an archive. It has the entire source and include files and directories in place and could be built immediately.
If library is cloned using Git it might require few more steps in order to be built. Boost is structured as main module with several submodules.
| 1. Clone Boost | Get the library files to your local drive. |
| 2. Get submodules | Get all submodules by executing Update command. It needs to be executed recursively to get all the libs into cloned Boost. |
| 3. Build Jam Tool | Jam interpreter "b2.exe" needs to be built from source using available tool-set, MSVC in this case. |
| 4. Rebuild include dir |
When library cloned from repository, 'boost' include folder is not created. All headers are stored in respective folders under libs directory. This is done to prevent duplicates in 'boost' and 'libs' directories. Instead, 'boost' directory is created with hard links to the original files by executing command |
Once library is downloaded/cloned we can start building it. It is advised to set BOOST_BUILD_PATH environment variable pointing to the root directory of boost library.
Project File
All projects in MSBuild started equal. It always starts like this:
<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
</Project>
There are few expected (but not required) sections ProjectConfigurations, Globals, ExtensionSettings, UserMacros, and ExtensionTargets. For more information about these sections see this blog.
ProjectConfigurations
ProjectConfigurations is an ItemGroup where Visual Studio configuration manager stores available Platform and Configurations. This section is created and manipulated by Visual Studio Configuration Manager:
<ItemGroup Label="ProjectConfigurations">
<ProjectConfiguration Include="Debug|Win32">
<Configuration>Debug</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Debug|x64">
<Configuration>Debug</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|Win32">
<Configuration>Release</Configuration>
<Platform>Win32</Platform>
</ProjectConfiguration>
<ProjectConfiguration Include="Release|x64">
<Configuration>Release</Configuration>
<Platform>x64</Platform>
</ProjectConfiguration>
</ItemGroup>
Globals
As you might already guessed it stores settings global to the whole project. This section is created either by template or manually and is not modified by Visula Studio directly. It usually has following elements:
<PropertyGroup Label="Globals">
<ProjectGuid>{9cd23c68-ba74-4c50-924f-2a609c25b7a0}</ProjectGuid>
<ProjectName>boost</ProjectName>
<BoostDir>$(BOOST_BUILD_PATH)</BoostDir>
</PropertyGroup>
It is important to note that element ProjectGuid has to be unique GUID across all projects in the solution.
We will add BoostDir variable pointing to directory set by BOOST_BUILD_PATH environment variable.
ExtensionSettings
ExtensionSettings and UserMacros are not required at the moment so we could leave them empty:
<ImportGroup Label="ExtensionSettings" />
<PropertyGroup Label="UserMacros" />
The only part left for us to do is to actually implement build targets and configuration property sheets for the project. It is a good programming style to leave project file and define all targets and tasks in a separate file with extension .targets. So we create boost.targets and import it into the project:
Imports
This is all the settings we need at the moment. Next comes import of default definitions for C++ project. This import is not required but it is one of these 'nice to have' things:
<Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
ExtensionTargets is where we add our code to the project. MSBuild recommends keeping .vcproj file for the settings and define Targets and Tasks in .targets files. We will move all the code into boost.targets file and include it as this:
<ImportGroup Label="ExtensionTargets">
<Import Project="boost.targets" />
</ImportGroup>
And we are done with the project!
Targets
Now we need to define what should be done during build and we do it by defining targets. Visual Studio has set of predefined targets with names: Build, Rebuild, and Clean:
<Target Name="Build" /> <Target Name="Rebuild" /> <Target Name="Clean" />
These are executed for Build, Rebuild and Clean operations respectively. We also remember that before we build boost we have to build Jamfile tool b2.exe and repopulate 'boost' include directory if cloned with git. We could do that by addind prerequisites to Build target with DependsOnTarget attribyte.
<Target Name="Build" DependsOnTargets="JamToolBuild;BoostHeaders;" >
<Target Name="Rebuild" DependsOnTargets="JamToolBuild;BoostHeaders;" >
<Target Name="Clean" DependsOnTargets="JamToolBuild;" >
To build Jamfile Tool b2.exe we need to execute build.bat located at "tools\build\src\engine\". We do that in target BuildJamTool:
<Target Name="BuildJamTool" Label="Building BJAM engine" >
<Exec Command="call build.bat" ... />
<ItemGroup Label="List of builtt exe files" >
<BJamTools Include="*.exe" />
</ItemGroup>
<Copy SourceFiles="@(BJamTools)" DestinationFolder="$(BoostRoot)\" />
</Target>
This target executes build.bat and copies *.exe files to boost root dir.
Now we can run build commands.
If boost project was cloned from Git repository 'boost' directory with all the include files will be missing. To populate it with headers we have to execute Jam tool like this "b2.exe headers". We do it in BoostHeaders target:
<Target Name="BoostHeaders" DependsOnTargets="JamToolBuild" >
<Exec Command="b2.exe headers" />
</Target>
Note that it depends on b2.exe being already built. Se we add DependsOnTargets="JamToolBuild" to make sure b2.exe is built and could be executed.
Now we are ready to build the whole library:
<Target Name="Build" DependsOnTargets="JamToolBuild;BoostHeaders;" >
<Exec Command="b2.exe" WorkingDirectory="$(BoostRoot)" />
</Target>
<Target Name="Rebuild" DependsOnTargets="BuildJamTool;">
<Exec Command="b2.exe -a" WorkingDirectory="$(BoostRoot)\" />
</Target>
...
Done.
Now comes the fun part: configuring Boost and integrating this configuration into Visual Studio property system.
Configuration
To integrate Boost into Visual Studio we need to create property pages which would allow us to set all the build options and switches. We can do it by creating XML file with ProjectSchemaDefinitions and Rules.
User Interface
For in-depth coverage of the topic please see my other articles: Part 1 & Part 2. Part One explains structure of property page schema and Part Two describes how to get these elements into a Property Page.
We start by creating boost.xml and adding it to the project. After we are done with adding all the properties we should see something like this:
This page would allow us to set the build options mentioned in Jamroot. If option has nothing defined (empty cell) the builder will use default value set in Jamfile. The settings are stored for each configuration individually and are completely independant of each other.
It is worth noting that each setting has short description shown below describing what it does. If more information is required pressing F1 will open URL with more in-depth explanation.
My personal favorite is Output Verbosity setting:
It allows you to choose how much information is displayed during the build and what information is displayed. These levels are not shown in default help screen for the b2 and are not usually apparent.
Selecting Libraries category will let us specify which libraries are included in the build. It only lists libraries requiring compilation and linking. If library implemented in header files and does not require to be built it will not be listed here.
Compression category will let you specify location of BZip2 and ZLib code or binaries:
And finally you could check all of your build switches on Command Line view:
The command line view also allows you to specify additional switches or options by typing them up into Additional Options box. It will automatically add them to the command line.
Tools Configuration
What if you want to tweak compiler or linker options. Disable a warning, or pass extra definition?
You can do it easily by passing parameters to compiler and linker via cxxflags and linkflags respectively.
Visual Studio comes with set of Property Pages for all built-in tools including compiler and linker. These property pages have all the correct switches already defined. We could just look at these and use them as a base for cxxflags and linkflags property pages.
CXXFLAGS
Compiler flags are defined in CL.XML located at C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\...\1033 directory. There a lot of settings which would not apply when building Boost but some are useful. We can create cxxflags.xml and copy all the relevant properties there. Once we are done and properly added the file to the project we should have something like this:
LINKFLAGS
We do the same with Linker property page LINK.XML and add it to project linkflags.xml:
Passing Configuration to Build Tool
Creating of XML property page is only half of the job. Now we need to pass these settings to the build tool. We need to get data from variables, format it properly and pass it alone to build command.
Normally we would do it in code by assigning values of variables alone with switch configuration to the build command line. But we have a better way:
Formatting data is already setup in XML file. All we have to do is to retrieve it and apply to project variables. One way of doing that is described in this article. But there is a better way. When it comes to processing XML data nothing can outperform XSLT so we will employ XSLT transform to do the job.
Options Integration
Up to this point everything was simple. Property pages, switches, command line...
Now we have to build hierarchical structure with C/C++ and Linker options being included as cxxflags and linkflags command line parameters in this format:
... cxxflags="/cxx1 /cxx2='Some dir here' ... " linkflags="/L1 /L2='Some dir here' ... "
We can do it by iterating through list of pages, building configuration options for each of them and concatenating results with proper prefixes (cxxflags, linkflags) added if required. We do it in target PrepareForBuild:
<Target Name="PrepareForBuild" Returns="@(boost-options)" >
...
<MSBuild Targets="ConfigHelper"
BuildInParallel="true"
Properties="PropertyPageSchema=%(PropertyPageSchema.Identity);
Context=%(PropertyPageSchema.Context)" >
<Output ItemName="boost-options" TaskParameter="TargetOutputs"/>
</MSBuild>
<ItemGroup Label="Build Settings">
<boost-options Condition="'$(AdditionalOptions)'!=''" Include="$(AdditionalOptions)" />
</ItemGroup>
In this code we instruct MSBuild to call ConfigHelper for each element in Item PropertyPageSchema passing it path to xml file and data stored in Metadata Context as a parameter. Returned value is stored in an Item boost-options. Following code adds all the switches defined in Additional Options box and list returned as a string of switches separated by space.
ConfigHelper is where all the magic takes place. It process each XML file in a few steps.
First it gets a list of all properties:
<XmlPeek XmlInputPath="$(File)" Query="/ns:Rule/*[not(contains(@IncludeInCommandLine, 'false'))]/@Name|
/ns:ProjectSchemaDefinitions/ns:Rule/*[not(contains(@IncludeInCommandLine, 'false'))]/@Name">
<Output TaskParameter="Result" ItemName="Names" />
</XmlPeek>
Next it creates a list of all the properties with values:
<data-name-map Include="$(%(Names.Identity))" Condition="'%(%(Names.Identity))'!=''" >
<Name>%(Names.Identity)</Name>
</data-name-map>
Note this notation: $(%(Names.Identity)) It tells the system to return value stored in variable with name which is stored in %(Names.Identity).
If variable contains a list of values separated by semicolon it will treat is as a list and add each of these values individually. For example if we have val1;val2;val3 in variable named xxList it will add it like this:
<data-name-map Include="val1" >
<Name>xxList<Name>
</data-name-map>
<data-name-map Include="val2" >
<Name>xxList<Name>
</data-name-map>
<data-name-map Include="val3" >
<Name>xxList<Name>
</data-name-map>
This effectively creates outer join for the data. Next it generates data elements in form of:
<Property Name="name" >value</Property>
<temp-data Condition="'@(data-name-map)'!='' And '%(data-name-map.Identity)'!=''"
Include="<Prop Name="%(data-name-map.Name)" >%(data-name-map.Identity)</Prop>"/>
And add them to XSL stylesheet like this:
<xsl:variable name="Data" >@(temp-data, '')</xsl:variable>
Once data is added it executes XSL transform:
<XslTransformation Condition="'@(temp-data)'!=''" Parameters="@(xslParameters, '')"
XmlInputPaths="$(PropertyPageSchema)" XslContent="$(raw-xsl)" OutputPaths="$(TempFile)" />
The rest is simple: It reads results of the transform from the file, adds prefixes if necessary and returns the options. After all of them were processed they are added to the command line when build tool is called.
Using the Code
Included archive has complete set of files required to build the project. You should be able to copy it into any directory, specify location of Boost root and build.
This article covers only small part of the code provided. For more informatin see next article on the subject.
Settings for cxxflags and linkflags are not thoroughly tested so proceed with caution. If you found something that does not work, drop me a line. Or you could just send me a pull request at GitHub.
History
03/13/2015 - Published
03/20/2015 - Added code to support project reference