Preprocessor Code Generation

I really do not like MACROs in C and C++, at least the way they have been traditionally used starting with C. Many of these uses are antiquated because of better feature support with C++. The primary uses are inline function calls and constant declarations. With the possibility of MACRO instantiation side-effects, and all of the different ways a programmer can use a function, it is very difficult to write it correctly for all scenarios. And when a problem does occur, you cannot be certain what is being generated and compiled unless you look at the output from the preprocessor to determine if it is what you had intended.

However, there is still one use of MACROs, that I think makes them too valuable for the preprocessor to be removed altogether. This use is code generation in definitions. What I mean by that, is hiding cumbersome boiler-plate definitions that cannot easily be replicated without resorting to manual cut-and-paste editing. Cut-and-Paste is notorious for its likelihood of introducing bugs. I want to discuss some of the features that cannot be matched without the preprocessor, and set some guidelines that will help keep problems caused by preprocessor misuse to a minimum.

• Download Alchemy.zip - 12 KB

Beyond Function MACROs and Constants

There are two operators that are unique to the preprocessor, and allow so much more to be accomplished with MACROs than implement inline functions. The first operator converts the input parameter to a string, and the second operator allows tokens to be pasted together to create a new symbol or declaration based on the input.

String Conversion Operator: #

40-years before that #-tag became a cultural phenomenon, this operator was available in the C preprocessor. This operator is used with MACROs that take arguments. When the # prepends an argument in the MACRO definition, the argument passed to the MACRO will be converted to a string literal. This means it will be enclosed with quotation marks. Any whitespace before and after the string argument will be ignored. Therefore, strings will automatically be concatenated if two or more are adjacent to each other.

Example

#define PRINT_VALUE( p ) \
  std::cout << #p << " = " << p << std::endl;

// usage:
int index = 20;
PRINT_VALUE(index);

// output:
// index = 20;

As the example above illustrates, the string conversion operator provides us with a way to insert the name of symbols that we are programming with into output messages. This allows one argument to provide the name of the parameter as well as the value in the output message.

The C++ Way

If we were solely interested in getting the name of an argument, C++ provides a mechanism to do this, if you have type information enabled. You must include the typeinfo header file. Then you can use the typeid operator to construct an instance of the type_info class. This class has a member called type_info::name that will return the human-readable string name of the symbol

#include < typeinfo >
// ...

int index = 20;
cout << typeid (index)::name() << " = " << index;

// output
// index = 20

Wait There's More

If all I wanted was to print the name of symbols, I would have sucked it up and adopted the C++ way years ago. However, anything that you place in the argument will be converted to a string. This allows this very well known MACRO sequence to be possible:

#define STR2_(x) #x
#define STR1_(x) STR2_(x)
#define TODO(str) \
  __pragma(message (__FILE__"("STR1_(__LINE__)"): TODO: " str))

// usage: TODO("Complete task.")

What does this do? This implementation is specific to Microsoft compilers; it will invoke the #pragma(message) directive, to print a message to the compiler output. Notice that it is not our message that we are converting to a string, but the line number where the message is declared.

That is accomplished with this statement: STR1_(__LINE__). You may be wondering why there are two STR_ MACRO definitions, with one nested inside of the other. That is because __LINE__ itself is a preprocessor MACRO. If we simply made this call STR2_(__LINE__), the literal output "__LINE__" would be created rather than the intended line number. The nested MACRO is a way to force the processor to resolve __LINE__ before converting the result to a string. See, MACROs can be tricky to get correct.

Finally, It will also prepend the text "TODO:" before the supplied message is finally printed. This allows the Task Manager in Visual Studio to recognize that line as a point of code that still requires attention. The final result is a message is printed to the compiler output window, that is clickable, and will take the user directly to the file and line where the TODO message was left.

file.cpp(101): TODO: Complete task

Since I first came across this MACRO for inserting messages on Microsoft compilers, I have adapted it to work for GCC compilers as well. At this point for all other compilers, the MACRO will do nothing.

// Add Compiler Specific Adjustments
#ifdef _MSC_VER
# define DO_PRAGMA(x) __pragma(x)
#elif defined(__GCC__)
# define DO_PRAGMA(x) _Pragma(x)
#else
# define DO_PRAGMA(x)
#endif

// Adjusted Message MACRO
#define MESSAGE(str) \
  DO_PRAGMA(message(__FILE__"("STR1_(__LINE__)"): NOTICE: " str))

I find the # operator is very useful for creating terse statements that improve readability and maintainability. As with any type of solution, always in moderation.

Token Pasting Operator: ##

When you want to create programmatically create repetitive symbols in code, the token-pasting operator, ## is what you need. When this operator precedes a MACRO argument, the whitespace between the previous symbol and the MACRO argument will be removed, joining the two parts together to create one token. If this is not obvious, the ## cannot be the first or last token in a MACRO definition. Here is a simple example to demonstrate:

Example

#define PRINT_OFFSET( n ) \
  std::cout << "offset" #n << " = " << offset ## n << std::endl;

// usage:
int offset2 = 20;
int offsetNext = 30;
PRINT_OFFSET(2);
PRINT_OFFSET(Next);

// output:
// offset2 = 20;
// offsetNext = 30;

This is only a demonstration for how the token-paste operator works. I will demonstrate more valuable uses in the next section.

Generating Code

My philosophy when writing code is, "The best code, is the code that is not written". Meaning, strive for simple elegant solutions. At some point, some code must exist in order to have a program. Generated code is a close second in my preferences. We can use the preprocessor with great effect towards achieving this goal. However, once again, code generation is not a panacea, because it is generally restricted to boilerplate type definitions.

Generally in practice, your software will be easier to build when there are fewer tools involved in the build process. I have written and used code generation programs, and when it is the right tool, it works fantastically. However, there is a threshold of effort that is required for a task to exceed before it becomes worth the trouble of developing and maintaining a new tool, and adding complexity to your build system. In these cases, code generation with the preprocessor is a great solution.

Before I continue, I would like to mention that Boost contains a preprocessor library[^], which is capable of some amazing things. Such as, identifying the number of arguments in a MACRO expression, iterating over the list of arguments, counting and many others. Think of it as meta-programming with the preprocessor. Many of the examples below are simplified and less general solutions to the same sort of capabilities provided by Boost Preprocessor.

Indexing with Recursion

Working with the preprocessor is yet another example of functional-programming hidden inside of C++. We are restricted from using state or mutable data (variables). Therefore, many of the solutions with the preprocessor become solutions that involve recursion, especially if a task must be repeated a number of times.

When I was exploring the Typelist construct a few days ago, I came across this cumbersome declaration:

// Typelist primary template definition.
template < typename T0,         typename T1 = empty, 
           typename T2 = empty, typename T3 = empty, 
           typename T4 = empty, typename T5 = empty,
           typename T6 = empty, typename T7 = empty, 
           typename T8 = empty, typename T9 = empty >
struct Typelist
{
  typedef Typenode< T0,
          Typenode< T1,
          Typenode< T2,
          Typenode< T3,
          Typenode< T4,
          Typenode< T5,
          Typenode< T6,
          Typenode< T7,
          Typenode< T8,
          Typenode< T9, empty > > > > > > > > > >     type;
};

This will be tedious to maintain, and error prone as we will need to update the type definitions anywhere the Typelist is used. For brevity, I only included 10 entries in the demonstration implementation. For my solution in Alchemy, I planned on starting with a minimum support for up to 32 entries. It would be nice if we could take advantage of the preprocessor to generate some of this repetitive code for us. Especially since these are hidden definitions in a library. The user will have no need to explore these header files to determine how to interact with these constructs. That makes me feel even more confident with a solution that depends on the preprocessor.

The first challenge that we will need to overcome is how do we build something that can repetitively process something until a given end-point is reached? This sounds like a recursive solution. To create this effect, we will need a utility file with predeclared MACROs. Boost actually has MACROs create these MACROs. For what I am after, I can get by with a small set MACROs specifically built to suit my purpose.

#define ALCHEMY_ARRAY_1(T)    T##0 
#define ALCHEMY_ARRAY_2(T)    ALCHEMY_ARRAY_1(T),  T##1
#define ALCHEMY_ARRAY_3(T)    ALCHEMY_ARRAY_2(T) , T##2
// ...
#define ALCHEMY_ARRAY_32(T)   ALCHEMY_ARRAY_31(T), T##31

// Repeats a specified number of times, 32 is the current maximum.
// Note: The value N passed in must be a literal number.
#define ALCHEMY_ARRAY_N(N,T)  ALCHEMY_ARRAY_##N(T)

This will allow us to create something like this:

ALCHEMY_REPEAT_N(4, typename T);
// Creates this, newlines added for clarity
typename T0,
typename T1,
typename T2,
typename T3

This will eliminate most of the repetitive template definitions that I would have had to otherwise write by hand. How does the chain of MACROs work? The result of each MACRO is a call to the MACRO of the previous index, followed by a comma, and the input T, token-pasted with the current index. Basically, we are constructing a MACRO as part of the resolution of the current MACRO. This forces the preprocessor to make another pass and resolve this instance. The process continues until the terminating case is reached.

This first version helps in two of the declarations, however, the declaration with the nested Typenodes cannot be solved with the MACRO in this form. The right-angle brackets are nested at the end of the definition. Pasting the index at the end of this expression would no longer be correct. Rather than trying to be clever, and create a MACRO that would allow me to replace text inside of a definition, I will just create a second simpler MACRO that repeats a specified token, N times.

#define ALCHEMY_REPEAT_1(T)    T 
#define ALCHEMY_REPEAT_2(T)    ALCHEMY_REPEAT_1(T) T
#define ALCHEMY_REPEAT_3(T)    ALCHEMY_REPEAT_2(T) T
// ...
#define ALCHEMY_REPEAT_32(T)   ALCHEMY_REPEAT_31(T) T

// Repeats a specified number of times, 32 is the current maximum.
// Note: The value N passed in must be a literal number.
#define ALCHEMY_REPEAT_N(N,T)  ALCHEMY_REPEAT_##N(T)

With these two MACROs, we can now simplify the definition of a Typelist to the following:

// I have chosen to leave the template definition
// fully expanded for clarity.
// Also would have to solve the challenge of 
// T0 not having a default value.
template < typename T0, typename T1 = empty, 
  // ...
  typename T30 = empty, typename T31 = empty>
struct TypeList
{
#define DECLARE_TYPE_LIST  \
  ALCHEMY_ARRAY_32(TypeNode< T ), empty ALCHEMY_REPEAT_32(>)

  typedef DECLARE_TYPE_LIST  type;
};

If you remember I mentioned that an implementation will need to be defined for each specialization of the base template above. We have already developed all of the helper MACROs to help us define those specializations.

#define tmp_ALCHEMY_TYPELIST_DEF(S)              \
template < ALCHEMY_ARRAY_ ##S(typename T)>       \
struct TypeList< ALCHEMY_ARRAY_##S(T)>           \
{                                                \
  typedef ALCHEMY_ARRAY_##S(TypeNode< T ), empty \
    ALCHEMY_REPEAT_##S(>). type;                 \
}

// Define specializations of this array from 1 to 31 elements *****************
tmp_ALCHEMY_TYPELIST_DEF(1);
tmp_ALCHEMY_TYPELIST_DEF(2);
// ...
tmp_ALCHEMY_TYPELIST_DEF(31);

// Undefined the MACRO to prevent its further use.
#undef tmp_ALCHEMY_TYPELIST_DEF

All-in-all, the little bit of work replaces about 500 lines of definitions. We will end up using those a few more times to implement the Typelist utility functions.

Guidelines

These are a few rules that I try to follow when working on a solution. MACROs in-and-of themselves are not bad. However, there are many unknown factors that can affect how a MACRO behaves. The only way to know for sure what is happening, is to compile the file, and inspect the output from the preprocessor. If I ever need to resort to using a MACRO function, I try to hide its use away within another function, and isolate that function as much as possible so that I can be sure it is correct. The list of guidelines below is not a complete rule set, nor are these rules absolute, but they should help build more reliable and maintainable solutions when you need to resort to the power of the preprocessor.

Make them UGLY

I think the convention of ALL_CAPS_WITH_UNDERSCORES is ugly. I also think it helps easily identify MACROs in your code, and as you saw in my definitions, I don't mind creating very long names for MACROs. Hopefully their use remains hidden in implementation files.

Define constants_all_lowercase

This one only comes to mind because I have been bitten by it when trying to port code between platforms. This is another convention that has migrated from the original C days of using #define to declare MACRO constants. However, this could leave you vulnerable to some extremely difficult to find compiler errors in the best case, or code that incorrectly compiles in the worst case.

What do I mean? Think about the result if there was a MACRO defined deep within platform or library header files that clashes with the name you have declared using the CAPITAL_SNAKE_CASE naming convention. Your constant value would be overwritten with the code defined behind the MACRO. In most cases I imagine this would result in a weird compiler error stating cannot assign a literal to a literal. However, the syntax could result in something that is valid for the compiler, but not logically correct. This is one of the potential downsides of the preprocessor.

Restrict MACROs to Declarations

If you limit the context when your MACROs are used, you will have much more control over the conditions that your MACRO can be used, and ensure that it works properly. There is no longer any reason to use a MACRO as a function with inline functions in C++. However, using a MACRO to help generate a complex definition should be much less of a risk, and improves the readability and maintainability tremendously.

One of my favorite examples of this use for MACROs is in Microsoft's ATL and WTL libraries to define Windows event dispatch functions. For those of you that are not familiar with Windows programming with C/C++, each window has a registered WindowProc. This is the dispatch procedure that inspects events, and calls the correct handler based on event type. Here is an example of the WindowProc generated by the default project wizard:

LRESULT CALLBACK WndProc(
  HWND hWnd, 
  UINT message, 
  WPARAM wParam, 
  LPARAM lParam
)
{
  int wmId, wmEvent;
  PAINTSTRUCT ps;
  HDC hdc;

  switch (message)
  {
  case WM_COMMAND:
    wmId    = LOWORD(wParam);
    wmEvent = HIWORD(wParam);
    // Parse the menu selections:
    switch (wmId)
    {
    case IDM_ABOUT:
      DialogBox(hInst, MAKEINTRESOURCE(IDD_ABOUTBOX), hWnd, About);
      break;
    case IDM_EXIT:
      DestroyWindow(hWnd);
    break;
    default:
    return DefWindowProc(hWnd, message, wParam, lParam);
   }
   break;
  case WM_PAINT:
    hdc = BeginPaint(hWnd, &ps);
    // TODO: Add any drawing code here...
    EndPaint(hWnd, &ps);
    break;
  case WM_DESTROY:
    PostQuitMessage(0);
    break;
  default:
    return DefWindowProc(hWnd, message, wParam, lParam);
  }
  return 0;
}

You can already see what a potential disaster is looming there. The code for each event is implemented in-line within the switch statement. I have run across WindowProc functions with up to 10000 lines of code in them. Ridiculous! Here is what a WindowProc looks like in ATL:

BEGIN_MSG_MAP(CNewWindow)
  COMMAND_ID_HANDLER(IDM_ABOUT, OnAbout)
  COMMAND_ID_HANDLER(IDM_EXIT, OnExit)
  MESSAGE_HANDLER(WM_PAINT, OnPaint)
  MESSAGE_HANDLER(WM_DESTROY, OnDestroy)
END_MSG_MAP()

This table declaration basically generates a WindowProc procedure, and maps the event handlers to a specific function call. Not only is this easier to read, it also encourages a maintainable implementation by hiding the direct access to the switch statement, and promoting the use of function handlers. I have successfully used this model many times to provide the required registration or definition of values. The result is an easy to read table that is compact and can easily serve as a point of reference for what events are registered, or definitions configured into your system.

Summary

I have demonstrated (ever notice the word demon starts out the word demonstrate) how to use some of the lesser used preprocessor features. I then went on to solve some cumbersome code declarations with these techniques, so now I have a fully declared Typelist type. Coming up, you will see how we will be able to take advantage of these utility MACROs to greatly simplify declarations that we use in the core of our library as we develop. In the end, the user of the library will be unaware that these MACROs even exist. While any type of MACRO can be risky to use based on what is included before your definition, I feel that this type of scenario gives the developer enough control over the definitions to minimize the potential for unexpected problems.

The link to download Alchemy is found below.

• The Typelist declaration can be found in the file: /Alchemy/Meta/type_list.h.
• The utility MACROs are located in: /Alchemy/Meta/meta_macros.h.

• Download Alchemy.zip - 12 KB