Download source - 2.76 MB

Introduction

This article describes a framework for writing small to medium size TCP servers, written in the C language with an object oriented approach, using the standard Berkeley sockets interface, and intended to be cross-platform. The development was done in a system with the Ubuntu Linux distribution installed, with later porting and a little testing in Windows 2003 with Visual Studio Express 2008.

Background

This framework comes as a consolidation of the many TCP servers I've come across in the many years that I have worked in the banking and retail automation fields. Those servers typically handled from a dozen to some hundred connections, running on private site-local networks.

Description

The framework uses non-blocking sockets to perform all communications functions, and all connections are managed by one thread only, therefore providing a high degree of scalability. Such scalability, however, bumps into the inherent limitations built into the standard Berkeley socket functions. Specifically, the limitations of the select socket function, which can handle connections in the hundreds range, but typically, not in the thousands range.

When writing a server using this framework, the business logic is provided by the application in the form of a dynamically configurable number of threads. Such threads are decoupled from the actual connections, so that applications don't have to worry about connection issues. It is relatively easy to configure the number and the type of the application threads.

Buffer management is also simplified, therefore minimizing a number of errors related to memory management. It is possible to configure static buffers, buffers associated to a connection, and ad hoc buffers, allocated if the other schemes fail. Also, copying is minimized: the only needed copy is the one from the operating system buffers to the framework buffers.

The messages exchanged on the wire use a counter and a flag, to provide for record boundaries and additional reliability.

A number of convenience functions are also provided, to aid in the writing of client applications, although the main goal of the framework is writing servers.

It is assumed that the servers written with the framework will generally be servers tied to specific business needs, and are meant for in house applications running on private networks. These servers are not meant to be general purpose servers running on the internet, like web servers and such.

Below is an informal UML structure class diagram, depicting the overall classes that comprise the framework.

The heart of the framework is a ConnectionManager object. It owns a ConnectionTable instance which owns many Connection objects. Each connection in turn owns a Socket instance which does the actual communications work. Each connection also owns either one or two Message instances, one for receiving messages from and other for sending messages to the connected client.

The connection manager, as its name implies, manages the TCP connections. For each connection, it keeps track of the messages being sent and received: how many bytes were already transferred, and how many are still outstanding. It also manages new connection requests, and handles the errors spawned by all socket operations, closing connections in error if needed.

The framework's general flow of operation is illustrated by the sequence diagram below:

Every application Thread sits in a loop, waiting for input messages to become available for processing. The waiting is done by calling the QueueManager::waitInputMessage method. When the connection manager finishes assembling an input message, it signals the queue manager by invoking the QueueManager::addInputMessage method. This method adds the complete message to the QueueManager's input message queue, and awakens an application thread to process the input message. Notice that the message is not guaranteed to be processed right away: this depends on the availability of an idle application thread.

The application thread, after processing an input message, may generate an output message that is to be sent to the client application. The application thread makes the output message available to the framework by calling the QueueManager::dispatchOutputMessage method. Upon receiving this call, the queue manager adds the output message to its output message queue, and notifies the connection manager of the existence of the new output message by calling the ConnectionManager::notifyOutputMessage method. The connection manager then removes the output message from the output message queue, and schedules its transmission by the connection associated with the message.

Notice that the application thread may reuse the input message as an output message. Or it can dispose of the input message by calling the QueueManager ::disposeMessage method (which returns the message to the free message queue) and then request a free message to use as output message, by calling the QueueManager::getFreeMessage method. It's up to the application thread how it manages its messages, as long as it does something with them: either calls dispatchOutputMessage or disposeMessage on any message it owns, before blocking again after processing an input message. Notice also that a Message object always belongs to another object: either to a MessageQueue, or to a Thread, or to a Connection.

There are three other classes in the class diagram that merit some explanation.

The Server class is a façade class providing convenient access to the most important methods for writing server applications. All the methods in the server class are in fact #defines of methods belonging to another classes.

The Client class is also a façade, like the server class, but for writing client applications. Although the framework's aim is to make it easy to write server applications, all its infrastructure can be easily taken advantage of, to also write client applications. The client class has a bunch of #defines to aid its role as a façade class, but also adds some functionality of its own, in the form of specific methods for writing client applications.

Finally, the Log class writes to a log file a huge amount of information about what the framework does when an application that uses the framework is run. This log facility has the usual levels of severity: information, warning, error, fatal, debugging. It can also be used to trace the communications buffers, and can easily be called from inside the application.

For more complete documentation about the framework, it's easy to change the doxygen configuration file to get a lot more information. (Doxygen is the utility used to generate the documentation for this project.) Specifically, two useful options that can be activated are "CALL_GRAPH" and "CALLER_GRAPH". The output generated by these options are highly useful, but for large projects it can mess up with the layout of the output shown in the browser, because of the big size of the images created.

A First Example: Hello ?

Ok, now for something a little more interesting. Let's see how to create a "hello world"-type application.

First, the main program:

int main(void)
{
   // does whatever initialization the framework needs
   server_init();

   // starts 3 threads of the same code
   server_addThreads(3, threadFunc, "example thread");

   // runs the server
   server_run();

   // that's it!
   return 0;
}

Now, the application thread. Remember, the framework will start 3 thread instances:

// a minimal thread

static threadfunc threadFunc(void* arg)
{
   for (;;)
   {
      // uses "server_printf" (and not plain printf) because of
      // contention issues between threads
      server_printf("Hello from a minimal thread...\n");

      server_sleep(3);
   }

   return 0;
}

Each of the threads above will print a message on the console, sleep for 3 seconds, and will keep doing it over and over again.

Now the header files used by this little program:

// common configuration options & declarations (always include first)
#include "config.h"

// application includes
#include "Server.h" /* server_xxx functions */

// prototypes
static threadfunc threadFunc(void*);

Yep, that's it, the whole program.

The points of interest are the server_xxx functions, and the threadfunc typedef, which are provided by the framework

Another Example Example Example: Echo

Now, the first server program in any book on TCP/IP: an echo server.

First, the main program:

int main(void)
{
   // the port on which the server waits for connection requests
   server_setServicePort(12345);

   // how much detail is written to the log file
   server_setLogLevel(LOG_LEVEL_DEBUG);

   // does whatever initialization it's needed
   server_init();

   // starts 5 threads of the same code
   server_addThreads(5, threadFunc, "example thread");

   // runs the server
   server_run();

   // that's it!
   return 0;
}

Now the application thread again:

static threadfunc threadFunc(void* arg)
{
   Message *msg;

   for (;;) // the thread main loop
   {
      // waits for a message from a client
      msg = server_waitInputMessage();
      server_logDebug("received a message");

      // echoes back the received message
      server_logInfo("ok, replying");
      server_dispatchOutputMessage(msg);
   }

   // yep, that's it!
   return 0;
}

The point of interest now is the Message typedef, which is provided by the framework, and contains the data sent by the client.

Another Example: Still Echoing, But Doing a Little More of Nothing

Now only the application thread is shown. It still doesn't do much, but at least it shows how to access and modify the data sent by the client. (In fact, the way it is shown here is a little insecure, but easy. There are some other methods that access the message contents in a slightly safer way, although as always, with C being C, you can do pretty much everything you want, including shooting yourself in the foot. And you can do it very fast, of course).

threadfunc threadFunc1(void* arg)
{
   uint size, count = 0;
   char *bufIn, *bufOut;
   Message *msgIn, *msgOut;

   for (;;)
   {
      // waits for a message from a client
      msgIn = server_waitInputMessage();

      // ok, message received
      bufIn = server_messageBuffer(msgIn);
      size = server_messageSize(msgIn);

      // requests an (output) message to be sent as reply to the client
      msgOut = server_getFreeMessage();
      bufOut = server_messageBuffer(msgOut);

      // processes the (input) message received from the client
      server_printf("* message: length=%02d buf=[%.20s]\n", size, bufIn);
      // blah blah blah

      // creates the (output) message to be sent as reply to the client
      // (uses the same bytes and size, just so the client can check them)
      memcpy(bufOut, bufIn, size);
      server_setMessageSize(msgOut, size);

      if (!(++count % 10))
         server_logInfo("%d messages processed now", count);

      // copies connection information from the input message to the output
      // message (this is needed so that the framework knows which client to
      // send the output message to)
      server_copyConnectionFromMessage(msgOut, msgIn);

      // releases the input message, it's not needed anymore
      // (this wouldn't be needed if the input message were reused)
      server_disposeMessage(msgIn);

      // makes the message available to be sent to the client
      server_dispatchOutputMessage(msgOut);
   }

   // that's it
   return 0;
}

There are several points of interest here.

The server_messageBuffer and server_messageSize methods, that provides access to the raw data sent by the client, and to its size.
The server_getFreeMessage method, through which the application thread acquires a new Message instance that it will use as reply to the client.
The server_setMessageSize which sets the size of the reply message.
The server_copyConnectionFromMessage which sets the IP address of the client to which the reply will be sent.
The server_disposeMessage that is used by the application thread to return an unused message to the framework.

Now What ?

There are several enhancements that can be done to the code as it is.

There are a great number of assert calls sprinkled liberally throughout the code. These calls should be reviewed, and proper error handling should be provided for the cases that make sense.

The central class ConnectionManager can be made more class-like by getting rid of the global static variables that exist now. Doing so will enable us to write applications having more than one connection manager, that is, serving in two or more IP addresses or service ports.

Writing some unit tests wouldn't hurt either...There's only one class with tests written, because they were absolutely necessary.

One thing of note is that the code uses lots and lot of uints and ushorts, something that may annoy people who likes their ints and shorts raw.

History

7^th April, 2010: First version of the article, and covers the version 1.0.0 of the framework
14^th April, 2010: Version 1.01 - Updated source code

Correction: Code inside "assert" calls taken outside
Included release builds
Compatibility of includes with the C++ language
Included C++ examples

29^th April, 2010: Version 1.02 - Updated source code

Correction: checks number of connections before creating new connection
Inclusion of the "generic client" class and example
Changed "CreateThread" to "_beginthreadex" in Windows
Reorganization of the code in ConnectionManager.c

A Small C-Language TCP Server Framework