Content

Introduction
Overview
Background
Design & Implementation
Points of Interest
What is Next?
References

1. Introduction

If you search for open-source electronics, you will most likely encounter Arduino [1] in the first place. The most important aspect to mention is that it combines hardware and software almost seamlessly. You can collect data from the real world and change something in the physical environment accordingly with your Arduino while it frees you from many hardware problems and many of the rest are solved with compatible shields (extension boards) [2]. Therefore, it gained great popularity all over the world.

Collecting data is the basic usage. You can detect whether a button pressed or not. You can measure luminance, temperature, humidity, or distance. You can detect movement, or vibration. Whatever you think of measuring, you can find a way to do it with Arduino.

Delivery of data to IT infrastructure follows data collection. You can accomplish this task in a variety of ways. Basically, you can communicate with an Arduino by a serial port of your computer. In addition to serial communication, you can deliver data packages over Ethernet, WiFi, or GPRS by using appropriate shields.

2. Overview

This project is about collecting data from a group of Arduino boards and storing them on a database. But what makes this project interesting is that it is an unattended system based on Windows Communication Foundation (WCF) windows services [3].

There exist two different WCF services in the project. The first one is for data management and makes use of Representational State Transfer (REST)[12] to communicate with its clients. The second service is for Arduino board management and depends on the data service as data source. It retrieves board and pin information from the data service, delivers them to boards, receives pin readings from boards, and finally forwards those readings to the data service.

I have used an SQL server database as the actual storage behind the data service. But, the storage logic and the service are separated from each other. Managed Extensibility Framework 2.0 (MEF) [6] connects them in the data service when a service instance is created.

As O/R mapper, Entity Framework 6.1 (EF) [7] is used. After designing and creating the data tables on the database, I leveraged database-first approach to create corresponding classes. Although some manipulation is needed, entity framework is a really great tool to map database onto your project.

Arduino side of the project is easier. It listens for the configuration information, and then it starts to send readings over the communication channel. I have implemented serial interface communication for this project, but Ethernet or other data transfer infrastructure can be used with appropriate hardware as well.

Figure 1: System overview

Figure 2: Installed Ardalog services

Figure 3: Service dependency

3. Background

3.1 What is WCF?

Simply, Windows Communication Foundation is the solution of Microsoft for service-oriented architecture [9]. It enables you to create isolated computational resources. When you want to retrieve an information from a resource out of your system, you basically need to know:

From WHERE to retrieve (what is the address, URI?)
HOW to talk it (what is the communication mechanism? TCP, HTTP, Text, Binary, in what format?) [10]
WHAT it provides (What is the functionality?)

These questions for a service are packed into endpoint concept in WCF. If you create a WCF service, you need to specify an address (where), binding (how), and contract (what) in an endpoint you configure. Therefore, any client can use your service as long as it knows your service's endpoint(s). [11]

REST

Representational State Transfer (REST) is an answer for binding issue [4][5]. It directly uses HTTP requests (e.g., GET, PUT, POST, DELETE, etc.) to exchange information. When you set an endpoint's binding as REST, it means that service expects clients to talk in pure HTTP language. Moreover, a RESTful service exposes its functionality as resources (objects) connected to a URI. You can perform CRUD operations on resources directly because CRUD functions have one-to-one relation with HTTP requests.

Hosting a Service

Your service needs a place to live. There are several ways to do it. You can either use ready-to-use solutions or design a host of your own. However, it is a good practice to use proven solutions when you have one. I usually prefer Windows services as host if I don't need my service on the Internet. You can use 'installutil.exe' utility tool to install your service as a Windows service on the machine [13].

3.2 What is EF?

Entity Framework (EF) is an object/relational mapping framework to reflect database tables and their relations onto the code side. Basically, your tables are converted to corresponding classes and relations between tables are reflected to the generated code. For EF, this assertion is not necessarily true since the reverse is quite possible. EF can take your classes and convert them to database tables. There are three approaches to map database and classes in EF.

You can design your database first, and then generate classes (database-first approach)
You can design your object classes first, then generate database (code-first)
Or use the EF-provided designer to create data model (model-first)

All those approaches work well as long as you know what you are doing. Code-first approach needs more time since you code everything manually. Using designer is a more elegant way in my opinion, but it will need more work on generated code to adapt it to your overall software design [14].

3.3 What is MEF?

Well, actually, there is a big debate about what MEF is. :) However, I can at least tell you what it does. It allows you to remove hard dependencies between your classes and perfectly matches to the concept of 'dependency inversion principle' [15] of SOLID design approach [16].

Key points of MEF:

Functionality (contract)
A provider which provides that functionality (part)
A client which needs that functionality (part)
Search place for candidate providers (catalog)
Management of parts (composition container)

When all ingredients come together properly, MEF can do its job, which is to connect parts. Therefore, client code can get served with the functionality that it needs.

So what? This approach adds a great power to your design. Please refer to SOLID design principles, because I think it is the purified explanation of what a software should look like. There are many good articles about the subject on CP. As an example from this project, you can easily detach the current data storage from the data service by replacing the assembly in which database access codes exist.

4. Design & Implementation

4.1 Database Design

Figure 4: Database tables

ARD_BOARD holds Arduino board information. The important point is that F_CONN_PROVIDER stores assembly-qualified names of connection provider classes for each board and F_CONN_STR is settings for corresponding provider. Provider is instantiated at runtime from its qualified name by reflection.

ARD_PIN stores pin information of boards. Each [FK_ARD_BOARD, F_PINNUM, F_IS_ANALOG] triple corresponds to a specific pin on a specific board. When a board is connected, its pin information is sent to it. Then, it starts to read from those pins according to F_READ_PER, which specifies duration between readings.

READING is for readings from pins as the name implies. F_VALUE can be 0|1 for digital pins and between 0-1023 for analog pins.

I have generated corresponding classes of database tables by using EF database-first approach. After generation, I have edited the model to differentiate digital and analog pins. F_IS_ANALOG column is the discriminator for this differentiation. It doesn't mean much in the implementation of services, but when it comes to data editors, this model will allow me more clear design.

Figure 5: EF generated classes

4.2 WCF Data Service (RESTful)

Figure 6: Data service

Data service functionality is provided by ArdaDataSvc which is registered by using 'installutil.exe' and hosted by Windows services. When it is started by Windows service host, it creates a WebServiceHost object (host) to provide RESTful communication. For each service request, host creates a service library instance (ArdaDataSvcLib) to expose underlying data. This instance is connected to the actual storage by Managed Extensibility Framework 2.0 (MEF). In this project, I have used an MS SQL database as the data storage which is proxied by ArdalogEntities first and then SvcDataSource. ArdalogEntities is the output of Entity Framework 6.1 (EF), and SvcDataSource uses ArdalogEntities for database operations.

IArdalogSvc is the interface which is agreed on by service-side classes and storage-side classes. If you want to change the storage, the only thing is to create a class which implements IArdalogSvc. MEF searches for IArdalogSvc and connects it to ArdaDataSvcLib instance at runtime. IArdalogSvc is also the contract exposed by the data service. Service clients use this interface when they are talking to the service over a channel.

// File: IArdalogSvc.cs

    [ServiceContract]
    public interface IArdalogSvc
    {
        [OperationContract]
        [WebGet(UriTemplate="boards")]
        List<Board> GetBoards();

        [OperationContract]
        [WebGet(UriTemplate="boards/{bid}/pins")]
        List<Pin> GetPins(string bid);

        [OperationContract]
        [WebInvoke(UriTemplate="readings", Method="POST")]
        void PostReading(Reading reading);
    }

ArdaDataSvc creates a web service host in its OnStart method to respond with REST to service calls.

// File: ArdaDataSvc.cs

    /// <summary>
    /// Ardalog data service (WCF Windows service + REST)
    /// </summary>
    public partial class ArdaDataSvc : ServiceBase
    {
        // base address of service (can also be specified in app.config)
        static string _baseaddr = @"http://localhost:45556/ArdalogSvc";

        // service host for REST
        WebServiceHost _host = null;

        protected override void OnStart(string[] args)
        {
            try
            {
                if (_host != null)
                    _host.Close();

                _host = new WebServiceHost(typeof(ArdaDataSvcLib), new Uri(_baseaddr));
                _host.Open();
                _host.Faulted += HostFaulted;
            }
            catch (Exception ex)
            {
                TextLogger.LogIt(ex);
                throw;
            }
        }
// other code

Following code fragments show how MEF works to combine related parts.

ArdaDataSvcLib tells MEF that it will [Import] DataSource which is IArdalogSvc.

// File: ArdaDataSvcLib.cs

    /// <summary>
    /// Data service library class
    /// </summary>
    public class ArdaDataSvcLib : IArdalogSvc
    {
        public static event EventHandler<EventArgs> Created;

        public ArdaDataSvcLib()
        {
            if (Created != null)
                Created(this, new EventArgs());
        }

        /// <summary>
        /// Underlying data source which is filled by MEF
        /// </summary>
        [Import]
        public IArdalogSvc DataSource { get; set; }
// other codes

SvcDataSource tells MEF that it [Export(typeof(IArdalogSvc))].

// File: SvcDataSource.cs

    [Export(typeof(IArdalogSvc))]
    public class SvcDataSource : IArdalogSvc
    {
// other codes

ArdaDataSvc utilizes MEF to combine parts in the Created event handler (ServiceInstanceCreated) when an ArdaDataSvcLib instance created. In the handler method, DirectoryCatalog is created with the assembly folder. A CompositionContainer uses that catalog to search for parts. If everything goes without an error, MEF composes parts for the ArdaDataSvcLib instance (lib).

// File: ArdaDataSvc.cs

        public ArdaDataSvc()
        {
            // some other code
            ArdaDataSvcLib.Created += ServiceInstanceCreated;
        }

        /// <summary>
        /// ArdaDataSvcLib.Created handler.
        /// MEF combines exports & imports for ArdaDataSvcLib when an instance created.
        /// </summary>
        /// <param name="o"></param>
        /// <param name="e"></param>
        static void ServiceInstanceCreated(object o, EventArgs e)
        {
            // get the assembly's directory.
            var dir = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
            // create a directory catalog for MEF
            using (var catalog = new DirectoryCatalog(dir))
            {
                // create a composition container with that catalog
                using (var container = new CompositionContainer(catalog))
                {
                    ArdaDataSvcLib lib = o as ArdaDataSvcLib;
                    if (lib == null)
                        throw new ArgumentException("service library is not ready");
                    // compose related parts
                    container.ComposeParts(lib);
                }
            }
        }
// other codes

Please note that database creation scripts (DbCreate.txt) and sample data (TestData.txt) is included in the Ardalog2.zip/ArdaDAL directory.

4.3 WCF Board Management Service

Figure 7: Board management service

ArdaListenerSvc is the service hosted by Windows services. When it is started, it initializes BoardListenerManager singleton instance. In the initialization process, BoardListenerManager retrieves board list from the data service, and creates all board listeners with their communication providers. CommProviderBase is responsible to keep a live connection with the Arduino board. When a connection is established, BoardListener instance sends board id and pin information to the Arduino in order to make it begin its pin readings. Each BoardListener tracks messages coming from its communication provider and takes actions accordingly. For example, if an incoming message is a pin reading, then BoardListener forwards that information to the data service.

After a connection is established, BoardListener sends configuration info to the board via CommProviderBase. Following code fragments show initialization logic.

// File: BoardListenerManager.cs

        /// <summary>
        /// initializes internal structures.
        /// call this when the service is started
        /// </summary>
        public void Initialize()
        {
            // some cleanup code here
            // ...

            // open data service channel factory
            this._channelFactory.Open();
            // create a communication channel
            var channel = this.GetNewChannel();


            // get boards
            var boards = channel.GetBoards();

            // create listeners for boards
            foreach (var b in boards)
            {
                if (string.IsNullOrEmpty(b.ConnectionProvider)
                    || string.IsNullOrEmpty(b.ConnectionString))
                {
                    TextLogger.LogIt("board skipped due to lack of conn info ({0})", b.Id);
                    continue;
                }
                var p = CreateCommProvider(b);
                this._listeners.Add(b.Name, new BoardListener(b, p));
            }
        }

        /// <summary>
        /// creates a new communication channel to the data service
        /// </summary>
        /// <returns></returns>
        public IArdalogSvc GetNewChannel()
        {
            return this._channelFactory.CreateChannel();
        }

        /// <summary>
        /// finds & creates communication provider for a board.
        /// it may be a serial interface, Ethernet, or WiFi.
        /// in this project, only serial port communication is implemented
        /// </summary>
        /// <param name="b"></param>
        /// <returns></returns>
        private static CommProviderBase CreateCommProvider(Board b)
        {
            var t = Type.GetType(b.ConnectionProvider);
            return (CommProviderBase)Activator.CreateInstance(t);
        }
// some other code

And board listener part:

// File: BoardListener.cs
        
        public BoardListener(Board board, CommProviderBase comm)
        {
            this._board = board;

            // get pins of the board from data service
            this._pins = BoardListenerManager.Instance
                .GetNewChannel().GetPins(this._board.Id.ToString());
            if (this._pins.Count <= 0)
            {
                TextLogger.LogIt("no pin defined for board: ({0})", this._board.Id);
                return;
            }

            this._commProvider = comm;
            this._commProvider.BoardMessageReceived += MessageReceivedHandlerAsync;
            this._commProvider.CommStateChanged += CommStateChangedHandlerAsync;

            // initialize comm provider with board id and connection info for the board.
            this._commProvider.Initialize(this._board.Id, this._board.ConnectionString);
        }

        /// <summary>
        /// executed when the state of communication changed.
        /// when communication established, it send configuration info to the board.
        /// </summary>
        /// <param name="sender"></param>
        /// <param name="e"></param>
        async void CommStateChangedHandlerAsync(object sender, StateChangedEventArgs e)
        {
            await Task.Run(() =>
            {
                try
                {
                    switch (e.BoardState)
                    {
                        case CommState.Closed:
                            break;
                        case CommState.Open:
                            this._commProvider.Send(string.Format("id,{0}$", this._board.Id));
                            this._pins.ForEach(p1 =>
                            {
                                this._commProvider.Send(string.Format("addpin,{0},{1},{2},{3}$",
                                    p1.Id, p1.PinNum, p1.IsAnalog ? "1" : "0", p1.ReadPeriod));
                            });
                            break;
                        case CommState.Aborted:
                            break;
                        default:
                            break;
                    }
                }
                catch (Exception ex)
                {
                    TextLogger.LogIt(ex);
                }
            });
        }
// some other code

4.4 Arduino Board

Figure 8: Arduino classes

Comm is the class which implements the communication protocol on the Arduino side and SerialComm is the serial port derivation of Comm. When a board receives an addpin command from the board management service, it creates (or updates if exists) a pin definition (PinDef) according to parameters, then starts to monitor it. Each reading is sent with sendReading function of Comm (in that case, SerialComm). If you use an ethernet shield as transport mechanism, then an ethernet derivation of Comm should be implemented and uploaded to the Arduino board.

/* File: ArdalogBoard.ino */

// board id from database
int boardId;
//  communication handler
Comm* comm;
// pin list constructed according to database
PinDefList pinlist;

// arduino setup
void setup()
{
    comm = SerialComm::getInstance();
    comm->setIdCallback(setId);
    comm->addPinCallback(addPin);

    comm->initialize();
}

// arduino main loop
void loop()
{
    // Serial handler update (to read serial data)
    comm->update();

    // scheduler update (to read pin and send data)
    Scheduler::getInstance()->update();
}

// callback for board id command
void setId(int id) {
    boardId = id;
}

// callback for add pin command
void addPin(int id, int pinnum, bool isanalog, int period) {
    PinDef *d = pinlist.Find(id);
    if(d == NULL) {
        d = new PinDef(id, pinnum, isanalog, period);
        d->setReadingCallback(sendReading);
        pinlist.Push(d);
    }
    else {
        d->setPeriod(period);
    }
}

// callback for pin reading
// sends value from pin to service
void sendReading(int id, int val) {
    comm->sendReading(id, val);
}

In the setup, communication handler (comm) is set. We have only serial port communication implementation (SerialComm) for now, and we use it as the data transfer mechanism.

Main loop of the application updates comm and Scheduler instance which informs all its PinDef clients when their periods are elapsed. A PinDef sends its reading to the service when its period is elapsed, by calling assigned sendReading callback function.

I don't want to dump class implementations here since there is nothing interesting with them.

4.5 Communication Protocol

Figure 9: Sequence diagram for communication protocol

When the board management service is started, it sends all configuration information to the boards.

id,<boardId>$ : A board sets its board id by this command. Responds with 'ok$'

addpin,<pinId,pinNum,isAnalog,period>$ : A pin configuration for the board.

ping$ : Service checks whether board is still connected

reading,<pinId,value> : A board sends a pin reading with that command.

An example talk:

id,1$ (from service, set your board id as 1)
ok$ (from arduino)
addpin,1,0,1,1$ (from service, read from analog pin 0 for every 1 minute. Its id is 1)
ok$ (from arduino)
...
ping$ (from service)
...
reading,1,254$ (from arduino, 254 is read from pin with id 1)
...

Actually, this implementation of the communication protocol is not reliable enough. Messages are not sequenced and numbered uniquely. Response messages for the commands are not tracked. Therefore, it needs more work on both sides of the communication.

5. Points of Interest

This project combines many technologies and many concepts. To summarize:

Arduino programming
Serial communication programming
WCF Windows services
Representational State Transfer (REST)
Managed Extensibility Framework (MEF)
Entity Framework (EF)

Each of those topics contains many concepts which cannot be covered in one article. I really enjoyed while developing this project to see that all of them came together in one.

In addition, I am planning to use Windows Presentation Foundation (WPF) [8] for UIs. With all editors and related reporting tools, this project can be used as all-in-one solution for any kind of physical environment monitoring (it doesn't harm to exaggerate a little :)).

6. What is Next?

This project is very open to extend. First, we need a data editor to manage boards and pins. Then a report utility comes in order to see readings. A setup application for database creation and service installation would be cool. When these user applications are completed, we can start to play with other communication mechanisms like Ethernet. Other data storage implementations can be done for the data service, for example, MySQL data provider. I welcome any suggestion.

7. References

Service-based Arduino Data Logger