Yet another Home Monitor

Introduction

What is the first thing technology types want to do with an embedded device? Read temperature of course. In this case, I had a wine cooler in my basement I wanted to monitor so that I knew that the wine was being kept at a consistent temperature. Well, that little project grew into some cloud integration and more than one sensor type.

This project uses a Netduino plus 2 to read data from several types of sensors. First, a serial OneWire temperature bus with a few thermocouple sensors on the bus. Second, an analog humidity sensor wired up to one of the analog ports. Finally, using some magnetic switches, I can monitor the status of my two garage doors through the digital ports available on the Netduino.

Of course, having this data is one thing, but having access to it, and potentially controlling the Netduino remotely is the goal. All data recorded by the Netduino will be sent to Azure and stored in an SQL database. In addition, a web REST interface will be created to enable grabbing the data recorded from the Netduino and displaying it on a mobile device. In this case, I will have a Windows Phone application that can read the sensor data from the Azure SQL instance.

Background

The premise of this project is the use of cheap endpoint devices (Netduino) to collect lots of disparate information and store this data in the cloud. Then, using common mobile platforms (smartphones) retrieve this data, raw and analyzed, and display to the user no matter where they are located.

The embedded device I chose for this implementation is the Netduino Plus 2 and more information on the device can be found here: http://www.netduino.com/. I am using a OneWire serial bus to read temperature data from a number of thermocouples. There is a lot of information out there on OneWire, some can be found here: http://developer.mbed.org/users/alpov/code/OneWire/docs/tip/. I am using DBS18B20 temperature sensors for placement in the actual physical locations where I am gathering temperature data.

The states for the garage doors are monitored by use of magnetic switches. When the doors are closed, the circuit is closed, and when the doors open the circuits open and this is captured by the Netduino device.

Humidity is monitoring using a HM1500LF humidity sensor (http://www.meas-spec.com/product/t_product.aspx?id=2448#) connected to the Netduino analog ports. Humidity is calculated using a formula that uses the voltage offset reported by the sensor to derive a humidity percentage.

Visual Studio 2013 is the basis for all the code development in this project. I am using C# along with the numerous libraries needed to talk to Azure and the Netduino. The Netduino runs a version of the .Net Micro Framework and the SDK and documentation can be found here: https://msdn.microsoft.com/en-us/library/ee436350.aspx

To get a VS 2013 environment up and running for Netduino development, follow the installation instructions here: http://www.netduino.com/downloads/

I will be implementing the storage and communication in Azure and all things Azure can be found here: http://azure.microsoft.com/en-us/. Specially, I will be using Cloud Services, the Service Bus, SQL Services and a web site.

Architecture

The architecture of Home Monitor is a multi-tiered service with one tier being the actual embedded devices with the sensors, a cloud service (Azure) providing the storage and intelligence and a mobile platform for the User Interface (UI).

This is based on a IofT pattern where there is one inbound service queue with a many-to-one relationship to the potentially large number of embedded devices. There is also a one-to-one relationship for a device command queue where commands to be executed on remote embedded devices are submitted.

The .Net Compact Framework running on the Netduino has a significant amount of functionality, but does not offer the full .Net or Windows environment. Therefore it is not feasible to run a “standard” windows based client that can speak directly to the services available in Azure. In this case, I am using a gateway type approach to provide communications to the remote embedded devices. The Netduino’s use socket based communications over TCP/IP to connect and transmit data to the hmEnqueue Azure cloud service that processes these inbound messages and creates entries on the InBoundTemps Service Bus queue for further processing.

Data transmitted to hmEnqueue from the Netduino is in a serialized XML format and placed directly on the InBoundTemps queue. XML data on the InBoundTemps queue is then de-serialized by the hmDequeue worker role and stored in the IofT SQL Database.

The hmNotify worker role is responsible for monitoring the IofT database and creating exception messages when data has not been received for a specific period of time. For this implementation, temperature data that has not been recorded in the last hour will raise the exception. I am using a App Service (SendGridEmail) that is providing the email interface to the hmNotify service. From configuration data in the IofT database, exception emails are generated to the appropriate party if temperature data from the configured sensors has not been recorded in the last hour. For this specific instance, I am using the email address of my smartphone to get SMS messages from Azure on these exceptions.

I also created the azHomeMonitor website to host the REST services that a mobile device can consume. These services allow the most recent temperatures, humidity readings, the last 24 hours activities on the garage doors and 24 hour averages to be displayed on a smartphone.

The hmEnqueue service also creates a specific queue for each endpoint device. This is generally for future use but envisioned to allow remote commands to be transmitted to the Netduino devices. I have this working in test code, but have not exposed a programmatic way of controlling the embedded devices at this time.

Using the code

Netduino

The embedded code performs the following tasks on startup:

• Initializes the network

• Uses a NTP server to set the current time (need DST support here)

• Creates the objects for the sensors we are going to measure

  • Initialize the OneWire interface and read how many sensors are on the bus

  • Initialize the digital ports for checking state changes

    • Pins monitored are 8,9,10,11,12,13

    • Interrupt handlers are created for each port monitored

  • Initialize the first analog port with the humidity sensor

• Creates timers for checking:

  • Temperatures (2 minutes)

  • Humidity (5 minute)

  • Digital Port monitoring (1 second)

  • Blinking the LED (.75 seconds short, 2.5 seconds long)

• Create interrupt handler for onboard button

• Start with the onboard LED in the OFF state

• Loop forever checking network communications

Network Communications

This run forever loop performs the following tasks:

1. If a socket connection exists, go into a do-while loop looking for inbound commands over the established TCP/IP socket

   1. On reception of data (remote commands posted on the Azure queue specific to this Netduino) perform the action. Currently supported actions are:

      1. Change LED blink timing

      2. Cause temperature data to be transmitted

      3. Cause digital port data to be transmitted

      4. Dump all current data

      5. Turn debugging on/off

   2. If no data available (timeout) or exception on receive command

      1. Send PING to hmEnqueue service
         This will cause a closed socket connection to be reopened and also let the queueing service know this device is still operational

LED timer

This routine is very simple, it just toggles the LED on/off. This allows me to get instant feedback on the status of the embedded device. The interval can be changed remotely, but the main purpose is for manual viewing of the status of the device. I believe there is a bug in the TCP/IP stack on the Netduino that can cause the entire device to hang up. This is why the LED is toggled. When I see the LED is not blinking, or I get a text from the system telling me no data has been recorded in an hour, I can reset the device and get it working again.

OneWire Timer (default 2 minutes)

This timer will read the current temperature on all the thermocouple devices on the OneWire bus whenever this timer fires. Temperatures are collected in a local list of temperature objects and transmitted to the hmEnqueue service every 5 timer interrupts (10 minutes default). When this timer is fired for the 5th time, a network message is generated and sent to the hmEnqueue service with the recorded temperatures. The local list of these temperatures is then reset and the process starts all over again.

Outbound data is serialized into XML format for transmission to the hmEnqueue service. While a bit chattier than JSON, I wanted to use XML in this case to balance out this solution showing more than one way (JSON) to serialize/de-serialize object models.

The actual code to get the temperature from the thermocouples is displayed below.

   public double getTemperature(byte[] device,DeviceType deviceType)
   {
        resetBus();
        selectDevice(device);
        convertTemp();
        resetBus();
        selectDevice(device);
        double temp = readTemp(deviceType);
        return temp;
}

   public double readTemp(DeviceType deviceType)
   {
        int theTemp = 0;
        double dtemp = 0;
        byte[] results = new byte[45];
        byte[] readTemp = new byte[] { W, 0, A, B, E, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, F, CR };
        writeData(readTemp, 24);
        if (readData(results, CR) > 0)
        {
            int half = 0;
            int sign = 0;
            switch (deviceType)
            {
                case DeviceType.DS18B20:
                    theTemp = convertHexCharByte(results[5]) & 0x07;                    // LSb of MSB and mask out the sign bits
                    theTemp = (theTemp << 4 | convertHexCharByte(results[2]));          // Shift and add in MSb of LSB
                    half = convertHexCharByte(results[3]);                              // Take LSb of LSB for decimal temp bits
                    sign = convertHexCharByte(results[4]) & 0x01;                       // Check if negative temperature
                    dtemp = (double)theTemp;
                    if (half > 0)
                        dtemp += ((double)half)/10.0;
                    if (sign > 0)
                        dtemp *= -1;
                    break;
                case DeviceType.DS1822:
                    theTemp = convertHexCharByte(results[2]);
                    theTemp = (theTemp << 4 | convertHexCharByte(results[3]));          // Shift in the two actual temperature bytes
                    theTemp = (theTemp >> 1);                                           // Shift out the decimal
                    half = convertHexCharByte(results[3]) & 0x01;                       // Check if decimal .5 is indicated
                    sign = convertHexCharByte(results[4]) & 0x01;                       // Check if negative temperature
                    dtemp = (double)theTemp;
                    if (half > 0)
                        dtemp += .5;
                    if (sign > 0)
                        dtemp *= -1;
                    break;
            }
       }
       return dtemp;
}

Humidity Sensor Timer (default 5 minutes)

This timer handler reads the analog port and creates a current humidity reading. The current humidity is calculated by reading the voltage on the analog input and calculating the actual humidity (in %). It also stores the date/time of this reading along with the humidity in the list of humidity readings. When 3 readings have been collected (default 15 minutes) this data is serialized into XML and transmitted to the hmEnqueue service.

public double readHumidity()
{
    double voltage = analogInput.Read() * _referenceVoltage;
    humidity = 0.03892 * (voltage*1000) - 42.017;            // Voltage in mv
    lastReading = DateTime.Now;
    return humidity;
}

Digital Ports Timer (default 1 second)

While the actual event, a digital state change, is handled by an interrupt handler, this timer routine looks to the digital state change list to see if any entries were created. If entries exist in the list, they are serialized into XML and transmitted to hmEnqueue. The list is then cleared.

OnButton Interrupt

This interrupt handler will create an outbound XML message and transmit that message to the hmEnqueue service. It will collect and transmit temperature, digital port data and humidity data and reset these outbound lists.

Inbound Messages

Messages sent to a device are handled by the processMessage loop. Right now, this is very limited but does allow for the blink rate of the LED to be changed and debug mode toggled on/off. You can also send codes to dump all or portions of the queued data on command.

azEnqueue

This worker role initializes the network stack, creates a socket to listen on, then listens for inbound connections. When an inbound connection is detected, a thread is created to receive data from that socket and the listener goes back to listening.

public netListen(IPEndPoint npEndPoint, ref QueueClient inboundQueue, addQueueDelegate addQueue )
{
    _inboundDataQueue = inboundQueue;
    listenSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);     //Initialize Socket class
    npEndPoint.Address = IPAddress.Any;
    listenSocket.Bind(npEndPoint);                                                                  //Request and bind to an IP from DHCP server
    listenSocket.Listen(10);                                                                        //Start listen for net requests
    log("Listening on configured endpoint...");
}

The thread that reads data is responsible for taking the inbound XML stream and placing it on the InBound Temps queue for storing in the SQL database.

public void readSocket()
{
    IPEndPoint clientIP = clientSocket.RemoteEndPoint as IPEndPoint;
    log("New connection from: " + clientIP.Address.ToString());
    byte[] buffer = new byte[_bufferSize];
    string message = "";
    int readByteCount = 0;
    clientSocket.SendTimeout = 10000;
    clientSocket.ReceiveTimeout = 90000;                    // 90 second receive timeout, should get pings every minute
    while (isRunning && (clientSocket != null))
    {
        message = "";
        try
        {
            do
            {
                readByteCount = clientSocket.Receive(buffer, _bufferSize, SocketFlags.None);
                if (readByteCount > 0)
                {
                    message += Encoding.UTF8.GetString(buffer, 0, readByteCount);
                    Thread.Sleep(80);                                               // Give the slower device a chance to catch up
                }
            }
            while (clientSocket.Available > 0);
            if (readByteCount != 0)
                processMessage(message);
            else
            {
                log("Closed connection from: " + clientIP.Address.ToString());
                RequestStop();
            }
        }
        catch (Exception ex)
        {
            log("RecvEx: " + ex.Message);
            RequestStop();
        }
    }
}
public void processMessage(string message)
{
    int index = message.IndexOf("<##>");                        // Look for first occurance of header (for multiple XML messages)
    string subMessage = "";
    while (index >= 0)
    {
        index += 5;
        int nextXML = message.IndexOf("<##>", index);            // Look for next occurance of header, if any
        if (nextXML > 0)
            subMessage = message.Substring(index, nextXML - index);
        else
            subMessage = message.Substring(index);
        processXML(subMessage);
        index = nextXML;
    }
}
public void processXML(string message)
{
    XmlDocument xDoc = new XmlDocument();
    try
    {
        if (message.IndexOf("</Home") >= 0)                 // Real XML message, not just a single line ping
        {
            xDoc.LoadXml(message);                          // Load to validate XML
            log("MsgRecvd: Putting on queue now.");
            BrokeredMessage brokered = new BrokeredMessage(message);
            inboundDataQueue.Send(brokered);                // put on inbound data queue
        }
        else
        {
            if( macAddress == null )
            {
                int client = message.IndexOf("Client");
                if( client > 0 )
                {
                    int fquote = message.IndexOf('"',client);
                    int squote = message.IndexOf('"', fquote+1);
                    macAddress = message.Substring(fquote + 1, (squote - fquote-1));
                    sendCommands = new Thread(readCommandQueue);
                    sendCommands.Start();
                }
            }
        }
    }
    catch (SystemException se)
    {
        log("Excp (processsXML): " + se.Message);
        log(message);
    }
}
public bool sendMessage(string message)
{
    try
    {
        int sent = clientSocket.Send(Encoding.UTF8.GetBytes(message), message.Length, SocketFlags.None);
        if (sent == message.Length)
            return true;
    }
    catch (SocketException se)
    {
        log("SendEx: " + se.ErrorCode.ToString());
        RequestStop();
    }
    return false;
}

The processMessage function has the ability to “see” multiple XML messages. For this effort I placed a simple tag between XML messages (<##>) that allows the processMessage function to separate out distinct XML data streams. These separated streams are used to create the BrokeredMessage objects that are then placed on the InBound Temps queue.

Ping messages, received every 60 seconds, are just discarded by this worker role now. However, they could be processed and relayed to some kind of system monitor for near real time monitoring of the embedded device.

IofT SQL Database

There are 7 tables in the IofT database. They are:

• ndTemps – used to store temperature data

  • Record ID

  • Device

  • Thermometer (unique OneWire device address)

  • Time of sample

  • Celsius Temp

• ndHumidity – used to store humidity data

  • Record ID

  • Device (MAC address)

  • Humidity

  • Time of sample

  • AnalogPort

• ndStates – used to store digital state data (i.e. status of garage doors)

  • Record ID

  • Device (MAC Address)

  • Port

  • Time of action

  • isOpen (circuit open indicator)

• ndDevices – Configuration table that holds all the devices in the system

  • Description (mostly location)

  • Device (MAC Address)

  • Type

  • Mail address for notifications

• ndHumidSensor – Configuration table that holds all the humidity sensors in the system

  • Record ID

  • Device (MAC Address)

  • AnalogPort – analog port sensor is hooked up to

  • Description (location)

  • Type

• ndPorts – Configuration table that holds port mappings for digital states

  • Record ID

  • Device (MAC Address)

  • Port

  • Description

• ndTherms – Configuration table that holds all the temperature sensors in the system

  • Description

  • Thermometer – Unique device address (unique to the OneWire bus, not device)

  • Device (MAC Address)

  • Type

azDequeue

This worker role initializes the InBound Temps queue and then starts a message pump on that queue. During initialization, the necessary SQL tables, listed above, are created if they do not already exist.

Inbound messages from a Netduino are processed and placed into an XMLDocument object. The nodes in the document are then inspected and the data retrieved from the XMLDocument and stored in the SQL database.

Digital port data is extracted and the following SQL statement is used to insert the new row into the ndStates database.

"INSERT INTO ndStates (Device, Port, Time, isOpen) values ('" + client + "', '" + port + "', '" + time + "', '" + open + "')"

Humidity data is extracted and the following SQL statement used to insert the new data into the table.

INSERT INTO ndHumidity (Device, Humidity, Time, AnalogPort) values ('" + client + "', '" + humidity + "', '" + time + "', '" + analog + "')"

Temperature data is extracted and the following SQL statement used.

"INSERT INTO ndTemps (Device, Thermometer, Time, Celsius) values ('" + client + "', '" + device + "', '" + time + "', " + temp + ")"

azNotify

This worker role is responsible for monitoring the data in the SQL databases and reporting back status. The role looks at the configuration database (ndDevices) and creates a list of the device objects in the system. It then runs a continuous loop checking these devices for data updates in the past hour. If no data is returned in the query, a sendMail procedure is called that will send an email to the configured email address for that device.

bool sendMail(string contactEmail, string message)
{
    bool sent = false;
    try
    {
        MailMessage newMail = new MailMessage("youremail@hotmail.com", contactEmail);
        newMail.Subject = "HM: Activity";
        newMail.Body = message;
        SmtpClient msgClient = new SmtpClient("smtp.sendgrid.net");
        msgClient.Credentials = new NetworkCredential("azure_<yourcredshere>@azure.com", "<key>");
        msgClient.Send(newMail);
        sent = true;
    }
    catch (Exception e)
    {
        log("hmNotify:SendMail() Exception: " + e.Message + ", " + e.InnerException.Message,true);
    }
    return sent;
}

This function uses a third party Azure application, SendMail, for the SMTP services.

azHomeMonitor

This is a simple web site that implements a Controller object that can handle the following requests:

• GET: /hm/devices/id

• GET: /hm/therm/id

• GET: /hm/temp/id

• GET: /hm/ports/id

• GET: /hm/state/id

• GET: /hm/humidity/id

• GET: /hm/averages/id

Here is the code for the humidity Action:

public ActionResult Humidity(string Id)
{
    List<hmHumidity> humidities = new List<hmHumidity>();
    sqlData.sqlCommand.CommandText = "select top 1 ndHumidity.Device, ndDevices.Description, ndDevices.Type, ndHumidSensor.Description, ndHumidSensor.Type, ndHumidity.Time, ndHumidity.Humidity from ndHumidity inner join ndDevices on ndHumidity.Device = ndDevices.Device inner join ndHumidSensor on ndHumidSensor.AnalogPort = ndHumidity.AnalogPort where ndHumidity.Device = '" + Id + "' order by ndHumidity.Time desc";
    SqlDataReader reader = sqlData.sqlCommand.ExecuteReader();
    while (reader.Read())
    {
        IDataRecord record = (IDataRecord)reader;
        hmHumidity humidity = new hmHumidity();
        humidity.device = (string) record[0];
        humidity.description = (string) record[1];
        humidity.type = (string) record[2];
        humidity.sensorDescription = (string) record[3];
        humidity.sensorType = (string) record[4];
        humidity.time = (DateTime)record[5];
        humidity.humidity = (double) record[6];
        humidities.Add(humidity);
    }
    return Json(humidities, JsonRequestBehavior.AllowGet);
}

You can see the API live here: http://azhomemonitor.azurewebsites.net/Help

Windows Phone Application

I did create a Windows 8.0 phone application that would consume the JSON from the web services listed above, but am replacing that with a Windows 8 Universal app that will have some ability to speak to the Netduino devices. So, stay tuned for more information on that.

Points of Interest

When I started this project, I am not sure that Subscriptions or Topics for Service Bus queues were around. Given my understanding of that mechanism, it will be something I will look into for the future. The ability to create a single outbound queue for all the devices, but specifically target a single device via a Topic would be interesting.

I also envision creating a portable class library and just using the XML/JSON serialization routines to serialize the classes so I don’t manipulate any XML or JSON data directly. I have implemented this in another Azure based system so know it works and has less code.

History

I started this project in early 2014 but had let it languish a bit. When I saw this contest I decided to resurrect it and finish it off. Of course, this has led to some of the desired improvements listed above and a re-write of the phone application. I guess no project is ever completely "finished".