Agumented Reality for IoTs

• Download mlidynamics.iot.zip - 801.7 KB

Introduction

The problem we are trying to solve is to augment the IoT device's sensory preception, to complement it when one of the sensor is decomissioned or faulty. A likely example would be when we use our phone as the primary locator within an office building. When we misplace the phone, our location identified is not accurate ( example an IoT based coffee maker stationed at the location, where is phone is located would suddenly turn on to serve us coffee, but we are no longer at that location ). However with IoT platform using augmented reality, the IoT device would be able identify our location using a-priori knowledge and sensory preception from other IoT devices we interact with.

Inorder to solve the problem, we will be using the azure infrastructure as a PaaS inorder to build the platform which provides augmented sensory preceptions for the IoT devices connected to the platform.

Background

A background on microprocessors and Azure platform would be useful in understanding the crux of this refrence architecture tutorial. The tutorial is laid out as simple as possible, for the reader to grasp the essentials in building a PaaS solution for IoT devices.

Using the code

The code provided in the solution file is for the PaaS reference architecture. This is a POC (proof of concept) to build an IoT PaaS platform for augementing sensor data using predictive analytics.

Arduino based sensor network

For this POC we are going to use the arduino based network with motion sensors. Each transreceiver has a unique id assigned using the CHILD_ID within the arduino sketch depicted below.

Using Nrf24l01 Radio Transreceivers

Arduino sketch for the motion sensor to capture location using HC-SR501 motion sensor. Please refer to details on connecting the sensor at, http://www.mysensors.org/build/motion.

#include <mysensor.h>
#include <spi.h>

unsigned long SLEEP_TIME = 120000; // Sleep time between reports (in milliseconds)
#define DIGITAL_INPUT_SENSOR 3   // The digital input you attached your motion sensor.  (Only 2 and 3 generates interrupt!)
#define INTERRUPT DIGITAL_INPUT_SENSOR-2 // Usually the interrupt = pin -2 (on uno/nano anyway)
#define CHILD_ID 1   // Id of the sensor child

MySensor gw;
// Initialize motion message
MyMessage msg(CHILD_ID, V_TRIPPED);

void setup()  
{  
  gw.begin();

  // Send the sketch version information to the gateway and Controller
  gw.sendSketchInfo("Motion Sensor", "1.0");

  pinMode(DIGITAL_INPUT_SENSOR, INPUT);      // sets the motion sensor digital pin as input
  // Register all sensors to gw (they will be created as child devices)
  gw.present(CHILD_ID, S_MOTION);
  
}

void loop()     
{     
  // Read digital motion value
  boolean tripped = digitalRead(DIGITAL_INPUT_SENSOR) == HIGH; 
        
  Serial.println(tripped);
  gw.send(msg.set(tripped?"1":"0"));  // Send tripped value to gw 
 
  // Sleep until interrupt comes in on motion sensor. Send update every two minute. 
  gw.sleep(INTERRUPT,CHANGE, SLEEP_TIME);
}</spi.h></mysensor.h>

Controller Gateway Using Arduino

The sensor network communicates to the controller gateway via RF and the message packets are posted from the controller gateway to MQTT hosted on azure. Inorder to accomplish it we are using the mqttclientgateway hosted on https://github.com/FotoFieber/MySensors/blob/mqttclient/libraries/MySensors/examples/MQTTClientGateway/MQTTClientGateway.ino

Nodejs MQTT on Raspberry PI

Install nodejs on Raspberry PI and MQTT server on Raspberry PI

Follow the posts on, http://workingmatt.blogspot.com/2014/05/install-nodejs-on-raspberry-pi.html.

Install mqtt on nodejs by running,

npm install mosca

npm install mqtt

var mqtt    = require('mqtt');
var client  = mqtt.connect('mqtt://127.0.0.1');
 
client.on('connect', function () {
  client.subscribe('MySensor');
  client.publish('MyMQTT/#', '');
});
 
client.on('message', function (topic, message) {
  // message is Buffer 
  // call rest api - to post location
  client.end();
});

Azure PaaS platform for Augmenting Sensors

MDM

The platform MDM is to be implemented to maintain the devices and its profile information that can be used by analytics operation to augement the sensor data.

Event Ingestion

Event ingestion is via a REST API exposed by the PaaS solution to be platform independent. The REST api layer is named mlidynamics.iot.io, this is a WEB API 2.0 published to Azure and hosted.

I have hosted the API at http://mlidynamics-iot.azurewebsites.net/Help. We would publish it from our solution into the Azure subscription -

We therefore need the IoT device connected to the internet to access this API and publish the event. Example Json is as follows,

{
  "DeviceId": "90bec401-36da-4750-a07e-7c666767190f",
  "Channel": "sample string 2",
  "Location": 3,
  "Temperature": 4.1,
  "Humidity": 5.1,
  "PresenceDetected": 6
}

The events are then published to the event hub also hosted in Azure. The devices are secured via the device id's registered in the MDM. Security is off utmost importance in IoT devices used for patient monitoring and this architecture is built with that in consideration. The code kept in the attached solution is simplified for wide audience who would be reading this article. The solution can be expanded as per requirements for the implementation domain.

[Route("Publish")]
        public async Task<ihttpactionresult> Publish(TelemetryModel message)
        {
            // do processing here
            var jsonString = JsonConvert.SerializeObject(message);

            _client.Send(new EventData(Encoding.UTF8.GetBytes(jsonString))
            {
                PartitionKey = "0"
            });

            return Ok();
        }</ihttpactionresult>

Notification hub is setup on Azure, which has the capability to ingest millions of events. As per this document, the events are generated from the nodejs hosted on the Raspberry PI which is the controller from the IoT sensor devices.

Quote:

Notification Hubs eliminate one major complexity: you do not have to manage the challenges of push notifications. Instead, you can use a Notification Hub. Notification Hubs use a full multiplatform, scaled-out push notification infrastructure, and considerably reduce the push-specific code that runs in the app backend. Notification Hubs implement all the functionality of a push infrastructure.

Pros of using Event Hub

• Log millions of events per second
• Simple authorization mechanism
• Time-based event buffering
• Elastic scale
• Pluggable adapters for other cloud services

Cons of using Event Hub

• No all features from Service Bus Topic exist (but is acceptable)
• The size of the event data is limited to 256KB (the size is pretty okay)
• Number of partitions is limited to maximum 32

There is ample documentation available on the microsoft site to setup the azure notification hub. In this I have set the notification hub named "mlidynamics-iot-telemetry". This is the hub to which the REST API publishes the events too.

private readonly EventHubClient _client =
            EventHubClient
                .CreateFromConnectionString(ConfigurationManager.AppSettings[@"Microsoft.ServiceBus.ConnectionString"],
                    "mlidynamics-iot-telemetry");

Once these events are published to the notification hub, I am using stream analytics another Azure component to process the events in about realtime and stage it on the Azure SQL database.

Quote:

Stream Analytics provides out-of-the-box integration with Event Hubs to ingest millions of events per second. Together, Event Hubs and Stream Analytics let you process massive amounts of real-time data so you can make business decisions in real-time.

A simple stream query is written for this POC, to create location information sent by the remote IoT sensors using the arduino coupled to a RF transmitter and proximitry sensor. This query will create a frequency count of the sensor triggers when someone comes with its proximity.

SELECT 
    DeviceId, Location, DateAdd(minute,-15,System.TimeStamp) as WinStartTime,
    System.Timestamp as WinEndTime, Count(*) as EventCount
    FROM [telemetry-input]
    group by TumblingWindow( minute , 15 ), DeviceId, Location

The stream analytics would aggregate and store the result in a Azure Sql storage. The schema used to capture the aggregrates,

USE [mlidynamics-iot]
GO

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[LocTelemetry] (
    [DeviceId]     CHAR (54)     NULL,
    [Location]     BIGINT        NULL,
    [WinStartTime] DATETIME2 (6) NULL,
    [WinEndTime]   DATETIME2 (6) NULL,
    [EventCount]   BIGINT        NULL
);


GO
CREATE CLUSTERED INDEX [LocTelemetry]
    ON [dbo].[LocTelemetry]([DeviceId] ASC);

Azure Ml (Machine Learning) for Predictive Analytics

Using Azure ML (Azure Machine Learning) to predict the closest location based on the location in the time series sensor data from the generated by the stream analytics. The machine learning predicts the closest location in that particular time window. In this POC, I have broken the time window within a 15 minute interval. The features are built on a week, minute schedule for the training set required by the Machine Learning algorithm. The stream analytics creates a frequency count of sensor hits whenever someone comes within the proximity. The analytics built is a time series segmented within a 15 minute interval. Using this time and location we would be able to predict the closes location that a person whose schedule was used in the training. Using machine learning we are augmenting the sensor data with predicted location. The sensor didn't have a notion of the location but only notion of whether an event occured.

The REST API that is exposed by Azure ML

https://ussouthcentral.services.azureml.net/workspaces/84c646bc34b0432d82f03dd5c236b66f/services/79793a0a274b44a7b74aa6336dacefaf/execute?api-version=2.0&details=true

The request json that is to be used in the POST REST API call

    {
  "Inputs": {
    "input1": {
      "ColumnNames": [
        "wk",
        "hr",
        "loc",
        "closest loc"
      ],
      "Values": [
        [
          "0",
          "0",
          "value",
          "value"
        ],
        [
          "0",
          "0",
          "value",
          "value"
        ]
      ]
    }
  },
  "GlobalParameters": {}
}

The response json that would be obtained from the REST API call

{
  "Results": {
    "output1": {
      "type": "DataTable",
      "value": {
        "ColumnNames": [
          "wk",
          "hr",
          "loc",
          "Scored Labels",
          "Scored Probabilities"
        ],
        "ColumnTypes": [
          "Numeric",
          "Numeric",
          "String",
          "String",
          "Numeric"
        ],
        "Values": [
          [
            "0",
            "0",
            "value",
            "value",
            "0"
          ],
          [
            "0",
            "0",
            "value",
            "value",
            "0"
          ]
        ]
      }
    }
  }
}

The sample code that can be used to predict the location, by calling the REST API exposed by Azure ML

// This code requires the Nuget package Microsoft.AspNet.WebApi.Client to be installed.
// Instructions for doing this in Visual Studio:
// Tools -> Nuget Package Manager -> Package Manager Console
// Install-Package Microsoft.AspNet.WebApi.Client

using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;

namespace CallRequestResponseService
{

    public class StringTable
    {
        public string[] ColumnNames { get; set; }
        public string[,] Values { get; set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            InvokeRequestResponseService().Wait();
        }

        static async Task InvokeRequestResponseService()
        {
            using (var client = new HttpClient())
            {
                var scoreRequest = new
                {

                    Inputs = new Dictionary<string, stringtable=""> () { 
                        { 
                            "input1", 
                            new StringTable() 
                            {
                                ColumnNames = new string[] {"wk", "hr", "loc", "closest loc"},
                                Values = new string[,] {  { "0", "0", "value", "value" },  { "0", "0", "value", "value" },  }
                            }
                        },
                                        },
                                    GlobalParameters = new Dictionary<string, string="">() {
}
                };
                const string apiKey = "abc123"; // Replace this with the API key for the web service
                client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue( "Bearer", apiKey);

                client.BaseAddress = new Uri("https://ussouthcentral.services.azureml.net/workspaces/84c646bc34b0432d82f03dd5c236b66f/services/79793a0a274b44a7b74aa6336dacefaf/execute?api-version=2.0&details=true");
                
                // WARNING: The 'await' statement below can result in a deadlock if you are calling this code from the UI thread of an ASP.Net application.
                // One way to address this would be to call ConfigureAwait(false) so that the execution does not attempt to resume on the original context.
                // For instance, replace code such as:
                //      result = await DoSomeTask()
                // with the following:
                //      result = await DoSomeTask().ConfigureAwait(false)


                HttpResponseMessage response = await client.PostAsJsonAsync("", scoreRequest);

                if (response.IsSuccessStatusCode)
                {
                    string result = await response.Content.ReadAsStringAsync();
                    Console.WriteLine("Result: {0}", result);
                }
                else
                {
                    Console.WriteLine("Failed with status code: {0}", response.StatusCode);
                }
            }
        }
    }
}

</string,></string,>

Testing the webservice, to predict the location based on the time series data we have. In this test, we are trying to find the location on the 1st week of the week, at the 1st hour and at the 30th second assuming the sensor in the study got triggered at about that time and the closest location assuming is the bedroom.

The results, identified the living room was the closest location based on the sensor readings.

Points of Interest

Most of the Azure components are preview versions and have its short comings. This article focuses on using the off the shelf components in Azure to accomplish the POC depicted in this article. In particular the article is kept less verbose and has guiding points to build a custom IoT PaaS solution. Its always challenging when we manage millions of IoT messages and process the streams of data. The focus was on pointing the required components required for building the PaaS platform for IoT. Cloud services such as Azure help in quickly creating the platform.

History

Keep a running update of any changes or improvements you've made here.