IoT System for Data Acquisition with REDIS and GO Language

• Download source code - 3 KB

Introduction

An IoT data acquisition system consists of a network of sensors that send your data to one or more central units that, in turn, process and store that data, allowing you to monitor and control devices located locally or remotely.

In this article, I will describe the architecture of the receptor module.

Background

The purpose of the receiver module is to allow the storage of data sent by a network of microcontrollers distributed locally or remotely.

• Each microcontroller device has a set of sensors whose measurements are sent to the central receiver through different communication protocols.
• The data is received by a concentrator similar to a broker or gateway.
• This data is then stored in a memory cache extremely quickly, being available for processing by the supervisory application.

To better understand, see the diagram below:

1. Sensor network → The sensors are connected to the microcontroller devices that send the data to the concentrator. There are no restrictions on the board models used, which may be Arduino, ESP8266, ESP32, PIC, etc.

   The boards, in fact, must be chosen according to the desired form of communication.

   However, there are operating and software standards that must be followed before the device can be connected to the system.

2. Communication forms and protocols → The following standards are provided for communication between the devices and the concentrator:
   • Wi-fi / Ethernet (Websocket / HTTP / REST)
   • RF / LORA
   • Serial / Bluetooth communication
   • GSM / GPRS
3. Receiving Routine → Routine in GO that receives the data from the sensors, processes it and sends it to the REDIS cache. There is a specific routine for each communication protocol described in item 2.

4. REDIS Cache → Stores sensor data temporarily or permanently. If necessary, the cache can be mirrored (replicated).

It is also possible to create a message publishing and subscription scheme similar to the MQQT protocol, but this will not be covered here.

Comments

1. Depending on the form of communication and the distance between the components, there may be additional layers of hardware / software for interconnection. This introduction is a generic and simplified description of the architecture;
2. The distribution of servers for each layer is not a decision defined by the architecture, but the ideal is to have a dedicated LINUX server for REDIS.
3. We are describing only the data acquisition module here. The control module will be detailed in another opportunity.

Technologies

We will now list the main software technologies used in the concentrator module:

1. REDIS is an in-memory data storage system with excellent performance that can be used as a NOSQL database, Cache or Message Broker.

   REDIS uses key-value data structures, such as linked lists, Strings, Sets, etc. In addition, data can be configured to be temporary or persisted in a variety of ways.

   I also highlight the support for the following features:

   • Replications
   • Stored Procedures
   • Partitioning;
   • Etc.
2. GO or Golang is a programming language created by Google and released in open source in November 2009. It is a statically typed language, compiled and focused on productivity and concurrent programming (Wikipedia).

   Go was designed for the multi-core reality of today's computers. Therefore, due to its robustness, it is widely used for programming on large-scale network servers and distributed systems.

Development Environment

We will now see how to prepare the development environment for the Linux OS.

Installing REDIS

To install REDIS on Linux, use apt:

sudo apt update
sudo apt install redis

To start the REDIS server as a service:

sudo systemctl start redis-server

To check the status of the server or stop the service:

sudo systemctl status redis
sudo systemctl stop redis

The REDIS package comes with a client application, redis-cli, which can be used for maintenance and testing. To see if everything is working, run the commands:

redis-cli
set hello world
get hello

If the system answered “world”, it means that everything is ok and you have just created a String data structure with the key “hello” and value “world”.

To exit the client, type: quit

Optionally, you can also install a client with more features and ease of use: Redis Desktop Manager

To learn more about REDIS:

• TutorialsPoint
• w3resource
• Documentação official

GO Language Installation

The first step is to download the installation package from the official website:

https://golang.org/dl/

Then unzip the file with the following command:

sudo tar -C /usr/local -xzf package_name

Where package_name is the name of the file you downloaded.

Example: go1.13.5.linux-amd64.tar.gz

The next step is to set up the environment.

I don't know if this is good or bad, but here among us, the GO language is authoritative in several aspects and you realize this already in the preparation of the development environment.

Before installing, you must already plan how your Workspace will be, that is, where your source code files will be. There is a pattern for this. It is not a rule, but it is better to obey • ᴗ •

Let's assume that your workspace will be in the /home/user/gocode folder. Then under that folder, 3 subfolders of the same level must be created:

• pkg → Folder where the packages and libraries are that you downloaded or created to reference in your programs
• src → Your source code files
• bin → Here are the executable files that you created and installed from the src folder. It is like a folder to deploy.

It is also necessary to create three environment variables:

• GOPATH → Path to your workspace. There can be more than one, in which case, separate by:
• GOROOT → Path where the GO was installed. Typically: /usr/local/go
• GOBIN → Folder where GO should generate executables. It may be the workspace bin folder. But it is up to you.

In addition, it is highly recommended to change the PATH variable to point to the GO binaries, usually: /usr/local/go/bin.

To create these environment variables, we use the export command. To do this, edit the ~/.profile file (sudo gedit ~/.profile) and place the export commands at the end of it.

In my case, it looked like this:

export PATH=$PATH:/usr/local/go/bin
export GOROOT=/usr/local/go
export GOPATH=/home/josecintra/code/go
export GOBIN=/home/josecintra/code/go/bin

Another thing: Golang emphasizes the use of version control systems, such as GIT. You will need one to install third party library packages.

If you have not installed one on your Linux system, see the installation instructions at this link.

Ready! Now we just need to choose an IDE.

IDEs for the GO Language

This is a personal matter. I like Geany a lot. It is already configured to edit, compile and run GO programs.

To install, use the following commands:

sudo add-apt-repository ppa:geany-dev/ppa
sudo apt-get update
sudo apt-get install geany geany-plugins-common

f you don't want to use Geany, here's an extensive list of IDEs with support for the GO language:

• https://github.com/golang/go/wiki/IDEsAndTextEditorPlugins

To learn more about the GO language:

• TutorialsPoint
• W3Adda
• A Tour of GO
• Effective GO

Data Modeling

Let's talk now about REDIS and how it will be used in this project module.

REDIS will function as an auxiliary database for the main DBMS of the supervisory system. The main function of REDIS will be to store the data from the sensor network to be displayed (and processed) in (almost) real time.

Due to its excellent performance, REDIS is ideal for this type of application and this will also save the system's main database.

In the supervisory system module, REDIS will have other functions not covered here.

The Scenario

The scenario we want to model is as follows: Several sensors arranged locally or remotely will periodically send their measurement data to a server. The collecting routine (GOLANG) will store the data in REDIS so that the supervisory application can process and present the information on a dashboard.

What we want to store from the sensors is just the following information:

• Identification of the sensor that is sending the information
• The date and time of sending
• The measurement value

Other information such as the type of sensor, location, unit of measure and others are NOT REDIS responsibility. This will be the responsibility of the supervisory WEB application.

The function of REDIS in this module is only to speed up the reception of data, leaving complexity to the supervision system.

Sensor identification: This is the sensor ID which is a String composed of 3 letters and a numerical sequence joined by the pipe separator (“|”).

For example:

• A temperature sensor, no matter which standard or model, may have the ID “TMP | 1”
• A humidity sensor can have the ID “HMD | 100” and so on.

Obviously, the IDs are not repeated, that is, there cannot be two sensors with the same identification, as in a primary key!

For REDIS, the sensor ID is just a dumb key and it doesn't know anything about it, unless it is possibly unique.

Date and time of sending: For reasons of agility, the date will be stored in the Unix Time Stamp format.

Measurement value: This is a String storing a numeric value. Similarly, REDIS does not know anything about this value, or where it comes from.

Note: The sensor ID and its measurement are provided by the microcontroller board to which the sensor is connected (Arduino family). The sending date is the responsibility of the receiving application developed in the GO language. This is for reasons of standardization and economy (the boards do not need to have an RTC module).

Data Structures

For those who work with relational databases, it may be strange how data is stored and accessed in REDIS. About this, we have already indicated several tutorials in the previous articles, so let's get straight to the point ...

Among the data structures made available by REDIS, we chose linked lists because the insertion and removal operations at the end of the list are extremely fast, which is what we need.

Let's see, as an example, what the operations of inserting and removing the list would look like.

Supposing that, on “2020/01/01” at “01:00:05” hours, the code sensor “TMP | 5” sent the value “25.25”.

The key to our list will simply be the sensor ID. Therefore, we will have a list for each sensor.

Converting the date to Unix, we have: “1577840405”. So the command to store this data at the end of the list would be this:

RPUSH TMP|5 1577840405|25.25

To obtain the last value sent by the “TMP | 5” sensor, we use the following command:

RPOP TMP|5

Note: Alternatively, for reasons of readability, we could represent the values in JSON format, a very common procedure in the REDIS world that has support for this technique.

Developing a Test Application

We will develop a test application (P.O.C.) for sending and receiving sensor data with GOLANG and REDIS.

For this particular test, we will use a NodeMCU board programmed in the Arduino IDE. This board will send, via WebSockets, the data from an NTC temperature sensor every 30 seconds.

The receiving application in Golang will receive the data, parse and store it in REDIS, making it available for the WEB application.

We will start by describing the application of NodeMCU.

Issuer Station nodemcu

Prototype

Components

1. NodeMCU card. In fact, it can be any ESP *. * Family board compatible with the Arduino IDE (ESP8266, NodeMCU, Wemos, ESP32, etc.)
2. 10K NTC thermistor type temperature sensor
3. Jumper wires and, optionally, a protoboard

Required Software

• To create the websocket server, we will need to install the ArduinoWebsockets which can be installed by the Arduino IDE itself through a library manager.
• The temperature data from the sensor will be sent every 3 minutes to the WEB server. To simplify our code, let's count the time with millis using the NeoTimer, library, which is also available in the manager.
• The temperature reading and calculation will be done through the following routine available on the Internet: Thermistor Interfacing with NodeMCU

Sketch

/*
  ESP8266-WS_Clent.ino - Esp8266 Websockets Client
  2020, by José Augusto Cintra 
  http://www.josecintra.com/blog

  This is free and unencumbered software released into the public domain.
  Anyone is free to copy, modify, publish, use, compile, sell, or
  distribute this software, either in source code form or as a compiled
  binary, for any purpose, commercial or non-commercial, and by any
  means.
  For more information, please refer to <http: unlicense.org="">

	This sketch:
        1. Connects to a WiFi network
        2. Connects to a Websockets server
        3. Periodically sends data from a temperature sensor to the server
        4. Receives return messages from the server
*/

// From library Manager
#include <arduinowebsockets.h> 
#include <esp8266wifi.h>
#include <neotimer.h>

const char* ssid = "x";  //Enter your SSID
const char* password = "y"; //Enter your Password
const char* websockets_server_host = "192.168.0.0"; //Enter your server address
const uint16_t websockets_server_port = 8080; // Enter server port

Neotimer mytimer = Neotimer(10000); // Time interval for sending sensor data
String tempString;
String sensor = "TMP|5";
using namespace websockets;

WebsocketsClient client;
void setup() {
    Serial.begin(9600);
    
    // Connect to wifi
    WiFi.begin(ssid, password);

    // Wait some time to connect to wifi
    for(int i = 0; i < 10 && WiFi.status() != WL_CONNECTED; i++) {
        Serial.print(".");
        delay(1000);
    }

    // Check if connected to wifi
    if(WiFi.status() != WL_CONNECTED) {
        Serial.println("No Wifi!");
        return;
    }

    Serial.println("Connected to Wifi, Connecting to server.");
    // try to connect to Websockets server
    bool connected = client.connect(websockets_server_host, websockets_server_port, "/");
    if(connected) {
        Serial.println("Connecetd!");
        //client.send("Hello Server");
    } else {
        Serial.println("Not Connected!");
    }
    
    // run callback when messages are received
    client.onMessage([&](WebsocketsMessage message) {
        Serial.print("Got Message: ");
        Serial.println(message.data());
    });
}

void loop() {
  // let the websockets client check for incoming messages
  if(client.available()) {
    client.poll();
  }
  // Periodic sending of temperature sensor data to the customer
  if (mytimer.repeat()) {
    tempString = readTemperature();
    Serial.println(tempString);
    client.send(sensor + "@" +tempString);
  }
}

// Temperature reading and calculation by the thermistor
double readTemperature() {
  // Code extracted from the 'Thermistor Interfacing with NodeMCU' tutorial available at:
  // https://www.electronicwings.com/nodemcu/thermistor-interfacing-with-nodemcu

  const double VCC = 3.3; // NodeMCU on board 3.3v vcc
  const double R2 = 10000; // 10k ohm series resistor
  const double adc_resolution = 1023; // 10-bit adc
  const double A = 0.001129148; // thermistor equation parameters
  const double B = 0.000234125;
  const double C = 0.0000000876741;
  double Vout, Rth, temperature, adc_value;

  adc_value = analogRead(A0);
  Vout = (adc_value * VCC) / adc_resolution;
  Rth = (VCC * R2 / Vout) - R2;

  temperature = (1 / (A + (B * log(Rth)) + (C * pow((log(Rth)), 3))));  // Temperature in kelvin
  temperature = temperature - 273.15;  // Temperature in degree celsius
  delay(500);
  return (temperature);
}

Golang Receiving Station

Required Software

We have previously shown how to install REDIS and the GO language. However, to establish a connection with REDIS and accept Node's websockets connections, we will need to install some library packages:

• GORILLA → Web Toolkit for the GO language
• REDIGO → Go Client forREDIS

To install the packages, enter the following commands on the virtual terminal:

go get github.com/gorilla/websocket
go get github.com/gomodule/redigo/redis

Let’s GO!

/*
  WS-redis.GO - Golang Websockets Server with REDIS integration
	2020, by José Augusto Cintra 
	http://www.josecintra.com/blog

  This is free and unencumbered software released into the public domain.
  Anyone is free to copy, modify, publish, use, compile, sell, or
  distribute this software, either in source code form or as a compiled
  binary, for any purpose, commercial or non-commercial, and by any
  means.
  For more information, please refer to <http: unlicense.org="">

	This routine:
        1. Starts a Websockets server
        3. Waits for client messages with sensor data  
        4. Parses the data received
        5. Stores data in a REDIS database 
*/

package main

import (
	"fmt"
	"strconv"
	"strings"
	"time"
	"log"
	"github.com/gomodule/redigo/redis" 
	"github.com/gorilla/websocket"
	"net/http"
)

// HTTP request handler
var upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
	CheckOrigin:     func(r *http.Request) bool { return true },
}

func main() {

  // Starts the server connection 
	http.HandleFunc("/", handler)
	http.ListenAndServe(":8080", nil)
}

func handler(writer http.ResponseWriter, request *http.Request) {
	socket, err := upgrader.Upgrade(writer, request, nil)
	if err != nil {
		fmt.Println(err)
	}
	for {
		// reading the message received via Websocket
		msgType, msg, err := socket.ReadMessage() // Dados recebido do sensor
		if err != nil {
			fmt.Println(err)
			return
		}

		// Tratar a mensagem recebida
		var msg_split []string = strings.Split(string(msg), "@")   // Splitted data
		currentTime := strconv.FormatInt(time.Now().Unix(), 10)    // Current date and time
		key := msg_split[0]                                        // Key to the linked list
                                                                   // on Redis
		value := currentTime + "|" + msg_split[1]                  // Value to the linked 
                                                                   // list on Redis

		// Send the data to REDIS
		conn, err := redis.Dial("tcp", "localhost:6379")
		if err != nil {
			log.Fatal(err)
		}
		defer conn.Close()
		_, err = conn.Do("RPUSH", key, value)
		if err != nil {
			log.Fatal(err)
		}

		// Optional: Returning the received message back to the customer
		err = socket.WriteMessage(msgType, msg)
		if err != nil {
			fmt.Println(err)
			return
		}
	}
}

Conclusion

The combination of the GO language and the REDIS database allows you to write robust and scalable applications with excellent performance, making them ideal for the IoT world.

The sample code shown here is didactic in nature. This application can be improved in several aspects:

• Encryption and security features
• Concurrency through GO routines
• Modularity
• etc.

At the next opportunity, we will show here how to develop the WEB application for control and monitoring.

Until then!

References

• REDIS Official Page
• Golang Official Page

History

• 2020-02-01 - Initial release