Data Cache Synchronization using AWS SNS and Lambda

Introduction

In this article I will present a use-case for the SNS publish-subscriber capabilities within Amazon Web Services (AWS). I have a hybrid SQL Server-MongoDB sample application for managing account holders at a ficticious financial firm. The SQL Server database serves as the system of record for the account holder data. All inserts, updates, and deletes of the account holder data is committed to the SQL Server database from the application using Entity Framework. The MongoDB document database seves as the cache and is used for searching account holders because of the significant performance gains by using NoSQL document databases.

While AWS may not be appropriate for all data synchronization problems, if your data cache system is hosted on AWS, SNS and Lambda provides an elegant solution to keep your AWS-hosted cache in sync. We will be covering the following AWS technologies in this article:

SNS is a pub-sub messaging services from AWS that can be used as the infrustructure for enterprise messaging in the cloud. https://aws.amazon.com/sns/

Lambda is a service from AWS that lets you run code with provisioning or managing servers. https://aws.amazon.com/lambda/

AWS SDK for .NET provided APIs for working with AWS services. https://aws.amazon.com/sdk-for-net/

CloudWatch is a monitoring service for AWS cloud resources and applications running on AWS. https://aws.amazon.com/cloudwatch/

Below is a diagram that summarizes how the data synchronization will work after integrating AWS SNS and Lambda.

I am assuming that you have already created an AWS account and you can access the IAM, Lambda, and SNS sections of the AWS interface. This is a Visual Studio solution so I am also assuming that you are using a Windows machine with Visual Studio and you know how to use NuGet for the AWS SDK package.

AWS Configuration

Create an AWS IAM user for sending messages to SNS

From the AWS IAM page, create a user and attach the AWSSNSFullAccess policy to the user. 

An access key ID and secret access code are created by AWS for the user. Once the user is created you will have the opportunity to view and record the secret access code just once. Once you leave the screen, for security purposes, you probably won't have the ability to view the secret access code from the AWS UI again for this user so you want to record the secret access code in a secure place outside of AWS. Refer to the AWS documentation for best practices regarding the security credentials.

You will need the access key ID and secret access code later on when connecting to AWS SNS from our application code.

Create the SNS Topic

From the SNS page, create a topic. Record the topic ARN and Region because you will need this information to send messages to the SNS Topic from your application code.

Create the Lamba function

For this example we are going to create a Lambda function using Python however you can use Node.JS as well.

First you will need to create a separate folder on your local machine to store the Python script as well as dependent Python modules which I will talk about later on. I called my folder "AWSLambdaUpdatePerson". Then I created the Python script called AWSLambdaUpdatePersonMongo.py in the new folder. The script defines an event handler that uses the MongoDB Python library to connect to my MongoDB instance on an AWS EC2 machine and update the person collection with the JSON in the event message body. I used the subject of the message to store the ID of the person record getting updated. The event handler in the Lambda function will be invoked by SNS when SNS notifies its subscribers upon receiving a message from the application. The event parameter in the event handler contains the message from SNS.

from __future__ import print_function

import json
import pymongo

def lambda_handler(event, context):
    # Open a connection to the Mongo instance on EC2
    client = pymongo.MongoClient("mongodb://****")

    # Get a reference to the footloosefs Mongo database
    db = client.footloosefs

    # The ID of the person we are updating will be in the subject
    personId = event['Records'][0]['Sns']['Subject']

    # The message will contain the JSON of the person attributes
    message = event['Records'][0]['Sns']['Message']
    
    print("From SNS: " + message)    
    print("From SNS: " + personId)  

    if personId:
        
         # Remove the document representing the person from the collection
        db.persons.remove( { "_id" : int(personId)  } )

        # Add the message with the updated person to the collection
        db.persons.insert_one( json.loads(message) )

        print("Processed insert or update for person: " + personId)

    return

The SNS message contains the data in JSON of the person to be updated in the persons MongoDB collection. The Python script connects to the MongoDB instance, removes the existing person document if any, and inserts a new person document using the JSON in the SNS message body.

Because the Python script depends on pymongo, the MongoDB Python library, we need to download the dependent Python libraries in this folder that contains the script. To do this we will need to install Python.

Because I am using a Windows machine, I am using Python 3.5 for Windows which I installed from the Python Windows download website. https://www.python.org/downloads/windows/. The good thing about Python 3.5 is that is comes with pip preinstalled. pip is the preferred Python installer for installing custom Python libraries..

I also installed 7-Zip for creating archives. http://www.7-zip.org/

In the same folder for convenience I created a BAT file that runs the following commands:

python -m pip install pymongo -t AWSLambdaUpdatePerson

7z a awslambda.zip AWSLambdaUpdatePerson\*

The python command runs pip which installs pymongo to the folder and all of its dependencies. Note that the pip command will use the Internet to access the Python central repository and download the MongoDB Python library similar to how NuGet works. The 7z command compresses the entire contents of the folder into a zip file awslambda.zip. How you create the zip is up to you but it must contain the Python script file and any depenencies that are not part of the AWS Python SDK aka Boto3.

Here is what my zip file looks like. The bson and gridfs folders are required by pymongo so they were downloaded by pip as well as the core pymongo library.

Finally we will go to the AWS Console and create the Lambda and upload the zip file.

From the AWS Lambda landing page click on the "Get Started Now" button. Then click the "Skip" button on the bottom-right (basically we are skipping the blueprint selection because we already have the code for the Lambda). In the next screen enter a name and description for the Lambda and select Python 2.7 as the runtime. You will need to remember the name of the Lambda for the next section when you create the SNS subscription.

Then select "Upload a ZIP" and upload the zip file that you created.

In the Handler input, change the name of the handler function to the name of the Python script that you created. So change "lambda_function.lambda_handler" to "AWSLambaUpdatePersonMongo.lambda_handler".

At this point you will be prompted to use an existing Role or create a new lambda basic role for the script.

Then click the "Next" button where you will have a confirmation page and then click "Create"

Create an SNS subscription

Now we need to create an SNS subscription so that the Lamba function is invoked whenever our application code sends a message to the SNS topic.

Go to SNS Topics listing page and select the topic that you created earlier. In the Action menu select "Subscribe to topic". In the dialog box, select "AWS Lambda" as the Protocol and then select the Lambda that you created. Then click the "Create Subscription" button.

Application Code to send notification to SNS

Install the AWS SDK using NuGet

Use NuGet to install AWSSDK.Core and AWSSDK.SimpleNotificationService libraries to your Visual Studio project. I used ASKSDK version 3.1 in my examples. Since my application involves serializing objects to JSON, I also installed the Newtonsoft JSON library using NuGet.

Store AWS user credentials

An application that uses AWS services must be authenticated with an identity that has permissions to use the services requested. Earlier we created an AWS user that has permissions to publish messages to SNS. Now I need to supply the credentials (access key ID and secret access key) to the application. Rather than hard-coding the credentials in the code or web.config, I created a file with the credentials on the local machine and gave the AWS SDK the location of the file.

The contents of the file contains the name of the profile, the access key id and the secrey access key in the following format (Note that the actual credentials have been masked with *****.

[UpdatePersonCollection]

aws_access_key_id = *****

aws_secret_access_key = *****

On my machine I named the file "credentials" in the location C:\awsprofiles.

Then I included the following key-values in the appSettings section of web.config. The AWSProfileName key references the name of the file in the credentials file and the AWSProfileLocation tells the AWS SDK where the credentials file is located.

    <add key="AWSSNSTopicArn" value="arn:aws:sns:us-east-1:127504306426:FootlooseFSPersonUpdate"/>

    <add key="AWSProfileName" value="UpdatePersonCollection"/>

    <add key="AWSProfilesLocation" value="C:\awsprofiles\credentials" />

There are other methods available for storing AWS credentials. They can be reviews in the AWS documentation link: http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/DeveloperGuide/net-dg-config-creds.html

Send a message to SNS using the ASK SDK

Now that we have installed the AWS SDK Core and SNS libraries and set up our credentials and web.config, we can write the code that sends message to the SNS topic. The code below will create a new PersonDocument object based on the updated Person EF entity, serialize the object using Newtonsoft JSON serializer, and send the message to the AWS SNS topic.

// Convert the Person Entity to a PersonDocument for MongoDB
var personDocument = new PersonDocument();

personDocument.PersonID = person.PersonID;
personDocument.EmailAddress = person.EmailAddress;
personDocument.FirstName = person.FirstName;
personDocument.LastName = person.LastName;
personDocument.PhoneNumber = person.Phones.Any(p => p.PhoneTypeID == 1) ? person.Phones.First(p => p.PhoneTypeID == 1).Number : string.Empty;

var address = person.Addresses.Any(a => a.AddressTypeID == 1) ? person.Addresses.First(a => a.AddressTypeID == 1).Address : null;

if (address != null)
{
  personDocument.StreetAddress = address.StreetAddress;

  personDocument.City = address.City;

  personDocument.State = address.State;

  personDocument.Zip = address.Zip;
}

// Serialize the PersonDocument object into a JSON string
var message = JsonConvert.SerializeObject(personDocument);

// Connect to AWS SNS
var topicArn = ConfigurationManager.AppSettings["AWSSNSTopicArn"];
var snsClient = new AmazonSimpleNotificationServiceClient(RegionEndpoint.USEast1);
          
try
{                
      // Send the message to the SNS Topic      
      var publishRequest = new Amazon.SimpleNotificationService.Model.PublishRequest();
      publishRequest.TopicArn = topicArn;
      publishRequest.Message = message;
      publishRequest.Subject = personId.ToString();

      snsClient.Publish(publishRequest);

      return true;
}
catch (Exception ex)
{
       // TODO Log exception
       return false;
}                       

When the AmazonSimpleNotificationServiceClient is instantiated, the AWS SDK will attempt to find authentication credentials at the location specified in the web.config. Note that we are storing the person ID in the subject of the message. With the topic ARN endpoint, subject and message, the AmazonSimpleNotificationServiceClient object publishes the message to the AWS SNS system.

Application Demo Time!

Now let's see the pub-sub process in action as a user of the application. The account holder (person) search screen gives the user the ability to query account holders by several attributes. These queries are run against the MongoDB person document collection on an AWS EC2 machine.

From this screen we select one of the persons to modify the person's information.

In this example we changed Marcia Aalderink's email address from hotmail.com to gmail.com. Then we saved the form. Saving the form, in addition to saving the information to our SQL Server system of record, executes the C# code snippet we saw earlier that creates a PersonDocument object, serializes the object, and sends a message to SNS.

Then we immediately go back to the search screen, which is refreshed when we return. In the search results, we see that Marcia's email address has already been updated in the MongoDB database. This means that in that short amount of time, SNS received the message, forwarded it to the Lambda subscriber, and the Lamba function was executed.

AWS Lambda executions are logged in the CloudWatch system. To see the Lambda logs, from the Aamazon Web Services landing page, go to "CloudWatch" and select "Logs". Then select the Lambda from the list of logs. You can see when the Lamnda was executed, the message subject, the content of the message that sent to the Lambda, which system sent the message, how long it took the Lambda to execute and the amount of memory used.

From CloudWatch you can also setup alerts to be sent out if there are any exceptions running the Lambda function.

Thanks for reading and if you have any experience working with AWS SNS and Lambda I would be interested hearing about them or any suggestions in general.