Click here to Skip to main content
15,065,328 members
Articles / Programming Languages / C#
Article
Posted 26 Jun 2021

Tagged as

Stats

6.8K views
91 downloads
11 bookmarked

A Simple Key-Value Store Microservice

Rate me:
Please Sign up or sign in to vote.
4.14/5 (7 votes)
26 Jun 2021CPOL3 min read
I have a very specific use-case where I need a microservice that manages a simple in memory data store, which I call a "bucket."
My specific use case is that I need to track who is using a resource and for how long. After listing the requirements about what I need and what I do not need, we will take a look at the API, implementation, helper classes and testing.

Introduction

I have a very specific use-case where I need a microservice that manages a simple in memory data store, which I call a "bucket." What surprises me is that my brief Googling for an existing simple solution came up empty, but that's par for the course as simple things grow into complex things and the simple things end up being forgotten.

My specific use case is that I need to track who is using a resource and for how long. The user has the ability to indicate that they are using a resource and when they are done using that resource, and the user has the ability to add a note regarding the resource usage. Other users can therefore see who is using a resource, how long they've been using, and any notes about their usage. Given this, my requirements are more about what I don't need than what I do need.

What I Need

  1. The ability to set a key to a value.

What I Don't Need

  1. I don't even need the ability to manage different buckets but it seems sort of silly not to implement that feature, so it's implemented.
  2. I don't care if the memory is lost if the server restarts. This is for informational data only that will be shared across different clients. Technically, one of the front-end clients (because the client would have the full data) could even restore the data for everyone else.
  3. I don't need complex data structures, just key-value pairs where the value is some value type, not a structure.
  4. I don't even care about one user stepping on top of another - in actual usage, this doesn't happen, and if it did, I don't care -- whoever updates a key last wins.

Redis

"Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. "

Yikes. Definitely major overkill for what I need. That said, please don't use this to replace something like Redis unless your requirements really fit the bill here.

The API

Browsing to http://localhost:7672/swagger/index.html, we see:

As you can see (sort of) from the Swagger documentation, we have endpoints for:

  • Get the contents of a bucket by its name.
  • List all the buckets currently in memory, by their name.
  • Get the bucket object, which will include the bucket's data and the bucket's metadata which is just the bucket name.
  • Delete a bucket.
  • Set the bucket's data to the key-value pairs in the POST body.
  • Update a bucket's data with the specified key-value.

Implementation

The implementation (C# ASP.NET Core 3.1) is 123 lines of controller code. Very simple. Here it is in its entirety.

C#
using System.Collections.Generic;
using System.Linq;

using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.ModelBinding;

using MemoryBucket.Classes;

namespace MemoryBucket.Controllers
{
  [ApiController]
  [Route("[controller]")]
  public class MemoryBucketController : ControllerBase
  {
    private static Buckets buckets = new Buckets();
    private static object locker = new object();

    public MemoryBucketController()
    {
    }

    /// <summary>
    /// Returns the contents of an existing bucket.
    /// </summary>
    [HttpGet]
    public object Get([FromQuery, BindRequired] string bucketName)
    {
      lock (locker)
      {
        Assertion.That(buckets.TryGetValue(bucketName, out Bucket bucket), 
            $"No bucket with name {bucketName} exists.");

        return bucket.Data;
      }
    }

    /// <summary>
    /// Lists all buckets.
    /// </summary>
    [HttpGet("List")]
    public object GetBucket()
    {
      lock (locker)
      {
        return buckets.Select(b => b.Key);
      }
    }

    /// <summary>
    /// Returns the bucket itself.
    /// </summary>
    [HttpGet("Bucket")]
    public object GetBucket([FromQuery, BindRequired] string bucketName)
    {
      lock (locker)
      {
        Assertion.That(buckets.TryGetValue(bucketName, out Bucket bucket), 
           $"No bucket with name {bucketName} exists.");

        return bucket;
      }
    }

    /// <summary>
    /// Deletes an existing bucket.
    /// </summary>
    [HttpDelete("{bucketName}")]
    public void Delete(string bucketName)
    {
      lock (locker)
      {
        Assertion.That(buckets.TryGetValue(bucketName, out Bucket bucket), 
           $"No bucket with name {bucketName} exists.");

        buckets.Remove(bucketName);
      }
    }

    /// <summary>
    /// Creates or gets an existing bucket and replaces its data.
    /// </summary>
    [HttpPost("Set")]
    public object Post(
      [FromQuery, BindRequired] string bucketName, 
      [FromBody] Dictionary<string, object> data)
    {
      lock (locker)
      {
        var bucket = CreateOrGetBucket(bucketName);
        bucket.Data = data;

        return data;
      }
    }

    /// <summary>
    /// Creates or gets an existing bucket and updates the specified key with the 
    /// specified value.
    /// </summary>
    [HttpPost("Update")]
    public object Post(
    [FromQuery, BindRequired] string bucketName,
    [FromQuery, BindRequired] string key,
    [FromQuery, BindRequired] string value)
    {
      lock (locker)
      {
        var bucket = CreateOrGetBucket(bucketName);

        var data = bucket.Data;
        data[key] = value;

        return data;
      }
    }

    private Bucket CreateOrGetBucket(string bucketName)
    {
      if (!buckets.TryGetValue(bucketName, out Bucket bucket))
      {
        bucket = new Bucket();
        bucket.Name = bucketName;
        buckets[bucketName] = bucket;
      }

      return bucket;
    }
  }
}

The Helper Classes

Try not to be awed. I really don't think this needs an explanation.

The Bucket Class

C#
using System.Collections.Generic;

namespace MemoryBucket.Classes
{
  public class Bucket
  {
    public string Name { get; set; }
    public Dictionary<string, object> Data { get; set; } = new Dictionary<string, object>();
  }
}

The Buckets Class

C#
using System.Collections.Generic;

namespace MemoryBucket.Classes
{
  public class Buckets : Dictionary<string, Bucket>
  {
  }
}

Testing

Some Postman tests suffice here.

Create a Bucket

curl --location --request POST 'http://localhost:7672/memoryBucket/Set?bucketName=Soup' \
--header 'Content-Type: application/json' \
--data-raw '{
"Name":"Marc",
"Resource": "Garlic",
"Note": "For the soup"
}'

and we see the response as:

JavaScript
{
  "Name": "Marc",
  "Resource": "Garlic",
  "Note": "For the soup"
}

Update a Bucket

Kate is taking over making the soup:

curl --location 
--request POST 'http://localhost:7672/memoryBucket/Update?bucketName=Soup&key=Name&value=Kate'

and we see the response as:

JavaScript
{
  "Name": "Kate",
  "Resource": "Garlic",
  "Note": "For the soup"
}

Listing Buckets

I'm adding another bucket:

curl --location --request POST 'http://localhost:7672/memoryBucket/Set?bucketName=Salad' \
--header 'Content-Type: application/json' \
--data-raw '{
"Name":"Laurie",
"Resource": "Lettuce"
}'

And when I list the buckets:

curl --location --request GET 'http://localhost:7672/memoryBucket/List'

I see:

JavaScript
[
  "Salad",
  "Soup"
]

Get the Bucket Itself

curl --location --request GET 'http://localhost:7672/memoryBucket/Bucket?bucketName=Soup'

And I see:

JavaScript
{
  "name": "Soup",
  "data": {
    "Name": "Marc",
    "Resource": "Garlic",
    "Note": "For the soup"
  }
}

Deleting Buckets

Dinner's ready:

curl --location --request DELETE '<a href="http://localhost:7672/memoryBucket/Soup">
http://localhost:7672/memoryBucket/Soup</a>'
curl --location --request DELETE '<a href="http://localhost:7672/memoryBucket/Salad">
http://localhost:7672/memoryBucket/Salad</a>'
curl --location --request GET 'http://localhost:7672/memoryBucket/List'

No more buckets:

JavaScript
[]

Conclusion

Simple, right? Makes me wonder why we don't do simple anymore.

History

  • 26th June, 2021: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Marc Clifton
Architect Interacx
United States United States
Blog: https://marcclifton.wordpress.com/
Home Page: http://www.marcclifton.com
Research: http://www.higherorderprogramming.com/
GitHub: https://github.com/cliftonm

All my life I have been passionate about architecture / software design, as this is the cornerstone to a maintainable and extensible application. As such, I have enjoyed exploring some crazy ideas and discovering that they are not so crazy after all. I also love writing about my ideas and seeing the community response. As a consultant, I've enjoyed working in a wide range of industries such as aerospace, boatyard management, remote sensing, emergency services / data management, and casino operations. I've done a variety of pro-bono work non-profit organizations related to nature conservancy, drug recovery and women's health.

Comments and Discussions

 
SuggestionWhat about ConcurrentDictionary<TKey,TValue> Pin
Alex Sanséau2-Jul-21 2:29
MemberAlex Sanséau2-Jul-21 2:29 
GeneralRe: What about ConcurrentDictionary<TKey,TValue> Pin
Nelek6-Jul-21 0:21
protectorNelek6-Jul-21 0:21 
GeneralMy vote of 5 Pin
raddevus29-Jun-21 3:25
mvaraddevus29-Jun-21 3:25 
GeneralRe: My vote of 5 Pin
Marc Clifton30-Jun-21 2:02
mvaMarc Clifton30-Jun-21 2:02 
SuggestionI am doing something similar in my IoT projects Pin
wmjordan29-Jun-21 2:32
professionalwmjordan29-Jun-21 2:32 
GeneralRe: I am doing something similar in my IoT projects Pin
Marc Clifton30-Jun-21 2:01
mvaMarc Clifton30-Jun-21 2:01 
QuestionRedis Pin
Amaury Fages27-Jun-21 22:15
MemberAmaury Fages27-Jun-21 22:15 
AnswerRe: Redis Pin
Marc Clifton28-Jun-21 9:00
mvaMarc Clifton28-Jun-21 9:00 
SuggestionLarge Object Heap Fragmentation Pin
Stylianos Polychroniadis27-Jun-21 9:57
MemberStylianos Polychroniadis27-Jun-21 9:57 
GeneralRe: Large Object Heap Fragmentation Pin
Marc Clifton28-Jun-21 8:59
mvaMarc Clifton28-Jun-21 8:59 
QuestionHmmm Pin
Tony 227-Jun-21 6:58
MemberTony 227-Jun-21 6:58 
AnswerRe: Hmmm Pin
Tomaž Štih27-Jun-21 22:50
MemberTomaž Štih27-Jun-21 22:50 
GeneralRe: Hmmm Pin
Marc Clifton28-Jun-21 8:55
mvaMarc Clifton28-Jun-21 8:55 
AnswerRe: Hmmm Pin
Marc Clifton28-Jun-21 8:34
mvaMarc Clifton28-Jun-21 8:34 
GeneralRe: Hmmm Pin
Member 1453048028-Jun-21 11:43
MemberMember 1453048028-Jun-21 11:43 
If this is going to be used by an SPA, shouldn't there be some sort of authentication on the endpoints?

I think Tony-2 came off as a bit critical, but to elaborate on his scalability/DevOps points:

RE Scalability:

- Since you're using instance memory, there will of course be a limit on how heavily the service can be used. If it gets used by too many consumers, you'll eventually get an `OutOfMemoryException`.

- Since you're using instance memory, it's not easy to just launch a second instance of the service and expect things to continue to work. You'd either have to move the storage to a distributed location (like Redis) or set up sticky sessions, so that consumers always connect to the same instance. I think these two points are what Tony was talking about from a scalability perspective.

RE DevOps:
- Assuming you're using a cloud provider (which you may not be), an extra service like this means new resources, roles etc need to be set up, and the service needs to be monitored for those potential memory issues. It'd be much easier from a DevOps perspective to just set up a caching service like ElastiCache.
- I realise that my previous point doesn't really apply if you're not in an organisation using cloud hosting.

My takeaway:

- You originally compared the service to something like Redis, but you really wouldn't want SPA clients connecting to Redis in the first place, for both security and latency reasons.
- I think if you add some JWT authentication to your service, you could have your SPA clients connect to your API, which uses Redis as an underlying data store, instead of instance memory. This would solve your scalability issues, and still be simple to consume.
GeneralRe: Hmmm Pin
Marc Clifton30-Jun-21 2:00
mvaMarc Clifton30-Jun-21 2:00 
GeneralRe: Hmmm Pin
Member 1453048030-Jun-21 11:39
MemberMember 1453048030-Jun-21 11:39 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.