Click here to Skip to main content
14,984,271 members
Articles / Database Development / Redis
Tip/Trick
Posted 16 Mar 2020

Stats

4.8K views
5 bookmarked

Leader Election with Redlock.net

Rate me:
Please Sign up or sign in to vote.
4.75/5 (5 votes)
16 Mar 2020CPOL3 min read
How to perform distributed lock between multiple instances of a microservice with Redlock.Net
Sometimes, there is a need to ensure that only one instance of a microservice executes a piece of code. This might be done to ensure application correctness or to implement a poor man's failover. The article shows how to perform this with Redlock.Net library.

Why Locking Things

Microservice architecture becomes widely adopted these days. One of the benefits it offers is the possibility of horizontal scaling which allows us to increase the performance of our application dramatically. However, there are situations when multiple instances of service face contention for some shared resource. Consider a service which apart from other functionality runs once per day some mission-critical job which should be executed in a single instance. At the same time, the deployment of a single instance is counterproductive because microservice bears other functionality which would benefit from horizontal scaling. One may argue that we can split such microservice into even smaller microservice but I’d be cautious against making microservices too granular. Another reason might be poor-man’s failover scenario, where one instance of a service executes work, while other is idle and waits just in case the first instance fails for some reason. As a solution, I offer to elect a single leader which would handle shared resource exclusively at a single point of time. Such well-known leader election algorithms such as Bully algorithm or Ring algorithm require a lot of ceremony and knowledge of the logical topology of your system in order to be implemented. That’s the reason why we’ll have a look at leader election using a distributed lock. You should use this pattern when the tasks in a distributed application need careful coordination and there’s no natural leader. As storage for a distributed lock, we’ll use Redis. Redis is in-memory key-value storage so we’ll take advantage of its speed. There is already a library that implements distributed lock over Redis. So we just have to make use of it.

The Code

The sample code can be accessed on github. Let’s break down what actually happens here. The idea behind leader election via distributed lock is whoever acquires lock over shared resource becomes a leader. So naturally, we have a lock key quite similarly to built-in C# lock construct.

C#
private const string _resource = "the-thing-we-are-locking-on";

Obviously, the storage is a single point of failure so we have to make sure that it is reliable. RedLock.net which we use for our case allows us to use multiple instances of Redis instead of single in order to improve reliability. Here’s how we create connection to Redis during start up.

C#
var endPoints = new List<RedLockEndPoint>
{
    new DnsEndPoint("redis1", 6379)
    new DnsEndPoint("redis2", 6379)
    new DnsEndPoint("redis3", 6379)
};
_distributedLockFactory = RedLockFactory.Create(endPoints);

Every instance tries to acquire a lock once in a given period of time. If it succeeds, it becomes a leader. If not, it will try once again later.

C#
private readonly TimeSpan _expiry = TimeSpan.FromSeconds(_expirySecondsCount);
_acquireLockTimer = new Timer(async state => await TryAcquireLock((CancellationToken)state), 
_cts.Token, 0, _expirySecondsCount * 1000);

However, the leader does not need to re-acquire a lock since it has auto-extend feature. At first encounter with RedLock.net, this might be unintuitive so it should be noted. Let’s have a look at TryAcquireLock method.

C#
private async Task TryAcquireLock(CancellationToken token)
{
    if (token.IsCancellationRequested)
        return;

    var distributedLock = await _distributedLockFactory.CreateLockAsync(_resource, _expiry);
    if (distributedLock.IsAcquired)
    {
        DoLeaderJob();
        _acquireLockTimer.Dispose(); //no need to renew lock because of autoextend
    }
}

As mentioned above, we get rid of re-acquire timer as soon as an instance becomes a leader taking advantage of auto-extend feature. Once the instance fails, the lock is released and is up to other instances for a taking.

Summary

As we can see, the implementation of leader election via distributed lock is pretty straightforward. Still, it should be used with care since every locking increases contention between instances of a microservice and thus reduces the benefits of horizontal scaling.

History

  • 16th March, 2020: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Bohdan Stupak
Software Developer
Ukraine Ukraine
https://twitter.com/BohdanStupak1

Comments and Discussions

 
QuestionRedlock safe or NOT? Pin
Sacha Barber18-Mar-20 1:02
mvaSacha Barber18-Mar-20 1:02 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.