ASP.NET Core: Implement a Load Balancer

• Download source code - 47 KB
• Clone from GitHub
• Open an issue

Introduction

I spent some time learning .NET Core functionalities and I wanted to test it on a real project, so I started a new one. I don’t know why but I decided to implement a software load balancer. There are many options in the market and a lot of them are open source. This project started only to give me the opportunity for experimenting the framework so reinventing the wheel did not scare me.

I thought of a load balancer because it is managed in most of the implementation by request filter according with the “pipeline pattern”. The middleware of .NET Core (or Owin too…) are very similar so it seems to be the right application to go in deep on this technology.

What the ASP.NET Core Load Balancer Will Do

The behaviour of a load balancer is quite simple so I avoid wasting time explaining what a balancer is. Anyway, I’ll spend a few words describing how I decided to implement it.

Requirements

• Be plug and play: no complex installation
• Be standalone or integrated in web server (nginx, apache, iis)
• Changing configuration will provide: a proxy server, a balancing server, both of them
• Use as much as possible what ASP.NET Core gives out of the box
• Keeping in mind performances

Modules

The main idea is to define a set of “modules” that can be activated or not based on configuration. It has to be possible to add new modules and allow third parties to develop their one.

Filters

This module provides an easy way to filter request based on some rules. All requests that match the filter will be dropped. Each url is tested over a set of rules. If the url matches the rule, the request will be dropped. Only one match determines the rule activations so, basically, all rules are "OR" conditions by default. Each rule can test a set of request parameters (url, agent, headers). Inside the single rule, all conditions must be true to activate the rule. This means we are working with something like this (CONDITION A AND CONDITION B) OR (CONDITION C) and this will support most cases.

Caching

By using standard .NET Core caching module, we can provide cache support for url, defining policy, etc. Caching has many options that are basically a wrap of the original module, so you can refer here for more details.

Rewrite

This stage will allow static rewrite rule. This is often demanded to the applications but can be implemented here to simplify server part or to map virtual urls over many different applications. This is mostly a way to couple external url with internal one in case there isn't a way to change balanced application. Balancer itself will balance the output of this transformation.

Balancing

This is the core module that defines, for each url what will be the destination. This step generates only the real path, replacing selected host. The host can be selected using one of the following algorithms:

1. Number of requests coming
2. Number of requests pending
3. Quicker response
4. Affiliation (based on Cookie)

Proxy

After Balancing stage completes the computation of right url, proxy module will invoke the request replying to the client.

.NET Core In Action

In this section, I’ll show the most important ASP.NET Core feature that I have used in this application to get the result.

The Host

.NET Core provides two built in servers that give you the capability to run a web application (Kestrel, Http.sys). The good part is that any application can run and act as a web server, and this is very interesting to run local Angular application, maybe based on electron framework. The bad part is that in most scenarios, this will have to run behind a proxy server due to their limitation. The first limitation I have in mind is that Kestrel doesn’t support host binding, but only listen on ports. So, if you want to have two different web sites in the same port, it is a problem. For the balancer is not a problem, because the main feature is to get the whole traffic and then route it to the destination, but in the real world, web server will have to provide multiple web sites on the same port, so you probably will need to use IIS or any other solution again. Another pain is about HTTPS: the configuration on Kestrel is not so easy and dynamic. So also in this case, staying behind a web proxy is preferable.

  1  public static IWebHost BuildWebHost(string[] args) 
  2  {
  3      WebHost.CreateDefaultBuilder(args)
  4          .UseStartup<Startup>()
  5          .UseKestrel(options =>
  6          {
  7              // some settings
  8              options.Limits.MaxConcurrentConnections = 100;
  9              options.Limits.MaxConcurrentUpgradedConnections = 100;
 10              options.Limits.MaxRequestBodySize = 10 * 1024;
 11              options.Limits.MinRequestBodyDataRate = new MinDataRate
 12                      (bytesPerSecond: 100, gracePeriod: TimeSpan.FromSeconds(10));
 13              options.Limits.MinResponseDataRate = new MinDataRate
 14                      (bytesPerSecond: 100, gracePeriod: TimeSpan.FromSeconds(10));
 15              //listening on http
 16              options.Listen(IPAddress.Loopback, 5000);
 17              // listening on 5001, but using https
 18              options.Listen(IPAddress.Loopback, 5001, listenOptions =>
 19              {
 20                  listenOptions.UseHttps("testCert.pfx", "testPassword");
 21              });
 22          })
 23          .Build();        
 24  }

In my opinion, to manage such settings as hard coded values are not the best option because there are mainly configuration issues. Anyway to mix values from configuration and some hardcoded can lead to some situation that is hard to understand and I suggest to manage as much as possible all settings in a single place.

Middlewares and Plugin System

Middleware are a very nice system and it is easy to implement your own. This is a sample:

  1  public class MyMiddleware
  2      {
  3          //store here the delegate to execute next step
  4          private readonly RequestDelegate _next;
  5  
  6          //get the next step on ctor
  7          public RequestCultureMiddleware(RequestDelegate next)
  8          {
  9              _next = next;
 10          }
 11  
 12          //do something and then invoke next step
 13          public Task Invoke(HttpContext context)
 14          {
 15              //do things here
 16  
 17              // Call the next delegate/middleware in the pipeline
 18              return this._next(context);
 19          }
 20      }

The part with “next” call is used to invoke next steps on the pipeline.

A good practice is to create an extension method to allow the registration on Startup simply invoking it:

  1  public static class MyMiddlewareExtensions
  2      {
  3          public static IApplicationBuilder UseMyMiddleware
  4             (this IApplicationBuilder builder, MyParam optionalParams)
  5          {
  6              return builder.UseMiddleware<MyMiddleware>();
  7          }
  8      }

There aren’t any limitations or rules to implement it: you just have to write the code inside a method. The way I don’t like is that there is a lot of freedom and there are lot of things left to convention and to the implementor. Of course, in normal usage, we need only to introduce middleware yet done and interact with their configuration. (Think about MVC one, you just have to include, then write files to let it work.) In this application, because middleware is the main part and we introduce lot of them, I preferred to give a scaffold to let these parts to develop new plugin without knowing how all other modules works. This is done by implementing an abstract class that gives to the implementor a way to define:

• if the plugin is active or not for the current request
• if the request has to be terminated or can flow to next steps
• write the code that does things (i.e., in balancer middleware, define which server is used as destination)
• write the configuration

The implementation of the module is abstract so user will have to implement. The other method has a default implementation and can be omitted (standard behaviour is: active basing on settings, never stops flow, register itself on startup).

Here is the abstract class definition. Default implementation are omitted to keep things readable, but you can inspect the full source code.

  1   public abstract class FilterMiddleware:IFilter
  2   {
  3       public virtual bool IsActive(HttpContext context)
  4       {
  5          //compute here the logic based on httpcontext 
  6          //to tell if this stage is active or not
  7       }
  8       
  9       public override async Task Invoke(HttpContext context)
 10       {
 11          var endRequest = false;
 12          if (this.IsActive(context))
 13          {
 14              object urlToProxy = null;
 15              // compute args here...
 16              await InvokeImpl(context /* provide computed args here*/);
 17              endRequest = this.Terminate(context);
 18          }
 19  
 20          if (!endRequest && NextStep != null)
 21          {
 22              await NextStep(context);
 23          }
 24      }
 25      
 26      public virtual bool Terminate(HttpContext httpContext)
 27      {
 28          //compute logic to tell to invoke method if request can be terminated
 29          return false;
 30      }
 31  
 32      //create an instance of filter and register it
 33      public virtual IApplicationBuilder Register
 34                     (IApplicationBuilder app, IConfiguration con, 
 35                     IHostingEnvironment env, ILoggerFactory loggerFactory)
 36      {
 37          return app.Use(next => 
 38          {
 39              var instance = (IFilter)Activator.CreateInstance(this.GetType());
 40              return instance.Init(next).Invoke;
 41          });
 42      }
 43       // Implementation of filter (must be implemented into child class)
 44       public abstract Task InvokeImpl(HttpContext context,string host, 
 45                            VHostOptions vhost,IConfigurationSection settings);
 46  }

The list of active plugins are written into config so that to add a new one, without changing the main application, you just need to create your DLL with the module, include it in bin folder with all dependencies and add an entry to config files.

This is the snippet to register all middlewares, the configuration is the topic of the next paragraph, so I show only the registration part here.

  1  //BalancerSettings.Current.Middlewares contains all middleware read from config
  2  foreach (var item in BalancerSettings.Current.Middlewares)
  3  {
  4    item.Value.Register(app, Configuration, env, loggerFactory);
  5  }

Configuration

The main topic about configuration is that it has to be dynamic and each middleware has to be able to read its part easily. I wanted also to use as much as possible the out-of-the-box way. Fortunately, ASP.NET Core configuration supports natively

• json deserialization binding section to objects
• getting single value by path (navigating the json tree)
• merging multiple settings file
• dynamically apply one config based on environment

Loading Main Settings

Main settings are stored in a conf file and is binded with a singleton element shared across all application parts.

Here is the json code:

  1  {
  2    "BalancerSettings": {
  3      "Mappings": [
  4        {
  5          "Host": "localhost:52231",
  6          "SettingsName": "site1"
  7        }
  8      ],
  9      "Plugins": [
 10        {
 11          "Name": "Log",
 12          "Impl": "NetLoadBalancer.Code.Middleware.LogMiddleware"
 13        },
 14        {
 15          "Name": "Init",
 16          "Impl": "NetLoadBalancer.Code.Middleware.InitMiddleware"
 17        },
 18        {
 19          "Name": "RequestFilter",
 20          "Impl": "NetLoadBalancer.Code.Middleware.RequestFilterMiddleware"
 21        },
 22        {
 23          "Name": "Balancer",
 24          "Impl": "NetLoadBalancer.Code.Middleware.BalancerMiddleware"
 25        },
 26        {
 27          "Name": "Proxy",
 28          "Impl": "NetLoadBalancer.Code.Middleware.ProxyMiddleware"
 29        }
 30      ]    
 31    }

Here is the code to bind it to the class, using dependency injection to make it available on all constructors.

  1  public void ConfigureServices(IServiceCollection services)
  2  {
  3    services.AddOptions();
  4    services.AddMemoryCache();
  5    services.Configure<BalancerSettings>(Configuration.GetSection("Balancersettings"));
  6  }
  7  
  8  public void Configure(IApplicationBuilder app,
  9                        IHostingEnvironment env, ILoggerFactory  loggerFactory,
 10                        IOptions<BalancerSettings> init)
 11     //here you can handle the injected value
 12  }

Apply Dynamic Config Based on Requests

This feature covered most of the issues, but I still need a way to apply different configuration based on request data. Yes, because all settings are static and we cannot run multiple instances of application with different settings. A solution would be to replicate the logic to get contextualized data into each middleware, but this way didn’t like because it will ask us to replicate lot of logic in many classes. Basically, if I am serving site1.com, I have to take different settings than siste2.com. Such rules usually are managed as application data, like storing in a database. But in this case, I wanted to use only configuration to introduce as few components as possible.

The solution I found uses all standard features and needs only config files. First of all, I have a map that defines for all host name the configuration file name. This allows to share same config across multiple domains, i.e., telling that www.site1.com and site1.com must route to the same cluster.

  1  "Mappings": [
  2       {
  3         "Host": "<a href="http://www.site1.com/">www.site1.com</a>",
  4         "SettingsName": "mycluster"
  5       },
  6       {
  7         "Host": "site1.com",
  8         "SettingsName": "mycluster"
  9       }
 10     ]

All configurations are named and linked by reference from the previous schema.

During the request processing, I get the host from request value so I can read the configuration section related to it. These are the few methods that read the section from host, resolving by configuration name.

  1  //get settings name from host (www.site.com=>mybalancer)
  2  public string GetSettingsName(string host)
  3  {
  4      //in memory map that reflects .json settings
  5      return hostToSettingsMap[host];
  6  }
  7  
  8  //get section from hostname (www.site.com=> read mybalancer section)
  9  public IConfigurationSection GetSettingsSection(string host)
 10  {
 11      string settingsName = GetSettingsName(host);
 12      return Startup.Configuration.GetSection(settingsName);
 13  }
 14  
 15  // bing settings to the class of a given type T
 16  public T GetSettings<T>(string host) where T : new()
 17  {
 18      var t = new T();
 19      GetSettingsSection(host).Bind(t);
 20      return t;
 21  }

So, all middleware has access to the configuration related to the current host and can find inside it their own section. See the balancer that read its’ information as example:

  1  //Balancer implementation
  2  public async override Task InvokeImpl
  3    (HttpContext context, string host, VHostOptions vhost, IConfigurationSection settings)
  4  {
  5    BalancerOptions options= new BalancerOptions();
  6    settings.Bind("Settings:Balancer", options);
  7    //.. continue doing real work..
  8  }

This is the part when I put all config files together:

  1  public Startup(IConfiguration configuration,IHostingEnvironment env)
  2  {
  3      Configuration = configuration;
  4  
  5      var builder = new ConfigurationBuilder()
  6         .SetBasePath(env.ContentRootPath)
  7         .AddJsonFile("conf/appsettings.json", optional: true, reloadOnChange: true);
  8  
  9      //get all files in ./conf/ folder
 10      string[] files = Directory.GetFiles
 11                       (Path.Combine(env.ContentRootPath, "conf", "vhosts"));
 12  
 13      foreach (var s in files)
 14      {
 15          builder = builder.AddJsonFile(s);
 16      }
 17  
 18      builder=builder.AddEnvironmentVariables();
 19      Configuration = builder.Build();
 20  }

Logging

Logging isn’t a new feature into programming and in .NET Framework, there is some tool to do it out of the box. The good news is that, today, we have a very complete logging framework that works in the way we like (NLog, Log4net). Logs can be routed to the default provider or to third parts framework like NLog, that I used into this project. The logger is provided from DI into constructor and as many things in .NET Core the best practices are to store into a local variable, something like this:

  1  public class MyController : Controller
  2  {
  3      private readonly ILogger _logger;
  4  
  5      public TodoController(ILogger<MyController> logger)
  6      {     
  7          _logger = logger;
  8      }
  9  }

To use an external provider is easy, here is my config that send logs to Nlog.

  1  loggerFactory.AddNLog();
  2  app.AddNLogWeb();
  3  env.ConfigureNLog(".\\conf\\nlog.config"); //I decided to place here...

Point of Interest

Is .NET Core Ready for Production?

Lot of people told me ASP.NET Core is not ready for the market because it is a lot younger than regular framework. Of, course, .NET Framework is a very mature framework, improved on each release and gives us a lot of certainty. It's also true that, compared to .NET Core, it is a lot more mature. By the way, this doesn’t means .NET Core is not enough to be used in production. If you remember in late 2001, when .NET Framework came out, it was not so mature too, but in 2005, just after couple of years from being born, .NET 2.0 was very reliable and has been chosen as the best solution in a large amount of project (I remember also version 1.1 that was working after a couple of hotfix and minor releases…). Also for .NET Core, the first year has gone and I found in it must of the features I need. There are a lot of third party libraries that are now available on .NET Core too and others are going to be ported. So, if you are looking for a technology to develop a long term project, it is an option to take in account. Even more, if you are going to implement a multi platform one, it is a very good solution to bring .NET power (C# and Visual Studio) on every workstation or server.

When to Choose .NET Core

• No dependency from .NET assembly or third party library available only on .NET
• Need to implement a cross platform application
• Starting an application that may need one of the above points in future
• Implement a local server to create a client application based on angular\electron
• Implement a pure API \microservice application
• Deploy on Docker
• Want to experiment

When to Choose .NET Framework

• Have any .NET Framework dependency (libraries or projects)
• Have to use COM object or any platform dependent technology

Next Steps

As this is a functionally working load balancer, there are some further steps to make it ready for the market. Of course, there may be a long list of things to do but, excluding the feature development, we can summarize to:

• marking performance tuning and load test
• package it, releasing multiple bundle depending on OS and mode

History

• 5^th September, 2017: First version published