AlwaysOn Windows-Service with Remote Control

• Download WindowsServiceV2_Executables.zip - 363.9 KB
• Download WindowsServiceV2.zip - 208.3 KB

Introduction

This article shows a method how to extend a Windows Service for running in an AlwaysOn manner.

Background

When we decided to move a SQL-Server instance to an Always-On Cluster. But only having the database clustered wouldn't mean much, if the depending systems fail to work, when the primary instance crashes.

So the first issue was easy to solve, we moved all existing connections to the servers databases to SQL-Listeners, which simply switch to another instance, when one goes down.

But on the current server we also have a service, which handles various task and it would have been nice, to have this service also clustered, instead of moving it to a third server outside of the DB cluster.

So this Article will show you, how such a concept could be accomplished

It's based on and extends my former article: Windows-Service for Multithreading with Remote Control GUI

1. Using the code

The source is based on Visual Studio 2013 and .NET 4.5. For the GUI, you also need this library: http://wpfanimatedgif.codeplex.com/.

The code will extend my Multithreading-Service, this parts won't be discussed here, so please refer to the former Article as base

Nearly all changes are done in the ServiceExecution.cs of the original service implementation (see linked article)

This GUI, which is included in the downloads, shows both services in action - The master is active, the slave is listening:

2. States

Initializing	Service is starting and initializing
Listening	Service is ready, and checking the partner which currently working
Running	Service is working
Shutting_Down	Service is shutting down
Preparing_Takeover	Service prepares to hand over processing to the partner
Waiting_for_Takeover	Service is waiting to take over processing from the partner service
Ready_For_Takeover	Service will take over processing
Stopped	Service processing is stopped

3. Additional properties

partnerService	The AlwaysOn partner WCF Interface
tcpFactory	The ChannelFactory for the TCPBinding
fallbackTakeoverTime	Stores the time, when a takeover is in initialized

One service has to be defined as MASTER, which is defined in the config file of the EXE

4. The main loop

public void StartServiceExecution()
{
	try
	{
		currentState = State.Initializing;
				
		InitializeAlwaysOnCluster();
 
		while (currentState == State.Listening || currentState == State.Waiting_for_Takeover)
		{
			Thread.Sleep(1000);
			NegotiateWithPartner();
		}
 
		while (currentState == State.Running)
		{
			CheckIntegrityOfPartner();
 
			...
		}
 
		// Here all open threads are closed, this takes as long as the last thread has been broken or has finished
		while (currentState == State.Shutting_Down || currentState == State.Preparing_Takeover)
		{
			using (LockHolder<Dictionary<Guid, ThreadHolder>> lockObj =
				new LockHolder<Dictionary<Guid, ThreadHolder>>(runningThreads, 1000))
			{
				if (lockObj.LockSuccessful)
				{
					...
 
					// If no more threads are left, set the state to stopped
					if (runningThreads.Count == 0)
					{
						currentState = currentState == State.Preparing_Takeover ? State.Ready_For_Takeover : State.Stopped;
					}
				}
			}
		}
	}
	catch (Exception e)
	{
		...
	}
}

When starting the main process, we need to initialize the AlwaysOn cluster, see more at 3.

While the service is in state "Running" it checks the integrity of it's partner at each iteration

Finally, when shutting down when the partner is waiting to takeover, we need to tell it, when takeover is possible, by setting the state "Ready_For_Takeover"

5. Initializing the AlwaysOn cluster

private void InitializeAlwaysOnCluster()
{
	for (int retry = 0; retry < 5; retry++)
	{
		OpenChannelToPartner();
 
		if (CheckIntegrityOfPartner() == true)
		{
			break;
		}
	}
 
	currentState = CheckIntegrityOfPartner() == false ? State.Running : State.Listening;
}

When opening, i put in a retry, to give the partner service some time to start up. When the partner responds, the service starts listening otherwiese it starts processing.

private void OpenChannelToPartner()
{
	tcpFactory = new ChannelFactory<iservicewcf>(
		new NetTcpBinding(),
		new EndpointAddress(Properties.Settings.Default.always_on_partner));
 
	partnerService = tcpFactory.CreateChannel();
}

I use a NetTcpBinding, because the services have to communicate between servers in our network. Each service has it's on config, where the corresponding partner binding address is maintained.

6. Integrity checking

public bool CheckIntegrityOfPartner()
{
	try
	{
		if (tcpFactory.State == CommunicationState.Closed)
		{
			OpenChannelToPartner();
		}
 
		partnerService.CheckState();
	}
	catch (Exception ex)
	{
		// If the connection is aborted by the other side, it stays open but returns an execption which tells you the connection is Faulted
		if (tcpFactory.State == CommunicationState.Opened && ex.ToString().Contains("Faulted"))
		{
			tcpFactory.Abort();
		}
 
		return false;
	}
 
	return true;
}

It the connection to the partner is closed, the first thing is to check if the partner service would be available and try to establish a connection.

Then try to get the state from the partner via the WCF Interface. No need the check if the connection is alive or not, because all Exeptions are catched.

A special behavior has to be mentioned here: Even if the CommunicationState of the connection is "Opened" the connection can be dead, which you only can determin in the Exception message, where the connection is declared as "Faulted". If so, abort the ChannelFactory.

7. Negotiating with the partner service

private void NegotiateWithPartner()
{
	if (Properties.Settings.Default.always_on_is_master == true)
	{

The service is marked as master for the cluster, so take the master´s way

if (partnerService.CheckState() == State.Listening)
{
    currentState = State.Running;
}

Only for clearance, if the slave is offline, it would run anyways

else if (partnerService.CheckState() == State.Running)
{
    partnerService.PrepareForTakeover();
}

If the slave is currently running, tell it, that the master will take over processing

else if (partnerService.CheckState() == State.Waiting_for_Takeover)
{
    partnerService.AbortTakeover();
}

If the slave already tries to take over, tell it to stop, hence the master is online again

else if (partnerService.CheckState() == State.Ready_For_Takeover)
{
    partnerService.StartService();
}

When the slave finished processing, tell it to restart, so state would go to "Listening"

}
else
{
    if (CheckIntegrityOfPartner() == false)
    {
        if (currentState == State.Listening)
        {
            fallbackTakeoverTime = DateTime.Now.AddSeconds(Properties.Settings.Default.fallback_takeover_delay_in_seconds);
            currentState = State.Waiting_for_Takeover;
        }

If the master isn't available and the slave is listening, start take over process. A delay time is defined, before the slave really can take over.

else if (currentState == State.Waiting_for_Takeover)
{
    if(fallbackTakeoverTime <= DateTime.Now)
    {
        currentState = State.Running;
    }
}

When the delay is over, start processing

		}
	}
}

8. The WCF service

In the original serice, the WCF provider was only used by the GUI to communicate with the service, and I used a NetNamedPipeBinding.

Now the is used by the GUI as well as by the partner service with the TCP binding

8.1 IServiceWCF.cs

[ServiceContract]
public interface IServiceWCF
{
	[OperationContract(IsOneWay = true)]
	void StartService();
 
	[OperationContract(IsOneWay = true)]
	void StopService();
 
	[OperationContract]
	string GetActiveThreads();
 
	[OperationContract]
	State CheckState();
 
	[OperationContract]
	void PrepareForTakeover();
 
	[OperationContract]
	void AbortTakeover();
 
	[OperationContract]
	bool CheckIntegrityOfPartner();
}

The interface now exposes some additional functions for the interaction between the services

8.2 WCFProvider.cs

The WCF provicer is now defined as NetTCPBinding

    class WCFProvider
{
	readonly ServiceHost serviceProviderTCP;

	public WCFProvider()
	{
		serviceProviderTCP = new ServiceHost(
			typeof(ServiceWCF), new Uri(Properties.Settings.Default.service_provider_uri));

		serviceProviderTCP.AddServiceEndpoint(typeof(IServiceWCF),
			new NetTcpBinding(), Properties.Settings.Default.service_provider_name);

		serviceProviderTCP.Open();
	}

	public void StopProvidingService()
	{
		serviceProviderTCP.Close();
	}
}

9 The GUI

Still I won't go much into detail, but as for now the WCF provider is TCP, the GUI isn't limited to run at the same maschine, the service does, but it can be started on any maschine in the network, which is able to access the server

Future improvements

None for now

History

• 24.02.2016 - Initial publication.