In this post, I will show you how to create a very simple Windows Service (I will call it Musketeer) that will collect information about other processes running on a server. Such a tool might be helpful if you host Windows services on some remote server and you want to store information about their performance in a database for further analysis. This tool might be also a cure if your admins didn’t give you enough privileges to connect the Performance Monitor to the remote server. :)

Database Model

Let’s start from a database model that the Musketeer will use. I’m using MySql syntax but it shouldn’t be a problem to adapt this code to any other database server:

create table services(
  Name varchar(100) primary key,
  DisplayName varchar(1000) null
) engine = InnoDB;

create table service_counters(
  Id int auto_increment primary key,
  ServiceName varchar(100) not null references services(Name),
  MachineName varchar(100) null,
  CategoryName varchar(100) not null,
  CounterName varchar(100) not null,
  InstanceName varchar(100) null,
  DisplayName varchar(1000) null,
  DisplayType enum('table', 'graph') null
) engine = InnoDB;

create table service_counter_snapshots(
  Id int auto_increment,
  ServiceCounterId int not null,
  SnapshotMachineName varchar(100) null,
  CreationTimeUtc datetime not null,
  ServiceCounterValue float null,
  primary key (Id, CreationTimeUtc)
) engine = InnoDB
partition by range columns(CreationTimeUtc)
(partition p20121018 values less than ('2012-10-19 00:00'),
 partition p20121019 values less than ('2012-10-20 00:00'));

The services table stores a list of all services (processes) we would like to monitor. Each service has a list of system performance counters assigned to it (stored in the service_counters table). At startup, the Musketeer service will create an instance of the System.Diagnostics.PerformanceCounter class for each row from the service_counters table. Then repeatedly at a specific interval, it will collect values from the created performance counters and store the collected data in the service_counter_snapshots table. As you can see in the script, the service_counter_snapshots table is partitioned by CreationTimeUtc which makes the logged data easily manageable: for instance, if we want to keep logs for only two past days, we can create a daily scheduler task that will drop all the older partitions. There is one caveat though: if we forget about creating a partition for the current day, all inserts from the Musketeer service will be rejected by the database. So just remember to add another line to your daily scheduler that will create a log partition for the next day.

Monitoring Service

To implement a monitoring service host, we will use the Topshelf library. The code of the service looks as follows:

class MusketeerWorker : ServiceControl
{
    private readonly LogWriter logger = HostLogger.Get<MusketeerWorker>();
    public static bool ShouldStop { get; private set; }
    private ManualResetEvent stopHandle;

    public bool Start(HostControl hostControl)
    {
        logger.Info("Starting Musketeer...");

        stopHandle = new ManualResetEvent(false);

        ThreadPool.QueueUserWorkItem(new ServiceMonitor().Monitor, stopHandle);

        return true;
    }

    public bool Stop(HostControl hostControl)
    {
        ShouldStop = true;
        logger.Info("Stopping Musketeer...");
        // wait for all threads to finish
        stopHandle.WaitOne(ServiceMonitor.SleepIntervalInMilliSecs + 10);

        return true;
    }
}

class Program
{
    static void Main()
    {
        HostFactory.Run(hc =>
        {
            hc.UseNLog();
            // service is constructed using its default constructor
            hc.Service<MusketeerWorker>();
            // sets service properties
            hc.SetServiceName(typeof(MusketeerWorker).Namespace);
            hc.SetDisplayName(typeof(MusketeerWorker).Namespace);
            hc.SetDescription("Musketeer - one to monitor them all.");
        });
    }
}

As you can read from the code, at the service startup, we create our monitoring thread that will execute the ServiceMonitor.Monitor method. Now it’s time to implement the ServiceMonitor class:

sealed class ServiceMonitor
{
    public const int SleepIntervalInMilliSecs = 120000;

    private readonly LogWriter logger = HostLogger.Get<ServiceMonitor>();
    private IList<Tuple<int, PerformanceCounter>> serviceCounters;

    public void Monitor(object state)
    {
        ManualResetEvent stopHandle = (ManualResetEvent)state;
        String machineName = Environment.MachineName;
        try
        {
            Initialize();
            var snapshots = new ServiceCounterSnapshot[serviceCounters.Count];

            while (!MusketeerWorker.ShouldStop)
            {
                Thread.Sleep(SleepIntervalInMilliSecs);

                // this would be our timestamp value by which we will group the snapshots
                DateTime timeStamp = DateTime.UtcNow;
                // collect snapshots
                for (int i = 0; i < serviceCounters.Count; i++)
                {
                    var snapshot = new ServiceCounterSnapshot();
                    snapshot.CreationTimeUtc = timeStamp;
                    snapshot.SnapshotMachineName = machineName;
                    snapshot.ServiceCounterId = serviceCounters[i].Item1;
                    try
                    {
                        snapshot.ServiceCounterValue = serviceCounters[i].Item2.NextValue();
                        logger.DebugFormat("Performance counter {0} 
                        read value: {1}", GetPerfCounterPath(serviceCounters[i].Item2),
                                            snapshot.ServiceCounterValue);
                    }
                    catch (InvalidOperationException)
                    {
                        snapshot.ServiceCounterValue = null;
                        logger.DebugFormat("Performance counter {0} 
                        didn't send any value.", GetPerfCounterPath(serviceCounters[i].Item2));
                    }
                    snapshots[i] = snapshot;
                }
                SaveServiceSnapshots(snapshots);
            }
        }
        finally
        {
            stopHandle.Set();
        }
    }

    private void Initialize()
    {
        var counters = new List<Tuple<int, PerformanceCounter>>();
        using (var conn = new MySqlConnection
        (ConfigurationManager.ConnectionStrings["MySqlDiagnosticsDb"].ConnectionString))
        {
            conn.Open();
            foreach (var counter in conn.Query<ServiceCounter>
            ("select Id,ServiceName,CategoryName,CounterName,InstanceName from service_counters"))
            {
                logger.InfoFormat(@"Creating performance counter: 
                {0}\{1}\{2}\{3}", counter.MachineName ?? ".", counter.CategoryName, 
                                    counter.CounterName, counter.InstanceName);
                var perfCounter = new PerformanceCounter
                (counter.CategoryName, counter.CounterName, 
                counter.InstanceName, counter.MachineName ?? ".");
                counters.Add(new Tuple<int, PerformanceCounter>(counter.Id, perfCounter));
                // first value doesn't matter so we should call the counter at least once
                try { perfCounter.NextValue(); } catch { }
            }
        }
        serviceCounters = counters;
    }

    private void SaveServiceSnapshots(IEnumerable<ServiceCounterSnapshot> snapshots)
    {
        using (var conn = new MySqlConnection
        (ConfigurationManager.ConnectionStrings["MySqlDiagnosticsDb"].ConnectionString))
        {
            conn.Open();
            foreach (var snapshot in snapshots)
            {
                // insert new snapshot to the database
                conn.Execute(
                @"insert into service_counter_snapshots
                (ServiceCounterId,SnapshotMachineName,CreationTimeUtc,ServiceCounterValue) values (
                @ServiceCounterId,@SnapshotMachineName,@CreationTimeUtc,@ServiceCounterValue)", 
                snapshot);
            }

        }
    }

    private String GetPerfCounterPath(PerformanceCounter cnt)
    {
        return String.Format(@"{0}\{1}\{2}\{3}", 
        cnt.MachineName, cnt.CategoryName, cnt.CounterName, cnt.InstanceName);
    }

In the main loop of the Monitor method, we iterate through all the initialized performance counters and ask them about their current values. When the process for which the counter was created is not running, the counter throws the InvalidOperationException and we log null as the counter value.

Example of Usage

As an example, we will run the Musketeer service to monitor an instance of a Notepad process and a mspaint process in our local system. First, insert the following values to the database:

insert into services values ('notepad', 'notepad process test');
insert into services values ('mspaint', 'mspaint process test');

insert into service_counters values (0, 'notepad', null, 'process', 
                                     '% Processor Time', 'notepad', null, 'graph');
insert into service_counters values (0, 'notepad', null, 'process', 'working set', 
                                     'notepad', null, 'graph');
insert into service_counters values (0, 'mspaint', null, 'process', '% Processor Time', 
                                     'mspaint', null, 'graph');
insert into service_counters values (0, 'mspaint', null, 'process', 'working set', 
                                     'mspaint', null, 'graph');

Now, start the Musketeer service (thanks to the Topshelf library, you may run it also from the command line) and query the service_counter_snapshots table. The data should appear shortly. On my machine after 5 minutes, I got:

mysql> select * from service_counter_snapshots;
+----+------------------+---------------------+---------------------+---------------------+
| Id | ServiceCounterId | SnapshotMachineName | CreationTimeUtc     | ServiceCounterValue |
+----+------------------+---------------------+---------------------+---------------------+
| 17 |                1 | LAPTOP              | 2012-10-20 19:50:00 |             2.74918 |
| 18 |                2 | LAPTOP              | 2012-10-20 19:50:00 |             5103620 |
| 19 |                5 | LAPTOP              | 2012-10-20 19:50:00 |             22.9386 |
| 20 |                6 | LAPTOP              | 2012-10-20 19:50:00 |            18325500 |
| 21 |                1 | LAPTOP              | 2012-10-20 19:50:10 |             3.06609 |
| 22 |                2 | LAPTOP              | 2012-10-20 19:50:10 |             5206020 |
| 23 |                5 | LAPTOP              | 2012-10-20 19:50:10 |                NULL |
| 24 |                6 | LAPTOP              | 2012-10-20 19:50:10 |                NULL |
+----+------------------+---------------------+---------------------+---------------------+
8 rows in set (0.00 sec)

Now it’s up to you what you can do with data collected from your services. You can process it online showing graphs, sending alerts, etc. or prepare performance statistics for your services (like peek hours, etc.). By combining those statistics with your services logs, you may find bugs in your services and fix them before they cause any bigger problems.

We are using Musketeer at work to monitor how fast messages from our queues are processed (\MSMQ Queue()\Messages in Queue) by our services and how much memory (Process()\Working Set) or CPU (Process()\% Processor Time) those services are using. We have a dashboard (on a web page) that renders the performance data as graphs (or tables) so that we can easily spot a moment of service malfunctioning.

Conclusion

As you can see, thanks to the System.Diagnostics classes with just a few lines of code, we can create a monitoring tool that might provide us with a good insight into the server. I added the Musketeer service to my .NET Diagnostics Toolkit so feel invited to download it and give it a try on your systems.

Filed under: CodeProject, Profiling .NET applications

Introducing Musketeer – The Performance Counter Data Collector