AlphaLog

• Download demo - 1.6 MB

Introduction

AlphaLog is a developer tool which is capable of parsing, merging and filtering a variety of log files. The tool supports a plug-in model which allows a developer to create a plug-in which is compatible with their particular log file type.

The tool is still very much under active development and there are a number of features still missing, However, I felt now was a good time to begin documenting the tool. I will update this article as features are added to the tool.

Background

During my work, I have experienced a number of scenarios where I need to study the log files generated by multiple systems in order to understand the chain of events which have resulted in an error. These systems can often generate log files in a variety of formats which can make merging the files challenging. Also, I often find when putting a systems logging into Debug mode the volume of information generated makes analysing the log files difficult. 

Generally I'm only interested in particular log entries within a certain timeframe. Therefore, I needed a tool which would allow me merge and filter log entries generated in a variety of different formats.

Using the code

The following diagram provides a basic overview of AlphaLogs structure:

I've tried to keep the API to AlphaLog as simple as possible. 

• When AlphaLog is provided with a new log file, it queries each Columniser Descriptor to find a valid columniser. 
• Once a match is found a new instance of the Columniser is created for that particular file. 
• One or more subscriber can subscribe to updates to log files.
• When the Update method is called on AlphaLog the log files are processed by the columnisers and any log entries are pushed to the subscribers.

Therefore, the AlphaLogParser is responsible for managing the lifecycle of the columnisers and subscribers.

Columniser

Before the log files can be merged or filtered they must be parsed into a common format. This is where the concept of a columniser was introduced. A columniser is essentially a plug-in which a developer creates to process their log file type. A number of generic columnisers have already been developed for Alpha log. These are as follows:

• RegexColumniser: A columniser designed to parse text files. The columniser utilises the named groups defined by a regular expression to parse the log entries into a dictionary. 
• CSVColumniser: A columniser designed to parse CSV files. The CSV file must defined headers as the column headers are used to parse the log entries into a dictionary.
• XMLColumniser: A columniser designed to parse XML files using a set of predefined XPath rules.

Users can also define proprietary columnisers to parse bespoke log files. One such example in the AlpaLog code base is the log4net XML columns. The columniser does not require any configuration, the columniser simply parses log files in the log4net XML format. An example class diagram for the CsvColumniser follows: 

Name	Description
IColumniserDescriptor	An interface for the columniser descriptors.
ColumnizerDescriptor	An abstract base class for ColumniserDescriptors.
CsvColumniserDescriptor	An implementation of a descriptor for CSV files.
IColumniser	An interface for columnisers
Columniser	An abstract base class for Columnisers.
CsvColumniser	A columniser for CSV files.

 

 

 

 

UI Plugins

When defining a new columniser, users can optionally defined a view which allows the columniser to be configured. 

This allows a developer to create a plugin for the UI which appears in the settings window. These columnisers can be seen under the columnisers node in the following figure:

AlphaLogParser

The AplhaLogParser is responsible for selecting a columniser for the provided file and performing type checking on the observed files. 

When a file is added the parser attempts to select a matching Columniser using the Columniser Descriptors. The parser then ensures that the selected columnisers datatypes do not conflict with the existing columnisers.

public Details Add(string file)
{
    if (this.disposed)
    {
        throw new ObjectDisposedException(typeof(AlphaLogParser).Name);
    }

    IColumnizer tempColumniser = this.GetColumniser(file);
    tempColumniser.Subscribe(this.alphaLogSubscribers.AsObserver());

    var columniserDetails = new Details(tempColumniser.File, tempColumniser.Name);
    if (this.UpdateMappingTable(tempColumniser))
    {
        columniserDetails = new ErrorDetails(file, "The data types conflict with another file.");
        if (this.progress != null)
        {
            this.progress.Report(ProcessState.Error, 0);
        }
    }

    return columniserDetails;
}

The Remove method, removes the columniser which maps to the specified files and rebuilds the mapping table used for type checking.

public void Remove(string file)
{
    if (this.disposed)
    {
        throw new ObjectDisposedException(typeof(AlphaLogParser).Name);
    }

    if (this.activeColumnisers.ContainsKey(file))
    {
        this.activeColumnisers[file].Dispose();
        this.activeColumnisers.Remove(file);
    }

    if (this.RebuildMappingTable())
    {
        if (this.progress != null)
        {
            this.progress.Report(ProcessState.Error, 0);
        }
    }
    else
    {
        if (this.progress != null)
        {
            this.progress.Report(ProcessState.None, 0);
        }
    }
}

Points of Interest

There were a number of technical challenges which had to be overcome whilst developing AlphaLog.

Matchers 

In the future, I plan to introduce a plug-in model for the filtering stage of AlphaLog. However for the first implementation I've built a simple interpreter for the filtering stage.

The Matchers library is an interesting part of AlphaLog as it is essentially a simple rules engine which could find applications in a number of different projects. The Matchers allow rules to be defined in the form of an Abstract Syntax Tree which can be serialised to XML. A WPF user interface is provided which allows the rules to be defined by users. A class diagram for the rules engine follows:

Name	Description
IMatch	An interface for Matchers.
MatchBase	An abstract base class for Matchers.
MatchFactory	A factory for the Matchers.
MatcherType	An enum for the available Matchers.
NumericBase	An abstract base class for Numeric based Matchers.
TextBase	An abstract base class for string/text based Matchers.
GroupBase	An abstract base class for Mathers that aggregate/group other Matchers.

 

 

 

 

 

 

A simple rule defined in C# code looks like this.

new And(
    new ContainsMatcher("Hello", "Message", false),
    new ExactMatcher("Hello", "Message", false));

Obviously defining the rules in C# code isn't particularly useful since you could just write the code. The real benefit comes from the ability to serialize and deserialize the rules from XML. 

<And>
   <Contains searchField="Message" searchValue="Hello" caseSensitive="false" />
   <Exact searchField="Message" searchValue="World" caseSensitive="false"/>
</And

Dynamic Columns

One of the more challenging problems I have had to solve whilst developing AlphaLog was defining the data grid columns dynamically. Since AlphaLog can handle multiple log file types it is not possible to define a fixed set of columns for the data grid. Therefore, I needed to be able to generate the column items at runtime.

I found the following articles pointed me in the right direction.

• http://www.reimers.dk/jacob-reimers-blog/auto-generating-datagrid-columns-from-dynamicobjects
• http://www.codeproject.com/Articles/575856/Windows-Forms-Binding-through-ITypedList-interface

Essentially the WPF Datagrid uses reflection to determine what columns need to be generated when using the "auto generate columns" feature. Since we can't create a strongly typed class, we must override information returned when reflecting on the log entry object.

Implementing ITypedList Interface allows you to provide different views of a collection. Since the individual log entries can have a varying number of columns, the DynamicItemCollection class merges the column information.

public class DynamicItemCollection<T> : ObservableCollection<T>, ITypedList
    where T : PropertyDescriptorCollection, ICustomTypeDescriptor
{
    /// <summary>
    /// Initializes a new instance of the <see cref="DynamicItemCollection{T}"/> class.
    /// </summary>
    /// <param name="toList">
    /// The to list.
    /// </param>
    public DynamicItemCollection(List<T> toList)
        : base(toList)
    {
    }

    /// <summary>
    /// The get item properties.
    /// </summary>
    /// <param name="listAccessors">
    /// The list accessors.
    /// </param>
    /// <returns>
    /// The <see cref="PropertyDescriptorCollection"/>.
    /// </returns>
    public PropertyDescriptorCollection GetItemProperties(PropertyDescriptor[] listAccessors)
    {
        var set = new HashSet<PropertyDescriptor>();
        foreach (T item in this.Items)
        {
            foreach (object i in item)
            {
                set.Add((PropertyDescriptor)i);
            }
        }

        return new PropertyDescriptorCollection(set.ToArray());
    }

    /// <summary>
    /// The get list name.
    /// </summary>
    /// <param name="listAccessors">
    /// The list accessors.
    /// </param>
    /// <returns>
    /// The <see cref="string"/>.
    /// </returns>
    public string GetListName(PropertyDescriptor[] listAccessors)
    {
        return null;
    }
}

PropertyDescriptorCollection

The LogDescriptorCollection provides a wrapper for an individual log entry.

public class LogDescriptorCollection : PropertyDescriptorCollection, ICustomTypeDescriptor
{
    /// <summary>
    ///     The log items.
    /// </summary>
    private readonly Dictionary<string, object> logItems = new Dictionary<string, object>();

    /// <summary>
    /// Initializes a new instance of the <see cref="LogDescriptorCollection"/> class.
    /// </summary>
    /// <param name="dictionary">
    /// The dictionary.
    /// </param>
    public LogDescriptorCollection(Dictionary<string, object> dictionary)
        : base(
            dictionary.Where(k => k.Value != null)
                .Select(k => new LogItemDescriptor(k.Key, k.Value.GetType()))
                .ToArray())
    {
        this.logItems = dictionary;
    }

    ...

    /// <summary>
    ///     Returns the properties for this instance of a component.
    /// </summary>
    /// <returns>
    ///     A <see cref="T:System.ComponentModel.PropertyDescriptorCollection" /> that represents the properties for this
    ///     component instance.
    /// </returns>
    public PropertyDescriptorCollection GetProperties()
    {
        var col = new PropertyDescriptorCollection(null);

        foreach (var item in this.logItems.Where(k => k.Value != null))
        {
            col.Add(new LogItemDescriptor(item.Key, item.Value.GetType()));
        }

        return col;
    }

    ... 
}

PropertyDescriptor

The PropertyDescriptor class exposes a dictionary entry (log item) for reflection purposes.

public class LogItemDescriptor : PropertyDescriptor
{
    /// <summary>
    ///     The m_prop type.
    /// </summary>
    private readonly Type type;

    /// <summary>
    /// Initializes a new instance of the <see cref="LogItemDescriptor"/> class.
    /// </summary>
    /// <param name="name">
    /// The name.
    /// </param>
    /// <param name="type">
    /// The type.
    /// </param>
    public LogItemDescriptor(string name, Type type)
        : base(name, null)
    {
        this.type = type;
    }

    ...

    /// <summary>
    /// When overridden in a derived class, gets the current value of the property on a component.
    /// </summary>
    /// <returns>
    /// The value of a property for a given component.
    /// </returns>
    /// <param name="component">
    /// The component with the property for which to retrieve the value.
    /// </param>
    public override object GetValue(object component)
    {
        return ((LogDescriptorCollection)component).Items[this.Name];
    }

    /// <summary>
    /// When overridden in a derived class, sets the value of the component to a different value.
    /// </summary>
    /// <param name="component">
    /// The component with the property value that is to be set.
    /// </param>
    /// <param name="value">
    /// The new value.
    /// </param>
    public override void SetValue(object component, object value)
    {
        ((LogDescriptorCollection)component).Items[this.Name] = value;
    }

    ...
}

Application Settings

Due to the nature of AlphaLog there are a lot of application settings which need to be stored. I designed a simple repository which stores the application settings in XML format. This can be seen in the following class diagram:

Name	Description
ISettingsManager	An interface for the main setting repository.
SettingsManager	The concrete implementation of the settings repository.
IRepository	An interface for objects which can be reconstituted from XML.
Repository	An abstract base class for repositories.
IPersist	An interface for objects which can be serialised to XML
MatcherDatabase	A repository which stores MatcherItem objects.
MatcherItems	An item which can be stored in the MatcherDatabase repository.

 

 

 

 

 

 

The SettingsManager class is based on the Service Locator pattern. The SettingsManager is a Repository of Repositories. The GetSettings method allows a Repository specified by its type to be obtained. The code for the GetSettings method is as follows:

public TSettings GetsSetting<TSettings>() where TSettings : IRepository, new()
{
    IRepository settings = null;
    if (!this.settingsCollection.TryGetValue(typeof(TSettings), out settings))
    {
        var newSettings = new TSettings();
        XElement settingsElement = this.GetElement(newSettings.SettingsName);
        if (settingsElement != null)
        {
            newSettings.Initialize(settingsElement);
        }

        this.settingsCollection[typeof(TSettings)] = newSettings;
        return newSettings;
    }

    return (TSettings)settings;
}

Given a settings manager instance the MatcherDatabase repository can then be obtained as follows:

[ImportingConstructor]
public ManageMatcherVM(ISettingsManager settingsManager)
    : base("Matchers")
{
    this.settings = settingsManager.GetsSetting<FieldSettings>();
    this.matchers = settingsManager.GetsSetting<MatcherDatabase>();
    
    ...
}

Source Code

If you would like to view the libraries source code and demo applications the code can be found on my Bit Bucket site. 

https://bitbucket.org/chrism233/alphalog

The git repository address is as follows:

https://chrism233@bitbucket.org/chrism233/alphalog.git

History

Date	Changes
07/01/2015	Initial release.
06/10/2015	Uploaded demo zip file containing executable.