.NET Configuration File Corrupts On Write

Introduction

We have a Windows service application which is driven from configuration information read from an application configuration file.  The purpose of the service is to run various steps at pre-scheduled times to do such things as generate static/cached data tables.  These steps are run sequentially within step groups and step groups are run in parallel.  The service is designed to be customizable via the configuration file both in execution parameters and the definitions of step groups and the steps within each group.  To help manage the configuration file a Windows GUI application was developed.  When adding new step groups to an existing configuration file and re-saving, the resultant configuration file becomes corrupt which prevents it from being read in the future viz. what the .NET mechanisms write out the .NET mechanisms can not read back in.

Background

.NET configuration files are XML files and follow a defined schema which allows for customization.  There are many standard configuration constructs which are designed to integrate the various functionalities supported by .NET, such as application settings, runtime serialization, service models, etc.  Customization is achieved by declaring the custom sections (System.Configuration.ConfigurationSection) and section groups (System.Configuration.ConfigurationSectionGroup) in the beginning of the configuration file as follows:

<configuration>
  <configSections>
            <!-- Custom Declarations Go Here... -->
  </configSections>
</configuration>

In our case the original configuration file (before attempting to add a new group) looked something like:

<configuration>
  <configSections>
    <sectionGroup name="ServiceConfiguration" type="System.Configuration.ConfigurationSectionGroup, System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" >

      <section name="Administrators" type="My.CustomSection" />

      <sectionGroup name="ServiceStepGroups" type="My.StepGroups" >

        <sectionGroup name="MyGroup1" type="My.StepGroup" />

      </sectionGroup>
    </sectionGroup>
  </configSections>

  <ServiceConfiguration>
    <Administrators ... />

    <ServiceStepGroups>

      <MyGroup1 ... />

    </ServiceStepGroups>

  </ServiceConfiguration>
</configuration>

Here, we declare in configSections the top-level custom section group ServiceConfiguration which contains all the business type configuration information unique to our service application.  We then have an Administrators section which is irrelevant for this discussion but is left in as a visual to demonstrate how custom sections are defined.

The ServiceStepGroups is defined as a custom section group which will in turn host other section groups like MyGroup1, each of which will define the steps within that group. 

Following the configSections declarations we start with the ServiceConfiguration definition.  The content is omitted for brevity and is not relevant.  What is relevant are the requirements that 1) custom sections and section groups must be both declared and defined, 2) the structure of the definition layout must match that of the declaration, and 3)  section groups must have unique names, even if they are of the same .NET type.

The Problem

Adding a new My.StepGroup (MyGroup2) to ServiceStepGroups programmatically in code and then re-saving corrupts the configuration file by violating requirements 2 and 3 (mentioned previously) as follows:

<configuration>
  <configSections>
    <sectionGroup name="ServiceConfiguration" type="System.Configuration.ConfigurationSectionGroup, System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" >

      <section name="Administrators" type="My.CustomSection" />

      <sectionGroup name="ServiceStepGroups" type="My.StepGroups" >

        <sectionGroup name="MyGroup1" type="My.StepGroup" />

      </sectionGroup>
    </sectionGroup>
    <sectionGroup name="ServiceConfiguration">  <!-- No type attribute, duplicate name and wrong place -->

      <sectionGroup name="ServiceStepGroups">   <!-- No type attribute, duplicate name and wrong place -->

        <sectionGroup name="MyGroup2" type="My.StepGroup" />

      </sectionGroup>
    </sectionGroup>
  </configSections>

  <ServiceConfiguration>
    <Administrators ... />

    <ServiceStepGroups>

      <MyGroup1 ... />
      <MyGroup2 ... />     <!-- This is in expected place -->

    </ServiceStepGroups>

  </ServiceConfiguration>
</configuration>

.NET Deep Dive

After laborious effort I was able to understand what was happening. Generally, I would consider this behavior as a bug with the .NET code since it can't read in what it writes out; however, the code seems structured and designed to produce this behavior; therefore, the behavior doesn't appear to be a mistake, just a sore and undocumented design with unintuitive behavior. Since this is such odd, undocumented and unintuitive behavior I felt compelled to share my findings with posterity.

Here is what I found:

Public APIs:

System.Configuration.ConfigurationSectionGroup

This is the configuration construct used to group sections and other groups, essentially a container. It can be subclassed but itself derives from nothing and offers little functionality so it offers little opportunity for customization.

System.Configuration.Configuration

Similar to ConfigurationSectionGroup in that it essentially acts as a container for sections and groups and derives from nothing; however, it is sealed and cannot be subclassed. It provides contextual information, some fundamental constructs for processing connection strings and application settings and the API for saving the configuration to file. It provides some flexibility for customization via delegate actions.

So, by and large, most of the processing of configuration files is behind the scenes and internal to the .NET framework. Therefore, to study these mechanisms required looking into the .NET code base and decompiling the necessary code in the VS debugger.

System.Configuration.MgmtConfigurationRecord
System.Configuration.BaseConfigurationRecord

These classes are internal to the .NET framework and provide much of the guts of configuration management. Simply, each configuration [file] is associated with one configuration record. The configuration record provides bookkeeping of what is in the configuration file and how the configuration information is written.

SaveAs(string filename, ConfigurationSaveMode saveMode, bool forceUpdateAll)

This method is called whenever a user wants to save the configuration file. It manages the save process, essentially providing a wrapper around save functionality and providing error handling. It calls CopyConfig(...) whose job it is to copy over all original configuration file content (comments and all) and integrate the new changes. It calls the following methods:

CopyConfigDeclarationsRecursive(...)

This method copies all content from the declarations section of the original configuration file. It then calls

WriteUnwrittenConfigDeclarations(...)

which writes any new declarations.

Herein lies the rub!

NEW SECTIONS AND SECTION GROUPS ARE WRITTEN AFTER EXISTING SECTIONS AND GROUPS.

Intuitive, yes? Wrong!

Consider again the corrupt configuration file introduced previously:

<configuration>
  <configSections>
    <sectionGroup name="ServiceConfiguration" type="System.Configuration.ConfigurationSectionGroup, System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" >

      <section name="Administrators" type="My.CustomSection" />

      <sectionGroup name="ServiceStepGroups" type="My.StepGroups" >

        <sectionGroup name="MyGroup1" type="My.StepGroup" />

      </sectionGroup>
    </sectionGroup>
    <sectionGroup name="ServiceConfiguration">  <!-- No type attribute, duplicate name and wrong place -->

      <sectionGroup name="ServiceStepGroups">   <!-- No type attribute, duplicate name and wrong place -->

        <sectionGroup name="MyGroup2" type="My.StepGroup" />

      </sectionGroup>
    </sectionGroup>
  </configSections>

  <ServiceConfiguration>
    <Administrators ... />

    <ServiceStepGroups>

      <MyGroup1 ... />
      <MyGroup2 ... />     <!-- This is in expected place -->

    </ServiceStepGroups>

  </ServiceConfiguration>
</configuration>

We have the main custom section group ServiceConfiguration which holds all business-type configuration information which is unique or specific to the application.  Inside this main group we have the ServiceStepGroups group which in turn declares various groups of steps for the application to perform (MyGroup1 and MyGroup2).

However, there are two (2) ServiceConfiguration declarations, a clear violation of the configuration specification which requires uniquely named groups. In the original configuration file we only had MyGroup1 defined where it and its containing groups were declared with the type attribute, necessary for declaring the .NET group type when loading a configuration file. But, we added MyGroup2 after loading the configuration file so that we could re-save the configuration file with the new group.

Here begins the unintuitive behavior .

Because CopyConfigDeclarationsRecursive(...) is called first it writes a complete ServiceConfiguration declaration as copied from the original file. When WriteUnwrittenConfigDeclarations(...) is called it has no choice but to write a second ServiceConfiguration declaration in order to maintain the declaration hierarchy; however, it does so without type attributes, presumably since the .NET type is not required for writing but only when reading. If a declaration (such as ServiceStepGroups) was not present in the original declaration then the type is written; if a declaration was present a type is not written.

So, this behavior would seem to be a design flaw in SaveAs(...). However, in the interest of open-mindedness I will hold back eternal judgment since there may be competing objectives or other behavioral scenarios of which I am not aware which drove Microsoft's design. Nonetheless, I think we can all agree that better documentation would have been helpful in understanding the behavior and function of programmatically modifying and saving configuration files. If there is better documentation I could not find it.

The Solution

After all was discovered and understood I was able to produce a proper configuration file by implementing a post-processing step . It was a workaround: manually modify the XML of the new configuration file (I used the System.Xml libraries) to move new group declarations to their proper place within the original ServiceStepGroups declaration and remove the duplicate hierarchical constructs.

I could have done this in the beginning but I had assumed I didn't need to (based on expectations of intuitive functionality and lack of documentation) and that the problem was with me and my code utilizing the .NET configuration mechanisms. While vindicated technically it was a good (but agonizing ) lesson in the underlying mechanisms of .NET configuration processing and its limitations.

So ending on a positive note maybe this technical dive and case study can provide some documentation or reference to others experiencing a similar issue in the future. I am reminded of a quote from John Adams:

Quote:

Posterity! you will never know how much it cost the present generation to preserve your freedom! I hope you will make a good use of it. If you do not, I shall repent in Heaven that I ever took half the pains to preserve it.

Posterity, I hope you make good use of this information.

Points of Interest

The original post of this issue can be found at: https://www.codeproject.com/Messages/5839778/Debrief-Configuration-Save-Writes-Invalid-Configur