Click here to Skip to main content
15,868,016 members
Articles / Programming Languages / C#

Cinchoo ETL - YAML Reader

Rate me:
Please Sign up or sign in to vote.
5.00/5 (1 vote)
1 Jul 2020CPOL34 min read 5.8K   3  
Simple YAML reader for .NET
ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

Contents

  1. Introduction
  2. Requirement
  3. "Hello World!" Sample
    1. Quick load - Data First Approach
    2. Code First Approach
    3. Configuration First Approach
    4. Code First with declarative configuration
  4. Reading All Records
  5. Read Records Manually
  6. Customize YAML Record
  7. Customize YAML Fields
    1. DefaultValue
    2. ChoFallbackValue
    3. Type Converters
      1. Declarative Approach
      2. Configuration Approach
    4. Validations
  8. Callback Mechanism
    1. Using ChoYamlReader events
    2. Implementing IChoNotifyRecordRead interface
    3. BeginLoad
    4. EndLoad
    5. BeforeRecordLoad
    6. AfterRecordLoad
    7. RecordLoadError
    8. BeforeRecordFieldLoad
    9. AfterRecordFieldLoad
    10. RecordLoadFieldError
    11. SkipUntil
    12. DoWhile
  9. Customization
  10. AsDataReader Helper Method
  11. AsDataTable Helper Method
  12. Using Dynamic Object
    1. DefaultValue
    2. ChoFallbackValue
    3. FieldType
    4. Type Converters
    5. Validations
  13. Working with sealed POCO object
  14. Exceptions
  15. Using MetadataType Annotation
  16. Configuration Choices
    1. Manual Configuration
    2. Auto Map Configuration
    3. Attaching MetadataType class
  17. LoadText Helper Method
  18. Advanced Topics
    1. Override Converters Format Specs
    2. Currency Support
    3. Enum Support
    4. Boolean Support
    5. DateTime Support
  19. Fluent API
    1. WithYamlPath
    2. WithFields
    3. WithField
    4. ColumnCountStrict
    5. NotifyAfter
    6. Configure
    7. Setup
  20. FAQs
    1. How to deserialize string as enum?
    2. How to load only the values from Key-Value Pair YAML?
    3. How to load only the keys from Key-Value Pair YAML?
    4. Deserialize YAML into dynamic object?
    5. How to convert YAML to XML?
    6. How to load selective nodes from YAML?
    7. How to load selective subnodes from YAML?
    8. How to convert YAML to DataTable?
    9. How to convert YAML to DataReader?
    10. How to convert YAML to CSV?
    11. How to deserialize an object?
    12. How to deserialize a collection?
    13. How to deserialize a Dictionary?
    14. How to deserialize from a file?
    15. How to deserialize with custom factory?
    16. Reading an entire file - Using a simple data structure
    17. Reading an entire file - Using Multiple Lists
    18. Using generic data structures (IDictionnary)
    19. Using generic data lists (IList)
    20. Reading an entire file - Dynamic structures
  21. History

1. Introduction

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

YAML is a human-readable data serialization standard that can be used in conjunction with all programming languages and is often used to write configuration files.

This article talks about using ChoYamlReader component offered by ChoETL framework. It is a simple utility class to extract YAML data from file / source to objects.

Features

  • Uses SharpYaml parser under the hood, parses YAML file in seconds and also handles large file without any memory issues
  • Stream based parsers allow for ultimate performance, low resource usage, and nearly unlimited versatility scalable to any size data file, even tens or hundreds of gigabytes
  • Event based data manipulation and validation allows total control over the flow of data during the bulk insert process
  • Exposes IEnumarable list of objects - which is often used with LINQ query for projection, aggregation and filtration, etc.
  • Supports deferred reading
  • Supports processing files with culture specific date, currency and number formats
  • Recognizes a wide variety of date, currency, enum, boolean and number formats when reading files
  • Provides fine control of date, currency, enum, boolean, number formats when writing files
  • Detailed and robust error handling, allowing you to quickly find and fix the problems

2. Requirement

This framework library is written in C# using .NET 4.5 Framework / .NET Core 2.x.

3. "Hello World!" Sample

  • Open VS.NET 2017 or higher
  • Create a sample VS.NET (.NET Framework 4.5 / .NET Core 2.x) Console Application project
  • Install ChoETL via Package Manager Console using Nuget Command based on the .NET environment:
    • Install-Package ChoETL.Yaml
  • Use the ChoETL namespace

Let's begin by looking into a simple example of reading YAML file having three fields.

Listing 3.1 Sample YAML data file (emp.yaml)
YAML
emps:
    - id: 1
      name: Tom

    - id: 2
      name: Mark

There are a number of ways in which you can get the YAML file parsing started with minimal setup.

3.1. Quick load - Data First Approach

It is the zero config, quick way to load a YAML file in no time. No POCO object is required. The sample code below shows how to load the file.

Listing 3.1.1 Load YAML File using Iterator
C#
foreach (dynamic e in new ChoYamlReader("emp.yaml")
                .WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}
Listing 3.1.2 Load YAML File using Loop
C#
var reader = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]");
dynamic rec;
 
while ((rec = reader.Read()) != null)
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

3.2. Code First Approach

This is another zero config way to parse and load YAML file using POCO class. First, define a simple data class to match the underlying YAML file layout.

Listing 3.2.1 Simple POCO Entity Class
C#
public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In the above, the class defines two properties matching the sample Yaml file template.

Listing 3.2.2 Load YAML File
C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

3.3. Configuration First Approach

In this model, we define the YAML configuration with all the necessary parsing parameters along with yaml fields matching with the underlying YAML file.

Listing 3.3.1 Define YAML Configuration
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id"));
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

In the above, the class defines two properties matching the sample YAML file template.

Listing 3.3.2 Load YAML File without POCO Object
C#
foreach (dynamic e in new ChoYamlReader("emp.yaml", config).WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}
Listing 3.3.3 Load YAML File with POCO Object
C#
foreach (var e in new ChoYamlReader<EmployeeRec>
        ("emp.yaml", config).WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

3.4. Code First with Declarative Configuration

This is the combined approach to define POCO entity class along with YAML configuration parameters decorated declaratively. Id is required field and Name is optional value field with default value "XXXX". If Name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object
C#
public class EmployeeRec
{
    [ChoYamlRecordField]
    [Required]
    public int Id
    {
        get;
        set;
    }
    [ChoYamlRecordField]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }
 
    public override string ToString()
    {
        return "{0}. {1}".FormatString(Id, Name);
    }
}

The code above illustrates about defining POCO object to carry the values of each record line in the input file. First thing defines property for each record field with ChoYamlRecordFieldAttribute to qualify for YAML record mapping. YamlPath is an optional property. If not specified, framework automatically discovers and loads the values from yaml property. Id is decorated with RequiredAttribute, if the value is missing, it will throw an exception. Name is given default value using DefaultValueAttribute. It means that if the Name yaml field contains empty value in the file, it will be defaulted to 'XXXX' value.

It is very simple and ready to extract YAML data in no time.

Listing 3.4.2 Main Method
C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

We start by creating a new instance of ChoYamlReader object. That's all. All the heavy lifting of parsing and loading YAML data stream into the objects is done by the parser under the hood.

By default, ChoYamlReader discovers and uses default configuration parameters while loading YAML file. These can be overridable according to your needs. The following sections will give details about each configuration attribute.

4. Reading All Records

It is as easy as setting up POCO object match up with YAML file structure, you can read the whole file as enumerable pattern. It is a deferred execution mode, but take care while making any aggregate operation on them. This will load the entire file records into memory.

Listing 4.1 Read YAML File
C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

or:

Listing 4.2 Read YAML File Stream
C#
foreach (var e in new ChoYamlReader<EmployeeRec>(textReader).WithYamlPath("$.emps[*]"))
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

This model keeps your code elegant, clean, easy to read and maintain. It also leverages LINQ extension methods to perform grouping, joining, projection, aggregation, etc.

Listing 4.3 Using LINQ
C#
var list = (from o in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]")
           where o.Name != null && o.Name.StartsWith("R")
           select o).ToArray();
 
foreach (var e in list)
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

5. Read Records Manually

It is as easy as setting up POCO object match up with YAML file structure, you can read the whole file as enumerable pattern.

Listing 5.1 Read YAML File
C#
var reader = new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]");
var rec = (object)null;
 
while ((rec = reader.Read()) != null)
{
    Console.WriteLine(e.Id);
    Console.WriteLine(e.Name);
}

6. Customize YAML Record

Using ChoYamlRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO Object for Each Record
C#
[ChoYamlRecordObject]
public class EmployeeRec
{
    [ChoYamlRecordField] 
    public int Id { get; set; }
    [ChoYamlRecordField] 
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of YAML load operation on a file.

  • YamlPath - Optional. YamlPath expression used to pick the elements to load. If not specified, the member value will be discovered and loaded automatically.
  • CultureName - The culture name (e.g., en-US, en-GB) is used to read and write YAML data.
  • Encoding - The encoding of the YAML file.
  • ColumnCountStrict - This flag indicates if an exception should be thrown if reading an expected field is missing.
  • ErrorMode - This flag indicates if an exception should be thrown if reading and an expected field is failed to load. This can be overridden per property. Possible values are:
    • IgnoreAndContinue - Ignore the error, record will be skipped and continue with next
    • ReportAndContinue - Report the error to POCO entity if it is of IChoNotifyRecordRead type
    • ThrowAndStop - Throw the error and stop the execution
  • IgnoreFieldValueMode - A flag to let the reader know if a record should be skipped when reading if it's empty / null. This can be overridden per property. Possible values are:
    • Null - skipped if the record value is null
    • DBNull - N/A
    • Empty - skipped if the record value is empty
    • WhiteSpace - skipped if the record value contains only whitespaces
  • ObjectValidationMode - A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
    • Off - No object validation performed (Default)
    • MemberLevel - Validation performed at the time of each YAML property gets loaded with value
    • ObjectLevel - Validation performed after all the properties are loaded to the POCO object

7. Customize YAML Fields

For each yaml field, you can specify the mapping in POCO entity property using ChoYamlRecordFieldAttribute. Only use this attribute if you want to use custom YamlPath to map to this field.

Listing 7.1 Customizing POCO Object for YAML Fields
C#
public class EmployeeRec
{
    [ChoYamlRecordField]
    public int Id { get; set; }
    [ChoYamlRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

  • YamlPath - Optional. YamlPath expression uses a path notation for addressing parts of a YAML document. If not specified, ChoYamlReader automatically discovers and loads the value from YAML object matching field name.

7.1. DefaultValue

It is the value used and set to the property when the yaml value is empty or whitespace (controlled via IgnoreFieldValueMode).

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute.

7.2. ChoFallbackValue

It is the value used and set to the property when the YAML value failed to set. Fallback value is only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute.

7.3. Type Converters

Most of the primitive types are automatically converted and set them to the properties. If the value of the yaml field can't automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter or TypeConverter converters.

There are a couple of ways in which you can specify the converters for each field:

  • Declarative Approach
  • Configuration Approach

7.3.1. Declarative Approach

This model is applicable to POCO entity object only. If you have POCO class, you can specify the converters to each property to carry out necessary conversion on them. Samples below show the way to do it.

Listing 7.3.1.1 Specifying Type Converters
C#
public class EmployeeRec
{
    [ChoYamlRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoYamlRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}
Listing 7.3.1.2 IntConverter Implementation
C#
public class IntConverter : IValueConverter
{
    public object Convert(object value, Type targetType, 
                          object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack(object value, Type targetType, 
                              object parameter, CultureInfo culture)
    {
        return value;
    }
}

In the example above, we defined custom IntConverter class. And showed how to use it with 'Id' YAML property.

7.3.2. Configuration Approach

This model is applicable to both dynamic and POCO entity object. This gives freedom to attach the converters to each property at runtime. This takes precedence over the declarative converters on POCO classes.

Listing 7.3.2.2 Specifying TypeConverters
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();

ChoYamlRecordFieldConfiguration idConfig = new ChoYamlRecordFieldConfiguration("Id");
idConfig.AddConverter(new IntConverter());
config.YamlRecordFieldConfigurations.Add(idConfig);

config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

In the above, we construct and attach the IntConverter to 'Id' field using AddConverter helper method in ChoYamlRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoYamlRecordFieldConfiguration object.

7.4. Validations

ChoYamlReader leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 7.4.1 Using Validation Attributes in POCO Entity
C#
[ChoYamlRecordObject]
public partial class EmployeeRec
{
    [ChoYamlRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoYamlRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In the example above, used Range validation attribute for Id property. Required validation attribute to Name property. ChoYamlReader performs validation on them during load based on Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

Sometimes, you may want to override the defined declarative validation behaviors that come with POCO class, you can do with Cinchoo ETL via configuration approach. The sample below shows the way to override them.

C#
static void ValidationOverridePOCOTest()
{
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    var idConfig = new ChoYamlRecordFieldConfiguration("Id");
    idConfig.Validators = new ValidationAttribute[] { new RequiredAttribute() };
    config.YamlRecordFieldConfigurations.Add(idConfig);
    config.YamlRecordFieldConfigurations.Add
           (new ChoYamlRecordFieldConfiguration("Name"));
 
    using (var parser = new ChoYamlReader<EmployeeRec>("emp.yaml", config))
    {
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

In some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 7.4.2 Manual Validation on POCO Entity
C#
[ChoYamlRecordObject]
public partial class EmployeeRec : IChoValidatable
{
    [ChoYamlRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoYamlRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate
    (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, 
                ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

The sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes the below methods:

  • TryValidate - Validate entire object, return true if all validation passed. Otherwise return false.
  • Validate - Validate entire object, throw exception if validation is not passed.
  • TryValidateFor - Validate specific property of the object, return true if all validation passed. Otherwise return false.
  • ValidateFor - Validate specific property of the object, throw exception if validation is not passed.

8. Callback Mechanism

ChoYamlReader offers industry standard YAML parsing out of the box to handle most of the parsing needs. If the parsing is not handling any of the needs, you can use the callback mechanism offered by ChoYamlReader to handle such situations. In order to participate in the callback mechanism, you can use either of the following models:

  • Using event handlers exposed by ChoYamlReader via IChoReader interface
  • Inheriting POCO entity object from IChoNotifyRecordRead / IChoNotifyFileRead / IChoNotifyRecordFieldRead interfaces
  • Inheriting DataAnnotation's MetadataType type object by IChoNotifyRecordRead / IChoNotifyFileRead / IChoNotifyRecordFieldRead interfaces
  • Inheriting IChoNotifyRecordFieldConfigurable / IChoNotifyRecordFieldConfigurable configuration interfaces

Note: Any exceptions raised out of these interface methods will be ignored.

IChoReader exposes the below events:

  • BeginLoad - Invoked at the begin of the YAML file load
  • EndLoad - Invoked at the end of the YAML file load
  • BeforeRecordLoad - Raised before the YAML record load
  • AfterRecordLoad - Raised after YAML record load
  • RecordLoadError - Raised when YAML record load errors out
  • BeforeRecordFieldLoad - Raised before YAML field value load
  • AfterRecordFieldLoad - Raised after YAML field value load
  • RecordFieldLoadError - Raised when YAML field value errors out
  • SkipUntil - Raised before the YAML parsing kicks off to add custom logic to skip record lines.
  • DoWhile - Raised during YAML parsing where you can add custom logic to stop the parsing.

IChoNotifyRecordRead exposes the below methods:

  • BeforeRecordLoad - Raised before the YAML record load
  • AfterRecordLoad - Raised after YAML record load
  • RecordLoadError - Raised when YAML record load errors out

IChoNotifyFileRead exposes the below methods:

  • BeginLoad - Invoked at the begin of the YAML file load
  • EndLoad - Invoked at the end of the YAML file load
  • SkipUntil - Raised before the YAML parsing kicks off to add custom logic to skip record lines
  • DoWhile - Raised during YAML parsing where you can add custom logic to stop the parsing

IChoNotifyRecordFieldRead exposes the below methods:

  • BeforeRecordFieldLoad - Raised before yaml field value load
  • AfterRecordFieldLoad - Raised after yaml field value load
  • RecordFieldLoadError - Raised when yaml field value errors out

IChoNotifyRecordConfigurable exposes the below methods:

  • RecondConfigure - Raised for YAML record configuration

IChoNotifyRecordFieldConfigurable exposes the below methods:

  • RecondFieldConfigure - Raised for each yaml record field configuration

8.1. Using ChoYamlReader Events

This is a more direct and simplest way to subscribe to the callback events and handle your odd situations in parsing YAML files. The downside is that code can't be reusable as you do by implementing IChoNotifyRecordRead with POCO record object.

The sample below shows how to use the BeforeRecordLoad callback method to skip lines starting with '%' characters.

Listing 8.1.1 Using ChoYamlReader Callback Events
C#
static void IgnoreLineTest()
{
    using (var parser = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]"))
    {
        parser.BeforeRecordLoad += (o, e) =>
        {
            if (e.Source != null)
            {
                e.Skip = !((IDictionary<string, object>)e.Source).ContainsKey("Name");
            }
        };
        foreach (var e in parser)
            Console.WriteLine(e.Dump());
    }
}

Likewise, you can use other callback methods as well with ChoYamlReader.

8.2. Implementing IChoNotifyRecordRead Interface

The sample below shows how to implement IChoNotifyRecordRead interface to direct POCO class.

Listing 8.2.1 Direct POCO Callback Mechanism Implementation
C#
[ChoYamlRecordObject]
public partial class EmployeeRec : IChoNotifyRecordRead
{
    [ChoYamlRecordField(FieldName = "Id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoYamlRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

The sample below shows how to attach Metadata class to POCO class by using MetadataTypeAttribute on it.

Listing 8.2 MetaDataType Based Callback Mechanism Implementation
C#
[ChoYamlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
    [ChoYamlRecordField(FieldName = "Id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoYamlRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

The sample below shows how to attach Metadata class for sealed or third party POCO class by using ChoMetadataRefTypeAttribute on it.

Listing 8.2.3 ChoMetaDataRefType Based Callback Mechanism Implementation
C#
[ChoMetadataRefType(typeof(EmployeeRec))]
[ChoYamlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead
{
    [ChoYamlRecordField(FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoYamlRecordField(FieldName = "Name")]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

public partial class EmployeeRec
{
    public int Id { get; set; }    
    public string Name { get; set; }
}

8.3. BeginLoad

This callback is invoked once at the beginning of the YAML file load. source is the YAML file stream object. In here, you have a chance to inspect the stream, return true to continue the YAML load. Returns false to stop the parsing.

Listing 8.1.1 BeginLoad Callback Sample
C#
public bool BeginLoad(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

8.4. EndLoad

This callback is invoked once at the end of the YAML file load. source is the YAML file stream object. In here, you have a chance to inspect the stream, do any post steps to be performed on the stream.

Listing 8.2.1 EndLoad Callback Sample
C#
public void EndLoad(object source)
{
    StreamReader sr = source as StreamReader;
}

8.5. BeforeRecordLoad

This callback is invoked before each YAML node in the YAML file is loaded. target is the instance of the POCO record object. index is the JObject node index in the file. source is the YAML record object. In here, you have a chance to inspect the object, and override it with new values if want to.

TIP: If you want to skip the JObject from loading, set the source to null.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.5.1 BeforeRecordLoad Callback Sample
C#
public bool BeforeRecordLoad(object target, int index, ref object source)
{
    IDictionary<string, object> obj = source as IDictionary<string, object>;
    return true;
}

8.6. AfterRecordLoad

This callback invoked after each JObject node in the YAML file is loaded. target is the instance of the POCO record object. index is the JObject node index in the file. source is the YAML record object. In here, you have a chance to do any post step operation with the JObject line.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.6.1 AfterRecordLoad Callback Sample
C#
public bool AfterRecordLoad(object target, int index, object source)
{
    IDictionary<string, object> obj = source as IDictionary<string, object>;
    return true;
}

8.7. RecordLoadError

This callback is invoked if error is encountered while loading JObject node. target is the instance of the POCO record object. index is the JObject node index in the file. source is the JObject node. ex is the exception object. In here, you have chance to handle the exception. This method is invoked only when Configuration.ErrorMode is ReportAndContinue.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.7.1 RecordLoadError Callback Sample
C#
public bool RecordLoadError(object target, int index, object source, Exception ex)
{
    IDictionary<string, object> obj = source as IDictionary<string, object>;
    return true;
}

8.8. BeforeRecordFieldLoad

This callback is invoked before each YAML record field is loaded. target is the instance of the POCO record object. index is the JObject node index in the file. propName is the YAML record property name. value is the YAML field value. In here, you have a chance to inspect the YAML record property value and perform any custom validations, etc.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.8.1 BeforeRecordFieldLoad Callback Sample
C#
public bool BeforeRecordFieldLoad
(object target, int index, string propName, ref object value)
{
    return true;
}

8.9. AfterRecordFieldLoad

This callback is invoked after each YAML record field is loaded. target is the instance of the POCO record object. index is the JObject node index in the file. propName is the YAML record property name. value is the YAML field value. Any post field operation can be performed here, like computing other properties, validations, etc.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.9.1 AfterRecordFieldLoad Callback Sample
C#
public bool AfterRecordFieldLoad
(object target, int index, string propName, object value)
{
    return true;
}

8.10. RecordLoadFieldError

This callback is invoked when error is encountered while loading YAML record field value. target is the instance of the POCO record object. index is the JObject node index in the file. propName is the YAML record property name. value is the YAML field value. ex is the exception object. In here, you have a chance to handle the exception. This method is invoked only after the below two sequences of steps have been performed by the ChoYamlReader.

  • ChoYamlReader looks for FallbackValue value of each YAML property. If present, it tries to assign its value to it.
  • If the FallbackValue value is not present and the Configuration.ErrorMode is specified as ReportAndContinue, this callback will be executed.

Returns true to continue the load process, otherwise returns false to stop the process.

Listing 8.10.1 RecordFieldLoadError Callback Sample
C#
public bool RecordFieldLoadError
(object target, int index, string propName, object value, Exception ex)
{
    return true;
}

8.11. SkipUntil

This callback is invoked at the start of the YAML parsing with custom logic to skip nodes. index is the JObject node index in the file.

Returns true to skip the line, otherwise returns false.

Listing 8.11.1 SkipUntil Callback Sample
public bool SkipUntil(long index, object source)
{
    return false;
}

8.12. DoWhile

This callback is invoked at the start of the YAML parsing with custom logic to skip nodes. index is the JObject node index in the file.

Returns true to stop the parsing, otherwise returns false.

Listing 8.12.1 DoWhile Callback Sample
C#
public bool DoWhile(long index, object source)
{
    return false;
}

9. Customization

ChoYamlReader automatically detects and loads the configured settings from POCO entity. At runtime, you can customize and tweak these parameters before YAML parsing. ChoYamlReader exposes Configuration property, it is of ChoYamlRecordConfiguration object. Using this property, you can customize them.

Listing 9.1 Customizing ChoYamlReader at Run-time
C#
class Program
{
    static void Main(string[] args)
    {
        using (var parser = new ChoYamlReader<EmployeeRec>("emp.yaml"))
        {
            object row = null;
  
            parser.Configuration.YamlPath = "$.emps[*]";
            while ((row = parser.Read()) != null)
                Console.WriteLine(row.ToString());
        }
    }

10. AsDataReader Helper Method

ChoYamlReader exposes AsDataReader helper method to retrieve the YAML records in .NET datareader object. DataReader are fast-forward streams of data. This datareader can be used in few places like bulk coping data to database using SqlBulkCopy, loading disconnected DataTable, etc.

Listing 10.1 Reading as DataReader Sample
C#
static void AsDataReaderTest()
{
    using (var parser = new ChoYamlReader<EmployeeRec>
          ("emp.yaml").WithYamlPath("$.emps[*]"))
    {
        IDataReader dr = parser.AsDataReader();
        while (dr.Read())
        {
            Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
        }
    }
}

11. AsDataTable Helper Method

ChoYamlReader exposes AsDataTable helper method to retrieve the YAML records in .NET DataTable object. It then can be persisted to disk, displayed in grid/controls or stored in memory like any other object.

Listing 11.1 Reading as DataTable Sample
C#
static void AsDataTableTest()
{
    using (var parser = new ChoYamlReader<EmployeeRec>
          ("emp.yaml").WithYamlPath("$.emps[*]"))
    {
        DataTable dt = parser.AsDataTable();
        foreach (DataRow dr in dt.Rows)
        {
            Console.WriteLine("Id: {0}, Name: {1}", dr[0], dr[1]);
        }
    }
}

12. Using Dynamic Object

So far, the article explained about using ChoYamlReader with POCO object. ChoYamlReader also supports loading YAML file without POCO object. It leverages .NET dynamic feature. The sample below shows how to read YAML stream without POCO object.

If you have YAML file, you can parse and load the file with minimal/zero configuration.

The sample below shows it:

Listing 12.1 Loading YAML File
C#
class Program
{
    static void Main(string[] args)
    {
        dynamic row;
        using (var parser = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]"))
        {
            while ((row = parser.Read()) != null)
            {
                Console.WriteLine(row.Id);
            }
        }
    }
}

The above example automatically discovers the YAML object members and parses the file.

You can override the default behavior of discovering fields automatically by adding field configurations manually and pass it to ChoYamlReader for parsing file.

The sample shows how to do it.

Listing 12.3 Loading YAML File with Configuration
C#
class Program
{
    static void Main(string[] args)
    {
        ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
        config.YamlRecordFieldConfigurations.Add
                              (new ChoYamlRecordFieldConfiguration("Id"));
        config.YamlRecordFieldConfigurations.Add
                              (new ChoYamlRecordFieldConfiguration("Name"));

        dynamic row;
        using (var parser = new ChoYamlReader
              ("emp.yaml", config).WithYamlPath("$.emps[*]"))
        {
            while ((row = parser.Read()) != null)
            {
                Console.WriteLine(row.Name);
            }
        }
    }
}

To completely turn off the auto field discovery, you will have to set ChoYamlRecordConfiguration.AutoDiscoverColumns to false.

12.1. DefaultValue

It is the value used and set to the property when the Yaml value is empty or whitespace (controlled via IgnoreFieldValueMode).

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute.

For dynamic object members or to override the declarative POCO object member's default value specification, you can do so through configuration as shown below:

C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id"));
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name") 
       { DefaultValue = "NoName" })

12.2. ChoFallbackValue

It is the value used and set to the property when the Yaml value failed to set. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute.

For dynamic object members or to override the declarative POCO object member's fallback values, you can do through configuration as shown below:

C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id"));
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name") 
       { FallbackValue = "Tom" });

12.3. FieldType

In the type less dynamic object model, the reader reads individual field value and populates them to dynamic object members in 'string' value. If you want to enforce the type and do extra type checking during load, you can do so by declaring the field type at the field configuration.

Listing 12.5.1 Defining FieldType
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id") 
       { FieldType = typeof(int) });
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

The above sample shows to define field type as 'int' to 'Id' field. This instructs the ChoYamlReader to parse and convert the value to integer before assigning to it. This extra type safety alleviates the incorrect values being loaded to object while parsing.

12.4. Type Converters

Most of the primitive types are automatically converted and set them to the properties by ChoYamlReader. If the value of the Yaml field can't automatically be converted into the type of the property, you can specify a custom / built-in .NET converters to convert the value. These can be either IValueConverter or TypeConverter converters.

In the dynamic object model, you can specify these converters via configuration. See the below example on the approach taken to specify type converters for Yaml fields.

Listing 12.4.1 Specifying TypeConverters
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();

ChoYamlRecordFieldConfiguration idConfig = new ChoYamlRecordFieldConfiguration("Id");
idConfig.AddConverter(new IntConverter());
config.YamlRecordFieldConfigurations.Add(idConfig);

config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

In the above, we construct and attach the IntConverter to 'Id' field using AddConverter helper method in ChoYamlRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoYamlRecordFieldConfiguration object.

12.5. Validations

ChoYamlReader leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual Yaml fields. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 12.5.1 Specifying Validations
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();

ChoYamlRecordFieldConfiguration idConfig = new ChoYamlRecordFieldConfiguration("Id");
idConfig.Validators = new ValidationAttribute[] { new RangeAttribute(0, 100) };
config.YamlRecordFieldConfigurations.Add(idConfig);

config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

In the example above, we used Range validation attribute for Id property. ChoYamlReader performs validation on them during load based on Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

P.S.: Self validation NOT supported in Dynamic object model

13. Working with Sealed POCO Object

If you already have existing sealed POCO object or the object is in 3rd party library, we can use them with ChoYamlReader.

Listing 13.1 Existing sealed POCO Object
C#
public sealed class ThirdPartyRec
{
    public int Id
    {
        get;
        set;
    }
    public string Name
    {
        get;
        set;
    }
}
Listing 13.2 Consuming YAML file
C#
class Program
{
    static void Main(string[] args)
    {
        using (var parser = new ChoYamlReader<ThirdPartyRec>
              ("emp.yaml").WithYamlPath("$.emps[*]"))
        {
            object row = null;
 
            while ((row = parser.Read()) != null)
                Console.WriteLine(row.ToString());
        }
    }
}

In this case, ChoYamlReader reverse discovers the Yaml fields from the YAML file and loads the data into POCO object. If the YAML file structure and POCO object matches, the load will succeed with populating all corresponding data to its properties. In case the property is missing for any Yaml field, ChoYamlReader silently ignores them and continue on with the rest.

You can override this behavior by setting ChoYamlRecordConfiguration.ThrowAndStopOnMissingField property to false. In this case, the ChoYamlReader will throw ChoMissingRecordFieldException exception if a property is missing for a Yaml field.

14. Exceptions

ChoYamlReader throws different types of exceptions in different situations:

  • ChoParserException - YAML file is bad and parser is not able to recover.
  • ChoRecordConfigurationException - Any invalid configuration settings are specified, this exception will be raised.
  • ChoMissingRecordFieldException - A property is missing for a Yaml field, this exception will be raised.

15. Using MetadataType Annotation

Cinchoo ETL works better with data annotation's MetadataType model. It is a way to attach MetaData class to data model class. In this associated class, you provide additional metadata information that is not in the data model. Its role is to add attribute to a class without having to modify this one. You can add this attribute that takes a single parameter to a class that will have all the attributes. This is useful when the POCO classes are auto generated (by Entity Framework, MVC, etc.) by automatic tools. This is why the second class comes into play. You can add new stuff without touching the generated file. Also, this promotes modularization by separating the concerns into multiple classes.

For more information about it, please search in MSDN.

Listing 15.1 MetadataType Annotation Usage Sample
C#
[MetadataType(typeof(EmployeeRecMeta))]
public class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

[ChoYamlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
    [ChoYamlRecordField(FieldName = "id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoYamlRecordField(FieldName = "Name")]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate
    (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, 
                               ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}

In the above, EmployeeRec is the data class. Contains only domain specific properties and operations. Mark it very simple class to look at it.

We separate the validation, callback mechanism, configuration, etc. into metadata type class, EmployeeRecMeta.

16. Configuration Choices

If the POCO entity class is an auto-generated class or exposed via library or it is a sealed class, it limits you to attach YAML schema definition to it declaratively. In such a case, you can choose one of the options below to specify YAML layout configuration

  • Manual Configuration
  • Auto Map Configuration
  • Attaching MetadataType class

I'm going to show you how to configure the below POCO entity class on each approach.

Listing 16.1 Sealed POCO Entity Class
C#
public sealed class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

16.1 Manual Configuration

Define a brand new configuration object from scratch and add all the necessary Yaml fields to the ChoYamlConfiguration.YamlRecordFieldConfigurations collection property. This option gives you greater flexibility to control the configuration of YAML parsing. But the downside is the possibility of making mistakes and it is hard to manage them if the YAML file layout is large.

Listing 16.1.1 Manual Configuration
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id"));
config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Name"));

16.2 Auto Map Configuration

This is an alternative approach and very less error-prone method to auto map the Yaml fields for the POCO entity class.

First, define a schema class for EmployeeRec POCO entity class as below:

Listing 16.2.1 Auto Map Class
C#
public class EmployeeRecMap
{
    [ChoYamlRecordField(FieldName = "Id")]
    public int Id { get; set; }
 
    [ChoYamlRecordField(FieldName = "Name")]
    public string Name { get; set; } 
}

Then you can use it to auto map Yaml fields by using ChoYamlRecordConfiguration.MapRecordFields method.

Listing 16.2.2 Using Auto Map Configuration
C#
ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
config.MapRecordFields<EmployeeRecMap>();

foreach (var e in new ChoYamlReader<EmployeeRec>
        ("emp.yaml", config).WithYamlPath("$.emps[*]")) 
    Console.WriteLine(e.ToString());

16.3 Attaching MetadataType Class

This is another approach to attach MetadataType class for POCO entity object. The previous approach simply cares for auto mapping of Yaml fields only. Other configuration properties like property converters, parser parameters, default/fallback values, etc. are not considered.

This model, accounts for everything by defining MetadataType class and specifying the YAML configuration parameters declaratively. This is useful when your POCO entity is sealed and not partial class. Also, it is one of favorable and less error-prone approach to configure YAML parsing of POCO entity.

Listing 16.3.1 Define MetadataType Class
C#
[ChoYamlRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordRead, IChoValidatable
{
    [ChoYamlRecordField(FieldName = "Id", ErrorMode = ChoErrorMode.ReportAndContinue )]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, 1, ErrorMessage = "Id must be > 0.")]
    public int Id { get; set; }

    [ChoYamlRecordField(FieldName = "Name")]
    [StringLength(1)]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool AfterRecordLoad(object target, int index, object source)
    {
        throw new NotImplementedException();
    }
 
    public bool BeforeRecordLoad(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }
 
    public bool RecordLoadError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
 
    public bool TryValidate
    (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor(object target, string memberName, 
                               ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public void Validate(object target)
    {
    }
 
    public void ValidateFor(object target, string memberName)
    {
    }
}
Listing 16.3.2 Attaching MetadataType Class
C#
//Attach metadata 
ChoMetadataObjectCache.Default.Attach<EmployeeRec>(new EmployeeRecMeta());

foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]")) 
    Console.WriteLine(e.ToString()

17. LoadText Helper Method

This is a little nifty helper method to parse and load Yaml text string into objects.

Listing 17.1 Using LoadText Method
C#
string txt = @"
emps:
    - id: 1
      name: Tom

    - id: 2
      name: Mark";

foreach (var e in ChoYamlReader.LoadText(txt).WithYamlPath("$.emps[*]"))
   Console.WriteLine(e.ToStringEx());

18. Advanced Topics

18.1 Override Converters Format Specs

Cinchoo ETL automatically parses and converts each Yaml field values to the corresponding Yaml field's underlying data type seamlessly. Most of the basic .NET types are handled automatically without any setup needed.

This is achieved through two key settings in the ETL system:

  1. ChoYamlRecordConfiguration.CultureInfo - Represents information about a specific culture including the names of the culture, the writing system, and the calendar used, as well as access to culture-specific objects that provide information for common operations, such as formatting dates and sorting strings. Default is 'en-US'.
  2. ChoTypeConverterFormatSpec - It is global format specifier class holds all the intrinsic .NET types formatting specs.

In this section, I'm going to talk about changing the default format specs for each .NET intrinsic data types according to parsing needs.

ChoTypeConverterFormatSpec is a singleton class, the instance is exposed via 'Instance' static member. It is thread local, means that there will be a separate instance copy kept on each thread.

There are two sets of format specs members given to each intrinsic type, one for loading and another one for writing the value, except for Boolean, Enum, DataTime types. These types have only one member for both loading and writing operations.

Specifying each intrinsic data type format specs through ChoTypeConverterFormatSpec will impact system wide, i.e., setting ChoTypeConverterFormatSpec.IntNumberStyle = NumberStyles.AllowParentheses will impact all integer members of Yaml objects to allow parentheses. If you want to override this behavior and take control of specific Yaml data member to handle its own unique parsing of Yaml value from global system wide setting, it can be done by specifying TypeConverter at the Yaml field member level. Refer to section 13.4 for more information.

NumberStyles (optional) used for loading values from Yaml stream and Format string are used for writing values to Yaml stream.

In this article, I'll brief about using NumberStyles for loading YAML data from stream. These values are optional. It determines the styles permitted for each type during parsing of YAML file. System automatically figures out the way to parse and load the values from underlying Culture. In odd situations, you may want to override and set the styles the way you want in order to successfully load the file. Refer to MSDN for more about NumberStyles and its values.

Listing 18.1.1 ChoTypeConverterFormatSpec Members
C#
public class ChoTypeConverterFormatSpec
{
    public static readonly ThreadLocal<ChoTypeConverterFormatSpec> Instance = 
    new ThreadLocal<ChoTypeConverterFormatSpec>(() => new ChoTypeConverterFormatSpec());
 
    public string DateTimeFormat { get; set; }
    public ChoBooleanFormatSpec BooleanFormat { get; set; }
    public ChoEnumFormatSpec EnumFormat { get; set; }
 
    public NumberStyles? CurrencyNumberStyle { get; set; }
    public string CurrencyFormat { get; set; }
 
    public NumberStyles? BigIntegerNumberStyle { get; set; }
    public string BigIntegerFormat { get; set; }
 
    public NumberStyles? ByteNumberStyle { get; set; }
    public string ByteFormat { get; set; }
 
    public NumberStyles? SByteNumberStyle { get; set; }
    public string SByteFormat { get; set; }
 
    public NumberStyles? DecimalNumberStyle { get; set; }
    public string DecimalFormat { get; set; }
 
    public NumberStyles? DoubleNumberStyle { get; set; }
    public string DoubleFormat { get; set; }
 
    public NumberStyles? FloatNumberStyle { get; set; }
    public string FloatFormat { get; set; }
 
    public string IntFormat { get; set; }
    public NumberStyles? IntNumberStyle { get; set; }
 
    public string UIntFormat { get; set; }
    public NumberStyles? UIntNumberStyle { get; set; }
 
    public NumberStyles? LongNumberStyle { get; set; }
    public string LongFormat { get; set; }
 
    public NumberStyles? ULongNumberStyle { get; set; }
    public string ULongFormat { get; set; }
 
    public NumberStyles? ShortNumberStyle { get; set; }
    public string ShortFormat { get; set; }
 
    public NumberStyles? UShortNumberStyle { get; set; }
    public string UShortFormat { get; set; }
}

The sample below shows how to load Yaml data stream having 'se-SE' (Swedish) culture specific data using ChoYamlReader. Also the input feed comes with 'EmployeeNo' values containing parentheses. In order to make the load successful, we have to set the ChoTypeConverterFormatSpec.IntNumberStyle to NumberStyles.AllowParenthesis.

Listing 18.1.2 Using ChoTypeConverterFormatSpec in code
C#
static void UsingFormatSpecs()
{
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    config.Culture = new System.Globalization.CultureInfo("se-SE");
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id") 
           { FieldType = typeof(int) });
    config.YamlRecordFieldConfigurations.Add
           (new ChoYamlRecordFieldConfiguration("Name"));
    config.YamlRecordFieldConfigurations.Add
           (new ChoYamlRecordFieldConfiguration("Salary") 
           { FieldType = typeof(ChoCurrency) });
    config.YamlRecordFieldConfigurations.Add
           (new ChoYamlRecordFieldConfiguration("JoinedDate") 
           { FieldType = typeof(DateTime) });
    config.YamlRecordFieldConfigurations.Add
           (new ChoYamlRecordFieldConfiguration("EmployeeNo") 
           { FieldType = typeof(int) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var parser = new ChoYamlReader("emp.yaml", config).WithYamlPath("$.emps[*]"))
    {
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

18.2 Currency Support

Cinchoo ETL provides ChoCurrency object to read and write currency values in YAML files. ChoCurrency is a wrapper class to hold the currency value in decimal type along with support of serializing them in text format during YAML load.

Listing 18.2.1 Using Currency Members in Dynamic Model
C#
static void CurrencyDynamicTest()
{
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Id"));
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Name"));
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Salary") 
                          { FieldType = typeof(ChoCurrency) });
 
    using (var parser = 
           new ChoYamlReader("emp.yaml", config).WithYamlPath("$.emps[*]"))
    {
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

The sample above shows how to load currency values using dynamic object model. By default, all the members of dynamic object are treated as string type, unless specified explicitly via ChoYamlFieldConfiguration.FieldType. By specifying the field type as ChoCurrency to the 'Salary' Yaml field, ChoYamlReader loads them as currency object.

P.S.: The format of the currency value is figured by ChoYamlReader through ChoRecordConfiguration.Culture and ChoTypeConverterFormatSpec.CurrencyNumberStyle.

The sample below shows how to use ChoCurrency Yaml field in POCO entity class.

Listing 18.2.2 Using Currency Members in POCO Model
C#
public class EmployeeRecWithCurrency
{
    public int Id { get; set; }
    public string Name { get; set; }
    public ChoCurrency Salary { get; set; }
}
 
static void CurrencyTest()
{
    using (var parser = new ChoYamlReader<EmployeeRecWithCurrency>
          ("emp.yaml").WithYamlPath("$.emps[*]"))
    {
        object rec;
        while ((rec = parser.Read()) != null)
        {
            Console.WriteLine(rec.ToStringEx());
        }
    }
}

18.3 Enum Support

Cinchoo ETL implicitly handles parsing of enum field values from YAML files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.EnumFormat. Default is ChoEnumFormatSpec.Value.

FYI, changing this value will impact system wide.

There are three possible values that can be used:

  1. ChoEnumFormatSpec.Value - Enum value is used for parsing.
  2. ChoEnumFormatSpec.Name - Enum key name is used for parsing.
  3. ChoEnumFormatSpec.Description - If each enum key is decorated with DescriptionAttribute, its value will be used for parsing.
Listing 18.3.1 Specifying Enum format specs during parsing
C#
public enum EmployeeType
{
    [Description("Full Time Employee")]
    Permanent = 0,
    [Description("Temporary Employee")]
    Temporary = 1,
    [Description("Contract Employee")]
    Contract = 2
}

static void EnumTest()
{
    ChoTypeConverterFormatSpec.Instance.EnumFormat = ChoEnumFormatSpec.Description;
 
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id") 
                                            { FieldType = typeof(int) });
    config.YamlRecordFieldConfigurations.Add
                             (new ChoYamlRecordFieldConfiguration("Name"));
    config.YamlRecordFieldConfigurations.Add
                             (new ChoYamlRecordFieldConfiguration("Salary") 
                                            { FieldType = typeof(ChoCurrency) });
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration
                                            ("JoinedDate") 
                                            { FieldType = typeof(DateTime) });
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration
                          ("EmployeeType") { FieldType = typeof(EmployeeType) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var parser = new ChoYamlReader("emp.yaml", config).WithYamlPath("$.emps[*]"))
    {
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

18.4 Boolean Support

Cinchoo ETL implicitly handles parsing of boolean Yaml field values from YAML files. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.BooleanFormat. Default value is ChoBooleanFormatSpec.ZeroOrOne.

FYI, changing this value will impact system wide.

There are four possible values that can be used:

  1. ChoBooleanFormatSpec.ZeroOrOne - '0' for false. '1' for true.
  2. ChoBooleanFormatSpec.YOrN - 'Y' for true, 'N' for false.
  3. ChoBooleanFormatSpec.TrueOrFalse - 'True' for true, 'False' for false.
  4. ChoBooleanFormatSpec.YesOrNo - 'Yes' for true, 'No' for false.
Listing 18.4.1 Specifying Boolean Format Specs during Parsing
C#
static void BoolTest()
{
    ChoTypeConverterFormatSpec.Instance.BooleanFormat = ChoBooleanFormatSpec.ZeroOrOne;
 
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Id") 
                                            { FieldType = typeof(int) });
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Name"));
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Salary") 
                                            { FieldType = typeof(ChoCurrency) });
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration
                                            ("JoinedDate") 
                                            { FieldType = typeof(DateTime) });
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Active") 
                                            { FieldType = typeof(bool) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var parser = new ChoYamlReader
          ("emp.yaml", config).WithYamlPath("$.emps[*]"))
    {
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

18.5 DateTime Support

Cinchoo ETL implicitly handles parsing of datetime Yaml field values from YAML files using system Culture or custom set culture. If you want to fine control the parsing of these values, you can specify them globally via ChoTypeConverterFormatSpec.DateTimeFormat. Default value is 'd'.

FYI, changing this value will impact system wide.

You can use any valid standard or custom datetime .NET format specification to parse the datetime Yaml values from the file.

Listing 18.5.1 Specifying datetime Format Specs during Parsing
C#
static void DateTimeTest()
{
    ChoTypeConverterFormatSpec.Instance.DateTimeFormat = "MMM dd, yyyy";
 
    ChoYamlRecordConfiguration config = new ChoYamlRecordConfiguration();
    config.YamlRecordFieldConfigurations.Add(new ChoYamlRecordFieldConfiguration("Id") 
                                            { FieldType = typeof(int) });
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Name"));
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Salary") 
                                            { FieldType = typeof(ChoCurrency) });
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("JoinedDate") 
                                            { FieldType = typeof(DateTime) });
    config.YamlRecordFieldConfigurations.Add
                          (new ChoYamlRecordFieldConfiguration("Active") 
                                            { FieldType = typeof(bool) });
 
    ChoTypeConverterFormatSpec.Instance.IntNumberStyle = NumberStyles.AllowParentheses;
 
    using (var parser = new ChoYamlReader
          ("emp.yaml", config).WithYamlPath("$.emps[*]"))
    {
        object row = null;
 
        while ((row = parser.Read()) != null)
            Console.WriteLine(row.ToStringEx());
    }
}

The sample above shows how to parse custom datetime Yaml values from YAML file.

Note: As the datetime values contains YAML separator, it is given with double quotes to pass the parsing.

19. Fluent API

ChoYamlReader exposes few frequent to use configuration parameters via fluent API methods. This will make the programming of parsing of YAML files quicker.

19.1 WithYamlPath

This API method sets the YamlPath expression to select the nodes to load using ChoYamlReader.

C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithYamlPath("$.emps[*]"))
    Console.WriteLine(e.ToString());

19.2 WithFields

This API method specifies the list of YAML nodes (either attributes or elements) to be considered for parsing and loading. Other fields in the YAML nodes will be discarded.

C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithFields("Id", "Name"))
    Console.WriteLine(e.ToString());

19.3 WithField

This API method used to add YAML node with YamlPath, data type and other parameters. This method is helpful in dynamic object model, by specifying each and individual YAML node with appropriate datatype.

C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").WithField
        ("Id", fieldType: typeof(int)))
    Console.WriteLine(e.ToString());

19.4 ColumnCountStrict

This API method used to set the ChoYamlWriter to perform check on field countness before reading the YAML file.

C#
foreach (var e in new ChoYamlReader<EmployeeRec>("emp.yaml").ColumnCountStrict())
    Console.WriteLine(e.ToString());

19.5 NotifyAfter

This API method used to define the number of rows to be processed before generating a notification event. This property is designed for user interface components that illustrate the YAML loading progress. Notifications are sent to subscribers who subscribed to RowsLoaded event.

C#
static void NotifyAfterTest()
{
    using (var parser = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]")
        .NotifyAfter(1000)
        )
    {
        parser.RowsLoaded += (o, e) => Console.WriteLine(e.RowsLoaded); 

        foreach (var rec in parser)
        {
            Console.WriteLine(String.Format("Id: {0}", rec.Id));
            Console.WriteLine(String.Format("Name: {0}", rec.Name));
            Console.WriteLine(String.Format("Salary: {0}", rec.Salary));
        }
    }
}

19.6 Configure

This API method used to configure all configuration parameters which are not exposed via Fluent API.

C#
static void ConfigureTest()
{
    using (var parser = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]")
        .Configure(c => c.ErrorMode = ChoErrorMode.ThrowAndStop)
        )
    {
        foreach (var rec in parser)
        {
            Console.WriteLine(String.Format("Id: {0}", rec.Id));
            Console.WriteLine(String.Format("Name: {0}", rec.Name));
            Console.WriteLine(String.Format("Salary: {0}", rec.Salary));
        }
    }
}

19.7 Setup

This API method used to setup the reader's parameters / events via Fluent API.

C#
static void SetupTest()
{
    using (var parser = new ChoYamlReader("emp.yaml").WithYamlPath("$.emps[*]")
        .Setup(r => r.BeforeRecordLoad += (o, e) =>
        {
            if (e.Source.CastTo<JObject>().ContainsKey("Name1"))
                e.Skip = true;
        }
        )
    {
        foreach (var rec in parser)
        {
            Console.WriteLine(String.Format("Id: {0}", rec.Id));
            Console.WriteLine(String.Format("Name: {0}", rec.Name));
            Console.WriteLine(String.Format("Salary: {0}", rec.Salary));
        }
    }
}

20. FAQs

20.1. How to Deserialize String as Enum?

ChoYamlReader implicitly handles the conversion of the enum text to enum value. The sample below shows how to load YAML with POCO object:

C#
public enum Gender { Male, Female }
public class Employee
{
    public int Age { get; set; }
    public Gender Gender { get; set; }
}

static void EnumTest()
{
    string yaml = @"
emps:
    - Age: 15
      name: Male

    - Age: 25
      name: Female
";

    using (var r = ChoYamlReader<Employee>.LoadText(yaml)
        .WithYamlPath("$.emps[*]")
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}

The sample below show how to parse the YAML with enum values in dynamic object model approach:

C#
static void DynamicEnumTest()
{
    string yaml = @"
emps:
    - Age: 15
      Gender: Male

    - Age: 25
      Gender: Female
";

    using (var r = ChoYamlReader.LoadText(yaml).WithYamlPath("$.emps[*]")
        .WithField("Age")
        .WithField("Gender", fieldType: typeof(Gender))
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}

20.2. How to Load Only the Values From Key-Value Pair YAML ?

With the custom YamlPath ($..^), you can load only the values from yaml.

C#
static void LoadDictKeysTest()
{
    string yaml = @"
Age: 15
Gender: Male
";

    using (var r = ChoYamlReader.LoadText(yaml)
        .WithYamlPath("$.^")
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}

20.3. How to Load Only the Keys From Key-Value Pair Yaml ?

With the custom YamlPath ($..~), you can load only the values from yaml.

C#
static void LoadDictKeysTest()
{
    string yaml = @"
Age: 15
Gender: Male
";

    using (var r = ChoYamlReader.LoadText(yaml)
        .WithYamlPath("$.~")
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}

20.4. Deserialize Yaml into Dynamic Object?

ChoYamlReader does this implicitly in the dynamic object model.

C#
static void DynamicEnumTest()
{
    string yaml = @"
emps:
    - Age: 15
      Gender: Male

    - Age: 25
      Gender: Female
";

    using (dynamic r = ChoYamlReader.LoadText(yaml).WithYamlPath("$.emps[*]")
        .WithField("Age")
        .WithField("Gender", fieldType: typeof(Gender))
        )
    {
        foreach (var rec in r)
        {
            Console.WriteLine(rec.Age);
            Console.WriteLine(rec.Gender);
        }
    }
}

In the above, the parser loads the YAML file, constructs and returns dynamic object.

20.5. How to Convert YAML to XML?

Cinchoo ETL provides ChoXmlWriter to generate XML file from objects. With ChoYamlReader along with ChoXmlWriter, you can convert YAML to XML format easily.

C#
static void Yaml2XmlTest()
{
     string yaml = @"
emps:
    - id: 1
      name: Mark

    - id: 2
      name: Tom
";

    StringBuilder xml = new StringBuilder();
    using (var r = ChoYamlReader.LoadText(yaml).WithYamlPath("$.emps[*]"))
    {
        using (var w = new ChoXmlWriter(xml)
            .WithRootName("Emps")
            .WithNodeName("Emp")
            )
            w.Write(r);
    }
    Console.WriteLine(xml.ToString());
}

Output

XML
<Emps>
  <Emp>
    <Id>1</Id>
    <Name>Mark</Name>
  </Emp>
  <Emp>
    <Id>2</Id>
    <Name>Tom</Name>
  </Emp>
</Emps>

20.6. How to Load Selective Nodes from YAML?

With YamlPath, you can load selective nodes from YAML into objects using ChoYamlReader.

For a sample YAML below:

C#
users:
    - name: 1
      teamname: Tom
      email: xx@gmail.com
      players: [1, 2]

wanted to load User data into object. Sample code uses YamlPath "$.user" to pick the node and parse it.

C#
public class UserInfo
{
    public string name { get; set; }
    public string teamname { get; set; }
    public string email { get; set; }
    public int[] players { get; set; }
}

static void SelectiveNodeTest1()
{
    string yaml = @"
users:
    - name: 1
      teamname: Tom
      email: xx@gmail.com
      players: [1, 2]
";
    using (var r = ChoYamlReader<UserInfo>.LoadText(yaml)
        .WithYamlPath("$.users[*]")
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}}

20.7. How to Load Selective Subnodes From YAML?

With YamlPath at each member level, you can load selective subnodes from YAML into object members using ChoYamlReader.

For a sample YAML below:

C#
users:
    - name: 1
      teamname: Tom
      email: xx@gmail.com
      players: [1, 2]

Wanted to load User data into object. The sample code uses YamlPath on each field to pick the subnode and parse it.

C#
public class UserInfo
{
    [ChoYamlRecordField(YamlPath = "$.name")]
    public string name { get; set; }
    [ChoYamlRecordField(YamlPath = "$.teamname")]
    public string teamname { get; set; }
    [ChoYamlRecordField(YamlPath = "$.email")]
    public string email { get; set; }
    [ChoYamlRecordField(YamlPath = "$.players")]
    public int[] players { get; set; }
}

In the above, each member has been specified with YamlPath to choose the node value from. This way, you can pick and choose yaml node value with complex YamlPath, give you more control to choose the data from.

C#
static void SelectiveNodeTest()
{
    string yaml = @"
users:
    - name: 1
      teamname: Tom
      email: xx@gmail.com
      players: [1, 2]
";
    using (var r = ChoYamlReader<UserInfo>.LoadText(yaml)
        .WithYamlPath("$.users[*]")
        )
    {
        foreach (var rec in r)
            Console.WriteLine(rec.Dump());
    }
}

20.7. How to Convert Yaml to DataTable?

ChoYamlReader provides a little helper method to convert yaml to Datatable, AsDataTable():

C#
static void ConvertToDataTableTest()
{
    string yaml = @"
emps:
    - id: 1
      name: Tom

    - id: 2
      name: Mark
";

    using (var r = ChoYamlReader<UserInfo>.LoadText(yaml)
        .WithYamlPath("$.emps[*]")
        )
    {
        var dt = r.AsDataTable();
    }
}

20.8. How to Convert Yaml to DataReader?

ChoYamlReader provides a little helper method to convert yaml to DataReader, AsDataReader().

C#
static void ConvertToDataReaderTest()
{
    string yaml = @"
emps:
    - id: 1
      name: Tom

    - id: 2
      name: Mark
";

    using (var r = ChoYamlReader<UserInfo>.LoadText(yaml)
        .WithYamlPath("$.emps[*]")
        )
    {
        var dt = r.AsDataReader();
    }
}

20.9. How to Convert Yaml to CSV?

Cinchoo ETL provides ChoCSVWriter to generate csv file from objects. With ChoYamlReader along with ChoCSVWriter, you can convert yaml to CSVformat easily.

Yaml is hierarchical object model, CSV is flat structure. Cinchoo handles them seamlessly and produces the output as expected.

C#
static void Yaml2CSV2()
{
    string yaml = @"
emps:
    - id: 1
      name: Tom

    - id: 2
      name: Mark
";

    StringBuilder csv = new StringBuilder();
    using (var r = ChoYamlReader.LoadText(yaml).WithYamlPath("$.emps[*]")
        )
    {
        using (var w = new ChoCSVWriter(csv)
            .WithFirstLineHeader()
            .NestedColumnSeparator('/')
            )
            w.Write(r);
    }

    Console.WriteLine(csv.ToString());
}

20.10. How to Deserialize an Object?

This sample deserializes yaml to an object:

C#
public class Emp
{
    public int Id { get; set; }
    public string Name { get; set; }
}

static void DeserializeObjectTest()
{
    string yaml = @"
id: 1
name: Tom
";
    Console.WriteLine(ChoYamlReader.DeserializeText<Emp>(yaml).FirstOrDefault().Dump());
}

20.11. How to Deserialize a Collection?

This sample deserializes yaml to a collection:

C#
static void DeserializeCollectioTest()
{
    string yaml = @"
emps: 
    - Tom
    - Mark
";
    var emps = ChoYamlReader.DeserializeText<string>(yaml, "$.emps[*]").ToList();
    Console.WriteLine(emps.Dump());
}

20.12. How to Deserialize a Dictionary?

This sample deserializes yaml to a Dictionary:

C#
static void DeserializeDictTest()
{
    string yaml = @"
id: 1
name: Tom
";
    Console.WriteLine(ChoYamlReader.DeserializeText<Dictionary
                      <string, object>>(yaml).FirstOrDefault().Dump());
}

20.13. How to Deserialize from a File?

This sample deserializes yaml to a Dictionary:

C#
public class Movie
{
    public string Name { get; set; }
    public int Year { get; set; }
}

static void DeserializeFromFile()
{
    Movie movie1 = ChoYamlReader.Deserialize<Movie>("emps.yaml").FirstOrDefault();
}

20.14. How to Deserialize with Custom Factory?

This sample deserializes yaml with custom factory to instantiate Employee instance for Person type.

C#
public class Person
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public DateTime BirthDate { get; set; }
}

public class Employee : Person
{
    public string Department { get; set; }
    public string JobTitle { get; set; }
}

static void CustomCreationTest()
{
    ChoActivator.Factory = (type, args) =>
    {
        if (type == typeof(Person))
            return new Employee();
        else
            return null;
    };
    Person person = ChoYamlReader.DeserializeText<Person>(yaml).FirstOrDefault();
    Console.WriteLine(person.GetType().Name);
}

20.15 Reading an Entire File - Using a Simple Data Structure

Often, you need to read the entire content of a file, so using a YAML Path filter does not help. What you want to do is to map your YAML file to a data structure that you can use directly:

C#
class PersonData
{
    public DateTime Creation { get; set; }
    public string Firstname { get; set; }
    public string Lastname { get; set; }
    public int Age { get; set; }
}

static void ReadEntireFile1()
{
    string yaml = @"
creation: 2020-12-26
firstname: John
lastname: Doe
age: 35";

    using (var parser = ChoYamlReader<PersonData>.LoadText(yaml))
    {
        foreach (var e in parser)
        {
            Console.WriteLine(e.Firstname); // John
            Console.WriteLine(e.Lastname);  // Doe
        }
    }
}

20.16 Reading an Entire File - Using Multiple Lists

Often, you have a file that contain multiple lists. These are simply defined using arrays of different structures.

C#
class ChildData
{
    public string Firstname { get; set; }
    public string Lastname { get; set; }
    public DateTime Dob { get; set; }
}
class AddressData
{
    public string Street { get; set; }
    public string City { get; set; }
    public string State { get; set; }
    public string Zip { get; set; }
    public string Country { get; set; }
}
class ParentData
{
    public DateTime Creation { get; set; }
    public string Firstname { get; set; }
    public string Lastname { get; set; }
    public int Age { get; set; }
    public ChildData[] Children  { get; set; }
    public AddressData[] Addresses  { get; set; }
}

static void ReadEntireFile2()
{
    string yaml = @"
creation: 2020-12-26
firstname: John
lastname: Doe
age: 35
children:
- firstname: Emmanuel
  lastname: Doe
  dob: 1990-01-02
- firstname: Elise
  lastname: Doe
  dob: 1996-10-02
addresses:
- street: 1234 California Ave
  city: San Francisco
  state: CA
  zip: 98765
  country: USA
- street: 666 Midtown Ct
  city: Palo Alto
  zip: 94444
  state: CA
  country: USA
";

    using (var parser = ChoYamlReader<ParentData>.LoadText(yaml))
    {
        foreach (var e in parser)
        {
            Console.WriteLine(e.Firstname); // John
            Console.WriteLine(e.Lastname);  // Doe
            Console.WriteLine(e.Children[0].Firstname); // Emmanuel
            Console.WriteLine(e.Addresses[0].Street);   // 1234 California Ave
        }
    }
}

20.17 Using Generic Data Structures (IDictionnary)

Even though it is more cumbersome and will probably not be used except in some very specific cases (see FAQ 20.19), it is useful to understand the logic of the ChoYamlReader and how it deals with what it reads. Here is the same content as in 12.15, but without assuming anything on the file. Note the ugly casts and manual work that you have to perform to get to the values.

C#
static void ReadEntireFileAnonymous()
{
    string yaml = @"
creation: 2020-12-26
firstname: John
lastname: Doe
age: 35";
    using (var parser = ChoYamlReader<IDictionary<string, object>>.LoadText(yaml))
    {
        foreach (var e in parser)
        {
            string firstname = (string) e["firstname"];                 // John
            DateTime creation = DateTime.Parse((string) e["creation"]); // 2020-12-26
            int age = (int) e["age"];                                   // 35
            Console.WriteLine(e.Dump());
        }
    }
}

20.18 Using Generic Data Lists (IList)

Under the same idea, you can read lists of anonymous data. Here again, this example is just here to explain how it works under the hood, the use of arrays as described in 20.16 is safer and more convenient.

C#
class ChildrenData
{
    public IList<object> Children { get; set; }
}

static void ReadAnonymousLists()
{
    string yaml = @"
children:
- firstname: Emmanuel
  lastname: Doe
  dob: 1990-01-02
- firstname: Elise
  lastname: Doe
  dob: 1996-10-02
";
    using (var parser = ChoYamlReader<ChildrenData>.LoadText(yaml))
    {
        foreach (var e in parser)
        {
            IDictionary<object, object> firstChild = 
                                (IDictionary<object, object>) e.Children[0];
            string firstname = (string) firstChild["firstname"];    // Emmanuel
            Console.WriteLine(e.Dump());
        }
    }
}

20.19 Reading an Entire File - Dynamic Structures

Sometimes, the file format can vary greatly and contains non-homogeneous data at the same level. We won't go into the debate of whether this is good practice or not, but here are some realities of computer science at times.

For example, here we have a list of possessions that have a type field and attributes that depend on the type of object. Combining 20.17 and 20.18, this gives:

C#
class OwnerData
{
    public PossessionData[] Possessions { get; set; }
}

class PossessionData
{
    public string Type { get; set; }
    public IDictionary<string, object> Description  { get; set; }
}

static void ReadDynamicData()
{
    string yaml = @"
possessions:
- type: car
  description:
    color: blue
    doors: 4
- type: computer
  description:
    disk: 1 TB
    memory: 16 MB
";
    using (var parser = ChoYamlReader<OwnerData>.LoadText(yaml))
    {
        foreach (var e in parser)
        {
            string carColor = (string) e.Possessions[0].Description["color"]; // blue
            foreach (var p in e.Possessions)
            {
                Console.WriteLine(p.Description.Dump());
            }
        }
    }
}

History

  • 1st July, 2020: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
-- There are no messages in this forum --