Cinchoo ETL - Parquet Writer

Cinchoo

Rate me:

0.00/5 (No votes)

17 Jun 2020CPOL27 min read

14.1K

Simple Parquet writer for .NET

ChoETL is an open source ETL (extract, transform and load) framework for .NET. It is a code based library for extracting data from multiple sources, transforming, and loading into your very own data warehouse in .NET environment. You can have data in your data warehouse in no time.

Introduction
Requirement
"Hello World!" Sample
Writing All Records
Write Records Manually
Customize Parquet Record
Customize Parquet Fields
Callback Mechanism
Customization
Using Dynamic Object
Exceptions
Using MetadataType Annotation
Configuration Choices
Writing DataReader Helper Method
Writing DataTable Helper Method
Advanced Topics
Fluent API
FAQ
History

1. Introduction

Apache Parquet, an open source file format for Hadoop. Parquet stores nested data structures in a flat columnar format. Compared to a traditional approach where data is stored in row-oriented approach, parquet is more efficient in terms of storage and performance.

This article talks about using ChoParquetWriter component offered by ChoETL framework. It is a simple utility class to save Parquet data to a file / external data source.

Corresponding ChoParquetReader, a Parquet reader article can be found here.

Features

Uses Parquet.NET library under the hood, to generate Parquet file in seconds.
Supports culture specific date, currency and number formats while generating files.
Provides fine control of date, currency, enum, boolean, number formats when writing files.
Detailed and robust error handling, allowing you to quickly find and fix problems.
Shorten your development time.

2. Requirement

This framework library is written in C# using .NET 4.5 Framework / .NET core 2.x.

3. "Hello World!" Sample

Open VS.NET 2017 or higher
Create a sample VS.NET (.NET Framework 4.x / .NET Core 2.x) Console Application project
Install ChoETL via Package Manager Console using Nuget Command based on working .NET version:
- Install-Package ChoETL.Parquet
Use the ChoETL namespace

Let's begin by looking into a simple example of generating the below Parquet file having 2 columns

Image 3.1 Sample Parquet data file (emp.parquet)

There are number of ways you can get the Parquet file be parsed with minimal setup.

3.1. Quick write - Data First Approach

This is the zero-config and quickest approach to create Parquet file in no time. No typed POCO object is needed. Sample code below shows how to generate sample Parquet file using dynamic objects

Listing 3.1.1 Write List of Objects to Parquet File

List<ExpandoObject> objs = new List<ExpandoObject>();
dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoParquetWriter("emp.parquet"))
{
    parser.Write(objs);
}

In the above sample, we give the list of dynamic objects to ParquetWriter at one pass to write them to Parquet file.

Listing 3.1.2 Write Each Object to Parquet File

using (var parser = new ChoParquetWriter("emp.parquet"))
{
    dynamic rec1 = new ExpandoObject();
    rec1.Id = 1;
    rec1.Name = "Mark";
    parser.Write(item);

    dynamic rec1 = new ExpandoObject();
    rec1.Id = 2;
    rec1.Name = "Jason";
    parser.Write(item);
}

In the above sample, we take control of constructing, passing each and individual dynamic record to the ParquetWriter to generate the Parquet file using Write overload.

3.2. Code First Approach

This is another zeo-config way to generate Parquet file using typed POCO class. First, define a simple POCO class to match the underlying Parquet file layout

Listing 3.2.1 Simple POCO Entity Class

public partial class EmployeeRecSimple
{
    public int Id { get; set; }
    public string Name { get; set; } 
}

In the above, the POCO class defines two properties matching the sample Parquet file template.

Listing 3.2.2 Saving to Parquet file

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoParquetWriter<EmployeeRecSimple>("emp.parquet"))
{
    parser.Write(objs);
}

The above sample shows how to create Parquet file from typed POCO class objects.

3.3. Configuration First Approach

In this model, we define the Parquet configuration with all the necessary parameters along with Parquet columns required to generate the sample Parquet file.

Listing 3.3.1 Define Parquet configuration

ChoParquetRecordConfiguration config = new ChoParquetRecordConfiguration();
config.ParquetRecordFieldConfigurations.Add
       (new ChoParquetRecordFieldConfiguration("Id"));
config.ParquetRecordFieldConfigurations.Add
       (new ChoParquetRecordFieldConfiguration("Name"));

In the above, the class defines two Parquet properties matching the sample Parquet file template.

Listing 3.3.2 Generate Parquet File without POCO Object

List<ExpandoObject> objs = new List<ExpandoObject>();

dynamic rec1 = new ExpandoObject();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
dynamic rec2 = new ExpandoObject();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoParquetWriter("emp.parquet", config))
{
    parser.Write(objs);
}

The above sample code shows how to generate Parquet file from list of dynamic objects using predefined Parquet configuration setup. In the ParquetWriter constructor, we specified the Parquet configuration object to obey the Parquet layout schema while creating the file. If there is any mismatch in the name or count of Parquet columns, it will be reported as an error and it stops the writing process.

Listing 3.3.3 Saving Parquet File with POCO Object

List<EmployeeRecSimple> objs = new List<EmployeeRecSimple>();

EmployeeRecSimple rec1 = new EmployeeRecSimple();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRecSimple rec2 = new EmployeeRecSimple();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoParquetWriter<EmployeeRecSimple>("emp.parquet", config))
{
    parser.Write(objs);
}

The above sample code shows how to generate Parquet file from list of POCO objects with Parquet configuration object. In the ParquetWriter constructor, we specified the Parquet configuration object.

3.4. Code First with Declarative Configuration

This is the combined approach to define POCO entity class along with attaching Parquet configuration parameters declaratively. id is required column and name is optional value column with default value "XXXX". If name is not present, it will take the default value.

Listing 3.4.1 Define POCO Object

public class EmployeeRec
{
    [ChoParquetRecordField]
    [Required]
    public int? Id
    {
        get;
        set;
    }

    [ChoParquetRecordField]
    [DefaultValue("XXXX")]
    public string Name
    {
        get;
        set;
    }

    public override string ToString()
    {
        return "{0}. {1}.".FormatString(Id, Name);
    }
}

The code above illustrates about defining POCO object with necessary attributes required to generate Parquet file. First thing defines property for each record field with ChoParquetRecordFieldAttribute to qualify for Parquet record mapping. Id is a required property. We decorated it with RequiredAttribute. Name is given default value using DefaultValueAttribute. It means that if the Name value is not set in the object, ParquetWriter spits the default value 'XXXX' to the file.

It is very simple and ready to save Parquet data in no time.

Listing 3.4.2 Saving Parquet File with POCO Object

List<EmployeeRec> objs = new List<EmployeeRec>();

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 1;
rec1.Name = "Mark";
objs.Add(rec1);
 
EmployeeRec rec2 = new EmployeeRec();
rec2.Id = 2;
rec2.Name = "Jason";
objs.Add(rec2);
 
using (var parser = new ChoParquetWriter<EmployeeRec>("emp.parquet"))
{
    parser.Write(objs);
}

We start by creating a new instance of ChoParquetWriter object. That's all. All the heavy lifting of generating Parquet data from the objects is done by the writer under the hood.

By default, ParquetWriter discovers and uses default configuration parameters while saving Parquet file. These can be overridable according to your needs. The following sections will give you in-depth details about each configuration attributes.

4. Writing All Records

It is as easy as setting up POCO object match up with Parquet file structure, construct the list of objects and pass it to ParquetWriter's Write method. This will write the entire list of objects into Parquet file in one single call.

Listing 4.1 Write to Parquet File

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var parser = new ChoParquetWriter<EmployeeRec>("emp.parquet"))
{
    parser.Write(objs);
}

or:

Listing 4.2 Writer to Parquet file stream

List<EmployeeRec> objs = new List<EmployeeRec>();
//Construct and attach objects to this list
...

using (var tx = File.OpenWrite("emp.parquet"))
{
    using (var parser = new ChoParquetWriter<EmployeeRec>(tx))
    {
        parser.Write(objs);
    }
}

This model keeps your code elegant, clean, easy to read and maintain.

5. Write Records Manually

This is an alternative way to write each and individual record to Parquet file in case when the POCO objects are constructed in a disconnected way.

Listing 5.1 Writing to Parquet File

var writer = new ChoParquetWriter<EmployeeRec>("emp.parquet");

EmployeeRec rec1 = new EmployeeRec();
rec1.Id = 1;
rec1.Name = "Mark";
 
writer.Write(rec1);

EmployeeRec rec2 = new EmployeeRec();
rec1.Id = 2;
rec1.Name = "Jason"; 

writer.Write(rec2);

6. Customize Parquet Record

Using ChoParquetRecordObjectAttribute, you can customize the POCO entity object declaratively.

Listing 6.1 Customizing POCO Object for Each Record

[ChoParquetRecordObject]
public class EmployeeRec
{
    [ChoParquetRecordField]
    public int Id { get; set; }
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available attributes to carry out customization of Parquet load operation on a file.

Culture - The culture info used to read and write.
ColumnCountStrict - This flag indicates if an exception should be thrown if Parquet field configuration mismatch with the data object members.
ErrorMode - This flag indicates if an exception should be thrown if writing and an expected field is failed to write. This can be overridden per property. Possible values are:
- IgnoreAndContinue - Ignore the error, record will be skipped and continue with next.
- ReportAndContinue - Report the error to POCO entity if it is of IChoNotifyRecordWrite type
- ThrowAndStop - Throw the error and stop the execution
ObjectValidationMode - A flag to let the reader know about the type of validation to be performed with record object. Possible values are:
- Off - No object validation performed. (Default)
- MemberLevel - Validation performed before each Parquet property gets written to the file.
- ObjectLevel - Validation performed before all the POCO properties are written to the file.

7. Customize Parquet Fields

For each Parquet column, you can specify the mapping in POCO entity property using ChoParquetRecordFieldAttribute.

Listing 7.1 Customizing POCO object for Parquet Columns

public class EmployeeRec
{
    [ChoParquetRecordField]
    public int Id { get; set; }
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Here are the available members to add some customization to it for each property:

FieldName - Parquet field name. If not specified, POCO object property name will be used as field name.
Size - Size of Parquet column value.
NullValue - Special null value text expect to be treated as null value from Parquet file at the field level.
ErrorMode - This flag indicates if an exception should be thrown if writing and an expected field failed to convert and write. Possible values are:
- IgnoreAndContinue - Ignore the error and continue to load other properties of the record.
- ReportAndContinue - Report the error to POCO entity if it is of IChoRecord type.
- ThrowAndStop - Throw the error and stop the execution.

7.1. DefaultValue

Any POCO entity property can be specified with default value using System.ComponentModel.DefaultValueAttribute. It is the value used to write when the Parquet value null (controlled via IgnoreFieldValueMode).

7.2. ChoFallbackValue

Any POCO entity property can be specified with fallback value using ChoETL.ChoFallbackValueAttribute. It is the value used when the property is failed to writer to Parquet. Fallback value only set when ErrorMode is either IgnoreAndContinue or ReportAndContinue.

7.3. Type Converters

Most of the primitive types are automatically converted to string/text and save them to Parquet file. If the value of the Parquet field aren't automatically be converted into the text value, you can specify a custom / built-in .NET converters to convert the value to text. These can be either IValueConverter, IChoValueConverteror TypeConverter converters.

There are couple of ways you can specify the converters for each field:

Declarative Approach
Configuration Approach

7.3.1. Declarative Approach

This model is applicable to POCO entity object only. If you have POCO class, you can specify the converters to each property to carry out necessary conversion on them. The samples below show the way to do it.

Listing 7.3.1.1 Specifying Type Converters

public class EmployeeRec
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    public int Id { get; set; }
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

Listing 7.3.1.2 IntConverter Implementation

public class IntConverter : IValueConverter
{
    public object Convert
    (object value, Type targetType, object parameter, CultureInfo culture)
    {
        return value;
    }
 
    public object ConvertBack
    (object value, Type targetType, object parameter, CultureInfo culture)
    {
        int intValue = (int)value;
        return intValue.ToString("D4");
    }
}

In the example above, we defined custom IntConverter class. And showed how to format 'Id' Parquet property with leading zeros.

7.3.2. Configuration Approach

This model is applicable to both dynamic and POCO entity object. This gives freedom to attach the converters to each property at runtime. This takes precedence over the declarative converters on POCO classes.

Listing 7.3.2.1 Specifying TypeConverters

ChoParquetRecordConfiguration config = new ChoParquetRecordConfiguration();

ChoParquetRecordFieldConfiguration idConfig = 
                new ChoParquetRecordFieldConfiguration("Id");
idConfig.AddConverter(new IntConverter());
config.ParquetRecordFieldConfigurations.Add(idConfig);

config.ParquetRecordFieldConfigurations.Add
       (new ChoParquetRecordFieldConfiguration("Name"));
config.ParquetRecordFieldConfigurations.Add
       (new ChoParquetRecordFieldConfiguration("Name1"));

In the above, we construct and attach the IntConverter to 'Id' field using AddConverter helper method in ChoParquetRecordFieldConfiguration object.

Likewise, if you want to remove any converter from it, you can use RemoveConverter on ChoParquetRecordFieldConfiguration object.

7.3.3. Custom Value Converter Approach

This approach allows to attach value converter to each Parquet member using Fluent API. This is a quick way to handle any odd conversion process and avoid creating value converter class.

Listing 7.3.3.1 POCO Class

public class EmployeeRec
{
    [ChoParquetRecordField]
    public int Id { get; set; }
    [ChoParquetRecordField(2, FieldName ="Name", QuoteField = true)]
    [Required]
    [DefaultValue("ZZZ")]
    public string Name { get; set; }
}

With the Fluent API, the sample below shows how to attach value converter to Id column.

Listing 7.3.3.2 Attaching Value Converter

using (var dr = new ChoParquetWriter<EmployeeRec>(@"Test.parquet")
    .WithField(c => c.Id, 
    valueConverter: (v) => ((int)value).ToString("C3", CultureInfo.CurrentCulture))
    )
{
    Console.WriteLine(rec);
}

7.4. Validations

ParquetWriter leverages both System.ComponentModel.DataAnnotations and Validation Block validation attributes to specify validation rules for individual fields of POCO entity. Refer to the MSDN site for a list of available DataAnnotations validation attributes.

Listing 7.4.1 Using Validation Attributes in POCO Entity

[ChoParquetRecordObject]
public partial class EmployeeRec
{
    [ChoParquetRecordField(1, FieldName = "id")]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
}

In the example above, used Range validation attribute for Id property. Required validation attribute to Name property. ParquetWriter performs validation on them before saving the data to file when Configuration.ObjectValidationMode is set to ChoObjectValidationMode.MemberLevel or ChoObjectValidationMode.ObjectLevel.

In some cases, you may want to take control and perform manual self validation within the POCO entity class. This can be achieved by inheriting POCO object from IChoValidatable interface.

Listing 7.4.2 Manual Validation on POCO Entity

[ChoParquetRecordObject]
public partial class EmployeeRec : IChoValidatable
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
 
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }
 
    public bool TryValidate
    (object target, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
 
    public bool TryValidateFor
    (object target, string memberName, ICollection<ValidationResult> validationResults)
    {
        return true;
    }
}

The sample above shows how to implement custom self-validation in POCO object.

IChoValidatable interface exposes the below methods:

TryValidate - Validate entire object, return true if all validation passed. Otherwise return false.
TryValidateFor - Validate specific property of the object, return true if all validation passed. Otherwise return false.

7.5. ChoIgnoreMember

If you want to ignore a POCO class member from Parquet parsing in OptOut mode, decorate them with ChoIgnoreMemberAttribute. The sample below shows Title member is ignored from Parquet loading process.

Listing 7.5.1 Ignore a Member

public class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
    [ChoIgnoreMember]
    public string Title { get; set; }
}

7.6. StringLength

In OptOut mode, you can specify the size of the Parquet column by using System.ComponentModel.DataAnnotations.StringLengthAttribute.

Listing 7.6.1 Specifying Size of Parquet member

public class EmployeeRec
{
    public int Id { get; set; }
    [StringLength(25)]
    public string Name { get; set; }
    [ChoIgnoreMember]
    public string Title { get; set; }
}

7.6. Display

In OptOut mode, you can specify the name of Parquet column mapped to member using System.ComponentModel.DataAnnotations.DisplayAttribute.

Listing 7.6.1 Specifying name of Parquet field

public class EmployeeRec
{
    public int Id { get; set; }
    [Display(Name="FullName")]
    [StringLength(25)]
    public string Name { get; set; }
    [ChoIgnoreMember]
    public string Title { get; set; }
}

7.7. DisplayName

In OptOut mode, you can specify the name of Parquet column mapped to member using System.ComponentModel.DataAnnotations.DisplayNameAttribute.

Listing 7.7.1 Specifying Name of Parquet Field

public class EmployeeRec
{
    public int Id { get; set; }
    [Display(Name="FullName")]
    [StringLength(25)]
    public string Name { get; set; }
    [ChoIgnoreMember]
    public string Title { get; set; }
}

8. Callback Mechanism

ParquetWriter offers industry standard Parquet data file generation out of the box to handle most of the needs. If the generation process is not handling any of your needs, you can use the callback mechanism offered by ParquetWriter to handle such situations. In order to participate in the callback mechanism, you can use either of the following models:

Using event handlers exposed by ParquetWriter via IChoWriter interface
Inheriting POCO entity object from IChoNotifyRecordWrite / IChoNotifyFileWrite / IChoNotifyRecordFieldWrite interfaces
Inheriting DataAnnotation's MetadataType type object by IChoNotifyRecordWrite / IChoNotifyFileWrite / IChoNotifyRecordFieldWrite interfaces.

In order to participate in the callback mechanism, either POCO entity object or DataAnnotation's MetadataType type object must be inherited by IChoNotifyRecordWrite interface.

Tip: Any exceptions raised out of these interface methods will be ignored.

IChoRecorder exposes the below methods:

BeginWrite - Invoked at the begin of the Parquet file write
EndWrite - Invoked at the end of the Parquet file write
BeforeRecordWrite - Raised before the Parquet record write
AfterRecordWrite - Raised after Parquet record write
RecordWriteError - Raised when Parquet record errors out while writing
BeforeRecordFieldWrite - Raised before Parquet column value write
AfterRecordFieldWrite - Raised after Parquet column value write
RecordFieldWriteError - Raised when Parquet column value errors out while writing

IChoNotifyRecordWrite exposes the below methods:

BeforeRecordWrite - Raised before the Parquet record write
AfterRecordWrite - Raised after Parquet record write
RecordWriteError - Raised when Parquet record write errors out

IChoNotifyFileWrite exposes the below methods:

BeginWrite - Invoked at the begin of the Parquet file write
EndWrite - Invoked at the end of the Parquet file write

IChoNotifyRecordFieldWrite exposes the below methods:

BeforeRecordFieldWrite - Raised before Parquet column value write
AfterRecordFieldWrite - Raised after Parquet column value write
RecordFieldWriteError - Raised when Parquet column value write errors out

IChoNotifyFileHeaderArrange exposes the below methods:

FileHeaderArrange - Raised before Parquet file header is written to file, an opportunity to rearrange the Parquet columns

IChoNotifyFileHeaderWrite exposes the below methods:

FileHeaderWrite - Raised before Parquet file header is written to file, an opportunity to customize the header.

8.1. Using ParquetWriter Events

This is more direct and simplest way to subscribe to the callback events and handle your odd situations in parsing Parquet files. The downside is that code can't be reusable as you do by implementing IChoNotifyRecordRead with POCO record object.

The sample below shows how to use the BeforeRecordLoad callback method to skip lines stating with '%' characters.

Listing 8.1.1 Using ParquetWriter Callback Events

static void IgnoreLineTest()
{
    using (var parser = new ChoParquetWriter("emp.parquet"))
    { 
        parser.BeforeRecordWrite += (o, e) =>
        {
            if (e.Source != null)
            {
                e.Skip = ((IDictionary<string, object>)e.Source).ContainsKey("name1");
            }
        };
        
        parser.Write(rec);
    }
}

Likewise, you can use other callback methods as well with ParquetWriter.

8.2. Implementing IChoNotifyRecordWrite Interface

The sample below shows how to implement IChoNotifyRecordWrite interface to direct POCO class.

Listing 8.2.1 Direct POCO Callback Mechanism Implementation

[ChoParquetRecordObject]
public partial class EmployeeRec : IChoNotifyrRecordWrite
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }
    
    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError(object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
}

The sample below shows how to attach Metadata class to POCO class by using MetadataTypeAttribute on it.

Listing 8.2.2 MetaDataType based Callback Mechanism Implementation

[ChoParquetRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordWrite
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError
           (object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

[MetadataType(typeof(EmployeeRecMeta))]
public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

The sample below shows how to attach Metadata class for sealed or third party POCO class by using ChoMetadataRefTypeAttribute on it.

Listing 8.2.3 MetaDataType Based Callback Mechanism Implementation

[ChoMetadataRefType(typeof(EmployeeRec))]
[ChoParquetRecordObject]
public class EmployeeRecMeta : IChoNotifyRecordWrite
{
    [ChoParquetRecordField]
    [ChoTypeConverter(typeof(IntConverter))]
    [Range(1, int.MaxValue, ErrorMessage = "Id must be > 0.")]
    [ChoFallbackValue(1)]
    public int Id { get; set; }

    [ChoParquetRecordField]
    [Required]
    [DefaultValue("ZZZ")]
    [ChoFallbackValue("XXX")]
    public string Name { get; set; }

    public bool AfterRecordWrite(object target, int index, object source)
    {
        throw new NotImplementedException();
    }

    public bool BeforeRecordWrite(object target, int index, ref object source)
    {
        throw new NotImplementedException();
    }

    public bool RecordWriteError
           (object target, int index, object source, Exception ex)
    {
        throw new NotImplementedException();
    }
} 

public partial class EmployeeRec
{
    public int Id { get; set; }
    public string Name { get; set; }
}

8.1. BeginWrite

This callback invoked once at the beginning of the Parquet file write. source is the Parquet file stream object. In here, you have chance to inspect the stream, return true to continue the Parquet generation. Return false to stop the generation.

Listing 8.1.1 BeginWrite Callback Sample

public bool BeginWrite(object source)
{
    StreamReader sr = source as StreamReader;
    return true;
}

8.2. EndWrite

This callback invoked once at the end of the Parquet file generation. source is the Parquet file stream object. In here, you have chance to inspect the stream, do any post steps to be performed on the stream.

Listing 8.2.1 EndWrite Callback Sample

public void EndWrite(object source)
{
    StreamReader sr = source as StreamReader;
}

8.3. BeforeRecordWrite

This callback invoked before each POCO record object is written to Parquet file. target is the instance of the POCO record object. index is the line index in the file. source is the Parquet record line. In here, you have a chance to inspect the POCO object, and generate the Parquet record line if needed.

Tip: If you want to skip the record from writing, set the source to null.