Click here to Skip to main content
65,938 articles
CodeProject is changing. Read more.
Articles
(untagged)

FileHelpers v3.1 - Delimited (CSV) or Fixed Data Import/Export Framework

0.00/5 (No votes)
17 Aug 2015 3  
An easy to use .NET library to read/write strong typed data from files with fixed length or delimited records (CSV). Also has support to import/export data from different data storages (Excel, Acces, SqlServer, MySql)

FileHelpers Home Page

FileHelpers v 3.1

Introduction

The FileHelpers are an easy to use library to import/export data from fixed length or delimited flat files (like the CSV). FileHelpers also has support to import/export data from different data storages (Excel, Acces, SqlServer).

The library has a set of converters for the basic types and can be easily extended to provide custom converters.

The main idea is pretty simple:

You can strong type your flat file (fixed or delimited) simply describing a class that maps to each record of the file and work with your file like a strong typed .NET array.

Basic Uses

Directly between the files and the .NET source code:

general read write of filehelpers 

It also can be used as an intermediate between files and MS Access or MS SqlServer tables:

 

General read write to data storage

Download Instructions

The recommend method to install the library is use the NuGet Package

www.nuget.org/packages/FileHelpers/

Via de NuGet Console:     Install-Package FileHelpers

Or you can use the Visual Studio NuGet Explorer to search for FileHelpers

add references

 

Quick Start Guide (easy steps)

To start using the FileHelpers Library you only need to add a reference in your project to the file: FileHelpers.dll. You can find it in the Release directory of the distribution. Tip: remember to leave the FileHelpers.xml file to get Intellisense support.

Next you need to define a class that maps to the record in the source/detination file. For this example we use a file with this format delimited by a |:

10248|VINET|04071996|32.38
10249|TOMSP|05071996|11.61
10250|HANAR|08071996|65.83
10251|VICTE|08071996|41.34
...............

The class that we refer can be like this:

[DelimitedRecord("|")]
public class Orders
{
   public int OrderID;

   public string CustomerID;

   [FieldConverter(ConverterKind.Date, "ddMMyyyy")]
   public DateTime OrderDate;

   public decimal Freight;
}

Later you must instantiate a FileHelperEngine and Read/Write files:

var engine = new FileHelperEngine<Orders>();

/// to Read use:
var res = engine.ReadFile("input.txt");

/// to Write use:
engine.WriteFile("output.txt", res);

Finally you can use the res array to access each item in the file, for example:

foreach (Orders order in res)
{
   Console.WriteLine("Order Info:");
   Console.WriteLine(order.CustomerID + " - " +
                     order.OrderDate.ToString("dd/MM/yy"));
}

Working with FixedLength files is exactly the same but you need to use the [FixedLengthRecord] and [FieldFixedLength] attributes.

More Examples

We have ton of examples in our web site to learn to use the library from every angle

Who needs the File Helpers Library?

In almost every project there is a need to read/write data from/to a flat file of a specified format. Writing the code to process these files is not hard, you can use String.Split or String.SubString, but if you do that the code sometimes becomes hard to read and change. If you want to add support later for quoted string, multiline, convert types, etc. these things tend to get complicated.

So with the FileHelpers you don't need to worry about changes in the file structure because you can change the Type, Converters, Triming, Length in seconds.

Another place where you can find the FileHelpers useful is for log parsing, data warehouse and OLAP applications, communication between systems, file format transformations (for example from a fixed length to an CSV file), load test data into your NUnit tests and a lot more!!!

This library aims to provide an easy and reliable way to accomplish all these tasks.

Roslyn Analyzer

Our Roslyn Analyzer helps you to use the library in the right way.

The current Analyzer implements:

  • Recomends to use the generic version when you use an engine with a typeof() in the constructor
  • Suggest to use [FieldHidden] instead of [FieldIgnored] or [FieldNotInFile]
  • Use FileHelpers.Dynamic instead of FileHelpers.RunTime namespace
  • Use IComparable instead of the obsolete IComparableRecord 

next version will implement

  • Check that record class contains valid record attributes
  • Check that record class contains any field
  • Check other obsolete methods
  • Convert engine.ReadFile in async version with foreach

You can join the development of the analyzer, join us in the Gitter chat:

https://gitter.im/MarcosMeli/FileHelpers

 

 

Record Class Wizard

Since version 1.3.5 the library has had a Record Class Wizard to generate the record class. The Record Class Wizard allows you to generate your record mapping class, save the definition, generate code based on snippets (Read File, Read with Generics, ReadAsync and so on).


Click to see how it works

Main Features

 

 Easy to use

The FileHelpers library is straightforward to learn and use.
Check the Examples

 Auto Converters

The library has a set of converters for common types and can be easy extended to provide custom converters.
Check the Example

 Event Support

The engines of the library contain events to make it easier to extend the behavior of the library
Check the INotifyRead andINotifyWrite Examples

 High Performance

The library doesn't use reflection to get or set the field values, it uses dynamic code generation

 Wide framework support

You can use the library in
.NET 2.03.04.04.54.6 and Mono!!

 Multiple Error Modes

You can configure the library to use different ErrorModes.
Check the example

 BigFileSorter

You can use the library to sort files with millon of records and MBs
Check the Example

 File Transform Engine

To convert files in one format to another (for example a file with CSV to a fixed length record format)
Check the Example

 FileDiffEngine

Provides the ability to compare files with the same record layout
Check the Example

 MasterDetailEngine

You can process hierarchical data with this engine
Check the Example

 Nullable Types

The library supports Nullable types in the core
Check the Example

 Record by Record Mode

You can use the library to read line by line and not the whole file.
Check the Example

 Open Source

The library is completely free to use in any kind of development
github.com/MarcosMeli/FileHelpers

 Format Autodetection

The library has an smart feature that based in some sample files can deduce the Record Class
Check the Example

 Wizard

We provide a Wizard app to help in the creation of the Record Class
Check the Online Wizard
 

 

Progress Notification

Here is a screenshot of the progress notification of the demo app.

Since versión 1.4.0 you can get notified of the progress of each operation of the library to show feedback to the user or whatever you want.

Class Diagram

Detailed Diagram

Detailed Diagram

Testing

The library contains more than 650 NUnit tests to ensure correctness:

Sample image

 
List of Converters Arguments

Here is a list of the parameters that you can give to the default converters:

ConverterKind.Date

Arg1: A string with the DateTime format that the engine passes to the DateTime.Parse function.

Arg2: A string with the encoding used for string convertions

Examples:

// By default the engines use: "ddMMyyyy"
[FieldConverter(ConverterKind.Date)]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   30-01-2006
[FieldConverter(ConverterKind.Date, "d-M-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  1-1-2006   01-1-2006   01-30-2006
[FieldConverter(ConverterKind.Date, "M-d-yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01/3/2006   30/02/2006
[FieldConverter(ConverterKind.Date, "d/M/yyyy" )]
public DateTime ShippedDate;

// Parse these dates:  01042006   30022006
[FieldConverter(ConverterKind.Date, "ddMMyyyy" )]
public DateTime ShippedDate;

// Parse these dates:  04/January/2006   02/July/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "en" )] // or "en-US"
public DateTime ShippedDate;

// Parse these dates:  04/Enero/2006   02/Julio/2006
[FieldConverter(ConverterKind.Date, "dd/MMMM/yyyy", "es" )]
public DateTime ShippedDate;

You can check all the supported format strings check the MSDN docs for DateTime.ParseExact 
and here all support cultures

ConverterKind.Double, ConverterKind.Single and ConverterKind.Decimal

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

Examples:

// "." is the decimal separator by default
[FieldConverter(ConverterKind.Double)]
public double Freight;

// "." is the decimal separator. (good for autodocumented code)
[FieldConverter(ConverterKind.Double, ".")]
public double Freight;

// "," is the decimal separator.
[FieldConverter(ConverterKind.Double, ",")]
public double Freight;

// The same for the other converters:
// ConverterKind.Decimal and ConverterKind.Single

Integer Converters

ConverterKind.Int16, ConverterKind.Int32, ConverterKind.Int64, ConverterKind.Byte, 
ConverterKind.UInt16, ConverterKind.UInt32, ConverterKind.UInt64, and ConverterKind.SByte

Arg1: A string with the character to be used as DecimalSeparator. Valid Values: "." or ",". By default: "."

WARNING: The library requires a decimal separator here and internally creates the group separator with the counterpart (for example if you provide "." it uses ",")

Examples:

// "." is the decimal separator by default
// allows you to parse:  12,125 or 1,458,385
[FieldConverter(ConverterKind.Int32)]
public Int32 Amount;

// using "," as decimal separator
// allows you to parse:  12.125 or 1.458.385
[FieldConverter(ConverterKind.Int32, ",")]
public Int32 Amount;

// The same works for the rest of the converts

ConverterKind.Boolean

Arg1: A string that represents the True value 
Arg2: A string that represents the False value

By default, this converter takes the strings "True" (case insensitive) and "1" as true. The values "True" and "False" are returned when converting the field to string.

Examples:

// By default it reads as true the strings "True" (Case insensitive) and "1"
[FieldConverter(ConverterKind.Boolean)]
public bool Shipped;

// Takes as true the string "verdad" and as false the string "mentira" (case insensitive)
[FieldConverter(ConverterKind.Boolean, "Verdad", "MenTiRa")]
public bool Shipped;

// Takes as true the string "1" and as false the string "0"
// You need these params if you want to write, because it use "True" and "False"
[FieldConverter(ConverterKind.Boolean, "1", "0")]
public bool Shipped;

// Takes as true the string "X" and as false the string "-"
[FieldConverter(ConverterKind.Boolean, "X", "-")]
public bool Shipped;

History

Version 3.1, July 2015

Hi everyone again !!

The FileHelpers has reached a big Milestone today we are releasing the 3.1 version!

Now with full support for .NET 4.0, 4.5 and Mono and with a ton of news and enhancements.

After some time off for different reasons we are now trying to update the library in a more regular fashion. We redesigned the HomeSite with a Material Design Template from GeeksLabs

Main changes

  • Docs rewritten thanks to the work of Ken Foskey and Matthias Cavigelli
  • NEW: Full .NET 2.0, 4.0 & 4.5 support.
  • NEW: Added support for MONO !!
  • NEW: FieldOrder: a new attribute to override the default order of fields.
  • NEW: Partial support to use Autoproperties
  • NEW: Async Methods returns IDisposable to allow using() statements that autoclose files on ends or exception.
  • NEW: Events for the Async engines.
  • NEW: The events args now have TotalBytes and CurrentBytes to allow you show progress while reading or writing.
  • NEW: FieldIndexers you can now get the values in the AsyncEngine like in the DataReader of ADO.NET =) You can access them via index or fieldName (at the moment case sensitive). This also completes the RunTime records support
  • NEW: SmartFormatDetector: A class designed to discover the format of delimited and flat files based on sample files. It is integrated with the wizzard
  • NEW: Better Delimited validations to make sure that the file contains the right number of fields
  • NEW: FieldValueDiscarted: a new attribute to allow the value of a particular field to be read but not stored in the record class (this was a very requested feature)
  • NEW: BigFileSorter: if you need to sort big files, you can use this feature that implements External Sorting
  • MOD: The error messages have been made more meaningful. In all exceptions you get: FieldName, ColumnNumber, LineNumber etc.
  • NEW: FieldNotEmpty: Indicates whether the target field needs to be populated with a non-empty value.

Breaking Changes

  • Removed support for .NET Compact Framework
  • [FieldIgnored] => [FieldHidden]
  • IComparableRecord<T> => IComparable<T>

Minor Changes

  • Better caching of internal classes
  • The core code of the library was refactored to make easier to extend it.
  • Better performance.
  • ADD: UpdateLinks properties, allow to specify how the library must handle Workbook links (thanks to Stefan Schälle)
  • A ton of new examples, autogenerated from FileHelpers.Examples project
  • FIX: A little problem with the quoted fields in the wizard
  • More documentation has been added to the Library source code. It is now even easier to get involved in development!
  •  
 

Licence (MIT)

FileHelpers Library is @Copyright 2005-2015 to Marcos Meli but it's source code and the binaries are free for commercial and non commercial use.

Vote for this article if you like the library.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here