|
In fact, I added a feature at the last minute (SupportMultiline) and my unit tests did not test it thoroughly For now, you can do as the examples with SupportMultiline=true. I fixed the bug so it works all the time now. Thanks for pointing me this out!
Update to the article on its way...
|
|
|
|
|
Hi,
I am tring to load the csv into a DataTable. How would you get the following unit tests to pass?
[Test]
public void LoadDataTableWithSampleData1()
{
System.Data.DataTable table = new System.Data.DataTable();
using (CachedCsvReader csv = new CachedCsvReader(new StringReader(CsvReaderSampleData.SampleData1), true))
{
table.Load(csv);
}
Assert.IsTrue(table.Rows.Count > 0);
}
//or for my sample data
[Test]
public void LoadDataTableWithBrentsData()
{
string s = @"""9990-3311-5"",1,2,07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""D"",01/01/1900 00:00:00,""D"",01/01/1900 00:00:00,""N"",07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""P"",07/28/1997 00:00:00,""N"",07/28/1997 00:00:00,""D"",01/01/1900 00:00:00,""D"",01/01/1900 00:00:00,""AN2"",""D"",01/01/1900 00:00:00,""D"",01/01/1900 00:00:00,""D"",01/01/1900 00:00:00,"" "","" "",01/01/1900 00:00:00,"" "",01/01/1900 00:00:00,"" "",01/01/1900 00:00:00,"" "",01/01/1900 00:00:00,"" "",01/01/1900 00:00:00";
System.Data.DataTable table = new System.Data.DataTable();
using (CachedCsvReader csv = new CachedCsvReader(new StringReader(s), false))
{
table.Load(csv);
}
Assert.IsTrue(table.Rows.Count > 0);
}
|
|
|
|
|
Hum, I must admit I did not try to load a DataTable with the CsvReader. Anyway, the correction needed is very simple:
In CsvReader.GetSchemaTable(), replace
DbType.String, with
typeof(string),
BTW, you do not need to use the CachedCsvReader, CsvReader is fine.
I will update this article tonight.
Sébastien
-- modified at 13:49 Monday 24th July, 2006
|
|
|
|
|
Thank you for the quick fix.
DataTable.Load(...) methods are new in the .NET Framework version 2.0.
|
|
|
|
|
I'm a .NET Newbie using VB.Net 2005 pro. I have need for a CSV Reader and thought if I could get this working with VB.Net it would be great. I've tried using the code below, but it doesn't find references or execute.
Imports System.IO
Public Class Form1
Private Sub GetCSVFile()
Dim Idx As Integer
Dim csv As csvreader
Dim FieldCount As Integer
CsvReader(csv = New CsvReader(New StreamReader("data.csv"), False))
FieldCount = csv.FieldCount
While (csv.ReadNextRecord())
For Idx = 0 To FieldCount - 1
If Idx = 0 Then
Debug.Print("--------------------")
End If
debug.Print(csv(idx).tostring))
Next
End While
End Sub
How do I get a reference to LumenWorks.Framework.IO.dll? Is there a document other than the source code that defines the methods, parameters, etc. for each CSV related statement (such as the 'False' in the 'New CSVReader' statement above)?
Does anybody have this working with VB?
Thanks,
Bob
|
|
|
|
|
You will find the dll in the folder "[...]\LumenWorks.Framework.IO\bin\Release". You should copy "LumenWorks.Framework.IO.dll" and "LumenWorks.Framework.IO.xml" to the location where you keep the libraries you use. Then you can add a reference to this dll in your project. It is important to also copy the xml file because it is what will give you intellisense support.
Your code should be the following:
Imports System
Imports System.Diagnostics
Imports System.IO
Imports LumenWorks.Framework.IO.Csv
Class Test
Private Sub ReadCSVFile()
Dim csv As New CsvReader(New StreamReader("data.csv"), False)
Try
Dim fieldCount As Integer = csv.FieldCount
Do While (csv.ReadNextRecord())
For i As Integer = 0 To fieldCount - 1
If i = 0 Then
Debug.WriteLine("--------------------")
End If
Debug.WriteLine(csv.Item(i))
Next
Loop
Finally
csv.Dispose()
End Try
End Sub
End Class
It is a good practice to declare variables as they are required because you will limit their scope, the lifetime of the object they point to and it will also be easier to look at them since they are closer to where they are actually used.
VB.NET support declaration and initialization on the same line.
Do While ... Loop is the more common way of writing while loops.
Declared this way, the loop variable i will be accessible only during the loop.
.ToString() is not necessary as the return type of csv.Item(i) is already a string.
There is no using keyword in VB.NET, so you need to replace it with a Try/Finally clause as you see above.
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
Thanks Sebastien for the fast response and the tip on "Do While...". The code works perfectly on a csv file that contains only one record type with the same number of fields in each record.
As in:
rec1, fld2, fld3
rec2, fld2, fld3
rec3, fld2, fld3
Unfortunately, the csv files I have to deal with have several 'record types' that contain different number and types of fields. The first field is the record type.
As in:
H, fld2, fld3, fld4
C, fld2, fld3, fld4, fld5,.... fld15
C, fld2, fld3, fld4, fld5,.... fld15
T, fld2, fld3, fld4, fld5
It appears that CSV Reader reads the first line of data and determines how many fields are in each record and processes that number for the entire file. That makes sense when using headers and only one record type within the file.
Is there any workaround using CSV Reader to process the kind of files I'm receiving?
Thanks,
Bob
-- modified at 3:15 Wednesday 19th July, 2006
|
|
|
|
|
Currently, the CsvReader does not support dynamic field counts. Adding that functionality would take a non negligeable amount of time which I prefer to use in other projects. Besides, this kind of CSV should be considered malformed anyway. Null fields should still be present and separated by a comma:
H,fld2,fld3,fld4,,,,,[...],
C,fld2,fld3,fld4,fld5,[...],fld15
C,fld2,fld3,fld4,fld5,[...],fld15
T,fld2,fld3,fld4,fld5,,,,,[...],
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
Thanks again. Guess I'll just have to skin this cat another way!
Bob
|
|
|
|
|
Just a small note. There is now a keyword "using" in VB .Net 2005, so I highly recommend using it if you're coding in VB as it's a great feature. The syntax is as follows...
Using reader As CsvReader = New CsvReader(...)
...
End Using
Bruce Dunwiddie
|
|
|
|
|
It does not mention which open source license is used with this software.
See http://www.opensource.org/licenses/[^] for more info.
In my case, I want to use it in a commercial product, which is, unfortunately, distributed without source code. So, all your copyright notices, mostly placed at the beginning of each code file, will not be visible to our clients. Are you okay with that? If you prefer that I do something in the product to give you the credit, let me know.
Thanks for the effort. I like your project.
|
|
|
|
|
Humm, I guess your software will have a licence or a readme file ... So I would like that you include a copy of the license at the end of your own. In fact, this is the only thing required by the MIT license which I use.
http://www.opensource.org/licenses/mit-license.php
Out of curiosity, may I know what is your product ?
Thanks for your kind comments!
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
Just to be as clear as I can, here is what I have in mind:
Your software licence
...
The Software includes a third-party component "LumenWorks.Framework.IO.CSV.CsvReader" licensed under the following terms:
LumenWorks.Framework.IO.CSV.CsvReader
Copyright (c) 2005 Sébastien Lorion
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE
FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
I have spoken to our legal department about modification of our license to include yours in it. However, this would have a great impact on many aspects of our business. Besides, this component is not even 1% of the product we are selling, so it will be very hard for me to sell your proposal internally. In other words, we need an alternative solution, such as inclusion of the notice at a different location.
Personally, I really like the way you have designed this component, as you have taken the time to implement interfaces to facilitate integration and benchmark the performance to show its competitive edge over other available solutions in the industry. So, I certainly want to give another try.
Thanks.
|
|
|
|
|
Humm, what can I say about that ... I chose the MIT license specifically to avoid such issues. The license is very short and clear, with no weird clauses. The only string attached to Open Source software is the mention of its authors and a company using Open Source component should be aware of that *before* using it. I have seen many commercial applications using OSS and they will all have addendum to their own license.
That said, a compromise would be to include the license in another file, but located in the same folder as yours. You could name it "CsvReader license.txt" or something similar.
Good luck!
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
Well, we have not included your component in any release of our product *yet*. I will certainly try to convince the decision maker to accept the alternative solution. If not, I will have to create one on my own or find another component that does a good job. All in all, thanks for your effort in explanation and clarification. I truly appreciate it.
I will get back to you when the decision is made.
|
|
|
|
|
Okay, we will put it in the license agreement that goes with the installer, as part of addendum. Thanks.
|
|
|
|
|
Wow, you have great diplomatic skills If you can tell me what is your product, I would be curious to check it out.
Thanks!
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
You can check out our web site at http://explorance.com/[^]. However, the demo is not available to the general public.
|
|
|
|
|
Hello, this script contain bug.
This is one record by http://tools.ietf.org/html/4180 but your reader it read as two records.
1996,Jeep,Grand Cherokee,"MUST SELL!
air, moon roof, loaded",4799.00
Sorry for my poor english...
|
|
|
|
|
I just tested your code and I got 1 record with a field count of 5. This example is really basic and is already covered *many* times in the unit tests.
Please provide me the source code if this is really not a joke...
using (LumenWorks.Framework.IO.Csv.CsvReader csv = new LumenWorks.Framework.IO.Csv.CsvReader(new System.IO.StringReader(@"1996,Jeep,Grand Cherokee,""MUST SELL!
air, moon roof, loaded"",4799.00"), false))
{
int count = csv.FieldCount;
while (csv.ReadNextRecord())
{
for (int i = 0; i < count; i++)
Console.Write(csv[i]);
Console.WriteLine();
}
}
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
-- modified at 22:21 Wednesday 12th July, 2006
|
|
|
|
|
Oh my mistake sorry. I thought that script automatic replace \n for " ". I am very sorry...
I am beginner...
BTW: My code isn't 1996,Jeep,Grand Cherokee,"MUST SELL! air, moon roof, loaded",4799.00
but is
1996,Jeep,Grand Cherokee,"MUST SELL! // here is \n
air, moon roof, loaded",4799.00
Anyway how replace automatic \n for " " or ""?
|
|
|
|
|
The parser will not do that for you. However, you can do as follow...
using (LumenWorks.Framework.IO.Csv.CsvReader csv = new LumenWorks.Framework.IO.Csv.CsvReader(new System.IO.StringReader("1996,Jeep,Grand Cherokee,\"MUST SELL!\nair, moon roof, loaded\",4799.00"), false))
{
int count = csv.FieldCount;
while (csv.ReadNextRecord())
{
for (int i = 0; i < count; i++)
Console.Write(csv[i].Replace("\n", " ");
Console.WriteLine();
}
}
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|
Hello,
first of all i like to thank you. The Csv-Reader is very comfortable.
The only thing i want you to ask is that if it is possible, to read the fieldcount again.
i. e. I have a csv-file with the first line only consisting of a white-space. so the fieldcount says 1. Next line i have all data i need, but i cant read them because i compare the fieldcount with greater or equal than one.
|
|
|
|
|
Humm, one thing you could do is call ReadLine() on your stream reader before passing it to the CsvReader. Would that solve your current problem ?
Thanks for your comment!
Sébastien
Intelligence shared is intelligence squared.
Homepage : http://sebastienlorion.com
|
|
|
|
|