Re: Parsing text files - C# Discussion Boards

4-Feb-09 6:59

I need to parse Log files with different operations that will require different grammars as each operation is organized differently. I have been using
char[] delimiterChars = { ',' 'WHATEVER' }; to detemine what gets parsed. the problem with that is, it does not seem to discriminate which parameter it chooses and when. Is there anyway to make it a little more automated instead of just creating a whole bunch of delimiter chars and then calling them in order at which they wouldd appear?

musefan4-Feb-09 7:10

musefan

4-Feb-09 7:10

I have no idea as to what you mean... please examplify

My opinion is... If someone has already posted an answer, dont post the SAME answer

PIEBALDconsult4-Feb-09 7:11

4-Feb-09 7:11

Can you provide a small sample of the various types of data?

alwaysthinking4-Feb-09 7:25

4-Feb-09 7:25

directly from the log file:

E 12:28:45 Line 1 : Dispense
tip 1 : dispense 0.00nl 1, 1 Deep Well, landscape [1,1]
10.00µl ">> Water << 23" Fixed <3 - 50µl> Single

E 12:28:47 Line 2 : Aspirate
tip 1 : aspirate 60.00µl 1, 1 Deep Well, landscape [1,2]
10.00µl "Water" Fixed <3 - 50µl> Multi

E 12:28:45 Line 1 : Dispense
tip 1 : dispense 0.00nl 1, 1 Deep Well, landscape [1,1]
10.00µl ">> Water << 23" Fixed <3 - 50µl> Single

E 12:28:47 Line 2 : Aspirate
tip 1 : aspirate 60.00µl 1, 1 Deep Well, landscape [1,2]
10.00µl "Water" Fixed <3 - 50µl> Multi

alwaysthinking4-Feb-09 7:36

4-Feb-09 7:36

scratch the last reply...

these three commands are the bread and butter of the operations. most are just combination of these 3...

E 12:28:45 Line 1 : Dispense
tip 1 : dispense 0.00nl 1, 1 Deep Well, landscape [1,1]
10.00µl ">> Water << 23" Fixed <3 - 50µl> Single

E 12:28:47 Line 2 : Aspirate
tip 1 : aspirate 60.00µl 1, 1 Deep Well, landscape [1,2]
10.00µl "Water" Fixed <3 - 50µl> Multi

E 12:28:57 Line 3 : Wash Tips
wash (2ml + 1ml), grid 26 and grid 26

now they look similar but each hasa different set of information needed to cut out or set as variables. I do plan on calling a "dispense", "asperate" and "Wash Tips" grammar structure when needed. if i tell it to parse at the comma's, i will get data that i dont want. I could remember each value of the word, and set that to a variable like word[1]=volume and word[5]=fluid type but i dont really want to do that. is there any way to make it a little more automated so that it knows the sequence to skip over bits not biased on charactor length as that will change as well.

Alan N4-Feb-09 7:46

Alan N

4-Feb-09 7:46

Hi,
I've reviewed your posts on the topic of parsing these instrument log files. What I haven't got from any of your questions is what information you want to extract.

So, for example, in the log just posted what output do you want?

Alan.

PIEBALDconsult4-Feb-09 8:07

4-Feb-09 8:07

Oh, this is the same thing I've seen posted before.

I'd use a Regular Expression. Not having much else to do today, I think I'll experiment a little... I'll be back.

PIEBALDconsult4-Feb-09 10:18

4-Feb-09 10:18

So far I've created this Regex:

System.Text.RegularExpressions.Regex reg = new System.Text.RegularExpressions.Regex
(
    @"(?'Type'\S)\s(?'Time'\d\d:\d\d:\d\d)\sLine\s*(?'Line'[^:]*)\s:\s(?'Operation'[^\r]*)\r\n(?'Data'([^\r]*\r){1,3})"
,
    System.Text.RegularExpressions.RegexOptions.Compiled |
    System.Text.RegularExpressions.RegexOptions.Singleline |
    System.Text.RegularExpressions.RegexOptions.CultureInvariant
) ;

It matches the three log entries you provided and provides the named groups:

E 12:28:45 Line      1 : Dispense
- --------           -   --------
|       |            |         |
|       |            |         Operation
|       |            Line
|       Time
Type
 
and then the following two or three lines go into Data

You can then use the Operation to determine further processing of the Data, perhaps each operation would have its own Regex to parse its data.

alwaysthinking9-Feb-09 8:35

9-Feb-09 8:35

Sorry for the late reply I thank you for the help. So far you have pretty much hit the pin on the head. The problem here is... I dont know what you mean by regular expressions and how I would build them for each operation. Ill show you the desired data and maybe give some pointers on how to create one for this operation and then Ill be able to use that to create more. This is the dispense operation.

tip 1 : dispense 0.00nl 1, 1 Deep Well, landscape [1,1]
----- | | | |
| | | | |
a b c d e

10.00µl ">> Water << 23" Fixed <3 - 50µl> Single
----- ----
| |
| |
f g

Also, my plan right now is to take each word that is parsed in the array, assign the desired arrays with the proper variables (time=word[8]) and work it that way. Im not sure if that is the most effective way to do it, im sure it will work but maybe not the best.

alwaysthinking9-Feb-09 8:36

9-Feb-09 8:36

Ignore the last post forgot the change HTML setting

Sorry for the late reply I thank you for the help. So far you have pretty much hit the pin on the head. The problem here is... I dont know what you mean by regular expressions and how I would build them for each operation. Ill show you the desired data and maybe give some pointers on how to create one for this operation and then Ill be able to use that to create more. This is the dispense operation.

tip 1 : dispense 0.00nl 1, 1 Deep Well, landscape [1,1]
----- | | | |
| | | | |
a b c d e

10.00µl ">> Water << 23" Fixed <3 - 50µl> Single
----- ----
| |
| |
f g

Also, my plan right now is to take each word that is parsed in the array, assign the desired arrays with the proper variables (time=word[8]) and work it that way. Im not sure if that is the most effective way to do it, im sure it will work but maybe not the best.

PIEBALDconsult9-Feb-09 11:59

9-Feb-09 11:59

You could put the stuff in a "code block" to have it appear properly in the message.

alwaysthinking10-Feb-09 10:01

10-Feb-09 10:01

Could you explain what each section of this line means?

@"(?'Type'\S)\s(?'Time'\d\d:\d\d:\d\d)\sLine\s*(?'Line'[^:]*)\s:\s(?'Operation'[^\r]*)\r\n(?'Data'([^\r]*\r){1,3})"

Im trying to find a legend for all the possibilities but i dont even know where to start...

PIEBALDconsult10-Feb-09 12:08

10-Feb-09 12:08

(?'Type'\S) There must be a non-whitespace character, capture it in group Type

\s Then there must be one whitespace character

(?'Time'\d\d:\d\d:\d\d) Then there must be a sequence of six decimal digits separated by colons, capture it in group Time

\sLine\s* Then there must be a whitespace character, the characters L i n e, and zero or more whitespace characters.

(?'Line'[^:]*) Then capture everything up to (but not including) the next colon in group Line

\s:\s Then there must be a whitespace character, a colon, and another whitespace character.

(?'Operation'[^\r]*) Then capture everything up to (but not including) the next carriage return in group Operation

\r\n There must then be a carriage return and a linefeed.

(?'Data'([^\r]*\r){1,3}) And finally capture everything from there through the next few carriage returns, there must be at least one and at most three carriage returns. Capture this into the Data group.

alwaysthinking17-Feb-09 12:22

17-Feb-09 12:22

thanks this is working perfectly except for one thing. under operation, serveral of the operations have spaces between them. I cant seem to find an expression that says "include everything here, and if it has a space include that too" into the operations group. is that even possible?

PIEBALDconsult17-Feb-09 12:28

17-Feb-09 12:28

The Operation group should contain everything up to the next carriage return. Doesn't it? Culd you post an example of a problematic string?

alwaysthinking17-Feb-09 12:32

17-Feb-09 12:32

example here:
E 10:45:05 Line 5 : Wash Tips

alwaysthinking17-Feb-09 12:53

17-Feb-09 12:53

so i was playing around and it seems it will include if there are spaces as it is. The problem seems to be from the data portion.

PIEBALDconsult17-Feb-09 13:21

17-Feb-09 13:21

OK, so what's the problem?

alwaysthinking17-Feb-09 13:32

17-Feb-09 13:32

these are the two different operations i have:

E 10:45:03 Line 4 : Dispense
tip 1 : dispense 20.56µl 6, 3 Deep Well, landscape [1,1]
10.00µl ">> Water << 23" Fixed <3 - 50µl> Single
and

E 10:45:05 Line 5 : Wash Tips
wash (2ml + 1ml), grid 20 and grid 20

the first line is great works fine, but with wash tips it doesnt seem to follow the pattern. when i replace the 2nd line in wash tips with the 2nd and 3rd line of dispense everything is kosher.

another problem im having is, id like to have the file read line by line but i cant seem to get that done too.

alwaysthinking17-Feb-09 13:42