Inconsistent file format

Question

0.00/5 (No votes)

See more:

Hi all,

1) In a little project I'm working on, I read a file, extract values, process them and output the final result. But this file format is not consistent.

i.e, The files are from a seabed survey, consisting of latitude, longitude, height, and GPS values. Sometimes there are 2 GPS's values and sometimes there is just one.

But all these files contain default variables that are used for calculations. How can I make this program consistent? Because, I have to change the swscanf function every single time, I have to read a different file format.

2) Another question :- As the calcultions involve calculations of mean I have to open the file many times, ie I have to read the file from first to last row many times,
1) After the first read- Calculate Total, count;
2) In the second read - Use the average value for some calculations on every line in the file and output.
3) After third do some calculations involving from values obtained from the previous step...

I read my file 3 times plus 1 (initially to get the start time and end time). Is there a way to get rid of this and make it simpler?

Note:
The file size varies from 320 KB to 2100 KB. The format:-

With 2 GPS
Type 1:
21:00:55.035,$GPGGA,210055.00,6102.0039902,N,00107.5920998,E,2,09,1.2,23.57,M,47.57,M,10.0,1011*4E,$GPGGA,210054,6102.00388,N,00107.62266,E,2,10,1.0,025.33,M,047.57,M,13,0777*76,$HEHDT,000.4,T*2B,0.67,
Type 2: 16:39:09 $GPGGA 163908 5838.966652 N 131.910585 E 2 9 0.8 67.4 M 0 M 8.4 1006*42 $GPGGA 163909 5838.94843 N 131.89993 E 1 10 0.8 22.77 M 46.33 M 0000*78 $HEHDT 292.3 T*25 293

With 1 GPS
Type 3:$GPGGA,140823.00,7434.7372191,N,05713.8933749,W,1,06,1.2,48.77,M,21.98,M,,*4E
Type 4:$GPGGA,154349.00,7434.7372530,N,05713.8941497,W,2,07,1.7,45.04,M,21.98,M,45.2,0081*5C,

Even the time format is different, For the first one I can output as it is but for the type 3 I got to format and then display as I use them for plotting graphs.

Posted 13-Aug-12 3:37am

Sumal.V

Updated 13-Aug-12 4:18am

v3

Add a Solution

Comments

[no name] 13-Aug-12 9:40am

Why don't you read the file once and do all the calculations once? Reading the file 4 times seems to be 4 times too many.

Sumal.V 13-Aug-12 9:49am

Hey sorry, I've updated my question. That is because after I read the first time, I calculate the average and in the second read I apply this average value to every line in the file n so on ...

[no name] 13-Aug-12 9:53am

And? Read the file once then you have all the data in memory... am I missing something here?

StianSandberg 13-Aug-12 9:50am

or 3 times too many.. :)

[no name] 13-Aug-12 9:53am

Was on purpose... :-)

StianSandberg 13-Aug-12 9:57am

lol

Sumal.V 13-Aug-12 9:54am

Nope sorry, I've updated my question.
B/w I must do that as I apply these calculated values to values in every single line and display them..

Mihai Vrinceanu 13-Aug-12 9:48am

Why don't you put a small raw file(binary or text) so we can see how are those values encoded. Also file size will be useful.

Richard MacCutchan 13-Aug-12 10:13am

Why read the file multiple times, it makes no sense? Read the file line by line just once, and as you read it extract the various fields depending on the structure of each line, and build this information into a table. Then you can traverse your table as many times as you like and produce the various different calculations that you need.

Sumal.V 13-Aug-12 10:28am

Do you mean use of Lists?

Richard MacCutchan 13-Aug-12 10:40am

Well that's one possibility, but it's really up to you to decide what class or structure will satisfy your needs. My point was really about getting the data into your program in a format that will allow you to do all the other processing and reporting in the best way.

Sumal.V 13-Aug-12 11:23am

Okay, I've never done anything involving lists and complex functions. WIll give that a try. Thanks

2 solutions

Solution 1

Quote:
Sometimes there are 2 GPS's values and sometimes there is just one.

Why not just read the whole file and interpret what's in it, not sure why your scanning of the file has to change.

Quote:
I read my file 3 times in total. Is there a way to get rid of this and make it simpler?

Yeah, do the calculations as you go, no need for you to open the file every time.

For example, calculate average weight (just example):
0. Read entry in file
1. Calculate: TotalWeight += EntryWeight;
2. Calculate: Entries += 1;
3. Calculate: AvgWeight = Entries / TotalWeight;
4. Go back to step 0 until you get to the end of file

...you just have to update your calculations to reflect where you are in a file, no need to read it over again.

Posted 13-Aug-12 4:13am

Albert Holguin

Comments

Sumal.V 13-Aug-12 10:27am

This is a sample code:
//1 read
if (csf.Open(fname, CFile::modeRead))
{
while (csf.ReadString(buffer) )

{
out = swscanf_s(buffer, _T("%[^,], %lf, %lf, %[^,], %lf, %[^,], %d, %d, %lf, %lf, %[^,], %lf, %[^,], , %[^,]"),
str1, 2046, &sTime, &db1, str4, 2046, &db2,str5, 2046, &in1, &in2, &db3, &db4, str6, 2046, &db5, str7, 2046,str10,2046);

double templat = int( lat1 / 100 ) + ( (lat1 / 100 - int( lat1 / 100 )) / 0.6 ); // latitude in degrees
double templon = int( lon1 / 100 ) + ( (lon1 / 100 - int( lon1 / 100 )) / 0.6 ); // longitude in degrees
//lat/lon calculations
sumLat = sumLat + templat;
count++; // int count = 0;
}
else
{
//error report
}

}
csf.Close();
}

meanLat = sumLat / count;

//Second Read
if (csf.Open(fname, CFile::modeRead))
{
while (csf.ReadString(buffer) )

{
out = swscanf_s(buffer, _T("%[^,], %lf, %lf, %[^,], %lf, %[^,], %d, %d, %lf, %lf, %[^,], %lf, %[^,], , %[^,]"),
str1, 2046, &sTime, &db1, str4, 2046, &db2,str5, 2046, &in1, &in2, &db3, &db4, str6, 2046, &db5, str7, 2046,str10,2046);
lattemp = db1;
lat = int( lattemp / 100 ) + ( (lattemp / 100 - int( lattemp / 100 )) / 0.6 );
latRad = lat * (ML::pi)/180;
deltaLat = latRad - meanLat ; //MeanLat is used for every single line.
.............

Albert Holguin 13-Aug-12 10:45am

Why don't you just read the values into a list of some sort (or vector)? Then you can do you delta calculations after the fact without reading from disk again.... alternatively, you can adjust your previously calculated deltas based on your changing mean (as you calculate it).

Sumal.V 14-Aug-12 8:28am

Yeah I'm trying to work with lists as explained in the solution 2. Thanks :)

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

JackDingler · Accepted Answer · 2012-08-13T06:43:00

Solution 2

Create a class that can contain all of the data for one line.

C++

class CSurveyLine
{
  public:
    CTime DateRecorded;
    CString DataType;
    Etc....

  public:
    virtual bool SetFieldData(LPCSTR szDateRecorded, LPCSTR szDataType, etc...);
    virtual bool SetFieldData(LPCSTR szDateRecorded, LPCSTR szDataType, etc...);

  protected:
    virtual bool ParseDateField(CTime & MemberDate, LPCSTR szDate);
    virtual bool SetDateRecorded(LPCSTR szDateRecorded) { return ParseDateField(DateRecorded, szDateRecorded); }
};

Since you have different input variants, you can create different SetFieldData() methods. If they have different arguments then you can overload the name, otherwise you may need to give them different names to differentiate them.
You may find it convenient to create methods to set the field data too, if you have to parse different field formats.
And one step further, you could make that conversion generic...

Create a vector to hold the data.

C++

std::vector<csurveyline> SurveyTable;</csurveyline>

Use push_back to add a new entry.

C++

{
   CSurveyLine SurveyLine;
   SurveyLine.SetFieldData(...);
   SurveyTable.push_back(SurveyLine);
}

Then walk through the table to calculate your stats,

C++

for (size_t Index = 0; Index < SurveyTable.size(); Index++)
{
   CSurveyLine & SurveyLine = SurveyTable[Index];

   /* Do Stuff to SurveyLine */
}

Posted 13-Aug-12 6:43am

JackDingler

Updated 13-Aug-12 6:48am

v2

Comments

Eugen Podsypalnikov 14-Aug-12 3:50am

Nice, without some keys "virtual" as well :)

Sumal.V 14-Aug-12 4:32am

Thanks so much for the wonderful example. It looks like an complicated exam question with lot of clues. Will give this a try :)

JackDingler 14-Aug-12 11:57am

This is stuff I do everyday...

It'll become easier for you with practice.

JackDingler 14-Aug-12 11:53am

Converting to a map wouldn't be too difficult. You could key of longitude and latitude, or any other criteria you like...

JackDingler 14-Aug-12 11:58am

Oh, perhaps you're pointing out the lack of a constructor and virtual destructor?

Gotta leave something as an exercise for the reader. :)

Sumal.V 14-Aug-12 8:52am

Hey Jack, the lines:
virtual bool SetFieldData(LPCSTR szDateRecorded, LPCSTR szDataType, etc...);
virtual bool SetFieldData(LPCSTR szDateRecorded, LPCSTR szDataType, etc...);
Did u mean to override the SetFieldData, for the rest of the variables?

Sumal.V 14-Aug-12 11:23am

And Can I also know what the SetFieldData () is for..

JackDingler 14-Aug-12 11:46am

Yes, they are intended to be overloaded. And yes, you would put in the list of variables.

Further, they should accept the results of your sscanf or whatever method you use to parse the lines.

Now if you have really long variable lists, then you could put them in an array of strings, then have a parse function to process that array.

JackDingler 14-Aug-12 11:52am

Later you could get fancier and overload different line types and have the vector contain a list of base pointers...

Sumal.V 15-Aug-12 6:35am

Thanks for that. May be I'm asking for too much,but could you please give me an example code, with some values, for a better understanding. I browsed the internet and found few examples, but they either look complicated or not relevant. Thanks again

I do have a book with a case study on lists but its starts with simple variables and by the time I could get to the middle of the program I had lost track of what I was doing.

JackDingler 15-Aug-12 10:40am

I thought that's what I gave you.

The line std::vector<csurveyline> SurveyTable; defines your list.

The class CSurveyLine holds the variables that maintain the values of each item, found in the lines in your input file.

If you want a complete working program, I charge money for that. As I mentioned I do this for a living. I don't do projects for free.

Sumal.V 16-Aug-12 9:25am

I don't understand why you have chosen "CString DataType" variable ; And what is the main function of SetFieldData()?
Can I not just use scanf function to read the file and add the variables into the SurveyTable..
And I want to know why the ParseDateField, setDateRecorded are for?
I'm surely not asking for you to make my project for free. You got my 5 for the answer.
But hey, how much do u charge ? ;)

JackDingler 16-Aug-12 9:36am

I don't know what variable names your going to invent to hold your scanned data. So I made one up. I figured you could invent your own names.

As to the methods to parse dates... You said the dates come in different formats. Rather than write the same block of code a dozen times or so, one for each field, it's helpful to write a generic function that can parse a date field in one place, and then call that function for each date field.

As to rates... if you have to ask....

Sumal.V 16-Aug-12 9:58am

Nope I'm not doing that mistake of asking ;) But thanks so much :)

JackDingler 16-Aug-12 11:52am

Are you with me so far on the explanation though?

This process will get easier for you over time. The best way to get comfortable in writing code, is to write lots of it.

Guidelines that I like to try to follow are...
A Class manages just one type of resource.
A Method performs only one function.
A Variable represents only one data element.

So a class may manage a handle for a file, a single record, or a single table, etc...
A class can contain a collection of elements that serve different roles, but in this capacity, the class should only have logic designed to manage it as a collection. If you have an class representing an automotive engine, then it shouldn't have logic that modifies the transmission internals, or logic that manages the butterfly valve in the carburetor. but it might send signals to the transmission, or control the carburetor's throttle assembly.

A method likewise, should just do one function. It will call other methods that do other functions, but it's scope should be limited to accomplishing one defined task.

For variables, don't reuse them for multiple purposes. Don't declare a variable for longitude, then later in the function use it for latitude...

JackDingler 16-Aug-12 9:40am

I simply blocked out a skeleton for your code that shows the essential elements. The idea was to show you how to store the data in memory for further processing, so you're not scanning the file for every pass. It's up to you to fill in the rest.

Sumal.V 20-Aug-12 5:35am

I finally got the program working. Took a while for me to understand, I'm a very slow learner. But Thank you :)

Sumal.V 15-Aug-12 10:04am

Hi that's alright. Just found a beginner level article on Lists :)