|
Hello
I use a Text file with following structure with about 4 millions lines and the separator is a ‘\t’ (tabulation).
"Date" "Time" "Signal"
20000103 1658 351
20000103 1659 352
20000103 1700 350
20000103 1701 352
20000103 1702 355
20000104 0900 354
20000104 0901 352
20000104 0902 350
I would like to copy the columns Date, Time and Signal into a STL vectors containers.
But I do not want to copy all the file. For example a just need to start at line 250000 and stop at line 3000000. The criteria to define the line where to start and finish copying is the column Date and Time
I currently use CFile and I copy the whole file into a buffer and I close the file.
This is very fast, less than 1 second.
Then I use the function strtok to read the buffer string by string.
And I count the lines until I find the date and time thatI want.
Unfortunately my method takes 57 seconds to cross all the file just to pick up the start and end line numbers.
So I am wondering if is there is a possibility to read the buffer or a file column by column?
Does someone knows or heard a method to swiftly read a string buffer or a file?
I thank you very much If you can advice solutions that help to speed up this process.
For information, I tested the function Tokenize of CString but is is very slow.
I also tested CStdioFile and ReadString function but it is also very slow.
|
|
|
|
|
Based on the sample data you posted the file format is "fixed length" therefore simple math with give you a location to the exact record you want to access.
led mike
|
|
|
|
|
Hi,
simple math with give you a location to the exact record
I am not sure to understand your solution.
Could you please clarify ?
|
|
|
|
|
Arris7 wrote: But I do not want to copy all the file. For example a just need to start at line 250000 and stop at line 3000000.
Use the CFile::Seek() method to go to offset 250000*length_of_line. Without testing, this is 4500000.
Arris7 wrote: I currently use CFile and I copy the whole file into a buffer and I close the file.
I would consider using CStdioFile with CMemFile. That way your file is processed in memory, rather than on disk, and you can utilize line-parsing functions.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Thnks.
CFile::Seek() doesn't help because actually I do not know at which line to start and finish. I just know a starting and ending date and time. So I need to read the String and to compare it with a variable.
Do you have a code sample on how to use CStdioFile with CMemFile together? Which functions are used for parsing? I just saw CStdioFile:: ReadString() which cannot be used with CMemFile.
|
|
|
|
|
Arris7 wrote: actually I do not know at which line to start and finish. I just know a starting and ending date and time. So I need to read the String and to compare it with a variable.
That contradicts your first post:
Arris7 wrote: For example a just need to start at line 250000 and stop at line 3000000
So which is it?
led mike
|
|
|
|
|
Sorry I was not clear in my first post. Actually I just know the date and the time where to start.
for example the starting date and time can be located at the line 250000 and I have to find the line.
|
|
|
|
|
Are the dates in order?
If the file starts with the earliest date and finishes with the latest date, then use the following method:
1) Fseek to the middle of the file
2) If the data is larger fseek to one quter of the way through the file, if smaller, fseek to 3 quarters of the way though the file.
3) Repeat.
Like guessing a number, if you ask someone select a number between 0 and 15, the quickest way to find the number is to ask,
Is it less than 8
If yes then ask is it less than 4
If no than ask is it less than 6
I'm sure you get the idea. Its a common technique.
|
|
|
|
|
thanks, Yes dates are in order.
the method you suggest is a binary search. I think it is the best solution for my problem.
|
|
|
|
|
Arris7 wrote: CFile::Seek() doesn't help because actually I do not know at which line to start and finish. I just know a starting and ending date and time. So I need to read the String and to compare it with a variable.
Fair enough, but you can still do it via a simple calculation, rather than using strtok() to find each line. strtok() is slowing you down as it has to examine each character to find the one you want. Since each line of the file is 18-19 characters in length, just compare the first 13 of those.
Something like:
void main( void )
{
char *szBuffer = "20000103\t1658\t351\n"
"20000103\t1659\t352\n"
"20000103\t1700\t350\n"
"20000103\t1701\t352\n"
"20000103\t1702\t355\n"
"20000104\t0900\t354\n"
"20000104\t0901\t352\n"
"20000104\t0902\t350\n";
char *p = szBuffer;
while (p != NULL && *p != '\0')
{
if (strncmp(p, "20000104\t0900", 13) == 0)
{
printf("Found it!\n");
break;;
}
p += 18;
}
}
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
many thanks
It sounds great. I gonna try it and let you know the results.
|
|
|
|
|
For very large files (millions of lines) this will still take a while. Seeing how the file is sorted by date and time I would use your original idea of using CFile::Seek to do a binary search of the file. A binary search will be slower if the required data is right at the start of the file, but a heck of a lot faster if the data is anywhere else.
You may be right
I may be crazy
-- Billy Joel --
Within you lies the power for good, use it!!!
|
|
|
|
|
PJ Arends wrote: Seeing how the file is sorted by date and time...
Was that a guarantee? If so, then a binary search via Seek() is the way to go.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
If it is not sorted a sorted solution would likely be optimal for 4 million records. Either sorting the orginal file or creating and index file or memory based index, perhaps a Database should be considered.
led mike
|
|
|
|
|
Many thanks for your solutions.
I think that your idea of using strncmp(p, "20000104\t0900", 13) == 0) in a binary search is a good answer.
I'll try it right now.
thnks again
|
|
|
|
|
Thnks for this solution.
yes the file is sorted by date and time so Binary search is a good answer to my question.
thnks again
|
|
|
|
|
I have a two-bay SCSI case, each with separate Target ID.
Running code that utilized SCSI_PASS_THROUGH_DIRECT reads works fine when only one disk is seated in the drive, but when both disks are seated, I'm getting errors on the reads. Any ideas?
Thanks,
Jim
|
|
|
|
|
hi i am using wmi concept to get the adapter details. i passed query to win32_networkadapterconfiguration as hres = m_pSvc->ExecQuery( bstr_t("WQL"),
bstr_t("SELECT * FROM Win32_NetworkAdapterConfiguration WHERE SettingID = AdapterIndex"),
WBEM_FLAG_FORWARD_ONLY | WBEM_FLAG_RETURN_IMMEDIATELY,
NULL,
&m_pEnumerator);
the value fo adapterindex is 9; but its returning as hres failed. how to pass the value of the variable n query
|
|
|
|
|
How about something like:
char szQuery[128];
sprintf(szQuery, "SELECT * FROM Win32_NetworkAdapterConfiguration WHERE SettingID = %d", AdapterIndex);
ExecQuery(bstr_t("WQL"), szQuery, WBEM_FLAG_FORWARD_ONLY | WBEM_FLAG_RETURN_IMMEDIATELY, NULL, &m_pEnumerator);
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
yes i got it. but how to get ipaddress and subnet from wmi. i am getting as 1and 2 respectivly
|
|
|
|
|
saisp wrote: but how to get ipaddress and subnet from wmi.
I use:
IWbemServices *pService = NULL;
IEnumWbemClassObject *pEnum = NULL;
...
HRESULT hr = pService->CreateInstanceEnum(_bstr_t("Win32_NetworkAdapterConfiguration"),
WBEM_FLAG_RETURN_IMMEDIATELY | WBEM_FLAG_FORWARD_ONLY,
NULL, &pEnum);
if (WBEM_S_NO_ERROR == hr)
{
IWbemClassObject *pClass = NULL;
ULONG ulReturned;
hr = pEnum->Next(WBEM_INFINITE, 1, &pClass, &ulReturned);
if (WBEM_S_NO_ERROR == hr)
{
_variant_t v;
hr = pClass->Get(L"IPAddress", 0, &v, NULL, NULL);
if (VT_NULL != v.vt)
{
LONG lBound;
BSTR Value;
SafeArrayGetUBound(V_ARRAY(&v), 1, &lBound);
while (lBound >= 0)
{
SafeArrayGetElement(V_ARRAY(&v), &lBound, &Value);
TRACE("IPAddress[%u] = %S\n", lBound, Value);
lBound--;
}
}
hr = pClass->Get(L"IPSubnet", 0, &v, NULL, NULL);
if (VT_NULL != v.vt)
{
LONG lBound;
BSTR Value;
SafeArrayGetUBound(V_ARRAY(&v), 1, &lBound);
while (lBound >= 0)
{
SafeArrayGetElement(V_ARRAY(&v), &lBound, &Value);
TRACE("IPSubnet[%u] = %S\n", lBound, Value);
lBound--;
}
}
}
}
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
hmm but how to know the ipaddress value. i have to print it in a file. sorry for disturbing you. its working fine but i cant see the value (192.168.168.168)
|
|
|
|
|
saisp wrote: hmm but how to know the ipaddress value.
That's what my code snippet was for. 
saisp wrote: its working fine but i cant see the value...
These two statements contradict each other.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
|
Means no error but i cant get the value. how to print the ip address value. you have used TRACE. i dont know how to get ip and subnet value from this. as im new to this i got many doubts.
|
|
|
|
|
saisp wrote: you have used TRACE. i dont know how to get ip and subnet value from this.
Are you not familar with printf()? It's contained in Value.
"Approved Workmen Are Not Ashamed" - 2 Timothy 2:15
"Judge not by the eye but by the heart." - Native American Proverb
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
