C / C++ / MFC

19-May-11 0:37

For merging together some recording data stored in CSV files I have used the class CtextFile (by Johan Rosengren via the Codeproject site) to read in and write out the processed files. I also used class CNewStringArray (which is a modified version of CStringArray by Anders M Eriksson, again via the Codeproject)
I use the MFC function AfxExtractSubString() to extract the various fields of the CSV record and CnewStringArray variables to hold and process the data.
Now I have a problem where the size of some of the files to be processed may be in the order of 250 to 700Gb.
I have never used files larger that about 10Mb, but will I have problems with files these sizes and do I need to consider new methods of processing these very large files.
Any comments and advice please.

Re: MFC Program and Very Large Text Files [modified]

Maximilien19-May-11 1:05

Maximilien

19-May-11 1:05

I'm not an expert on that topic, but I think you have to look at Memory Map Files.

That will let you navigate the file without having to load the whole file in memory.

Watched code never compiles.

federico.strati19-May-11 1:26

federico.strati

19-May-11 1:26

Yes, you definitely are better off using memory mapped files,
namely, you should use "CreateFileMapping", "MapViewOfFile", "UnmapViewOfFile" and the likes.

It is a bit outdated, but the book from Jeffrey Richter
"Programming Applications for Microsoft Windows" has a good introduction to such an API.

Maybe, looking into MSDN with these pointers will lead you to the correct API to use
for your Operating System version.

Cheers
Federico

yeap, it is still the same API in latest versions of Windows, just checked on MSDN:
http://msdn.microsoft.com/en-us/library/aa366537(v=VS.85).aspx

modified on Thursday, May 19, 2011 7:51 AM

Andy20219-May-11 22:04

19-May-11 22:04

Thanks Federico for your post.

I have used these API's, but for large auto generated data structures ~ 50k bytes.

I did look at he link you gave and the following concerns me:-

If the file mapping object is backed by the operating system paging file (the hfile parameter is INVALID_HANDLE_VALUE), specifies that when a view of the file is mapped into a process address space, the entire range of pages is committed rather than reserved. The system must have enough committable pages to hold the entire mapping. Otherwise, CreateFileMapping fails.

With files sizes of ~ 500GB will these APIs work?

Andy.

federico.strati20-May-11 23:25

federico.strati

20-May-11 23:25

Hello Andy,

what follows is extracted from the book from Jeffrey Richter
"Programming Applications for Microsoft Windows"

It is written for a 32bit O.S. so be careful to adapt it in case you work on Win 7

--- start ---

Processing a Big File Using Memory-Mapped Files

In an earlier section, I said I would tell you how to map a 16-EB file into a small address space.
Well, you can't. Instead, you must map a view of the file that contains only a small portion of the file's data.
You should start by mapping a view of the very beginning of the file.

When you've finished accessing the first view of the file, you can unmap it and then map a new view
starting at an offset deeper within the file. You'll need to repeat this process until you access the complete file. This certainly makes dealing with large memory-mapped files less convenient, but fortunately most files are small enough that this problem doesn't usually come up.

Let's look at an example using an 8-GB file and a 32-bit address space.
Here is a routine that counts all the 0 bytes in a binary data file in several steps:

__int64 Count0s(void) {

// Views must always start on a multiple
// of the allocation granularity

SYSTEM_INFO sinf;
GetSystemInfo(&sinf);

// Open the data file.

HANDLE hFile = CreateFile("C:\\HugeFile.Big", GENERIC_READ,
FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, NULL);

// Create the file-mapping object.
HANDLE hFileMapping = CreateFileMapping(hFile, NULL,
PAGE_READONLY, 0, 0, NULL);
DWORD dwFileSizeHigh;

__int64 qwFileSize = GetFileSize(hFile, &dwFileSizeHigh);
qwFileSize += (((__int64) dwFileSizeHigh) << 32);

// We no longer need access to the file object's handle.
CloseHandle(hFile);

__int64 qwFileOffset = 0, qwNumOf0s = 0;
while (qwFileSize > 0) {

  // Determine the number of bytes to be mapped in this view
  DWORD dwBytesInBlock = sinf.dwAllocationGranularity;
  if (qwFileSize < sinf.dwAllocationGranularity)
    dwBytesInBlock = (DWORD) qwFileSize;
  PBYTE pbFile = (PBYTE) MapViewOfFile(hFileMapping, FILE_MAP_READ,
                            (DWORD) (qwFileOffset >> 32), // Starting byte
                            (DWORD) (qwFileOffset & 0xFFFFFFFF), // in file
                             dwBytesInBlock); // # of bytes to map
  // Count the number of Js in this block.
  for (DWORD dwByte = 0; dwByte < dwBytesInBlock; dwByte++) {
    if (pbFile[dwByte] == 0)
    qwNumOf0s++;
  }
  // Unmap the view; we don't want multiple views
  // in our address space.
  UnmapViewOfFile(pbFile);
  // Skip to the next set of bytes in the file.
  qwFileOffset += dwBytesInBlock;
  qwFileSize -= dwBytesInBlock;
}

CloseHandle(hFileMapping);
return(qwNumOf0s);
}

This algorithm maps views of 64 KB (the allocation granularity size) or less.
Also, remember that MapViewOfFile requires that
the file offset parameters be a multiple of the allocation granularity size.

As each view is mapped into the address space, the
scanning for zeros continues. After each 64-KB chunk of the file has been mapped and scanned,
it's time to tidy up by closing the file-mapping object.

--- end ---

Cheers

Federico

Andy20225-May-11 21:33

25-May-11 21:33

Thanks Federico for the information. The requirement has gone away, but I thought that it would be good to do this task (a worked example) should I ever need to revist this problem again.

Just one follow up question you suggest 64 KB (the allocation granularity size)?

Is this from expericence? And the best value.

Andy

federico.strati26-May-11 2:14

federico.strati

26-May-11 2:14

Hi,

the allocation granularity size, if you look at the code, is system dependent:

SYSTEM_INFO sinf;
GetSystemInfo(&sinf);

[... snipped ...].

  // Determine the number of bytes to be mapped in this view
  DWORD dwBytesInBlock = sinf.dwAllocationGranularity;

I don't know the values for recent Windows OS's, I cited 64Kb just to say a size,
you may just get them from the API (GetSystemInfo).

As far as what would be the best value, I would say that you should map in multiples of
such allocation size and other considerations (as available memory for the system and/or the single
process) come into play.

You may have to experiment a bit to find the best for your requirements.

Cheers

Federico

Chandrasekharan P19-May-11 1:22

19-May-11 1:22

250 to 700GB of data or is it a typo error???? if its 250 to 700 Gb of data, where are your data stored??

jschell19-May-11 8:21

jschell

19-May-11 8:21

Andy202 wrote:
For merging together some recording data stored in CSV files I have used the
class CtextFile (by Johan Rosengren via the Codeproject site) to read in and
write out the processed files.

Presumably you are doing something more than just creating one file out of two.

Consequently your design with much larger files, needs to take into account specifically what you need to do.

And if it was me I would also look into the business requirement that requires that the output be another file. Pretty pointless to parse a large file and then parse it again.

Andy20219-May-11 22:10

19-May-11 22:10

Thanks for your post jschell.

Two input files to generate one output file.

Input1 sampled at 50 msec and Input2 sampled at 200 msec, so using timing information merge the two sets of data into one output file; interporlate as required.

I can do what is required if the files were as follows:-

Input1 = 1 Mb; Input2 = 2Mb and Output = 5Mb.

Andy

Embedding Excel in a dialog

jschell20-May-11 10:13

jschell

20-May-11 10:13

Andy202 wrote:
Input1 sampled at 50 msec and Input2 sampled at 200 msec, so using timing information merge the two sets of data into one output file; interporlate as required.

So you have one 'block' (perhaps one line or longer) from file 1 that exists every 50msec. Call this X.
You have another 'block' from file 2 that exists every 200msec. Call this Y.

Thus you have 4 (200msec/50msect) X entries before each Y entry.

Steps
1. Build a buffered reader for each file. Buffered in the case reads N 'block' entries and allows it to read M more on request. The timestamp is exposed (parsed from the block.)
2. Read X via the buffer where N is 200.
3. Read Y via the buffer where N is also 200 (could be less as well.)
4. Now 2 will have data that fits into 3. Because you have read enough to overlap.
5. Step 4 represents a starting point. Basically the two buffers will mostly be off by 4.
6. You can't assume it will always be offset by 4 so continue to compare the two timestamps as you read.
7. On start up you need to sync the two buffer reads, since one file might have a much different start time from the other.

Performance impacts
- Play with the stream buffer sizes (actual file read versus buffered readers above.)
- Profile it with a tool for some large and real files, say 50 meg at least.

efraimyy18-May-11 22:49

18-May-11 22:49

I would like to create a dialog box which contains an embedded Excel workbook. (Excel is installed on the computer, of course.) MS provides example code showing how to do something like this with a Document/View based app, but not in a dialog.

(I saw from searching this forum that there was a discussion about "Embedded Excel", but when I try to access it, I get an empty list with the message "Unable to load messages due to high load.")

Chandrasekharan P19-May-11 1:35

19-May-11 1:35

You can go to the project tab and select Add to Project Sub menu. This menu has another sub menu. select the last option. Components and control
When you select that a windows dialog will pop up with 2 folders. Select the first one. Search for Microsoft Office SpreadSheet. Select that and click OK.
The spreadsheet will be embedded in your dialog.

efraimyy19-May-11 2:58

19-May-11 2:58

I'm not sure what you mean by the "Project Tab" and "Add To Project". If I right-click on the solution, I get an item "Add New Project" and under that one of the possibilities is Visual C#/Microsoft Office/Excel Workbook, but I don't see how this will help me. Won't it just create a whole new project unrelated to my current one?

efraimyy19-May-11 3:00

19-May-11 3:00

I should have mentioned that I am using VS2008

Chandrasekharan P19-May-11 22:04

19-May-11 22:04

Forgot to mention, In the menu bar. There should be an option called Project. You should select that.

efraimyy21-May-11 21:40

21-May-11 21:40

I still don't see this.

In VS2008, all I have under the "Project" menu or the right-click menu on the project is "Add New Item" and "Components and controls" is not a selection from the dialog I get from that.

Chandrasekharan P22-May-11 18:42