|
Please use the <pre> tags (use the 'code block' button)to post code snippets.
What do you mean with 'wrong result'? Please elaborate.
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
For merging together some recording data stored in CSV files I have used the class CtextFile (by Johan Rosengren via the Codeproject site) to read in and write out the processed files. I also used class CNewStringArray (which is a modified version of CStringArray by Anders M Eriksson, again via the Codeproject)
I use the MFC function AfxExtractSubString() to extract the various fields of the CSV record and CnewStringArray variables to hold and process the data.
Now I have a problem where the size of some of the files to be processed may be in the order of 250 to 700Gb.
I have never used files larger that about 10Mb, but will I have problems with files these sizes and do I need to consider new methods of processing these very large files.
Any comments and advice please.
|
|
|
|
|
I'm not an expert on that topic, but I think you have to look at Memory Map Files.
That will let you navigate the file without having to load the whole file in memory.
Watched code never compiles.
|
|
|
|
|
Yes, you definitely are better off using memory mapped files,
namely, you should use "CreateFileMapping", "MapViewOfFile", "UnmapViewOfFile" and the likes.
It is a bit outdated, but the book from Jeffrey Richter
"Programming Applications for Microsoft Windows" has a good introduction to such an API.
Maybe, looking into MSDN with these pointers will lead you to the correct API to use
for your Operating System version.
Cheers
Federico
yeap, it is still the same API in latest versions of Windows, just checked on MSDN:
http://msdn.microsoft.com/en-us/library/aa366537(v=VS.85).aspx
modified on Thursday, May 19, 2011 7:51 AM
|
|
|
|
|
Thanks Federico for your post.
I have used these API's, but for large auto generated data structures ~ 50k bytes.
I did look at he link you gave and the following concerns me:-
If the file mapping object is backed by the operating system paging file (the hfile parameter is INVALID_HANDLE_VALUE), specifies that when a view of the file is mapped into a process address space, the entire range of pages is committed rather than reserved. The system must have enough committable pages to hold the entire mapping. Otherwise, CreateFileMapping fails.
With files sizes of ~ 500GB will these APIs work?
Andy.
|
|
|
|
|
what follows is extracted from the book from Jeffrey Richter
"Programming Applications for Microsoft Windows"
It is written for a 32bit O.S. so be careful to adapt it in case you work on Win 7
--- start ---
Processing a Big File Using Memory-Mapped Files
In an earlier section, I said I would tell you how to map a 16-EB file into a small address space.
Well, you can't. Instead, you must map a view of the file that contains only a small portion of the file's data.
You should start by mapping a view of the very beginning of the file.
When you've finished accessing the first view of the file, you can unmap it and then map a new view
starting at an offset deeper within the file. You'll need to repeat this process until you access the complete file. This certainly makes dealing with large memory-mapped files less convenient, but fortunately most files are small enough that this problem doesn't usually come up.
Let's look at an example using an 8-GB file and a 32-bit address space.
Here is a routine that counts all the 0 bytes in a binary data file in several steps:
__int64 Count0s(void) {
SYSTEM_INFO sinf;
GetSystemInfo(&sinf);
HANDLE hFile = CreateFile("C:\\HugeFile.Big", GENERIC_READ,
FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_FLAG_SEQUENTIAL_SCAN, NULL);
HANDLE hFileMapping = CreateFileMapping(hFile, NULL,
PAGE_READONLY, 0, 0, NULL);
DWORD dwFileSizeHigh;
__int64 qwFileSize = GetFileSize(hFile, &dwFileSizeHigh);
qwFileSize += (((__int64) dwFileSizeHigh) << 32);
CloseHandle(hFile);
__int64 qwFileOffset = 0, qwNumOf0s = 0;
while (qwFileSize > 0) {
DWORD dwBytesInBlock = sinf.dwAllocationGranularity;
if (qwFileSize < sinf.dwAllocationGranularity)
dwBytesInBlock = (DWORD) qwFileSize;
PBYTE pbFile = (PBYTE) MapViewOfFile(hFileMapping, FILE_MAP_READ,
(DWORD) (qwFileOffset >> 32),
(DWORD) (qwFileOffset & 0xFFFFFFFF),
dwBytesInBlock);
for (DWORD dwByte = 0; dwByte < dwBytesInBlock; dwByte++) {
if (pbFile[dwByte] == 0)
qwNumOf0s++;
}
UnmapViewOfFile(pbFile);
qwFileOffset += dwBytesInBlock;
qwFileSize -= dwBytesInBlock;
}
CloseHandle(hFileMapping);
return(qwNumOf0s);
}
This algorithm maps views of 64 KB (the allocation granularity size) or less.
Also, remember that MapViewOfFile requires that
the file offset parameters be a multiple of the allocation granularity size.
As each view is mapped into the address space, the
scanning for zeros continues. After each 64-KB chunk of the file has been mapped and scanned,
it's time to tidy up by closing the file-mapping object.
--- end ---
Cheers
Federico
|
|
|
|
|
Thanks Federico for the information. The requirement has gone away, but I thought that it would be good to do this task (a worked example) should I ever need to revist this problem again.
Just one follow up question you suggest 64 KB (the allocation granularity size)?
Is this from expericence? And the best value.
Andy
|
|
|
|
|
Hi,
the allocation granularity size, if you look at the code, is system dependent:
SYSTEM_INFO sinf;
GetSystemInfo(&sinf);
[... snipped ...].
DWORD dwBytesInBlock = sinf.dwAllocationGranularity;
I don't know the values for recent Windows OS's, I cited 64Kb just to say a size,
you may just get them from the API (GetSystemInfo).
As far as what would be the best value, I would say that you should map in multiples of
such allocation size and other considerations (as available memory for the system and/or the single
process) come into play.
You may have to experiment a bit to find the best for your requirements.
Cheers
Federico
|
|
|
|
|
250 to 700GB of data or is it a typo error???? if its 250 to 700 Gb of data, where are your data stored??
|
|
|
|
|
Andy202 wrote: For merging together some recording data stored in CSV files I have used the
class CtextFile (by Johan Rosengren via the Codeproject site) to read in and
write out the processed files.
Presumably you are doing something more than just creating one file out of two.
Consequently your design with much larger files, needs to take into account specifically what you need to do.
And if it was me I would also look into the business requirement that requires that the output be another file. Pretty pointless to parse a large file and then parse it again.
|
|
|
|
|
Thanks for your post jschell.
Two input files to generate one output file.
Input1 sampled at 50 msec and Input2 sampled at 200 msec, so using timing information merge the two sets of data into one output file; interporlate as required.
I can do what is required if the files were as follows:-
Input1 = 1 Mb; Input2 = 2Mb and Output = 5Mb.
Andy
|
|
|
|
|
Andy202 wrote: Input1 sampled at 50 msec and Input2 sampled at 200 msec, so using timing information merge the two sets of data into one output file; interporlate as required.
So you have one 'block' (perhaps one line or longer) from file 1 that exists every 50msec. Call this X.
You have another 'block' from file 2 that exists every 200msec. Call this Y.
Thus you have 4 (200msec/50msect) X entries before each Y entry.
Steps
1. Build a buffered reader for each file. Buffered in the case reads N 'block' entries and allows it to read M more on request. The timestamp is exposed (parsed from the block.)
2. Read X via the buffer where N is 200.
3. Read Y via the buffer where N is also 200 (could be less as well.)
4. Now 2 will have data that fits into 3. Because you have read enough to overlap.
5. Step 4 represents a starting point. Basically the two buffers will mostly be off by 4.
6. You can't assume it will always be offset by 4 so continue to compare the two timestamps as you read.
7. On start up you need to sync the two buffer reads, since one file might have a much different start time from the other.
Performance impacts
- Play with the stream buffer sizes (actual file read versus buffered readers above.)
- Profile it with a tool for some large and real files, say 50 meg at least.
|
|
|
|
|
I would like to create a dialog box which contains an embedded Excel workbook. (Excel is installed on the computer, of course.) MS provides example code showing how to do something like this with a Document/View based app, but not in a dialog.
(I saw from searching this forum that there was a discussion about "Embedded Excel", but when I try to access it, I get an empty list with the message "Unable to load messages due to high load.")
|
|
|
|
|
You can go to the project tab and select Add to Project Sub menu. This menu has another sub menu. select the last option. Components and control
When you select that a windows dialog will pop up with 2 folders. Select the first one. Search for Microsoft Office SpreadSheet. Select that and click OK.
The spreadsheet will be embedded in your dialog.
|
|
|
|
|
I'm not sure what you mean by the "Project Tab" and "Add To Project". If I right-click on the solution, I get an item "Add New Project" and under that one of the possibilities is Visual C#/Microsoft Office/Excel Workbook, but I don't see how this will help me. Won't it just create a whole new project unrelated to my current one?
|
|
|
|
|
I should have mentioned that I am using VS2008
|
|
|
|
|
Forgot to mention, In the menu bar. There should be an option called Project. You should select that.
|
|
|
|
|
I still don't see this.
In VS2008, all I have under the "Project" menu or the right-click menu on the project is "Add New Item" and "Components and controls" is not a selection from the dialog I get from that.
|
|
|
|
|
What you can do is, Right click on the Dialog. You will get an option. Insert ActiveX Control. You will get the same component here also. Here you can select Microsoft Office SpreadSheet 11.0 or 10.0.
Hope it helps
|
|
|
|
|
Hi!
I've added Vertical and Horizontal Scrollbar to a Dialog in MFC. I'm facing the following problems.
(i)I've loaded a Flash movie inside the Dialog. My movie is larger than my Dialog. But both the scrollbars are not appearing at
the beginning. After I reduce the dialog size only,the scrollbars appear.
(ii) When I reached the extreme of Horizontal Scrollbar or Vertical Scrollbar the movie is not fully displayed. Part of the movies
is cut.
(iii) Some kind flicking occurs during scrolling(both Vertical and Horizontal).
I've developed this application using VC++ 2008/MFC. Here's the code:
BOOL OnInitDialog()
{
m_nScrollPos = 0;
m_nHScrollPos = 0;
}
void OnSize(UINT nType, int cx, int cy)
{
CDialog::OnSize(nType, cx, cy);
m_nCurHeight = cy;
m_nCurWidth = cx;
SCROLLINFO si;
si.cbSize = sizeof(SCROLLINFO);
si.fMask = SIF_ALL;
si.nMin = 0;
si.nMax = m_rcOriginalRect.Height();
si.nPage = cy;
si.nPos = 0;
SetScrollInfo(SB_VERT, &si, TRUE);
SCROLLINFO si1;
si1.cbSize = sizeof(SCROLLINFO);
si1.fMask = SIF_ALL;
si1.nMin = 0;
si1.nMax = m_rcOriginalRect.Width();
si1.nPage = cx;
si1.nPos = 0;
SetScrollInfo(SB_HORZ, &si1, TRUE);
}
void OnVScroll(UINT nSBCode, UINT nPos, CScrollBar *pScrollBar)
{
int nDelta;
int nMaxPos = m_rcOriginalRect.Height() - m_nCurHeight;
switch (nSBCode)
{
case SB_LINEDOWN:
if (m_nScrollPos >= nMaxPos)
return;
nDelta = min(max(nMaxPos/20,5),nMaxPos-m_nScrollPos);
break;
case SB_LINEUP:
if (m_nScrollPos <= 0)
return;
nDelta = -min(max(nMaxPos/20,5),m_nScrollPos);
break;
case SB_PAGEDOWN:
if (m_nScrollPos >= nMaxPos)
return;
nDelta = min(max(nMaxPos/10,5),nMaxPos-m_nScrollPos);
break;
case SB_THUMBTRACK:
case SB_THUMBPOSITION:
nDelta = (int)nPos - m_nScrollPos;
break;
case SB_PAGEUP:
if (m_nScrollPos <= 0)
return;
nDelta = -min(max(nMaxPos/10,5),m_nScrollPos);
break;
default:
return;
}
m_nScrollPos += nDelta;
SetScrollPos(SB_VERT,m_nScrollPos,TRUE);
ScrollWindow(0,-nDelta);
CDialog::OnVScroll(nSBCode, nPos, pScrollBar);
}
void Htmlbr::OnSize(UINT nType, int cx, int cy)
{
CDialog::OnSize(nType, cx, cy);
m_nCurHeight = cy;
m_nCurWidth = cx;
SCROLLINFO si;
si.cbSize = sizeof(SCROLLINFO);
si.fMask = SIF_ALL;
si.nMin = 0;
si.nMax = m_rcOriginalRect.Height();
si.nPage = cy;
si.nPos = 0;
SetScrollInfo(SB_VERT, &si, TRUE);
SCROLLINFO si1;
si1.cbSize = sizeof(SCROLLINFO);
si1.fMask = SIF_ALL;
si1.nMin = 0;
si1.nMax = m_rcOriginalRect.Width();
si1.nPage = cx;
si1.nPos = 0;
SetScrollInfo(SB_HORZ, &si1, TRUE);
}
void Htmlbr::OnHScroll(UINT nSBCode, UINT nPos, CScrollBar *pScrollBar)
{
int nDelta;
int nMaxPos = m_rcOriginalRect.Width() - m_nCurWidth;
switch (nSBCode)
{
case SB_LINERIGHT:
if (m_nHScrollPos >= nMaxPos)
return;
nDelta = min(max(nMaxPos/20,5),nMaxPos-m_nHScrollPos);
break;
case SB_LINELEFT:
if (m_nHScrollPos <= 0)
return;
nDelta = -min(max(nMaxPos/20,5),m_nHScrollPos);
break;
case SB_PAGERIGHT:
if (m_nScrollPos >= nMaxPos)
return;
nDelta = min(max(nMaxPos/10,5),nMaxPos-m_nHScrollPos);
break;
case SB_THUMBTRACK:
case SB_THUMBPOSITION:
nDelta = (int)nPos - m_nHScrollPos;
break;
case SB_PAGELEFT:
if (m_nScrollPos <= 0)
return;
nDelta = -min(max(nMaxPos/10,5),m_nHScrollPos);
break;
default:
return;
}
m_nHScrollPos += nDelta;
SetScrollPos(SB_HORZ,m_nHScrollPos,TRUE);
ScrollWindow(-nDelta,0);
CDialog::OnHScroll(nSBCode, nPos, pScrollBar);
}
What else need to be done to rectify the above problems?
|
|
|
|
|
Are you getting the size of the movie and then putting that value in the scroll bar?? why dont you try for Direct Show() API's?
|
|
|
|
|
Chandru080 wrote: wDirect Show() API's
Can you be little bit more elaborate on this?
|
|
|
|
|
|
voice chat in a socket client/server program.
assume the server has 3,000 clients connected, one client chats to others in voice (maybe a song).
I think it is not a good idea to submit the song to server from the client then server broadcasts to all other 3,000 clients through sockets, because a song may last for long time interval and consumes too many server's resources.
what do you do in the case?
.
|
|
|
|
|
I check out what Peer-to-peer[^] networking is?
> The problem with computers is that they do what you tell them to do and not what you want them to do. <
> //TODO: Implement signature here<
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
