|
I'm writing a Windows-XP application that reads a large binary file (more than 3GB) 1MB at a time. I write the data to a DAC card, then go back and read another 1MB from the file. I've malloc'ed 1MB, with a pointer, to store the data. Because this is a real-time DAC application I need the file input to be fast.
I notice that when the file I'm reading is smaller than some limit (about 2.6GB) the file input is fast enough, but when it's at all larger than this limit, it slows down unaccepably. This behavior is very consistent.
I don't think this problem has anything to do with my hard drive or fragmentation of the files. It's too specific to the file size.
I'm using the MFC CFile class to open and read the file:
buffer = (char *) malloc (1048576);
for (i=0; i=3000; i++) {
bytes_read = cfile.Read(buffer, 1048576);
"write to DAC"
}
It's suspicious that the 2.6GB limit is very close to the "available memory" as reported by Windows Task Manager, although I don't see this number change much while the program is running. I'm not trying to read the whole file at once, just 1MB at a time, always overwriting the same 1MB buffer. It's as if it's trying to allocate 3GB and therefore actively paging.
Any ideas would be much appreciated.
Best Regards,
Jason
|
|
|
|
|
For something like this I'd look at using Memory Mapped Files. This should give optimal access to the file and you can just write it directly to your output device. With MMF you don't actually read the file, you access it just like memory. I added MMF support to pugXml which you'll find here on CP.
Neville Franks, Author of ED for Windows www.getsoft.com and Surfulater www.surfulater.com "Save what you Surf"
|
|
|
|
|
Thank you. I think that's worth looking into. Long term, though, I might need to read files even bigger than 4GB, which I think is the limit for a memory-mapped file. I thought a "cfile.read" would work, and it seems to work up to a point. It's as if the system is trying to allocate memory for the entire file, even though I'm only reading 1 MB at a time.
Regards,
Jason
|
|
|
|
|
Have you tried profiling the code? I like Glowcode for this. Also have you tried it on another PC, another version of Windows etc.
With MMF I think you can slide a window to different segments of the file so there may not be an issue with file size limits. I've not used this though.
Neville Franks, Author of ED for Windows www.getsoft.com and Surfulater www.surfulater.com "Save what you Surf"
|
|
|
|
|
A few more things. You aren't testing what Read() returns. I think you'll find that CFile can only handle files up to 2G.
Neville Franks, Author of ED for Windows www.getsoft.com and Surfulater www.surfulater.com "Save what you Surf"
|
|
|
|
|
Without profiling your code, I can only guess, but I suspect the system is cacheing the file's data (both what you just accessed and look-ahead), and when the cache fills up available memory, the system starts discarding pages and swapping. The result is your program starts having to wait on the disk access, which is far slower than memory.
If that indeed is the cause (again, you need to profile to be sure) then I'd try using CreateFile() with the FILE_FLAG_NO_BUFFERING flag.
--Mike--
LINKS~! Ericahist updated! | 1ClickPicGrabber | CP SearchBar v2.0.2 | C++ Forum FAQ
Strange things are afoot at the U+004B U+20DD
|
|
|
|
|
Thanks for your reply. It's just like you said. Judging by the Windows Task Manager, the system is cacheing the file data. As long as the file is smaller than the memory available for the system cache, the execution can keep up. For large files, though, it's getting killed by the swapping.
I switched to FILE_FLAG_NO_BUFFERING. Now I no longer see the system cache filling up, but the transfers have actually slowed down! Does that make any sense?
1. I wrote a loop in which I repeatedly read from the file until the end of the file. It executed fastest when the file was declared as "SEQUENTIAL_SCAN", slowest with "NO_BUFFERING", somewhere in between when "SEQ_SCAN | NO_BUF". I would have expected the last one to be the fastest.
2. I wrote a loop in which I read 1MB into a buffer, then wrote it to a new file. The results weren't conclusive here. I wouldn't be surprised if "NO_BUF" was slower, though. There could be an advantage to cacheing the 1MB of data if you're going to immediately write it back out again.
Please let me know if you have any suggestions.
Best regards,
Jason
|
|
|
|
|
Try overriding CFile:Open and add the FILE_FLAG_NO_BUFFERING (and possibly FILE_FLAG_OVERLAPPED) flag to ::CreateFile().
|
|
|
|
|
Thanks to everyone for your help. Here's an update:
I saw a lot of improvement using "FILE_FLAG_SEQUENTIAL_SCAN".
To use FILE_FLAG_NO_BUFFERING I need to make sure that the buffer is sector aligned. I had been using malloc, but I've switched to VirtualAlloc to control the buffer address alignment. It looks like VirtualAlloc will give me a page-aligned buffer, but is that necessarily sector-aligned? How do I report the sector size? GetDiskFreeSpace will give me that, but only for volumes of less than 2GB.
Regards,
Jason
|
|
|
|
|
Hi ...
i am writing this code to test if there is internet connection :
**********************************************************
void CServerInternetDlg::OnOn()
{
HINTERNET hInternet= ::InternetOpen("BSAtomicEdu", INTERNET_OPEN_TYPE_PRECONFIG , NULL, NULL, 0);
if(!hInternet)
{
AfxMessageBox( "Internet open error! Test your system and try again.", MB_OK | MB_ICONERROR );
return;
}
else
{
AfxMessageBox( "Internet open Success", MB_OK | MB_ICONERROR );
return;
}
}
**********************************************************
but i am getting the following error:
Server InternetDlg.obj : error LNK2001: unresolved external symbol __imp__InternetOpenA@20
Debug/Server Internet.exe : fatal error LNK1120: 1 unresolved externals
**********************************************************
i think i have missing library i am using
#include <afxdb.h>
#include <afxinet.h>
#include <afx.h>
#include <wininet.h>
#include <stdio.h>
#include <tchar.h>
#include <wincrypt.h>
any idea about the error ??!!???
Best Regards
|
|
|
|
|
In order to get rid of your error, simply link the Wininet.lib library to your project. That oughta clean it up.
|
|
|
|
|
Thank you very much its working now...
|
|
|
|
|
I want to make project like Tweak Manager .. any one can help me with open source code or any information about it....
Thanks
|
|
|
|
|
Hi.
I have a thread that adds values to a vector variable in a class. also main program reads same thread an to display data in it. I havn't used critical section or other sync methods but program works ok, is it necessary to use critical section ?
thanks.
|
|
|
|
|
main program reads same vector variable not thread.
|
|
|
|
|
Hi,
Because u r using multi-threading application you SHOULD sync ur work using critical sections, events, semaphores,...
your program runs well now, but when the thread writes to the variable at the same time as the main thread reads it, the value read by the main thread will be out-of-date, it is not the current value of the variable.
Furthermore, if you don't use the keyword volatitle in declaring shared variables, the compiler may perform wrong optimizations to speed the code up, and you may get wrong results.
-Always take the safest way
Regards,
Mohammad Gdeisat
|
|
|
|
|
|
Another concern is a possible program crash. When vectors are changed, all iterators are invalidated. So if the main function has an interator and is in the process of reading the data, the thread could interrupt and add something to the vector. Now the main function has an invalid iterator. The problem with not using synch methods is that the errors may not be noticed for months. You could think all is well and then one day, your program crashes while doing the same thing it's been doing all along.
Shawn
|
|
|
|
|
i need to be able to delete the browser history. also it would be a big help to be able to delete temporary internet files. thanks
|
|
|
|
|
deleteurlcacheentry is a function, use it.
however you need to know the exact url that you want to delete
hope this helps you
peace
|
|
|
|
|
I declared a variable through ClassView>Add Member Variable in my Cview.
I included header in my MainFrm.cpp.
How to use the variable because it doesnt recognise my variable. Do I need to make an object inside my MainFrm.cpp?
I tried to use
CProgView cview;
cview.myvariable;
but it says cannot access protected member declared in class.
Thanks
|
|
|
|
|
Add Member Variable with 'public',not 'protect', than you can access it.
|
|
|
|
|
It is public variable from CView. I use it in CDoc, how to declare it? THanks
|
|
|
|
|
Thx Astham
There are 3 kinds of member variables/functions:
public:
Can be used/called by objects of any class
private:
Can be used by objects of this class only
protected:
Can be used by objects of this class and derived classes only.
So if you want to use the variable from somewhere else than a CProgView member
function, you need to make the variable public. If you use Add Member Variable
there's three radio buttons at the bottom of the dialog. Or you could just write
public: at the line before the variable declaration.
THis is my header code,
public:
float pointX[100];
float pointY[100];
virtual ~CMyProgView();
#ifdef _DEBUG
virtual void AssertValid() const;
virtual void Dump(CDumpContext& dc) const;
#endif ]
This is my public variable declaration. Why cant I use it?
How to use it from my CMyProgDoc? I have #include CMyProgView.h and use it such as below:
CMyProgView obView;
obView.pointX[i] = something;
|
|
|
|
|
When you right click a RichEdit Control.
<italic>Work hard and a bit of luck is the key to success. You don`t need to be genius, to be rich.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
