|
A. Don't use threads where you don't need them. Depending on the size of these files, it doesn't sound like there's a reason to break the processing up into multiple threads. Only other reason I can think of to break this up into threads is if there's a large number of files and you can process them simultaneously, then look at creating a thread pool.
B. Again, it depends on the size of the file, if they're small text files, this doesn't really matter, if they are large binary files (or images, videos) then it might be beneficial to only read chunks at a time instead of loading the whole thing. I've had to process files > 4GB in size in which case loading to memory is just plain ol' impractical (or impossible depending on memory).
C. Once again, it's going to depend on file size and how much processing there is to do.
You really don't provide enough information to give a good recommendation, would have to know:
0. File size range.
1. Amount of processing involved.
2. Type of processing involved.
|
|
|
|
|
I agree, that the requirements as stated don't seem to make threads necessary.
I'll add though that processing requirements can make threads beneficial. If there is a resource the processing depends on that might see long delays, such as a remote data source, SQL query, etc..., the threads can keep data moving, if one file is taking a while to process. Note that this can require that you implement additional queues if the files must be delivered in some sort of relational order.
In this case, where a file must be processed every X number of seconds, it doesn't sound like the added effort of introducing threads is worth it.
I would entertain the concept of adding threads if...
1. The process can't finish a file before the next file enters the queue.
2. Another process needs the files to be processed faster because of some time sensitive constraint.
|
|
|
|
|
If you need it only once, and all sequentially, then don't store in memory what you don't have to store there. Use a streaming approach: get some, do some, store some.
And don't create lots of threads if you don't have to; a main thread for GUI, and app management; and one, or a few, for performing the boring, long winding tasks, is all that makes sense to me. Separating read/process/write when you don't have to is a recipe for disaster, as you would need to synchronize all these threads and the data they share. The better way would be to have say two threads both dealing with half of the files. Having lots of similar threads is not a good idea either, as they would contend for the same resources, such as disk I/O bandwidth.
|
|
|
|
|
Hello All,
I am writing an application to get some zipped files from a remote location [over WAN but within the domain]. The file data is huge [around 1.5 GB]. I need to unzip the contents of the file to a temporary location and then read an unzipped database to get required information.
Currently what I do is this:
1. I have written the code using FindFirstFile/FindNextFile.
2. Download the files to a temporary location.
3. Unzip the contents of the file.
4. Read the extracted database file.
5. Display the result.
But this process is slow and communicating over the network is slow as well. I understand that the download speed will be a major factor in getting the files faster, but is there any other tweak that I can follow to make this work in a better way?
P.S. I cannot do this on a on-demand basis as the datastructure in the zipped file doesn't support this and is not likely to be changed.
You talk about Being HUMAN. I have it in my name
AnsHUMAN
|
|
|
|
|
How about unzipping "on the fly", so as you receive the file from the network, start unzipping it right away getting along as it downloads?
> The problem with computers is that they do what you tell them to do and not what you want them to do. <
> If it doesn't matter, it's antimatter.<
|
|
|
|
|
Is there more than one zip file? Do you've to get all the zip files and then extract them to a single large file (as in a large file was split into several small ones by a compression utility)?
"Real men drive manual transmission" - Rajesh.
|
|
|
|
|
Affirmative! There is more than one zip file I need to extract but not to a single large file. All zip's have to be downloaded and unzipped individually.
You talk about Being HUMAN. I have it in my name
AnsHUMAN
|
|
|
|
|
So you can use something like the classic producer-consumer design. A thread keeps downloading zip files and another thread will simultaneously extract the downloaded zip files (and perhaps another thread will process the extracted files, depending on how fast all these things happen).
"Real men drive manual transmission" - Rajesh.
modified 7-Dec-11 2:04am.
|
|
|
|
|
Rajesh R Subramanian wrote: producer-consumer design
Can you provide some good links for the above Design pattern with Code Samples?
|
|
|
|
|
|
I think the producer consumer example is the way to go, but maybe a version of that using asynchronous workers would help the most. What I have in mind would limit the threaded complexity to a minimum while still achieving very good performance.
If you are using Qt then my answer would be slightly different but in short I would do like this
Assuming I understood you correct and you do not have access
to c++11 or concurrency library features like future or asynch run., I am also assuming that it is OK that the zip files are unzipped in a non-deterministic order.
1. A thread, maybe the main thread, tells a worker to start downloading one or several zip files.
2. As soon as a zip file is downloaded it is handed over to another worker that is unzipping the file at the wanted location
3. When a zip file is downloaded and unzipped then this is communicated back to the original calling thread
4. Communication between "participants" is done solely through message queues. No shared memory except the queue, that way data sharing and thread synchronization issues are minimal.
5. The "design pattern" I have in mind for this is the active object. There is plenty of material if you google for it. My favourite is Herb Sutters take on it.
I wrote my version of active object, which should be easy to use, maybe after making minimal change to use whatever thread library you use.
Another example., although more of the fire and forget nature than your scenario., is the asynchronous logger g2log
that I recently wrote about. Check in the code how the messages are passed from the g2logworker to it's thread internal and you will see how easy it is
modified 7-Dec-11 2:55am.
|
|
|
|
|
Sounds like you need a client/server approach. Since a majority of your time is spent downloading, and only for the sake of opening the file, leave the file on the server and simply send the server a request to unzip the file and extract the necessary information from it.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous
|
|
|
|
|
hi all,
please help to convert from 'LPCTSTR' to 'const char *'
thanks.
|
|
|
|
|
Your question is a bit vague.
The short answer would be: "LPCTSTR tstr = (LPCTSTR)constCharPtr";
The long answer is most likely this: If your project setting is "MBCS" or "not set", then LPCTSTR should be "const char *" already, if your project setting is "unicode", then try using MultiByteToWideChar[^] to convert your string to a widechar string.
> The problem with computers is that they do what you tell them to do and not what you want them to do. <
> If it doesn't matter, it's antimatter.<
|
|
|
|
|
If you look at the definition of LPCTSTR in your program you may find that is already defined as const char *; that would give you a clue as to what you need to do next.
Unrequited desire is character building. OriginalGriff
I'm sitting here giving you a standing ovation - Len Goodman
|
|
|
|
|
|
Hello,
Case-1
SendMessage((*m_pIPedt).GetHWnd(), WM_SETFOCUS, NULL, NULL);
Result-1
Edit control is getting focus. But it is not accepting any text. Hanging sometimes.
Case-2
SetFocus((*m_pIPedt).GetHWnd());
Result-2
Edit box is getting focus and I am able to type text in edit control
I want to know difference between these 2 cases.
Please help me.
Regards
msr
|
|
|
|
|
The WM_SETFOCUS message is sent to a window by the Windows system, as the result of a call to SetFocus() or the user setting the focus to the window by mouse or keyboard. Thus option 2 is the correct way to do it.
Unrequited desire is character building. OriginalGriff
I'm sitting here giving you a standing ovation - Len Goodman
|
|
|
|
|
|
i am reading excel file .
Excel::_ApplicationPtr pApplication;
pApplication.CreateInstance( _T("Excel.Application") );
but i dont understand why pApplication.CreateInstance( _T("Excel.Application") ) failed on my PC while its working fine on another PC.
please help me for this.
thanks.
|
|
|
|
|
Le@rner wrote: but i dont understand why pApplication.CreateInstance( _T("Excel.Application") ) failed
Because you didn't bother to check the return value.
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
no chk this
if ( FAILED( pApplication.CreateInstance( _T("Excel.Application") ) ) )
{
AfxMessage( _T("Failed to initialize Excel!") );
return ;
}
|
|
|
|
|
It is no use printing a message such as the above when something fails, as it provides no useful information. You need to capture the system error code from GetLastError() and print its details to find out why your program fails. I am assuming that you have all the necessary libraries installed on your PC in the first place.
Oops: sorry you need to interpret your HRESULT as CPallini suggests.
Unrequited desire is character building. OriginalGriff
I'm sitting here giving you a standing ovation - Len Goodman
|
|
|
|
|
I believe that's the optimal way with standard API (like Win32). On the other hand, COM calls provides error code in return value.
If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.
This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke
[My articles]
|
|
|
|
|
You are, of course, correct; it was my bad (mea culpa) and I have fixed it.
Unrequited desire is character building. OriginalGriff
I'm sitting here giving you a standing ovation - Len Goodman
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
