C / C++ / MFC

26-Feb-09 1:03

Hi,

I'm working on an application where it loads a huge file (like 4-6GB) and parses the file, searching for regular expressions, converting from Hex to Human readable form, filters etc.

Since the file is of very large size, I'm facing lot of performance issues where it takes hours to process and show the file information in a window.

To overcome this, what I did is, break the entire file into logical segments of 1 MB each and process 3 segments at a time. When the user is scrolling, the next segments will be processed. With this, a certain amount of improvements were done. But it is limited to files which are less than 300MB or so. For files which are over GBs, it is just crawling and pathetic.

Wondering if there is any better way to handle these large files in a better way so that the user need not wait for hours?

Will a buffered read and caching the decoded information in a temporary files using a thread would help? How can we make sure that the GUI doesn't hang during these intensive parsing process?

Really appreciate your suggestions.

Thanks
Vikas

Vikas K.26-Feb-09 1:04

26-Feb-09 1:04

by the way, the application is written in MFC

«_Superman_»26-Feb-09 1:54

«_Superman_»

26-Feb-09 1:54

You could do memory mapping for the file segments.

You should get hold of a good profiler and check where the actual bottle neck is.

«_Superman_»
I love work. It gives me something to do between weekends.

Nishad S26-Feb-09 2:06

Nishad S

26-Feb-09 2:06

Vikas K. wrote:
How can we make sure that the GUI doesn't hang during these intensive parsing process?

What about using a worker thread?

- ns ami -

Stuart Dootson26-Feb-09 4:02

Stuart Dootson

26-Feb-09 4:02

I've had pretty good performance with memory mapped files (I've only used files up to about 1GB). What you need to do is create a file mapping with CreateFileMapping and then use MapViewOfFile to map segments of the file.

I worked with 16MB windows on the file, which actually meant I mapped 32MB at a time, 16MB before and after the nominal file offset, so that negative random accesses from the file offset didn't fault over into the previous window - something like the diagram below:

+----------------+----------------+
|                |                |
|                |                |
|                |                |
+----------------+----------------+
^                ^                ^
|                |                |
-16MB        Requested        +16MB
          offset into file

I packaged this up into a MemoryMappedFile class which had an associated randomly-access iterator class (like a vector has an iterator). The memory mapped file could only be accessed using the iterator, which was exposed using 'begin' and 'end' methods of the memory mapped file class.

On a reasonable workstation (2.4GHz Core 2 Duo), I could traverse the file at around 100MB/second when counting data packets whose size was roughly 200 bytes.

Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p

Vikas K.3-Mar-09 19:22

3-Mar-09 19:22

Thank you for your suggestions. Willwork on them and update Smile | :)

win 32 dll

hrishiS26-Feb-09 0:10

hrishiS

26-Feb-09 0:10

Hi to All,

I am trying to make a win32 dll.
I just wanted to make sure if we can use the win32 dll in all types of application project or all type of languages. I am sure about Vb. What about other languages? Will it work?

Thanks in advance
HimangshuS

I am a learner Always

CPallini26-Feb-09 0:14

CPallini

26-Feb-09 0:14

himangshuS wrote:
I am trying to make a win32 dll.

Well, have fun with it!

himangshuS wrote:
I just wanted to make sure if we can use the win32 dll in all types of application project or all type of languages. I am sure about Vb. What about other languages? Will it work?

The correct answer is no. The practical, common sense answer, is yes (provided the target OS is Windows and you're ready to put some effort in the task...)
Smile | :)

If the Lord God Almighty had consulted me before embarking upon the Creation, I would have recommended something simpler.
-- Alfonso the Wise, 13th Century King of Castile.

This is going on my arrogant assumptions. You may have a superb reason why I'm completely wrong.
-- Iain Clarke

[My articles]

hrishiS26-Feb-09 0:18

hrishiS

26-Feb-09 0:18

Hi
thanks for your reply

Could you please put some more light into it...

thanks

I am a learner Always

CPallini26-Feb-09 1:02

CPallini

26-Feb-09 1:02

Most languages allows using Windows API (on Windows, of course) that comes inside DLLs, hence you will be able to use, as well, your DLLs, in such languages.
Smile | :)

Iain Clarke, Warrior Programmer

Iain Clarke, Warrior Programmer26-Feb-09 1:05

26-Feb-09 1:05

Your question "Tell me all about DLLs" is not something we can answer in a few lines - you'll have to do a bunch of learning yourself. Luckily, other people have written articles on the subject!

I put "DLL" in the search bar near the top of the code project window, and can see these useful articles:
http://www.codeproject.com/KB/DLL/RegDLL.aspx[^]
http://www.codeproject.com/KB/cpp/howto_export_cpp_classes.aspx[^]
http://www.codeproject.com/KB/DLL/XDllPt1.aspx[^]

Good luck on your journey,

Iain.

Codeproject MVP for C++, I can't believe it's for my lounge posts...

Hamid_RT26-Feb-09 6:15

Hamid_RT

26-Feb-09 6:15

But my search bar is powerful than your search bar I could to see more. Big Grin | :-D

Of one Essence is the human race
thus has Creation put the base
One Limb impacted is sufficient
For all Others to feel the Mace
(Saadi )

Iain Clarke, Warrior Programmer

Iain Clarke, Warrior Programmer26-Feb-09 6:39

26-Feb-09 6:39

Well, I got all impatient! That cut and pasting gets tiring you know...

Iain.

Codeproject MVP for C++, I can't believe it's for my lounge posts...

add-in for outlook express

nikhil3125-Feb-09 23:44

nikhil31

25-Feb-09 23:44

Hello all,

I have been given the work which includes writing a add-in for Outlook Express.
The working of this add-in should be as follows:

Whenever a mail is opened in a new window & a right-click is made on a sender's name,then a context menu should pop-up which will include some menu items apart from
existing ones.
I would also like to provide a separate tool bar which will have some buttons on it,
providing same functionality.

Is there a way of doing this without making use of Third-party libraries.

reply me at : n.chudekar.31@gmail.com

Thanks n Regards,
Nikhil.

Re: add-in for outlook express

Vikas K.26-Feb-09 1:07

Re: add-in for outlook express

26-Feb-09 1:07

I don't know about Outlook Express but for MS Outlook, you can do the operations that you have given using Microsoft VSTO. Please read more about it and see if it helps you.

Crazy Kiya re26-Feb-09 17:36

Crazy Kiya re

26-Feb-09 17:36

Hello All,
I am also facing same problem.I want to add context menu and tool bar in Inspector Window of Outlook Express Frown | :(

. I have done it for MS outlook.
How to archive above task in Outlook Express

Thanks & Regards,
Jitendra Shirolkar

setting background color of the dialog

Member 163194525-Feb-09 23:08

Member 1631945

25-Feb-09 23:08

hello,i have seen all your forums to set the background color of the dialog.i am able to set the background color.But i want to change the background color of the main dialog during run time on selection of different menus.i want to call repaint ,how i do this?.i am using mfc dialog based application

software enggineer

CPallini25-Feb-09 23:19

CPallini

25-Feb-09 23:19

Member 1631945 wrote:
software enggineer

Sure?

Hamid_RT26-Feb-09 6:11

Hamid_RT

26-Feb-09 6:11

Yeah but it seems we need to a duel for it!
Big Grin | :-D

Of one Essence is the human race
thus has Creation put the base
One Limb impacted is sufficient
For all Others to feel the Mace
(Saadi )

«_Superman_»25-Feb-09 23:22

«_Superman_»

25-Feb-09 23:22

Calling Invalidate will redraw the dialog.
You can handle WM_CTLCOLORDLG and return the appropriate brush.

«_Superman_»
I love work. It gives me something to do between weekends.

Nishad S26-Feb-09 0:41

Nishad S

26-Feb-09 0:41

Another approach than the other reply.

You can map WM_ERASEBKGND and write like...

BOOL CMyDlg::OnEraseBkGnd( CDC* pDC )
{
 CRect rect;
 GetClientRect( &rect );
 pDC->FillSolidRect( &rect, m_clrBK );
}

You can set the m_clrBK where ever you needed and call Inbvalidate( TRUE ); to update the background.

- ns ami -