|
People, I found that CFileDialog class causes serious problem.
I had a large free memory block in the address space of my prog: 1200 MB
Then I create CFileDialog and delete it:
CFileDialog *pdlg=new CFileDialog(TRUE);
pdlg->DoModal();
delete pdlg;
I check address space again and found only 700 MB:
The reason was very simple: I found some dlls in the address space of my process like urlmon.dll, netapi.dll, modemInst.dll and so on. These dlls was loaded into my address space by CFileDialog class and was not unloaded by its destructor. As a result, I couldn't allocate memory for a large file- address space had been fragmented. Any idea how to solve this situation? I think to create CFileDialog in a separate process. May be, there are better variants?
|
|
|
|
|
If you're sure the memory is being taken up by those dlls, why not try unloading them yourself with
FreeLibrary() ?
How are you trying to allocate the memory? Have you tried the VirtualAlloc() function?
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
[quote]If you're sure the memory is being taken up by those dlls, why not try unloading them yourself with
FreeLibrary()?
How are you trying to allocate the memory? Have you tried the VirtualAlloc() function?[/quote]
Yeah, I'm sure because I saw list of modules, downloaded in the address space before creation of CFileDialog and after it had been destructed. I tried to use FreeLibrary for such modules. After it (not immediately) program failed, giving me some inaudible error messages. Yeah, I used VirtualAlloc(). It is the best if you need to allocate large memory block. Currently I created a new process, which works with CFileDialog, leaving my address space untouched. It works, but I think that it is not very elegant approach. And one mysterious dll still presents- modemInst.dll (it is not connected with CFileDialog).
|
|
|
|
|
Freeing libraries you didn't load is dangerous, it's like releasing a reference you don't hold on a COM object.
Steve
|
|
|
|
|
Thank you, I didn't realize that.
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Read here, especially the last part.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous
|
|
|
|
|
A program using DLLs may delay loading them unto the point when they're first needed. This appears to have happened here. The problem with that is that the DLLs will then stay until you exit your program, or manually unload them (as suggested above).
Why CFileDialog loads all these DLLs isn't clear to me. My guess would be file preview settings - but then I would have thought the DLLs used for previewing would be loaded to system memory, not application memory. Anyway, this is not in fact a memory leak, "only" a shortage of memory.
Moreover, you should normally not even be able to notice memory fragmentation, except for the fact that the operators new and delete notably slow down when you're low on memory. The system memory manager hides the true physical memory addresses from you, and translates them to a virtual application memory address space. So even if you get addresses from allocations that indicate a state of fragmentation, this may be misleading.
That said, the memory manager will easily allocate huge blocks of memory for you even if fragmentation does split up your memory into unusably small chunks: it has the ability to pretend a contiguous block of memory, even if the physical memory is spread out all over the entire available physical memory space.
Your problem is a different one however: you cannot load a large file into memory as a whole. that means your application is using more memory than available. Note that even with 16 GB RAM, the memory space of an application is limited to no more than 4 GBm or in Windows in fact only about 2-3 GB, depending ont the OS version.
The solution is to split it up. It isn't a good idea to load such big files into memory anyway - exactly because of the problem you encountered. Depending on what you wish to do with that file, you don't need the full file at any point in time anyway. Instead use a buffer of at most a few MB in size, load a chunk, process the data, store the intermediary results, then repeat until you're done.
|
|
|
|
|
To unload libraries which you did not load is unsafe op- just as Stephen wrote. try delete twice the same object and you'll see what will happen. To perform this op, one has to know how they were loaded and unload them exact in opposite order. Moreover, he must know, that nobody else will try to unload these libraries too. I tried, and program failed. CFileDialog loads all these libraries for the very simple reason. It offers such service: one may open file from remote disk for example (or you may type an URL in address bar). Dear Stefan, you are not right about memory manager. I do not see physical addresses, but only virtual ones. In my virtual address space I have to have a continuous block of free memory in virtual address space to allocate large file (and no matter, what I use- VirtualAlloc, new or MapViewOfFile). I can clearly see fragmentation of my address space, when I run my prog under debugger. As I wrote, I had 1.2 GB continuous block before creation of CFileDialog and only 0.7 GB after it. If I can to download large file, I will do it. To split file is the last chance. Any idea, what headache will I have, if for example I need to rotate large file, which cannot be downloaded in memory?
|
|
|
|
|
Yes, unloading is a risk. I've read Stephens comment and therefore omitted that warning. Maybe I should have pointed it out nonetheless.
Regarding fragmentation, maybe I haven't been clear. Indeed, all you see is virtual addresses. That's what I meant when I said the MM hides the true addresses from you. As a consequence, you won't know if the addresses you see point to two blocks of memory that lie next to each other, or if they are sperated by a 5GB gap. In fact, you don't even know if one of those blocks is in fact contiguous - part of it may have been moved to the page file!
In any case, as pointed out before, the state of fragmentation is irrelevant to your problem. Your problem is you're running out of memory.
You only have a few options:
a) reduce memory usage elsewhere; apparently that may not be easy or even work
b) make your application a 64 bit application, enabling it to use a bigger address space; of course this only helps if your total memory is big enough, but you can increase available memory by increasing the page file on your system.
c) split up your file and adapt your program to work on it bit by bit.
|
|
|
|
|
Have you ever read what I wrote? My problem is NOT that I'm running out of memory. I had enough of it before I created CFileDialog. I wanted open a file of size 880 MB, having 1.2 GB of free memory memory. But I had not a continuous block of memory after CFileDialog was destructed. One very simple but not elegant way to resolve this situation was to allocate 1.2 GB using VirtualAlloc, then create CFileDialog. It downloads its modules in memory anyway, but large block will remain intact. Memory would not be as fragmented as it was if I previously created CFileDialog. Another way, to which I came up was to create a separate process, create CFileDialog within it and return what I needed (path to file) to the parent process using one of appropriate methods (I used memory mapped file). It is a programming- there are usually many ways to do same thing. I thought about which method is the best. May be, there is a way to get CFileDialog to work more correctly than it really does. I did not dig in its source code. I do have too much problems with algorithms to waste time for such thing. When you are short of resources (few memory, weak processor and so on) it definitely makes you more disciplined. And I have no intention to buy another comp and 64 bit OS - let's my boss do it.
|
|
|
|
|
Sorry, apparently I've skipped part of your message in reading. So what you're seeing is a fragmentation of the virtual address space of your process, and that prevents the allocation of a block that doesn't fit into that space.
My next suggestion then would be to allocate a big block of memory before calling CFileDialog, like you did. You don't even need VirtualAlloc for that, you could just call new, and then create your buffer using in-place construction.
This of course requires that you know in advance the size of your file. But then, if the size would vary, there would be no point in even trying to load it into memory as a whole: if you don't know the size, you have to assume that you will eventually encounter files that don't fit into memory, no matter what!
That said, even if the file is always the same size, where is it that this aplication will run? Is it on your PC? If not, a client running the program may have less memory available than you do!
One last idea is to split the file buffer into multiple parts, allowing you to use up most of the available memory in spite of fragmentation. That way you could keep all of it in memory, just not within one contiguous block.
You could even write a wrapper class that does behave like a contiguous block of memory, but internally translates an address into a location within one of those buffers.
P.S.: here's some code to illustrate my suggestion:
template <unsigned int total_size, unsigned int chunk_size>
class DistributedBuffer {
unsigned char* blocks[1+(total_size-1)/chunk_size];
public:
DistributedBuffer() {
for (unsigned int i = 0; i < 1+(total_size-1)/chunk_size; ++i)
blocks[i] = new unsigned char[chunk_size];
}
unsigned char operator[](unsigned int offset) const {
return blocks[offset/chunk_size][offset%chunk_size];
}
unsigned char& operator[](unsigned int offset) {
return blocks[offset/chunk_size][offset%chunk_size];
}
};
int foo() {
DistributedBuffer<1024*1024*1024, 1024*1024> myFileBuffer;
myFileBuffer[12345678] = 17; unsigned char c = myFileBuffer[12345678]; return (int)c; }
modified 27-Feb-13 9:16am.
|
|
|
|
|
Thanks for tips. I think, that such class will slow my prog. As I wrote, I tried to allocate large memory block and only after it create CFileDialog. But even if I can preserve large memory block for my own use, I'll loose some volume of memory anyway- all sh*t that CFileDialog loads in my address space will stay there even after CFileDialog will have been destructed. I prefer to keep my address space clean. One way is for example to change base addresses for my libraries like FreeImage.dll, jpeglib.dll and so on- there is such useful utility named Rebase.exe. On the other hand, I'm really short of memory and this is why I prefer do not load all unnecessary libraries into address space. What I did- another separated process- gave me what I wanted. I cannot change base addresses of system libraries- it may help only if my prog runs at my own comp. There are many useful things, which can be implemented, but now the main problem for me is effectiveness of algorithms which I use.
|
|
|
|
|
Hi all. Nice to meet you. Could you tell me how can I fix the memory leak of this code? I don't have a clue. Thanks.
#include <stdio.h>
#include <pthread.h>
#include <stdlib.h>
#define MAX 100
#define BUFSZ 10
int i;
int buffindex = 0;
char* buffer;
void* producer(void* p);
void* consumer(void* p);
pthread_cond_t cond_prod = PTHREAD_COND_INITIALIZER;
pthread_cond_t cond_con = PTHREAD_COND_INITIALIZER;
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
int main(void)
{
pthread_t prod, con;
if((buffer = malloc(sizeof(char) * BUFSZ)) == NULL)
{
perror("malloc");
exit(EXIT_FAILURE);
}
pthread_mutex_init(&mutex, NULL);
pthread_cond_init(&cond_prod, NULL);
pthread_cond_init(&cond_con, NULL);
pthread_create(&prod, NULL , producer, NULL);
pthread_create(&con, NULL , consumer, NULL);
pthread_join(prod, NULL);
pthread_join(con, NULL);
pthread_cond_destroy(&cond_prod);
pthread_cond_destroy(&cond_con);
pthread_mutex_destroy(&mutex);
free(buffer);
return EXIT_SUCCESS;
}
void* producer(void* p)
{
for(i = 0; i < MAX; i++)
{
pthread_mutex_lock(&mutex);
if(buffindex == BUFSZ)
{
pthread_cond_wait(&cond_prod, &mutex);
}
buffer[buffindex++] = '@';
printf("Produce: %d\n", buffindex);
pthread_cond_signal(&cond_con);
pthread_mutex_unlock(&mutex);
}
pthread_exit(EXIT_SUCCESS);
return NULL;
}
void* consumer(void* p)
{
for(i = 0; i < MAX; i++)
{
pthread_mutex_lock(&mutex);
if(buffindex == -1)
{
pthread_cond_wait(&cond_con, &mutex);
}
printf("Consume: %d\n", buffindex--);
buffer[buffindex] = '\0';
pthread_cond_signal(&cond_prod);
pthread_mutex_unlock(&mutex);
}
pthread_exit(EXIT_SUCCESS);
return NULL;
}
modified 27-Feb-13 15:48pm.
|
|
|
|
|
This is a good example of how NOT to ask a question on Code Project.
You're just dumping a bunch of code and asking others to do your work.
What have you tried? Have you run this code in the debugger? Exactly which allocation is leaking?
No wonder you don't have a clue!
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Thank you for the greetings sir. First of all, I'd like to say that I'm a self taught person so... this thing of "do you think we are really going to do your homework for you?!" doesn't exist for me (did you know there are freaking bad colleges in the world?). Ah... I study in a college but the place where I study simply sucks (all my teachers try to hide the C implementation of the theory from us as much as they can and I mean ... always!). So, I decided to learn everything (as much as possible) from Computer Science by myself. So, please, take easy. I don't have a clue because I don't have anyone to teach me anything  .
Back to the code ...
sorry if I didn't post the output:
Produce: 1
Produce: 2
Consume: 2
Consume: 1
Consume: 0
==2737== Thread 3:
==2737== Invalid write of size 1
==2737== at 0x400D6C: consumer (prodcon.c:72)
==2737== by 0x4E38E0D: start_thread (in /lib64/libpthread-2.15.so)
==2737== Address 0x53f203f is 1 bytes before a block of size 10 alloc
==2737== at 0x4C2ABED: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==2737== by 0x400B6D: main (prodcon.c:24)
==2737== LEAK SUMMARY:
==2737== definitely lost: 0 bytes in 0 blocks
==2737== indirectly lost: 0 bytes in 0 blocks
==2737== possibly lost: 0 bytes in 0 blocks
==2737== still reachable: 1,552 bytes in 4 blocks
==2737== suppressed: 0 bytes in 0 blocks
I appreciate the help.
|
|
|
|
|
I apologize for not understanding your situation better, but you didn't make it clear in your first post. 
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
Quote: I apologize for not understanding your situation better
ok, no problem
Quote: but you didn't make it clear in your first post.
sorry for that too.
|
|
|
|
|
noislude wrote: ==2737== at 0x400D6C: consumer (prodcon.c:72)
...
==2737== by 0x400B6D: main (prodcon.c:24)
Are these line numbers?
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous
|
|
|
|
|
yes. David you don't need to bother with this program. few hours ago, I took a detour from Modern Operating Systems to Programming with POSIX Threads to get a better understanding about threads. I apologize for the trouble.
|
|
|
|
|
The error output points at the function consumer, and the program output indicates the state before hitting the error: You're writing to an array index of -1 in this line:
buffer[buffindex] = '\0';
, after decreasing buffindex from 0 to -1 in this line:
printf("Consume: %d\n", buffindex--);
Suggested fix: change
if(buffindex == -1)
to
if(buffindex == 0)
, or better yet:
if(buffindex <= 0)
(in case you accidentally decrease buffindex multiple times; your current code doesn't seem to be able to achieve that, but code changes...)
The leak information may go away after fixing the error, it may just be a consequence of the abnormal error condition. if not, please come back with your new results.
|
|
|
|
|
I fixed the problem with the buffer index. Now I can notice three behaviors:
the first one, always producing one stuff and consuming it (in this case, I get a memory leak)
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
Produce: 1
Consume: 1
possibly lost: 272 bytes in 1 blocks
for the second behavior, I produced and consumed ten stuffs (and I got a bigger memory leak)
Produce: 1
Produce: 2
Produce: 3
Produce: 4
Produce: 5
Produce: 6
Produce: 7
Produce: 8
Produce: 9
Produce: 10
Consume: 10
Consume: 9
Consume: 8
Consume: 7
Consume: 6
Consume: 5
Consume: 4
Consume: 3
Consume: 2
Consume: 1
Produce: 1
still reachable: 1,552 bytes in 4 blocks
for the last behavior, everything went fine (without memory leaks)
Produce: 1
Produce: 2
Produce: 3
Produce: 4
Produce: 5
Produce: 6
Produce: 7
Produce: 8
Produce: 9
Produce: 10
Consume: 10
Consume: 9
Consume: 8
Consume: 7
Consume: 6
Consume: 5
Consume: 4
Consume: 3
Consume: 2
Consume: 1
Produce: 1
All heap blocks were freed -- no leaks are possible
|
|
|
|
|
I don't really see how your code can leak more than one block of memory when the entire program only allocates one!? Those leaks must be elsewhere - if those leak messages are indeed correct.
I suspect that the pthread library internally uses some kind of memory pool that doesn't get freed before the end of the program. I've seen the same thing in a CAD library that produces tens of thousands of "leak" messages because of that. It's very annoying becasue that way it's extremely difficult to pinpoint actual leaks, or find out if there even is one. 
|
|
|
|
|
I see. Are those internal leaks a bad thing for a bigger program?
|
|
|
|
|
Well, if my assumption is correct, technically these aren't leaks - they just appear as such because memory pools are designed that way. If used correctly, these 'leaks' will not be a problem, except when it comes to the problem of locating real leaks.
Memory pools are used as a faster way to allocate and free lots of small blocks of memory. Basically they take up the role of a memory manager for specific types (or sizes) of objects. They may be capable of freeing unused blocks at runtime (reducing the number of "leak" messages), but doing so complicates the management effort considerably. As a consequence, most implementations never free blocks at runtime, they only grow when running out of free memory within already allocated blocks. That means the total cost of memory is the maximum load at runtime.
|
|
|
|
|
Just in case someone wonders, it was written in the book Programming with POSIX threads (the book I mentioned earlier), that we don't need to destroy (with pthread_mutex_destroy()) a mutex initialized statically i.e. with PTHREAD_MUTEX_INITIALIZER. Thank you people for all the attention given to this topic
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
