|
I don't think you meant to post this to me, or (maybe) even to this thread.
Use the best guess
|
|
|
|
|
I meant it for the thread anyway...
|
|
|
|
|
thank you for the suggestion. yes, i'm going in the woring direction. i have to find another solution.
|
|
|
|
|
What is the problem that you are trying to solve?
|
|
|
|
|
i'm trying to solve "out of memory" problem, it's the no-source third party lib who eat up my memory. but i'm rather my usage is right, and i have to do this way. so, i'm trying to find a way to make the third patry lib occupy less memory.
|
|
|
|
|
If the leak is due to a bug in a third party library then there may be no workaround to your issue. The operating system already caches out memory to a page file on disk. If you're exceeding the 1 or 3gig limit set by the linker, then you're hitting the wall on addressable memory.
You can't allocate more memory than you can address. With 32 bit pointers the theoretical limit is 2^32 bytes or 4gig. In practical terms it's less because some of the bits in the upper end of the pointer can have a special meaning.
I've seen server side systems live with leaks like this, by periodically restarting.
If you can prove that the third party library is leaking the memory with a simple test case, then I would raise this issue with your vendor.
|
|
|
|
|
As a library presumably it is using the standard windows libraries and it links to those dynamically. That means it is using an external calls to manage memory.
So it should be possible to re-link and/or wrap the library to use a different memory management calls.
I doubt that it is easy however.
You might also want to carefully examine what problem you think this will solve. It certainly won't give you more memory. It is going to make it much slower, and if you are not careful about how your allocator works it could make it vastly slower.
|
|
|
|
|
Falconapollo wrote: My problem is how to avoid thrid party lib to allocate memory from physical memory.
No, that is not your problem - that is a suggestion on how to solve your (as of yet unknown) problem. And it won't work for several reasons:
1. The operating system will do all memory allocations; most importantly even the most basic function you can call to allocate memory will allocate that within physical memory. There's no way to avoid that short of writing your own OS.
2. The OS will also move memory from and to hard disk as needed in order to keep the entire system running smoothly. Repeatedly. At runtime! Again, you cannot influence that short of rewriting the OS.
3. Technically, no application or library actually allocates or processes data in physical memory; instead they work on virtual memory. Under the hood, the OS will put the actual data anywhere into physical memory or on hard disk, but neither your application nor that library would even be able to tell where their data are currently residing, because from their point of view all data always resides in virtual memory. There is no way (and no sensible reason) to change that. Well, short of rewriting the OS 
So, please start again, and start by describing your problem, not by suggesting a solution. Are you running out of memory? Are you experiencing performance issues? Are you suffering from memory fragmentation? Any of these could be a problem, and there may be reasonable solutions. But either solution will not solve all of these problems, so knowing the problem would help.
|
|
|
|
|
my real problem is "running out of memory", this make me crazy, i don't know how to solve the issue. because it's the no-source lib who eat up my memory. but, i must use the lib.
so, i'm stuck here. please give my your suggestion?? any help will be appreciated!
|
|
|
|
|
Ok, there are probably several ways to solve this, but first let me clear up a misconception:
The amount of physical memory available to your computer does not limit the amount of memory available to your application. Therefore, forcing a library to not work on physical memory - if that were possible - wouldn't change a thing! It is the OS that decides where allocated objects reside, and, yes, that does include the hard disk. The OS already moves objects from physical memory to the hard disk if it is running low on physical memory. And it does that at runtime.
Your actual problem is that you are running out of virtual memory. Virtual memory is an abstract address space that the OS grants every application. Depending on your Windows version and settings, that address space may encompass anything between 2 and 3 GB at most, even if your machine has 16 GB of physical RAM.
There are two probable causes for your problem:
1. The amount of data you have to deal with is simply more than this library can handle.
If this is the problem, then you should consider ways how to partition your data into smaller chunks, and adapt your operations to process that data bit by bit. Loading a 4 GB file into memory simply does not work!
2. You may not be using the library as intended: many library functions create objects and return them as results. The intention is that you call appropriate destruction functions once you are finished with those objects. If you don't do that, the amount of memory you need will continually increase over time until you run out of memory. (and you may think it's the library that is the problem because the library does the allocation
You should check the library documentation to see which of the functions create objects and expect you to release those objecs again, or call corresponding functions that clean them up properly. Also check your algorithms to see whether they 'lock' these objects unnecessarily, or for too long.
|
|
|
|
|
Thank you for your advice. Yes, maybe I can do some cleanup work.
|
|
|
|
|
That's the best place to start.
Make sure you have no memory leaks.
If there is some reason that you need gigabytes of data in memory to do some heavy complex processing (perhaps DNA analysis, simulations etc...,) then upgrade your hardware and go 64 bit.
|
|
|
|
|
|
Stefan_Lang wrote: 2. The OS will also move memory from and to hard disk as needed in order to keep
the entire system running smoothly. Repeatedly. At runtime! Again, you cannot
influence that short of rewriting the OS
I am rather certain that all desktop OSes provide access to some very raw memory management APIs. Without that there are whole classifications of applications that could not be written.
As an example of one API for windows, I presume there are others, the following insures that memory is not written to the hard drive.
http://msdn.microsoft.com/en-us/library/aa366895%28VS.85%29.aspx[^]
|
|
|
|
|
Quote: I am rather certain that all desktop OSes provide access to some very raw memory management APIs.
yes, i can't agree more. But I don't find what i need.
|
|
|
|
|
If your problem is that your application fails because over time this library keeps consuming memory then the solution is to remove the library from your application. You can do that as follows.
1. Create an executable that wraps the library.
2. Provide a communication idiom: files, stdio or sockets. That provides the necessary API for your business needs (basically a proxy for what you are already doing.)
3. In your actual application use WinProcess (or whatever the method is called) to execute 1 and manage it.
4. Use the communication from 2 from your application in place of the straight to library calls you are currently using.
Other variations on the above depend on your needs
- Create multiple instance
- Stop each instance after X (1 or more) uses
- Have min/max instances and use a locked queue to send requests for processing
- Detect failures, restart and retry a request.
- Detect excessive memory usage, terminate and restart.
|
|
|
|
|
Thank you for the specific suggestions. i will have a try.
|
|
|
|
|
I want to convert a binary file (.dll) to a base64 string and store it in my application, then write it to the hard disk using the stored base64 string but i am failing miserably. any help is appreciated thanks.
edit: basically, exactly what this site does: http://www.motobit.com/util/base64-decoder-encoder.asp[^]
|
|
|
|
|
base64 conversion is actually simple, you shouldn't fail. However you may also find ready-to-use implemantations, for instance, a quick Google search gave me: "Encoding and decoding base64 with C++"[^].
Veni, vidi, vici.
|
|
|
|
|
Member 8417954 wrote: but i am failing miserably. any help is appreciated thanks.
It's impossible to help unless you explain exactly what the failure is. You might also like to explain exactly what problem you are trying to solve, as this seems a rather pointless exercise.
Use the best guess
|
|
|
|
|
CString ftitle;
fscanf_s(f_ptr_seed_inf,"%s\n",ftitle[ifl].GetBuffer(MAX_PATH));
ftitle[ifl].ReleaseBuffer();
I got error message:
Unhandled exception at 0x776015de in MSW_Vib_Model.exe: 0x00000000: The operation completed successfully.
Please help
|
|
|
|
|
Try taking out the [ifl] from the ftitle object. I don't think you can call GetBuffer() on a character returned from the [] operator.
CString ftitle;
fscanf_s(f_ptr_seed_inf,"%s\n",ftitle.GetBuffer(MAX_PATH));
ftitle.ReleaseBuffer();
|
|
|
|
|
|
mrby123 wrote:
fscanf_s(f_ptr_seed_inf,"%s\n",ftitle[ifl].GetBuffer(MAX_PATH));
How does this even compile? I would have expected a C2228 error.
"One man's wage rise is another man's price increase." - Harold Wilson
"Fireproof doesn't mean the fire will never come. It means when the fire comes that you will be able to withstand it." - Michael Simmons
"Show me a community that obeys the Ten Commandments and I'll show you a less crowded prison system." - Anonymous
|
|
|
|
|
As Richard said, I've already answered this question.
You need to change the fscanf_s line as follows -
fscanf_s(f_ptr_seed_inf,"%s\n",ftitle[ifl].GetBuffer(MAX_PATH), MAX_PATH);
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
