|
Hi Forum,
I am a newbie in C# even though I have been coding since in MATLAB since last 7 years. Now, I want to develop few library files for my scientific calculations. My calculations involve large matrix manipulations, say a calculating the inverse of a 10000*10000 matrix. I have tried this in C and my code compiles all right, but gives a runtime error. I think this is due to the fact that the computer memory can't handle such a long matrix. I have not tried it in C# though. But I would like to know if there is anyway I can do manitrix manipulation of large matrix in C#.
I have found many matrix manipulation library files in C# by different people in the CODEPROJECT website, but I ain't sure if they will handle such large matri manipulation.
Regards,
Amit
|
|
|
|
|
You are right: Your runtime error will most probably result from the lack of memory. Having such a large matrix with for example a 8 byte type (double, long etc) will need over 760 mb in ram. When calculating the inverse you will need another 760 mb. This logically gets worse when adding or multiplying 
Since windows processes have only a limited usage of memory (I think 2 gig is the boundary) no language you choose will handle several such large matrices in memory. If you really need this you will have to use some 64 bit OS.
One solution for this is having those matrices not directly in memory but on your harddrive and accessing and writing directly to your files, but this will result in much (!!!) work by yourself.
|
|
|
|
|
Thanks Rob!!! You gave a really helpful insight into the problem. I would like to know more about how to write the matrix into the harddrive and access from there instead of memory. Do you have any suggestions where should I start?
Regards,
Amit
|
|
|
|
|
Principally its relatively easy.
First you have to specify some sort of file structure. In this case the file could start with 2x4bytes containing the info about the size of the matrix. After that I would just place all elements of the matrix row by row, column by column into the file.
If you have such a file you should implement some sort of access wrapper which holds an open stream to the file and positions it to the correctly depending on what elements should be read. The formula is relatively simple. Assuming each element has a size of 8 bytes position x,y would be at the position 2 * 4 + x * 8 + y * matrixWidth * 8.
The ByteConverter class will help you to transform specific datatypes to pure byte arrays which can be written to the stream. To boost performance the access wrapper will also need the ability for stack reading (means a whole row or several rows in one catch) and probably also some caching or buffering mechanisms. But this greatly depends on what calculations you need.
I once made such a thing for some very special matrix types (but those were only 4 byte floating numbers) and if you put enough brain into it it can work really well. But keep in mind that for example calculating the inverse of a matrix means that youll have to read through the whole 760 mb file and then write it to another 760 mb file. So the bottleneck with this solution will always be the IO operations no matter how fast the algorithm behind is. The advantage is that you can nearly work on as many matrices at the same time as you want.
If you can reduce the size of each element as far as you can, e.g. take floats instead of double or short instead of long. Probably even byte will do it. Think carefully as it will greatly influence the performance and data size.
|
|
|
|
|
hi,
You probably know math better then I do (e.g you do scientific calculation  ), so consider this more like my curiosity... do you really need so big matrix in memory? I mean if matrix is somehow special (e.g has lot of zeroes or something), IIRC you don't have to store it whole in memory... something like if rank of matrix NxN is lower then N you can store only base columns and coeficients of linear combination for other columns... hmm at least my lecture text say so  If it's "just damn big matrix" then there isn't much you can do, IMO.
As Robert said there must be some techniques for working with matrixes in external memory, too.
It could be interesting topic for an article...
David
Never forget: "Stay kul and happy" (I.A.)
David's thoughts / dnhsoftware.org / MyHTMLTidy
|
|
|
|
|
Thats right. But thing is that if you want to have an inverse, the rank of a n*n matrix would have to be n. but, it was really nice of you to reply and give a thought to my problem. In fact, if you really look at reply of Rob, it really helps the cause I guess. Thanks Rob for your reply.
Now, one of the questions come to my mind is that is matrix manipulation so hard or so difficult. Is there some secret algo for its calculation? I mean the best selling (and of course, expensive) software for doing mathematical calculation is , no doubt, MATLAB. Why can't we have some more software like MATLAb, so that MATLAb gets cheaper due to the competition and we have a little heavier purse.
:P
Amit
|
|
|
|
|
amitmohanty wrote:
But thing is that if you want to have an inverse, the rank of a n*n matrix would have to be n
 okay it was bad example. There is one more (bad) example: assuming you somehow managed to compute determinant(A), you can compute A-1 as (1/determinant(A))*adj(A). I know that compute determinant is expensive* and you still need big matrix in memory, but only one... not two (assuming you would compute inverse by Gauss-Jordan elimination of A|Id)
Sure it looks like storing matrix on disc is way to go, just thinking about further optimalizations
amitmohanty wrote:
Is there some secret algo for its calculation?
I don't think so... but maybe there are "secret" implementation tricks and modifications to make known algorithms even faster. 
amitmohanty wrote:
Why can't we have some more software like MATLAb, so that MATLAb gets cheaper due to the competition and we have a little heavier purse.
*but OTOH it's just one number; once you manage to compute it, store it's value for reuse.
btw I am looking forward to be corrected again
David
Never forget: "Stay kul and happy" (I.A.)
David's thoughts / dnhsoftware.org / MyHTMLTidy
|
|
|
|
|
Hi all, here is myproblem:
I have two window applications ( with c#). I want these applications can share a value, say it a String value.
I tried to build a dll to bridge between these applications.
First application store that value in dll, and another read/get it.
Ofcourse that dll have method get and set value.But I don't know how storing that value without writing to file or database.
How can do that? any idea?
|
|
|
|
|
How does your solution work? Your dll will get loaded in both the applications and each will have its own copy of data.
You need to use remoting, which is the natural way for inter process communication in .NET.
Regards
Senthil
_____________________________
My Blog | My Articles | WinMacro
|
|
|
|
|
Woow, dot net Remoting ?
what is this ?
Ok, thank's,
You are the first one introducing me with that word.
do you have good online resource about Remoting?
But in Linux so simple,with pipe, for inter process communication.
Does Remoting like pipe?,
ok,I 'll search 'remoting'
|
|
|
|
|
hi,
create a class library from first application and added to second Application.
then you can read first applciation value..
regards,
pubudu.
|
|
|
|
|
Hello Every Body
How can I handle a system event that is not exposed by the SystemEvents class using c# ? I read somthing about "InvokeOnEventsThread" method in MSDN :
public static void InvokeOnEventsThread(Delegate method);
But I don't undrestand how I can use it.Please Help Me.
Thanx 
|
|
|
|
|
What are you calling a "system event"? What "event" are you looking to a notification of?
RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome
|
|
|
|
|
Hi...
I want to handle system events such as mouse clicks,mouse move and any action that happens out side of my application(When my application is not focused.Such as clicks on desktop or on any other application). Like babylone dictionary and keyloggers.I read an article in code project about system hooks(Global System Hooks in .NET) But hooking is very heavy action.The CPU usage of this sample is 80% to 100% when application is running ! I want a way to trap this events when they happen.
Please Guide me !
|
|
|
|
|
The only way to grab the mouse and keyboard events are global system hooks. If written properly, the hooks will not consume any measurable CPU time for days, maybe even weeks. But, of course, this all depends on what your doing with the captured messages.
RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome
|
|
|
|
|
Thanx For Your Attention Dave. Your Guid was very useful for me.![Rose | [Rose]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/rose.gif)
Good Luck !
|
|
|
|
|
Hi folks,
I've written a custom installer / uninstaller for my application and everything works fine
But...
How do i delete my uninstal.exe??
I can't tell it to delete itself but i have been told there is a way to get windows to delete a file next time it starts up. Anyone got any ideas on how to do this?
Kev
|
|
|
|
|
|
Very close but not quite.
Assume i have installed my application and there is a file called
c:\uninstall.exe
Uninstall.exe is registered with the windows add / remove programs.
The user tries to remove my application and all files and registry
entries are deleted without any problems. All except one, Uninstall.exe.
I can't tell it to delete itself as it is currently running and would
obviously crash.
Anyone know the answer or able to point me in the direction of an article?
Kev
|
|
|
|
|
|
I wasn't aware that i could put commands into the registry.
Could you show me how ot do it using the c:\Uninstall.exe as an example?
Thanks for your help
Kev
|
|
|
|
|
Colin pointed out the link to help. Here's the pertinent line:
The data value for a key is a command line.
Use a command line command, like "del", as in "del c:\uninstall.exe"
Remember good ol' DOS?
Marc
MyXaml
Advanced Unit Testing
YAPO
|
|
|
|
|
Marc,
Thanks for clearing that up, that's the "Idiot Proof" answer i was after but...
When i create my key with the value of del c:\uninstaller.exe i just get a message saying Cannot find del... when i restart my PC.
I assume i am missing something very trivial and i'll probably kick myself when i get it sorted but can you tell me what i'm doing wrong. Does the registry entry's name have to be anything specific? oir is that irrelevant in this case?
Cheers
Kev
|
|
|
|
|
exhaulted wrote:
Cannot find del... when i restart my PC.
Ew. I had assumed that it would be able to run DOS commands. You might need to do something like c:\uninstaller /u, where you pass a command line parameter to itself. BTW, it won't crash if you tell it to delete itself--why would it? The running version is in memory. There is the potential, though, of a locking issue--a running app might have its file locked by the OS, but I'm not sure.
Strange stuff!
Marc
MyXaml
Advanced Unit Testing
YAPO
|
|
|
|
|
Lol, after all that you are right, you can just tell it to delete itself!!
I just assumed there woudl be file locking issues but there's not.
Thanks to you all for your help.
Kev
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
