How do I manage large databases and create displays based on the data?

See more:

C++

Perl

database

My capabilities: I did math as an undergraduate at a solid university. I have taken a couple of classes that involved coding for narrow purposes and a couple of intro CS classes. Limited experience with C++, Fortran, Perl.

What I want to do: I want to create a database environment of about 10,000 x 10,000 blocks with about three numerical values associated with each block, plus establish boundary conditions.

Next, I want to iterate through my blocks, running calculations, and updating values.

From a logic standpoint, I have no problem coding this. I can build logical loops that cycle through arrays in the desired order and things of this nature. On a theoretical basis (from math side), I am familiar with different routines that optimize your computational time required to do various mathematical tasks. I did a computational physics course where the instructor gave us a database of algorithms that implement different routines with varying accuracy/efficiency balances and I know how to select what is best for my purpose. I don't have that anymore, it was just for one language, and I wouldn't know where to look now.

I realize this is computationally expensive and I might need to reduce my array size. That's fine - I can tinker and see what my limits are here.

I want to associate a color with a block based on my values in it, then be able to display each "time step" in an animated way. Additional functionality that would be nice is the ability to zoom into certain areas (since I don't have a 10,000^2 pixel monitor). That's not mission critical however - I can do something crude like change the code to display only a selected slice if it's too much trouble. I would (perhaps naively) think that the display of this data would require some sort of averaging of a cluster of blocks to one color on my end of things then feeding out 1000x1000 pixels or something like that but I have no clue really.

It would also be really nice to be able to layer images on top of this color coded background (like a simple -> looking arrow, perhaps with variable length tails that I compute).

Finally, and perhaps least important, it would be nice to be able to implement a drop down menu or something that lets me change certain variables in the code - the equivalent functionality of being able to halt a routine at the command line, give input, start it back up.

I will be coding on a relatively new mac, preferably in xcode unless you have reasons for me to do something else.

So, the broad questions:

What language do I use for this? Anything with good documentation online I will learn. I'll have to consult documentation anyway for above languages - it has been a while.

What literature should I read about optimizing really large computational tasks? I do have some vague familiarity with this; for instance, I have implemented code that was multi-threaded in the past. The logic that updates my grid will require multiple passes per cycle so I might be able to get a big speed up from a lagging thread covering 2nd or 3rd passes based on what the leading thread already updated. Beyond that, I think my primary efficiency gains are fairly restricted because I need to iterate through the grid block by block - just picking the right routines to do my math at each step is where any significant speed up might derive on top of multi-threading. Enlighten me otherwise if you can though!

Can someone explain how the display side works? Is there some prepackaged code that does a similar function? If it's relatively close I can tinker with it. I wouldn't even know how to make a box and print hello world in it; I've only worked in terminals and spit out data to files in the past.