|
What is the best way to store *MASSIVE* amounts of data? I'm thinking [EDITED] millions or billions of images totalling 10TB or so. Not sure of the exact size yet... need to get some data samples... but somewhere in that arena.
I guess at a high level, they could be split by state.
I know that SQL can not handle this kind of size.
Performant lookup is also desired.
How is this usually handled?
Needs to be backupable of course as well.
Had an old boss who was a big fan of storing the path in the DB and the files on disk. From personal experience, you end up with so many directories, the file system breaks down. Try navigating to a folder with 1000+ directories.
I don't think I'll need to do too complex of queries on the data, just simple look ups. Inserts should be fast as well.
-- modified 10-Oct-13 23:06pm.
|
|
|
|
|
Never mind the DB you choose, how will you connect that many hard drives to one machine?! 
Have you looked at these guys MongoDB[^]?
The difficult we do right away...
...the impossible takes slightly longer.
modified 10-Oct-13 22:46pm.
|
|
|
|
|
Hmm... you're right haha... my estimates were a bit over the top. I did the math just now, and thats like 95,000 TB... lol...
I think 10TB - 100TB total data is more in the right area... oops 
|
|
|
|
|
That sort of volume is going to required some serious iron. I would suggest you need to look at Oracle, I loathe Oracle but SQL server struggles with serious volume. We were looking at record volume about the same but data size substantially smaller and I think the 3 server cost topped $1m for the Oracle licences.
Then you are going to want to hire an Oracle consultant/DBA to design and tune the blasted thing.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
As I responded to the other guy, I got a little over zealous with my estimates... 10TB to 100TB is probably closer.
|
|
|
|
|
If the number of records is over 100m then you are going to struggle with SQL Server, even when you go down the path suggested by Garth - that is really your only option anyway. There is no way you want the images anywhere near your searchable data.
Never underestimate the power of human stupidity
RAH
|
|
|
|
|
ok, there's the images, and then there's the 'meta data', or think of the index fields telling you how to locate an image ..
So, to locate the correct image or pointer to image, you need the metadata - this is what you search on .. the question is, do you really need to store the images WITH the meta/index data ? I'd suggest 'not' .. so, you have a searchable DB of metadata/indexes on fast tier 1 storage, then, once the right record is round, you then retrieve the image from maybe a worm drive, or 'tier 2 storage' using the pointer/location from the search.
Plenty of storage providers provide WORM storage for example
|
|
|
|
|
|
SledgeHammer01 wrote: Performant lookup is also desired.
You need actual requirements - not off the cuff statements.
For example you could take the above statement and claim that the the system must be capable of serving all of that data at the very same time. If so then the company is going to need to buy an IP backbone company just to deliver it.
The reality is that system has business cases that dictate usage. You need to start with those.
SledgeHammer01 wrote: Needs to be backupable of course as well.
Think about that in terms of the above requirements. How long is it going to take to restore the entire system from scratch? Obviously too long some something has to be different.
SledgeHammer01 wrote: I guess at a high level, they could be split by state.
You start with requirements, business cases and usage patterns and then define categorizations. Categorizations will impact how it is stored.
One final suggestion - if that estimate is a pie in the sky dream then go do your own research. If it is a hard business requirement then the business needs to pay for consultants (very likely plural) with experience in very large data systems rather than trying to roll their own.
|
|
|
|
|
SQL is a language.
I have a few hundred gigs that I generate google map tile images on the fly so it can work. Appropriate indexing and caching is the real key.
How it is usually handled? Testing, scaling, testing, and measuring.
Are you on an intranet exlusively? Inexpensive servers with 4 port NICS serve as really nice in-house CDN's and scale fairly well. Not on an intranet? Well there are public CDN which will host the images then it is merely about hosting the indexes.
|
|
|
|
|
Hi,
do you know any c# code generation framework (.net 4.5)?
Any resources / best practices for that topic?
|
|
|
|
|
Here is a code generation framework: CodeWorker[^]
The difficult we do right away...
...the impossible takes slightly longer.
|
|
|
|
|
The truth is you can compile C# programs on CommandPrompt (aka DOS).
And with .NET Framework you can create text files with any extension including cs (for C#).
And also with .NET Framework you can issue CommandPrompt commands.
So, in theory you can make a C# program that creates c# programs and also that runs them.
|
|
|
|
|
|
Hi Team, i have a master page in my .net application and from master page i call a helper file (.cs)and my requirement is i would like to execute one method(returns a string value) in that helper file, my application connects to master page each time upon loading of different pages i would like to keep the first method call value and persist the value between different page calls
i am unable to user httpcontext.session object as it gives me null values always ,
can some one suggest me the best way to preserve the method call value between round trips.
|
|
|
|
|
|
Hi This is related to C# code only but i have given the information related to session which i tried to get my requirement, it is purely c# related issue.
|
|
|
|
|
You mentioned session variables and httpcontext which is why I asked if it is an ASP.NET application. If it is pure C# then you just save the information in a class variable somewhere, in the same way you would for any variable.
Veni, vidi, abiit domum
|
|
|
|
|
No, while you are using C# as your code, the issue relates to ASP.NET development. This isn't a general purpose C# question, is it? This is asking how to store the value so that different browser page refreshes see the same value.
|
|
|
|
|
Hi i am looking for options in C# code to use a c# method call value between different pages, anyway i have posted my request in asp.net section.
|
|
|
|
|
What Pete has said the issue is an ASP.NET issue rather than a C# issue, even though you are coding using C#
The C# forum covers a multitude of sins that are not related to ASP.NET in particular.
you will more than likely get an answer quicker using the ASP.NET forum
Every day, thousands of innocent plants are killed by vegetarians.
Help end the violence EAT BACON
|
|
|
|
|
for (int i = 0; i < t.Rows.Count; i++)
{
dataGridView1.Rows[i].Cells["Column1"].Value = t.Rows[0]["Address"].ToString();
}
error occur :
index was out of range,must be non-negative and less than the
size of the collection. parameter name:index
how i solve this prob,
|
|
|
|
|
Kawshik_itbd wrote: for (int i = 0; i < t.Rows.Count; i++)
{
dataGridView1.Rows[i].Cells["Column1"].Value = t.Rows[0]["Address"].ToString();
}
Change to
for (int i = 0; i<t.Rows.Count-1;i++)
{
...
}
This should do it..
The signature is in building process.. Please wait...
|
|
|
|
|
if i do
for (int i = 0; i <= <pre lang="xml">t.Rows.Count-1; i++)
{
dataGridView1.Rows.Add();
dataGridView1.Rows[i].Cells["Column1"].Value ="roy";
}</pre>
my prob is solve but i don;t add dataGridView1.Rows.Add() prog can not be execute through a exception like Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index whatever i do either t.Rows.Count-1 or t.Rows.Count
plz help what can i do
|
|
|
|
|
Change
...
dataGridView1.Rows[i].Cells[...
...
to
...
dataGridView1.Rows[dataGridView1.Rows.Count-1].Cells[...
...
Your counter is based on t, which is not equal to the dataGridView.
The signature is in building process.. Please wait...
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
