V. wrote:The user can choose to view just a day a year or even the entire history. (eg.
the last 2 years wich still means 1 500 000 000 records)
Simple math will demonstrate that a user (person) will NEVER look at that many records at once.
They will either look at a summary, or a summary with drill downs or a will be looking for a very, very small subset of that.
V. wrote:Has anyone got any experience with this size of database and how to solve the
issue of getting the data quickly?
Get real requirements.
V. wrote:The data are datapoints that need to be plotted on a graph in a certain time
Excellent example. Graph 1.5 billion data points (from your first requirement) - so xactly how many pixels are on your screen? Again simple math will demonstrate that you can't view that many data points on a graph. So either there will be a much smaller time period or a summary of the entire period.
If a summary one solution is to build summary tables. So for example at the end of every data you create summary data of the day. Then a graph that display every day, uses the summary table rather than the raw data.
V. wrote:1. Keep table size reasonable, so how to split the data into different tables
You start with real usage patterns. For example what percentage of the time does a user want to look at data for the last week, month, year? Or hour, day, week? Or by collection point via week. Etc, etc, etc.
Second you get realistic estimates and anticipated growth. So is 3 billion that average or the maximum? Will it be 3 billion next year or 30 billion. Keep in mind it must be realistic, not pie in the sky.
Third how long must you keep it? 1 year, 10 years 100 years?
And if it no one seems willing to discuss reality then you might want to look into price tags for a really hug SAN system, and submit a request to buy it because you will need it for testing - because the system will need that to run.