Click here to Skip to main content
15,907,677 members
Home / Discussions / Database
   

Database

 
GeneralRe: large database Pin
V.16-Jul-15 22:10
professionalV.16-Jul-15 22:10 
GeneralRe: large database Pin
Wendelius16-Jul-15 22:20
mentorWendelius16-Jul-15 22:20 
GeneralRe: large database Pin
Mycroft Holmes17-Jul-15 14:02
professionalMycroft Holmes17-Jul-15 14:02 
GeneralRe: large database Pin
V.18-Jul-15 0:12
professionalV.18-Jul-15 0:12 
GeneralRe: large database Pin
Mycroft Holmes18-Jul-15 0:45
professionalMycroft Holmes18-Jul-15 0:45 
GeneralRe: large database Pin
V.18-Jul-15 8:25
professionalV.18-Jul-15 8:25 
GeneralRe: large database Pin
Jörgen Andersson20-Jul-15 11:01
professionalJörgen Andersson20-Jul-15 11:01 
AnswerRe: large database Pin
GuyThiebaut18-Jul-15 1:19
professionalGuyThiebaut18-Jul-15 1:19 
I have worked with 300 million row medical datasets where analysis jobs took two days to run.

A lot can depend on how the data is modelled - sometimes a very flat table structure can help where in effect you denormalise the data - this is a very useful way of modelling the data if you are just trawling through one table. At other times more EAV style tables can speed things up.

A lot of it depends on what you are doing with the data - there is no simple answer as to how to structure/re-structure data as much of it comes down to some experience and trial and error.

One thing you may want to consider is when running queries on the data - have an initial set of queries that will extract the general population of your data that you are working on. Then build tables with indexes on this 'candidate' data and work from there. This way you may be slimming down your data from 3 billion rows to a few million rows.

You probably do not want to be running queries on a 3billion row recordset other than for the purposes of extracting your initial data which you then work on in separate tables.

This is how I did things with 300 million rows, although I realise that this is smaller by a factor of 10 compared to your dataset.

[edit] I noticed you also mention in a later post that writing is an issue.
Your limitation is going to be down to hardware. Depending on your budget splashing out an a decent SAN solution may help - although SAN technology is beyond my current level of experience, I am just someone who uses it while other people configure it.
Be sure that the disks where the database lives are not shared with the OS or anything else that may cause contention when reading/writing data.
“That which can be asserted without evidence, can be dismissed without evidence.”

― Christopher Hitchens


modified 18-Jul-15 7:31am.

GeneralRe: large database Pin
V.18-Jul-15 8:23
professionalV.18-Jul-15 8:23 
GeneralRe: large database Pin
GuyThiebaut18-Jul-15 8:37
professionalGuyThiebaut18-Jul-15 8:37 
QuestionSQL Linq, updating multiple records Pin
jkirkerx15-Jul-15 10:50
professionaljkirkerx15-Jul-15 10:50 
QuestionSave value of PREPARE into a variable Pin
Jassim Rahma14-Jul-15 22:23
Jassim Rahma14-Jul-15 22:23 
AnswerRe: Save value of PREPARE into a variable Pin
Wendelius14-Jul-15 22:45
mentorWendelius14-Jul-15 22:45 
GeneralRe: Save value of PREPARE into a variable Pin
Jassim Rahma15-Jul-15 22:22
Jassim Rahma15-Jul-15 22:22 
GeneralRe: Save value of PREPARE into a variable Pin
Wendelius15-Jul-15 22:51
mentorWendelius15-Jul-15 22:51 
AnswerRe: Save value of PREPARE into a variable Pin
jschell15-Jul-15 10:45
jschell15-Jul-15 10:45 
QuestionSome help with SQL to Linq Conversions Pin
jkirkerx13-Jul-15 7:04
professionaljkirkerx13-Jul-15 7:04 
AnswerRe: Some help with SQL to Linq Conversions Pin
Richard Deeming13-Jul-15 7:18
mveRichard Deeming13-Jul-15 7:18 
GeneralRe: Some help with SQL to Linq Conversions Pin
jkirkerx13-Jul-15 7:34
professionaljkirkerx13-Jul-15 7:34 
General[Got It] Pin
jkirkerx13-Jul-15 7:58
professionaljkirkerx13-Jul-15 7:58 
GeneralRe: Some help with SQL to Linq Conversions Pin
Richard Deeming13-Jul-15 8:00
mveRichard Deeming13-Jul-15 8:00 
QuestionSQL Linq, query and then take result to query again Pin
jkirkerx10-Jul-15 7:43
professionaljkirkerx10-Jul-15 7:43 
Answer[Solved] Pin
jkirkerx10-Jul-15 8:53
professionaljkirkerx10-Jul-15 8:53 
QuestionSELECT then INSERT data in a matrix way Pin
Farhad Eft9-Jul-15 9:14
Farhad Eft9-Jul-15 9:14 
AnswerRe: SELECT then INSERT data in a matrix way Pin
Richard Deeming9-Jul-15 9:22
mveRichard Deeming9-Jul-15 9:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.