|
I need some ideas or pointers on storing several hundred gigabytes of data. Currently, the files transferred range from 1 KB up to 600 MB and vary in sizes.
What I would like to do is break the files down into small (8K?) blocks and index them with a hash. Then in a database, create a chain so the files can be reconstructed. I like this because it will allow duplicated blocks of data to be identified and reduce the size of the storage. Bad idea? How might a directory structure look to accomplish this so it isn’t impossible to enumerate through the files?
I thought about storing the files blocks in a database as blobs but I think that would be too much strain on the database resources. SQL 2008 will have some nice features to accomplish this but it will be a year or two before we get there.
Any ideas? Thanks in advance.
|
|
|
|
|
Whats wrong with the filesystem for storing your data, with a seperate index if necessary? We probably need more information on what type of data you have and how you intend to index it.
When you said "I have a few hundred gig of files and I need to store them somehow" the first thing that comes to mind is NTFS 
|
|
|
|
|
Thanks for the response Mark.
The system I am working on is a pub/sub that distributes files to multiple subscribers. It is an in-house system that transfers manufacturing data – application, documents, collected data, test results, etc. I currently have it where the data is uploaded to a file server in the sky and the subscribers then download – nothing too complicated.
It is currently architected where the subscribers connect through a web farm (load balanced) to download small chunks of 64K until the transfer is complete. Each chunk is a new http request which is causing heavy I/O between the web servers and file server as it opens the file, reads until the requested chunk and then returns the data. It would be ideal to just stream the file using the same connection until the transfer is complete and resume with network hiccups. The problem here is that some of the third-party sites use older proxy servers which won't allow for that. In addition, there is a firewall(beyond my control) which limit the amount of time a connection is allowed to remain open.
My new plan is to store the files in smaller chunks so it is more efficient when downloading so it doesn't have to navigate through the large files up to the position of the chunk being downloaded. I could also leverage this to reduce duplications of data stored on the server. Unfortunately with this design, I would then have to open a database connection to identify where the blocks of data are stored and how to piece them together. I would probably end up in a worse scenario with the IO to the database and calculating what to return. This is where I thought storing the data in the database might be better as I would already have a connection and I can query the data and return the exact requested chunk. I am torn here because of the associated costs for storing that much data in a database.
I am just curious if there are any ideas. I probably should not worry about how the data is stored and work on solutions to solve the number of http requests.
Thanks
|
|
|
|
|
rcardare wrote: I probably should not worry about how the data is stored and work on solutions to solve the number of http requests.
Yes, it sounds to me your current solution is not using chunked-encoding, or not using it correctly.
|
|
|
|
|
Hi,
It seems like the issue with some legacy clients not handling large downloads may be solvable by the clients requesting the data in chunks. This doesnt mean the data needs to be stored in chunks though
I'd solve the issues with how the clients retrieve the files first. Then you can see how appropriate your storage mechanism is (I'd suggest NTFS with additionally indexing in SQL Server would work fine.
|
|
|
|
|
Here is my two cents for what it's worth:
avoid moving the data around, why not use the database to tell the client where the data is located?
whereever there is data there should be services to provide the client with the "knowledge" about the data he is looking for.
when you are working with large chunks of data, avoid moving it around, and especially across a network.
make the local service perform as much work as possible and return an "answer".
David
|
|
|
|
|
We had a similar problem with large file distribution.
I wrote a system where a client would download the file and then send broadcast messages out so that other local hosts could pick up the file and download it from the lan. Also if your using windows clients look at the Background intelligent file transfer.
a programmer traped in a thugs body
|
|
|
|
|
Dear friends,
I guess and must have this thing been discussed many times before also, but now a fresh question. 
What should messages on user interface say to user while these situations (and many more)(user is in US).
Also before, at the moment of doing and after performing the action...
File Copy
File Not found
File Delete
File Send
Database Update
Database Insert
Database Delete
Please help with the syntax and approprite sentences.
thanks in advance,
|
|
|
|
|
The convention for a file copy is to show a file being copied animation, and to provide feedback on the file being copied. For instance, if there are multiple files being copied you would normally tell the user where they are in the copying process, e.g. Copying 10 of 24 and it is common to show a progressbar detailing performance.
Anyway, the standard way to design your application is to write the messages out in your own language (stored in a resource DLL), and then have someone translate the commands out in the relevant language.
When it comes to standards, take a look at what the tools you use on a day to day basis tell you. What does Word say in these situations? Visual Studio? Explorer? It's generally best to follow what others have done rather than reinventing things yourself.
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
Hi Guys,
I'll be starting to work on web app that would send out massive mails for our clients campaign (approx 20k per batch), the webserver's and mail server are being hosted on our end so we have the complete control of the platform.
I'm just wondering:
1. What would it takes me to maintain a high volume of mails being received by our mail server
2. Would it be possible to have another mail server where we could crate a load balance for the volumes of mails?
3. Can you give reference / URL where I could check on the design for this kind of web apps.
Thanks
Dom
|
|
|
|
|
1) cross posting is discouraged here ( that means you posted this same message in more than one CP forum)
2) people don't like SPAM including the people at CodeProject
However hope is not lost, since you think this is a "Web App" with any luck you will never get it to work.
|
|
|
|
|
Hello comrades !!! I need your help. Here is what we have:
We have a jpg image that contains gray background and a lot of white eggs on it. We need to get the number of eggs and the size of each eggs. (i think in pixels).
How can i do it ?
The hardest problem is to get the whole egg (To mark the edges) and smart divide screen into some parts, for easiest count.
Thanks for help.
One nation - underground
|
|
|
|
|
Gosh. Cross posting homework. Excellent.
Deja View - the feeling that you've seen this post before.
|
|
|
|
|
I work for a state agency that has a multiple page, multiple question application form. Because the application centers around eligibility for a payoff, there are different paths that an applicant may take through the application. For instance, depending upon their answers, they may only need to fill out one or two pages of questions, but could conceivably need to fill out more pages depending upon whether or not further information is required.
What sort of approach/technology would be ideal to handing this sort of thing in .NET? Currently, it is handled in a very clumsy manner involving multiple .aspx pages and a lot of unnecessary, hard-coded session variables and logic checks. In this environment it's easy for things to become unnecessarily complicated, and changes to a single question might require changes in multiple places throughout the application (especially to database calls).
One method I was thinking of would be a single .aspx page with a multiview control backed by XML files for each page of questions where the questions are given attributes that define what type of question they are (i.e. yes/no, checkbox, text), as well as the name they are to be given in the database (i.e. "absence_eligibility1"), but I am unsure of the best way to implement the dependencies.
Any ideas?
|
|
|
|
|
Depending on how much time you are planning to invest in this, it may be more cost effective to get an off-the-shelf solution that can handle this type of skip logic. These types of product range in price from hundreds to thousands of dollars.
*DISCLAIMER* I am the architect for a company that produces this type of software, but there are many options so you might want to check around. I can tell you from first hand experience that once you get into conditional logic and/or have to create and deploy multiple questionnaires you've got quite a bit of work to do.
|
|
|
|
|
I am just trying to learn OOAD using c# at present, by designing a simple song management system.
Within this system there are songs which have one to many artists. I therefore designed classes as follows:
<br />
class Song<br />
{<br />
Artist[] artists; <br />
}<br />
<br />
class Artist<br />
{<br />
String name;<br />
}<br />
<br />
class SongList<br />
{<br />
List<Song> songs;<br />
}<br />
<br />
This seems to work OK I could in my output list each 'Song in songs' and show the artist by looping through each one in that song.
Where I have got really confused is another requirement which I might have to list all Artists along with their songs. How can I build a collection of artists and list the songs for each artist if artist is an attribute of a song. I can after all do song.artist, but I cannot access song from the artist: artist.song.
On thinking further I decided that an artist is not actually an attribute of song, but just a loose relationship. If they are just to seperate objects in a collection how do I implement this loose relationship between the two objects.
Any help, including references to books or other material which may be of help would be greatly appreciated. I son't think that I am thinking correctly to understand this, so any help that will assist me in thinking the right way would be greatly appreciated.
Thanks,
Richard
|
|
|
|
|
rb157 wrote: How can I build a collection of artists and list the songs for each artist if artist is an attribute of a song. I can after all do song.artist, but I cannot access song from the artist: artist.song.
You could have a dictionary with the artist as the key and a list of songs as the value, e.g.
Dictionary<Artist, List<Song>> songsByArtist = new Dictionary<Artist, List<Song>>(); You'd then have to loop through all the songs, loop through the artists, check if they already exists in the dictionary (if not add the key-value pair) and then add the song to the List in the value part.
As for the further thinking I'm also having a think about it (too early in the morning (oops, afternoon  ))
|
|
|
|
|
rb157 wrote: How can I build a collection of artists and list the songs for each artist if artist is an attribute of a song. I can after all do song.artist, but I cannot access song from the artist: artist.song.
Sure you can
<Artist><Song><Song>class Song
{
List<Artist> artists;
}
class Artist
{
String name;
List<Song> songs;
}
class SongList
{
List<Song> songs;
}
</Song></Song></Artist>
|
|
|
|
|
Thanks, I did wonder about these sort of circular relationships and whether they are wise... i.e. if for some reason the Song and Artist objects do not replicate each other correctly then a different result is obtained when outputing song.artist[i] than artist.song[i]. However if circular relationships are generally considered appropriate, then I am quite happy with that approach.
My concern is that in the real world the artist is probably not an attribute of the song and the Song not of the artist, in that one is not part of the other, just has an other type of relationship: In the same way that the Driver is not an attribute of a car. However if a driver is modelled in the same way as a steering wheel in its relationship then that is fine. I would appreciate comment on this point.
The approach provided by Ed... This would work artist to Song, but I think that another list would be required to output song by artist:
Dictionary <Song, List <Artist>>
Which is fine but would appear to be an amount of processing which is far in excess of mike's suggestion.
Thanks,
Richard
|
|
|
|
|
rb157 wrote: I did wonder about these sort of circular relationships and whether they are wise
I apologize. I was in no way suggesting it was wise, only that it could be done. In order to know if it is wise much more information is required. I would also guess that a design utilizing one or more interfaces would most likely be superior.
|
|
|
|
|
Thanks, can you give me an idea of the sort of things that I would need to consider in establishing if this was an apropriate approach, you also mention interfaces, in what respect might these be utilised? I understand what interfaces do, but can't quite work out how they assist in sorting out the relationships.
Thanks, Richard
|
|
|
|
|
I think it a classic design question.
Do you keep data "heavy" to optimize queries or do you keep the data light and don't really care about performance for queries ?
One other question to ask yourself is :
What is the most important basic information ? is it the artist ? the song ? the CD (for example) ?
Will you enter an artist and then all the song that he wrote ?
Will you enter a CD with the artist(s) and all the songs?
You have to remember that the data, in this particular case is really static; keeping cross reference should be easy.
|
|
|
|
|
Thanks, my feeling at present is actually that I need to design for the mosst likely scenario, then provide a dictionary for the alternative. I expect that the default view will be of songs with the artists as a part of them. I feel really quite unsure about the cross referencing as it would seem to create far too much dependency between the two objects. It may work perfectly well, but I could find that I add a further attribute to song which is similar in the future, for example what if I add a rating to the song, made by a user I then have a link through user to rating and through rating through to song. The cross referencing may then become quite complex if I want to do some association regarding user rating of artists.
I would rather then follow a common approach and provide a one way reference, with the return referencing created when a requirement develops to provide a dictionary type linking between user and artist.
Does this make sense or would you think that the cross referencing up and down the 'tree' is still an appropriate way, or is my second scenario completely different and therefore it be appropriate to employ two approaches.
I no that there are a lot of assumptions required here, but I would appreciate views on whether my thinking is sound.
Many Thanks,
Richard
|
|
|
|
|
is a songlist really a class or a collection of objects?
David
|
|
|
|
|
Why don't you look at providing a Many:Many relationship between the songs and artists. Ok, sure this is best performed back in the data, but you can then easily place a "LINK" class between the Songs and Artists. That way you don't have a true circular reference but you will reap the benefits.
Songs (1) ---> (M) SongArtistLink (M) <--- (1) Artists
I'm using this approach in our own in-house solution on a number of various entities. In the songs class, add a Readonly Artists property which references the "LINK" class. In the artists class, add a Readonly Songs property which references the "LINK" class.
We've seen no perceived or measurable performance penalties using this approach and because the link is done in "data" you can then query the data at any time using any of the available tools.
Hope this helps.
Edward Steward
edwardsteward@optusnet.com.au
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
