|
Hi there.
I am working on an application that shall be used as a local frontend to an online image database. The application shall display a list of images matching specific criteria as thumbnails and, upon click, will show the images in full size. The server has a list of newly added content which shall be presented to the user on startup, ideally showing every new content since the last synchronisation.
The problem is that a rather "small" number of images (~30 that is) will eat up several megs of disk space (~10 MB). I don't want to rely on the users having a broadband connection and also can't afford to let my application eat hunderts of megs of disk space, so apparently there is a need of some clever caching behaviour.
Now I have never done this before and don't even know how a regular web browser handles caching, so I need some information to get me started or even better, some ideas on how to solve this.
If it matters, I am working on .NET with C# since I hope that this will give me the possibility to port the project later to tge Mono platform.
Any help would be appreciated!
Regards,
Markus
|
|
|
|
|
From what I understand you will create a website showing a gallery of images. You will also create a Windows application that will access this website and show the images as thumbnails and full-size. If this is the case then there is no way around the connection speed problem - while thumbnails will be small (a few KB each) the full-size images will always be large and no amount of caching can solve this.
A web browser caches things by simply storing them on disk. If it hasn't been used in a certain amount of time it is then deleted.
|
|
|
|
|
Guess that question is answered. I thought caching would be somewhat more sophisticated ... well, it was early that day ... and the lack of coffee ... you know. 
From what I've seen so far, there are two major caching solutions for .NET; The ASP Cache and the Caching Application Block. What I need is a cache that is persistent over application restarts (basically the ASP cache isn't, but I think this could be easily implemented) and configurable during runtime - but I have seen no possibility to restrict the memory used on either one.
Any idea how this is done or could be done? Or is there any other cache implementation that features this already?
Thanks so far!
-- modified at 11:42 Thursday 19th July, 2007
Regards,
Markus
|
|
|
|
|
I'm looking at using one of these two to provide a means for our support people (not raw users) to create and maintain definitions for exporting data to a text file, to be imported into accounting packages. The prime candidate would be SSIS, but I fear that the expertise level would be a bit high their, so my other candidate is Reporting Services and RDL, to provide a user friendly WYSIWYG interface to design the layouts for text files, but how do I create a report that renders directly to a text file?
|
|
|
|
|
Hello all,
I'm new to the .NET universe and there for a little lost in the water.
I have a small task to complete and I found myself blocked and pressed for time so decided to ask for help. Any help will be greatly appreciated.
Here is my problem... as clearly explained as I can:
I have two MSSQL database servers set up that ensure redundancy and back for a data base used by several applications.
I also have a service set up that periodically checks to see witch server is up and sets an internal value according to the result of it's tests.
My task is to make all the applications interrogate this service to obtain the address of the current SQL server to use.
Applications are written in VB, C++ C# and are on variouse machines.
Anybody has any idea what would be the best way to do this ?
Thank you in advance.
|
|
|
|
|
The answer that springs immediately to mind here is to use remoting for this. I would have a remotable object in the service which would provide the information you need. Then, your clients would simply use remoting to get this information from the service.
|
|
|
|
|
A few colleagues of mine are in disagreement as to what is the best practice for coding boolean flags. Should you use the actual words "true" or "false" or use the integers "1" or "0" (or -1 in some languages) respectively.
I favor a boolean flag to show values as "true" or "false" because it makes reading 800 lines of code easier. 0's and 1's can be confusing, in my scenario, because we are constantly setting variables back to a regular integer values of 1 or 0.
Thanks!
-- Steve
[ Don't do today what can be done tomorrow ]
|
|
|
|
|
I believe you are right. In terms of readability if the language you are using actually supports the boolean type and the true,false keywords then it means is recommended to use them. In certain cases this will not be true. For example C did not have a book type until C99 came out and then C++ provided bool type and true/false keywords as a built in feature. If you do use C++ then it is recommended to use the true/false type since they are provided. Even in C , it was a good coding practice to #define the keywords TRUE and FALSE to 1 and 0 respectively to provide a more readable code.
Just my humble opinion 
|
|
|
|
|
Steve
I have to agree with you here. True/false are better, primarily because they are unambiguous.
Using 1/-1 or 0 causes confusion - is it a number or a boolean condition? What happens if I add them together?
|
|
|
|
|
Hi,
Of course I agree with the true/false syntax preference over 1/0.
I want to add one big caveat: in C integer values other than 0 and 1 will fail the
if (x==TRUE) test but pass the if(x) test, yielding some very nasty bugs.
So the recommendation might be:
1. to use !=FALSE rather than ==TRUE
2. to use IS_TRUE(x) and IS_FALSE(x) macros instead
3. to use FALSE and !FALSE rather than FALSE and TRUE (but not ==!FALSE)
None of these could be enforced however.
|
|
|
|
|
Thanks for the input...good to know that I am not the only one who thinks that way about Boolean logic. 
[ Don't do today what can be done tomorrow ]
|
|
|
|
|
Hi All,
I am looking for very efficient solution for this problem. Here is my problem description - I have serveral database,
from which I have to pull data (All databases differ in physical design implementation), and insert them in standardized
common database. Let me give you an example
Database - 1
Table - 1
Col_1 - int
Col_2 - text
Col_3 - float
Database - 2
Table - 2
Col_1 - int
Col_2 - int
Col_3 - float
Col_4 - text
Col_5 - text
Common database
col_1 (id) - int
col_2 - int
col_3 - float
col_4 - text
So mapping between database - 1 and common database would be
col_1 --> col_2
col_2 --> col_4
col_3 --> col_3
and between database - 2 and commond database would be
col_1 --> col_2
col_2 --> col_2
col_3 --> col_3
col_4 --> col_4
col_5 --> col_4
How can I achive this? One more thing, this mapping can be changed during runtime.
Thanks,
Ashish Patel
|
|
|
|
|
Well - for a start, you might want to try posting in only one forum. Go ahead, pick one.
|
|
|
|
|
Hi.
I wish to create a 'circular buffer' that one thread can write to, and another read from. The data is going into the buffer very fast, and there is a lot of it. I will probably use malloc() to create a buffer, however I wish to use a large buffer, but due to the speed the data is going in and out, anything that will slow the whole process will result in an incomplete stream being read out of the buffer.
I am very worried about using a too large buffer, and the memory being put into a page file. Could someone please inform me how likely this is, when it will happen, how I can prevent it, the largest memory allocation I can create etc?
Thank you all!!
|
|
|
|
|
For the record I am using VC++ (native). Also it looks as though I may have posted in the wrong category, sorry.
|
|
|
|
|
Hi,
please elaborate on the context: what is large ? what is fast ? and what kind
of data is it (byte oriented, record oriented, fixed record size...)?
|
|
|
|
|
The actual reading & writing is easy. The data is of fixed length, so no problems coding the buffer itself, I just really really cant afford any lost time due to paging or hard drive accesses.
The data will be stored in 13 byte blocks, with data being added at about 53mb per second. This data will simultaneously be read & written to hard disk, the aim is to go as long as possible before the buffer fills up & data/stream integrity is lost.
Thank you.
|
|
|
|
|
Hi,
there seems no need to have just one (large) buffer.
this is how I would go about it:
- pre-allocate a number N of memory blocks of 52KB each (yes exactly 52*1024 bytes);
- have two queues, one with empty buffers, one with full buffers;
- initialize with all the available buffers in the "empty queue";
- let the producer take an empty buffer, and fill it up completely, before
putting it in the full queue;
- let the consumer take a full buffer, empty it, and return it to the empty queue;
take care of performance by:
- not using function calls in producer/consumer other than the ones absolutely
necessary, including one queue_get and one queue_put.
- using a single disk I/O function that transfers an entire buffer at the time
(that's why it best is a multiple of 4KB); and use the lowest-level I/O function
available, in C that might be fread/fwrite or even read/write. Not sure.
- trying to keep the buffers in cache (although they probably would have to
be copied to/from memory to perform the disk's DMA); that's why I do not
insist on having large buffers nor many buffers; I would try not to exceed 1 MB
in total.
- possibly: not using a locking queue; a queue that serves one producer and
one consumer can de devised to lock only on extreme conditions (that is
full and empty), but not while putting/getting an item (use a circular buffer,
a get and a put index; only one party is allowed to write to the get index,
the other to the put index)
- possibly and somewhat tricky: play around with producer/consumer priorities:
if there are more empty buffers, raise the producer; if there are more full
buffers, lower the producer. Or something similar.
- experimenting with N: this is your one degree of freedom, you can add memory
to keep it alive longer (maybe with less cache efficiency), that is to overcome
longer disturbances due to external causes.
If you do this right, I would not be surprised you could make it run "forever",
since 53MB/s is well below today's disk bandwidth. The thing that probably
matters most is what else the PC is doing (networks, optical disks, etc; turn
them of as much as possible!).
I would not be concerned about the paging mechanism inside the PC; it knows
how to handle bigger jobs than this. You only would be using around 1 MB of
memory, and all your use is rather static and in multiples of 4KB, so
dont worry.
One final remark: I would first try a subset of the above, to get things
working, so I could observe how well it already is doing, then decide on further
improvements.
Good luck, and dont hesitate to post more questions, or results...
|
|
|
|
|
Thank you, this gave me quite a few ideas to think about. A previous attempt by someone else 2 years ago managed a run time of about 20 seconds with half as much data, I need to increase by at least 200% and double the datarate. The system will be the same to start with but I am going to try to get it upgraded.
Thank you once again.
|
|
|
|
|
I hope you're running it on a system with high speed hardrives or a RAID array. A 7200RPM drive only has a sustained transfer rate of 66mbps. You'd either need a raid array striped over 2+ disks (with a real hardware controller) or a 15k RPM drive to hit 106mpbs sustained. If you're doing reads and writes concurrently seek time will lower the maximum throughput farther.
--
You have to explain to them [VB coders] what you mean by "typed". their first response is likely to be something like, "Of course my code is typed. Do you think i magically project it onto the screen with the power of my mind?" --- John Simmons / outlaw programmer
|
|
|
|
|
Hi.
The system is currently using RAID stiping across 2 IDE disks. The idea is for a batch system so data is only going to be written to start with, reading will come after the job has finished (initially, depending on performance obtained).
I will be trying to get faster hard drives for the project, I'm not sure how fund are at the moment.
I am having a hard time obtaining sustained drive datarates, where should I be looking?
Thank you.
|
|
|
|
|
Mr Simple wrote: I am having a hard time obtaining sustained drive datarates, where should I be looking?
Dunno, IIRC the 66mpbs number is a benchmarking average. I'd assume performance of single drives would scale linearly.
--
You have to explain to them [VB coders] what you mean by "typed". their first response is likely to be something like, "Of course my code is typed. Do you think i magically project it onto the screen with the power of my mind?" --- John Simmons / outlaw programmer
|
|
|
|
|
|
Have you considered using shared memory? There are implementations of circular buffers using shared memory and they are really fast. If speed is what you need I thing you should consider this solution.
|
|
|
|
|
Hi,
I agree shared memory is fast, but I fail to see how a two-process approach
with shared memory would be preferable over a two-thread single-process approach
using local memory ?
With two processes, you need some interprocess communication (you might use
part of the shared memory for that), but you also have to take care of proper
startup sequencing, and the possibility one process exiting for whatever reason
without the other process knowing it...
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
