Click here to Skip to main content
15,867,488 members
Home / Discussions / Algorithms
   

Algorithms

 
AnswerRe: Lock free algorithms Pin
Michael Gazonda31-Jul-14 18:51
professionalMichael Gazonda31-Jul-14 18:51 
GeneralRe: Lock free algorithms Pin
Joe Woodbury31-Jul-14 19:21
professionalJoe Woodbury31-Jul-14 19:21 
GeneralRe: Lock free algorithms Pin
Michael Gazonda31-Jul-14 19:25
professionalMichael Gazonda31-Jul-14 19:25 
AnswerRe: Lock free algorithms Pin
SledgeHammer016-Sep-14 8:58
SledgeHammer016-Sep-14 8:58 
GeneralRe: Lock free algorithms Pin
Joe Woodbury6-Sep-14 9:50
professionalJoe Woodbury6-Sep-14 9:50 
GeneralRe: Lock free algorithms Pin
SledgeHammer016-Sep-14 11:35
SledgeHammer016-Sep-14 11:35 
GeneralRe: Lock free algorithms Pin
Joe Woodbury6-Sep-14 11:49
professionalJoe Woodbury6-Sep-14 11:49 
GeneralRe: Lock free algorithms Pin
SledgeHammer016-Sep-14 12:55
SledgeHammer016-Sep-14 12:55 
Joe Woodbury wrote:
What, again, was your definition of scalable? Smile | Smile | :)

I meant if your task was to deliver that kind of scale from a home grown solution in a few weeks, that would be tough. If you managed to do that, that's very impressive.

Joe Woodbury wrote:
Actually, I have, though the use was very narrow and specialized. Again, our testing found that all the commercial solutions had performance bottlenecks or just plain fell apart when trying to use them with massive amounts of data.

BTW, in one app, switching from ZLib to LZ4 was a significant performance gain that overwhelmed any decrease due to slightly larger message sizes.


Yes, there are performance penalties due to all the configuration options.

A slight aside on this topic... we found with our testing that sending large payloads through a message queue (and we originally had a home grown solution as well) is very inefficient.

Say for example, your payload is 1MB files.

Your producer needs to:

1) load the file / deserialize it (or just load it as a binary blob)
2) possibly serialize it back to a binary blob, compress it
3) post it to a queue (whether your own or a commercial one)
4) possibly do some preprocessing on the blob before sending it over the wire

The client needs to:

5) get the large blob
6) decompress it
7) deserialize it
8) do the work

It is more efficient to put the payload on a file share and in the payload just send the UNC path to the file share. This way you get rid of the whole compression / decompression step / multiple serialize / de-serialization steps.

In our project, we have to process 4 BILLION 1MB payloads, so...

ANYWAYS... back to your original question...

Whatever you use, the producer "send" should be instant fire-and-forget and this includes if it needs to get a response. It should do that in an async callback (push notification). There should be pretty much 0% CPU usage when the producer is idle and it should never be blocked. If it has something to submit, it should be able to at any time. The server needs to be able to deal with a flood. Unless your server has a ton of memory and you are guaranteed the clients can keep up at any load, you'd need to have a backing store of some kind when the memory gets filled up.

As far as the client side goes, it sounds like you are polling your circular queue for work. NOTHING in your design should be polling. Polling is a big no-no. Everything should be async push notifications. Sleep / poll is not a good design. Should be push / WaitForSingleObject. WaitForSingleObject does not consume any CPU resources.
GeneralRe: Lock free algorithms Pin
Joe Woodbury6-Sep-14 13:11
professionalJoe Woodbury6-Sep-14 13:11 
GeneralRe: Lock free algorithms Pin
SledgeHammer016-Sep-14 13:37
SledgeHammer016-Sep-14 13:37 
GeneralRe: Lock free algorithms Pin
Joe Woodbury6-Sep-14 13:44
professionalJoe Woodbury6-Sep-14 13:44 
QuestionAlgorithm for comparing word with randomly distributed substring Pin
asdf2321130-Jun-14 23:06
asdf2321130-Jun-14 23:06 
AnswerRe: Algorithm for comparing word with randomly distributed substring Pin
Sanmayce3-Jul-14 7:15
Sanmayce3-Jul-14 7:15 
AnswerRe: Algorithm for comparing word with randomly distributed substring Pin
Sanmayce5-Jul-14 7:19
Sanmayce5-Jul-14 7:19 
QuestionCubesort Pin
Igor van den Hoven22-Jun-14 4:09
Igor van den Hoven22-Jun-14 4:09 
AnswerRe: Cubesort Pin
Sanmayce28-Jun-14 8:02
Sanmayce28-Jun-14 8:02 
GeneralRe: Cubesort Pin
Igor van den Hoven29-Jun-14 9:58
Igor van den Hoven29-Jun-14 9:58 
AnswerRe: Cubesort Pin
Sanmayce30-Jun-14 6:48
Sanmayce30-Jun-14 6:48 
GeneralRe: Cubesort Pin
Igor van den Hoven12-Jul-14 4:01
Igor van den Hoven12-Jul-14 4:01 
AnswerRe: Cubesort Pin
Sanmayce3-Jul-14 4:30
Sanmayce3-Jul-14 4:30 
QuestionReduce a Q2SAT formula Pin
Apurvgupta15-Jun-14 20:48
Apurvgupta15-Jun-14 20:48 
QuestionFastest textual decompression in C Pin
Sanmayce10-May-14 8:54
Sanmayce10-May-14 8:54 
AnswerRe: Fastest textual decompression in C Pin
Richard MacCutchan10-May-14 21:46
mveRichard MacCutchan10-May-14 21:46 
GeneralRe: Fastest textual decompression in C Pin
Sanmayce12-May-14 0:18
Sanmayce12-May-14 0:18 
GeneralRe: Fastest textual decompression in C Pin
Richard MacCutchan12-May-14 1:24
mveRichard MacCutchan12-May-14 1:24 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.