|
What tech/architecture is recommended to handle a high amount of real-time data From Server to the clients?
Please note - Gathering data from Client to server is not difficult. We have ingestion points to collect the data.
But for sending data from Server to Client. Looks like it needs some good study and evaluation.
The cave-man idea that immediately comes to me is using Sockets and keep streaming data back.
Is there any recent frameworks/services available that does this best?
Google Cloud/Firebase has the options, we do use. But anything better out there?
Think of it as even a Share Market Application. Where you have to update all the connected clients every few seconds with Real-Time data This is not domain I'm into but the requirement is technically same. It's all about sending data in real-time.
Client - Web/Android/iOS
modified 26-Apr-20 10:00am.
|
|
|
|
|
Just to be sure I understand, you want to send lots of data from a server to the clients? Could there be more than one server doing this?
|
|
|
|
|
Sorry, just updated my question to be more clear.
It's Server to Client.
Greg Utas wrote: Could there be more than one server doing this?
Yes, but we could even discuss a single server scenario sending data to 1000 clients, to begin with.
To keep the use-case simpler.
|
|
|
|
|
I don't know which frameworks are suitable for this. If you have to start from scratch, try TCP. If it doesn't scale to meet your needs, look at RTP[^]. If message loss or reordering is acceptable, you could even go with basic UDP. It shouldn't be hard to change the protocol if your software is well structured.
You might find my article Software Techniques for Lemmings (link in .sig below) useful. It describes things that should generally be avoided in serious servers.
|
|
|
|
|
What about a multicast scenario? Would that work? Do all clients want the same data? I honestly don't know anything about multicast, other than there's such a thing. It might work for you, but you'd have to look at routing issues and maybe a way to resend when a client misses some data or goes offline.
Keep Calm and Carry On
|
|
|
|
|
Various protocols support multicasting. If you start investigating RTP, you will find related protocols for multicasting (streaming) video. They usually allow dropped frames, so it's a question of whether it's OK for clients to not have all of the server's information.
Off the top of my head, I would implement this with three threads:
- The first thread writes the data that will be streamed to clients into a large circular buffer.
- The second thread services the clients by reading from the buffer to
TCP-send each client its next batch of data. - If clients acknowledge reception so as not to lose data, the third thread uses
TCP-poll and TCP-recv to handle acknowledgments on the client sockets.
The buffer's start pointer advances over data after it has been streamed to, or acknowledged by, all clients.
Edit: You might want to look at the Robust Services Core (see link in .sig), an open-source framework that I developed. It could probably be used to develop a server that supports this kind of application. What's in its nb and nw directories would probably be adequate.
modified 26-Apr-20 11:06am.
|
|
|
|
|
|
Nand32 wrote: It's Server to Client.
Like a generic download, similar to downloading a DVD?
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
Use pull technology; "sending" / pushing rarely makes sense. You have better control of the "cycle" time, which you say in your case is "seconds".
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Cool. I'm also thinking of a hybrid idea. I've done this before.
The hint that the new data is available happens through "Push".
But the decision to fetch or not & the actual data-pull happens separately.
The notification mechanism will not be required if it's a consistently high-speed requirement like Sharemarket.
If the solution involves stopping data production intermittently, then a Push-notification mechanism to trigger start/stop could be useful.
Will try it out.
Thanks a lot.
|
|
|
|
|
Nand32 wrote: The hint that the new data is available happens through "Push".
I'm happily querying a webserver for a file by requesting its header along with the last datetime of my file. If my file is from the same date, it will only send a header back with a 403 (Not modified) (from cache).
If the clients need to be useable during the download, I'd go for the BITS-service that Windows uses to download its own updates. If the fetching of the data is more important than the clients responsibility, I'd search CodeProject for a download-manager and open 10 connections to the server from the client and have each download 1/10th of the file.
If speed is paramount then I'd recommend QNX, not Windows.
Bastard Programmer from Hell
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
|
|
I have an app that has 3 sets of features. The Agent, Dispatcher, and Admin.
Each's ones use the same Domain Models but the context of use is different.
And now, the Admin gets the Ticket Object that has data in it used by the Agent and the Dispatcher.
The app has a backend in Java and frontend in Angular. The communicate overt HTTP + JSON.
Should I separate them by domain, and create 3 separate microservices?
Would that be an overkill, as the app is not big?
Or should I keep the app as a monolith, and just reorganize the code in packages Admin Package, Dispatcher Package, etc... ?
|
|
|
|
|
Since all those features use the same bounded context there is really no need for splitting up the system in microservices.
Quote: and just reorganize the code in packages Admin Package, Dispatcher Package
This is nice idea. Sort of vertical slices.
|
|
|
|
|
I remember this from when I was a student, I'm sure my lecturer called it something like "state shadowing" (permutations of that name covered by the "shadow state" and "variable shadowing") - basically what I'm remembering is related to hardware programming where there's a time cost associated with querying the state of a connected hardware peripheral, so when setting the hardware state you also set a variable in-memory - that way whenever you need to know the hardware state you can just reference the variable.
Does anyone know what this kind of pattern would be called?
I'm looking for some inspiration on how to design something, so I'm just looking for any kind of related literature, but it's a bit hard to find anything when you don't know the name.
|
|
|
|
|
I would call it a cached result, but maybe some other term is in common usage among embedded system developers who work close to the hardware. Objects sometimes do the same thing to avoid calculating a result that clients frequently use.
|
|
|
|
|
A virtual machine.
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Hi,
We are starting a new project and are discussing about which architecture to use in our project.
We want to decide between Monolithic and Microservices architecture.
Just to give a overview of our project,
We have four functional modules, each with a specific functionality.
Each of these module is part of a process flow of the project.
After completion of each stage of the flow, that data will be used by the next module for further processing.
This will be a web application which can be used by anyone by registering.
Number of hits to this application may not be as much as with other applications, as it will be used only by people with specific expertise.
May be by around one million people.
Please suggest your thoughts on how to go about this project.
|
|
|
|
|
The only time "monolithic" works, is when making bricks. And you believe there is only one alternative?
It was only in wine that he laid down no limit for himself, but he did not allow himself to be confused by it.
― Confucian Analects: Rules of Confucius about his food
|
|
|
|
|
Hard to say, I don't feel like there is enough info. But it sounds like you could have 1 db that each process (.net service, azure function, console app, whatever it is) can then access.
Social Media - A platform that makes it easier for the crazies to find each other.
Everyone is born right handed. Only the strongest overcome it.
Fight for left-handed rights and hand equality.
|
|
|
|
|
Member 14786750 wrote: Which architecture
x86. Any other questions?
Bastard Programmer from Hell 
If you can't read my code, try converting it here[^]
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
My thinking is that microservice architecture brings a lot of operational complexity and inter-service communication (i.e. you should handle failures during communication gracefully, manage contract changes between services, etc). So you should consider them carefully not just use them as a new default.
I also feel that there are not enough details in your questions. But I think you should use microservices:
- if some of processing stage is more resource-consuming and you need to deploy multiple instances of it.
- if your processing stages belong to different bounded contexts or business domains.
If the goal is just to reduce code complexity then SOLID and vertical slices are your friends.
I guess the most sensible approach is to start with a monolith and extract microservices once needed.
|
|
|
|
|
Hi,
Thank you for sharing your thoughts. I will be sharing the same with my team and will take a call.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
