Welcome to the Lounge

1. It is just possible to handle this amount of data with a dedicated 10 Gbps connection (the actual data rate is 6.6 Gbps), but once you take into account framing, collisions, etc., it looks very iffy.
   [Probably have multiple systems receiving the data]
2. The interfaces (NVMe, etc.) can handle this data rate, but building a storage system that can handle this sort of sustained write rate is non-trivial.
   [Probably use multiple disks running in parallel]
3. Once you have the data stored locally, you must read it off the storage at the same rate (otherwise you will eventually run out of space), process it, and store it somewhere else.
   [The initial processing of this much data would presumably require a massively parallel system, with all the communication and synchronization issues that this entails. Have at least one primary processing node for each receiving system]
4. How will secondary, tertiary, etc. processing be done?
   [Whether you have one secondary processor for one or more primary processors or vice versa depends on the amount of data and the processing required. Again, we have synchronization and communication issues]
5. Presentation of the results?
   [Presumably requires that the results of the processing be sent to a single node. Synchronization, communication issues...]

1. Does all the data from all around the World end up in single data center?
2. Does this data center do all the processing?
3. Is this really needed or can processing be distributed around the World?
4. Sure at some point of time you may need all your data in one location for some kind of analysis. But does this have to be real time? Do you need "raw" data or processed data from remote servers can work fine?

Jörgen Andersson wrote:

Well, you obviously need to go distributed and pre-crunch the data locally, before sending it to regional servers for analysis.

Quote:

OT: How are you doing?

Quote:

What's life like under lockdown?