Welcome to the Lounge

OriginalGriff11-Mar-20 3:40

OriginalGriff

11-Mar-20 3:40

Yep. You'll be talking about a 10gigabit EAD leased line, with fibre optic direct to your building. Depending on where you are that'll be expensive. As in "Cheaper to build a new building somewhere else" expensive, I suspect. Laugh | :laugh:

"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
AntiTwitter: @DalekDave is now a follower!

dandy7211-Mar-20 4:44

dandy72

11-Mar-20 4:44

OriginalGriff wrote:
As in "Cheaper to build a new building somewhere else" expensive, I suspect

I've got it - move the building into one of the big cloud providers' data centers.

Save the bandwidth, the data never needs to get exposed through the public internet, and it's still cloud-based. Win-win-win.

OriginalGriff11-Mar-20 4:58

OriginalGriff

11-Mar-20 4:58

That's brilliant!
Solves all the problems with the cloud I can think of... Thumbs Up | :thumbsup:

"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
AntiTwitter: @DalekDave is now a follower!

dandy7211-Mar-20 6:18

dandy72

11-Mar-20 6:18

In hindsight, I should've just called it "redefining what 'on-prem' means".

RickZeeland11-Mar-20 2:09

RickZeeland

11-Mar-20 2:09

The term "Render farm" springs to mind: Render farm - Wikipedia[^]
How To: Building Your Own Render Farm | Tom's Hardware[^]
$Shucks | :-\$

Nand3211-Mar-20 4:08

11-Mar-20 4:08

Sounds funny, but they are definitely talking something similar. But I also feel if this is all about testing our solution/architecture abilities. The easiest answer I had was to try HPCs on Azure. $Shucks | :-\$

lopatir11-Mar-20 2:19

lopatir

11-Mar-20 2:19

just because doing graphics processing what's the need for so much storage?

sounds like somebody got suckerd by the sales goons, or just some idiot trying to impress with big number-words.
for instance city-wide traffic monitoring systems that record plates etc don't need that much, city facial recognition systems don't need that much.

1. the figures quoted are just nonsense, throw those away
2. get rid of the goons that came up with that crap, clueless / out of their depth / making irrational crap up
then:
3. be more specific on the nature of processing, volume and retention to get useful recommendations
-- without that it's just other peoples guesses.

pestilence [ pes-tl-uh ns ] noun
1. a deadly or virulent epidemic disease. especially bubonic plague.
2. something that is considered harmful, destructive, or evil.
Synonyms: pest, plague, people

musefan11-Mar-20 2:28

11-Mar-20 2:28

Depends what the system does though. CCTV for example would require the data to be kept, as least for X amount of time. No point in having CCTV if the video isn't there to review when you need it.

Imagine also a city-wide service that takes peoples faces and puts a hat on them. Then at any point in time, a citizen can log in and see how good/bad they look in a hat... gotta store all those images to do that. Silly example, but you see my point Big Grin | :-D

Amarnath S11-Mar-20 2:52

Amarnath S

11-Mar-20 2:52

Yes, 0.5 petabytes per week seems too high. For example, I had visited the Nuclear Medicine department of a Cancer hospital, where they have four PET-CT machines, each spewing out 300 MB imaging data every 15 minutes or so (say 5 GB per hour, with all the four machines running). With the hospital working for 12 hours every day, we have 60 GB accumulated from that one department alone on a single day. Then, the CT and MR machines themselves spew out comparable sizes of data. So, we have about 150 GB per day from all departments put together. Taking 6 days as a week, we have about 1 TB per week.

Much less than 0.5 petabytes per week.

musefan11-Mar-20 3:57

11-Mar-20 3:57

Amarnath S wrote:
I had visited the Nuclear Medicine department of a Cancer hospital

Using big scary words doesn't equate to big scary amounts of data. You seem to suggest there should be a correlation between importance of software and the amount of data it produces. That would probably make Youtube one of the most important bits of software in the universe Laugh | :laugh:

Nand3211-Mar-20 4:09

11-Mar-20 4:09

Bingo.

lopatir11-Mar-20 4:41

lopatir

11-Mar-20 4:41

0.5 petabtyes per week, and you want to process and store it in the cloud

ok on a 10 gigabit link (which you won't have) it'll take around 937 days to upload 0.5 petabytes

yes, you would need 937 days per week to upload the weeks data
-- The fastest link currently available is 18.2Gb (South Korea), still looking at >400 days
-- 5g promises 100Gb, that's >90 days per week

your figures and information are just ridiculous, just pathetic.
clearly whoever is coming up with those is totally clueless
and you want to put it on the cloud or the clown???

don't care you crossed it out,

asking people for advice?
provide real and proper information, not this bogus crap

BINGO THAT!

pestilence [ pes-tl-uh ns ] noun
1. a deadly or virulent epidemic disease. especially bubonic plague.
2. something that is considered harmful, destructive, or evil.
Synonyms: pest, plague, people

Nand3211-Mar-20 5:05

11-Mar-20 5:05

I never thought that a Petabyte word could have troubled someone so much.

Please Ignore the message if it did not interest you. Just like I'm doing for your message now.
And go home and have a chill beer on my name Smile | :)

Bingo that? Smile | :)

musefan11-Mar-20 5:48

11-Mar-20 5:48

Maybe I am doing the numbers wrong but I get 4.6 days to transfer 0.5 PB at 10Gb/s.

0.5 PB = 500,000 GB
10Gb = 1.25 GB

500,000 / 1.25 = 400,000 (seconds)

400,000 / 86400 = 4.63 days

Nelek11-Mar-20 10:20

Nelek

11-Mar-20 10:20

Are your 10 Gb/s GigaBytes? Or GigaBits?

M.D.V.

If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.

musefan11-Mar-20 21:14

11-Mar-20 21:14

10 Gb = 10 Gbits (hence small b)
1.25 GB = 10 GBytes (hence big B)

Nelek11-Mar-20 21:25

Nelek

11-Mar-20 21:25

I now realize the capitalization in your previous message... (it was late at night, when I wrote)
but I think your comparisons in this message are mixing them up as I I did yesterday Wink | ;)

musefan wrote:
1.25 GB = 10 GBytes (hence big B)

I think you wanted to say
1,25 GB (big B) = 10 Gb (small b)

M.D.V.

AndyChisholm12-Mar-20 5:41

AndyChisholm

12-Mar-20 5:41

I was Solution Architect a couple of years ago for a system that stored 2PB per week of video and telemetry data from self driving car development.

You are correct that that is a large amount of data, roughly 3.6 GB/s if you think of it as a constant stream. In addition to the challenge of simply storing it, it also has to be simultaneously backed up and analysed.

Data arrive at the data center from the many cars in the field not via cables but on SSD based cartridges that have to be read in via reader stations. The cartridges are then returned to the field to collect more data.

Storage for production and "backup" is provided by hundreds of Dell-EMC Isilon NAS nodes. Analysis of the data is done in more than 300 servers each with 32 cores and 0.5 TB of RAM.

It sounds mind blowing but there really are companies collecting, storing and processing data on that scale.

Andy

OriginalGriff11-Mar-20 6:08

OriginalGriff

11-Mar-20 6:08

musefan wrote:
That would probably make Youtube one of the most important bits of software in the universe

And everybody knows that's FarceBok.

"I have no idea what I did, but I'm taking full credit for it." - ThisOldTony
AntiTwitter: @DalekDave is now a follower!

Forogar 11-Mar-20 4:27

Forogar

11-Mar-20 4:27

I concur. I developed a system for a hospital that recorded all activity from around 70 cameras 24/7 and it never went above needing about 800GB. Certainly less than a terabyte, never mind petabytes!

- I would love to change the world, but they won’t give me the source code.

musefan11-Mar-20 2:22

11-Mar-20 2:22

Is the application 3rd party? If so, ask them for recommendations on how to balance workload across machines. Most applications that have high processing requirements will support this kind of shared workload scenario.

If it's in house, then the developers should have a pretty big say in the best way to maximise performance.

Any serious amounts of image processing are best done across multiple machines, but requires the application to support it.

Storage is a separate issue, so design it as so, you only really need to make sure the connection between processing machine(s) and storage machine(s) is fast enough to keep up. Other than that, it's 2 separate requirements.

Nand3211-Mar-20 4:15

11-Mar-20 4:15

musefan wrote:
Is the application 3rd party?

The client is a subsidiary to one of the top Oil & Gas industry. They want to work with us for building the application. They've hired people from AMD on their side. I guess this is just for the hardware department. & They also own the AI/ML teams. We are just focusing on the application that collects data.

Now most probably, as I've updated on my OP, the data seems to be fairly huge. But the intent of the contact person from this company looks to be testing our capacity. He's watching if we'd run away looking at the scale of the application. We did not run, because we don't know what it means to handle Petabytes of data. Laugh | :laugh:

W∴ Balboos, GHB11-Mar-20 2:24

W∴ Balboos, GHB

11-Mar-20 2:24

Considering the data requirements you need I suggest the following storage system:
1 - transport layer[^]
2 - storage[^] (note hack-proof encryption in progress)

Ravings en masse^
"The difference between genius and stupidity is that genius has its limits." - Albert Einstein
"If you are searching for perfection in others, then you seek disappointment. If you seek perfection in yourself, then you will find failure." - Balboos HaGadol Mar 2010

musefan11-Mar-20 2:30

11-Mar-20 2:30

I have the same system in place for work emails, and can confirm it is very effective Thumbs Up | :thumbsup: