Welcome to the Lounge
For discussing anything related to a software developer's life but is not for programming questions. Got a programming question?
The Lounge is rated Safe For Work. If you're about to post something inappropriate for a shared office environment, then don't post it. No ads, no abuse, and no programming questions. Trolling, (political, climate, religious or whatever) will result in your account being removed.
|I used to think that backups could be thought of as being overly cautious, but please read the following, totally true, cautionary tale.
Way back when (90's) our company decided with great wisdom to out-source our email. They chose a company that was well-known for telecoms at that time - I shall call them Cable & Hopeless (C&H) as they still operate today, in some remoter parts of the world.
While I was happy to let our department's emails be managed and stored elsewhere I was less than happy about trying to access 30 Outlook mailboxes over a 64 kb/s ISDN line. I therefor implemented a policy of setting up Outlook to keep a local copy of the inbox and seting up individual local folders for all users on a local server. This was immediately criticised by the upper powers as they couldn't access the entirety of my users e-mails. (My immediate boss quickly informed them that it was illegal to do so and that all our mail was top secret anyway, so they could take a walk).
Fast forward a couple of years, my system is humming along nicely despite Outlook's attempts to corrupt .pst files if there was the slightest network time-out. I had GFS backups of everything mail-related and the world was bright and shiny, until... "Mail server is off-line".
Day 1: We contacted C&H who said "don't worry, it looks like a disk failure but it's RAID 5 so you'll be OK. Please wait while we swap out the disk and the array rebuilds".
Day 2: "Erm it looks like more than one disk has failed, we can't understand it, we're trying to rebuild with 2 disks" (I start to hear alarm bells)
Day 3: "Erm well it appears that you can't rebuild a RAID 5 array when 2 out of 3 disks have failed"
"So this is the 2nd disk to fail in the array then?"
"Looks like it"
"So why didn't you replace the first one when you got the alert?"
Day 4: "It looks like the alerts were set up to email someone who has since left the company. Anyway we're re-installing your Exchange Server and will be able to restore from backups."
Day 5: "Please be patient, we're looking for the best backup."
Day 8: "Erm, it looks like all our backups are blank! Anyway your server is up and running and we're adding accounts as we speak"
Day 9: "Apologies but it appears that we forgot to plug the SCSI cable in to the tape drive, there have never been any backups of the server!"
"Didn't you get any alert messages?"
wait for it ...
"It looks like the alerts were set up to email someone who has since left the company."
My 30 users were very happy to find all there past and present emails available to them. The rest of the company (200 accounts) had to grin and bear it.
At a later stage the powers that be decided to bring email back in-company, even then they still managed to omit to back up Active Directory. This blew up in their faces when a trainee with admin privileges hit "Delete" on the main tree and then tried to fix it himself.
Guess who did have a backup?
(Finished blowing my own trumpet, but don't forget the first law of IT, "If anything can go wrong, it will do so in the worst possible and least expected manner")
So old that I did my first coding in octal via switches on a DEC PDP 8
General News Suggestion Question Bug Answer Joke Praise Rant Admin
Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.