|
Been there, done that! I think it's time i got the t-shirt. 
|
|
|
|
|
Nobody will remember:
- Your salary
- Your fancy title
- How "busy" you were
- How stressed you were
- How many hours you worked
People will remember:
- Your commit that caused a production issue.
And I did just that this morning, immortality is mine!
|
|
|
|
|
 you plagiarized that to an extent.
Decades ago I worked in a shop that had a rule - you shut down the wrong server, you buy lunch for the team. There I am rebooting my workstation, except I was logged in to the main server with 2k+ users on it.... yeah, I bought lunch.
Charlie Gilley
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
Has never been more appropriate.
|
|
|
|
|
|
A friend of mine was dogged by slow Internet, well below the service levels he paid for. Since it was DSL based he figured it was phone company nonsense, and was an ongoing battle with the phone company over his service. He couldn't or wouldn't switch to cable, although I don't remember why.
Anyway, eventually it comes up in conversation that he has a Windows 2008 Server machine that is Internet facing, running out of his home network.
*headdesk*
As soon as he said that, I was like "here's how to fix your Internet"
Sure enough, half the network was basically a botnet for Kevin only knows who.
He doesn't run servers out of his home anymore.
To err is human. Fortune favors the monsters.
|
|
|
|
|
honey the codewitch wrote: He doesn't run servers out of his home anymore.
Maybe he just shouldn't run server anymore, period 
|
|
|
|
|
Correct. 
To err is human. Fortune favors the monsters.
|
|
|
|
|
I entered kill 0 (instead of kill 9) on a SunOS server one time and it failed to come back up. The problem is it was a core server for a large number of administrative people. Turns out it had previously corrupted the file system. It was a week before it was available again.
|
|
|
|
|
Also decades ago I worked for a rather large broking company. Sysadmin could not work out who owned a particular server (there were a lot of servers) so he shut it down to see who screamed. It turned out to be a company that had been sold off 4 years before and moved 2 floors down.
Never underestimate the power of human stupidity -
RAH
I'm old. I know stuff - JSOP
|
|
|
|
|
Ah, the 'ol scream test. An oldie but a goodie. But, potentially career limiting if the wrong person screams.
|
|
|
|
|
I've done it more than once. The screams usually come pretty quickly.
|
|
|
|
|
I've had to do this as well since no one seems to know what their resources are.
|
|
|
|
|
One chap I was called to help around 25 years ago explained to me that every time he pressed a key on his keyboard his computer would restart.
So I went over to his desk and observed what was happening and yes the computer did shut down every time he pressed a key on the keyboard.
Now my Sherlock Holmes senses kicked in as I noticed that he also had a folder resting on his lap.
Every time he leaned forward to press a key, on the keyboard, the corner of the folder pressed the restart button on his computer...
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
modified 26-Aug-22 3:43am.
|
|
|
|
|
I once deleted ALL of the car loans for FL because someone changed the "mechanical" terminal Switch,
and I was NO LONGER on the DEV box.
OUCH!
A month later, someone else did a different state. And I patched the operating systems for production to block the command (renamed it to another command we would NEVER type accidentally)...
|
|
|
|
|
|
|
I accidently tested a database shutdown script I was developing on the PROD system, but who hasn't done that?
One day I found out that a DEC AlphaServer 1000 has a switch which kills power if the case is opened while the system is running. I mean, WTF? Definitely over-engineered. All I was doing was determining whether or not there were any spare RAM slots. : sheesh : At least that was a DEV system.
|
|
|
|
|
Hmm,
Done both of those! 8)
Worst for me was - on a production system - deleting an 'old' copy of a database from the backup filestore during a very busy morning. You guessed it - not the 'old' copy at all! This was all long before SQL (or even Linux) was a thing on PCs.
Had to restore from the overnight tape backup (we did have backups, and they were regularly tested, fortunately) and then get the reservation staff to go through all the paper printouts of that day's bookings to recover the mornings transactions, whilst suspending new ones to guard against double bookings. (Fortunately I also produced a paper copy of every transaction as - back in the day - neither the power nor tape backups were 100% reliable).
I was not Mr popular that day, I can tell you!
I learnt my lesson though and added an hourly local backup to a .zip file of transaction data just in case. That saved my bacon a few times too!
|
|
|
|
|
|
What the ![Elephant | [mastadon]](https://codeproject.global.ssl.fastly.net/script/Forums/Images/elephant.gif) !? 
Was this guy fired?
Did he die in a freak "accident"?
This is closer to sabotage than to ignorance 
|
|
|
|
|
Sander Rossel wrote: Was this guy fired?
Nope. The guy who did it, just did what he was ordered to.
The explanation is: big OEM company.
It is not that the right hand doesn't know what the left hand is doing. There are many times when not even the fingers of the same hand know what the others are doing.
M.D.V. 
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Back in the mid-80’s at a former employer who did photo processing they did a major upgrade of the DEC PDO-11/70 systems located in their multiple film processing plants to Microvax. Each of the new Microvax systems was setup with two large hard drives with one being mirrored on the other. They were hot swappable so if something went wrong with one drive they could stop it and keep going with the other one while the bad drive was sent for repairs. So one morning at work we were alerted to a problem that a bunch of invoices were missing in one of the plants. It turned out that at this particular plant they had to get a hard drive repaired and when it came back a few days later they started mirroring the drives again only someone (not me, thankfully) did the mirroring in the wrong direction so the drive that had the old data overwrote the drive with the current data. I’m not sure what they ultimately did to get the data back since I don’t think there was a tape backup because of the disk mirroring (I think it was probably the same person who thought they no longer needed the redundant DECNet cross links between the plants) but I’m thinking one of the devs had to write something so some poor workers in the plant could do a bunch of data entry.
|
|
|
|
|
Classic!
Actually iTunes did that to my iPod once.
First (and last) time I used it.
Don't even own an iPod anymore now.
No Apple a day keeps the doctor away
|
|
|
|
|
Congrats on your immortality! And welcome to the club
|
|
|
|
|
I was immortalized many years ago actually
You don't get to be in IT for twelve years without accidentally deleting a table or two in production (well, I never did that, but locking an entire database was pretty bad too) 
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
