I am having the worst professional day of my life and it will probably end with me getting fired.

I work as a systems admin for a small financial firm and made a bad decision today which cost my company a lot of time and $$$. If I am lucky I will just get a reprimand. What happened you ask? Well, I allowed an outside service technician to replace a faulty controller in our corporate storage array during business hours and naturally he caused it to crash. My company was without access to any of the critical data for over 2 hours.

Yes, yes, yes…I know, this should have been done after hours but I was assured by the vendor not once, not twice but three times that nothing would happen. The storage array is redundant in every way imaginable. It is practically unbreakable. Well guess what…it broke. I think I need to check the tech’s last name…perhaps it was Murphy.

Anyhow, pretty soon, I may have a lot more time to do my models ;-)

OUCH - Having been everything from IT Director to a cable monkey - I know exactally where you're coming from. Boy did that one go TU or CF

Well, I"m hoping for the repremand for you. Unemployment doesn't pay much.

Hi Martin!

Sorry for the bad news! But if I'm right you are not fired yet... so there is still hope. I think everybody as the right to have a second chance... no? Anyway, I wish you won't have that many time to model in the next days...

Jean-Luc

yikes... hope it turns out alright.

Well that isn't good. I can't imagine them letting you go for one mistake, no matter how critical. I make mistakes all the time. However, the attitude around here is "if you don't make mistakes occasionally it is a sign you're not trying anything new."
As long as the were informed decisions, incorrect choices have never been a problem.

Good luck. It must be tough sitting around waiting to find out if you still have a job.

Shaun

Well Marty, we may be in the same boat soon. I have a meeting on Thursday with a representative of the School's Human Resource Provider (Yeah I think it's a bloody stupid title as well). I think my face is out of favour because I refuse to be bullied by Senior Staff. Teaching ain't what it used to be! Good luck with yours Marty, meanwhile I'll put my angry head back on for Thursday!

Vinnie

Don't know how your company works, but unless you violated policy you were making an informed decision. I t kinda depends on how important the repair was. That in itself might have crashed the system if you hadn't made the decision to replace it. I'ld go in swinging. Years ago I had a manager pick an argument with me then haul me into HR trying to get me to fire off again so they could have a reason to terminate me. It turned out because of the tact I took, I put him on defensive on several issues. He had to totally back off and I eventually survived him when he got promoted up and out.

Good luck Marty. I can't imagine a company firing somebody for a single mistake, but I'm sure it happens. We'll keep a good thought for you.

You can always look for the other guy on the unemployment line.

Hell my place would have new IT people every month if that were the case!!!

Is it possible to claim damages from the vendor that fixed your controllers considering that they claimed nothing would happen? I'm pretty sure your company would not fire you.

Cheers

Ciao Marty,
Sorry to hear about that. Hopefully, you will just get a butt-chewing and be done with it. I haven't a clue about what you did (computer idiot I am), but it sounds like you were trying to do the right thing, and something went wrong. Good luck!
Ciao for now
Rob

Marty, hopefully you wont end up on the wrong side of the office door tomorrow... But I know how you feel... Earlier this year I took a service call late on a friday.. Drove 2 hours to the forklift and found the harmonic balancer had fallen off.. Ok, I cant do anything but order parts on Monday.. Following monday, cust calls wants truck running.. Call for parts, drive 2 hours south to pick them up and then 4 hours north, back to the site.. Install parts.. Job runs about 2000 bucks.. 2 weeks later, same cust calls and says, truck is broken again. We cart the truck to the shop this time. I take a look at it, same thing... Well isnt that interesting.. Look the truck over and find nothing out of the ordinary.. Replace all parts again under warranty.. About 1500 this time.. No extra labour involed for all that driving.
A few weeks go by, same cust, same problem.. Only this time.. Crankshaft is broken.. Oh boy.. Bring truck back into the shop.. Cartage company literally pulls a steer tire off the forklift after it hooks the edge of the bed. Trucks back in shop.. Pull the motor and order a new crank and bearings. Put it together and am checking the valve timing..
Something isnt right.. Do a leak test, found four bent valves. Pull the heads and have the valves done.. Put heads back on and rebuild motor. Bolting on the 3rd new balancer and the crank bolt stops.. And its not all the way in. Ok, Try to back it out.. nope, its not coming.. get it out finally and find the new bolt has stripped the threads outta the new crank. At this time, I sit down and have a smoke.. Now I quit smoking three years ago.. Pull the crank out and have it helicoiled.. Finally get this motor back together... All this over a 30 dollor motor mount...Final total on that job about 8000 dollors...
Next month, Im due for my annual review.. Talk about any issues from the previous year, raises, that kinda thing.. im thinking I might have to give back a buck or two an hour to cover that one...


Dave

Hmmm let me see: frank i need you to cut 92 inches of hot rolled, i end up cutting 92 inches of colled rolled. But a 4*4*1/4 tuber 69 inches, well i cut it 4*4*3/16!

Man, the worst was when 4 important pieces came back to my workplace for stupid mistakes... 22 inches in stead of 21 inches on a shaft for hte dump body, wrong measures, wrong material... everything all at once.... and they kept me! lol

i felt so miserable but they guys said" dont worry frank, days like these happen to the best of us"

Frank

Just out of curiousity how far into a new forklift is $8000.

This particular unit would cost about 30,000 - 35,000 new... So 8000 isnt really a stretch...

Dave

Thanks everyone for your concern. Luckily I am still employed. Later on today I will I will have to go through a series of question-and-answer sessions but I have a feeling they will be more of a "what can we do to avoid this in the future" kind of a a thing. Man, oh man I was really upset yesterday. I thought for sure I would be fired but I think that things will work out fine in the end.

Two lessons learned here:

1. Never, ever do any maintenance on production systems during business hours, no matter how insignificant it may be

2. Just when you think you have all the bases covered, something will definitely go wrong in the most inopportune moment.

Good news. A sound reaming and keeping your job is way better than the alternative...

Shaun

Glad things are looking beter today.

Gotta love Murphy.

3. Cover your butt, however be as open and honest as possible.

Jeff

well i am really happy that u still have ur job . We usually pay mistakes that are made by others and thats the risk of beeing in a position of taking decisions . I wish u the best my friend

awesome!

Thanks guys!

Glad things sorted themselves out. Next time things are looking grim, just tell yourself it could be a lot worse. If I am incompetent, it usually results in a visit from two guys in a green sedan.

Ciao Marty,
Great news for you!! I told you that you would probably just get a butt-chewing.
Ciao for now
Rob

When I was a supervisr and things went south, my attitude was "Fix it, solve it and lets keep on rolling. We don't have the time for finger pointing. Let's learn from this and figure out how not to let it happen again rather than worry who's fault it is" Sounds like your company has the right idea.