r/talesfromtechsupport

Me: Thank you for calling the IT help desk this is (My name). May I have your name name and ID?
Customer: Yes it's (Name and ID) so I sent an email on Friday but I haven't heard back from anyone.
Me: That's weird, I know Friday was a holiday but I we should have had someone working. I don't see any open tickets under your account. Did you happened to get an automated email with a ticket number?
Customer: No nothing is here
I check the mailbox and can't find email from this customer
Me: OK sir I just checked the inbox and I'm not seeing anything. It's possible the email didn't arrive or was moved, I should be able to help you though. What issue are you having?
Customer: Can't you see it in the email? I'm not able to sign into (name of company Website)
Me: No I never got the email. I know they recently revamped the homepage the sign in process is different. What error are you receiving when you sign in?
Customer: It's in the email I sent you.
Me: I'm not seeing the email it may not have gone through. If you want to you can resend it. Are email address is (email address). In the meantime since we're talking I should just be able to assist you with signing in what error message are you receiving? Is this an incorrect username or password, a site cannot be found error a blank page?
Customer: It's in the email. Why aren't you helping me. Can't you just see it?
Me: Sir I only have info you send me. and right now you are not giving me anything. If you like to I can sign into computer. What is the computer number?
Customer: It's a personal computer.
Me: OK that makes it more difficult. The software we use to connect is only for inside the company.
Customer: I don't understand Why you are not able to solve this. Would it help if I resend the email.
Me: Yes it would, can you resend the email.

Customer: I'll do it later.
Customer hangs up. I just checked the records today and they never sent the email

I’m on a small team of slightly deranged individuals responsible for providing world wide technical support for the dealers who sell my company’s products. Many of the technicians we support are competent individuals and a pleasure to work with. However, there are a few who don’t know which way is up.

So no shit, there I was. . . Sitting at my desk, minding my own business, and doing absolutely nothing unexpected, when Jake Tucker from Family Guy calls in.

Jake is working on a product that is essentially a bank of input cards used to monitor a variety of sensors and it is not going well. The sensor readouts are reporting Fire and brimstone coming down from the skies! Rivers and seas boiling! Forty years of darkness! Earthquakes, volcanoes... The dead rising from the grave! Human sacrifice, dogs and cats living together... MASS HYSTERIA!

The conditions outside are in fact somewhat better than that so we get to work straightening out the sensor inputs. It doesn’t take long before I realize there is something a bit off about what he’s telling me.

This type of device has a row of analog and digital sensor input cards that need to be configured using an online tool. The physical placement of these cards matters when you start configuring them through that tool. The input cards also contain several jumper blocks that need to be adjusted to accommodate the supply and input voltages. We quickly find out Jake has mounted the entire assembly sideways and has chosen to refer to the top as the bottom and bottom as the top.

This at least explains all our issues as the configuration assignments are all in the wrong places and the jumper blocks are a mess. I then inform Jake of the error and attempt to proceed with fixing it all. The problem is that Jake is entirely incapable of reorienting himself to view the assembly from the correct point of view and we are going nowhere fast.

I could have spent the next 3 hours painstakingly walking Jake through each step of the process and correcting each mistake multiple times but I had a far more elegant solution.

I flipped my manual so the bottom now faced the sky and proceeded to give Jake all his instructions upside down and backwards. Jake would then do the opposite of what I told him which resulted in everything being in perfect working order after 15 minutes. Sometimes it pays to behave like a Looney Toons character.

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.