A Day in the Life of an IT Pro: Struth! A down webserver, Down-Under

In this ongoing series Kent Row (seasoned IT admin evangelist) reveals the trials and tribulations of an IT professional, he wrote his own bio:

Kent Row is a seasoned evangelist for IT admins at SolarWinds. He is at the forefront of a technological age of hosting, firewalling, trolling, tweeting, blocking and CTRL-ALT-DELETING.

He is fluent in both technical and non-technical liaison; an organisational guru who has averted more crises than you’ve stacked applications. Don’t let his excellent communication skills fool you though – he has his very own privacy policy and will maintain customer confidentiality at all times.

One of the perks of working as an IT consultant is the ability to travel the globe. Or at least speaking to people who have colleagues based in other countries, which is kind of the same thing, right? Pro Tip – IT guys in Australia do not like talking about Neighbours, Russell Crowe or ‘I’m a Celebrity Get Me Out of Here’.

There’s nothing like an international IT crisis to make me feel alive. Recently, I was asked by a customer to check his router configuration for his down webserver in Australia. After my initial thought of… “FLAMIN’ HECK!” I was suddenly confused as to why in the blazers we needed to check the router configuration for a webserver issue, with an overwhelming urge to question him on the matter.
He said he was able to ping the webserver, but in the browser he got a server response timeout.

IT detective mode: Engage

Like a good foot soldier he’d already checked the server’s CPU, RAM etc., and the webserver process, but all seemed OK. Reluctantly, I checked the router and the configuration seemed OK, besides no one had touched it in days… or so I thought.

I leant back in my arm chair, puffed on my pipe and had a good long think. What was I missing? I paced at length around the room, which often helps me unravel a conundrum such as this.
Just to make sure I wasn’t missing anything, I leapt towards my computer and pulled the router’s nightly backup configurations and selected a quick view of the current running configurations vs. yesterday’s.

As I expected, a couple of pink lines jumped right out, displaying a miss-match. Lo and behold, there was an Access Control List (ACL) blocking port 80 returns. I pulled up the Simple Network Management Protocol (SNMP) device details and there it was: a restart at 02:50. The ACL had been lying in wait in the startup configuration.
So the life lesson here is to always pay more attention to the running vs. baseline vs. startup change reports – it mitigates the issue where someone else makes configuration changes but neglects to tell anyone about it.

You can read Kent Row's other adventure "Emailgeddon" here, stay tuned the people's IT hero will return