A Day in the Life of an IT Pro: How VoIP ruined Valentine’s day

In this ongoing series Kent Row (seasoned IT admin evangelist) reveals the trials and tribulations of an IT professional, he wrote his own bio:

Kent Row is a seasoned evangelist for IT admins at SolarWinds. He is at the forefront of a technological age of hosting, firewalling, trolling, tweeting, blocking and CTRL-ALT-DELETING.

In this blog Kent Row shares an "anonymous" guest post from a love struck IT Geek

Last year on Valentine’s Day, not content with causing problems in my work life, the IT support profession wreaked havoc on a very important date with a very special lady friend.

I met Sandra from HR while fixing her flickering screen. She admired my technical prowess and offered me a Malteser chocolate as a means of thanks, which was very kind. It would become clear in the coming weeks that Sandra would purposely forget her password several times so that she had an excuse to see me. Normally I would find this to be an irritant of the highest order, but Sandra had a very nice way about her, and I was keen to get to know her better. With February 14th approaching, I built up the courage to ask her out on a date.

I booked a table at a little Italian restaurant around the corner from the office, and, dreaming of cannelloni and canoodling, walked to work with a spring in my step. Sandra had insisted she wanted to video call me at lunchtime, testing out the new VoIP system I had been tasked with installing that week, to discuss our plans for the evening.

[caption id="attachment_112748" align="aligncenter" width="497"]

Love is in the air

Sandra loves facts[/caption]

Now, the company I worked for had a large distributed network, disaster recovery sites and a huge data centre; the CIO was keen to ensure that we kept up to date with the latest technologies, from telepresence to BYOD. We implemented a new Internet Small Computer System Interface (ISCSI) storage system and decided that we should backup all business data to the data centre at the end of each working day. Configuration changes were made and the network seemed fine.

As lunchtime rolled around, the helpdesk was suddenly flooded with tickets about VoIP call drop outs. Not only did this mean I missed that all important call from Sandra, but I was unable to reach her to tell her why. Frantically trying to solve the problem, we took a look at the traffic between the WAN and MPLS links of the branch routers using NetFlow data. Looking at the WAN exit link conversations, we immediately saw what went wrong. The backup job traffic was being put under scavenger class QoS to keep it from affecting business traffic. Unfortunately, the network guy who configured the ACLs to match backup traffic had set a rule to capture VoIP under scavenger class. Backups ran under Best Effort whereas VoIP got squeezed out, causing drops. DSCP QoS markings in the NetFlow reports saved us, otherwise troubleshooting would have been painful. By 2pm, the network was back on track, but my relationship with Sandra was experiencing duplex mis-match: While my attraction was set to FULL, hers was very pointedly running at HALF (or less).

The great news in all of this is that for IT Pros managing VoIP, so long as we have effective monitoring tools, the reports will reveal all. And all was not lost for me either. The rest of the afternoon was pretty quiet, and I managed to do enough groveling to Sandra to ensure the date went ahead. More than one lesson has been learnt, and I am a wiser man for it.

You can read all about Kent Row’s other adventures “Emailgeddon”, and “A Down Webserver, Down-Under“, and "When pre-historic laptops go rogue on the corporate network".

Image credit: Berk Club