Cisco Live is a yearly gathering of Cisco customers, partners, and employees, eager to experience the latest Cisco technologies. Cisco engineers from throughout the company converge to create a pop-up city that enables content sharing, interactive labs, exciting demos and live streaming.
The show is powered by a dynamic network, the same network that enables businesses to run and thrive in this digital world. The team of individuals responsible for managing the network, which has grown every year as the size and complexity has grown, ranges from early-in-career associate systems engineers to experienced CCIEs (Cisco Certified Internetwork Experts), technical leaders and distinguished engineers. Many of these people return every year, bringing a richer set of experiences and feedback for improvements. Let’s explore the year-over-year evolution of the network engineering team, examining how the type of work has changed at each level.
When the team first began building out the network for Cisco Live in Europe, the Network Operations Centre (NOC) was a relatively small team and operated in a heavily siloed manner. Not because they hadn’t worked together before, but rather because the different tasks involved in the build-out didn’t overlap. The entire group consisted of routing, switching, wireless, access, data centre and network management teams. Teams knew their part of the network and did what they thought was best to accomplish their goals. There was no centralised automation, which meant a lot of configuration by hand and per device. Switches were unboxed, powered on, and configured via copy and paste. Complex Quality of Service (QoS) configuration on edge routers was accomplished by hand or via copy and paste
This manual configuration approach brought challenges that were compounded by the fact that the Cisco Live network was very dynamic, often needing last-minute changes either driven by stakeholder needs or venue oddities. When there was a problem, it often meant breaking out a console cable and running across the venue to troubleshoot. Network visibility was limited to the tools that were on hand. Because of the lack of strong collaboration, sometimes work was duplicated. The team was very reactive and, because there wasn't pervasive network visibility or strong holistic automation, this led to long hours.
In 2015, we needed to redesign the data centre for Cisco Live Europe. I led this project. If I had to add a new VLAN, it meant manually configuring four switches, four UCS fabrics, and 16 compute hosts. Carefully applying the necessary commands on all of the devices took a good 20 minutes – while the network was running. I didn’t want to ever do that again. So, since I had been spreading the message of automation at Cisco Live in various breakout sessions, it was time to practice what I preached. I began to build scripts that would automate the whole process.
With this goal in mind, I developed a set of Python and shell scripts that would automate the process of configuration. Parameters required to create an end-to-end VLAN were pushed to each device and were automatically validated. These scripts didn’t simply automate the command-line interface, they also enabled the data model-driven APIs embedded in the devices. This meant the configuration was applied more quickly and more reliably. At the 2016 Cisco Live, none of the configuration had to be applied by hand. Instead of taking 20 minutes to configure one VLAN, four new VLANs were created in two minutes.
This successful use of automation was addictive. The next year, since we were using a similar data centre architecture, I spent more time building scripts to monitor the health of the data centre, the services running in it and the network itself. I started to build Python scripts that gathered data using SNMP, device APIs, and application APIs; processed the data; and then pushed informative messages to various Spark rooms monitored by NOC staff. This allowed all of us to proactively mitigate potential issues, such as routing table changes, DHCP pool exhaustion, interfaces going into an error-disabled state and devices becoming unreachable. We also created a bot that would tell you where a given user, MAC address or IP address was in the venue.
Growth and change
Over the years, both Cisco Live and the network operations team have grown. Eight years ago, there were about 5,000 attendees and a team of 35 operations engineers. At the 2018 Cisco Live in Barcelona, there were about 15,000 attendees and a team of 70 network operations engineers, many of whom were new to Cisco and participating in the NOC for the first time. Each one of them played an important role in delivering an automation-centric, production-class network; many of them had feedback for how we can do better next year. It’s interesting to note that while the size of the event has tripled in eight years, the NOC team has only doubled. The capacity of the network, as well as the number of services, will continue to increase, and because of increased use of automation, the NOC team can more than keep up.
Network engineering in transition
Network interactions are changing - from manual configuration to driven by automation and orchestration and now highly intent-focused. However, what will not change is the need for engineers at every skill level. Junior engineers will use more web-based portals or Application Programming Interface (API) invocations to interact with the network. They’ll use the terminal less. Senior engineers will use web-based tools, too, but they will be simulating new network designs and architectures. They will also use APIs to build custom integrations that tie the network tightly to the core business, creating a true digital differentiator.
The role and contribution of today’s network engineers are more valuable than ever. Tasks are moving from manual to automated, and the workflow is focused on faster, more scalable delivery. Engineers are now capable of and expected to quickly create reliable network interactions and spend more time on higher-value network designs and business integrations.
The network operations centre at Cisco Live provides an example of how the role of the network engineer is transitioning. The tables below offer more detailed insights into the changing roles at each level.
Joe Clarke, distinguished services engineer, Cisco Systems
Image Credit: Flex