Shifting to automated notifications and alerts for IT incidents

When CIOs, CSOs and other enterprise decision makers think about mass notification systems, it tends to be in the context of a non-recurring “emergency” event such as an evacuation, severe weather, active shooter or terrorist threat, when the ability to deliver real-time alerts to employees and other individuals is critical.   

But intelligent notification systems are increasingly becoming a core tool for enterprise CXOs seeking to not only keep employees safe, but to ensure reliable risk management and business continuity of operations – including crisis management, IT service management, corporate communications, customer relations, supply chain management, event management, or any area that needs reliable two-way notification for groups from one to many thousands. In particular, forward-thinking businesses are expanding their perspective to look at how automated IT incident management alerts and notifications can enhance day-to-day business operations and productivity. 

Growing IT need for automation  

Today’s IT environments are more complex than ever before, placing a whole new set of demands upon IT managers—not the least of which is the need to accomplish more with fewer resources. These pressures demand a holistic approach to the identification of use cases for automated notification and how IT can benefit from the notification enablement for key processes.   

As long as there are events that interrupt the status quo, there will be a need to inform those who are affected by the interruption, as well as those who can get things back on track. Nowhere is that more evident than in the modern IT department.       

The use of technology to reach key IT personnel started with the advent of pagers many years ago. The adoption of “rotational pagers” meant there was often a need to have one individual make sure the next on-call staff individual received the pager when responsibilities transitioned from one to another. While one-way and two-way paging for IT staff has been in use many years, their capabilities are much more limited than the technologies available today for automated notifications. 

To address the growing demand for quality delivery of services from IT, the operations team must rely on tools to support IT Infrastructure Library (ITIL) processes within the organization. Some of the most prevalent tools are based around IT Service Management (ITSM), like incident tickets, helpdesk issues, etc. In many cases, multiple ITSM tools are used to address some of the same issues for IT operations. While manual notification solutions can assist in the process, IT organizations typically don’t have the time to identify an incident, then manually launch and monitor responses from a notification solution. This scenario drives the need for automated integration between service management tools and notification solutions, a need that is not exclusive to ITSM departments.

Shifting to automated IT alerts and notifications  

While automated notification cannot solve all the challenges associated with IT management, there are many IT issues that can benefit from the integration of a notification solutions throughout ITIL processes. IT operations and management have the benefit of being a service in high demand, and with that comes the added importance and expectation from stakeholders for consistent and timely communications in the event of an IT incident or disruption of services. 

Notification enablement in this type of environment follows the same principles of planning for messages about general business interruptions or crisis communication. Before the integration can be initiated, a process for messaging must be defined that identifies who should get the notification, what the message should say, when should it be delivered and, importantly, when escalation might be required.   

Don’t forget that the IT incident might require the definition of unique groups and different messages. Obviously, the IT resources that have responsibility for resolution must be notified, but information must also be provided for IT management and stakeholders (employees, customers, suppliers, etc.) who have a need to know about the incident. And each incident and the criticality or severity of that incident might require a different definition of these groups for notification and the timing for those notifications. 

IT incident management alerts and escalations is a prime example of where automated alerts prove valuable. While manual notification solutions can assist in the process, IT personnel typically don’t have the time to identify an incident, then manually launch and monitor responses from a notification solution. When events happen, automated alerts are sent to the appropriate staff, taking into account whether they are on duty, on call or out of office, letting recipients quickly fix problems before they cause downtime or delays and ultimately delivering a recognizable return on investment by reducing response time for IT incidents. 

Ability to address IT-specific issues  

Beyond the standard notification feature set that is seen in many of the industry-leading systems, IT notification solutions must also be able to address IT-specific issues, including:   

  • IT skill-based resources: The ability to escalate through specific skills for specific employees  
  • Multiple shifts with on-call responsibilities that rotate based on schedules: Robust schedule and shifts with intelligence to automatically send notifications to the correct individuals, depending on the shift and current on-call schedule  
  • IT management oversight and need for situational awareness (possibly with or without call to action): Escalation through management layers for awareness of developing incidents and the individuals who take ownership of the resolutions  
  • Monitoring and threshold management of critical components and applications: Integrated notification capabilities to recognize triggers or thresholds and automatically launch pre-determined notifications to address the developing incidents  
  • IT service management tools:  Automated integration of notifications both to receive information from ITSM tools, and then to launch notifications, with the ability to then return information to the ITSM tool and update information  
  • Complex operating procedures: Need for an automated workflow that can be configured to interpret information and take actions based on defined criteria, including launching multiple notifications and integration into other tools within the IT environment  
  • Integration to enterprise applications: Strong API for notification solutions and a run book automation tool for building custom logic 

Understand implementation requirements  

Finally, once the decision has been made to take the steps to define and implement a notification solution for the IT environment, specific implementation steps will ensure the organization gets the full value of the notification solution. 

1. Begin by analyzing the notification behaviors you want to achieve – Don’t make a decision on a tool based on just a few features. Take the time to analyze what features are most important and prioritize those in your selection of a tool. 

2. Develop standard operating procedures – The tool can then automate the process. Make sure to define the incidents and the criticalities that will drive the development of your notification processes. 

3. Map your notification process – Include who should be notified, what the message should be, what the escalation tiers should include, and what actions must be taken for the responses returned from the notification. 

4. Design your framework – Understand how notification will integrate throughout the organization. Integration concepts should include which enterprise applications would benefit from notification enablement, and how that will be accomplished. 

5. Integrate notification with ITIL processes – While notifications are a critical part of incident management, notification technology can be used throughout many ITIL processes, including but not limited to change management, configuration management, availability management, service continuity management, problem management and many others. 

6. Perform technical integration – Use a variety of tools that work with the notification solution for complex and customized integrations. Whether it’s the use of a Web services API or run book automation, integrations can be accomplished in a cost-effective manner. 

Automated notification has proven to be an invaluable tool for incident management, quickly delivering a recognizable return on investment by reducing response time for IT incidents. These benefits are not unique to the IT department, and can provide great value when extended throughout the enterprise, ultimately helping to streamline business operations, increase efficiency and enhance productivity. 

Alex Tsepetis, CTO at OnSolve 

Image Credit: SFIO CRACHO / Shutterstock