Skip to main content

Achieving the holy grail of software testing: What CIOs need to know

(Image credit: Image source: Shutterstock/TechnoVectors)

Do we really know the business risks associated with the new software we design, build, test, and implement?  Will the new capabilities we invested so much time and resources in actually function and perform and expected, or will they draw the ire of customer? Getting answers to these questions is the last “Holy Grail” of IT.   Unfortunately, most IT stakeholders focus on the number of tests that are successfully run, not whether the tests are providing insight into critical business risks. 

It is a rare enterprise CIO who likes to dive into software testing methodologies. But, IT leaders at all levels might be intrigued to learn what an archaic and ineffective process most enterprises are using to test the innovations driving their digital transformation initiatives. The way it’s commonly done, testing is an expensive activity that holds back your innovation—without providing clear insight into the business risks of your applications.

As an industry veteran with experience across a broad swath of industries, I can tell you that transforming testing is emerging as a stealth strategy to take your digital transformation initiatives to the next level.  Many organisations are quietly transforming their testing process to deliver instant insight into the business risks of each release candidate. At the same time, they’re slashing testing costs, which frees up additional resources to dedicate to innovation.  With this new competitive edge, they’re surging ahead.  

How this process holds us back

What might surprise IT leaders—and especially CIOs—is that amount of time and money that testing requires vs. the business value that it actually delivers.

Most testing focuses on the number of tests that are run, not whether the tests are addressing the risks that are most important to the business. Often the last stage of testing is conducted by a dedicated independent testing organisation with inadequate understanding of their company’s business risks or objectives with the new software system.  To make matters a bit worse, defects (or even false positives) discovered late are triaged for impact, and perhaps not addressed because of mounting pressure to release.  And load testing – streaming transactions and queries against a system to make sure the system can handle full production volumes – usually gets short thrift. 

I understand that many IT leaders find software testing boring. However, upon learning that this process often consumes 40 per cent of an IT application budget, people tend to find it more intriguing.

Why is testing so ineffective yet expensive? Thanks to agile methodologies and DevOps, software development cycle times have been reduced significantly, yet today’s software testing continues to operate at a slower pace—creating a process cadence mismatch. This is because testing is dominated by manual tests. In fact, 70-80 per cent of software testing is manual, tedious, and time-intensive.  At the end of the day, we do not really know whether the right tests were done, nor do we know if the software system and databases can perform well in production.

Transforming testing for a competitive edge

So, should a CIO care about software testing?  Yes – but not about how many tests were completed.  What is far more important is whether the tests that were completed actually address the business risks of the enterprise in a way that gives confidence in test results.

Risk-based test automation provides a fast and cost-efficient way to control top business risks in software releases. This can’t be reliably achieved by simply throwing more people or more tools at the problem. It requires a deliberately architected approach to continuous risk-based testing of what is important to you and your customers.  

Risk-based testing involves knowing that the organisation designed what the user needed, that what was designed was built to handle the risks of the business and perform to the requirements of the business, and what was released can handle the scaling of volume loads once in production.

How does this create a competitive advantage? Let’s look at a few specific ways…


The success of a business’s innovative ideas depends on how quickly your team can develop and deploy those ideas in software. Any hiccup that slows down the process could hinder the success of your project.  There's a delicate balance between moving fast enough to be first to market and moving cautiously enough to ensure that you won’t crash and burn.

It’s never going to be feasible to find and fix every issue with the application system before each release. But risk-based testing, which balances speed with quality, tells us whether we each detected issue is a small glitch that can be fixed later or a true showstopper.  

Time to Market

Manual testing can rapidly become a roadblock to speeding new release candidates to market. But that’s just the beginning. Manual testing, even with scores of testers often employed for even a single application, becomes exceedingly costly because it takes time and knowledge to know whether the critical risks to the business are being validated.   The lost revenue and profitability in getting to market late can often dwarf testing costs, or allow a more nimble competitor to seize market advantage. If the cadence of software testing doesn’t match development, there’s an issue.

The adoption of risk-based test automation is rapidly becoming a priority for companies in the early stages of the development process, for the simple reason of keeping up with market demand. Enterprises must be up for the challenge of handling frequent changes to their system landscape and able to conduct end-to-end testing across all enterprise systems and remain stable in the production environment.


Now that software is now the primary interface to the business, a software failure is a business failure and brand reputation risk.  Automated risk-based testing is key to reducing the overall amount of testing while achieving the highest possible risk coverage. It’s exactly what fast-moving DevOps teams need in order to immediately assess whether their changes have impacted the overall user experience.  In addition, knowing that the new release can satisfy core functionality expectations, perform acceptably under the expected volume of transactions, and handle surges in volume is key for building confidence at the CXO level.

Testing might not be exciting as the innovation process, which involves brainstorming ideas, watching them come to life as prototypes, and witnessing the process of bringing prototype to final production-ready software.  Nevertheless, transforming testing is an often-overlooked, yet powerful, catalyst for innovation. With a modern approach driven by risk-based test automation, your organisation can free additional IT resources for innovation, bring your innovations to market faster, and ensure that they have a positive impact on the customer experience.

Terry Milholland, Senior Partner, TVM Consulting
Image source: Shutterstock/TechnoVectors

Terry Milholland is former CTO & CIO for the IRS; CTO for Visa Inc., CIO and CTO for EDS; and CIO for Boeing.