Skip to main content

Tech has a validation problem. How we innovate needs to change...

(Image credit: Shutterstock / aorpixza)

Digital transformation remains a top priority for senior tech leaders, especially where rushed implementations have begun to creak under the accumulated weight of expectations and the changing pressures of the ‘new normal’.

In fact, most companies can’t actually achieve ‘true’ digital transformation. Why? Because technology has a validation problem that most leaders are simply not aware of. 

Organizations are still rushing to meet the needs of an increasingly connected world. Many are still changing to cloud systems, rolling out internal apps, or facing periods of user uncertainty as they get to grips with new devices, apps, and processes on a rolling basis. The trouble is that typical software testing halts the process of transformation.

Developers are creating code, but getting feedback is often so costly and time-consuming, that this part of the process decreases from the expected productivity. Furthermore, software bugs have caused major complications, often pausing digital transformation. For example, earlier this year bugs exposed Honda employees to security risks, delayed NASA’s flight on Mars, and brought down Amazon, Spotify, and eBay for hours. Almost the whole world will have noticed Facebook, Instagram, Messenger, and WhatsApp going down in early October, given its billions of users being nearly a quarter of the global population.

It’s a supply chain problem

Nowadays not a lot of applications need to be entirely developed in-house with developers writing custom code. Very often apps combine pre-existing code from libraries: A ‘supply chain of code’. It is common for organizations to offer a fully functional service, like for online shopping, that has relied on a supply chain of third-party services, or dependencies. This might include Facebook to authorize visitors, Shopify for e-commerce, and a provider like Stripe for payments.

But guess what? In the past year, modern software supply chain attacks have increased by 650 percent. The dependency confusion attack is most common when an automated software development tool is duped into downloading a malicious package from a repository. Dependencies allow developers to move at pace that speed should not be an excuse not to look at if they are secure.

All these dependencies relied on by software teams have become incredibly complex, and vulnerabilities are missed. It doesn’t matter how skilled the engineering team is, people cannot comprehend all the components in the products. Yet the business still demands delivery of software fast and error-free.

Managing this complexity requires moving the focus from the application to the myriad sources of change affecting it. Change validation. 

Innovation requires change validation

Facebook’s outage is a prime example of the importance of change validation. In this case configuration changes on the backbone routers went wrong. An update to the Border Gateway Protocol records basically removed the ability for everyone online to find its online properties. It’s as if an error erased them from the ‘map’. What’s more, Facebook employees reported not being able to work, as their own tools and systems were affected, and consequent automated door permission failures meant that engineers could not physically get to the affected systems to make changes. 

Change validation acts as a guard rail for innovation by automating the process of verifying new software in production. This makes releasing software to production a consistent, scalable and self-driving process. It helps companies innovate faster and more securely, digitize more efficiently, and release new features and capabilities reliably. It is effectively the key to true digital transformation.

Engineering teams are key to ensuring that innovation is delivered the right way and become the standard bearers for digital transformation. Of course. Software is still eating the world. Users’ quality expectations are stratospheric. Software teams are shipping faster but in the face of a development process that has become massively complex. The way engineers build today is incredibly interconnected. Sources of change no longer exist solely in a repository, making it impossible for a single developer to understand the entire process.

This can lead to a crisis of confidence in a teams’ code. That confidence is the foundation for speed. Without it, the process founders at the start.

As software becomes a competitive differentiator for more organizations including beyond the tech industry, across all verticals, software teams are optimizing for faster delivery, and a few trends have appeared. Firstly, there’s a greater use of third-party services and tools; there are microservice architectures; and there are ever larger data sets.

While these factors have helped refine and optimize software development, they have also reduced the ability to be confident in complete validation within a pre-production environment. The presence of these factors increases the cost of achieving the same confidence in the code before it is shipped.

CI/CD is where you start

Something to note. The point where the cost of preproduction validation surpasses that of in-production validation isn’t static. The cost of testing in production is somewhat fixed and the cost of thorough pre-production testing increases with the growing complexity of development processes. The threshold is something that needs to be dynamically assessed.

Change validation is founded on best practices like continuous integration/continuous delivery and release orchestration. The new learning is that in this world of software supply chains and the cloud, testing in production has become an important additional tool to know if changes to code work as intended before releasing to the world.

No one can adopt new technologies like AI and machine learning without knowing if changes happening in their apps are effective. In fact, where advanced technologies make changes to the code themselves, tracking the source of changes is critical!

Validating changes as they happen, where they happen, addresses errors when you want to find them. Failures happen in complex situations, and it can be managed. Where engineers can seamlessly roll back releases, they can ideate, solve, and innovate. Change validation empowers experimentation.

What’s more, beyond functionality, uptime, and experience, the cost for testing in production now is considerably lower. Better design and best practices help but testing and proving the build is essential. The world economy runs on software and services - and so does our ‘new normal’...

Rob Zuber, CTO, CircleCI

Rob has been the CTO at CircleCI for 7 years, giving him extensive knowledge of the technology. As a CTO, his responsibilities include overseeing the development and dissemination of technology for external customers, vendors, and other clients to help improve and increase business. CircleCI is the leading continuous integration and delivery platform for software innovation at scale. With intelligent automation and delivery tools, CircleCI is used by the world's best engineering teams to radically reduce the time from idea to execution.