Most organisations have DevOps or CI/CD initiatives, yet still struggle with Continuous Delivery. Here are the 10 most common reasons why enterprises get continuous delivery wrong.
1. “Agile” and “Release” are used in meetings
Continuous Delivery doesn’t mean “release more than once a year or quarter.” It means deployments are continuous, never-ending, ongoing, nonstop – and so on.
Agile was something dev teams aspired to achieve 10 years ago when weekly or monthly release cycles were considered valuable.
Today, daily production deployments have become the norm. Today, all the planning, meetings, approvals, tickets, politics, etc. associated with managing releases will actually slow you down or kill your business. In 2019, you don’t want to be riding a horse when all of your competitors are driving cars.
Once code is committed, it’s down to your deployment pipelines to tell you if it’s ready for production or not.
2. you don’t commit to trunk/master
Branch, commit, merge, and resolve conflicts – more tasks that slow you down. The whole point of Continuous Delivery is to deliver and deploy independent software components fast and frequently using dev teams that work in parallel.
If a build, test or deployment pipeline fails you should stop, understand what happened, fix it, and learn from it. This is what happens in production, so why not do it during development and test? Committing to Trunk/Master will drive the right team behaviour and urgency required for Continuous Delivery.
3. Fixing builds/deployments takes 30+ mins
The whole point of a deployment pipeline is to kill your build before it kills customers in production. When a deployment pipeline or test fails, it should be treated as a “stop-the-world” event where everyone stops, focuses, and fixes the build so things can progress. If your production canary or deployment fails, you should be able to rollback instantly or roll forward with a rapid fix.
4. Deployment pipelines take hours to complete
Let’s imagine your new build or artefact is perfect. How long does it take for that artefact to be promoted through all your deployment pipeline stages (dev, QA, staging) and into production? One hour? Two Hours? Six Hours? Longer?
Feedback loops for deployment pipelines (and stages) need to happen in minutes, not hours. This may require more test/tool/feedback automation so you can eliminate manual tasks and approvals throughout your deployment pipelines. For example, if one pipeline stage succeeds, it should automatically move on to the next until a stage fails.
5. Your deployment pipelines rarely fail
If 95 per cent or your deployments succeed in dev/QA/staging, something is wrong. Your deployment pipelines either lack basic test coverage or they lack integration with your existing toolsets. If you’ve already invested money in tools, why aren’t you integrating and leveraging them to gain insight into your pipelines? Many enterprises leverage APM tools and log analytics to help identify performance or quality regression in deployments. A company that does continuous delivery should be able to send out a ton of deployments and have their failed deployments be detected without customers being impacted. Test automation, an integrated toolchain, and the ability to verify deployments are key components of Continuous Delivery.
6. It takes a village to deploy & debug deployments
It’s entirely possible you’ve automated your deployment process. By automated I mean you’ve stitched together tens of deployment scripts written by 20 different people, who at some stage, tweaked and refactored a few hundred lines here and there. So, when something breaks, it literally takes a village to debug and troubleshoot a deployment. Next time you deploy, count the number of people involved and multiply that number by how long the deployment process lasts. Village deployments are expensive.
7. You have a dedicated deployment/release team
If you have multiple devs or product teams, the last thing you need is another team that bottlenecks all your deployments. Tickets, change control, approvals, handoff, documentation, and so on are more activities that slow down your deployment timelines.
If a deployment/release team does multiple deployments a day and the first deployment goes down, then they’ll probably spend the next 6 hours debugging versus deploying, and thus bottlenecking other deployment pipelines. This is bad.
8. Developers don’t deploy/debug their own code
Most well-oiled CI/CD organisations let DevOps teams build deployment pipelines, and let their developers deploy using these pipelines. Let me say that again: DevOps manages the tooling/automation/framework and developers use this platform-as-a-service so they can deploy their own code. When you let developers deploy their own code something natural happens – they end up debugging their own code should something fail. This is a very good thing.
I hear too often from customers that Deployment/Release teams often drag DevOps or SREs teams into firefights when deployments go south. If code fails in production, then you need the right set of eyes, context, and knowledge to rapidly troubleshoot. You also need the ability to automatically roll back if you can’t roll forward.
9. You don’t know the business impact of deployments
The whole reason for doing CI/CD isn’t so you can spend a ton of money on people, technology, and tools. You do it to grow your business and make it more competitive. Organisations do CI/CD so their applications and services deliver a better service and experience than their competitors. End of story.
10. You think DevOps is continuous delivery
DevOps is a culture or mindset, whereas Continuous Delivery is a practice or set of principles that teams follow to deliver software safely, quickly, and in a sustainable manner.
A DevOps culture makes Continuous Delivery principles easier to implement, but it’s not going to magically re-architect your application so more components can be deployed more frequently in the cloud.
DevOps is popular initiative, but this is more about people, culture, transparency, and collaboration across teams. It’s not about tools or technology.
What other signs or mistakes do you see other teams and organisations making?
Steve Burton, DevOps Evangelist, Harness
Image source: Shutterstock/violetkaipa