In the space of time it takes you to read this blog post and finish your morning coffee, a company at the vanguard of DevSecOps, such as Etsy, Amazon or Netflix, will have completed yet another deployment – one of potentially thousands per day. Deployment frequency has accelerated to a pace that would have been unthinkable just six years ago, often at the cost of robust security assurance of the code under development. So, the natural question is how companies can effectively scale their security processes to keep pace with the velocity of development we see today? My experience has been that a focus on automation alone is insufficient, instead it takes a blend of automation, cultural change and integration of security processes throughout the development lifecycle to achieve effective layered security in such agile environments.
Many as four out of five companies leveraging a DevOps approach to software engineering do so without integrating the necessary information security controls, underscoring the urgency with which companies should be evaluating “Rugged” DevOps (also known as “shift left”) to build security into their development lifecycle as early as possible.
Rugged DevOps represents an evolution of DevOps in that it takes a mode of development in which speed and agility are primary and integrates security, not just with automated tools and processes, but also through cultural change emphasizing ongoing, flexible collaboration between release engineers and security teams. The goal is to bridge the traditional gap between the two functions, reconciling rapid deployment of code with the imperative for security. The siloed thinking and processes that so often characterize DevOps are replaced by increased communication and shared responsibility for security tasks during all phases of the delivery process, adding increased security, transparency and a clearer understanding of exposures and business risk. In this agile approach, security parameters are put into practice at the start of the project and the security assurance process is applied throughout the development cycle.
For many companies, a common pitfall on the path to implementing rugged DevOps is implementing the approach all at once rather than incrementally, underestimating the complexity of the undertaking and producing cultural disruption in the process. To increase the probability of successful adoption, companies should instead take a multi-phase approach to rugged DevOps. The first step is identifying and developing the skills of security champions, members of the development and operations teams who help bridge the divide between engineering and security through basic security cross-training, peer education as well as code and configuration reviews. Security champions can act as experts and catalysts for security that anticipate potential design or implementation problems early-on in the development process. Champions support the adoption of secure coding practices by providing development teams with real-world examples of past security issues which provide a better basis for driving remediation than abstract, less relatable errors. As a first step, technology leaders should identify developers who demonstrate an interest in security and provide them with an expanded role as security champions. Once trained, their purpose will be to scale limited security training more effectively and provide a degree of security oversight in agile and DevOps teams.
Another key challenge for companies transitioning to rugged DevOps are attempts to solve for security solely through tooling when, in reality, effective transition involves both a cultural and technical aspects which should be approached with an eye toward the same principles (flexibility, responsiveness, collaboration, et al.) that define agile software development. Effecting cultural change requires radically changing well-defined roles and responsibilities by adding complementary security duties focused on ensuring the security of code shipped to production.
For successful deployment of ‘Rugged’ DevOps, continuous readiness should also be a focal point. Through continuous readiness DevOps engineers are constantly leveraging learnings from security cross-trainings. To achieve a state of continuous readiness, organizations must realize that traditional reactive approaches to security just won’t cut it. Instead, companies should implement a “push” model—proactively delivering security learning to DevOps engineers at regular intervals and in bite-sized modules. They should also shift from consumption-centric learning metrics (e.g., Did DevOps engineers complete the learning module?) to assessment-based ones (Can DevOps engineers perform the key skills the learning module sought to impart?).
In addition, as part of continuous-readiness programs, development, operations and information security should work jointly to train DevOps engineers on the following:
- Code analysis – Deliver code in small chunks so that vulnerabilities can be identified more readily. This stands in contrast to conventional multi-week sprint cycles producing large amounts of code to be reviewed.
- Change management – Developers can push certain changes to continuous integration and delivery (CI/CD) platform but very often there is no attestation as to whether they validated the security of the code first. Organization need a security attestation process which verifies that effective code analysis took place.
- Compliance monitoring – Organizations should strive to ready for an audit at any time, which means maintaining a constant state of compliance, including gathering evidence of General Data Protection Regulation (GDPR) compliance, Payment Card Industry (PCI) compliance, etc. Do this by leveraging outputs of static application security testing (SAST)/ dynamic application security testing (DAST) tools to support continuous compliance and auditing.
- Threat investigation – Identify potential emerging threats with each code update and be able to respond quickly. Do this by having a policy document which sets out what the most prevalent security findings are and develop practices that ensure that developers can solve for issues in a self-serve manner.
- Vulnerability assessment & KPIs – Discover new vulnerabilities with code analysis, then analyze how quickly they are being responded to and patched.
Organizations can make further progress in the transition to rugged DevOps by recognizing that zero-day vulnerabilities (previously unknown vulnerabilities) do exist and can be prepared for. Once discovered, developers need to contact security experts, work toward remediation or mitigation and add lessons learned to their threat investigation and playbooks. We live in the era in which security breaches are a common occurrence. DevOps engineers need to be trained on how to react to zero-days impacting the product stack within hours of its discovery. This training must include an execution plan for such an event, as well as tools for building and delivering security against such attacks quickly.
The velocity of business and software development today means that DevOps teams of all types need foundational, continuous and reactive strategies to proactively ensure the security of code being released to production and effect a successful rugged DevOps transition. Putting these plans in place is not a one-and-done process; instead the approach should continuously evolve to support the various scenarios and needs that DevOps teams encounter.
Swapnil Deshmukh, Security Head for Emerging Technologies at Visa
Image Credit: Profit_Image / Shutterstock