The benefits of DevOps for automation, traceability, and especially, collaboration among previously siloed teams and stakeholders are widely accepted. But as DevOps teams are increasingly tasked with shifting operations to containerized Kubernetes environments, long tried-and-tested DevOps practices can fall short. Security concerns — and risks — are also manifested in different ways.
The good news is that GitOps serves to fill in a number of gaps in DevOps for security of distributed environments. This is because GitOps processes are highly conducive for DevSecOps, defined here as security best practices that apply to the entire application lifecycle.
In this article, we look at how GitOps provides an essential framework for DevSecOps, for security checks that extend throughout CI/CD, as well during the post-deployment stages of application management on Kubernetes clusters.
A single immutable source of truth
GitOps can be defined as:
An operating model for Kubernetes and other cloud native technologies, providing a set of best practices that unify Git deployment, management, and monitoring for containerized clusters and applications.
A path towards a developer experience for managing applications; where end-to-end CI/CD pipelines and Git workflows are applied to both operations, and development.
Git is the single source of truth, as the desired configuration is declared here. There is a GitOps agent that runs inside Kubernetes, it continually compares the actual state inside Kubernetes with the desired state stored in Git. Any new changes merged into the monitored branch in Git are automatically applied to Kubernetes. Conversely, any manual changes directly applied to Kubernetes are automatically reverted back to the desired state declared in Git. Configuration drift is eliminated.
Git is often aptly described as the single source of truth as it applies to GitOps, thanks to its immutable structure. Among other things, it helps to maintain a boundary — not unlike a firewall — that separates the concerns between CI and CD. In this way, the number of steps involved in application development as part of CI — pull requests, testing, commits, etc. — remain separate on Git.
For the developer making a pull request, the pull request, once reviewed and approved, is merged and automatically applied to the cluster on the next reconciliation — which typically takes 15 minutes.
By the default, the process is bi-directional — this means that a change made directly to Kubernetes is reciprocated on Git when the next reconciliation loop runs (typically every 15 minutes). However, this scenario is less than ideal, when a DevOps team member or, worse yet, a bad-actor intruder makes changes directly to the clusters. These direct-to-cluster changes have not been properly vetted through merge requests and approvals, and thus violate the GitOps principle of Git serving as the immutable source of truth as drift occurs.
As a solution to help mitigate drift from occurring, a GitOps monitoring tool can send alerts if changes are made to the cluster that were not first applied in Git. When this happens, thanks to the audit trail, the application code on Git can then replace the erroneous changes made to the clusters, hrough controllers in its deployed state in the runtime environment.
Conversely, when immutability is not achieved, drift can occur. This might happen in the event of a network attack or when a DevOps team member inadvertently changes the cluster configuration so it is different from what is on Git. The incongruity is flagged when this happens — through the use of proper GitOps tools — thus representing a quintessential DevSecOps that GitOps processes provide.
Proper tools automate the process of continued monitoring to ensure that the desired state of the configuration on the Git repository matches the actual state in the Kubernetes cluster. It is also used to reconcile and complete deployments once they are properly committed in the declared state of the repository.
The audit functionality that GitOps provides is also key for DevSecOps support. By remaining on the Git repository as the single source of truth, all applications, code, and configurations are versioned, and retain a full audit trail, which is a main requirement for any secure environment. This audit trail is also typically available to both the developers and the operations team members, in order for them to observe what is running in the cluster (most users are limited to read-only access to the cluster configuration).
The developer may not necessarily need to rely on audit trails as much as operations and DevSecOps team members do, but they can make use of the capability to understand what changes have happened and what the motivation was behind changes to the repository that did occur. In a nutshell, for the developer — as well as for all DevOps team members — everything is just a Git log away.
Audit trails on Git are also accessible and simple for developers to look up when needed. This is because the complete history of the system is captured in the Git system of record, which developers understand.
With the availability of the audit trail, it is also possible to easily roll back changes to applications that have caused issues. This is especially helpful when clusters have been compromised or misconfigured. In this case, instead of having to rebuild a cluster from scratch, the audit trail contains the desired state in the repo. The cluster configuration and applications with the desired state from the audit trail are then deployed and the rebuild process is automated.
Few ‘keys to the kingdom’
Developers typically rely on Jenkins, an automation server, to build, test, and support CI/CD for production pipelines for Kubernetes environments. Without GitOps, the developers might otherwise have direct access to Kubernetes clusters as they deploy their code directly. In other words, the developers — whether they are part of the organization or are a contractor working remotely — would have direct access to the production environment and would have the “keys to the kingdom,” so to speak. This is hardly an ideal security scenario.
All it would take, in the above case, is for the wrong user — or worse yet, an intruder — to have KubeConfig or Kubectl command access to clusters in production that they could run from their laptop. If an attacker has compromised the CI system and the credential set, for example, then they can also have access to any cluster the CI system has access to.
The fact that GitOps helps to prevent users — or attackers — from changing cluster configurations without leaving any trace is especially critical for operations and security teams — in addition to helping reduce the developer's mental load. Through access control, for example, the developer should typically not have direct access to Kubernetes nodes and/or kubectl command lines. GitOps thus reconciles anything that developers define in Git but will not allow for manual access to Kubernetes clusters or production environments unless developers have special access-control permissions.
GitOps’s DevSecOps functionality as a way to thwart this attack-vector scenario for CD thus protecting the “keys to the kingdom.” When GitOps operators, such as open source Flux — more about this below — run inside Kubernetes and thus have access to clusters, the access controls remain within Kubernetes security framework. User access is determined by assigning privileges to the namespaces of different teams and team members.
As an example of how DevSecOps and GitOps are highly conducive to one another, the U.S. Department of Defense (DoD), Nicolas Chaillan, chief software officer of the U.S. Air Force, recently described how DevSecOps and GitOps — with Flux — play a critical role in its software development that supports the entire spectrum of the U.S.’ security forces.
“Safety and security are non-negotiables but we also want developer self service to boost productivity and velocity,” Chaillan said.
As an example, GitOps is the “key for our success in building and rolling out Platform One across the entire DoD,” the DoD says. According to the DoD, Platform One is "a collection of approved, hardened Cloud Native Computing Foundation (CNCF)-compliant Kubernetes distributions, infrastructure as code playbooks, and hardened containers," with baked-in security pipelines.
Checks and balances
DevSecOps processes integrate with GitOps offer checks and balances. In this way, the access control it provides prevents developer — or intruder — from accessing a source-code repository in order to introduce, for example, a backdoor, zero-day, or another kind of vulnerability during the CI process. This is because all pull requests are peer reviewed.
With DevSecOps supporting the CI process, the checks and balances apply before code is committed on Git and deployed in the cluster. The developer will typically submit a change as a pull request, which is peer-reviewed. Once reviewed and approved, the code is merged on Git, and the YAML file of the desired state is then altered accordingly.
The desired state will be automatically applied to the Kubernetes cluster. As mentioned above, proper GitOps tools with DevSecOps functionality constantly monitor the actual state within the target system, to ensure it reciprocates what is declared on Git. An alert is issued if there is any difference, so corrective action can be taken.
A number of GitOps tools support DevSecOps. Open source Flux, for example, helps to maintain the Git repository as the single immutable source of truth. Flux’s capabilities also extend to access control for checks and balances during CI/CD for code that is committed and deployed.
For a more opinionated Flux experience, open source Weave GitOps Core simplifies the first steps to help automate CI/CD across multiple clusters. The process of setting up a GitOps and DevSecOps process with Weave GitOps Core involves a few simple commands on a console.
Weave GitOps Enterprise has emerged as the first GitOps platform that automates continuous application delivery and automated operational control for Kubernetes at any scale across hybrid cloud, multi-cloud, and edge architectures.
All three tools help to automate the monitoring to ensure that the cluster configuration always matches what is on Git to help prevent drift. A complete and accessible audit trail allows for application and cluster-configuration rollbacks to be made on an as-needed basis — without having to rebuild the cluster from scratch.
In summary, GitOps represents the evolution of DevOps for distributed Kubernetes environments, while DevSecOps has emerged as an essential way to maintain CI/CD security for GitOps, throughout the application lifecycle.
Steve Waterworth is Technical Marketing Manager at Weaveworks.