Skip to main content

Avoid multi-million dollar mistakes in Kubernetes refactoring: Four best practice to follow and two pitfalls to avoid

(Image credit: Image Credit: Everything Possible / Shutterstock)

Today, data breaches can cost companies millions of dollars to fix. A recent report from IBM Security estimated that breaches cost companies nearly $4 million on average and those figures are climbing. Some of the main causes of data breaches can be traced to human error and code errors that were built into the original monolithic application. With the advent of microservices managed through Kubernetes orchestration, developers now have the ability to refactor those monolithic apps into containers and correct those old security flaws in the process, thereby providing better security.

But don’t get complacent thinking that containers will provide fool-proof security. Without proper security protocols, they can be left vulnerable to attack. This will require a paradigm shift in thinking. We’re used to working on systems where everything is safe by default and we have to actually mess something up to make it unsafe. With containers and Kubernetes however it’s exactly the opposite: container networking security and Kubernetes infrastructure is open by default and the security needs to be incorporated into it, which is something you need to get used to and always keep in mind.

This article will look at four advantages and two common mistakes that DevOps and Security teams should be aware of in order to set up security policies during refactoring that will allow them to quickly monitor, investigate and isolate vulnerabilities from CD to Runtime before they are allowed to inflict serious monetary damage.

Four things to take advantage of:

  • Utilise network policies - Network policies enable users to identify specific guidelines for how different groups of containers (or pods) are allowed to communicate with each other. The ability to set up such rules and guidelines is critical, especially since modern applications require that pods be able to communicate with each other. However, for security reasons, communications cannot be left open (which is the Kubernetes default).
  • Managing Secrets - Secrets are the hidden keys which help you manage code and allow containers to communicate with each other. API keys, passwords and tokens are all examples of secrets. These secrets should be encrypted and a plan should be put in place to rotate in new secrets and deactivate old ones on a regular basis. Secrets are a security feature which goes a long way toward protecting against data breaches. From a technical perspective, a single Kubernetes worker-node keeps secrets in memory for several pods running on that node. But no pod, assuming certain runtime privilege configurations are met, can access the secrets of any other pod. Secrets can also be used by a kubelet when pulling images for the pod. This communication between the user and API server, and from API server to the kubelets, is protected by TLS. When a pod using a secret is deleted a kubelet, a Kubernetes worker-node agent, deletes the local copy of the secret data as well. 
  • RBAC Authorisation - Role-Based Access Control (RBAC) is a method of regulating access to network resources based on the roles of individual users within an enterprise. Kubernetes allows admins to dynamically configure access policies through the Kubernetes API, and to drive authorisation decisions. For example high ranking executives can be granted access to sensitive information, such as financial data, while low level employees would not be allowed to access such information. The Kubernetes API discovery endpoints are inaccessible to unauthenticated users by default, greatly reducing attack surface and increasing the overall cluster security. Review & control workloads that require API access privileges to the Kubernetes API server - specifically, permissions that allow users to read Secret Resources, create workloads of any type (pods, deployments, etc.), and create services that can potentially open the cluster to the outside. In general, the principle of “least privilege” should be followed - running workloads, pods & microservices, in general, must be able to access only the information and resources that are necessary for its legitimate purpose, and nothing else beyond. This will prevent opening the door to cluster-wide or workload level take over attacks.
  • Use Audit Logs in Kubernetes - These logs provide a chronological record of each call made to the Kubernetes API Server, the central access point used by all users. These records are critical for investigating suspicious API requests that may indicate a data breach. The dynamic nature of a Kubernetes cluster means that workloads are constantly being added, modified or removed. When it comes to database security, for example, it’s not a matter of an auditor focusing on access to a few specific workload containing a database – it’s a matter of identifying which workloads contain a sensitive database at each particular instant in the audited time period, and which users and roles had access to these workloads.

Two things to be wary of:

  • Remain continuously vigilant on security - Unlike other types of code, network policies in Kubernetes are open and therefore unsafe by default and security measures need to be built into them to enable them to be secure. It’s also important to always be alert and aware of aspects like Kubernetes being backward compatible with older versions. This includes those that don’t have important security updates. This is why it’s crucial to pay attention to the little details, like merging code from an older version and ensuring the new code remains secure.
  • Mislabelling - Labels allow users to specify in detail how they would like pod-to-pod communication to work in their application. Kubernetes provides a list of recommended labels for every object you create and have a shared prefix. The prefixes ensure that recommended labels do not get mixed up with custom labels defined by users. Label your pods appropriately so that they can be set up to be whitelisted or to deny access as required. This can seem like a painstaking task but labelling pods as you go will help with Kubernetes orchestration in the long run. Creating an extensive list with labels can provide more functionality and context to your objects.

Working with containers through Kubernetes orchestration requires a new way of thinking when it comes to software development. While refactoring in Kubernetes allows developers to correct code errors for their original monolithic applications, strict policies need to be adhered to in order to prevent opening the containers up to wider security threats. By following these four rules from CD to Runtime and avoiding these two common mistakes developers can ensure their new Kubernetes deployments are optimised for security and function.

Amir Ofek, CEO, Alcide