The use of open source has become a crucial part of the modern development process. It helps everyone build software faster than ever, but it also puts developers at risk of unknowingly introducing security vulnerabilities into the applications they develop, and placing their own organization or their customers in jeopardy. A single bug in one of the code dependencies can affect an entire application, making it susceptible to compromise. In a cloud native world, these risks are amplified due to the continuous stream of potentially vulnerable code.
The need for speed: The rise of open source
Over the past few years, open source has seen massive growth. Today, over 2.5 million developers contribute to open source on platforms like GitHub, which is the largest open source community in the world. It hosts millions of open source projects, some of them not only having thousands of contributors but also acting as dependencies for numerous other repositories. Following the popularity of open source, its use in commercial software has also surged. According to the Synopsys 2020 Open Source Security and Risk Analysis Report, “open source components and libraries are the foundation of literally every application in every industry.”
The reality is that building an app completely from scratch these days is extremely rare. The proliferation of DevOps and cloud computing created a new, dynamic and fast-paced environment. More and more organizations are adopting containers and microservices to build cloud native applications, taking advantage of the speed and agility these new technologies have to offer. Amazon engineers allegedly deploy code every 11.7 seconds whereas Netflix engineers deploy code thousands of times per day.
To keep up with this velocity, developers are increasingly using open source packages and libraries throughout the software lifecycle. It is estimated that 99% of today’s codebases contain open source components, and up to 70% of enterprise code is open source. Building software is becoming more like assembling Lego cubes from the ready-made open source components - with the remaining code largely serving as a “glue”.
The explosive growth of both open source and cloud native is not accidental. Open source projects dominate the technology scene across an entire cloud native compute stack: Docker, containerd, Kubernetes, GitLab, Jenkins, MySQL, Redis, Harbor, Rancher, and many others. The cloud native world is essentially powered by open source which, in turn, allows for velocity and rapid development lifecycles.
While open source brings many benefits, it also introduces new and potentially serious security threats for organizations.
Why is open source a security risk?
Open source software is free for anyone to use, explore, and enhance. Developed and maintained by large numbers of contributors, it provides unlimited access to the source code, so that users can modify it as they wish. Like any software, open source is written by humans and, hence, has bugs. According to Sonatype 2020 State of the Software Supply Chain Report, one in ten open source software downloads are vulnerable, and on average there are 38 known open source vulnerabilities per application.
The main reason behind the vulnerability of open source lies exactly in its public nature. Numerous developers can get involved with little vetting - including bad actors. Open source is, well, open and built by sometimes unaccountable contributors. While security awareness within the open source community is slowly improving and there have been initiatives to address security issues (e.g. GitHub Security Lab), the lack of central control gives plenty of opportunities for attackers to find holes and vulnerabilities. The governance of open source software varies quite a lot, and with the exception of the very popular, commercially-driven projects, one cannot simply rely on quality and security standards to be applied equally rigorously across the board. Unlike commercial software, there’s no standardized process for dealing with new updates and fixes, and often open source projects lack resources to find and patch bugs. Who knows how many unpatched vulnerabilities there are?
Driven by the fast pace of DevOps, developers who pull and use code from public repositories can unknowingly be incorporating vulnerable, risky, unlicensed, or out-of-date components into their project. 75% of commercial codebases contain open source security vulnerabilities, and nearly half contain high-risk vulnerabilities. Because of the distributed nature of the open source, a vulnerability can remain undetected for a long time, and an attacker exploiting it can do so for a significant period.
In a cloud native environment, containers add another layer of risk on top of open source concerns. For example, 86% of images from Docker Hub were found to be configured to run with root privileges as their default. This broadens the attack surface and enables an easy way to privilege escalation. If attackers manage to exploit a vulnerability in one of the open source components inside the container running as root, they get full access to the host and the application running on it. Combine open source with a poorly configured container, and you are on a path to a security nightmare.
A lucrative target
As more and more organizations are leveraging third-party components, attacks that leverage open source have become very attractive to bad actors. They are investing more time and resources in targeting open source components with malware to infect organizations earlier in the software supply chain. Popular package repositories, such as npm, PyPI, and RubyGems, have become for threat actors “a reliable and scalable malware distribution channel”. Over the past year, the sheer number of supply chain attacks has surged by 430%.
The attacks are getting more sophisticated and have the potential to reach an extremely large scale. In 2018, an attacker compromised a popular npm package, event-stream, which had at that time over 1.5 million weekly downloads and was depended on by nearly 1,600 other packages on GitHub. A malicious user managed to gain ownership of event-stream by making several significant contributions to the package and asking the original author to take over its maintenance. Any developer who used this particular dependency in their application was immediately put at risk.
We are also seeing an increase in attacks targeting the software supply chain of cloud native environments, placing hidden malware in publicly available container images. Just recently, Aqua’s cyber security research team, Team Nautilus, uncovered that even the SaaS services used by container developers, GitHub, Docker Hub, Circle CI and Travis CI, can be abused by attackers for cryptocurrency mining.
The challenges for organizations
Despite serious risks, the use of open source is generally poorly understood and inadequately controlled by organizations. Most breaches are caused by failure to update software components that are known to be vulnerable for months or even years (remember the Equifax hack?). Even though over 85% of open source security vulnerabilities are disclosed with a fix already available, many organizations don’t have the process in place to implement those patches. Enterprises are also struggling to keep up with the increased rate of reported open source vulnerabilities. In 2019, their number skyrocketed to over 6000. That makes keeping track of the newly disclosed vulnerabilities and their patches almost impossible to carry out manually, especially at scale.
Open source vulnerabilities are not going away as humans write code and will continue to make mistakes. Because of the nature of today’s software development, there are always going to be third-party components used in the process. Even if an application is deployed without a vulnerability today, at some point in the future a new vulnerability may come out for a particular component version within it. The question is not whether to use open source or not, it’s about how can you safely take advantage of open source while addressing the risks associated with it?
Leveraging open source in a secure way
To minimize the attack surface and overall risk exposure, it’s vital for organizations to gain comprehensive visibility and control over open source. As the use of open source continues to grow, there’s a clear need for SCA (Software Composition Analysis) – an automated solution to evaluate and manage the risks caused by open source vulnerabilities within third party components.
In the context of cloud native applications, SCA is the basis for more contextual and comprehensive risk-based controls. SCA tools look at open source components in general, but don’t necessarily provide visibility into the image structure, dependencies, layers, or configuration, all of which may present additional issues, or amplify the risk from known vulnerabilities. And securing open source can be done using open source tools, such as the Trivy image scanner.
Finally, it’s not enough to identify vulnerable open source components, you need to understand their actual impact on your specific environment and, based on that, act to effectively remediate or mitigate them. As a customer told me once: “It’s great that I can see all the vulnerabilities in my images, but now I can’t sleep at night.” A holistic cloud native security strategy goes beyond identifying vulnerabilities and security issues to include a complete and automated workflow to address them.
Cloud native: More secure software, faster
While the widespread adoption of open source components poses serious risks, with the right strategy and tools you can efficiently address them in your cloud native CI/CD pipeline and significantly reduce the attack surface. Cloud native is a great opportunity to do security right from the outset and empower developers to be aware of security risks and deliver more secure software faster. In case vulnerable images slip through into your production environment, you will be aware of the risks and have compensating security controls in place.
Benjy Portnoy, Senior Director of Solution Architecture, Aqua Security