Securing the DevOps deployment pipeline

The growing popularity of DevOps practices in recent years has led to an increase in automated deployments to test and production systems. The benefits of automated deployments are numerous but these deployment systems are often designed without taking security into consideration.

An informal survey of Internet domains recently concluded that one in 600 sites currently serves up a Git directory when a request is made for the base URL appended with .git. This happens when a git based deployment system is used and the web server is not configured to deny these requests.

Unfortunately it is still somewhat common for developers to include credentials for test or even production systems when they initially create a project, especially in config files. Even if these sensitive values are removed from the project at a later date, they are still exposed in the projects commit history and can be viewed by anyone with access to the Git directory.

This is just one example of a security issue that can arise due to a legitimate mistake. While it’s always a good idea to try and minimise mistakes like this, it’s impossible to eliminate them completely as long as humans are involved in the deploy process.

The guidelines below are designed to limit the likelihood of a security mistake happening and in the cases where a mistake does occur they will limit the impact of a security breach.

Cleanup and prepare

Before a project is checked in to a repo, or deployed to any public facing location it needs to be cleaned up. This includes removing any test code which could be a security problem as well as cleaning up any test values in configuration files.

All configuration files need to have placeholder values and not credentials for test or production systems. The actual values for these config files should be stored somewhere secure and access needs to be limited only to those with a compelling need. Alternatively, credentials and config files can be encrypted and then decrypted at deploy time once they have been transferred to the target environment.

There any many automated tools that can help with the task of encrypting these values and performing decryption at deploy time. You will want to choose a tool that meshes well with your existing deployment processes. If you’re unsure about which tool to use, consider consulting with someone who specializes in the field of deployment and automation.

Isolate credentials

Each of your systems and environments should have a separate set of credentials dedicated to that system. Developers should also have separate credentials. No authentication mechanism should be shared between multiple systems or individuals.

In some cases it is also helpful to have separate sets of credentials for various tasks within a single system. An example of this would be a read only login for an individual that is not authorised to make changes to a system. The full read/write privileges would be given only to individuals with a need to make system changes.

Some systems have better support for limited privileges than others so this approach has to be implemented on a system by system basis.

Train staff

Security awareness training is absolutely essential to any security plan. Without awareness training it is very difficult for even the most conscientious staff to know what to consider when making security related decisions.

Awareness training is also one of the most effective countermeasures to prevent social engineering attacks which are becoming an increasingly common element of sophisticated hacks.

The type of training each staff member receives should be tailored to their position and expertise. Customer support representatives for instance would generally receive more training around social engineering while a software engineer would probably need more specialized training in security practices for particular systems or frameworks.

Log everything

Logging system activity actually has two benefits. If an intrusion is detected, logs allow you to know exactly which systems were changed and how severe the effects of a breach might be. This is a reactive approach to security however and there is an even better way to detect intrusions.

Instead of reading logs after the fact, they can also be monitored in real time and alerts can be triggered when suspicious activity is detected. There are a number of tools that are capable of this type of monitoring but they are generally system specific and require specialized knowledge to set up.

Log collection systems greatly facilitate both of these approaches and should be an essential part of any security plan.

These security strategies aren’t comprehensive but they are a great start for any team making the DevOps transition. DevOps is all about continuous improvement so make sure you stay up to date on the latest security best practices as you go along.

Caleb Fornari is a Co-Founder and Senior Consultant at StartOps Group