In O'Reilly's “State of Microservices Maturity” survey, more than half of all respondents found that their new software development projects were based on a microservices architecture. By breaking down the more traditional three-tier application to a service that is comprised of independent units, aka microservices, developers have found they are able to make changes faster and avoid some of the problems that can occur during change management for those applications. By working at the API level rather than at the application level, these components can be altered and upgraded whenever is necessary.
Microservices has also been assisted by the growth of software container technologies such as Docker and the rapid rise of Kubernetes for container orchestration. These approaches have made it easier to manage scaling microservices applications up and down in the cloud.
However, it is important to remember two things: first, microservices is not being adopted because it is a great theory for software development. Instead, it is proving its value: businesses using microservices can address new requirements and roll out those changes faster. In that same survey by O’Reilly, 86 per cent of respondents stated that the migration to microservices had been at least partially successful. Second, these applications create data, and that data has to go somewhere beyond the application itself.
So, how do databases fit into the new world of microservices?
Microservices applications and data - stateless apps need stateful data
Microservices applications are stateless by design – the components exist in-memory for as long as they are needed to fulfil their compute requirements and are torn down and deleted or scaled back. However, those components still have to interact with a database to retrieve transactional data when it is needed to complete a function, or they will write data to store it for later.
Integrating with a database or multiple database instances will therefore have to be designed into the overall application. This will also be an opportunity to fulfil all the other requirements that companies have around the data they create, from creating additional copies of data for resiliency, for long-term storage or for compliance reasons.
Typically, each microservice would have its persistent data created in a single private location so that no other microservice could access that data directly. In order to follow a microservices architecture pattern, there are several ways to achieve this. The first option is to create a set of tables for each service that the application component can write to; alternatively, you could create a dedicated private schema for that component; the final option is to create a dedicated database for each service component to use. Private tables or schema will tend to have the lowest overhead and be suitable for components that create little volumes of metadata about their transactions. However, most high throughput services that create or use large volumes of data should benefit from having their own private database instances. In general, using a private schema or database per service is most appealing since it clearly defines ownership and provides encapsulation.
Where might you use microservices and databases together?
Alongside the technical aspect of combining microservices and databases, it is worth looking at where this approach can deliver business value to you. One common example of where microservices applications can be combined with a database per service model would be an application that uses customer data for an activity, such as an eCommerce app or a recommendation service in a customer-facing application.
In this kind of application, a separate set of microservices will be responsible for handling and managing data on customers, products, orders, and payments data management. These components will all be plugged together in a specific order to manage the process through from beginning to end, creating and using data all the way through. These microservices can be a mix of internal applications or third-party services consumed through APIs, and they can reside across multiple data centres or cloud providers.
In this example application, each customer should be able to get instant access to their order data or their shopping cart. For the company, these services should also fit alongside looking at analytics on behaviour or performance of marketing campaigns. Each set of data from the microservices components can be called up, combined and used to meet that request. Additionally, the retailer can use this approach to address high availability with no latency, and be in compliance with local regulations by keeping data within a certain geography.
To deliver this well, many companies are looking at how to use cloud services to support their microservices applications. Cloud can scale up and down based on demand levels, which fits nicely with microservices designs, and can be used to host the application components and data. Using containers, these services can run across multiple environments in multi-cloud or hybrid cloud designs.
Microservices, databases and cloud design
To run in these new environments, you have to consider how your database instance is designed. Traditional relational databases (RDBMS) may not be a good fit for this combination of hybrid or multi-cloud with microservices. The reason for this is rooted in the relational approach to managing data, which will tend to involve using fixed schema and architectures based on having one lead instance in charge of the entire database instance. RDBMS works well for smaller data volumes that are located in a single place, but it tends to become more difficult to manage and fragile at scale.
RDBMS systems can only scale vertically for both read and write operations, meaning that larger CPUs, memory, and storage resources have to be added to an existing host machine to accommodate growth. With cloud, this can be less of a problem, but even these instances will eventually hit their limits. Alongside scaling a single image, it is possible to scale out through sharding and using a ‘primary-replica’ architecture; however, it introduces additional complexity and increases the chance of failure.
The alternative to RDBMS is the NoSQL family of databases that have been developed to fit better with the needs of cloud-native applications. For applications based on microservices, getting the database instances as close to the software components can help reduce latency and improve performance. For databases like Apache Cassandra, it is easier to scale up and handle the higher volume of writes that software components can create, as nodes can exist in the same cloud instance or region.
Cassandra also has a ‘masterless’ architecture where all the nodes are in constant communication with each other. To support more data, additional nodes can be created in the same way as microservices can create additional container instances to cope with load. These database nodes then organise their handling of data so that all information can be easily replicated across multiple geographies where other instances of microservices are deployed.
Alongside this choice of which database to deploy, it is also important to consider who will run your database. For developers, standing up and running a database cluster might be an additional workload that they wish to avoid. Instead, looking at a Database as a Service offering might be a better option. This can take on the management and update overhead for the application while also providing tuning and performance improvements.
Facilitating microservices with the right database approach
As more companies want to adopt cloud-native patterns and best practices, the aim is to improve performance for applications and meet goals faster. Microservices applications support these objectives, but they are not enough to deliver this on its own. Alongside the applications, you have to consider your database strategy and how this will work alongside your new applications.
The next level of application development has to be dynamic in nature, distributed, and highly available by design, and seamlessly deployed to any cloud – your database will have to be able to achieve the same goals as well.
Patrick Callaghan, Enterprise Architect and Strategic Business Advisor, DataStax