Skip to main content

Why the success of your Microservice architecture migration is determined by the data layer challenge

(Image credit: Image source: Shutterstock/Kalakruthi)

We all want to get the advantage microservices offer. The challenge is that there are lots of moving parts —and the trickiest part to get right is the one you may leave to last after you’ve sorted out the infrastructure and the applications elements: how will your great new microservices stack work with information and business data?

On paper, you could write microservices in whatever technology you want. That's because they are an idea, not a specific product; all they are there to do is to break up an application into smaller chunks with as few dependencies as possible between those chunks. Why that matters to developers is that apps then become easier to develop, but also easier to maintain, modify and upgrade. 

And that starts mattering to the business, as today your pace of change as an organization is directly related to the ability to change whatever application, webpage, anything that you're doing, quickly. If your competitor adds a shopping basket, you can too; if you come up with a new idea where you want to offer purple widgets every Tuesday, then you want to be able to do so and do not want your IT team to tell you that it is something they’ll try and look at in six months if they get a chance, which is what used to happen in the commercial data processing.

Flexibility, scalability, resilience

Having lots of independent microservices coordinated and working together is great for the flexibility and agility of a digital business. It also gives greater resilience: when you log into Netflix (a very committed microservices user), it offers some options customized to you, but if that microservice is unavailable for some reason, it goes off and it asks another microservice, what's the best in the genre. If that’s down, what's the most popular in this part of the world, and so on until it gets a hit. Instead of losing your whole website, which is, again, what used to happen, and we’re glad we’ve moved on, if little bits of it don't function, they can be substituted with content from other places and a good customer experience is maintained. 

You also get scalability; instead of having to depend on one big server, you can put it in something like Kubernetes and scale up when the system gets busy and scale back when services aren’t so busy. It’s clear that the move to microservices for an enterprise makes a lot of great architectural but also business sense. People are on this microservices journey, where some are a long way down the path, but there a lot only in the middle or just starting. 

It’s you I want to talk to now. I know you are reworking your major platforms, and planning on how to move from monolithic applications to the point where it’s all microservices. But what I need you to focus on is that building all this has some consequences--most assuredly around not the application side, but the business database side.

What can happen here: you’ve left the database service in the same monolithic Oracle/DB2 database you’ve always had. The problem is that traditional databases just can’t act as a dynamic, elastic, scalable service underneath your great new microservices. You have moved the application layer to microservices, and have gone for the best available tool, but you haven't got a database that is geared up for what they're doing. 

‘It turns out that the database side of a microservices move is actually a lot trickier than you might have expected’

To be explicit, you haven’t got there if you have moved to PostgreSQL, which is fantastic but doesn't make geo-distribution easy, or to a cloud-native database like Google Spanner or Aurora which have their own limitations. By the way, Postgres is the right interface for modern databases in a microservices/cloud architecture, but you don’t want a kludge like a translation layer to make it work, as that just adds more complexity back into the system.

It turns out that the database side of a microservices move is actually a lot more challenging than you might expect. There are a lot of complicated workarounds to get what you want, like having extra replication and complex environments, or using established products that mean you still need to juggle cost and complexity without the advantage of cloud-native aspects being built in. But you want to be multi-cloud or at the very least on an on-prem hybrid, using microservices for all they are worth, and with a nice, agile modern interface.  What can be done?

I would argue that only systems that have taken the inherited power of SQL, open-source and relational that you get with native layer cloud-agnostic and multi-cloud modern databases can even start to help you. You need something that spans the intersection of what you want from a traditional transactional database like Oracle and what you want from a modern database like Postgres. Even better: the high availability and the enterprise features you get from being cloud-native.

Don’t spoil the project with a database fudge

The reason this is important to any migration to microservices is that, however good your development of microservices is, it’s going to be compromised in what it's trying to achieve until it has an equivalent storage layer sitting behind it. So, if you are going to build a modern application layer with microservices, then you want a modern data layer to hold the data underneath that; it’s just not sufficient to have a good application, but not have the equivalent at the storage layer to manage the data in the way the business needs.

In summary, the best practice I have seen and what customers are telling me is that any move to microservices can falter if you don't fix the backend data storage. Your microservices need to be able to use the best-of-breed database for their applications, so you don't want a big Oracle database or something similar underneath it; what you want is best of breed for transactional, analytical, whatever you need for each microservice. I’d even say that if you asked the little beauties, microservices themselves inherently want to use the best-of-breed database for the type of data storage layer that they're doing, and therefore, picking the right databases for your data layer (and for your core transactional systems) is critical. 

If you don't do that, you compromise on the value that you get from your microservices.  Perhaps even likely, make the whole project too expensive to get the value from the idea—or even, worse, see it crash and burn completely.

Really—don’t do that.

David Walker, Field CTO, EMEA, Yugabyte