Everyone remembers when Zuckerberg came out with the Facebook motto: ‘Move fast and break things’. Few recall when he changed it to ‘move fast, with stable infra’. This was a Zuckerberg joke that recognized that moving fast had made things a buggy house of cards and no developer wants to build on something ready to fold up at any moment.
Today, every business wants to move faster and Covid-19 has only accelerated the urgency. However, this leads to some difficult potential trade-offs on how to deal with data and make the most of new, cloud-native approaches. How can we move faster with our data and not break things as we continue that cloud migration? How can we make the road simpler to build, faster to get onto, and easier to use?
The cloud-native puzzle
Cloud-native is defined by the Cloud Native Computing Foundation as running applications in modern, dynamic environments that can scale more easily using technologies like containers, microservices and declarative APIs. Containers have solved the problem of basic infrastructure portability as they wrap and isolate apps into infrastructure agnostic images. Microservices deployed in containers communicate with each other and the world through declarative APIs. This component level delivery model makes it easier to scale your applications up and down based on demand.
At the bottom of this infrastructure stack, you have technologies like block storage and databases. Communications take place between microservices using messaging with network and service discovery services helping glue things together. When this architecture is deployed on a network they need to communicate efficiently and with service discovery, they need to be able to find each other. Put it all together and you have a reasonable chance at stable infra.
So how do we embrace cloud-native data without tripping over ourselves?
The answer has been around for a long time in distributed data and NoSQL. These approaches were first developed over a decade ago, as we went through a transformation of the internet, where we saw millions of users all wanting to use applications and websites at the same time. The likes of Google and Facebook emerged and had to develop new technologies to cope at this scale, as traditional relational databases couldn’t keep up.
As more companies want to find some of the same magic and work with data at scale, they need the same kind of data infrastructure that these companies developed. The entire NoSQL database category has experienced explosive growth as companies want to run operational and transactional databases that can cope with millions of users. This scale problem meant that adoption has been a long process, but today, we are seeing more development of AI driven operations to help in that process too. This automation is needed as implementations are becoming so large and complex, there are crucial areas where we need to have machine helpers.
Are we there yet?
I tend to compare the progress of cloud-native databases with the evolution of the self-driving car: we may have Waymo opening up more access to its fully driverless ride-hailing service in Phoenix, Arizona and Tesla supplying a full self-driving beta to “careful drivers”, but we don’t have cars driving us to our destinations yet. What we are seeing is the common application of AI assistance for safety reasons.
When I drive my Volvo it assists my driving. If a car pulls sharply in front of me, my car hits the brakes before I can even think about it. In a similar way with cloud-native databases, we need that automation. There’s so much configuration and much of it falls on the shoulders of a single person. It’s too much. I feel it’s almost going to break the internet if we’re not careful. We need that assistance.
Right now, large companies are using and relying on Kubernetes to automate and manage their applications, and the same approach is needed for cloud-native data. For databases like Apache Cassandra, using Kubernetes operators can help automate the management process and help this process. When the community started building Apache Cassandra and NoSQL systems, we built it way ahead of what’s happening in cloud-native, and now we want to bring cloud-native and Apache Cassandra together to create the best combination.
Failures are an inevitable part of any infrastructure, so another aspect of cloud-native and Kubernetes that helps you sleep at night is self-healing. When something bad happens to one node, Kubernetes can fix that. It detects that the state was supposed to be and just adds another node into the system to replace the failed one.
The Cassandra & Kubernetes combo
Apache Cassandra takes a very similar approach and it does resilience like no other database. It provides the ability to run across multiple locations and cloud services, so that there is no single point of failure. From a cloud-native perspective, this means that the data implementation alongside any applications is available and reliable, as well as able to scale.
Cloud-native data is going to be a significant part of the next phase in Kubernetes adoption. So, what we have been putting forward in cloud-native compute and to cloud-native communities is Cassandra as the underlying infrastructure for a lot of your dealings.
The world we’re living in right now has the database at the bottom layer and below this is block level storage. But cloud-native is affecting everything: developers want to work with APIs rather than directly with infrastructure, while every part of the infrastructure should be dynamic, scalable and self healing. If you had this deployed as your application infrastructure, I think you would be very happy, because you can answer questions, you can respond to changes in the world, and you can grow as you need to. If the business wants to extract value out of another part of your business you have the ability to do it quickly.
So what does this mean? Over time, the focus will shift from what databases and infrastructure is in place, and more onto how to use the data for applications and services. This will mean having database infrastructure that is so solid and reliable that it can be taken for granted, regardless of any issues that take place or any scaling requirements. For developers, using data based on APIs will make things easier for them, while DevOps and IT Operations teams will automate the database side as much as possible, so it can be available as a service to the rest of the business.
What we’re seeing in this data-driven enterprise world is that this declarative application infrastructure is accelerating. It’s moving very fast within Kubernetes, but what's after this is how the operations and processes around data will become cloud-native as well. This set of Data Operations processes - or DataOps - is almost the last frontier for large scale application infrastructure. Over time, there will be no more of this separation of concerns.
What that means is that DataOps is going to be more of a service; you're probably going to be renting your data operations, you're not going to be deploying it. It will be cloud-native, so wherever I am in the world and whatever I need data for, it's available. The impact here is that data will be more available and more useful for developers to build applications and services with. Looking at the road ahead, support for moving fast around cloud-native data and infrastructure will be essential.
Patrick McFadin, Vice President Developer Relations, DataStax