 Welcome everyone, thanks for joining us. Today we're going to be talking about taking your database beyond the border of a single cluster, specifically Kubernetes cluster. My name is Ty Morton and I'm a Google Cloud customer engineer and I am joined by... Hi, I'm Christopher Bradford. I'm the Kate Sandra Project Lead and I work for DataStacks. So let's get started here and discuss the high level goals. So right now we're assuming that we're running a database and a single Kubernetes cluster in a single region. In this case, we're talking about Apache Cassandra running on six nodes, two per zone across three zones in a region. If you look at our example here, we're using the zone name for GCP, but it works for other clouds too and even on-prem. And here's where we wanna be, right? We want two regions, each with a logical Cassandra data center and that in turn translates to two Kubernetes clusters, one per region. Why would we wanna do this? High availability, better performance, geolocality, we wanna scale up. This effectively gives us doubly capacity, but you did not join the talk on why Cassandra is awesome. We don't have enough time for that. So let's stick into how we make this happen. So it seems kind of simple at first, right? Maybe I already have some automation to spin up a Kubernetes cluster that might be Terraform scripts, that might just be some sweet command line scripts. I might be clicking through the console. We can just deploy it, spin it up, call it done. Right, maybe it'll just work. Well, depending on your environment, it might work. Well, parts of it might work. Some parts might work a little bit. Some may make you think that they're working and they're not. You might have a stray firewall rule hanging out. In some cases, it could be just a couple of weeks and you're on to the next task, but in others, maybe not so much. So what we really wanna equip you with here is many of the common pitfalls that you're gonna run into as you go into this multi-cluster scenario and make sure that you're ready to go. And even if some of these things are working out of the gate for you, you should understand what those pieces are because maybe as you grow into a multi-cloud environment or you expand into another Kubernetes version or maybe you have different types of clusters out there, understanding these basics and the fine building blocks underneath the hood can really enable you to come up with some advanced solutions and ultimately something that fits your exact needs. So your choice of cloud provider, Kubernetes distro, command line flag and even DNS server or service can lead you down a dark and stormy path. So let's start by looking at networking, specifically how networking works inside of Kubernetes. To go back to our first cluster and take a look at the layers of networking involved, at the VPC level, we have a side range representing the addresses for instances launched within it. Here it's 10.100 and the instances, as you can see here with the slash 32s are given addresses out of that space. In some cloud environments, those subnets are tied to availability zones. So you might have a side range per subnet that your nodes are launched into. And when I say nodes here, I'm talking about Kubernetes workers. You might have other VMs inside of your VPC but that's what I'm trying to illustrate here. Now your Kubernetes cluster is gonna have a different side range for its own network. That's a network that's just used within the Kubernetes cluster and specifically that's the network where your pods are running. A pod is a collection of containers that have shared resources such as storage, networking, and process space. So here in our example, we have 10.100 for the physical well, even if it's virtual for the actual nodes themselves and then 10.200 for the Kubernetes level. And then down here, you can see the workers each get a slice of that 10.200 range for the pods that are running on that individual instance. So in this first worker node one, the pods are given a 10.200.10 address space and you can see hot A as an IP out of there. Same thing for node two and node three. Now, in some cases, there are multiple side ranges. Those would represent the pods and the other range would represent a service. It's not uncommon to see two sets of IPs within a single Kubernetes cluster but you should just be kind of aware that that exists. Now, if we think back to our target structure what happens if both clusters utilize the same or overlapping IP spaces? Well, it takes me back to think about this error message. When I was first learning about networking and honestly typing in IP addresses and hoping that the two computers still talk to each other, I remember seeing these messages come up. Maybe you see them from time to time on your machines. It doesn't look like this with Kubernetes. You don't have a little Windows system error that pops up in the GCP console. But it's worth understanding what's happening here. If I have a cluster that has one IP space and then I have another cluster with the same IP space or where they overlap, how does that cluster know when that particular packet needs to leave the bounds of its address space and route through the VPC network over to the other cluster and then into that cluster to outer space? If they're operating in the same area, there's kind of no hint there. I know Ty's gonna disagree with me a little bit because there are some ways around this, but at a high level, if you're overlapping, you're asking for a bad time. So this leads us to our first point really. You wanna understand your address space for each cluster and carefully plan the assignment and usage of those IPs. Now you might be saying, Chris, we don't have enough IPs. I've had that conversation with users before. We can't give every worker and every pod their own IP address. It's literally not possible. And if that's the case, what you can do is take a step back and say, okay, well, what services absolutely have to be uniquely addressed? And what services can be running in the same address space? Maybe because they aren't talking to each other. A good example of this is maybe your day-based tier requires it because each pod needs to be able to talk to each and every other pod, but your application tier, any East Coast and the West Coast, maybe they have their own dedicated Kubernetes clusters. Those can be in the same size or space because they're not reaching out the way and talking to each other. They're just talking to their local data layer. They're a data Kubernetes cluster. So that's an option to get around this. And that's what that particular user ended up doing. But really we need to be clear on where the deniation is of these services and how those addresses get assigned. Now, ultimately what we're doing here though is we're flattening out the network. We had non-overlapping standard ranges. Now we can move on to routing packets between pods in each cluster. And just to call out the particular values here, we are very specific. You can see the West Coast is 10.100, the East Coast is 10.300. The nodes obviously have IPs coming from those ranges. The Kubernetes clusters have their own IP space, 200 versus 400 here. And then the pods again are sliced off just like they were previously. So we have a bunch of IP addresses. We have uniqueness to those addresses. How do we handle the routing of this data? And how do we handle the communication and discovery and all this? Like what's next, Ty? Can you hook us up with that? You bet. And the answer is it depends. Not the best answer. I think really what we need to do is focus on how do we go from it depends to giving you some options, if you will. In some cases, it may just work out of the box. You may get lucky. Some cloud providers provide VPC routing. And so a lot of that purple smoke and magic is left into the cloud. And you don't have to do a lot of heavy lifting around that. In other cases, it may require you creating automation or worst case, manually creating routing tables. But the reality is that, Daemon sets, BGP tooling, things of that nature, VKNs are all certainly possible answers. That may not always be the best answer. And some of that's gonna depend on what this multi-cluster architecture looks like. Is it within a single cloud provider? Is it multi-cloud? Is it hybrid cloud where it's a combination of on-prem and maybe two cloud providers? But while you could certainly go instrument all that across all those different environments, it really can be a little bit overwhelming and require a lot of time and effort. So how do we take things up a notch? What is the, maybe not the easy button, but an easier answer is overlay networks. In this model, you would build out separate IP address space for the application, in this case, standard database. And you would run that on top of the existing CUBE network, leveraging proxies, sidecars, gateways, ingress and egress controllers, whatever the case may be. It's probably a little bit too much to go into specifically, but I know there's lots of content around how to do that at a high level. So what's next? Assuming that we get the network in place and we have all the appropriate routing, some connectivity between these clusters exists, at least at an IP layer. The next thing that you're gonna have to deal with and can often be a little bit of a challenge as well is service discovery and DNS. Yes, we have IP connectivity, pods in cluster one, can talk to pods in cluster two, but we have some new things that we have to take into account. Those pods are going to need to be upgraded at some point or you may need to redeploy them or you may need to add additional pods to scale the application. In that scenario, now we're introducing a lot of dynamic IPs and how in the world do we resolve DNS in a dynamic IP environment across multiple cloud providers and my on-prem data center potentially, like what does that look like? This could also be done via automation or some other manual intervention if you will. If you wanna leverage DNS services within each cluster and you really wanna manage all that, you can tell Kubernetes to forward requests from cluster A to cluster B and vice versa. And you would do that through some sort of a stone, right? And so we have an example here of forwarding requests for US West 4 from US East 4 to the specific DNS server that lives in that zone. And then I have to go manage all the entries in there. The big gotcha here is that it's gonna require a different DNS suffix which is set through a Kubelet Fag is probably not an option on every flavor of Kubernetes, specifically if they're in a managed environment, you're just not gonna have this ability to do this. It is possible to work around it by tweaking the namespace names in each cluster and then setting up stubs and all this requires at the end of the day, a lot of DNS knowledge. And if you have that expertise and that's what you really wanna do, more power to you. Again, back to the, it depends answer. You know, what works for one environment may not be the best solution for others. So what other solutions do we have that are like that? Well, the next option would be a managed DNS offering. In this scenario where we're talking about cloud DNS, it abstracts away a lot of the traditional overhead that you would have. The cloud provider is going to manage the scaling, the monitoring and security patches and all of that stuff, if you will, as a managed offering. And so that will offload certainly a lot off your plate. There are some added benefits to some cloud providers such as GKE provide, you know, node local DNS cache. So now I can reduce the latency by running a DNS cache, you know, at a lower level. And I'm not waiting on DNS responses, you know, clear across the globe from East to West as was the example. You know, obviously you gain all the things such as integration to, you know, their operational tools as well. And you can even set some scopes, you know, for multi cluster, you know, VPC wide or maybe within the single Kubernetes environment. So there's certainly a lot of options and this helps offload a lot of that heavy lifting. I think, you know, long-term though, when you start looking at a managed offering, specifically for DNS, if you're only in a single cloud, that's great. But if you're talking about spanning clusters across multiple cloud providers and your on-prem environment or one cloud provider in your on-prem environment, those managed offerings may only help with part of the equation. So what are some of the strong alternatives that we could look at? And, you know, keep in mind, these are just a few examples. You know, the CNCF provides a multitude of options and there are tons of open source projects that really have come a long way in helping to alleviate some of these pain points and really abstract away a lot of the heavy lifting and provide a better alternative with a lot of really big benefits, especially, again, in that cross-cloud, multi-cloud, hybrid cloud type of scenario. This example, we're looking at, you know, Istio and Silium running side by side. So it doesn't have to be an either or, it can be an and. If you already have Istio deployed, you can certainly add, you know, something like Silium on top of that. You know, pod IP routing is really kind of the foundation of multi-cluster capabilities and obviously both of these tools provide that, you know, functionality today. And really the goal is, you know, how do we streamline, you know, pod to pod connectivity and ensure that, you know, they're able to perform multi-cluster IP routing? Like that's where we want to get to. How do we do that? Well, as we mentioned earlier, you know, as I mentioned with overlay networks. So it is possible to tunnel all of this through encapsulation. With that comes pros and cons. You know, the advantage is that, you know, the pod IPs are never visible on the underlying network. You get the added benefit of encryption. This advantage to that is, is that, you know, those things require additional CPU or you need some kind of server-side offloader. Not a big deal. It's pretty common these days. The other potential disadvantages, you know, around MTU, we're encapsulating another packet. So we're gonna give a little bit up there. I think, you know, at the end of the day, there are lots of options that exist in the CNI space and service mesh and discovery and all those great things that can help kind of eliminate all that heavy lifting that you would have to manually do around not only IP routing but DNS service discovery and ensuring, you know, end-to-end connectivity. And, you know, these products not only provide all of that functionality kind of bundled up into a single solution or maybe a couple of solutions, but they also have some really big additional benefits that you're not gonna get out of the box like you would if you're, you know, manually configuring DNS and you're manually configuring IP routing and having to map all that out and, you know, document and automate and orchestrate all of that heavy lifting. One of the huge benefits that you get, you know, Cilium has done a great job creating a plugin architecture that runs on top of, you know, EVPF and it really gives you application-level visibility. So you can actually start to create policies that are beyond what you may have seen in, you know, or leveraged today. Instead of saying, I wanna create a firewall rule to ensure that, you know, this application can only talk to this, you know, Cassandra server. You can actually take it down a few notches and say, hey, I only want it to have read access or you wanna restrict it to, you know, specific records or tables. That's just not something that's possible with a lot of, you know, the existing tooling that we've used in the past, whether that's, you know, again, VPNs and firewalls and all that great stuff. The other thing is that all of this has created a lot of complexity and, you know, cube section, if you will around, you know, layers upon layers, overlay networks. And so how do you ensure that you have visibility and really how difficult is it to instrument all of that stuff that, you know, we just talked about? And especially in the case where maybe you're managing DNS on your own, that it's just one more thing that you're gonna have to go collect logs, you know, gather metrics, create dashboards, like it just creates a lot of additional overhead. When you look at projects, you know, galley or Hubble from the same folks that are bringing an assilium, I think the big advantage is not only do you get all this instrumentation out of the box and observability and it builds, you know, a visual representation of who's talking to what, but also showing you kind of the health of all those pods and just tons of like fine-grained visibility that you won't get traditionally. You know, I think that the key thing here is that not only does it do all that, but by the way, it can also, you know, help instrument on the traditional kind of, you know, monitoring metrics side of the house where most of these tools have the ability to integrate tightly with Prometheus. And now I'm getting, you know, metric data on connectivity between all of the pods and the applications and where the latency may or may not be as well as, you know, what policy is potentially being impacted. And yes, you could certainly, you know, instrument that in the past. I think, you know, we've all been there and done that where we've, you know, collect logs to some central log aggregator. We build custom searches, we create dashboards, all of that good stuff. But at the end of the day, this kind of happens out of the box. So how do we get from all the great things we've talked about in these slides to actually deploying this into an environment? And with that, Chris is gonna talk a little bit about how do we deal with all of this? Yeah, so you're not in a single region or cluster anymore. There's gonna be a bit of juggling involved. You might be pushing manifests and resources to each cluster one by one. Maybe you're templating things out and using tools like Helm or maybe some GitOps or other pipeline tools to make sure that you are staging appropriately and you're working through different environments. But really there's still a lot more that is required when you're in these types of deployment. So like one example here is Helm. If you're using Helm, you're gonna have a release per cluster and you're gonna have to maintain and manage switching between those various contexts and making sure you're upgrading the right way. And then dealing if things goes, like when things go sideways, how do you stage a change or roll back a change before you switch over and do operations in the other cluster or the other region? And when you go beyond two regions, there's even a bit more complexity there. Now I'd like to call out the operator framework here and really the operator SDK and the individual operators that make up a number of the things we talked about today. Some of these are starting to level up now with multiple cluster functionality where in some cases you're running instances of their operator inside of each of the clusters and they communicate and lock and perform when they go to perform various actions. In other cases you might have a control plane where you're running the operator and then it's operating on its reconciling resources in the downstream clusters. So maybe we have an ops Kubernetes cluster or maybe just US West 4 is running the operator but it's communicating with the Kube API and US East. We're currently doing that in the Kaysander project where we're going from Helm charts to an operator that has Kube configs and service account credentials to talk to remote API servers and to reconcile resources across those boundaries because some operations need to happen serially. Maybe if a node is down in one DC, we don't wanna do a certain operation in another DC. And so having operators that can communicate across this cluster boundaries can be really advantageous especially when you're talking about orchestrating these pieces together. And again, we talked a lot today about manually managing things versus having cloud native technologies that can manage them for us whether that be service discovery or routing tables or even just adjusting the packets in flight to indicate what cluster they need to go to and eventually what they need to get down to. So really as we go to wrap this up though and think about the application of this, what's the big moral of the story? Well, first you need to plan your deployments, get the right people. I would say in the room COVID has changed things a little bit get the right people together though whether that's on Zoom or what have you and hash out like how are you gonna do this, right? Where do you have limits in your system? Maybe you have a scarcity of IP addresses. Maybe you were looking at running one big cluster and now you're talking about many small clusters. Maybe you do clusters along the lines of business or certain ops teams. How are you going to start to venture into this multi-pluster multi-region space and ultimately building the plumbing and the pipes between those systems to communicate with each other? And that could be a single team, it could be but really you're probably looking at operations teams you're looking at people that run the cloud accounts you're looking at people that run if you're a hybrid or multi-pod you probably have some network people involved too. Maybe there's firewalls that need to be adjusted in certain ways and really doing a lot of this planning up front it's really beneficial and will pay out huge down the road as you're getting into implementing. In some cases after you've launched the cluster it's not something you can just change I can't just change the site or space that my pods are running and after I've launched the cluster I would then need to migrate. And so if I do some of this planning up front I can avoid some of those unfortunate situations. So in addition to this, Ty and I we built on some amazing projects here today we've got a bunch of references here just covering a bunch of the pieces that we talked about whether you're going to roll your own solution in the space or you're going to rely on some of these really amazing technologies like Scyllium, SCO, Linkerdee if you're using managed services like Google's Cloud DNS it's really amazing to see some of the stuff out there. So we've included a bunch of references here check out the slides and with that, are there any questions?