 appreciate it. This meeting is being recorded. So nice to meet you all as my name is Keith McClown and the director of partner solutions engineering here at Cockroach Labs and I'm joined by my friend and frequent project partner Raphael Spazoli. Do you want to go ahead and introduce yourself Raphael? Sure Raphael Spazoli. I work at Red Hat in consulting architect in around everything around open shift both application and infrastructure and it's a pleasure to be here with you Keith. Awesome well you know that we're big fans of what Red Hat does in the Kubernetes community and glad to have you here today. So what we're going to be talking about today is a presentation based on a blog that I wrote I guess earlier this year around whether or not Kubernetes is kind of the next fault domain and the argument that I'm making is largely that Kubernetes has become the common operating system for the cloud, for hybrid cloud deployments, multi-cloud deployments, multi-region deployments, anything where you need to be able to run a lot of the copies of the same application across multiple sites and maybe route users to them. The is largely getting run on Kubernetes these days and in my opinion Kubernetes meets all of the criteria for an operating system right, manages work and runtime right, manages the resources without explicit user interaction like when was the last time you picked which CPU core on your MacBook was was running your application you don't do that because the operating system handles that for you. It's got a common input and output layer so as a as an application developer, as an administrator, as a user I interact with the common API gateway and the system handles all of this for me and it abstracts away the complexity of running an application across the cluster of machines right. So the the thing that Kubernetes that's special about Kubernetes is really that that operating system isn't limited to a single machine. Not only does that open up some challenges to running an app across nodes in Kubernetes but it opens up this concept of having to build apps that not run not just across servers but across Kubernetes clusters. Effectively your Kubernetes cluster becomes your fault domain for building your application for building highly resilient applications. Now the reality is there's a lot of multi cluster challenges in Kubernetes. Some of them have been solved today some of them haven't right. There's challenges around networking there's challenges around load balancing and service discovery identity and trust. So like with networking we're talking things like there's there's CNIs there's EBPF there's VPNs there's these things called vans or virtual application networks. For load balancing you've got concepts like API gateways and service meshes for security there you have to have a consistent view of what a user is allowed to do in multiple different locations. That's a really challenging problem. Identity and trust. This is really how do I validate that the user is who they say they are right to try to avoid situations where a user that's working in one Kubernetes cluster isn't also someone isn't trying to pretend they're them in another cluster and do bad things. There's of course infrastructure and performance challenges. There's this whole concept of how do you manage failure recovery observability monitoring and then of course you know state which is someone of the problems that Cockroach TV can help with. Raphael you've built a lot of these applications over the years. Where do you see some of the biggest challenges in building kind of multi cluster applications on top of Kubernetes? Yeah one place where I had significant challenges is the fact that Kubernetes operators are cluster bound. So they can control they can automate and control everything inside the cluster but when you go multi cluster and multi region it's not that obvious how you can declare your intention as a user with something that impacts multiple clusters. You have to come up with some solutions that is not unfortunately that is not just limited to writing an operator that doesn't work. Other things that I had to deal with is trust and identity is an important one that you have here because in Kubernetes there are Kubernetes results so it's not just a failure domain in my opinion it's all a trust domain. It's very easy to validate that a pod is a pod and you know how to destroy and connect to another pod is actually what it's saying it is it's easy to generate certificate within a cluster but when you have to connect with between multiple clusters how do you how do you create that trust is a problem that you have to solve. And then another one is unavoidable but still something to consider that is latency right that's not really just a problem of Kubernetes it's a problem of it's just the way it works when you do deployment across multiple regions but latency will impact several decisions so you have to take that into account. Yeah I mean we spend a lot of time talking about latency in cockroach DB right. We've got a bunch of optimizations to make the read path super fast but we do if we're in a multi-site configuration we largely are always going to be impacted on the right side when we're writing to multiple regions we've got to wait until a quorum of them are updated. I'll double down on your operator comment we have an operator for cockroach DB but it is it only works for a single site deployments right for customers who want to use cockroach DB across multiple physical Kubernetes clusters we recommend other install patterns that that allow the administrator to to be more explicit making decisions. The last thing we want to do is have like a API controller effectively that is making decisions based on incomplete information about the cluster right. What we've talked about what we could maybe do to solve that problem in the future. I know that the Kubernetes ecosystem and there's like kube federated and kube multicluster there are a couple of sigs that they're trying to find different ways to tackle this problem. I know this is something that the community has been talking about for a long time. We have the advantage of theoretically we could bootstrap this with our own database if we wanted to but we haven't we haven't gone down that path yet. I know we're going to talk a little bit about kind of the failure recovery mechanisms here a little bit later but I'd love to dig in a little bit on the networking right so because there are a lot of different options here there's kind of the C and C and I layer stuff ebpf right. There's there's also VPN based solutions like Submariner that we've talked about in the past and and then vans like Scupper. What do you think is going to be the long-term solution for multicluster networking in in Kubernetes? Yeah so the premise here is that obviously when you deploy these solutions across multiple clusters they need to communicate right and I call that kind of communication east to west communication as opposed to north south where you know that where which is the traffic that comes into the cluster and it's your workload. Here we're talking about the workload issues having to communicate with each other. So for east to west across clusters and across possibly you know data centers and cloud regions there are some solutions I think that that you have mentioned I think I I think that ebpf has a good chance of becoming the the preferable one. I think that Selium already has a solution in the space but in general I see that that could be a good technology to build this east west tunnel tunnels between between clusters. There is enough abstraction there where you know you could make it at the same time very efficient and and abstract a lot of the underlying complexity that you may find when you do this setup. So as an extension of that right there are definitely circumstances where pods in one Kubernetes cluster need to be able to discover pods in another Kubernetes cluster either for high availability purposes or potentially just for synchronization purposes. So what do you kind of see as the state of the art for like multi-cluster load balancing service discovery? Are you like a service mesh fan or you're an API gateway fan? Do you prefer like the CDN style approach where where you have set kind of like a global load balancer? Like what's your what's your philosophy there? My hope is that there is a specification for multi-clusters multi-cluster services. I think it's called MCS within the Kubernetes one of the Kubernetes sig. That's that's a specification that works well for tunnels between SDN tunnels between clusters where you don't need to involve egress and ingress to do that. My hope is that we can we can you know standardize around that kind of approach possibly that kind of specification. The other yeah the multi-cluster service mesh is another option here but there's no need to have the override of the sidecar you know for for just for discovery and and load balancing. I will mention because I forgot to at the beginning of the of the call we are we are accepting questions in the Q&A and we will be raffling off some t-shirts too. I think it's the five best questions that get asked so if anyone has questions please drop them in the webinar chat. I'd be happy to we'll be happy to try to field them live or any that aren't completely on topic for what we're we're talking about at the moment but are related we'd be happy to get to at the end of the webinar. Before we move on Raphael are there other multi-cluster challenges that we're not talking about here? Something I missed maybe. I think it's a complete list at least for as far as I can tell the monitoring and observability is a big one right it's yeah collecting collecting those metrics across multiple clusters it's not only you know you need to have something else where that uses a collector and then you have to solve the scalability problem and it's often you know as as you are more and more successful it can become an issue and it can become a bottleneck. I think I think we're seeing a lot of evolution in that space and the open telemetry standardization it's a very good thing. I know this is a thing that we had to build into cockroach DB was kind of monitoring capabilities for cluster wide monitoring capabilities so that we could see what any given nodes opinion was of the health of the rest of the cluster to help us diagnose these types of issues but that's a single that's a specific implementation. Do you see do you see any common patterns emerging around how how we're going to do this in the kind of a multi kubernetes cluster world across arbitrary applications? Yeah no the pattern is to you know produce the the metrics and the traces and collect them towards a common collector maybe applying sampling if need be but I see and so seeing solution like tannox tannox and I don't remember the other cortex right those two are good approaches but I see what in the actual implementation phase I see a lot of struggle within my customers in achieving the correct level of scalability scale I should say the correct level of scale. I mean distributed state management in general is challenging right we've solved it for for kind of relational transactional workloads but on an arbitrary basis there are a lot of different solutions that are out there and they all have their their strengths and weaknesses maybe we'll talk a little bit about that later here in the in the chat but I imagine that observability and monitoring has both that challenge plus kind of the single pain of glass challenge right where I need to I care about the the health of my application not necessarily the health of a given kubernetes right pod or even a cluster in a multi cluster world and I certainly care about when users have a degraded experience because they're being routed to a more remote look at remote location than than they necessarily need to be so there we do have a question in the in the q&a what's the solution to have georedunds redundant kubernetes clusters that are hosted on prem right I will say that in general the the customers that that I've worked with that had kind of either an on prem or hybrid cloud deployment most of them ended up using open shift right the I think the important thing with on prem clusters is at least from a cockroach db perspective is making sure that you have similar performance characteristics out of all of your different sites and you have reliable connectivity between those sites those seem to be the most important things for cockroach db right I feel you've built more hybrid application stacks than I have is there any is there anything else that you'd have to add on top of that yeah I um I think I don't see a lot of this difference between the cloud and on prem in terms of building you know geographical distributed solutions but I would say just the way the question is phrased we look for solution to our applications you know availability not necessarily to the cluster availability I would say it's more important to think about the application than the cluster let's treat the clusters as disposable um so at that point it doesn't really matter where they run it only matters that you can once some the disaster happened or some problem you have some problem you can bring them back right and you can reapply quickly the configuration possibly with gtops and then we can focus on making the applications available you know and resilient to a disaster um so rephrasing the question what is the solution to making application available um and especially stateful application when you want to deploy this application across multiple cluster possibly hybrid or you know multiple on prem geography I think we're gonna go down a little bit deeper into that question the rest of the presentation so so um we got we got one more question that I'm going to field while we're we're on this topic which is around upgrading cross cluster applications to a newer version so with cockroach db we um we support this out of the box and in a multi cluster deployment we generally recommend you upgrade one region at a time now cockroach db can roll pods and continue to have the service live even in the region that's being upgraded and the database is designed to be able to support running in kind of a mixed version mode during an upgrade so we can do that without an outage um my experience largely is if you solve that from a data layer perspective the the application services end up being pretty easy to upgrade um the uh the challenge is usually at the data layer when you have to like make schema changes or some other sort of data migration um and cockroach db was the designer of the box to solve that problem uh rafael have you experienced any other kinds of challenges with that type of uh pattern no I agree with you it's interesting I'm in the middle of those kind of discussion with a customer but yes I would yeah take care of doing the schema changes and database upgrades first and then the stateless piece of the application hits way easier to deal with so I'm going to kind of summarize my my take on this right in Kubernetes in general but particularly in multi cluster Kubernetes it's important to design your application stack to survive when failures happen rather than to have kind of a disaster recovery strategy which was the traditional way of handling applications um and and that largely means we need to rethink how we build systems we need to plan for the system continue to operate when something's broken rather than uh planning to try to recover from uh outage when something's broken and and largely the reason for this is in a sufficiently distributed system something's always broken uh you know you're running hundreds of thousands of pods for microservices you're running say 150 nodes of cockroach db across three sites maybe um the likelihood that you don't have some problems somewhere in your environment at any given time is is pretty close to zero so it becomes imperative to design the system to kind of be able to handle the the types of events that used to be considered a disaster now Raphael I know you're an expert in the in how how to build systems to survive disasters in Kubernetes so uh do you want to kind of lead this part of the conversation sure thank you Keith um so this uh table is uh from a um white paper that we wrote for the cncf tag storage that talks about this idea of cloud native disaster recovery I'm going to try to talk about it in in comparing and contrasting it with traditional disaster recovery so in traditional disaster recovery the detection of the disaster is usually a human decision right so something starts to go wrong there is a meeting the decision is made that we cannot recover with the usual you know ha type of approaches somebody press a big red press a big red button and we are now in dr mode right then then there are then a lot of stuff usually needs to happen some of it is automated some not but it's usually a manual decision in cloud native disaster recovery we want to or disaster management we want to we want that decision to be made by the system so the system autonomously realizes that a piece of it is not responding anymore and takes action and then the recovery procedure itself like I said usually what we see you know in large organization is a mix of automated and manual actions and that's the reason why you know there is this b annual biannual um you know disaster recovery exercise in in some places and sometimes it's well done sometimes it's not so well done but um it's there is manual action there is there are actions that are not commonly executed that people you know are not used to they may fail why are they doing why they do that and do them in cloud native disaster recovery everything needs to be fully automated so the machine makes the decision to that something needs to be done because there is a failure and the actions that we take are fully automated then rto and rpo are the two main disaster recovering metrics right one is rto is the time that it takes to recover the service rpo is the elapsed of data that we lose because of the disaster so in in in traditional disaster recovery depending on how you architect the system you could be between minutes and hours of downtime right in cloud native disaster recovery we we want to be near zero downtime essentially the time that it takes for a nail check to take place and the detect that there is something going wrong and start redirecting the traffic to the healthy locations and for a recovery point of jet rpo in traditional disaster recovery you can have from from zero to hours depending on the storage solution that you use right if it's backups it's probably ours if it if it's more of a synchronous file or volume synchronization it's probably it could be around zero but for cloud native disaster recovery disaster management it it can be zero especially if we are talking about strongly consistent solutions which is one of them and then now um thinking about the ownership of these processes right the disaster recovery processes so a little bit out of the technology more about the organizational aspects normally um in in large in large organization um the the ownership of the business continuity of an application is always in the application team but normally what happens is that the application team team turns to the storage team or the infrastructure team and says what guarantees can you can you give me for this database or for this uh volume right and and then they take whatever the answer is whatever is available in the in the organization sometimes there are tiers but essentially that the very limited choice my argument is that in cloud native disaster recovery because the the the responsibility of owning the disaster recovery process is going to be in the in the team and and because they will have to choose the correct middleware and they will be the owner of the middleware so and so it could be cocker cb or it could be etcd or it could be kafka but they are gonna own it or maybe they're gonna purchase it as a service and but still essentially they are the owner and then the other thing that I noticed doing these kind of deployments and this kind of exercise is that in traditional disaster recovery the actual technical capabilities that essentially enable the ability to recover it's for the disaster recovery procedure are backup uh log shipping or you know volume synchronization something that happens at the storage layer instead in in cloud native disaster recovery we need more capability from the networking layer in particular we need that east-west communication that we were mentioning before and we need a good load balancer global load balancer that can detect um issues and can make a decision about how to steer the traffic yeah so so we have a question in the chat about whether or not backup is still an option for for very large environments specifically I think the questions around data um at least with cockroach dbd answers yes um we we do distributed backup to each location so so that we can solve for that problem I think in general the the the question really then becomes how fast can you get those backups off-site right so so with cockroach db the data is replicated to multiple regions so even if you lose a single region you still have a full copy of the data backed up in another region but for regulatory reasons sometimes you do need to have like a tertiary storage location for for that type of stuff so is it an option yes but it does if you end up having to go to that it looks a lot more like a traditional dr type of a solution where it takes hours to get the data backloaded potentially um which is why largely we architect these solutions to survive what traditionally would be a disaster and treat it as an ha event because the because true disasters end up being incredibly expensive from like a human um perspective if it's a if it's a financial application or whatnot it can have a monetary impact on the company there's usually a lot of fallout to to having a true disaster happen so anything we can do to avoid um a disaster and treat be able to treat as an ha event is is usually best the one thing that rafael you didn't mention that i'm gonna i'm going to mention is that 70 percent of disasters are either caused or exacerbated by the human interaction one person doing one thing and in the wrong order um can can turn uh set like a failover from say a couple of minute type of an event to a couple of hour or a couple of day type of event um someone fat fingers a command someone deletes the wrong file somebody does something that they probably didn't even mean to do it's not like there was necessarily even intent but but people are the most fragile element in um in disaster recovery so the more you can remove the people the less likely you are to have a disaster so um thank you for for running with that um rafael jevony other context you'd like to add on backup i think it's an interesting question uh it comes up often i would say we if you if you can deploy a cloud native distributed solution you know whether with caucus or other um other of this new generation of um state distributed workloads you you probably still want to do backups but not to protect yourself against a disaster it's more to protect yourself against logical errors right say say you make a wrong deployment and you drop a table you know your code drops a table that was not supposed to be dropped now you still have a backup right so you have backups for those reasons not for protecting you for against disaster recovery against disaster there's always the chance that there will be a real disaster and you will need those backups so it's good to have them um you just actually answered one of the the other q and a questions about um um how do you account for replicating corruption so in cockroach db um the the replication happens at the raft log layer and the actual file system work happens through a kv store so um so you would someone would have to have submitted bad data for us to have um have like file system corruption propagate to towards multiple nodes in cockroach db if the if the node notices file system corruption it will remove itself from the cluster um and since all of the data is replicated at least three times that will still leave us with a quorum of replicas to process against and then after a handful of minutes the cluster will take the two remaining good copies and and create a new third clone from um and if you're really really concerned about that you can up replicate to five or seven uh x replication to for your most important data um there there was a kind of a follow-up question around uh does storage level replication make sense in a cloud native world um or would you only rely on on database level replication for me personally i'd like uh storage replication for some use cases in a single cluster perspective but multi cluster there tends to not be enough data guarantees uh to be assured of that consistency so you're i generally like either using some sort of a message bus like a Kafka to transport data between regions or a database like a durable state store like a database to transport data uh across a across Kubernetes clusters rather than block or file system level replication which um doesn't necessarily guarantee that um but you'll have a replica that's consistent as of a point of time but you're not guaranteed of what exactly that time is and you won't be able to verify there's no like transaction semantic to verify that the data has been updated right and so that's the more distributed distributed yet the more likelihood that those types of um issues will pop up and so I generally recommend either a message bus based solution or like a database based solution Raphael do you have any other thoughts on that no I agree I don't see file level replication or block level replication as a something in cloud native perhaps a block of just storage replication right that that's uh you know if you have enough storage and behind the scenes it's it's replicating it's doing things that yeah I guess that's a good point right for replicating backups off-site so that you might use a solution like that to do to a tertiary location right um but that's not going to be live block storage that's going to be file level replication not block level replication anyway right um so isn't this expensive right so in a traditional dr type of environment you have one site that's your primary hot environment it's processing all of your traffic and you have the second one that's either warm or cold and by warm I mean maybe you're doing some non-business critical work in the secondary site like running bi dashboards or something like that or it's truly cold where it's just there to receive data and context from from the the hot site right and both sites have to be able to handle a hundred percent of your production workload so that means that you have to be provisioned at two x minimum of everything plus whatever added infrastructure you need to keep those those two sites in sync and as Raphael mentioned the the failover and failback is expensive and risky even when the failover works really well the failback actually is a lot of times even more risky than the failover because everybody only tests in one direction I don't know if you've ever had this experience where Raphael where the failover works great and then they realize that they don't have any runbooks on how to fail back you've been have you lived that world before I yes I can attest to that yes yeah so um when you started to talk about cloud native disaster recovery um all of your sites are supporting users are hot effectively and that and that means when you lose a site your remaining sites are are there to support uh you distribute whatever work was on that site that failed to the remaining sites which reduces your total overhead of operating the solution so effectively in cloud native disaster recovery you're going to recommend three or more sites right that allows you to do consensus-based replication it allows you to not have an external observer deciding when a disaster has happened right at least with cockroach db when you lose if a site gets say has a network partition the that the database knows to shut itself down if it can't see a quorum of the nodes in the cluster so that you know that business operations aren't continuing to be processed in that site it's really important um but the real key thing is the more distributed you are and by more distributed I mean across the more sites that you're distributed the less overhead you have to survive any particular event right so if you're across three sites you need to have 160 percent of your total infrastructure provision to be able to continue to to run your production operations during a site failure there are four sites it's 35 to 45 percent five sites 25 to 35 percent etc etc at five sites you could also potentially if you wanted to design your system to survive two site failures can currently you could provision your infrastructure to do that as well now that would change your overhead numbers here but now all of a sudden we're in the rarefied era of having to worry about multiple region cloud regions going down simultaneously and because there's no failover or failback all you're doing is routing users to a different site when those sites come back online they just rejoin the cluster get caught back up and then start processing user transactions right cockroach db makes this really really easy we actually spent an enormous amount of time building the SQL engine to enable us to do that and the the great thing about using coverage the end of the covers for problem to solve these types of problems is it gives the apps that are running on top of it those same superpowers right so assuming that you're using either the database for all of your state or the database plus say a distributed message broker like a Kafka that also can survive these types of events all of a sudden now you're in a situation where a site failure say aws us one goes down or azure west two goes down or google's central one goes down you still in full operating mode maybe your performance isn't as good as it was when you had all of your sites up and running but you're able to continue to process all of your important business logic there is one question in the chat that I want to talk to here around what's my opinion for about data consistency in a multi cluster environment personally I think it's super important I wouldn't be a cockroach to be if I didn't it is kind of the problem we were designed to solve the one thing that I'll point out is that kind of traditional consistency requirements and either a single server or a single site are kind of have a limited window of vulnerability just based on the physics right a single server when you're in a like you're running a lower level of consistency let's say you're running like snapshot or repeatable read in your database or maybe running a non posix compliant file system right your vulnerability window to to like data entropy problems is really small right and on a single server you're talking microseconds and a multi server but single like physical site situation you're probably talking still less than a millisecond of actual vulnerability window as soon as your multi site your best case vulnerability window is is one round trip between your site and its next near site right just a couple of orders of magnitude more vulnerable there are certain use cases that can tolerate that type of vulnerability and if you can there are some great like no sequel type solutions that they can help you do state synchronization right in those types of scenarios but if you're dealing with anything that's like financial transactions inventory management anything that require like transaction or like accounting stuff anything requires any kind of auditable results i think it's exceptionally important to think about this before even you get started if you can right now granted cockroach db being postgres while protocol protocol compatible means that we can kind of come in after you've already built some of your app and and help you get moved but it's it's a little harder right so um let's go ahead and and talk a little bit about what this looks like with cockroach db right so we do a lot of multi cluster kubernetes um whether it is multi region across a single cloud multi cloud provider like i'm showing in this diagram a hybrid like a lot of the work that i've done with rafael in the past where you have some physical on-prem sites and you've added some cloud sites in all of them basically have the same fundamental challenge right how do you manage an app that runs across multiple kubernetes clusters so on a single basis cockroach db is actually really good at doing this on its own now for our um dedicated and serverless offerings our database of service offerings where we're running fleets of systems that are running across multiple kubernetes clusters we have built our own kind of control plane uh i think we're using an epb ebpf based networking solution under the covers and um we've certainly kind of built our own um identity and access management um plug-ins for both the on the administrator side and on the app side so that users can you know use the same credentials to log into the database regardless of which node or which site they've logged into um you know we've had to try to solve a lot of the the distributed backup problems that we were talking about here right um the thing that i like the most about cockroach db in general is um the database doesn't care um so the database works by when it when a node joins the cluster it it announces its location in physical location in the hierarchy and that allows the database to kind of self manage replicas to make sure that we're um replicating the data is appropriately for both the performance requirements the application and the resiliency requirements that the administrator has put in place by default we're going to deploy data in the most resilient pattern possible um and it also gives you a lot of flexibility on the network and um as well as having workloads be um hot in in all of these sites right um raffle and i didn't have time to get the the kind of the demo set up um for today but um we have on uh on a pretty regular basis have have done demonstrations of of just this thing where um well we'll set up red hat on all three of the cloud providers um or open shift on all three of the cloud providers and then run a single cockroach db cluster across it right um there's we have a lot of options as far as as that goes and one of the things i like the most about open shift is because it is not so tightly tied to a single cloud provider um you can have a common kubernetes uh operating experience by using open shift as your kubernetes distro in all of these locations so what does the future look like um i have started to already see the kubernetes ecosystem converge around some of these solutions like i think ebpf raffle i agree is probably where we're going to land for for a lot of the cross cluster uh network routing the east west network routing types of problems that we've been talking about um there's some great kind of multi cluster controllers and like cross plane and kubu multi cluster that are very promising you know identity management with thought said and spice uh spice db um key cloak like there are a lot of great pieces of future tech that some of which are usable in production today like ebpf some of which are still kind of in in development but um what's your thought on what kind of what's the next what's the next thing up that needs to be solved in the community i think uh um global load balancing is is the next is is a thing where maybe the gateway api can help but um i haven't really i still have to see a a solution that is not very cloud provider specific to declaratively say i want this traffic to be load balanced to these three regions these three cluster clusters and this the pods deployed in these clusters right um yeah there is as far as i know and i could i could have missed some some steps but as far as i know there is no specification there uh the the gateway api could probably support that semantic but i haven't seen any real implementation for doing it so that's important because the global load balancer like is like we said is not just a traffic load balancer at this point you certainly need that but it needs to be more it needs to be something that realizes that something is going wrong and steers the traffic so it needs i call it a smart global load balancer in the sense that also has the concept of l checks um and possibly also has the concept of advanced um traffic load balancing policies like when you're doing these things you know really geographically distributed you probably want the clients that are closers to a location to be always redirected there right can you configure that policy or is it just random um we obviously need those those capabilities i haven't seen something in in in the kubernetes area that really addresses there are some attempts but there is nothing in my opinion that is mature that really addresses that problem and i think we need to work on that yeah so um there were a couple of questions in the chat around how the data replication works in cockroach db so i'm going to take that real quick um we um yes the replication happens in real time so cockroach db is split up in these things called ranges each of which is its own raft group and those are synchronously replicated and so what what happens is in say a three x replication uh configuration for a transaction to be uh committed um a full quorum of those replicas have to be updated before the transaction is committed um and then any remaining replicas are updated within what we call our closed timestamp window which is uh i think by default i think we have set to 250 milliseconds um so the data replication happens in real time and then what we do on the read end is we distribute the authority to act to the raft leaders um so the raft leader for any particular segment of a table or database can respond to read requests without having to do a quorum check so that's how we kind of avoid some of the like the the worst the worst side effects of the uh of the cross region latency um so we had a couple other questions we are um uh we are about out of time so i'm going to go ahead and run through the last slide or two and then anyone that wants to stick around and uh get a couple more questions i'm happy to do that so um if you'd like to to learn a little bit more about um how to run cockroach to be on Kubernetes there are a lot of different options uh you can go and and check out our website um i've got a bunch of blog posts on this topic rafael's got some bunch of blog posts on this topic some of them are ones we co-authored together um and there's a lot of other great content on the cockroach labs website around all of this oh press the wrong button and um for those of you don't want to stick around for the q and a uh thank you for joining um so um thank you we do have a couple of other questions um one is what tool do you consider best to manage kubernetes services um i don't think there's a one size fits all answer here um i know a lot of folks um like helm as a package manager but that's a single cluster kind of the solution um i'll tell you that most of our multi cluster deployments outside of our um outside of our dedicated and our deep database and service um offerings are all running basically like static stateful set configs um they provide the most flexibility for when you're trying to deploy a single stateful app across them like a multi cluster type of deployment um have you seen any good like like application abstractions application abstractions rafael um i don't want to name um tools but i'm just saying in terms of approach i would i strongly recommend to go with a github's approach both you know both for day two configuration which is usually okay the cluster is up how do i do i want to configure it and what operators i need to install those things but also for tenants so whoever if you're doing multi tenant cluster whoever is the user of the cluster make them use github's to configure what they need from the cluster done you know so instead of accessing the cluster directly and creating objects there set up a github's workflow for that um there was another question around how do you manage situations where maybe a kubernetes cluster got deleted i think that's also an answer for that right if you just get ops then you can you can uh you can kind of rehydrate your environments pretty quickly um i'd like that as a pretty good general solution to that problem um so one remaining question um is it easy to deploy a cluster application remaining across three azs with it's it's easy to run an application across three azs in a region challenges when you run um active active across two or more regions how does the combination of open shift and and cockroach to be helped with this use case right so um so from an open shift perspective my the more distributed you are the more important is to have a common operating layer uh across all of your sites most of my customers that are trying to do this type solve this type of problem are all in some sort of either a hybrid or multi cloud type of a solution right and so as soon as you've got multiple cloud providers or you've got and like a combination of on-prem and cloud offerings um it's really hard to make an argument not to use open shift for that use case because the um it there is a like an impedance mismatch when an operator a human operator is moving between different kubernetes distributions that you can just avoid by using open shift um as far as um um does it make sense to run cockroach to be on top of open shift then in that scenario generally that's what we would recommend right um are there use cases where you might want to run the database outside of kubernetes sure but um in general kubernetes gives cockroach kubernetes superpowers because if we have a transient pod failure those pods can then get started up on another node and then the database is in a degraded state for a much shorter period of time right um can you make simulate making that work by installing it on on VMs yes you can you can write system descripts that will try to try to restart the database and or restart the node if there's a problem and all that kind of stuff yeah you absolutely can but you end up doing a lot of work to to make that happen um there's some performance trade-offs but we do have a pretty robust cockroach to be in kubernetes performance tuning guide on the website um largely most of the overhead comes from the cni layer and there's always the option of of moving away from that if you need to so um that was the last um um um there that was pretty much the last question so thank you everybody for taking the time to talk with us today um kandace do you want to do you want to close us out yes thank you so much keith and raffael for your time today and thank you everyone for joining us as a reminder this recording will be on the linux foundation's youtube page later today we hope you join us for future webinars have a wonderful day