 Hi everyone, welcome to the data services office hour. And today we have a special session with one of our favorite guests, Annette Cluitt. Good morning Annette, how are you? Good morning, Michelle. Good morning, so okay, so today's gonna be an AMA session and ask me anything session for disaster recovery for OpenShift. And the format's gonna be kind of like a podcast but what demos and I'm gonna ask questions and Annette's gonna answer them. And if anyone has questions, please put them in the chat and I'll do my best to keep my eye on it and we'll get to your questions. So, my first question is the big question which is, what is disaster recovery for OpenShift? Yeah, thanks Michelle. So disaster recovery for OpenShift is not any one product, it's a combination of Red Hat products but the main thing is what we're trying to achieve is the ability for containers to have a disaster recovery plan which in terms of the Kubernetes and such that's been sort of a difficult or it's been a long time since that solution has been being baked and this also is in coordination with the CNCF and we are contributing to the different projects but mainly the deal is that this is application based, it's not cluster based. So we're creating, we have some new custom resources which I'll try to show but that allow on an application basis to fail over to what's called the failover cluster. So, and it's all sort of, well, we'll talk about it more but it's all wired together with a few Red Hat products that essentially do the automation. So it's a completely automated solution. Wow, okay, so let's talk about the Red Hat products which how many are involved and what are our acronyms? Okay, all right, yeah. There's a lot of acronyms. Maybe take notes, everyone take notes, acronyms. Yeah, so everything starts with our age, that means Red Hat. It makes it easier, right? Yeah, so starting with sort of where I come from in Red Hat or have been involved in is more on the sort of the storage solution. So we have the OpenShift Data Foundation and that is a software-defined storage solution based on a very mature upstream project called SAP. I think it's 10, 11 years now as a project and that provides not only the persistent storage because when you failover an application, it's probably obvious but what's really important is that your persistent data follows, right? And that the persistent data is in sync as much as possible, meaning your data loss is as close to zero as possible. So ODF provides VSF the ability to do the replication and that then, so we can do the replication of the storage level, we can create persistence of the storage level but then we need to do all the automation. And that's where Red Hat Advanced Cluster Management gets into the picture. And if you're familiar with what's called ACM, it allows you to schedule applications and create clusters and manage clusters all through one interface. And I can just quickly show that, let me share my screen and if you'll let me do that. And if you'll let me do that. So I have right here, I'm logged in to ACM. And what we see here is I've imported three clusters. One of them is actually the cluster where ACM is running. So it's the OpenShift cluster where ACM is running. And the other two I imported for the purposes of creating disaster recovery so are enabling applications to do disaster recovery that are scheduled on the two clusters on the bottom there. So these clusters, you could log into that on their own. Like here's the first one, this is one of them, what we call managed clusters. Here's the second one. And then this is the one that's actually hosting ACM. So the ACM clusters like this sort of off to the side, you don't run anything else on it, it's just responsible for minutes. Well, actually no, that'll be, I think something we're gonna chat about in a minute, which is I, and let me just stop sharing so we can go back to this. The last part of it that I didn't talk about yet is how do you glue ACM together with ODF? And the way that we do that is, again, an upstream project, it's been going for about two and a half years now. And that this is the project that we're working with the CNCF data protection working group with, it's called Raman. And what Raman does is it provides two operators and they're called in downstream and they're called OpenShift Disaster Recovery cluster operator and hub operator. Upstream obviously to the Raman hub and cluster operator. But these operators provide new custom resources that allow us to basically when we create an application, we can put this new custom resource in the same namespace and it will tell the application and via ACM what, where is its failover cluster? And that's how it's actually stitched together. So these new custom resource, we call them DR placement control, DR policy, that's really the part that is new. And the way that you as a customer would get access to it is by the OpenShift Data Foundation at EventSkip. So even if you already had OpenShift Data Foundation, if you wanted to use these new custom resources to enable application failover via ACM, then you would need the OpenShift Data Foundation at EventSkip. So it's really sort of three different things that have come together. And we're, I think the Raman project has had a lot of contributors, a lot of interests in the CNCF as well. So again, DR for OpenShift is not one product, but it's a combination of solutions. Yeah. And of course, along with that, this is our second release of the solution. So I think we'll get into sort of, how do you deploy it and such? So I'll, I'll always get into that, but yeah. All right, so that's my next question, which is what is the recommended way of deploying what are the different like layout deployment and so on? Yeah, so there's sort of two different ways that we can provide this. And again, the goal is always to get, if you have to failover on an application level, what you wanna do is have as little data loss and as little application downtime, right? I mean, that's the goal of any disaster recovery. And in disaster recovery world, we call that recovery point objective is what is the duration of time that you lost data and then recovery time objective, which is from the time that you're out, say the application is not available to the time you recover. So those are, so you're focused on, did I lose any data and how long did it take me to recover? Okay. So with the solution that we came out with first, we call it regional disaster recovery. And what that means in these terms, we made them up obviously, Red Hat did, but we use it to distinguish the idea of, if you have two clusters, and right now this is when we set up these relationships of what we call the preferred cluster and the failover cluster, it's a two cluster solution. So even if you had a hundred clusters, you'd still divide them into twos right now, okay? So each cluster will have a failover cluster for the applications. If those two clusters are far apart from each other, let's say in the US, one is in, I don't know if it was in AWS, let's say you're in a region in Ohio and your failover clusters in a region and say Oregon. That's about 50, 60 milliseconds round trip. That's way too much latency to be able to support synchronous replication of your data, okay? So what the solution there is what we call asynchronous replication and via the SEF, it's called SEF mirroring, we can set up scheduling on an interval. So every say five minutes, the volume, the Delta data that has been written into a volume from an application will get replicated to the failover cluster to a persistent image or a persistent volume sitting there waiting for you to failover. So none of the other application resources are actually on the failover cluster. They're not taking any CPU or M because they don't exist there yet. The only thing that exists is the Delta data every five minutes being sent to the alternate cluster waiting for the application to failover. Because it's asynchronous and let's say it is five minutes there is potential for you to lose up to five minutes of data because think about it, if I replicated T zero five minutes later, I have a failure. I don't have any data except what I got five minutes ago. Now you know, probability is you know, you'll fail over two or three minutes or something. So but the maximum you could lose is five minutes. So that's an asynchronous solution. It's per volume. You can also set different replication schedules for different applications. So for some reason you might want an application on a five minute replication but another application on a 15 minute replication. So you can do that. So that's one deployment strategy in that case you would install and I'll just, I can just, I think we have time, right? Yeah, absolutely. So I'll just show, just share one more thing here. So if it was, of course, these are going to be having me log in before. So I'm going to log into one of the managed clusters with my little cheat sheet here because I can't actually remember this password. So when we do this, what we call regional DR, ODF it'll take a little bit to render these servers are in D actually. So we install on the managed clusters, we install OpenShift Data Foundation and I'm just going into installed operators here and we create the storage cluster. So in the case of regional DR, we are creating the ODF, the OpenShift Data Foundation, the stuff cluster. We're creating it what we call in an internal mode, which means it's hosted on OpenShift. And if I was to log in over here to this second managed cluster, we would see exactly the same thing. So that when we talk about the next deployment method, that'll be sort of one of the major distinctions is, do I have two stuff clusters on, meaning one on each different managed cluster or do I have a single storage cluster? Okay. Okay, so and again, if I go in here to the managed cluster, the hub cluster, which is the third cluster, does not have a storage cluster installed on it because it is not actually hosting or the storage or the applications. It's just, it has ACM and it has the OpenShift DR Hub Operator and all that. And just an important distinction about that, the OpenShift disaster recovery hub operator needs to be, and we'll talk about in best practice, but it needs to be able to not be failed at the same time because it's going to orchestrate ACM and the OpenShift DR Hub Operator are going to orchestrate all the failover, right? So if this cluster went completely away, one of the managed clusters that we would be able to orchestrate the failover of the applications because the third cluster is still working. So let me stop sharing again. And the other way of deploying is what we call MetroDR. And with MetroDR, it sort of implied in the name that it's a metropolitan area. I mean, this is sort of old term these days, but we used to have metropolitan area networks, which meant that they were in a metro area, but the deal about it is the latency, right? So the latency between the two peer clusters that you're going to set up preferred and failover cluster needs to be, and this is really a gauge. We say below 10 milliseconds around trip delay. If you had some applications that were latency sensitive, you would reduce that in my experience because we're going to be doing synchronous replication. In my experience, synchronous replication, you might need to go to five milliseconds, or in terms of distance, that's probably somewhere around 100 to 500 miles between the two sites. So the synchronous replication has a huge advantage, which is you don't lose any data. So the RPO, the recovery point objective equals zero. You still have a recovery time objective because it's going to use the same way of failing over as original DR. So the application, if you lose the application or you decide to take the application down and failover, that takes time to recreate the application. But in terms of data loss, because it's synchronous, you haven't lost anything. So in order to facilitate that synchronous data loss, or zero data loss because of synchronous, you have to take the storage cluster and move it outside of OpenShift. Now this is called Red Hat Ceph Storage, RHCS, and this is a product that Red Hat definitely has available. It's currently, the way that you would get access to it is again, via the OpenShift Data Foundation Advanced SQ. And when you move the storage cluster outside of OpenShift, you essentially stretch it between the two sites. So you have some of the OpenShift, or excuse me, some of the Ceph storage nodes in one site and some in the other. And it's using a Ceph mode called Stretch Mode, which with an arbiter that will allow you to lose basically the storage nodes in one site and still be able to have complete read-write access storage to the volumes. The arbiter node, similar to the ACM node, the arbiter is called a monitor. And the monitor has to be in quorum with the other monitors, it's friend monitors. There needs to be three actually in quorum. So what you wanna do is take this monitor, maybe put it at a colo or put it even in a cloud environment where it will survive. And it can be a lot of latency for the monitor. It could be 100 milliseconds, which is basically within the realm of anywhere, say in the US or almost anywhere in the world, you'd be able to move it far enough away. So it is not latency dependent, the arbiter. But it has to survive in the case of an outage. So that's why you put it somewhere else. Okay, so in my mind, I'm imagining if I were working at a large enterprise, I might do something where my Metro DR consists of maybe a production cluster with a development cluster that if my latency requirements for data, if my tolerance for data loss is extremely low, I might do something within the city or even if I need to within the same data center, just for immediate failover, but I can still use that other for other things. And then for true disaster, like massive power outage on the East Coast, we're looking to flip it over to something much further away and I have to be willing to tolerate data loss. Exactly. And actually you speak of something, the company that I worked with prior to this had a lot of data centers and COLA's in the New York, New Jersey area. And if anybody was around during Sandy, everybody learned a lot during that day of time, or days. Like we have too much concentration in one place. Okay, so all right, so I'm gonna go to the next question because I think we, I know we're gonna get into more detail about all of this, but we may have answered this already, but how many OpenShift clusters are recommended for the DR solution? Yeah, the recommended, and I showed it is three. If you're really serious about your recovery time objective, you need to have that third cluster and the advanced cluster management development team, they're working and releasing every release, they're releasing more capability to have hub backup. I think with the new ACM version 2.5, you can backup the hub cluster. So essentially you can, or the components that you need to have on a cluster to do ACM so that you can literally recover the hub. But, and there are some strategies to do only two clusters and host advanced cluster management and the OpenShift DR hub custom resources on one of the managed clusters. But if you do that, your recovery time objective is definitely gonna suffer. So, yeah, so those solutions are coming along, but today, given, you know, we're just a couple of releases into this, our recommendation is to have the ACM, and the OpenShift DR hub operators on different cluster than the managed clusters. Okay. And the managed clusters again are gonna, yeah, the managed clusters are going to be, that's where your storage is, whether it be external or internal to OpenShift and that's where your applications are deployed. Okay, okay. All right, so next question, slightly different question. So active, active versus active passive for application access. Yeah, that comes up a lot. And these terms, you know, come from legacy disaster recovery solutions and plans. I mean, it's not like we made up disaster recovery for containers. So active, active, the sort of the hope of active, active is that I could be able to have connections inbound to whatever application I have on two different clusters, one being sort of my preferred cluster or cluster one and the cluster two. And then if I fail over all my connections inbound from whatever, you know, they all switch to the cluster that's working and nobody knows anything and it's just great. That there's some difficulties and have been for a long time with that about what about data, right? I mean, data has a hard time residing in two places at the same time and being consistent. So the way that this solution works is on a per application level. An application, an open shift application that's been deployed can only be active on one cluster at a time. Okay. So each application where you put the DR placement control will have a failover cluster. That failover cluster could be different. So like an open shift one, I could have an application and it's failover clusters, open shift two. I could have another application on open shift two whereas failover cluster is open shift one. So what I would say is from a cluster point of view it's active, active because we're using resources on both clusters for the active application but from an application level, it's not really active passive because we don't have any on the alternate cluster or the failover cluster, we don't have anything running. It's not running until you need it. So it's not really passive, it's sort of waiting. Or it doesn't actually exist yet, right? It doesn't exist, like I said. The only thing that exists in either the metro or the regional is the ability to use the volume. The volume is being updated with either Delta data or it's actually synchronous. Meaning it's totally consistent. Okay, but from a cluster point of view both clusters can be totally different. Okay, all right. Okay, so next question. Replication from one cluster to end clusters where end is greater than one supported. Yeah, we talked about that and I'll just make it clear for the current release each cluster pair and you could call them cluster peers there can only be two. So you have your failover, you have your preferred cluster and your failover cluster. And we call that in the custom resource DR cluster set or DR cluster pair. So you could have a hundred clusters which you wanna set up disaster recovery for but you're gonna have to divide them into sets of two. So you would end up with 50 sort of DR cluster pairs and those DR cluster pairs could, some of them could be doing Metro DR based on their latency and some of them could be doing regional DR. So that's not required that they all be the same but right now it's not one to many. That is a future possibility but it's not there right now. Okay, gotcha, all right. Okay, so next question. Which volume modes are supported? RWX, RVO? Good question. So it's gonna be a different answer for our two different sort of disaster recovery configurations. For regional disaster recovery we're using today, we're using the self-meering or RBD mirroring and RBD is a RWO solution. So read write once is what we support today for the regional disaster recovery. So if you create a volume using your ODF storage classes, the only volumes we're gonna be able to replicate today are the ones that are RWO or CFRBD. Okay. So the CFFS volumes will not be supported by this with our current release today, which is version 4.10 for OpenShift Data Foundation. For MetroDR, because we're stretching the storage cluster across and it's really just Red Hat self-storage and it's just one cluster, all of the storage types, RWO, RWX, all of those are supported and there's no constraints on that because it's just one self-cluster and it's in a synchronous mode. Okay, that makes sense. Okay, so that's an additional advantage of doing it in a Metro way, the disadvantage being you're close enough. Yeah, so actually I'll wait till we get to the question about Valsync, but I do have an RWX solution coming. Oh, okay, nice, so we get to see something. All right, so next question and I remember this question. Okay, so is OADP, that's Valero, used in the DR for OpenShift solution? Yeah, I wanted to address that because it gets to be alphabet superior. So OADP is OpenShift APIs for Data Protection. See, you can't even remember. There's so many, excuse me. Yeah, for Data Protection, let's continue here. So that's OADP, OpenShift APIs for Data Protection. What that really has been and still is today is exposing the Valero restore backup and other APIs into OpenShift so that you can use them easily. It's a certified Red Hat operator and is nice if you're going to do backup into same cluster, meaning currently it doesn't move anything off of OpenShift so it's not like a full meal deal backup and restore. But it's nice and it will have a data mover similar to other backup and restore solutions coming along, I don't know, in a couple of months. But it has currently no intersection with DR for OpenShift solution. There's no intersection today. We are looking at possibly because if you think about backup and restore and disaster recovery, they're not the same thing but they're complimentary. So if I'm doing replication and I replicate something that's corrupted, then it's still corrupted even if I replicate it, right? So there's still a very good reason to do backup and be able to recover to a known good state. So eventually we do see OADP merging in as that complimentary solution so that you really not only do you have a disaster recovery alternate cluster but you also have the ability to recover from corruption or maybe it's not, who knows, but you have snapshots and these are called user snapshots that you can recover back to, which is what OADP does. Okay, and if a while ago, I think I did a show with using OADP for migration and there are no time constraints on it. It's not about time. It takes as long as it takes. Like it's not concerned with like, oh, I have to have this synchronized with them. Yeah, the only thing with OADP, the word timing would be is if you had a scheduled backup and you couldn't complete the backup, just like replication. If you can't complete the replication in the time of your schedule, then you would wanna look into that. And you have problems. Okay, all right, so one more question and then maybe we can start look at a few things. Is Valsync used in the DR for OpenShift solution and you're gonna have to explain to me what Valsync is. Okay, yeah, I'm addressing that because that also comes. So Valsync was and is an upstream project. I think it's still called Scribe. It's again about two and a half years or so and one of our Red Haters, John Strunk is basically the parent of this project. So what it does is it takes the OpenShift Data Foundation and the Container Storage Interface capability of Snapshot and Clone and it creates, it is using that to create a replication capability. So the custom resources are essentially called replication source and replication destination. And you set up using the Snapshot and Clone capability. You're basically doing similar to self-marrying, you're transferring the Delta data to the alternate cluster, which is defined in your replication source and replication destination. So it is another replication method that is based on, it's asynchronous and it's based on the interval that you set, say five minutes. Okay, can we just go ahead and say that? It supports every kind of storage, even outside of SAF or ODF that is using a CSI compliant Snapshot and Clone capability, which is Container Storage Interface. So it would support EBS GP2 that's using the CSI driver. It would support really any, it's meant to be sort of storage diagnostic as long as you have implemented the Snapshot and Clone capability and you're with your CSI driver for your storage. So what we're going to be using in version for ODF 4.11, which will drag along the OpenShift DR operators is we are going to be using that under the covers, this will all be automated that if it is not a SAF RVD volume and it's a SAF FS volume, Volsync will do the replication. So when 4.11 is released, it'll probably come out as probably what we call development preview. I'm not sure it'll be tech preview, but it's a nice solution because again, it's going to be totally automated via the ramen or the OpenShift DR cluster and hub operator and ACM. So all of that, it'll behave from a user point of view exactly the same. You won't even know you'll just, if you create volumes with RWX and you put a DR placement control in that namespace, then it will get replicated and you'll be able to set the interval just like you do for the block volumes. So it sounds like Volsync gives you flexibility on the other side, right? Like let's say your primary cluster looks one way because it's, I don't know, an AWS and your other cluster maybe is on the, another continent in the data center and you've got different storage there. You've got different, so it gives you like lots of flexibility around. It gives you a lot of flexibility. Yeah, I think our first release of it, we made limit that to just what ODF provides, which is CFFS, the file volume. But yeah, in terms of the upstream capability and this actually Volsync is already integrated with ACM. It came out in ACM version 2.4. So one release behind, we're now at 2.5. So it's already integrated and it's already in the ACM documentation as a replication method. But what we're doing now is we're taking and integrating it in with all this automation so that again, the user doesn't have to create the replication source or the replication destination. All that is automated. Okay, all right. So that's it for the planned questions and I don't see anything in chat. So can we look at something? Can we? Yeah, actually the... Only if it's... Yeah, we have looked at some things currently. I was hoping to show sort of the next version, but I got a little bit held up on that. So... Nope. Okay, so... Yeah. All right, so we can end early if you like if since there are no questions at this time, I think we've covered a lot actually. And I'll be sure to add some chapter markers with these questions so that people can go right to the question and see the answer right there. So any additional repos or anything you would like me to include, I'll be ensured to include those. Yeah, actually, you know, can I put a link somewhere? Yeah, yeah. If you can give me the link, I'll drop it in. So... Okay. Okay, well thank you so much. Yeah, cause I... And then also if you want, we can do a future show when you can give us a preview, we can just do another show, showing the preview. Yeah, what I was gonna do is just give you, let me just make sure this thing doesn't start, give you a link, I don't know if you can, can I put it in this interface or? That's a good question. Well, what I'll do, anyway, this link should go out as well, which is it shows how the, how you fail over and fail back on an application level. And it's, it is going to be exactly the way it works now. And it's also gonna be how it works for our new version. So it'll, yeah, so it'll show people how that works. All right, so I'll make sure it's in the description. Okay. People can try it. Awesome. All right, so thank you. Thank you, Michelle. All right. Thank you so much. Thanks, bye.