 storage as an intermediary. But and then we'll help you move a few buckets of things. So you can kind of think of your migrations in three buckets. You've got your Kubernetes objects. Those are pretty lightweight. You can usually stream them out as JSON objects, serialize, deserialize them. And then secondly, you've got your application state, which a lot of people consider to be probably the most important part of their workload. And then thirdly, you've got your images. So we'll also move your images as well. And we have some different modes that will allow you to say, disable PV migrations if you want to handle that yourself, move your own data. You can also disable images. We've had some users actually write their own tools using Scopio to migrate their own images as well. But we will also do all three and you can kind of tweak those. So what we're looking at here is, I set up two clusters before we met. So we've got an OpenShift three cluster that have got some example workloads on. There's just some mini like Nginx examples with a very small PV as sort of a toy example. And then we've got an OpenShift four, six, I wanna say. Yeah, four, six, 21. That's what we're looking at here. So this is the OpenShift four console. And then in another tab, Michelle has up is the MTC. You'll hear me say MTC. That stands for Migration Toolkit for Containers. That's the product that we're looking at that handles this. Can I just pause you for a second? How does this look to you guys? You needed smaller, larger, like columns are nice. What do you wanna, just let me know now before someone says anything. I think it looks good. If audience, if you can't see it, let us know please and we'll move on. Okay, sorry, Eric. Go ahead. That's about where I was wrapping up. So that we're looking at the Migration Toolkit for Containers console. So that's gonna be, it lives at a different route. You can see the, you can sort of see the addresses separate from the console. We're looking at integrating that with at a deeper level into the console. But yeah, so this is the MTC GUI. That you use in order to execute migrations. Okay, so Chris, have you ever used MTC at all? Do you know? I have not. I have not used the migration. Okay, so Eric, if you could explain to Chris the way you explained to me, like how the different parts. Sure, oh yeah, absolutely. Yeah, so I guess the way that I generally think about the, of migrations, like, you know, we sat down when we initially started this and just sort of thought about it generically and just like abstractly, what are the concepts that you need in order to actually move your workloads? And so obviously you need your clusters. So you're going to have at a minimum two clusters, your source and your target destination. I should also say that there is nothing about MTC that we do have some sensitivity to the underlying versions of the clusters that come down to the cube versions. So we can talk about that if we want to get into that. Of course you're going to have like group version kind differences and API differences and drifts between like a an open shift like 311 cluster, which is I think cube of 111, up to like a more recent open shift four cluster is going to be APIs that are present or different. Maybe you've got to be one beta one API, V1 API and your target. But generally MTC is mostly indifferent aside from those slight edge cases, I guess. And there's some stuff that's built into it in order to try to handle that or at least warn you if that's the case. So when you initially fire up MTC and you navigate to this GUI, it will automatically register the host cluster. So it's aware of the cluster that it lives on. Usually people install MTC in their four cluster. So what makes up MTC is MTC at its core is an operator. I'm sure folks have heard that before. And the operator is responsible for installing a number of components at a high level. Those components are the UI, the controller. So we have a controller that is kind of the brains of the operation. And then the controller is going to be reconciling a data model that's constructed out of CRDs. So those of you who are familiar with CRDs and Kubernetes will feel right at home. We designed the whole thing to be API like CRD driven first. So even the UI that you're seeing here is really just another client that uses the vanilla Kubernetes API. So it's talking to the cube API surfaces directly. So this is the tooling formally known as CAM, right? Yes, so yeah, I think that transition happened, let's say six to eight months ago, I wanna say. So it was rebranded. It formally was known as, I think it was container application migration. Migration, yeah. Yeah, so someone in the audience is asking, that's why I'm asking. Yes, yep. So this is CAM I think was like a one one and a one two. And when it became MTC, that really was a, it was released alongside a number of features, but it wasn't like a one to two version break in that it was like a major feature change. It was really mostly branding. So that we could, it sort of fit into this umbrella effort that we've got going on, which is conveyor, which I should talk about a little bit. Which is the, is Red Hat's kind of cloud migration federation of projects, if you will. So that's our upstream organization. And so there are other projects there. We've got a migration toolkit for virtualization that folks may be interested in to pull VMs into cube vert. So that'll help you do that. This one's really focused on application migration. So that was the reason for that rebrand. Okay. All right. So hang on. So you, it's installed, the MTC is installed here on the forecluster. It's also installed on the three cluster. Yes. Right. And then, so sorry, go ahead. I started to talk about what the operator manages. Those main components are, you've got your controller. It installs the API surface on your fore, on your forecluster, which is a set of CRDs. And then below that, we are built on top of a project called Valero, which is, I think it came out of Heptio originally. Yeah. And so this is a project that's mainly focused on backup and recovery, which is it's nuanced in terms of its differences with migrations. We actually orchestrate a migration using backups and recoveries. And you'll see that's why actually we require an intermediary repository. That's how Valero operates. So we're orchestrating, we're doing like a backup. And then we kind of snapshot some things, take stuff down, do another restore on your target side. So it does sort of a dance between backups and recoveries in order to migrate everything over to the target side. So I say that because on the OpenShift3 side, it's almost like an agent. In OpenShift3, there is no OOM. So we do install ourselves as an operator, but we are not installed and managed via OOM, which is the operator lifecycle manager. So that's a manual manifest creation. You'll do like a cube cut all create on it on a set of manifests. And that, there's only a single controller. So that lives inside of your four side. And then on your three side, we do require Valero, because Valero is actually what's getting everything out of your cluster. So that's kind of, that's what gets installed on your source. So. Okay, all right, cool. So, all right, so do you want to go here and talk about the MTC console and how you planned it out? Or we can, okay. So I started on this path a little bit. I'm in this, I'm in the trenches here every day. So pull me back if I'm getting a little too deep. We got you. So when you think about a migration, you obviously need your clusters. So when you install MTC initially on your four side, you'll get the host cluster. You see that in the first row here. And then separately, I pre-registered our OpenShift3 side with this cluster. So you do that by creating a MIG cluster CR. And if you open add cluster there, that's actually what it's doing. So this is how you register a cluster. Primarily you give it a name just to identify it. And then you have your URL, which is going to be the coordinates to the API server. This is a Kubernetes API server. And then secondly, when you install our source side, it will generate a token that's privileged such that you'll be able to pull and access everything out of the OpenShift3. Can I ask you a question? Can you have more than one source all going to the same target? Yes, yeah. So there's no limit. And also when you register a cluster, it's not designated as a source and a target until you actually create a plan. So that means you could have a OpenShift4 cluster as it could serve as your host and your target. And actually we had a user recently contact us. They had done a migration to their forecluster. And then they went through an X, like I think accidentally deleted their images or purposely deleted their images on the source side. And at a later point in time, they said, oh, we actually need those. So they just changed the designations and then brought the images back over from the target side. Nice. They worked in both directions. Yeah, that's cool. Okay, nice. Okay, so this is how you add a cluster. Okay, you want to go into replication repositories or just remember you said that we need an intermediary place to put all of this stuff, right? Yeah, so that's what your replication repository is gonna be. This is generally an S3 bucket. So if you hit that, you can choose, we support a bunch of different kinds. There's AWS S3, which is distinct. Like they both, Valero under the covers expects an S3 API. So AWS is distinct from S3 here and that S3 could be a self-hosted S3. So I'm sure folks are familiar with Nuba. Nuba is something that some of our disconnected customers, they can set up their S3 on-premises so that they don't require some cloud provider like AWS. And then we also support GCP in Azure. So I think Azure Blob Storage and GCP object. There's also a lot of vendors out there that are now supporting that S3 API by default, like the normal move, copy, remove kind of operations. So yeah, that's awesome that you have AWS and the generic S3. That's cool. So you set this up? So that's, I'm actually using just a vanilla AWS S3 container here. Okay, migration plan. Are you ready? Any questions, Chris? Is there any other questions? No questions yet, but please audience, if you have any questions about what we're talking about, feel free to ask. No question is dumb. Trust me, I am learning this stuff as you are. So I will have stupid questions at some point too. So are we gonna add a migration plan? You wanna talk about EngineX, what would you like to do? Yeah, so I created an initial one as part of my testing to make sure that everything was not gonna blow up on us. Thank you. We can also see actually what it's gonna look like when it's finished here. So we can either go into that and kind of explore what that looks like and what we expect to see when we actually do a migration or we can go ahead and just create the new plan. I should also talk about what a plan is in general. Okay, so I'm happy, let's look at this first so people know what to expect and like the terminology with migration and stage and all that other stuff, because we talked about the Fortnite questions. Sure, so let's talk about a migration plan. So when you create, the first thing you do, you set up your source and cluster target and then you've got your replication repository. Now you're ready to go. Now you're actually ready to migrate stuff. So when you go into the migration plans, the plan is where you actually tell us what you actually wanna migrate and how. So when you create the plan, that doesn't mean you're executing it, it's just a plan for you to execute at some point in the future and we have a few different ways of for you to actually execute that. So we see actually two migrations here that I ran previously. We see a stage and a migrate. So the purpose of, go ahead. Shaggy, you want me to go into one? Do you wanna see stage first? Okay, go ahead. Yeah, I'll just talk about the difference between the two and why we have a distinction. So this is all in an attempt to reduce your downtime. So when you're doing a migration, we're trying to get all of the data off of your source cluster as possible, prior to you actually doing that final cutover when you actually need to take things down and then bring them up. So generally, if you think about it, you got a database, you've got a ton of data in there. Let's call it a terabyte of data. There's probably only a fraction of that that's gonna change between your migration window, right? So we'll get 98% of that moved over ahead of time. You don't need to take down your application. And then that final migration is gonna capture that final delta so that while your, and the difference between the two is that during stage, your application is still up while during a migration, a final migration, your application is down. So we've reduced the downtime to be just as much time as we need in order to capture that final delta between your last stage and your final migration. So that's a distinction here. You and I, we were talking about this before. So you could potentially, you could, you would do a stage Thursday, Friday, and then Saturday morning, do your migration. You could decide, let's say two weeks go by because you haven't, you did your stage phase, but for some reason you didn't do the migration, you could decide you wanna redo your stage, you can decide, nope, I just wanna do the migration. There's all kinds of like combinations you can do with the idea of you've taken your first cut of this and the migration you said is, it's a repeat of the stage, but it recognizes all the stuff that's already been moved over and picks up only the delta, is that correct? Yeah, there's some details in there that get kinda complex that I probably would need my guys in order to really describe in depth, but it goes through like sort of a few stages and then it goes through like, it's trying to capture everything in the original state that it was in and then it brings stuff down. But the crux of it is it goes through, the general pattern is like, stage your data upfront to get ready for it and then you can do that as many times as you want. So like you said, maybe I did it two weeks ago and I got a bunch of it. I'm gonna do it like on Friday before the Saturday when I wanna do my cut over and then Saturday comes, I do my final migration. I'm just gonna get the little bits that maybe changed on Saturday. Hopefully it's not very much and I'm cut over and that will actually queuesce your application on the source side. So I just wanna show people inside one of these phases, like the stage phase, you can actually go and see what was done, which is kind of nice, right? If you remember we're doing this application at a time, right? So you can go in and see which is there, yeah, and then it was done. That's all, I thought that was nice. So a question came in from Keith Lawrence. So stage equals copy and migration equals move. I think stage is just what you labeled this, correct? Yeah, stage is like abstractly the concept of trying to get, stage your data on the target side so that you can do your final cut over. I'll get into the difference between copy and move because there is a distinction. There's different ways that you can get your state over to the target side and there are different reasons why you would wanna use one over the other and we have varying levels of sort of, yeah, depending on your interests and what you care about. So I can get into those details in a little bit. Awesome. So just to have a quick look at migration, right? So stage really just had a few, are any of these of interest you want me to go into? Do you wanna talk about, restore these details? So this should look like- It's probably more interesting like this, if we watch a migration happen, you'll see like these, all the pipeline kind of go walk through. It seems like some folks are actually familiar with CAM. You'll recognize this looks totally different than it used to previously. We really didn't have a lot of visibility into progress, which is frustrating when you're trying to do a migration. Part of that just wasn't available at the Valero layer. We're built on top of Valeros, so we're sort of limited by what's available below that. We've added a bunch of other really cool features like now we support direct migrations. So we'll go from, previously we had to use this intermediary location because we were built on Valero, which is a backup recovery tool. So if you had one gig of data, you wanted to get to your target side, you had to put it one gig up and then bring one gig down. Now we'll actually go directly to your target. So in theory, that should be twice as fast. And we also control that layer more. So we are under the covers actually really using R-Sync. And so we'll R-Sync the data directly into your target cluster. We also support that for images. And what that also allowed us to do is to build this like much more improved progress reporting experience. So now you actually get granular progress for how many Kubernetes objects have I moved over so far? How much data, like how much data am I moving? What percentage of it am I at currently? So we've gotten a lot more access to that data and we're surfacing that. So that's an improvement from a MTC compared to CAM previously. Want to do, can we do the out of migration plan? Yes, let's do it. Let's do it. Can't see this, all right. My plan. All right, source cluster, source cluster, close. Yeah, so here I said like clusters don't have designations when you originally register them. This is where you actually give them like meaning. Is this a source cluster or a target cluster? And it only has meaning within the context of a plan because I could go out and create another plan where my host is my source cluster. Thanks. Okay, and this is the repository. Okay, next. Okay, so we have a controller that is called a discovery service. And basically that's inspecting your source cluster that you told us about on the prior step of this wizard. So here we're discovering the namespaces that are available to you in order to choose for your migration. We do some filtering, like we don't want like system namespaces sort of stuff. So in this cluster, I've got four Nginx example namespaces. So I think I mentioned this before when you're migrating the unit of like migration for MTC is at the namespace level. You can choose more than one. So it operates as a set of namespaces. And that's what you're seeing here. So you can select your namespaces that you want to move over. Okay, so these should be like if I, the same one, like basically I should see them here. Okay, just checking. All right. Any, which one would you like to do? I mean, they all seem to have the same PV claims. You want to do- Yeah, they're about the same. If you look at the pods, so you can see here, I actually already ran the Nginx example. So that's why the pod is at zero. There's no pod, there's actually deployment there that's been queuesced to zero. So why don't we go ahead and choose two or three? They're actually the same. Which one do we, I can't remember. Let's do three. I thought the one that we have a plan for already is on two. I could be wrong, but it's definitely not three. So let's do this. Okay. Okay. So now that you've chosen it, that gets written out to the CR and our controller is inspecting. It's doing some, it's like sort of discovering what PV's and that you actually have in use. Prior versions of CAM, I think actually required a PV to be bound to a running pod. So we actually checked to make sure it was in operation, like actually in use. We had some RFEs from folks who were interested in, they said like, hey, I've got a bunch of PVCs that are just kind of around. You can think of like a functional, what's the name of it? Sort of like a lambda where it only comes up when it needs to be. Yeah, serverless, thank you. So maybe they've got data sitting around, but they don't actually, they're not actively using it in terms of having a running pod that has it mounted. So we started to expand that so that it'll actually discover everything that's in here. So it doesn't require them to be mounted. But in this case, we had one PVC that's a gig in size. And we should also point out this is on Gluster. So on my three side, I've got OCS three and on my four side, I've got OCS four. So three side is running Gluster, four side is running stuff. Okay. Is this a good time to talk about, you mentioned to me the one case where the migration got held up because the PVC was to a PV that no longer had anything attached to it. And like the idea that when you do this migration, you actually clean up a lot because all of these dangling PVCs out there and PVs, like, can you just talk about that experience with that particular customer? I thought that was an interesting use case. Yeah, sure. So the fun part about migrations, especially at when you are focused on evacuating entire clusters, that can be on the order of 100,000 namespaces or something like that. You find a lot of problems. So as it turns out, MTC is sort of a decent health check. And in a lot of cases, the people who are using this tool don't really know what's inside of their namespaces. They're usually like cluster administrators. So they're not the app owners themselves. So in this case, this is some speculation, but what we think happened was that, let's say there was an application owner who had an application with a pod and it had a PV mounted, and behind it was Amazon EBS. So what we found was that the, so, well, they came to us and said, there's an error when we're trying to migrate this volume, we're not sure why. So we started to dig into it and we realized, oh, this volume doesn't exist in Amazon. So, but as far as Kubernetes was concerned, the PV was there. So at the Kubernetes layer, the PV was there, the PVC was present, but the volume itself, like the actual data had been deleted in Amazon. In addition to that, we started to look at the pod and it didn't have any mounts. So it didn't actually mount this PV either. So what that led us to believe was that at some point they decided they didn't care about it anymore and pulled it out of the pod. And then they went into the Amazon console and manually deleted it. So, CUBE had no idea any of this had happened. And MTC operates at a CUBE layer. So MTC is doing this discovery and it found, hey, I found a PV, so let's try and migrate it. So when we did that at our copy layer, we bring up a pod and we try to mount the volume. And so, of course, when we went to mount the volume, it didn't exist and so that there was an error. So that's an interesting case. And you'll see stuff like that often. Typically the types of problems that we see often, especially with the current version of MTC, it's usually environmental. It usually comes down to things like that. Maybe somebody goes into their Gluster console and they've expanded their PV size past what the PVC, sorry, they've expanded their volume at the Gluster layer past what Kubernetes thinks it's at. So Kubernetes maybe the claims one gig and the volume's actually two gigs and they filled it up with two gigs. So when we go to actually request a volume on the target side, we request one gig because that's what we found on the source side. And we try to stuff two gigs out of the volume into the one gig volume on the target side. So we actually have some tooling in place in order to address that. So we'll prompt you and say, hey, we actually found way more data here than you've claimed. Do you want to make a bigger volume on the target side? Nifty. Well done. That's good, good stuff. Okay, so I'm gonna hit next, let's, okay. All right, so copy method. Yeah. So this is getting into the, actually it might have been on the previous screen. I can talk about the copy versus move, if that makes sense. Sure. Do you wanna look at this? Yeah, it's just gonna rediscover it. Okay. So here's when you designate the way that you wanna actually get your data to the target side and to say the way that you wanna get your, I'm being careful not to say copy because copy is not a move. Right. So move, you can kind of think of, move is relevant to shared storage. So like NFS, for example, it's similar to like unplugging the disk and plugging it into the target cluster. So that's really gonna be preferable when you've got a lot of data and it's shared. So it's really only an option for certain types of storage that are shared. We actually just confirmed this morning, Cinderworks. NFS is a classic example. I think there's tons of people who are using NFS at the last time I looked at it. But that will take the actual underlying NFS volume and we'll recreate the PV definition on the target side so that it will remount that data. So you're not copying terabytes of data, it's just remounting it on the target. So that's gonna be really preferable if you care about speed a lot. Of course, one of the drawbacks is you aren't cloning your data. So if you have an app on the target side and it starts writing to that data, that's your primary data. So that's not a clone, you can't roll that back in the same way that you can a copy. Okay, that makes total sense. All right, so any questions about copy? Anything, let me know. Not yet. All right, I'm going to next. Okay, so since we chose copy. So we have a couple of different types. Now we get into the types of copy that you can do. You've got file system copy that you can literally think of like an R-Sync and in fact, the newer versions literally are using R-Sync. Prior versions used a project called R-Stych. That R-Stych, I believe it ships with Valero but that's generally what will be used when you're backing up state with Valero. So we do that separately. That just came in after, like right as you said, any questions have dropped. So if the source is cluster and this target is Ceph, RDB, block volume, if the application requires RWS shared, how does that work when source is R-W-O? Yeah, there are some cases where I think, so some of the, I don't know the exact details of some of the access mode. Like there's a lot of logic that's in place in order to handle like sort of these edge cases. The, we do warn in the cases that we think maybe of concern. I would have to get some of the other folks who are in this day-to-day to give some more detailed answers to that. If also, if there are some further questions to it that I can't answer, well, we'll give out. R-W-X. R-W-X means, yeah. Yeah, yeah, R-W-X. I'll give out the mailing list and Slack addresses so folks can get some more details. Also, we have a Discord for the channel. So if you wanna jump in there and ask your questions, I can get them answered as necessary, but yeah, thank you, Keith. R-W-X. All right, question. Are edge cases discussed for the documentation? Sorry, sorry, Chris. Like, is there, do you have a place in the documentation where those kinds of cases access modes are discussed or? Yeah. Okay. Yeah, the private docs. I'm gonna grab a link to the doc real quick. Yeah, I'll just do that. All right, cool. All right. Okay, so what am I doing here? Am I doing file system copy? You wanna do a volume snapshot? Yeah, let's do file system copy. I'll just mention volume snapshots too. We do support volume snapshots with like, with providers that support it. So AWS, like EBS, it'll do a snapshot. It can recover from the snapshot on the target side. Similar with GCP and Azure as well. So here we'll do a file system copy though. And I've got a bunch of storage classes here on my target side, but we'll just choose RBD. Nice. Okay. Yes, we already have RBD selected. Okay, so we're good. Next. Anything we need to do here? No, but I should talk about this. These are the new features that have been added to do this direct migration. So it's a lot faster. So it skips the intermediary storage. The caveat is your source needs to see the target cluster and that's not always true for a lot of people. Okay. All right, so hang on. All right, so are we ready? Yep. Okay. Are we doing this? Any hooks? Yeah, we need to hook. Do you wanna talk about them? Yeah, what do hooks do? Sure. So hooks are a way for you to, like this is a generic migration tool and like one of the, a lot of folks have like, they'll want to hook into the process of the migration. The general, migrations are a state machine. So this hooks allow you to hook into certain parts of the life cycle of your migration to do custom logic. Here you can see there's an opportunity to, we support Ansible natively. So you can add an Ansible playbook here and we'll actually go build it, drop it into a container and run your Ansible. You'll have access to your environment. So you can do your work, use the Kubernetes modules, do whatever you'd like to do. You can talk to Boto, you can talk to your DNS layer. So if you need to reroute DNS to your target side, that's one of the like classic examples of a hook, right? Is that DNS, we made a decision. Everybody's got a different opinion about DNS. So instead of us trying to solve this, why don't we delegate that out to you so that you can handle your specifics. We've seen customers who have like a lot of people at like an enterprise environment, they're gonna have to tackle stuff like SSO, they have to deal with certs, they have to deal with internal routing, internal DNS. It's really complicated. So the best way to do that is, hey, let's delegate out to you to handle that. So you can either handle that with some Ansible playbooks or you can even actually just specify a container that will just go run with some input in order so you can do whatever you'd like. You can bake a go binary into that and we'll run that. That's awesome. And did you also say the hooks can be placed anywhere along the migration plan? Are you only at the end? Is there any value and do you get to choose? Like, oh, you're coming out of the source, but before you hit the repository, do this hook. And then again, do some other hooks so you get control. Okay, all right. You can designate when you want it to run and where you want it to run. So it can run on the source and the target cluster. And in addition to that, here on the left-hand side, you can see the hooks left nav option. So we recently made this like a global concept. So previously you had to actually craft it and bind it to a plan. Now they're generically created and they're independent of plans. And so why that's cool is that we're hoping that or you could foresee, let's say there's a shared GitHub of generically useful hooks. So maybe every WordPress app needs a hook that's gonna go in and correct some configuration when it gets migrated to the target side. So people can go in and kind of pull this stuff off the shelf, add it to their global hooks and then reapply it to each of their plans so that they're reusable. Okay, so that's awesome. I think that's awesome, by the way. So just to recap stuff a little bit, I'd like to come to this place where we are. So when you go to think about your migration from OCS three to four, clearly you're gonna have a source and a target cluster. You're gonna have the MTC operator on both, but you're probably gonna call your target cluster the host, which is where the MTC this console would run would probably run on four. In addition, you're gonna have to think about a couple of things like underneath are you going to do a copy versus a move? And that will probably depend on whether or not like NFS and share and stuff like that. And if those two clusters are in the same area so they can actually talk to each other. And then hopefully if you're either way, even in a copy you have to go down and think about if you're going to do it directly or if you're gonna use the intermediary repository. Okay, so then in addition, that's just shifting data around in a couple of different ways. On top of that, you can actually create global hooks and then decide where you need to place them based on what has to change such as DNS and maybe some other actually, there's probably a million things that need to change. So, okay, awesome. Like I think that's great. I think that's very well thought out actually. Yeah, the hooks are like not, they're not necessary. And actually a lot of folks just decide not to use them because they don't need to. But it is a nice way for you to execute some custom logic if you would like to do that. If you need to. Yeah, one question here. You may, like we may have said this, but can you go from many sources to one target as long as you have enough capacity on that target? Like I take many pools of storage and put them into one. Is the question like, can I go from many PVs to a single PV or is it many clusters to one cluster? Many, well, that's a good point. I would assume PVs based off how the question was phrased but Keith, if you feel like highlighting or clarifying, let me know. The, we don't go from any PVs to one PV. So it won't consolidate PVs. There are probably ways you could do that if you were interested in doing that. But out of the box, it will recreate PVs for each PV on the source. Okay. Okay. So yeah, that would be like a post-step or pre-step even maybe, you know. No migrations. Yeah, I mean, you could write a hook to do it. Okay, that's cool. Okay, can I run this? It's ready. Yeah, so it does a lot of validation. We skipped the validation screen. There's actually a validation screen that will show like warnings. But at this point, we can just, it's all ready. So there's nothing really to see there. I guess there's a couple of things to point out too before we start. Hang on, Keith just clarified many OCP clusters to one OCP cluster. Yes. Is that possible? Yes, you can do that. Not within a single plan, because a single plan has one source plan target. Yeah. Okay. But if you wanted to, if you had multiple source clusters or whatever, you could do three to four. You can do four to four. Nice. I have a question. Do you always have to define a replication repository even if you know you're not going to use it? Yes, and the reason for that is even if you choose to do direct states and images, we still use the replication repository as an intermediary store for the Kubernetes objects. So that's where they get serialized and deserialized from. But it could be much smaller. You don't have to size it to the stuff you're moving. Yeah. If you've got like a terabyte volume, that won't go there if you're doing direct. Okay. Okay. Are we ready? Can I go? Sure. All right. So what am I, so here I get to choose as I'm staging or migrating or whatever else I'm doing. So I start with stage. If I did a rollback now, it would just tell me I can't do it because I haven't done anything yet, right? Might be a no-op. A rollback plan. Let me try. I don't know what's gonna happen, so. All right. So let's go with what, it's up to you, Chris. Lord knows I've broken enough on my office. So I'm happy to just proceed with the plan. Don't roll back if you haven't rolled forward yet. Maybe it's what we should say. Okay. So if I do stage now. I think it would be a no-op. It'll create a rollback migration. Nothing will happen, then you can roll forwards. Okay. Can I go into the stage and watch it running or is it just gonna come back with? Yeah. So the link immediately to the right of the name it's a second column. This one, there you go. So this is how you get into the pipeline view. So this is the list of all your migrations. And then these are the details. So this is the pipeline. So this is your Villero backup and there's your status. You can see like there were five out of five objects. Since we're doing a stage it is a smaller subset of the total amount of Kubernetes objects that make up your workloads. So all we care about during stage are your volume related objects. So there's probably some PVCs in there. You're gonna see a larger number of those when we actually go to do the final migration when it captures like the deployment and the other objects. Okay. All right, so now it's doing the other one. Hopefully this will start soon. Yeah, the pod volume backups is another that's the Villero. Okay. All right. Nice. All right, so it's in progress. Okay. Okay, cool. So yeah, it did a backup, a Villero backup to your intermediary storage. Now it's doing, it created a restore on the target cluster. This is the controller creating all of these objects. And really what the controller is doing is it's creating all these instances of CRs and then it watches the status to determine whether or not it can progress to the next state in the state machine. Nice. Is it, can we see any of this reflected in the OCB4 console under workloads? Yeah. When you watch parts come up and, okay. Yeah, so if you were to open the console right now, you should see, there's probably a PVC that's been created. So the PVC is where your staged data has transferred to. If you search your projects. Okay. See if there's an Nginx project. Yeah, three. Three, right. Remember, this is a target. So this did not exist before. So here's my PVC. Okay. Cool. Bound. So that's where you staged your data. So, and it should be on Ceph there. So you've actually gone from Gluster to Ceph. Yeah, right there. Okay. Nice. That's very slick. Yeah, there it is. Okay. Awesome. Okay, so then, anything else we should see while this is in action? Anything else you think, okay, so these have completed. That's the stage. It seems like it didn't. Yeah, so the stage completed and you can see it actually, there's like a runtime there. So you can figure out how long that actually took. This one's gonna be really fast. This is probably a good opportunity. So it like took two minutes roughly. It's a really small volume. There's not barely any data in there. This is a good opportunity to talk about planning your migrations. I'm not, you're gonna treat different types of migrations differently or like at least you should conceptualize them differently. So for example, a migration that's made up of many, many small workloads with small volumes, the bulk of your time there is gonna be spent in like the transactions of going through the states rather than actually being bottlenecked by the state transfer itself versus say fewer applications but with much, much larger volumes. Most of your time is gonna be spent in that data transfer. So that's where your bottleneck is gonna be. So depending on like in that case, you probably wanna consider move if you've got a huge volume, if it's an available option to you because that's gonna be the fastest way for you to do it. If you super care about, like if you wanna make sure that you clone your data, maybe move isn't for you, maybe you wanna do a file system copy. And maybe you wanna consider direct instead of the intermediary one. Maybe you don't, your source can't see your target. Direct isn't an option for you. You can go back to the legacy indirect method. Okay. And do you find like you want users to think about what kind of migration they're gonna have so they'll know just where the bulk of the time, where the work is being done, where the time is being spent as opposed to worrying about it and say, oh, I know what it's doing. It's building it up, it's tearing it down. It's kind of, but the shipment is small or something like that. Okay. Any other considerations for migration that people should, you know, cluster admins should think about? You talk stateful, stateless, how many PVs we're talking about? I mean, aside from looking at each PV and kind of just comparing and contrasting, how is there a verification step that happens during the migration that like actually checks the ones and zeros on disk? Yes, actually. Okay, cool. Yeah, there's a checksum verify. It's very expensive. So that's gonna add a lot of time to your overall migration, but there is a checksum verification that will compare that to ensure. I think that's even a, you know, there's some like a... Is that RESTIC? Regulatory like requirements for that. Yeah, is that, I'm kind of curious what funk or tool is doing that. RESTIC does it? Yeah, okay. And I think at a bottom, I don't actually know specifically what RESTIC is using. I know our direct migration uses the R-Sync checksum. Cool, nice. Okay. So do I move on to migrate now? Stage is successful? Yep. So simulating this like, I've staged my data on Friday, now it's Saturday, I wanna go ahead and actually do my cutover, so. Right, okay. All right, so. I'll let you go. Do you wanna read this and maybe ask me some questions or I can explain what this is asking as well. Okay, hang on. All right, so this is the shrink it to zero on the source so that we can make the cut. This is actually when you need people to stop writing to the volume so that you can just capture the delta. Yeah. The reason why we prompt you for this is that maybe you know your application is fully capable of remaining up. While we do this final cutover, it kind of depends. We explored some other avenues where we would actually delegate to the user to define something that's custom here. So maybe they can put their app into a read-only mode. But as a default, and it seems to be acceptable to most people, we just queues it to zero. Okay. Cool. All right, so we're gonna hold it. There we go. Okay, so now if I go in here, I should see my migration, okay. This might actually be an opportunity to show the debug resources view, I think that we talked about too. So we can actually see the tree of objects that make up a migration. Okay, where am I going? So if you go back to the top level, I think is where it's at. Under the kebab menu, the view migration plan resources. So here is where you're gonna see kind of the underlying objects that make up a migration. So this is the plan. And all of the resources that are spawned by the plan that we're watching. So you can do things like copy your OC get command. You can actually view the raw JSON here to see the objects. So if you wanted to pull that open, this is actually the Kubernetes object if you wanted to get into that via the UI. So maybe you wanted to do it on the CLI. You got the copy command. If you wanna view that in the UI, you can do that. This is really helpful because earlier versions, like this is another improvement that we've made since CAM. It was pretty, you actually had to string together, like sort of follow all the breadcrumbs manually in order to get down to these objects, which is not a good debug experience. So this actually builds that tree and presents that to you. So you get the name of the objects to everything that actually matters to you. So, and then you can view those objects here as well. So it's just an easier way to kind of, we string everything together for you so you can skip that process of like following this ID as you over here. And then you can get a whole sort of tree overview. At least I have a question. So I understand the first one, right? So here we have our, we're backing it up first and then eventually we restore it. But why do I see a second one? Just curious. So there's two migrations here. One is the stage, which is that bottom one that you've got highlighted. And then the second one is the final migration. Oh, gotcha. Got it. Thanks. Wow, look at that. All right, are we, where are we? Migration. It's running. Let's see. Wow, almost towards the end. Here we go. Oh wow. Store time. You did it. Oh, that was quick. That was quick. You did it. Migration successful. Okay. All right. Logs that we can look at after the fact. Do we drill down? We're having long conversations about this. Yes, there are logs that are available. We have sort of a pseudo log beer here. Let me talk about a few things that we're working on right now. We have a consolidated logger that's kind of a bolt-on, like add-on. I don't think it gets deployed by default. It can be kind of expensive. If you're interested in that, you can hit us up on the Slack channels or on the mailing list, we can dig into that. So this version is 1.4.2. 1.4 is our, I want to say it was the first release that we really had a major, we added a lot of stuff to 1.4. Part of it was this like new phase status and progress and everything and the pipeline and like visualization. The other one is direct. So that was like a major release. But we're in a 1.4.2 release right now. We're expecting to do some sort of monthly Zstream releases. So we're going to do 1.3.1.4. And as part of that, our main focus right now is just making this tool more ergonomic in that. And what I mean by that is like when this, it works, we've come to see that it works really well when it works, when things break, it's pretty opaque. And there's a lot of parts that you need to go trace in order to figure out really what's happening. So we are really focused on improving that experience. And part of that we think is a better log experience. So logs are a divisive topic. There's a lot of philosophy around how much do you log? What do you log? How much is too much? Are you actively taking away from the ability to figure out what's going on? If there is too much, do you log stack traces? So a lot of conversations is going on around this. You can read up on those conversations. They're on JIRAs, they're on enhancements. Everything we're doing is out in the open. And but what we're actually trying to get to now is with 143 we're expecting to really focus on improving that log experience. So there's a few problems. Logs are distributed. So we're trying to consolidate those all into one place so that it's easy to access. You don't need to jump around the 10 different places to get them. Another thing is we think it's kind of under logged from a user perspective. There's a lot of developer focus logs like stack traces. We don't think that's not gonna mean a lot to users. So one of the things that we're trying to do and we're having a lot of conversations around is, let's say like the controller, I mentioned it was a state machine. So it goes, you can actually see it as it's going through all those phases. So when the controller comes to a decision point and it's trying to figure out, can I continue to the next step? We need to make sure that we log that if it's not actually, if it's unable to progress to the next step because that's gonna tell you, oh, you've halted at this place. Something might be wrong. Let's say if a pod isn't able to come up because the volume can't mount. So that really helps in aiding you try and figure out what's actually going on underneath the covers. And the overall goal is to help folks get from, like I mentioned, a lot of these problems are environmental, but there is a time, the time between seeing that something went wrong and actually figuring out what the root causes, we need to reduce that time because often it ends up being the next step is, oh, I've got an environmental problem like my storage layer is down or something like that. So that logging strategy is an attempt to shorten that time to help you get to the root problem faster. So decision points that the controller is making is really informative for users. And then as a final step, there was another one, oh, structured logs. So you have to be able to fill, there's a lot going on from a lot of different components. I need to be able to say, give me all the logs for this plan and this migration and get those logs without any other noise to understand like over time what decisions the controller has been hopping through. So that's another thing that we're expecting to release with some good support around that. So this is the old version. And then, okay, just flipping, poking around. Well, this will probably turn into a, this is a good way to like dump your logs and like kind of send them over to us. We also have a must gather. If folks have heard of that, that'll grab everything. There's a lot of sensitive information in that potentially. But yeah, this will probably turn into either, you know, we're rethinking this, this part of the app. And then we're sort of thinking, we may want to actually link directly into the OpenShift console itself here since they've got such a good log viewer. So that's on the topic of logs. And then I guess the second piece of that is Kubernetes events. So, cube events are a pretty native way of sort of figuring out what's going on in your cluster. And we want to leverage those more than we are today. So the example that we keep coming back to is like when you're trying to debug what's going on in the pod, you're not going to the pod controller logs typically unless something's really gone wrong. You're usually looking at the events, you're just describing the pod. So we want to get to that point as well. So a more natural way of debugging the way you debug everything else in Kubernetes. Okay, all right, that makes a lot of sense actually. We only have, I don't know how much time we have left. Any questions? I was just before I asked you. Not really. James Lelaki came in and dropped a link to the OCP best practices doc. So that was super cool. Thank you, James. And yeah, I think people are looking at that. If you're watching out there and they have questions, get them in now. Yeah, this was awesome. Thank you for coming on and telling us all this stuff. And part of this for me came out of doing some workshops with people where they're on OCS 3.x. And I was like, we have a toolkit but I don't know how to use it. I don't know anything about it. So I'm really happy to bring this to people if this is something that you're experiencing. I don't want any speed bumps between OCP 3.x and OCS 3.x to going to four. So I thought this was really well thought out and then maybe showing them. You have some scenarios where you have break fix, kind of like you actually set up a broken migration and we go in and fix it. That's something maybe we could do in the future if people are interested. I don't know. I don't know how many of our users are on 3.x right now, actually. I mean, I would assume quite a few of our users are on 3.x. I'm not sure if we can like actively pull stats for that or anything from the limited knowledge we have of what's running out there in the fleet of clusters. But I would imagine it's pretty big, right? Like storage is something that you muck with very judiciously and cautiously, right? Like, you don't just, hey, we're moving our storage today, right? Like it's not something you wake up in the morning and it's like, all right, we're gonna do this and off we go, right? You plan it, you think about it, you make sure that your destination's solid, you make sure that your source is ready, that kind of stuff, right? But this makes it, wow, like the fact that we migrated to anything in less than an hour is pretty impressive, in my opinion, right? Like when it comes to, when I think storage migrations, I think long times waiting, right? And this does a lot of the verification steps and checking and just makes it very seemingly easy. And I use that word very rarely. That is the goal. We have said like the, we throw around easy button frequently in our conversations. Right, yeah, I mean, that's like, when I think, yeah, going back to the storage migration thing, it's just like, I think it's just a nightmare always to migrate storage and this is not the case anymore, I feel like. It's much better. I was gonna say, Chris, when you think they're, I'm waiting for hours and hours for storage migration, I think there goes my weekend, honestly. Right, yeah. Like, I mean, I've, you know, I've been part of like some large telco organizations where it's like, yeah, we're doing a massive, you know, storage array migration this weekend kind of thing. And it's, you know, we're in data centers like migrating storage all weekend long. Like I distinctly remember a trip to Florida one time where like I was actively, we were driving back and I was actively working part of this plan on a tether, right? Like, and yeah, it's always, always usually like a weekend or weekend plus plus kind of deal like you start on Friday night and you end up like Monday morning at like eight o'clock, everything's done. Eric, I wanted to ask any, can you discuss loosely discussed roadmap? I know you've mentioned, okay, focus on logs and shortening the debug time, but can you just, I don't know what's publicly available yet, but if you could just give us rough timelines for things. Sure, let me give you one second and I'll pull this up so I don't miss anything. Good call. No, I always mess up the roadmap. So remember, this is roadmap stuff. It's, you know, future it may or may not land in whatever version we're saying it may or may not land in. So, you know, roadmap take with a grain of salt, dates are TBD. So we've got our 143 release. That's our next Zstream release coming out. Mid-April roughly. So I mentioned debug maturity. We're working on the logging as I kind of went through. We're talking about that debug resources view. So we're trying to ask, like move that into a place where it's more easily discoverable than behind that kebab menu. We actually had a funny case where a customer was asking like, hey, it's really hard to like, like trace this tree of objects. It'd be really nice if we were able to see, get that automatically. And it's like, funny, you should say that we actually have a debug menu over here. So they weren't aware that it was there. So that's something we need to make sure is like actually, like clear. Yeah. And then if you're looking at that and you've got like 10 different PV pod volume backups for Valero and one of them is bad, you wanna be able to easily pick that out from the list. So we're trying to do some like sort of UX improvements in order to help you just like set off alarms where the place is like where you need to look in order to just make like reduce that time between problem and figuring out what's actually going on. Another thing we're looking at is just like continuing to work on the DVM, which is with the direct volume migrations. We're trying to make that as resilient as possible. So like if an R-sync timeout happens or something like that, we'll get like exponential back off and it'll pick up from where it left off. So it is R-sync. So if you've got like, if you've done half a terabyte, you pick up from half a terabyte, you don't have to redo the whole thing. So you get the incremental changes. We're talking about storage layer improvements. We're pretty involved in Valero upstream. So we're like fixing bugs up there, race conditions. And we're talking about also just sort of expanding the storage classes that we support move for. So I mentioned Cinder is one, we have some folks who are interested in Cinder move support and Ceph in general. So that's one for three. We've got, we're doing like monthly releases. So in May, we're just continuing on the debug maturity story. So we're working on the CUBE events. We're talking about standalone state transfers. So today there are ways for you to actually do just get your data, move it from one place to another using DVM, which is this new feature that we've got. It requires you to kind of jump through some hoops in order to actually make that happen. Namely, you need like one of our engineers sitting next to you telling you how to do that. So we're trying to make that more accessible and improving the hook experience. So like having hook terminal states that are surfaced in the UI nicely and you get your hook logs and everything nicely. And then like longer term, we're just talking about a lot of hardening. So like validation pre-flight, making sure that like you're just doing as many checks as we can to make sure that your migration is gonna go smoothly upfront when you have the time to actually investigate, oh, my source actually can't see the target. So trying to do those validations upfront, integrating with the OpenShift console. And then, yeah, that's mostly it. And then we're expecting kind of a one of five release later on this year, let's say. And we have some other sort of, it's a little inside baseball like CR, like V1, beta one CRDs are going away with OpenShift 4.9. We need to address that stuff like that. Nice, okay, yeah. Well, it's a lot, it's good. Very cool, good stuff. Anything you wanna squeeze in the last minute and a half here? Let me think if I wanna plug anything. I definitely wanna plug Red Hat Summit coming up. I will drop a link to that in chat for everybody. Check out Red Hat Summit, the site there. We can, we're doing it in three parts this year. So it's definitely worth looking into, figuring out which part or which parts you want to attend. And I would highly encourage you to check that page and you'll get more knowledge like this in the Summit content as well. So it would be great. Anything you wanna promote, Eric or Michelle? Yeah, I would just call attention to the conveyor community. There's like a lot of really cool stuff that they're doing there. Meetups, it's not even just Red Hat people. Like we're really just trying to grow that into a community of people who are interested in this space and growing our projects. Like all of our stuff is upstream. Like I mentioned, we've got mailing lists. There's a conveyor slack channel on the Kubernetes Slack. We'd love to talk to you if you're interested. And, you know, I'm in there. Our engineers are in there. Everyone's kind of talking all the time. So if you're interested in digging deeper into any of this stuff, we're available. Thank you, JW Matthews for dropping the link to conveyor.io and chat. Appreciate that. Awesome. Awesome. Great show. This is fantastic. So thank you very much, Eric. Thank you, Michelle. And coming up next on the channel, we will have the Dev Nation crew. Sebi from the show is coming on. I always love Sebi shows. They are fun filled and educational at the same time. So he does a very good job with those. So thank you very much. I will be back on the air later today, myself, with Christian Hernandez and the folks from WaveWorks who invented the term GitOps. So GitOps Guide to the Galaxy will feature the people that created GitOps, the term. So tune in for that today at three Eastern 1900 UTC and the Render, sorry, DST has this all out of whack. So you will be back on track later in the month, I think. Thank you. So thank you all and until next time, definitely please stay safe out there and let us know if you need anything. Thank you very much. Thanks for having me. No, thank you. Thanks for coming.