 Good morning. Good afternoon. Good evening and welcome to the level up hour here on Red Hat live streaming I am Chris short host and showrunner of Red Hat live streaming as You can see the usual suspect Langdon is missing We wish Langdon the best of luck. He has taken on his Professorship duties at Boston University starting this week and if you weren't here last week yet Langdon has no longer at Red Hat so Just to let you know. He is off to being a professor. So me calling him illustrious and a professor on the show Might have done him some good. I don't know. Maybe I'll take credit for it though So today I'm joined by the new usual suspects The one and only Randy Russell Jafar Chorabi and then we have special guest Eric Nelson here Let's just go around the horn real quick. Let's do some intros, you know, Randy I'll toss it over to you as the quote new host of the show What? Yeah, so yeah, hello everybody I'm Randy Russell. I'm the director of certification at Red Hat and really excited to be joining the level up hour and Having these discussions with you approximately every week or ish. Yeah, you know when we're not dark for various reasons Jafar Hey, so my name is Jafar. I work as a tech market market manager for OpenShift and I'm really glad to be here. I'm trying to grow my verb So it looks like a bit like Langdon, but it's gonna take a while. So yeah, so hopefully in 40 episodes It might be might be Eric's almost there I Will say Scott McBrian is also gonna be on the show regularly, but he is off this week So he's on PTO. We're gonna talk about conveyor today and specifically Where's mine, where's yours? I'm sorry. Oh, I can definitely get you on. Oh, there you go They are six dollars, I think on the cool stuff store for what it's worth So yeah, Eric you're here Hi, hello, how are you? Welcome back to the channel. Thanks for having me. I am Eric Nelson I'm a manager in an engineering manager in migration engineering. So I work on a project called crane, which is the topic of this conversation Downstream it's known as migration toolkit for containers. I've been an engineer at Red Hat for Six don't know five years at this point and prior to that I'm a computer engineer and actually I went to school at Boston University. So I know that's a pretty good program There you go. Yeah, only you delay your education. Yeah, I should have done that Yeah, you could have been taught by the illustrious one But oh well go back for your masters or something. Yeah, I Yeah, I'm in Raleigh now. So but I do miss Boston quite a bit. It's gonna be I based migration There you go. Exactly So just let folks know we will still at least for the very short term have our usual points and everything else I like and that's sweet all that off to me. The sweet sweet internet points are here As far as their value that is up in the air Still sorry, I haven't heard anything otherwise But I know that something is coming so we'll keep the points going for now So we're talking about Crane migrating container workloads with crane Where so I already got a question fielded for you, Eric You but let's let's go ahead and talk about what crane as first before we dive into the question that I have Sure So crane is Should I show that the slide that kind of gives an overview Maybe before we even dive into crane, let's talk about conveyor conveyor. Yes. Thank you, Randy I know can everybody see my this this diagram here. Yeah, okay cool. So Conveyor is I think there have been some other projects from the conveyor org that have have been here So folks may have heard about it, but I'll just quickly give an overview. So conveyor is kind of an umbrella like collective of projects that All sort of have a shared objective to just move people to cloud native environments specifically kubernetes And and so that that can be like there's a lot of paths there There's a lot of problems that are involved. So there's a lot of projects that seek to solve a bunch of different problems So you can see here. There's re-hosting. So maybe you've got like one cluster in one place. You want to move it around? crane concerns itself with container workloads So like the traditional kubernetes migrations, maybe you want to go from one cluster to another and it's already been containerized Then there's a project called forklift which concerns itself with actually picking up vms and moving them to kubernetes clusters Which is pretty cool uses cube dirt So it's kind of forklift and crane or kind of sister projects in that respect Then you've got replatforms. So move to cube expects a source. That's not a kubernetes cluster So other platforms such as like docker swarm or cloud foundry This will pick stuff up kind of analyze it spit out kubernetes objects And then that'll help you get like to sort of convert your workloads to kubernetes Tackle I think was the one that was actually on here if I'm correct But this will help you actually refactor your applications I think we'll probably talk a little bit about this, but All applications are not just created cloud native out of the box. There's a lot that actually goes into architecting and designing and writing like Software so that it expects to be running inside of a cloud environment and that it can take advantage of all the benefits That are associated with that. So tackle helps you helps you actually look at your application and make some recommendations about how you can manipulate that And so that's turning into migration toolkit for applications Which is our supported downstream offering You'll hear me say like MTC or MTV or MTA. All of those are like our downstream references And then we've got our upstream projects and we're trying to move towards this like upstream first model Let's go easy on the acronyms Did the move to cube people not get the memo right on the new conventions because I love crane and forklift and tackle because they are in fact very descriptive of what these things do move to cube also descriptive, but that's not from the moving industry It's not for the building industry. Come on. Yeah, it's a well We know that like in software the hardest thing is like actually naming things. So yeah But actually move to cube is a is an IBM project. So it started with IBM So that's actually the background there and why it's not it doesn't it hasn't adopted one of those types of names Fair enough. And then finally there is Polaris So this is like sort of metrics like measuring the changes and impact of like how you've delivered things It's almost in order of like maturity Crane's been around since 2019 So I guess I can jump into crane any questions about like conveyor in general or what we're doing here Well, let me just throw a really high-level question out of there on this and so this project got kicked off really earlier this year, right? Crane or conveyor? I think it's been around Maybe let's say a year and a half Crane crane did predate conveyor as a as an organization But it's true that it it's been around for let's say under two years Yeah, so we have some of these independent projects that have sort of come under the the conveyor umbrella Is all being related in the sense of being able to Manage this big picture right in different ways whether it's replatforming Rehosting refactoring. It's there's there's a tool for everything, right? So here's the thing I kind of wonder as just sort of first high-level question is You know, we have this need right now to have this sort of tooling Do we anticipate that this is the sort of that that the conveyor project and that these Things under that conveyor umbrella will continue for for years to come because this is going to be an Ungoing need because I think a Polaris is sort of the end of the cycle But it takes you right back to the beginning of the cycle Yeah, for sure There is an expectation that this is going to live on And and it's not just migrations like migrations are sort of like a point-in-time event Once you're there you sort of but You know, we've learned that that's not really true Like like you pointed out Polaris is something that you're going to use to For day two operations monitoring things upgrades all that sort of stuff and then crane The genesis of like crane was to move people from open shift three to four because we didn't have a migration path there, but it quickly became apparent that I can get into this but you know In order to take advantage of hybrid cloud right like on-premises and like cloud environments where you have burstable resources and stuff like that Your applications have to be architected to be portable And and and not only that like there's a whole environment around them where they need to live Like they need to be decoupled from the environments that they live inside of And exist in kind of a golden source location. You could call this a get look get location a get repository that deploys them out so In an environment agnostic way so you can like point it at a cluster and kind of deploy it there So in that respect, it's an ongoing problem And and especially when you consider hybrid cloud where things are moving around You need things to be fluid and so a lot of these these projects seek to help you do that And maybe your source code 100 and portable apps is an ongoing concern So it's not it's not a single point in time migration So a lot of these projects underneath conveyor are seeking to help that and in addition to that conveyor is kind of a living breathing thing It's not just a set of Repositories. We also are running like meetups. There's conversations going on on mailing lists. So It's a community really is a more accurate way to describe that and And again, like that's anybody's interested in this is is this is a great place to kind of connect with other people Who are also interested in the problem and work on tools that will help you achieve that Great awesome So, uh, do we want to talk a little bit drill into The wonders of crane Sure, I'll I'll send my questions for a bit or when we have A bit of understanding of what what it does Okay, I already have a few questions So, um, I'll go back a little bit. Um The I mentioned that like crane Crane is the upstream name at the time. It was really only mtc, which is migration toolkit for containers Um, I'm just going to call it crane We'll call it crane v1 because there is a crane v2 that's on the horizon Um, and that's kind of the interesting part of like what I was hoping to discuss here um crane v1 Concerns itself with like mask cluster evacuations. So This is something that a cluster administrator and that's like an important point It is only available to cluster administrators To pick up everything that's in a in a cluster and move it over to a second clusters This is a total like a whole whole migration It does not concern itself with the control plane. So, um, it operates at the application workload layer um, and primarily, um Usually when you think about a kubernetes migration, uh, it's kind of three buckets You've got the kubernetes resources that make up the application. So you're like pods deployments Stuff like that your images that make up that back the containers So often these are images that are inside living inside of like a registry that you've either got inside of the cluster somewhere else maybe and then finally there's the The state which is kind of the the hard part Exactly that lives inside of the pvs. So crane Picks all that up and it moves it over in a specific manner. Um, I can get into like the architecture of how it's architected but Yeah, that's the objective and it will do it at like a sort of a mass scale. So We've had I mentioned in kind of the show description We've done like large scale migrations and I would call those like we're talking multiple clusters and many geos with thousands of namespaces crane operates at a namespace level. So that's the like unit of It's the atomic unit that we use in order to migrate things Often people will organize their applications it within namespaces on a namespace basis That's not always the case. So you could have like microservices that have the interdependencies spread across namespaces In which case you're you're going to want to evaluate that and make make sure that you bring them all over together as part of a plan But at a high level, um, it's effectively a it's an operator backed application that Is controller based and its api is composed of crd So it's using the kubernetes extensions as its core api So you can communicate with it as you would any other cube object And so it uses that sort of declarative reconciliation pattern that kubernetes is famous for To execute these migrations across clusters Um Nice. Yeah, so we already have a question in chat, which was the question I got before the show even started Go ahead. It's a far You sure? Okay. Yeah And jw matthew's just answered it in chat, uh, but air gap environments talk to me about that Can we use crane? Yes in okay, like Tell me more because yeah, we have a it's a it's a Primary concern for a lot of customers. Um, of course, they don't have these things just living out on the internet They're pretty important and they live within Disconnected environments So there there are a couple levels of disconnected that you can kind of think of air gap would be one which is like This is a hundred percent a hundred percent not accessible which is possible and then separately you've got Pseudo disconnected which is sort of it has limited external connectivity Maybe it can see the outside world through a proxy or something to that effect. Both of those are supported um And we've had a lot of people use mtc in that manner. So uh The way that that works is with with open shift four, um their open shift four uses olm which is kind of its Packet the operator repository, right and there are mechanisms that allow you to mirror the images into The environments so that you can actually deploy mtc In order to actually perform the migration we we of course do have the requirement that both of those clusters can see one another Or at a bare minimum Well, there's also this is getting into kind of how mtc is architected Underneath the covers mtc uses a project called valero Some folks may have heard about that. I've heard of it. Yeah It is a backup and recovery solution for kubernetes The world's kind of coalescing on it as like the the backup and recovery solution dr solution for native kubernetes So we're built on top of that So actually mtc what it is is a controller that orchestrates a migration using a series of backups and recoveries Communicating with valero, which kind of executes underneath the covers. This has changed over time, but that's kind of the crux of it For that reason we actually require an intermediary repository where We back things up to and then we recover things out of so We we recommend for a disconnected and air-gapped scenario to actually deploy you can run like ocs which is our container storage solution on An open shift forecluster and there you can set up your s3 repository that's self hosted so it's all on your let's call it your target cluster and You'd be able to use that as your intermediary repository So the controller would basically move all your data out of the source into that target repository And then it would recover from that repository into the target cluster Trying to think what else is relevant as far as disconnected Is the operator and the certified index or their operator index? I'm pulling up. Okay. That's what I figured Yep mtc. You'll find migration toolkit for containers there And then there's also a community operator called crane. Yep. Awesome So I do have a question Which is related to what so I kind of like the way you dissected the main Areas of interest when you are doing this kind of migrations You spoke about like the resources The communities resources which are basically like the the whole the animal fires you have to export in order to be able to push them somewhere else You have the data and then you have the images And so I know that we started this project a while ago like two years ago when we started addressing the migration from three to four And at that point in time get ups was not yet like a very hot topic So the whole thing was I had customers asking me while I was still, you know doing pre-sale stuff like so what's the proper way to export stuff like my resources and get rid of all the cluster specific details like creation date ids and stuff like that in order to be you know to to make them generic and then push them somewhere else um, so that that was sort of a manual process and I think that uh mtc At that point in time already was able to solve that kind of questions by introspecting the cluster and being able to To to migrate items without necessarily having the definitions at hand but uh, so my question now is now that we are also playing in the gilabs field where we have this this notion of now that I have my resources descriptors like all my animal finds stored somewhere I can push them across multiple clusters and I can also Keep that consistency like if I update something from the source cluster, then I can automatically Maybe you know update it And mirror it back to the remaining clusters So is it something that uh mtc is Is is working with is it something that you know, so how does it like because the way I see it is You have the operator that does this thing it introspects stuff and it moves resources somewhere else but Does it make sense to to sort of plug it a gilabs? I would say engine somewhere inside mtc to have that you know that Reconciliation loop Uh capability and all of those, you know new stuff that came up Uh recently I would say Yeah 100 um, I'm happy that you're kind of teeing up perfectly what crane 2 seeks to do. Um, okay, so um mtc one And I didn't know that yeah Yeah, that was uh, yeah I mean, it's it's interesting because you're pointing out like a real pain right like this is what people want to be doing um, and so we've spent a lot of time talking with Services folks and people in the field and customers trying to figure out like What are the things that you want to be doing and how is mtc was a little bit early? Believe it or not like it sort of missed uh like There were still a lot of like people sort of figuring this space out a little bit when we did a lot of the initial design But we still believe that there is space for mtc Um and crane 2 and I can get into crane 2's objectives Um, they're they're kind of they're they're targeting two different problem spaces the first one with mtc This is sort of its pro like some of its faults are sort of also core to its design It's a tool that does a like a mass cluster Migration so and it's it's a tool for cluster administrators So they can pick up a whole cluster pick up all the app workloads and move them over to a target cluster the um We what we found while uh interface like while seeing this in actually in production migrations Is that oftentimes cluster administrators have no idea what's actually inside of their namespaces? So imagine there's a large university right that has um all of their students running their word press blogs inside of namespaces Some of them are crashing like the whole cluster is kind of a disaster They don't really know what's in what's inside of those. They don't know if the like a crash loop pod is a problem um They don't know that like the interdependencies between namespaces the application owners do And the developers do there are the people who are deploying into these environments. So That was the first thing is that the fact that mtc is a cluster admin required tool in addition to the fact that The admins kind of don't know what's inside of those namespaces Means that there's kind of a gap that needs to be filled and that's where crane 2 comes in So the app owners are really the people who understand their applications and their interdependencies and everything about that Let's like shift that paradigm a little bit and have the developer the app owners themselves actually migrate their own applications So that distributes the load of and the burden of a full cluster evacuation It also allows you to do it in piece mail Which mtc can do depending on how you how you slice it up but um So that's one of the the first objectives of crane 2 is to actually allow for app owners themselves to migrate their applications the second thing is You're right like you kind of nailed a lot of the points that crane 2 is is thinking about There's I like to kind of describe hybrid cloud like crane 2 allows you to unlock hybrid cloud because There are a lot of these applications that are like snowflake applications in that they're very tightly coupled to the environments that they live in They've probably had like kind of duct tape like changes made to their resources over time that are specific to the environment We know of a lot of examples of that. Let's call them node selectors resource quotas like stuff that you pointed out So in order to actually access hybrid cloud the first you need to a portable application And what that means is you need to strip out a lot of that Like cluster specific information And then you need to version it you need to put it into get ops model It actually could be like kind of any real version the point is to version it in an in a cloud in a cluster agnostic way and then So really and then continuously deliver it to your various clusters And maybe you want to actually layer on some more cluster specifics as you do that So that's starting to sound kind of like a pipeline, right? And so what we're interested in is with crane 2 is is actually building that pipeline where you have that discovery aspect Where even app owners may not know what's inside of the namespaces So it goes out it exports all of the stuff that's inside of the namespace And then it applies some transforms to those objects that are by the way commonly Like a lot of people are going to have common interests Let's say they want to strip the status which is not actually relevant for a target cluster for example And also it's not it's not a live application. So it doesn't necessarily have status at that point in time So you want to strip that stuff out? Maybe you want to like there's another example with routers Like maybe you want to strip out the the host that it lives behind because of course that's cluster specific um, and then from there Once you've applied these transforms and we have a plugin model around this that's I think is pretty exciting that allows like services folks with shared problems to Build their own plugins that are reusable you can pull them off the shelf and apply them and then um So you'll get this kind of like stripped down Portable representation of your workload that can be version. So you check that into a git ops a git repository You flip on your git ops and then you can use this pipeline to deploy it out into whatever clusters that you're interested in deploying it in so Crane 2 is designed for that and in a lot of ways like it's a migration tool And it allows you to migrate from an application to cluster to cluster But really while you're doing that migration It's a great opportunity to modernize it too and and to to unlock that hybrid cloud by adopting that git ops model And so we want to help people get to there as well There's a lot of unsolved problems there state is one that's tricky And there's a lot of conversation that's happening that we're involved in in some of the upstream kind of like SIG groups data protection stuff like that where How do you like fluidly replicate your state across a cluster? How do you deal with the fact that these applications are live? So maybe you need to quiesce the applications move them into a read-only mode so that you're not corrupting your data while you're moving it around Stuff like that um are tricky tricky problems to solve but um, that's that's really what crane 2 is seeking to solve is is and how it differentiates itself from mtc whereas mtc will continue to live on as this like mass cluster evacuation tool and crane 2 seeks to be kind of that um tool that's designed for app owners to export transform and Modernize kind of their applications into a git ops flow so that they can be then redeployed in a cluster agnostic way Beautiful So yeah, thanks a lot for this like very comprehensive answer and i'm i'm happy to see that uh, you know it's sort of it's going to solve some of the questions that i saw within the field so That's that's cool to see that we are You know addressing those real problems So I I have another question if nobody wants to step in We have some questions in chat but go ahead Okay, so you mentioned one key aspect I believe is the understanding of the application and its ecosystem like its Uh dependencies and when we mean dependencies, it's uh, it can be things like The data it's relying on like databases So you can have the pod running in certain namespace But if you don't migrate a database that runs in another namespace or another microservice Or something like that the application is not going to be fully I would say Working so So I was doing uh for for a few years some enterprise architecture consulting Consentancy and and one of the key things that we did before, you know when we wanted to address these types of migration was Assessing the application and like properly Having an application topology where we Model the application and its dependencies like in enterprise modeling solutions So without going into that much of advanced stuff Do you guys think uh I haven't seen MTC recently, but is there a way to sort of Gather the dependencies of a specific application. So you said for instance The admin doesn't know the application. It doesn't know to what services it needs to talk et cetera, so I believe it would be very helpful When you look at an application definition within the those tools to to have some something that says Here's the application and here are all the services that it relies on like It can really point to some specific namespaces services, et cetera Do you have something like that that like gathers objects from the existing Cluster to to to give an overview or a topology of the application and its dependencies Or even if we can you know just describe it by text or something like that Is it something that you have that you plan to have? What's what's the the take on that? And it's definitely important. Um, and MTC doesn't do that. Um, and neither does crane And and it's kind of a gap in kubernetes in general kubernetes doesn't have a high level abstraction that kind of encapsulates like this is my application and all of it's interdependencies, which is A gap and I know that there are conversations. There's a lot of conversations that are going on around that In the six gaps. I think they're trying to address that hundred percent And that's what we would build our migration based on instead of like a the unit of migration and like the closest we can get to that It's kind of a namespace Out of the box the problem. Yeah, everybody kind of represents them in different ways Yeah, because sometimes you can have several components living in different namespaces for the same application 100 and and that's where you know, it becomes harder to get the whole application Which relies in like 10 namespaces or even five namespaces or whatever Yeah, our recommendation is um, we we kind of expect you expect that legwork to have been done Um, it is a problem that I think there's there's opportunity to solve and make that easier for sure I think that I've I've heard some like rumors of other tools that may seek to do that um, but it seems it seems like it would be an opportunity and a perfect problem to kind of be explored underneath the conveyor umbrella. Um, We also have mechanisms for actually onboarding new projects stuff like that. Um, but yeah MTC and crane don't concern themselves with that sort of analysis Cool. All right Good to know. All right, randy. Do you want to tackle these questions and chat? Do you want me to handle it? Please be my guest. Okay um What if I want to migrate my ocp cluster from one data center to another Just because I heard the word for larrow is just covered by these conveyor tools Yes, um, so if you have an ocp cluster in one data center, you've got another one. Um, That's that's like the bread and butter of what mtc does. Um, so you you Just to quickly describe kind of the the data model and how you would Do that You install, uh You have to install mtc on one of your clusters and actually it doesn't have to be the source or the destination It could just be a third cluster. In fact, like I've seen people run it on their laptop and like a One of the developer clusters um connect out But um, so you have to you install it somewhere and then we've got this, uh, crd data model that represents You know everything that we need. Uh, that's our api and so you can communicate that with um, so we have a ui That can be used and that's super helpful. It's gotten uh, very very useful. Um as we've moved forwards um, it's particularly useful when things go wrong and um as we've learned with migrations These almost never go smoothly. So it's very helpful. Um to have um And uh, so you register your uh, the clusters that are involved in your migrations So you'd register the two clusters, uh, you you give us like the coordinates to the cluster as well as your credentials So we can access it Then you're going to go set up your replication repository, which is this intermediary, um storage where we're going to export and import our data, uh to and from between the two clusters and then you're going to um draft a plan and what that means is, um You're going to pick the namespaces that in your source cluster You're going to pick the pvs and how you want to handle them Maybe you want to change your storage classes. You're converting from like cluster to saff or something like that and then, um There's there's a few more details, but at its core you kind of the mig plan represents This is what I want to move. Um, and and there's a discovery process that helps you Figure out what's available to you to move Next up, uh, and the way that we've we've built mtc. This is getting into A lot of people want to reduce your downtime as much as possible, of course, so Some of these applications can be quite heavy. We've done terabyte volumes before so What you're going to want to do is stage as much of that data as possible This is assuming a stateful migration, of course, stateless is a lot easier where you don't have to deal with this Um, so you can you can leave your source application up and running You'll stage as much of your data as possible onto the target without taking your application down So let's say you get like 99% of your data onto the target Um, and then you do your final cutover migration, which uh queueses the source application By default you can actually switch that off Uh, and then we'll grab that final delta of data that's been written between the last time that you staged your data and Currently, so we'll get that last piece. We'll move it over and by staging all of that data The time that it takes for your final cutover and your downtime should only happen when during the time that we need to capture that final bit Um I think that's it. Yeah, that's mtc in a nutshell So that's going to help that's how you can migrate from one data center to another Of course, there are these like caveats where both of them actually need to see each other They need to have access to your application repository. Um, but that's that's how that works So I do have another question go for you for yeah related to what you just talked about so Um, when we're speaking about the state state of the applications As you mentioned one of the problems is to to get that live data So it's easy to to migrate static data that doesn't you know change But uh, if you want to get the last updates to to the data while you are already, you know Planning or doing the migration It gets a little trickier and one of the things that happened in the in that space in the data Migration or synchronization is what we call cdc like change data capture Uh, where we have tools that are able to plug onto a source database Where you say so here's my e e-commerce database and here's my my target database and It does actually migrate and copy the data But as as soon as you have a change that occurs in the source data, it also replicates it into the destination Database and so I know that red hand has an extreme project about that called divisium Are there any plans to sort of you know try to Plug in to tools like that that could help address that issue of you know capturing data, but also Keeping that you know update easier or any more I would say industrial way With with with pain will it make sense to have something like that back been and How synchronize the data Actually, I haven't heard of that project. I'd be interested to look at it a little bit. Um We we don't have uh, we don't there's no plans on the roadmap currently to integrate with a tool that helps you do that Um, I actually don't feel but we haven't had a ton of interest in or problems that have been raised around actively like active applications that are writing during a migration or something like that. However, we do have um Concepts that will help you do that. Um, we have something called a hook uh that um So our our migration is broken down into a 19 area of steps And in reality, it's it's a state machine behind the scenes. So what you can actually do is write arbitrary logic into hooks during the migration So let's say like pre quiesce of the application. You'd be able to so we'll delegate out to the user It's hard to write like a fully generic uh migration tool that's going to satisfy everybody's needs Um, so as part of that we want to develop a hook mechanism so that you can hook into that process and do some custom work I've seen people use it for things like dns like changes during migration at a specific time That that might be an opportunity to use it. So maybe you want to hook in and do some intelligent Operations with like your you can take advantage of some data apis that you that are available to you to switch things on So that you can replicate it across clusters stuff like that So you can hook in and the way that it's implemented actually is out of the box It uses ansible, but actually it's just a we run an image And we'll provide it kind of a there's a there's like a Yeah, there's a there's a contract so you can rely on like you'll get some specific information about the migration that's going on And it's just using an entry point. So you it doesn't actually have to be ansible So you can provide us a playbook. It'll we can do ansible or um, you can provide us a uh Just an entry point and then it can be anything that you want so you can execute some arbitrary logic So that means maybe you're communicating with external systems or something like that to trigger like some type of replication during the migration There isn't that like ongoing data sync post migration But there's a lot of investigation that's going on we have folks in some of the like data the SIG groups that are looking at data movers and Volume replicators stuff like that that are that are lower level kubernetes concepts that I think seek to To achieve similar ideas where you're sort of like propagating data across and changes across multiple clusters in pvs and There there's also like I think some of the storage providers will actually provide that as well. I think like Um, we have a question. It's actually outside of yeah outside of kubernetes in general. It'll it's able to um Propagate data across so that like to cluster like storage leaders Storage layers will be synced and look similar So speaking of like all right storage layers and things like that If we're using something like open data foundations formerly known as open container storage or open shift container storage um How do we migrate how does crane migrate those pvcs? Is this still using velero at that point or is there something that we're leaning on in the ods phase? um We have a few different mechanisms There's i'm picturing a slide that we have that's floating around that's kind of like a pyramid There are a bunch of options that are available to you and like you're going to choose one depending on What makes the most sense for like the storage that you're using so There's two mech in two ways that you can actually move your data I'm struggling. I don't want to call it move because that's one of the ways is a move versus a copy so I think something like 70 of of people are using actually just nfs um So with nfs that's interesting because like if you've got a ton of data like a terabyte volume Copying that data is going to take a very long time obviously. So uh One of the better approaches there is like, oh, I've got shared storage and both of my clusters can see the shared storage Why don't we just pick up one of those volumes and plug it in into the destination? So that's called a move and that's going to be your fastest way to Get your data and make it accessible in the target if you're using something like nfs Of course, there are caveats there where that is the data. That's the golden source. So you're not cloning it Which means you need to be careful And there are there are implications for rollback and stuff like that when you're actually picking up your your data Of course backup all like healthy backup practices apply here. Um, so And then separately there's a layer below that where Depending on your storage provider. Let's say you're using amazon Uh, they're I'm blanking on the name of their storage. Yeah, ebs Whatever exactly they have a few so most of them have snapshot apis csi is one of the formats that Everybody's talking about and crane 2 is interested in so we can actually create a snapshot and recover from a snapshot on target side So that that operates at a different level than the file system And then one layer down from that is actually like a file system level copy So we had a couple ways of doing that um in the initial versions of of mtc we had We used a project called rustic indirectly, which is what valero actually uses. Yeah, so Um, it would copy data into the replication repository and out of it. Um, I understand that it was similar to an rsync mechanism I think it had, um Incremental backups Functionality we had some limitations with that we ran into some limitations And so we actually optimized that with some of our recent mtc releases where we offer a direct data migration So now we offer direct volume migration and direct image migration Which means you can skip that intermediary replication repository and we'll actually rsync the data using like an s tunnel mechanism Directly into your target cluster. So in theory that should cut your migration time in half for that And that's something that that we developed in house. So that's our solution And actually I have an interesting story about that too as of our our most recent release where there's This is moving towards the crane 2 world world We we kind of learned a lot and and and have codified that knowledge into a library called crane lib and so We're moving some of that logic out of mtc into reusable Libraries and then we're bringing that we're refactoring it and bringing it back into mtc So mtc now depends on crane lib. The interesting part about that is now crane 2's cli Is able to actually use the same mechanism and so we're developing all of that in in one core place There's also another project known as scribe. That's that's connected to red hat and that we're interested in Contributing to because it's also solving a lot of similar problems And of course, this is like kind of the open source way. Let's combine forces and start to do our development upstream So we've moved a lot of that that knowledge and logic into into crane lib And we're interested in contributing that into scribe as well Nice cool So lots of work Is what I hear lots of work is considered, you know being considered around state here Good good good good. Um And thank you. So I had a question that uh, if I got Far away back, you know, you were talking about crane 2 and some of the design goals and and things that you were looking at doing there and You know, if you think about crane 1 It's pretty much Sort of seen as a one-shot deal, right? And it's let's get you through that migration. I'm simplifying but with crane 2 it sounded like you're also guiding The application owner towards some practices that will be better in the future Did I understand that correctly? Absolutely. Um, that's the goal, right is uh, let's um while you're doing this mic it's it's almost like a um We try to avoid saying this but it's almost like an anti pattern to just pick up something and put it into the destination, right? Because you're You're carrying with you all of that cruft and and that legacy Just, you know weight that's associated with this like environment specific Application that's just not truly cloud native in the sense that it's like fluid and portable. So Um, we have the opinion that you know get ops is sort of the the solution towards that and and so that's one of the goals with crane 2 is to also help you adopt and help customers adopt the best practices because um, there's a lot of people who are not doing that. Um, and so when you when you're using those best practices, you're sort of it unlocks a lot of Things that are become available to you like hybrid cloud and all the benefits that are associated with it So that's one of the goals is to help people migrate to like not only migrate their application Which is their core interest but also get you like leave you in a better place than when you started Right. Yeah. Well, so, you know to make a metaphor of it. I think about moving one's home and you know what If you are have a bunch of cruft in your attic That first move that you make you'll go. Wow. I don't ever want to do that again And then you discover that there's other things along the way that you think well, okay If only somebody had told me That you know getting a storage unit, you know that I could that I could take the stuff and stage it there before Actually moving it to the home that would not be ready or you know until the the moving day You know, there's that's kind of what I'm hearing about crane 2 is that it does start guiding some practices Towards what we'll you know from from from the you know from experience, right? Is here's what has been a challenge in the past these are the hurdles that we have encountered Let's Just kind of set you up so that the next time you're having to do a migration You are going to have a much more painless one. You know, is that about right? Yes, that's it. That's I really I love that metaphor because it's actually really apt like When you're when you're doing it, you know, you you sort of like acquire all of this junk and then like And that's your source application But when you do that move it's an opportunity to kind of get rid of all of that make yourself a lot more Fluid and able to move again in the future without all of that burden So that's a that's a great. Um, that's a great analogy and then on top of that, um, the the It's almost like You you remove the necessity for migrations at all because you've moved yourself into an environment where you are portable enough where It's like I sort of it reminds me of when get started to become popular It was like for a long time branching was a very expensive operation It was not something that was to be taken lightly Get just said hey, why just branch early and often um, and it sort of changed a lot of those patterns like and and we're better for it and so, um It's almost like moving into this get ops world. It's just like I'm Uh redeploying into different environments and hybrid cloud. That's just a matter of course that that's a normal day to day thing And so you you remove the problem entirely Because you're fluid enough and you're designed to take advantage of of hybrid cloud Yeah, well, so I think that's the difference is the difference between being cloud native versus hybrid cloud native In a hybrid cloud native world The notion of migration starts to actually disappear a little bit because you've already built things in a way that You're not tackling it as a migration It simply just comes with the way you actually are detected the the app in the first place Yeah, you're you're basically just instantiating new apps Yeah um, so so I do have a question regarding that, uh, you know the notion of hybrid cloud approach so one of the the the things I liked with the with the two vert and open shield virtualization is the ability to have a hybrid application That is composed of containers and of virtual images, uh, you know within the same app so I was wondering What the approach would be or you know, what are the plans for that type of applications? So say I have a containerized app that talks to a vm that leaves outside of Of communities for for the moment, but I do want to migrate them so As of today, I understand that I have to use the two separate tools like I have to use mtc for the container part and I have to use mtv for the vm And at some point they're going to land. Maybe they can land in the same cluster So I have two questions. So the first one Would the correct way of doing it today like first migrating the vm from outside Kubernetes to the source cluster within the same like namespace of the application or something And then migrate the two together with mtc. So the question then is is mtc able to understand a CRD like virtual machine or is it just like CRDs that Kubernetes understands by default So that's the first part of the question and the second one is like if we think a little ahead Are there plans to have like an umbrella UI that allows you to combine both like I have Container coming from this and I have a vm coming from this and they do migrate as a single entity In a target namespace or something like that. Although they both come from two different tools Yeah, I think in that scenario, I actually have not had any I don't know of anybody who's actually tried to use mtc and mtv which is the container and the Tools in conjunction with one another crane and forklift I haven't seen it done and I haven't seen actually really any I haven't had interest come across my desk so to speak but I think There's no that doesn't mean that they can't be used in conjunction with one another mtc does understand crd's and actually it will Crawl and discover like cr's instances of the crd's and and actually migrate the crd's themselves Which is interesting because crd's are actually cluster scoped And it does handle some cluster scoped Resources when it recognizes that there's a dependency that's necessary Like the operator and it can go back to the operator to have the crd's. Yeah Operated workloads are actually an interesting problem that mtc in some cases has difficulty with because there's sort of an interesting This is a bit of a side tangent, but like chicken and migrate the Do you migrate the cr that the operator is actually operating on and delegate to the operator to reconstruct them the Application on the target side. That's actually probably the correct way to do it and then helm is another whole thing so I can get into that but to come back to your your question I would recommend in that scenario. I think crane 2 in conjunction with mtd would be an interesting option for you crane 2 is particularly Uh, I didn't I sort of touched on this a little bit But like we we built crane 2 to be a collection of tools and we like to call it like just kind of it's the classic unix philosophy full box 100 it's It's it's a bunch of like really small sharp tools that do one thing well that uh that can be composed to do Larger more interesting things and like as it turns out like they figured that out a while ago And it continues to be like a really good solution And so that's that's kind of the way that we're building that and we're intending on building An experience kind of around that where you can build these like pipelines that are executing those tools But like under the covers and we found this to be extremely valuable during some of our like more difficult production migrations We need those like those failure loops to be as tight as possible so that you're not waiting on timeouts You can't have this like monolithic migration. You need to be able to verify like the steps that are happening And and when you need to go backwards because something Breached your expectations. You need to be able to go backwards like like one little step not not back to the beginning of the migration So we've taken a lot of those Learn it like lessons learned and incorporated them into crane tool crane 2 So as a result because it's this collection of tools it's very Very flexible for its use so you can kind of pick and choose what's useful for you For example, like crane has a the crane cli has a command that says export If all you care about is just discovering and exporting everything that's inside of one namespace It will do that for you and it will output that then it has a transform command that will allow you to layer on those changes that jafar mentioned To strip things out really actually do arbitrary things because it runs it runs using a plugin mechanism And so you can use it you can choose not to use it And so you'd be able to use crane 2 and maybe the cli you can build solutions on top of it That you can then integrate with something like mtv Which is by the way also architected in exactly the same way that mtc is so that it has an api surface that is represented by a series of CRs and crd's So you can automate that you could build something that's calling into crane 2 and then maybe it's creating crs and and watching status of those crs To wait for mtv to migrate vm's and then maybe you go do something else. Maybe you move around to load balance or something like that so That's kind of our philosophy is that we're trying to build these like core building blocks And then we're going to offer like an opinionated solution that uses those We'll take sort of a Position on how we think you should use those but of course you can get into the toolbox and start building your own solutions easily as well Nice, that's cool. But that'd be to say that mtc2 would be the opinionated usage of all the tools in crane 2 Yeah, most likely And we're we're trying to to integrate with things like tecton and argo Um to build these pipelines that really like if you're familiar with tecton, it's going to look like a pipeline I slogged it for us and as you saw that said that yeah And and so maybe those those steps are running these crane commands and that you're you've got validations That are going to stop a pipeline in its tracks if if you've validated something and it's filled that validation so So I have another question. Oh my god so many questions. Yeah, so but that's that's a that's a lighter one Yeah, so so we'll mtd2 bill is just getting hired jafar. I mean, yeah, you realize you're going to be invoiced for this, right? Yeah, but this one this one is like is lighter. So will mtv2 be named vh1? I'll need to run that by uh I'll have to bring that one up. Um, that's not the first time we've people really like the mtv name Awesome. All right. That's what it beautiful. Thank you. Uh, let's get through some uh point sharing over here real quick I've realized we're we're kind of already diverging from our normal Schedule here to an extent. Uh, so this is the level up hour. You've been watching that for a little while. Let me know if you can't see my screen So about us so randy is here, but randy does not have A twitter so that's fine So I made him a shortcode just for uh, apparently it's not sharing the right screen. That's fun Uh, new share Share desktop 2 No, I was serious when I said that Wow zoom is just being fun today. Thanks zoom. All right. There's points. Um, enough inception so Randy has linked in Just read that ec slash triple r dash linked in they'll take you to randy's profile. Uh, scott mc bryan who I said is off today You can follow him on twitter Jafar is on twitter as well I am on twitter, uh, and we have our world famous discord where you can check in with us at any given moment in time Yeah, small peer pressure No big deal. Um, but If you are curious, uh show notes from last time I put those together. Uh, they are up there on the show notes page as well as Show notes from this episode are already done Uh, and you can find those in the level up our github repo, which I just dropped in chat But what you all really want to know is this sweet sweet internet points Who's got what and what's got who so if you want to participate in what we call sweet sweet internet points All you have to do is go to the form that's shown on on screen here and I'm going to drop links for everybody in chat Um, you too can get your sweet sweet internet points, which at this point Have no real value other than bragging rights but We hope to have some some real value to them in the near future Uh, we keep saying that and you know, maybe that'll happen. Uh, but the rend of is Clearly in the lead with and I'll hack them right behind them And then no affriction followed by debt conan kudo bacon fork and joe fuzz So if you want to see your name on this list Feel free to check the archive of shows and go back to get other codes to drop in Uh, to the form to get you more points and I think that's the last slide. Yes, it is so Any other questions from the audience any other questions from Jafar or randy or anybody at this point Erica, did you feel like you covered everything? Hopefully I didn't talk too much. Oh, this this was great and like that's like the I know this show is in good hands with jafar randy and scott because jafar can just ask questions that are related to everything um, so Yeah, uh, please check out the conveyor community itself length drop there and then When in doubt, uh, please like subscribe and share as always and uh Coming up next on the channel assuming there's no more questions or anything from the audience I'll give you a few seconds to chime in Coming up next on the channel today is ask an open shift admin with the one and only uh, andrew selivan But we also have a special guest my friend john spinks will be coming on We'll be talking about red hat insights for open shift. This is like having A a highly technical person sitting in your data center saying hey something's broken over here But you don't know that because this config mismatch is a very unusual kind of thing so It'll be interesting to see how that works in open shift I know we've talked about it on the channel as far as rel goes But in open shift clusters, that'll be a very interesting topic of discussion So Eric do you feel like you got everything off your chest? You needed to mtc mtv nta vh1 bet whatever yeah All that's gonna share one other thing. Oh, yeah, if you are interested in mtc, okay red hat does offer a course On Migrating so if you want to find that if this is all something that speaks to you and you need to know about this Please look up do 326. That's the course code Or our migrating applications course that we offer at red hat And you've got me googling now, so hang on. Let me find the link for that for everybody do 326 is the number Wow, that is actually used in a lot of different things Uh Okay, randy if you have a link to that. Yeah, I gotta include red hat with it. Otherwise, you might pull up Yeah, digital ocean. Who knows? Uh There we go red hat migration lab Ta-da Lincoln chat And there you go So, yeah, you can get uh you can get your training on For all the only other thing i'll mention too is that um conveyor is an organization is actually on the kubernetes slack instance So if you want to actually reach like all the engineers are there that are working on this you can talk to us every day We're in there. So that's um the kubernetes slack and then it's like just pound conveyor. You'll find us. Yeah Let me uh find the slack.kubernetes.io link real quick to drop in chat And you can find them on the conveyor channel that is conveyor with a k And they'll be happy to help you out I am also in that slack so feel free to ping me with anything as always you can reach me on twitter at kreshort Or if you have any questions or suggestions for future shows feel free to email me short at redhat.com And uh, yeah, thanks everybody We really appreciate y'all tuning in and thank you randy jafar And eric thank you chris. Thank you eric. Thank you. Thank you And see you soon. Yeah, we'll see you soon and uh check us out here at 11 a.m. Eastern 1500 utc for ask an open shift admin that is an office hour You don't have to just tune in for only the topic you can ask whatever questions you want during that office hour so feel free to jump on that and uh We'll answer all your questions for you. If not, we'll find somebody that can So thank y'all Thanks. Bye. Bye. Thank you