 I'm Eric, I'm the team lead. For now, the team lead of Converturell, which is managing a protocol Converturell. It's a team of seven, something like that, and I'm, yeah, God, this is frustrating. There we go, all right. So, I'm gonna be talking about how you can convert a system, right? And it's gonna be primarily about CentOS. So, most of the systems that are similar to CentOS are based off of REL, so you have Alma, you have Rocky, you have Oracle Linux, CentOS as well. CentOS Stream is actually upstream, so it doesn't really count, but that's what REL is based off of. So that's why it makes it quite easy to convert. So you have relatively simple process, everything's basically the same, but when it comes to something like Debian, it's a lot more difficult because then you have to deal with a completely different architecture and ground, right? Because it's not similar in that way. But converting is still difficult in a way. The actual process is easy, but there's always gonna be some edge cases and other difficulties, so there's a few tools that people develop over time. I don't think Converturell was the first one. I think that was actually CentOS to Oracle Linux, if I'm not correct, but yeah. There's a few tools, though. There's Elevate, which allows you to upgrade systems, so like from CentOS going to Alma and some of that. They also have an Alma Linux deploy, which is actually converting from CentOS or CentOS Stream over to Alma. For Oracle, there's CentOS to Oracle, and that's, you know, they're a bit simpler. They have Converturell, which is a relatively big product, which converts you to RHEL, and then for Rocky, I have Migrate to Rocky. So there's a few tools available. Anyway, so I'm from the North Pole. I come from Sweden. I'm currently making some kombucha. I'm trying to learn piano. I tried to do some mountain biking when I was living here in Brno, but now I'm just, it's very flat where I am, so I have to go quite a bit to get somewhere. I've been a developer for five years. Started like a front-end kind of environment where I was mostly working by myself, but I learned a lot, and then I think since two, three years ago, I started with Python and actually developing Converturell. Yeah, and being a team leader, it's something I'm gonna transition over because I can't really be a manager as well as being a team leader, but yeah. And I'm also a crazy cat lady because I have some cats, right? There's Isolde on the left side, and then you have Lisa on the right, both Mankun's, they're gonna be quite big. Oh, and Mimi's a lose as well. So what are the differences between the tools? Well, first of converting like a system, why would you even want to do that, right? Like it's, you can just redeploy. And in most cases, I would say redeploy is probably the better option because when you redeploy, you get a free system, you're not gonna have any like weird things that you didn't think of before. Like when you convert a system, like you're gonna have something, even if you don't know what, you're gonna have something that is steal from the old system. So if you convert from centers to Alma, there's gonna be some centers parts, right? But you're not gonna have any redeployments when you're converting place. There's gonna be all the corporations and files. And then when you have converted over, you know what you haven't converted with the packages at least. So there's some security there. And I would say most of the time, like it still makes sense to redeploy, but like it's, if you can't redeploy, then converting in place is the way to go. And for redeployments, it's probably the better option, right? You have better like safety margin if you can. Like everything, you can make sure everything is working and can have stage migration. So like you can keep the old systems running when you redeploy the new ones and then after a while just shut it off, right? And that's usually what bigger companies do, right? I think, sorry, I think Facebook did it where they went from route to center stream. They also had like gradual staging migrations. So the differences, I don't know, maybe it's gonna be a bit heathen, but yeah, I think converter all folks is a bit more on like stability in testing. So a lot of the tools have a lot of lines of code, but I think the converter all is probably the biggest project out there so far. If we exclude elevate, which is kind of leap, which is rarely upgrades, but I'll get to that soon. And then elevate, which is a play on enterprise Linux, which is why they have EL capitalized. They kind of treat center seven as like an upgrade path. So they disregard that it sent us all together and they say, yeah, well, it's similar. So we can just upgrade it to Alma, to Rocky, to center stream, anything like that. And then Alma Linux deploy centers to Oracle and migrate to Rocky. They're basically just a bash script. So they're very, they're automating the tedious parts, but there's gonna be some manual steps to fix things. So the general execution flow of like converting a system is kind of easy, right? I mentioned that before. You just have to clear the cache. You have to update the factors to the latest version to make sure that you're on the latest at least. You change the repository to the new version, or a new distro by keeping the same major version. So if you're on center seven, and you want to go to rel seven, then keep the same major version. You don't go to center seven to rel eight, because it's more complexity. So it's more difficult to know what's going on. And then you just replace the packages with their new distro repo counterparts, right? So if you have Apache on centers repos, you just reinstalled that. So it's now pointing to, it is now the package from rel. And that's it. That's the whole process of converting. Talk over, right? But there's always edge cases. Like UFI is one of them. We're like on centers. There are some edge cases that doesn't really work when you just try to install a new kernel and so on. You have different custom scenarios where like if you're doing something with your company, there might be some things there that you're gonna have to work around. Bad packages, which is more of like, well, for example, CentOS logos is one of them, right? That's very CentOS specific. So you want to like have to try and mitigate that somehow, right? And then package depends on mismatches, et cetera. The package between mismatches probably won the biggest issue, I would say. Like there will be something and a lot of the tools try to just like, oh, well, either we just ignore it if it doesn't work or we try to mitigate it somehow or just like say fix it before you convert, depending on the tool. So yeah, there's endless configurations, right? Every system is gonna be unique. There's gonna be packages that don't work, right? For example, model the app is one of the packages that you can see on the right side. That just does not work when you try to convert it from CentOS to rel because there's some dependency things that you can't really resolve. So those things we're gonna have to ignore. And then it's in general easier to convert to Alma and Rocky from CentOS than for example, from CentOS to rel because they are, I mean, there's a successor to CentOS. So they have some like, they're pretty much similar. I would say it's almost the same system. So it just makes it easier, but rel actually packages their own systems or packages and then it makes it a bit easier to convert to Alma and Rocky. And then CentOS Stream, that's like the big scary one because CentOS Stream, since it's an upstream from rel, there's gonna be downgrades when you go from CentOS Stream to rel or something else because you have so many, you have so many different downgrades. So either you're just preset in place for a long while which is not secure and not something you should do. And then you just hope for the best, but it's still a scary subject to like try to do it yourself. And the reason for that is like that you're gonna have downgrades and they're not usually tested. So package maintainers, they usually test that upgrading a package is fine, but when it comes to downgrading, they don't usually do that, which in general, it's not gonna cause an issue, but if let's say package updates on configuration file, like a format or something, then they can't really go back to that previous format. So let's delve into some more facts about how to actually do it. Yeah, as I mentioned, it's all based off of rel, which is in turn based off, well, it's based off of CentOS Stream, which is based on Fedora, right? But almost the same thing. Packages are mostly fine when upgrading since it's, you know, it's tested, you have like some security there. I don't think it's gonna be an issue if you go from CentOS to rel, CentOS to Alma and have to upgrade them because the version difference, it kind of works out. But the downgrades, we just don't know. We can never tell. It's like, okay, we have to try, see what breaks. If nothing breaks, it's probably fine. And as I mentioned, they don't usually test that downgrades work, but CentOS Stream is still something that you can convert. There are some tools that allow it. Unbounded deploy has an option for it, where if you put in dash dash downgrade, I think, then you will be able to convert CentOS Stream. And then convert to rel is trying to do it as well, but we, since we're trying to do resilience and make sure everything is going to get smoothly, it's still, it's, we have a semi-working draft that's been up for quite a while. But it's such a big topic that we have not really tackled like how we should actually do it yet. And then configuration migrations, they will happen. They can't really be reversed. Like when you have a breaking change for a configuration file, yeah. You're probably gonna have to fix it manually. At least that's what I believe, right? Or just talk to the package maintainer, see if they actually have some script that you can just reverse engineer, maybe. So bootloader is another thing that's a bit edge casey. So bootloader, if you don't know, you just have a bunch of options. It has an order to go through for the boot. And this is from my machine. So I have in running a Fedora silver, blue, or well, Kinoite, which is KD. For some reason I see rail there, but I don't know why, because it's a Fedora machine. But it's booting from Fedora. If you're trying to convert this to something else, you would change the boot order, you would just add a new entry and it would show a different file. And that may cause some issues in some system, as I mentioned, right? Secureboot is another thing that will not work when you convert because Secureboot wants things to stay as they are, you should not modify things, right? So you're gonna have to turn that off when you're converting. Then for CentOS 7, there will be some edge cases for using UEFI because when we're creating a new system, converting to rail, for example, when we create a new boot order, it's not really gonna work because there's gonna be a difference with the actual, I'm gonna let them go back. In the actual file pass, there's gonna be an issue because we're not gonna be pointing to the same to the correct files. We have to fix it manually and create our own entry. And then you may be naive in thinking, okay, well, I can use LSBLK to get all the block devices and then I see the minor thing there and then it's gonna be the minor number that they use so I can just use that and do it automated. Which we thought worked until we got some customer complaints that it doesn't work and as you can see in the lower picture, you can see that it says SDI 1 which is the first partition of SDI but the minor version or the minor block number is 129 which is not the partition number. So that's getting that was actually quite difficult. In the end, we managed to find this command here BLKID and we just get the part entry number from there and that actually worked. And another thing that I don't think any of our team realized that boot number entries they're actually hexadecimal. So you can get something that says boot 001A and that's just gonna break everything you just automate this like assuming it's all numbers because why would you need like 9999 configurations, right? I mean it's, I don't think I've ever seen that so but they use hexadecimal so that's why I have to like think that's an edge case. And then with the actual tools, I mean there's gonna be more variants, right? You have the normal AMD Intel X86 which is the like easy part. You have ARCH which is getting, oh I'm sorry, which is getting up into like popularity because it's usually a bit better than X86 in a lot of regards. You have IBM Power Systems, you have different major and minor versions and then you have different life cycles and non-minor versions and major versions, right? So you have like long-term support or extended upgrade support, I think that was the answer or something like that. And that means even more packages to test. So if you want to like make sure that things are good then you have to go through, the best way to test is just convert it, right? You can just convert like a test system or like spin up mirror and try to convert that and see if anything breaks but there will be some package specific or architecture specific versions for each package unless they're using like no ARCH which works and everything. Like if they have a Python package then as long as the system can have Python, it should work so it's just no ARCH or no ARCH maybe. And then kernel packages. I haven't looked into this but they assume that kernel package should be different on like ARM versus X86. So that's another thing because packages from kernel is gonna be, I mean, that's the most important part, right? Because when you're converting you want to make sure that the kernel is actually from the new repository, the new distribution. So that's one of the things. And then if we want to make it more complicated we have cloud conversions, right? So for subscription-based services like Oracle Linux or REL they have like two options. You have bring your own subscription, B-A-Y-O-S and pay G, pay S-E-Go. And B-A-Y-O-S is like images that the cloud providers provide and then pay S-E-Go is like, well you don't pay for the subscription right away, you pay for the use end. So it's like add an on top of the hourly rate that machine server like the minute rate. B-Y-O-S is similar, like it's similar to normal conversion. It's gonna be quite easy. Everything should work as normal. There might be some subscription issues but in general it shouldn't be. And then pay G, that one is a bit more of a difficult thing because actual cloud providers they hook the billing to the actual machine. So when you're converting that from Oracle over to REL for example, you're gonna get double bills. So you're gonna get the bill from Oracle and you're gonna get a bill from Red Hat, which it's not ideal and the cloud providers are trying to solve that. I know AWS, I think AWS is the one. They're trying to fix this somehow. And then there can also be slight differences in ISOs provided by the cloud providers. I think that Google, they provide their own centers Linux builds because they want to pack their own, they want to introduce something. And then AWS, they actually use secured hardened images from a third party that's partnered with AWS. And without knowing the difference between these and the actual images from the official repositories, it's difficult to know whether it would pass or not, right? So you have to test that as well. And then you have normally images from non-enterprise Linux like Alma, Rocky, so on, like all of those should be available and you can just convert and we kind of treat them as a normal conversion because that's usually easier part, but the enterprise ones are more difficult. Oh, and actually testing the conversions. I mean, this is a big topic. Every OS, every architecture, every version is different. So it's going to be huge test suite of all kinds of different environments. We want to stay as close to the environment of the tool that you're trying to run. So if you're converting center seven, that's using some IDM server. You kind of want to reproduce that in a test environment first and then see if it converts. Ideally, you just want to like spin up a backup and then see, okay, can I convert? If not, then something needs to be changed, right? Containers, they are actually great for converting. I know that Elevate, Elevate is actually going into some boot management options so container doesn't really work there because they spin up an innate RAM FS and kind of boot entry where it finishes the conversion in and that doesn't really work with containers. Correct me if I'm wrong, Peter, but I assume you don't really run it in containers, all right? So, but most of the time, you can use a unit test with containers. That's like an easy part. But it's not a good picture. You can just test the actual unit, but it's not going to test like the whole conversion. So you have to actually use VMs or some kind of parameter machine to try to convert. And I know that Converterell extensively uses it. So we have a whole, we're using testing front, it's a whole array of machines and then we also have our own VMs. We try to use Vagrant if you want something simple and we can spin up pretty much any of the systems that Converterell can support. And then you just try to install as many packages as you can. If possible, install all of the packages that are from the distribution from the repositories because then you will know if there are any packages that are not going to be converted if there's any issues and so on. And ideally just clone the system because otherwise it's just going to be, it's not going to be the complete mirror of what you're actually trying to convert. And make sure that you have backups. I didn't put it in here, but backups are like important because if there's an issue and you're into production and it's like, oh, well, my production system is down, I didn't do a backup. Shit, that is kind of ruined, right? And then you have to try and fix the actual machine that was restoring from backup and so on. So, God, I was so fast. But yeah, so, any questions? Because I feel like there's going to be quite a few when it comes to converting. Yeah, I don't know. Right, so the question is when you're converting, Cento is streamed to row and you have to like freeze all the packages. Can you just allow security patches to go through? That's basically, yeah. I'm, I don't think so. I think there's still going to be some issues with the downgrades. I mean, it's going to make the system more secure, right? Because we need some security, but when you're actually converting the system over from something, there will be downgrades. And if it's a security-related package, then you will have to solve it. But it depends on how much resources you have. Like if you have enough resources that you investigate, okay, this package is going to be an issue, how can we mitigate that, how can we fix it and you just create some custom scripts to like fix that? It's fine, but there might be a lot and that's not ideal. So it's, you have to try and see. Yeah, yeah, any questions? I think that's it. You can always reach me afterwards. I'm available on Mastodon as well as on Twitter. It's just SpyTech, so you can find it there. And I also go by Agustos on that. Yeah, thank you.