 All right. I just started recording. My name is Arun. I work for Yahoo. I've been at Yahoo for more than, what, four years now. I also contribute to Federal Project. Not that active anymore, but then I still send bug reports to these people because I mostly work on themselves these days at Yahoo. I also work on the OpenStack team now. I do a little bit of contributions to OpenStack Project as well. This talk is about how to customize Anaconda, like add your own custom patches and how to rebuild Anaconda for Fedora. I have done a couple of patches for Anaconda because one thing is like, okay, it is challenging to work on Anaconda because the first thing you search on the internet for Anaconda, you get this one. So it's really a distracting experience. Like you keep clicking this and then it never goes away. I don't know whether the Anaconda does here experience the same thing. But yeah, it happens. So why we should modify Anaconda? I mean, a lot of people never even use Anaconda these days. They just use FedUp to upgrade their software and distribution. So why? So if you're building remixes, so you might want to replace the name of the installer with your own branding and own name. And if you're adding new packages to Anaconda runtime, so why we need to do this? So at Yahoo, so we have like several hundreds and thousands of servers. And most of the hardware comes with like four hard drives. And most of them comes with more than four hard drives. Like you have multiple hard drives. The moment you have multiple hard drives, it comes with the RAID controller card. And the RAID controller kind of comes with its own radio treatise. So the person, the property, the project that wants to deploy to this hardware, they want the hardware right configured. So only way to do with Anaconda right now is to either include that hardware utility in the Anaconda runtime so that they can run their whatever prescripts, and configure the hardware right in the way they want and then install the operating system. So this is one of the things that we can, or you can include your custom utility in the Anaconda runtime where you want to do something in your post or prescripts. So I don't know why these people can answer. Anaconda never added hardware right support. I mean, yeah, I mean, so maybe I should file a bug. Yeah, so hardware right is really important because for us, a person who works in a big environment because on a day we have like a normal day, it's 200 machines coming up and going down. On a like a really active day, you have like thousands of machines coming up and re-imaging. So like new soft new, you know, product launch or something. So you have 1000 machines. So every 1000 machines, some person in the data center goes there manually raise a ticket and then somebody goes there and manually contrives this rate with a special RAM disk. And it's really, I mean, it's really time consuming. Like a week in advance, the particular person has to say, okay, I want to write a file on this, you know, this, and then they go there and manually modify it. This is like really, really bad problem for us because it makes like, you need to prepare a card, you know, you cannot just go and deploy software without configuring, right? So we needed some kind of, you know, they can the property themselves go and, you know, modify the kickstart file and do that. If there is a way to do instead of script running script, if there is a way for in an account that to do the right configurations with like, okay, hardware rate, put this on rate five and 10, that'd be very useful for a company like Yahoo where we have several thousand machines. There is one. Another one is adding new features to Anaconda. So you might want to add, so company like Yahoo, sometimes we always try to point out bugs and submit to Red Hat and folks to Anaconda team, but sometimes it takes a loop, you know, it takes a while to further bug fix to come in, but we cannot wait around for that. So we will time to time add patches to Anaconda and, you know, push it out. So this is where you add your own patches so that you'll be like ahead of what comes from Red Hat or what comes from Fedora. Another thing, so one more thing, this is a little bit Yahoo, we noticed is that when you are, you know, building 1,800 machines, most of the machines, you know, like the failure rate is like, the failure team style is like, let's say one percent, still it's like, what, one percent of 1,800 is 10 to 20 machines, 18 machines, yeah, 18 machines, thanks. So the person who's deploying the operator, who's deploying, has to actually go into console of this machine. We do not use Graphical User Interface, it's all text modes. So the person has to go into Graphical User Interface, look into the console, into the tool server, and figure out what the heck happened there, whether it's a bad hard drive or something blown up. It's really, again, a time consuming job. So we are looking for something like, okay, the Anaconda when it starts installing, so it starts a service where you query from a centralized dashboard where say, okay, why it failed, what's the status of the current installation, and some kind of a status API. So there are other installers that do that right now. I mean, we are trying to add this feature to Anaconda, but we might, we might, I don't know, it's a tricky thing, we might implement it or we might use the other installer. So there is an installer called IPA, which does that. So they have an API where it keeps telling, okay, what's happening in the machine, whether it's, you know, formatting files, file system or a particular hard drive went down. So that kind of status messages will be very useful for data center use cases in case of Anaconda. I don't know what's he thinking. I think it's the wrong model. What do you think is the right model? Did you see yourself with the Nebula GIFT of what they uploaded on their private cloud stuff? So the way it works is you've got one, like you've got a rack of systems, right? It's a brand new rack. You employ one system manually to pick it away, and it's a pixie server, basically, and you tell your hardware vendor to be able to configure the pixie for the first time. Every time you rack something, it sees a pixie doing it, immediately sends an agent up to run the machine. The agent runs IPMI tool, sets the IPMI password, and now everything happens from the top on IPMI right now. And so it can ask, what, it's never appreciate this status, this status, what's that status, and it's running from the reverse crossing. So basically they have a custom agent running. Yeah, okay. And then we can configure it to the loader to do whatever and sort of install it into whatever we get after that. Which is a stuff above where Anaphonda is involved, right? Yeah, I mean, something like installation logs, like how do we get that? It's completely automated process. Rack, machine, and five minutes later it's running as well. Okay, so these are some of the things that, I mean, we can discuss about it, like what, how we can do something like monitoring of the installation status of all these things. Okay, so what are the tools people use normally to build Anaconda? So there are three tools we're going to use, we're going to use Mark. So Mark is, you know, is used for RPM building. So if you want to build on a, if you have a Fedora machine, if you want to build for L6, you will need a Mark Sherwood to build Anaconda on that. So we'll set up Mark Sherwood and run Anaconda in there. And then in case of, REL6 used to use this tool called building style, he might know a lot about it. It's a shell script, oh gosh. So you have to modify shell script if you want to add another package, anything to this runtime installer, you have to modify shell scripts. So it's a totally building style. We'll, we'll look briefly about it. We'll mostly focus on something a low-rex, for a sense. It's so much better. Yes, low-rex is nicer version of building style. It's, I think REL6 uses building style, REL7 uses low-rex. And Fedora, if you're today, want to build images for Fedora, you'll be using low-rex. Okay, so this is important. So I'll be like switching between terminals. It's not a demo. I already built a stage to SquashFS outside because with this internet, I cannot build anything right now. So, so this is like a mock configuration file. So we need additional settings. I mean, correct me if you, I want to, I still don't understand why there's nothing wrong. Okay, I'll set it down. So basically, we need to enable bind mounts to lose SquashFS images. And, and also some people want to lose they, so mark every time cleans the children. So they, what to do is they bind mount their Anaconda directories into the mark zero so that it doesn't clean anything. So, if you look at my, I have a custom mark configuration file. They have, you can see mark, flock. Yeah, so I get a new mark configuration file, flock 21. So because yahoo compose some yahoo stuff here. And the important point point is the line 10 to 10 to 13. So we have a bind mount enabled and we just buy more, buy more, better and previous and same. So we, we do not use the existing mark file. So we create another one so that when, when you're building RPMs, you don't really need all this. We also do some Jenkins magic for, you know, get the environment variable and precedent. This is something related to yahoo. All right, so, so what are the installer components? So once you build a, build an installer, there are three components in installer, like as usual the pixie kernel that you won't need. And one of the, one is called RANDisk. So RANDisk, Anaconda is a big installer. It's a two-stage installer. RANDisk has your breaker stuff that actually prepares the stage two, downloads the stage two, downloads the kickstart file stage one. And there is a stage two. Stage two is a splash of this image that is in the install tree. So it's a three-stage installer. So we'll be building all these three. We'll not be building but I'll be showing I have logs of the build slip. What happens? Okay, build install. So what's the problem? It's shell script. Let's look at build install. The problem is not that it's a shell script. What is that? The problem is not that it's a shell script. The problem is that it's a really bad shell script. Yeah, let's look at it, like, see how bad it is. I had to work on this to fix it. Yeah, I do. I wrote a lot of it. So. So, I was like, why do we do that? Matt and Jeremy have been. Uh-oh. What? You've done them last. Oh, Vim. Control save, yeah. Yeah. Skateboard, skateboard here. Now, demo or less? No, both. What's the command? That's not correct. Demos. Go into 13. So this is, this is the one comes with rel. Centaurs. Scripts. Oh, it's hard. Build install. Okay, so this is a build install shell script. Actually the main build install stuff is everything is in something else. Another shell script. Which is called up-instrued. So if you have to add a new package. So you have to modify this shell script. So, up-instrued, it's hard. Yes. Okay, I'm going to sit here. Yeah. So, this is the, so that shell script calls this one. And it's almost the same as, so if you look at this packages, these are the list of packages. That's fair. So if you want to add a new package into your Anaconda runtime, you edit this thing and add this. And make sure that you use the right build install to build this. Because I mean, the build install, there is a build install outside and there is a build install part of Anaconda. So, these are the set of packages that gets insert in the squashes and images and then this is the place where you edit it. All right. Yeah. That is the reason they created, I mean, I don't know what's the exact reason. Is this the reason, like, is the, every time you guys have to do a release, you have to modify this? Is that, what's the reason for Lorex? The reason for Lorex is that if you keep on reading a build install other than that, there's a terrifying code. Actually, it might not be an intro, but it does things like try to figure out which LDSOs you need to have by guessing and entering through LVD with the thing that makes it do, uh... Yeah. And then it does a depth sorting of it, a depth solving thing and tries to sort them to try to make sure we've got all the right things in it, but it doesn't actually break up the, the, the, the requirements loops. No. No. So, so I think it's, it's written by Willem Woods, right? Lorex. Um, you know my, uh, my name. Oh, okay. So, so it's called that, it's called the tree builder. So basically it is used to generate the, like your DVD tree in there. What do you see in a federal DVD? This is what the use is generated. So why build install is hard to customize? That's what I heard from the IRC in places. And Lorex is Python based. And it uses a system called template system. So everything is a template. So they have a template file. I think it's the, what, what's the template system? Is it Jin? That's Jinjab. It's something else. Um, um. I don't know the name. Yeah. It's just in here. Okay. Is, it uses a template system where it's the templates are customizable. And if you want to add a build step, you can add your own custom templates. So it's, so it's more modular if you want to customize it. Um, yeah. It's the. So let's look at the templates. Yeah. Use the share Lorex. Make up. Make up. Yeah. So, so if you look at these pvmpl files, these are the, um, templates that are used to customize the whole image. Um, so by default it's based on the actualizing first x86 template. Um, let's look at what it is. What's that? So all the stuff for x86 stuff. A bootloader and the runtime image, the squash of this image is created by, you can go through the core layer. Um, runtime. Install post install cleanup. So everything is a template. So if you look at here, this, the instead of like having a, uh, bash variable you have like, uh, yeah, it's kind of easy to, uh, you know, add, modify things here. So, so if you want to add a new package to on a color runtime, this is the place you add, or you add your own template and pass it on, uh, here. All right. I'm going to, I think this talk is going to be like, very, we'll complete, you know, you know, soon. Uh, okay. So the patches. So what patches do we add? So I have a small patch, like a welcome patch. You know, you change, you know, I think most of the distributions, if you want to do a remix, they do this. Like they modify the, um, modify the, you know, trademarks and all the names and stuff. So let's look at this patch. It's called the welcome patch. I actually applied this patch on a machine. So what, what this patches does is like, it just says, um, welcome to flock 15, from welcome to Fedora. And I think this, this, I don't know. I didn't get it working. I think it's only works on a live in live image. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. But, but when you're building with Lorex, you can say the product name. So it changes the product name. We'll at least see that in the demo. And, uh, so, so these are the, uh, two patches. I ordered, I have another patch. It's like a bigger patch. I can show you. So this patch is actually running, running in production at Yahoo. So April 2015. Um, okay. So this is a patch that adds top payload support for a well, six in seller. So the concept is that we have a lot of, we don't install into the lot. You know, we have, if you, if you are, if you came to talk like last year, at, uh, product. So we do a lot of tar balls. So tar balls are easy to like transfer and export that individual arguments. It just reduces time. So, uh, we currently have support. Thanks to our kind of team. They have added support for tar balls on some point one nonversal. Yeah. Some point one. It landed on some. So even before they added, we, we've been running that patch now. Well, six, we needed, we needed real, we have had a customer in seller and, uh, we wanted to move away from it, try different things. So Anaconda is one of them. So we added a patch to Anaconda, to add support for top payload. How it works is that you, uh, you specify during installation, you specify the current command line top payload equals the URL of the top ball. It automatically passes this top payload argument and downloads this top ball, creates partitions and extracts them onto the disk. So it's very simple like a, and, but it kind of, uh, we have to hack into the installer to divert to do this. Otherwise the installer will do something else. So I think this is some form of change. I did, I don't know why. Okay. I shouldn't have. Yeah. Oh, old codes. Like, uh, the old Anaconda code is lately a little bit hard to read. The new code is really good. I mean, uh, from looking at the old code and the new code, it's well-organized and obviously well-organized. The new code is well-organized. But I, I mean, working on OpenSec, you see like things like docs and unit tests, but I don't see anything, anything here so far. Do you guys have plans to add it? There's, there's some unit testing out of the test group. Okay. There's some stuff that they've been involved with. Yeah. Oh, really? Yeah. Okay. I'll talk to you guys. I mean, with that, I'm also going to send a picture of the key to the RunOpen group. Okay. So I'm going to ask the group, it's happened to touch the whole thing out of the unit tests, but... Yeah, so this, this after applying patch, what we, what I do is normally, uh, create a ISO image, and then boot the whole thing in a VMS like a test. Uh, if the VM works, we just push it to, uh, like a centralist or positive, and then it goes to production automatically. It's like a small dump test, but, um, I think the current team does more tests where they actually deploy the installer, use the installer on a bare-model machine, and do the tests. Um, so, so this flag actually comes from, uh, is sent by the code that process the, the kickstart files, sorry, the kernel command lines. So if you find the top payload flag, it calls the top payload back in. It's both for Fedora and Dell. And if you look at the top payload back in, um, so you just, what it does is it finds out the size of the, uh, top ball, see if it can write it to the temporary file system on the disk, and it downloads it, and, uh, uses a progress bar that's a progress bar. It's almost like equaling to the one that you guys have in live image payload, uh, except it's for L6, which is like the older installer, and it extracts the whole thing, um, onto the disk. So this part is little important. We use no ACLs because there's a bug in tar. If you use ACLs, it sets the default ACLs on all files. So we got a, this, we had to disable it because it's kind of a security vulnerability to set the default ACLs on all the files. Um, so we extract, uh, actual, uh, files, the top ball onto the disk using this command. And then are you, are you certain that we label or what? Um, no. Okay. Should we? I don't know. So we don't use, we do not have ACLinux enabled on any way. Yeah, so, but the ACLinux flag is just there, you know, in the future if we decide to do that. We should probably trigger it, relabel. Yeah. So there was two bugs open on, uh, against redact. So it's fixed on Fedora. That patch is not, yeah, the patch was not never put back to tar. So to, I think it's tended to, I think it would be released with 7.2 and 6.8. I don't know why 6.7. Um, but, um, let's see. Yep. So we do some sync and then, uh, we, you know, this, this all runs in a thread and this is the patch. So this patch is supplied against rel6. Um, okay, let's go here. Okay, this is the command actually, uh, to build the, um, stage two scratch ups image. Um, so what it does is this is a product with, I have a make find that does this. I mean, I have published everything on my page. Um, I'll give you the link later. You can go look at the patch and all they make for if you want to do it yourself. Uh, so Lorex command is the product name. So this is your new distro. Uh, and this is your, uh, version of your distro and the release. So this, yes, this is the report. So normally it points at the upstream report, which is Fedora. And this report is probably the updates and the final one in Anaconda. So you have built Anaconda. So I, what it is, it is like, I applied the patch to Anaconda. I built Anaconda. Now the Anaconda version has to be a little bit like you bump the release so that it's a little bit higher than the one that's available at, you know, the Red Admirers or Fedora. Yes, to get in the image and, uh, you point your, um, appoint this to the Anaconda. And finally you say it is final. This is a little bit important. Otherwise, Anaconda will keep telling people that, oh, you're running a pre-released version. Yeah. And another important thing, another thing I noticed is, uh, uh, if you're building and building this on a RL6, but you're building for a RL7 on Achirud, sometimes this, um, I think it's the Fedora image. I think if you're building Fedora on a RL6 machine, what happens is that, um, the Mac good stuff. So basically the HFS, uh, they don't have HFS drivers. So you probably need to say no Mac good, uh, in the low-res tamales. Somebody added a patch recently, I think like a month, two months ago I added a patch to disable no Mac good. It wasn't there. So I used to do just sit down. Yeah. We certainly never built it on RL7, but that would add between the two and so that it would be a Fedora feature before the RL7 was built. Yeah. I, I, I totally understand what it comes but when you're like working in working in an enterprise company, like you, you have like RL6 popular thing right now at RL7. We are just adopting into RL7. Um, so no, no Mac good options there. That's for people. So it writes all the things into trees. So let's look at the Mac file. That's the like, uh, important thing. So how do you build the whole thing? I, I'm not going to run the Mac file. I just ran before this talk and I have the logs and the image. I'll show you that because if you run it, it will take another hour with this entire connection. Um, so I have a Mac file named macfile.lorex. So as you can see the, all the files are like, uh, it's a plot and 21 and exactly 64 and, uh, so I have a command called a small macfile variable called ranmark. So it uses it's going to use the block 21x664 compass uh, mark configuration file uh, which has the bind mounts enabling everything. And, uh, so I have a one I'm not going to use the updates mirror. So I have a, um, mirror study of MIDI. I think a MIDI is closer here. So just a little faster. So I have a Fedora mirror, which is like a 21 mirror here, base mirror, no updates. My Anaconda, I have a web server running and I copy it on my Anaconda 2. I mean, I don't know. I try to use the file flag. It just didn't work. File, colon, colon can. What is it? Uh, I didn't. Yeah. Yeah. I forgot. Sorry. I don't know why it didn't work. Yeah. Okay. So, um, so this Anaconda mirror, um, so right now, so to work around that, I just started my HDTBD. So you can see, um, I can have one more WWW STML Anaconda, so it so it just has uh, all the Anaconda RPMs with the bump release pump dash 2 which has a patch in it and then I put it up. Um, yeah. So it will retire from that. So I also, I'm also going to test this image. So once I have the image, I will just start an EM and see if it works. You know, it's like a smoke test. See. Everything works. Okay. You can ignore that. So the main, main thing is the mock tree. So I should have run like dash dash scrub all. I normally scrub everything and you need the mock tree again, but with this internet connection I cannot afford to scrub anything. Yeah. So what I do is I, uh, install, uh, low-rex into the mock tree first and then I copy my current working directory which has the actual, this main file and the actual code to build the whole thing into the into the, uh, mock tree. And then here I can do, you know, do the seeding to the work directory and then I call make file dot low-rex to, I don't know why you play. So I call them low-rex clean make, uh, and I don't know what's there. I think it's probably uh, low-rex, uh, tree. It's probably clean and tree. So it's probably first clean. Clean the whole thing and then again, uh, because the tree it builds, uh, the low-rex it uses the low-rex command to actually build the thing into the tree and then, um, finally, uh, this is the clean. So it just cleans, just deletes all the unwanted stuff and, uh, this is where the whole thing runs. Oh, this is important in case of Fedora. So... Oh, yeah. Yeah. Okay. So this is, uh, so if you don't exclude this, um, you know, product, this package is, it'll, it'll, you'll, you'll run into Depsol when you're creating the runtime image. Um, so the product of this is right now stored here. So we have EFI images, ISO Linux, and Live OS. The Live OS has the squash of this image, which is stage two. I think EFI... ... So that, I think ISO Linux has the kernel and the round disk. Um, so, so this is basically, uh, and then, I think images, images has the boot.iso. ... Yes. So this is basically like a DVD tree, uh, the whole thing is created. Let's, let's test this. So this is created, um, I have the build lock here, um, I don't know, I don't think I have to show this, but, um, so I'm gonna just, uh, do a meek. So what I did is, I built the ISO and, uh, I patched the whole thing up, and I, I'm gonna run from here. Okay, I have the, I'm just going to do a quick test using VM. So I just created a boot.iso. I'm injecting a kickstart file in there. But it will show you like the actual boot and say flock 21, um, I haven't replaced the, you know, the logos and all this, but just want to show you like flock 21. Uh, yeah, there you go. Just gonna go show the actual graphical UI, so it'll show flock 21 there as well. So, I don't know, a lot of people may not have, you know, don't need to really like do all this because, uh, Anaconda just works most of the time. But, uh, if you're working in a company, big company enterprise, which you want to use, wants to use Anaconda, if you want to reduce dependence on, uh, red ads, you know, release model, they take a little bit of extra time, you want to fix it fast because internet companies are like fast. Uh, things move fast. Um, so, uh, we, uh, you have to do this. Okay, any questions? Uh, so, if you're just adding an RPM, would it ever just, uh, for hardware? What is that? Uh, instead of building a whole, uh, new image with your back RPM, would you just, uh, use the stocking there for the hardware? Updates that image? Yeah. Certainly, we have the ability to add updated images, but I mean, it's just normal, like, um, Oh, with the driver? Yeah. Okay. So, maybe you guys can send me the instructions on how to do that. Uh, yeah. Documentations on this, I think most of the things I brought up from BCL, BCL is here? No. No. So, he, he, he has a tricky page, his user page has some instructions. And, uh, if you guys can send me stuff on, like, uh, the driver image. Yeah, it's a little hard. It's hard to find. You have to ask, um, probably, you know, this kind of starts, it's like an initial thing. If anyone wants to get started with, uh, playing with Anaconda, want to contribute to the Anaconda project, um, it'll be a good point to start. It comes to whatever kind of you mean, you know, local engine, like, updates, and all those updates that I mentioned. Okay. Maybe you guys are here, right? Maybe I can talk to you guys about it. Huh? I'll, I'll catch him. He's not hiding. All right. I'm going to stop the presentation and thank you everyone for coming and, uh, despite the uh, Fesco. Yeah. Yeah.