 Welcome to some of the work that I've been doing. I just wanted that I've been doing with the community projects and has gotten me excited about working more and getting more deeply involved, not just with the Fedora work, but also just lots of things that are associated with the open source and building images, creating things that other people can use. It's been really fun for me. I'm a solutions architect, so just a little bit about me. I'm a partner solutions architect for Amazon, and I spend my daytime... Let's see if my slides will advance. There we go. I spend my daytime working on Red Hat Solutions for Amazon and Amazon Web Services customers. So this is pretty dear to my heart, right? And before I worked in Amazon, I worked at Red Hat. My first computer was a ZX80 that I soldered together with my own hands and with a lot of guidance from my dad. And I live pretty much my whole day in org mode and trying to make everything more like literate programming than it probably should be. So this was an exciting thing for me to sort of take on as a quest. I wanted to build on... Primary goal is I wanted to build with Ansible on Amazon Web Services and to determine where the rough edges were and how I could really communicate that back to the Ansible team, people like Adam and Jill, who are doing really great work around the development while this whole new thing with collections is happening. It's a much more... It's a complicated time, right? And then I wanted to demonstrate to myself and maybe even to others that I could participate in iterative progress in a way that as an architect, I don't necessarily get to spend a lot of time doing. So I'm glad to see, Michelle, that you also enjoy literate programming. It makes me very happy and high-nick. But I have some personal roadblocks I feel like because I spend a lot of my time working on solutions that are kind of key in the business area and not as technical as I would like them to be. And then I rely heavily on strong engineers like Neil, like Nick, like Jonathan to help me sort of redirect the guidelines around projects that I'm working on. And so this was an opportunity for me to kind of dig into the toolbox and sort of feel around and find out how these things really fit together. So there's my primary goal. But then as a secondary goal, I wanted to really jump out there and make this work for a real project, right? And the real project that I chose was to build out two things. One was a CentOS-based image that I could then publish into the AWS Marketplace in China that would match the requirements of the community and be available for Chinese customers, where we didn't have a presence at that point. And obviously, you know, working with the team to make this a part of the process. And so here I am with my goals. I want to build a competency-enhancable and related build process on the distribution of my choosing. And that's mainly, you know, vis-a-vis Fedora for all of us who participate here. I wanted to figure out a lot more about the user collections and where the AWS collections, both community and the ones that Adam and Jill and... I can't remember. There's a couple of others who are supporting this. Beyond just the community at large, it's doing the majority of the work for the AWS so that I had a very strong voice internally to talk about this with our engineering teams. And then finding entryway into the tools and techniques that are being used by my friends like Dusty and his teams and Luca to do the work and maybe make myself even a better candidate for helping them to get their solutions working than I am today. So, and to continue working on that over time. So as issues evolve, you know, editing this, I want to have a couple, you know, this CentOS image and then want to be able to publish the Microsoft SQL image process so that anyone can follow it. And it can be a guideline for how to do this in the real world, but then also to be the exact same as the process that's being used internally for deploying those things for the rest of the world to use in the context of AWS. And then, man, my pipeline skills are bad. So this was a super exciting opportunity for me to get to understand more about the way that Jenkins works. Obviously Jenkins and I have struggled and Ansible and I have struggled together over the course of this and to bring some awareness to some of the best practices around the system that I'm finding out from doing this. So here's my milestones. Build an Amazon EC2 image using ImageBuilder. So ImageBuilder is sort of the focus and I know Neil, you and I have had some conversations around this in the past, but, you know, with the work around OSBuild and the way that it's functioning today, I'm super excited to, you know, to sort of stay on that track and try to bring that up not only through the Fedora work, but also into the Red Hat work and then onto CentOS where I feel like a lot of this work is being done and to be able to do this all in the context of sort of a cloud native style, right? And then ultimately deliver this artifact that is the base AMI for use and a process for reviewing that in such a way that both the build itself is discoverable just like it is in Koji and then to be able to determine how to improve the real processes which are the ones that are being used by the Fedora engine, REL engine, the CPE team today. So I got started and I started working on this in sort of an iterative way, starting with, you know, just the hand-holding and the figuring out how things work, installing things over and over again to figure out how I could better build my process in. But there are so many choices out there in the world on what to do and what tools to incorporate and how to test and what QA looks like. I really had to narrow it down and narrow it down quickly became very key to me. So finding the tools that I wanted to work with quickly instead of figuring out, trying to figure out what was the absolute best tool was I thought the right way to go. So I settled on a couple of things. I think I just went backwards. I'll go forward two slides now. Give it a second. Probably should have done this in a PDF. Sorry about that, everyone. Okay. Yeah, there's a lot of choice out there. And just inside of the Fedora teams, there's a lot of things that are very different and extremely malleable, right? Very different kinds of tools that I could have used. So choosing these was something that was just an arbitrary choice. And I kind of want to make that more or less clear. But I had some goals in mind. So the goals kind of help. Keeping the community project affiliation was really important. And making sure that I was using things that I could see were a part of, at least a component part of the deployment process that everyone was using around me was something that I wanted to keep in mind and make a part of my decisions. I also wanted to be able to read whatever it was that I was using. So important enough that it be open source because that was a part of my process, was figuring out how these things work and then filing bugs on those things, being able to patch them and look at them. And Image Builder and I had a lot of those and I marked a lot of things as bugs and then I marked a lot of things as not a bug after I figured out what I was doing wrong. But then I got to be a part of a lot of the test patches and figuring out what was going on upstream. And Brian Lane helped me a lot to get to the next steps on building images with Image Builder. And some of the things that I learned applied directly to those downstream commercial goals that I have, like building RHEL with Microsoft SQL on it. And I found some things that were really interesting about the build process that I've implemented in the context here. I'm one of those that I don't have really illustrated in my slides, but I wanted to talk about a little bit is the ability to provide a group of scripts from a GET repository and present them as an RPM, dump them into a specific location, and then execute on that in the context of the blueprint. That was a huge assistant when I was looking into the concepts of building the SQL images because I couldn't figure out how I was going to get all of that script information material onto a system that was freshly installed without doing some sort of a crazy charoute or something else that didn't feel like it was going to fit with the CLI commands or even the cockpit model for doing the deployments. But I was trying to stick to the CLI commands or anything that I could script in the context of Python so that I could move forward with the deployment hands-off and keep it moving in a long-running configuration. So what does it look like? So first off, AWS Code Pipeline became a component part for me because, well, I'll talk about this later, so I'll talk about that part of it later. So first off, the Ansible scripts are part of the deployment. That's pretty much the trigger, and it hooks back to the GitHub repository through push events to the Code Pipeline. The Ansible script runs, and the first thing that happens with the Ansible script, and of course, this is a chicken and the egg problem, the Ansible script has to run once manually. So my goal was to have something that I could run once. It would build the infrastructure that I need, and then once that infrastructure was built, I would be able to maintain it with the same scripts. So first off, the Code Pipeline, there are Ansible modules to support building the Code Pipeline itself. I mean, you have to go in and dink with the permissions for the push events. If you want to use the push events, you'll have to do the authorization from GitHub to the Code Pipeline so that it can see your... So it has access to the repositories. And then, find the images that I want to use. In the current state, or in the state that I'm using ImageBuilder, I'm obviously OSBuild. I should be able to use... When I'm using that, I should be able to use a single distribution version to do more than one other version. But for now, I need an individual ImageBuilder instance for each one of the distribution versions that I want to build for. And that means I also need one for each one of the version architectures that I need to build for. Something that I found kind of interesting there was that the... Obviously, I had to craft a little bit more handily the configuration. I've got some snippets here. Let me see if I can pull something up in the Emacs here so you can see it. So here in the server, in the server definition for the Jenkins, and there's a few variables that are missing. But I thought it'd be interesting to kind of show this. One of the things that I wanted to make... One... Or an important distinction in the way that this Ansible script works... Yeah, exactly, Neil. So... But Neil was grumbling about the lack of cross-arch support. It gets complicated. And so good news is that I have what I need to pull that out. And so where's my... Army definition? There we go. So if I have an instance arch, I'm using that instance arch in the definition of the Army query. So grabbing these thanks, Dusty, for making this so easy to review. And then grabbing the instance architecture that I'm expecting to build for. Once I have the instance architecture, I can make a decision in the variables about the configuration. But I've got it here, specifying the instance arch in the filters. And then same in the name, I obviously could have gotten away from doing it one place or the other, but I've got it in both, and I'm comfortable with that. And then the Genja template of functionality was something that I had not worked with previously, but then started to work with in more of an interesting way. And I found a whole lot of ways to do this, and this is one of those places where I feel like I really need to dig in, that it's the place where you'll see the most functionality in Ansible go the way of one line scripts where you said something into grep and then awk it, right? There's so many places in here where you can just transform data in different directions. So that's an essential component part of the deployment of the Jenkins server. Also, this is interesting here, there's a number of these modules, so from the collection, the Amazon collection, and then there's some from the community collection, and it's not clear which direction to go with which one today, and I feel like that probably will clear up more in the future, but it was an interesting experience to go back in and see where those are. So this is a little bit more about the pipeline itself and what my goals are and the experience around that are. I've got an extra helmet in here, sorry about that. So the helmet is an IAM role, and this is something that I'll point out a little bit more, but I'm associating an IAM role with the Jenkins instance because the Jenkins instance is in the playbooks. So initially, you have to run the Ansible playbook from somewhere, but next time it can be, the Ansible playbooks will land on the Jenkins instance or in the jobs themselves, so it'll land in the pipeline, and this is essentially what's going on. So here's me and anybody else can help me get better at what I'm doing. So Jonathan, it's not yet... Jonathan asked if this was somewhere public that we can access the playbook. It's somewhere that I can share it with you because I would be super excited to have any of you help me get the work done. But right now, I have questionable security on the artifacts that are in the repository, so I want to make sure that I'm not sharing out old secrets. Yeah, I know. Neil gave me the side eye on that one. But yes, I'll get it out. I'll make sure it's available, but yeah, there you go. It will be available to everyone to look at. So the way it works now is you update the GitHub repository, and actually I've got it split into two repositories. There's one repository that's there for the image builder to collect the information that's associated with the builds. So for the SQL server, there's a repository specific to the build of the repos. So the instances that I'm using are cloud access instances, and the cloud access instances come up. I register them with Red Hat, and then I'm doing a full repo sync onto the image builder instance. I have to use a pretty beefy instance to get it to work because it either has to be metal or one of the larger, more sophisticated nitro instances to get the virtual machine configuration to run in a timely manner. I mean, I could run a smaller instance for longer, but why? If I'm just going to shut it down. So from there, the pipeline push event happens in the code pipeline, and the code pipeline is initiating a longer run on the Jenkins server. So the Jenkins server only has the standard port open. I haven't done anything to secure it at this point. That's a roadmap item for a different time or for someone who knows how to do it in seconds instead of me doing it in hours. So the Ansible Playbook then runs and creates, generates the instances, and you see the little Lorax here, a little history on the image builder. The instances are then generated using the blueprints that are in the configuration from the GitHub, and the blueprints themselves are just pulled from S3. So the whole goal of this obviously is to have several of these instances running, creating either REL7 and REL8 or CentOS7, CentOS8 images with the component parts. And then to have essentially a little solutions builder here so that we can vary the blueprints that are being used and produce some very interesting sort of collections of what I'd love to do, and this is my goal here on what I was excited about was building a way to have solutions that are already there in the Fedora repertoire or scope and to provide images of those that can be shared to customers in ways, really not customers, but everybody users who want to leverage that in the context of the cloud and to provide kind of a guideline model for an architecture that spins up works and then spins itself down, but is still fully functional. Because, well, we'll talk about that in a little bit. So the and then to have an associated role for this instance here so that we don't have to ever provide any of the any of the credentials that would be required so almost everything that you read about using Amazon EC2 has somewhere where you're storing you know in a semi or permanent or permanent way credentials that then are out there in the big in the big world with with little safety net and of course we don't want nobody wants that. So Neil, you had a question. You said, is the idea to have a layering model for producing omis? Yes, the idea is actually to have a way to produce artifacts that are not just omis. I mean, the goal is to be able to produce isos to produce anything that people, you know, that you would want in the same way that you would do that in the context of Koji, right? So, I mean, I believe that there's nothing there would be nothing from just just outlining and deploying a pipeline with the or deploying a a request to build the images with the scripts that Dusty has in place today, like with the Corus builder or whatever, you know, or the image builder in the context of the CPE but this is kind of a I want to understand this model as much as as you do. And some of that requires some building, right? Building on my part. So, using the AWS code pipeline, so jumping right into building out and I know it's fairly simple to build a standard pipeline configuration for Jenkins, but it's not really on my radar of things that I need to be deeply involved in. So, I chose to take this to take this path with the code pipeline to begin with because I felt like I was I needed to crawl before I was walking and I also needed to have something that was there available and servicing the configuration at times when I didn't have any hardware in place and wasn't really ready to deploy or what didn't really wasn't really active, right? So, let's say there's no changes in the GitHub repository, there's no modifications to the code that's associated, you know, that's intermediate I don't have any way to I don't have any need for infrastructure I want that to go away and then to integrate with as many the tools that I mean, so knowing that many people were not going to have the same kind of limitations around Jenkins not going to have the same limitations around I wanted to have around the other tools that I had chosen I wanted to make sure that the people who were willing to help could jump in and really help me not only expand my understanding but also to expand the capability and the ability for others people to use it and there's like some cool tools out there that I learned about just, you know, in the past day like TMT, like the test tool that Miles and those guys were working on and it's pretty amazing how much of that could extend the functionality and something like that Code pipeline, no, so Neil asked if it's a pipeline Jenkins based or something else it's something else, so it's a it's actually based on an internal tool that is called pipelines for Amazon and it was originally written I believe it was still written in Java but it was originally I mean, now it's written at scale in microservices and it's given to the AWS microservice model integrating into a whole lot of other tools on the back end okay, I accept that but it's a, when I say that I mean that by itself it's not Jenkins based and in fact benefits from an extension to using Jenkins for standard pipeline technology you can build your own plug-ins for it but it's a, in my opinion it is a fairly simple tool for the process and it expects you to incorporate custom components into your build process and somebody else has a different opinion of that and I'd love to hear it and like when you're not yet, well, not yet yeah, so it's very limited in the scope of the repositories that it will use so you can, having a native Git repository is not supported having a GitHub repository is supported having a code commit repository is supported having an S3 object that has SNS notifications is supported but like I said it's one of those things that I thought was a really good it's like a really great fit for where I am today and that may end up being replaced by other open tools just because my goal is obviously open default to open so the long runs, so all the things that are doing here was a community decision so I want to learn with my friends this is not as simple for me as it should be and I think that if I give a lot of people an opportunity to really shine that I get a lot out of it that's something I'm really grateful for and I think we all are grateful for the mentorship that this project has brought to us all but it was a great opportunity for me to reach out and say here's a simple tool where you can hone your mentoring skills and I can benefit so there's outside of the GitHub flavors like Travis and Bamboo and all that there was a it's a pretty solid rock solid choice GitLab has their own hooks and everybody's got some way that they can integrate with it it's the legacy soldier that keeps us all working and keeps sent to us and check and so there's a lot of I think there's a lot of benefit to working with it obviously it facilitates my needs and AWS Batch was something that I toyed with the idea of putting in here because it also is great for scalable work and like massively scalable work it's used for in a lot of cases where people are driving parallel computation in ways that are different than they used to it's not just your basic batch scheduler like platform or lava so I considered it pretty heavily because it's one of those things that I hear a lot of customers say oh well we have thousands of virtual machines and it's going to cost us a lot more money these thousands of virtual machines on the cloud and when I look back at them I see that they're purpose built they have very specific functionality and that keeping them around isn't really necessarily in anyone's best interest it's just a good thing to have those purpose built configurations and so I thought this might be an interesting place for me to understand better how I could help people disaggregate those workloads and understand how to save how to save money in the way that they do it and so I mean I'd love to so long picture I think of this Serge's work on Koku and the things that are out there that that are integrating cost analysis and billing review into their process and like billing alerts I'd love to sort of build that into this process so that people can understand exactly what it is that they're getting in the opportunity and make something that I think is a good reference architecture in the context of Fedora and Fedora based tools okay so here's my spoiler which is that I'm not done like I was super excited to do some big time demos for Flock but I haven't made it so let's talk about this as being the first iteration of the talk and the first iteration of the tool so I mean I can show you some of the things that I've done and we'll do that if we have time and you're interested but the truth is is that I'm still running a lot of this by hand and I'm not getting any of the Jenkins stuff built but in an automated fashion so it's burning a hole in my pocket to get more time to work on this only building the image builder so far and producing fundamental images for like our instances I guess I should say instances for CentOS 7 and 8 and the upstream Red Hat 7 and 8 so now a CentOS 7 and a CentOS 8 image in either architecture on and a 7 and an 8 on either architecture I don't really feel like I have to worry about doing that on the Fedora side because Dusty's doing it all for me but to have those with some customized configurations the same thing I would love to do the same thing in terms of the spins but I don't have I haven't really pulled in the spin configuration so that I was just churning out machine images but at some point I would love to see that and then creating sort of a state equals stop or a state equals terminated or a state equals absent I guess for the EC2 instances hasn't really happened yet and one of the things that I do want to experiment with it's not really available yet but it's still I mean it's open it's an open item to integrate EC2 hibernate into support and onto AWS within the context of Fedora so Neil I hope you'll review that spec file when I get it polished for addition so the EC2 hibernate while the page is changing one of the things about the EC2 hibernate what was exciting for me is that hibernation in the context of a server and definitely in the context of a cloud server is something that's fairly new but I think it's kind of catching on and there are people who are using it in the context of spot instances so they're like hey you know this is a lazy process so I don't have to really think about whether or not I want to do it I just would go back to the service and then and then I can look at you know I can I can look at the spot price and then when the spot price is right I just wake the instance the instance continues to work if the if for some reason there is a you know there's a spike in price then that can go back into the hibernation state and continue forward there are some challenges that have been kind of interesting because that we've run into and I think are kind of sort of notable one is the SE Linux policy around writing to a swap file it turns out that it's better obviously it's a better practice to not force customers to have like a second volume for hibernation you want to you don't want to swap volume by creating a monthly bill for an additional device that's going to hold a swap partition so the swap partition itself is kind of rolled into in or it's it's a it's basically a five gets created as a swap file and that swap file gets written to well it turns out that the FS isn't really friendly to writing to that swap file in a block fashion with transparent huge pages so you have to turn you have to move the move from the default configuration of thp on to mAdvise so that the so that the huge pages aren't used in the in the that is that's a great first step in preventing file system corruption but then SE Linux domain for hibernation for the for the shutdown that domain doesn't have context to write to the swap file so any attempt to run an actual hibernation lands you have to relax the SE Linux context so here's something that I have I found very interesting so one you know we're just in the early stages of the community and the supported can I go backwards I did we're just in the first stages of the ansible support for collections and while it's gone on for now for a little while from you know for over the course of the you know early in 2.8 and then now in or late in 2.8 and now in 2.9 the 2.9 has brought brought us the first releases of this and like I said I'm super excited about where this is going and I think quite a bit for more active participation on you know from the from the teams that are responsible for these for the services underneath these modules so that we have some full support but for now we have really great community support for a lot of the configurations and just building in the support for or building this into my process of to the local directory has been beneficial in running the ansible configuration so now I'm building those ansible configurations and the local the local setting I'm not setting them up in just across the system with the ansible playbook I'm setting them up with the with a config that has the current working directory as a base I think that's the right way to do it the system role for image builder I can't remember the name of the guy who's responsible for this but I can ansible galaxy it into support I think so I do an ansible galaxy collection full system roles image builder is it info collection yes it's an Ubuntu laptop it's my work laptop you know so you'll ask if I was using an Ubuntu laptop my my fedora laptop has four gig of memory and my Ubuntu laptop has 32 gig of memory so when I thought about where I wanted things to go yeah it's got to be in here somewhere so yeah it'll give you an idea of what I'm doing to run this I'll just leave it at that like I said I can't remember who's I can't remember the command anyway so this builder role was critical but then the other thing that I thought was really important and something that I thought was interesting to everyone was that the storage role in the context of building for AWS is tough and Dusty we've talked about this and this goes back to the Udev rules that we talked about before and that we were talking about the upstream bug for this where the devices on EC2 the EBS volumes they come in as pseudo PCI devices in the way that the hyperconvergence works Terry Bowling thanks Neil and it's fabulous he's doing it constantly increasing the robustness of the playbook all the time so I'm super grateful to the work that he's done and just want to call that out so the storage devices though the way that we call them out in the configuration as a device the device names are old school devices slash dev slash SDG SDF but that enumeration doesn't carry over into the operating system once you get into the operating system that device name that you've associated with the EBS volume the ID goes away it's in the NVMe info for the device but it's no longer but it's not a sim link so whatever you named it that doesn't exist you have to go back and rediscover that or use the Udev rules that Luca provided in CoroS to to develop that and I actually have a copper repository for the EC2 utils that came out of the Amazon Linux instances so I built this EC2 net utils so that it can include both the network utilities for managing the ethernet or the network interfaces and then this the EC2 utils so that you could enumerate the devices but Luca's rules in my opinion are they are more I think they're simpler right so he creates a simpler rule and gets the same result so I would encourage you if you want to you know to leverage this package or doing it the way that Amazon is doing it in Amazon Linux but if you want to see a very effective and efficient way of doing it I think that Luca made a great set of rules so well I've got a heavy hand on this pad so the storage configuration becomes rather important so one of the other things is that the volumes that you're writing the images to is a volume that you can restore from snapshot this is not something that I've put in just yet but it's on my wish list for things that I need to do but the one day this will advance is that the devices themselves have to be enumerated correctly you have to go back in and discover them the NVMe CLI gives the information that you need but if you're in Ansible that's not necessarily something that you're using unless you install the NVMe CLI so this role storage you have to remember to go back and reestablish what the devices are or add the EC2 Utils or Luca's UDev rules into the system to get the right device information and I think that's handy just in general not just specifically in the context of this this configuration so I pointed out that I did the IAM role associated with the Jenkins instance and that was permanent credentials or what I say here a waste of your time the same instance you have for Ansible Engine can drive your Jenkins server it's possible that we could do an AWX server at some point or an Ansible tower and use that for callback functions one of the things that I think is completely underutilized in the context of public cloud is the cloud config and cloud init and I'm sorry Ignition Dusty, I'm talking about Ignition so bringing that whole process together is something that I think is really important so there's nothing stop us from killing the entire architecture and leaving one pipeline in place and with that I'll I've left two minutes for questions so Neil asked does AWS have a fancy native way for Ignition configs? No the answer to that is that in almost all the documentation I see a lot of just stepping away from the cloud config or the JSON configuration that you would require for Ignition config to be laid down to just issuing bash commands at the user data level and if you look at some of the things that are done with the the EFS team they have done some things that are similar to what I would expect to have happen in vendor data but they injected into the user data configuration at launch and it is always done in a scripted fashion rather than yeah Neil, your sentiment there is exactly like mine so I tend to and I think I have an example right here so I tend to start and end with I don't have an Ignition script up right now so just bear with me but I am doing this like I said CentOS 7 and RHEL 7 and 8 so for me the most important thing is to ensure that the configuration is set so that you can so that it can be read by cloud and yet I can get it out of the analyze so if I need to do a cloud and it analyze I can figure out what happened so no shebangs for me if you are doing a run command it should happen in the context of a list like it is supposed to in sub process anything else? Any other questions? Thanks I am super excited about it and I will tell you right now but it is getting better so you grumble about it let's build a little bit but I tell you everybody is getting a lot more excited about it with Lars in there doing a lot of work on it but it carries that legacy and a lot of the things are going in the right direction and there is a lot of focus on being able to build a much broader set of tools with just the one or broader set of results with the one tool so I am really excited about it and will continue to work on integrating it and building this iterative process so we can all look at it together I will submit this talk again and we will look at what it looks like the next time yeah thanks everyone for participating I really enjoyed this and I hope we can work on it together and I am always on IRC I will just say that so if anybody wants to reach out I am super happy to respond back and I am DAVDUNK everywhere