 So while they're joining in, I'll say, I'm David Duncan. I'm a partner solutions architect, and I work at Amazon, but I worked at Red Hat first. So obviously I've spent a lot of time around this space and I have been a Fedora contributor for more than 14 years. And so I feel like this is my first family and definitely a group of people that I want to make sure are having the right kind of experience around the public cloud and looking at some of the things that we all run into, there are complications and trying to find some ways that maybe we can address those. I gave a talk at the last CentOS Dojo around the kinds of things that were necessary to get from what the expectations of public cloud providers, service teams, like just the service teams that want to have big goals around what they want to bring in into their programs, they want to bring those into the operating systems like Fedora. And so we spent, well actually, their goals usually are around commercial requirements. And so they're talking about, they think that they're going to go straight from start of nothing to a hundred. And thanks, Jerry. And so they're doing great work, but we wanted to make sure that we were taking that in the right direction. So we want to go from Fedora into the commercial distribution red hat, right? So take that from the upstream and then bring it back in. And so it's always fun to look at how we can work together to get that done. And some of the things that we've done over the years is to have joint sponsorship around different things like the cloud in it package. We usually do a summit every year that includes representatives, people who engineers who are working on the cloud in it come together for discussions, obviously not in times of COVID, but in general, and there's been like overall from a perspective of the groups that support open source software, we kind of, we know how we're gonna work together to make sure that we sponsor these things and don't make it a burden on each other. So just wanna give everybody an opportunity to kind of ask questions, talk about what goes on and enjoy finding some of the things that they want in that experience around Fedora and the CentOS experience and even RHEL. So I worked really closely with the Ruby team and was just praising Andre and the rest of the composer team for what they're doing all with OS build and the ability to automate the upload of images. So I'm gonna leave it right there. I kind of open the floor and say, I'd love to have other people who are working on projects or who want to understand better how they can take advantage of the work that is being done around supporting open source work and around these distributions, which is where I spend most of my time. Anybody have any specific questions or are they working on something? I mean, I'd love to hear about cool stuff too. I mean, that'd be great or we can have dead air. Well, I'll tell you some of the things that I'm super excited about. So one of those is hibernation. That wasn't something that was in any way expected, I think to be a benefit on cloud platforms and it really took off. But not without some generous help from sort of a community-minded approach to kernel requirements, supporting the swap file, making changes to the SE Linux domain that was necessary for writing to the swap file. And then part of what we did in the kernel was to work with the Red Hat team to make some upstream changes to the way that speedup is defined in queuing rights to disk and made some modifications to the way that the swap file was written through to the NVME devices, which affects platforms all over and the ability to leverage hibernate in larger and larger memory spaces based on the time to shut down. So things like that are making headway. They're changing the way that things are going. In fact, two years ago, there was a joint presentation done at Plumbers to talk about changes in the kernel that were related to hibernation and how that was a benefit to customers who were using it and getting the opportunity to have fast recovery time and to leverage spot pricing for opportunity workloads. So what about CLI work or support for services? I'm really not thinking about this from the, I'm only talking about it from the perspective of AWS because that's exactly what I know, but I still am interested in how I can help to make this a better experience around Fedora regardless of where it is that you're using it. Anybody doing anything fun with like automated security? I know we have some, we had, there was at least one talk about building more automation around some of the security profiles and making that more automated and something that I think has been a lot of fun. So Miroslav, you asked about EKS on AWS. So there's been a lot of work around building up support for that within the context of like building solutions out and manage nodes with Fedora. And we have, I know that there are a lot of people who use CentOS and even rail for their managed nodes. But the thing that's really exciting to me is the Red Hat OpenShift on AWS. Can we get that set up? Because now we are using EKS basically, but I would love to use OpenShift for my selfish reasons. So I didn't even ask. So maybe this is a very good place to ask if that would be possible, for example, on our Fedora CI account. Oh yeah, totally. Wow, cool. So should I reach out to you or where should I file an issue or how? Kevin Finsi, yeah. So Kevin will not. Cool, awesome, yeah. Because we had some issues there because we are sharing the account between multiple people, multiple teams, right? That's fine. So it's just a little bit long turnaround when we need something, but it's not such a big problem. So cool, I will. Yeah, I was just gonna say, I think a lot of that came from just not having, so in the early days of setting up a lot of that infrastructure, we didn't have all the permissions hammered out. So we were building out a better and better, more well-architected system. And I think that that's something that working with Kevin has been possible. And then there was a small matter of some credentials that had been pushed to GitHub and located very quickly, disabled very quickly. Kevin, in fact, had disabled them, but the follow-up on the security review was slow. And so we had a period there where we couldn't increase the number of instances that we had available or make changes to the account because there was a concern about compromise, which is a good thing. I mean, I definitely want people to be looking out for my best interest when it comes to security. Of course, like we are encrypting our credentials and we keep it in private repos, but you're encrypting with Ansible Vault currently. What would like to use some magical vault or something at least that maybe that will be even better, but yeah, not enough time. And not enough expertise, I would say. Right, yeah, yeah. Well, so I think a lot of the work that's been done around the identity management has been to establish the SAML off and to use tokenized credentials. And the tokenized credentials are great because they expire pretty quickly and they have a defined open time and you can narrow that down. And for the automation accounts, I know that MoBruyern is rotating those so that he is reaching out to people and rotating the keys for automation users. I need to follow up on that. Okay, cool. So if you say that OpenShift on AWS is possible, I will file an issue and we can follow up. What would that take to set it up because? Yeah, 100%, it's 100% possible and we can definitely make that a part of the infrastructure. Awesome, one thing I wanted to ask, so how possible is it to, for example, also have QBird enabled there and the OpenShift virtualization? So is that something that we could also use? Because we have some workloads, a lot of our workloads are basically VMs. And yeah, VMs, we can just use EC2 instances, but still like for some reasons, for example, for internal reason and so on, we want to have capability to also use QBird. So that's an interesting question. Yeah, so I haven't used QBird myself on AWS, but I do know that we have instances where the KVM functionality is exposed, it's just not supported. And so it's possible and obviously, we have our process for doing that is typically through Firecracker, we like to get rid of the BIOS and take away a lot of the IO subsystems to make it more lightweight, but the QBird falls right into that plan. So I'd say we should look at profile some of the workloads and figure out what instances are the right ones for that. And in regards to the nested virtualization, I guess there is no nested virtualization on EC2 instances like the virtualized ones. I need for the metal. Yeah, for the metal, yeah, we can do that. And if the workload calls for it, we can, we'll figure out how to make that work. And a little bit selfish on the questions, maybe others had also questions, don't want to take all your time. Yeah, no, that's good. I was actually also excited to hear your question about Composer. So I use Composer to build images, right? And so for me, that was an exciting thing to say is, hey, there's a community model for building images with the solutions that you want and the stacks that you want to build. And so when you started talking about that, that's one of the things that I have is like a part, it's a great part of my work process. And granted, like I lean towards making sure that I'm doing work with the open source tools because that's what I'm supposed to have an expertise in. But Ansible is one of those things that falls right into there, right? So I use Ansible, Red Hat Ansible all the time for building workloads, building out workloads and automating security review and those kinds of things for the things that are within my responsibility. But I run into these little places where I think, man, I should spend more time developing for the Ansible team because these community modules are missing things like the hibernation, right? I was talking about the hibernation. And so when I first started writing up my automation for testing of hibernation, it was like, man, I've got to add that. That's like, that's an option I need to add to the EC2 hibernate or Big Jill for. So, and that's exactly what I was thinking about when we were talking about, or when I was talking about building this meetup was to think, there's a lot of those spots in there where there's just tiny little slips around the support that you get super excited about it and then you move in and you're like, well, hold on a minute. I'm gonna have to do this a different way. And so most of the time, I will go as far as I can, but then if I have to pull back, if I see that there's something that's gonna be clearly missing like, I don't know, I need to define prefix lists in my security groups, right? Or something like that with the Ansible side, then I'll back out and leverage a CloudFormation template. And then have that CloudFormation template be deployed in the, from within the Ansible stack. If we need now to automate something on AWS Terraform, but we are looking at Pulumi. So that is like also a tool that is there and it's close to Python because we do a lot of Python. So yeah, an experience with those, like... Terraform, yes. I mean, I've spent a lot of time with Terraform and I love the guys at HashiCorp, right? But like I said, my day job is supporting Red Hat on AWS. And so I'd like to go straight in for the Red Hat tools so that I can talk to customers about exactly how it works. But yeah, so a lot of the base OpenShift installer work or after four and looking at how that works and finding little gotchas there. One of the ones that I had, so there's a Jira open on the Bootstrap instance for the OpenShift installer. And the reason that Jira is open is because during the last or the last couple of months of the year, we have re-invent and in order to make sure that everybody can try everything they want to as an employee, I help to make room for the participants to enjoy the platform in the easiest way. So one of the things they limit is our ability to launch like an M5 instance type, right? And it just so happens that the Bootstrap instances for an OpenShift installer absolutely will not boot on anything, you can't modify that, right? You can modify the manage node instance type or the management nodes, but you can't manage the Bootstrap. And so that's one of those things that I see and I'm like, under specific conditions, this can be, this is rough. Yeah, and then another thing that I keep thinking about is network utilities. So I think in many cases there are ways to use. So one of the things that I'm gonna put a link to it in here is the easy to net utils. Let me make sure I've got the right link because I build them in the COPA repositories. But this is a group of utilities that we've been talking, I've been talking to Dusty about this a lot. So in the context of CoreOS, there's no way to just, you can't just like tack these back on, but these are Udev rules and some modifications to the network configuration that are specific to the way that network devices are discovered and arranged on the EC2 instances. And it's not quite, so it's written with Amazon Linux in mind, although there are lots of people who work on it and there are pull requests that have come in from a number of places. But the SUSE team decided that this didn't work, this didn't fit with their requirements, but they still wanted to be able to do things like, use the IP prefixes for configuring, for narrowing their widening the scope of their network as they see fit. And then they wanted to be able to add multiple IP addresses. So they built what they called the cloud net config as a part of the Insulatus project. And it is, it's a place where I really, I mean, I feel like they did a really good job of pulling that together, but it's focused around Wicked, right? Not network manager. And this is close to focused on network manager, but it's not really the same kind of, it's not the same tools that are being used today. So I'd love to see something change there. I'd love to see a little bit more dynamic support for that. And then the same thing happens around volumes. So it's a little bit weird in the virtualized world. I mean, from just booting on KVM all the way up into the platforms where there's a whole lot of management around or plumbing around how volumes are connected. The network utilities and the discovery of the devices is a little bit more complicated than just plugging and unplugging the same device, right? The PCI IDs are different, but the volume is the same. And so I'd love to see some more effort put in around getting those tools in. So talking to Dusty about it, Dusty May, we, somebody else on the CoreOS team from the original CoreOS team, Luca was talking back and forth. He actually made a little utility for doing the network utilities and handling the storage devices. And when we talked about it, I said, it would be great if we could push this up into system D and have just sort of a general library for cloud that was managed there. And there's an issue on this. I can't remember exactly where it is right now, but Leonard said that he was open to the idea of having some of those rules for the more popular options in system D. And I would love to see that go upstream instead of being an individually maintained package that doesn't have an upstream test bed and sort of get to that upstream first model. So... Interesting, I'm not watching this area. So by the way, like currently like the open shift on AWS, how does the use CoreOS, is it? How is it deployed? Yeah, it does. So you're using Red Hat CoreOS underneath. And it's, I mean, it's obviously configured by the installer. So you can use the IPI or the UPI installer. And you can, even if you're not, I mean, you can install it in pretty much every region now. And then, but you can also deploy it to Outposts. So we have some customers who have deployed it all the way out on the edge with the Outposts configuration and the UPI. So IPI, the automated installer doesn't support some of the changes in the way that the subnets are defined. But with the UPI, you can definitely do a solid install on the AWS Outposts. Nice. Thank you. Yeah. I also may be wanting to discuss and it's about snapshotting because we are using OpenStack downstream mostly and we have this feature that we are able to quickly, like OpenStack makes very simple to snapshot VM and then basically recreate that machine from that snapshot. We use that to run some tests which break the test environment. We use this snapshot. And it's like, it's of course, like it takes one minute or whatever it takes some time. So it's not a cheap operation, but still for some of the test which break the system and you cannot do basically anything about it, it's something we want. And when we want to implement this also on AWS because we scale out some of our workloads. I don't know if you know, not on this account, but we have a different account in AWS 4.0 and we scale out and cloud burst basically from my team, I guess. Yeah, we have hundreds as to like a little red, but we cloud burst some of those testing from OpenStack. If it's full, we cloud burst to AWS. We have it connected to the internal network even that we have a VPC there in North Virginia. And for that, this use case of snapshot thing is still not solved on our side. So we don't know how to implement it because I was looking at it and it seems like in AWS, this is not such an easy operation as in OpenStack. So. So, I mean, so in OpenStack you've got, you've got this unified storage, which you don't have on the AWS side. You're essentially taking a network snapshot of your device. And behind the scenes, an incremental snapshot can be fairly quick, right? So you can take continuous snapshots, just sort of rotate out snapshots. And then when you need to have a fast snapshot, you have that pre-cache and then your copy on write is much smaller. So you get effectively a much smaller snap for the last one. But the other thing you can do is to leverage LVM and you can take advantage of LVM volumes to do a base snap. Better idea is to have a golden image, right? And then just make sure that when you're doing your deployment, you can drop from that golden image pretty much any time. So what we try to do is capture, the packer process is great for getting started, but then you have this continuous process and you have some specific instances that you wanna bring up fast. So usually that's what we do is we tend to have, we expect fail-only architecture. So if I have a running VM and I want to snapshot it or maybe can I use that hibernation for it? And I want to basically recreate it from that snapshot later on. And the IP should stay the same, is that possible? Well, I don't know about from the hibernate side, you're gonna lose your swap, right? When you snapshot it, but because you're not gonna be on the same hardware. And there's no guarantee that you'll have, you'll have exactly the same footprint, right? But for an instance, like if I decide that I want, so let's say I'm messing around and I know that I want what I've got point in time on the instance that I have today. The fastest way to do what you're talking about is through the LVM and to create a snapshot and have sort of a double-sized volume. And the volumes themselves, if you get to a point where you think that this is the right size volume, you snapshot that volume, you extend it, you create the LVM size that will work for your fast snapshot, you do your fast snapshot. And if you need to throw away the whole machine and start it back up with the right size snapshot, because it's XFS, you don't wanna shrink it. I need to try it out, thank you. Yeah. Yeah, but the, like a logical volume for specific, for specific mount points is probably a great way to go and then just leverage that. And the AMI process is always better. Like if you can create a machine image instead of just creating a snapshot, that's usually my preference there is that you grab that machine image and then the machine image is consistent metadata and all. The snapshot itself doesn't necessarily carry with it the billing, like the billing identifier. So if you've got, let's say you're using, you wanna use the Rui and there's Rui in all sorts of, in lots of clouds. And you wanna use the Rui, you're usually bound to specific identifiers for the machine images or the images that are being used to pull those instances. And then you have to make sure that all of that is consistent. Thank you. Creating an image seems that it's what we basically want. Anybody else? Looks like we just get to have our own conversation. Anything else you're working on that's been? I don't know. I wanted to ask about the current quota setup. Like what are the limits? Because like we are hitting now, like doing a lot of federacy testing lately and it will go up. So I'm just, as I'm not the administrator of the account, I'm not even watching like what we are doing there. So I just wanted to ask what are the, what we are expecting. Right, so the cool thing is, is that you have what's called a personal health dashboard and Kevin can extend you access to the personal health dashboard and you can get that content from the command line. So you can bring the metrics from that personal health dashboard into something that is consumable by you. And so you can see where you are in terms of that. But sometimes it's not about quotas, it's about capacity. So you may see that you find yourself running up against limits. I mean, it's good to run up against limits. And then the next thing there is that there's an automated, there's an automated customer service request that you can make. And you can, so you can put in an automated service customer service request for a limit increase when you have time. And when you don't have time, you know, you can, you can hit me up, we'll talk about it. So, you know. Correct, we didn't have any issues. Like there was this, just these spot instances as we were enabling spot instances because basically for all of the CI testing we can just use spots, right? But we didn't because it didn't work, right? So spot instances, yeah. And I know that we are very hitting there some limit because it's like all the limit is on the whole account. So we had like 150. Yeah, well, so that spot instance issue was directly related to the open case on the credentials that had been discovered in GitHub. And so once, I can't believe how long that took. So it literally, the case hadn't been touched in so long that it had fallen off of the, you know, had fallen off of the dashboard took us a while to figure out where it was. Finally, you know, we were talking, I think Kevin was talking to someone in threat management and they said, you know, they wanted to, they wanted to know about this open case. And he was like, I thought I'd handled that, but clearly, you know, like we didn't agree on that. And so that he went back and handled it. And now we're actually able to make those, the changes in the limit increases that are necessary to support the workloads. Awesome, I need to set that up because like currently we are using, I think, non-spot instances and I would rather use spot instances and then failover to using non-spot ones. So we are able to do that. Yeah, and have you looked at launch templates too? Not yet, no, we have our old provisioning system. We have a presentation about it tomorrow that we use for that failover and cloud bursting from Open Stack WS. So we have that baked in somehow so we don't use launch templates directly using. But you use a kind of a similar, it sounds like you're using a similar kind of model though. Like it's the structure of it. Yeah, we have some golden images. So we have prepared CI, special images which have already baked in CI stuff which I built actually with Pector because like we have Ansible Playbooks. That's why I was... Yeah, yeah. Yeah, no, I like... Yeah. So Packer was a... So the interesting thing about Packer is that you have that option to side load things and side loading with a Packer configuration can separate the snapshot from the instance itself and then reassign that. And if a customer is using a metered billing like they're using an hourly price instance. So one of the things that they may run into that you won't run into because you're probably using just a standard instance with a cloud access image is that when you separate the snapshot from the metadata of the machine image, that the billing product code doesn't necessarily follow the snapshot until later. And then if you do mix things like root volume swap or something like that, you may have to go back and review what it is, what the provenance of your snapshot to ensure that you're not getting a different billing code reassigned on top of the ones that are appropriate. Very interesting. Yeah, currently we do the building internally in OpenStack and we just sync then the queue calls from OpenStack to AWS. So that's what we are doing. But yeah, I wanted to look into building directly on AWS but we don't do it now. So we'll just look at... Yeah, look at the composer. So that's another thing is the... So the image builder or OS build supports a lot of that build process through the same kickstarts we're leveraging to create the machine images through the group. So like Brian Stenson uses those for the CentOS images and Dusty uses them for the cloud images and that's expected. And although I don't think either one of them, I don't think they're using OS build right now but the process is the same and they're leveraging the same kickstarts. And you might be able to sneak a lot of the things you need in there before building a build. Yeah. So cool. I think we can call it. I don't think we got any more questions and thanks for joining me. Appreciate that. Thanks for answering my questions. Really glad and thanks for everything. Yeah, absolutely. Nicola, thanks for proctoring here and making this really easy. Oh, well, actually the whole team that is working behind us is helping us a lot and helping me a lot as well. I wanted to thank you for this nice conversation that you had. I didn't really get much of what you said but I think it was interesting for the people who were into the topic because I mean... Yeah. Well, I mean, Miroslav is talking almost immediately about a lot of the Fedora infrastructure that has been, we've built out on AWS over many years and supports things like the copper bills and a lot of the continuous integration. And so we wanna make sure that that runs really well. That wasn't necessarily exactly what I was hoping to talk about. I was hoping that we'd have a few more people who wanted to talk about their cloud journey and to discuss some of the pain points that we've all experienced together or some of the joint, a lot of the things that I think we've done that are jointly lifting for the community, like sponsoring the cloud init group for support or getting together for specific cloud summits around different technologies that we all share and seeing how those are supported. So had a very celebrated career doing this and have gotten to meet a lot of people in open source from a lot of different places. And it's been a beautiful career and journey around that experience. So just wanna share or make sure that we all get to do it. Well, if you want to share more, you have still 10 minutes. And if you want to share maybe some contacts or some resources for maybe some people that are in the chat or that are just reading this later or seeing this later to reach back to you maybe or even today or in the next days, just feel free to write whatever you want here. And because I think that, I mean, as far as I saw it, there are people interested in talking, but people are still very shy to join live conversations, right? So maybe they just need some time to understand and have the courage to reach back to you. Cool. Yeah, I will put some things in the chat there, but Yuri, so it looks like you've got your microphone off mute. Am I saying? Yeah, I thought I will use my second moderator privilege to jump in because I have some experience with Becker and I was wondering when do you trigger an image rebuild when you are using Becker for infrastructure? Is it something that you prefer to do manually or every time something changes in a repository, which then means you have to have separate repository for each image, which is quite impractical. But if you don't trigger it, then you can keep accumulating changes and then when you trigger a build after a week or so, then you figure out the build doesn't work and you don't know which coin broke it. So you have experienced situations like this? Well, so I know that a lot of the, so I know that there's a lot of work that's been done around this with EKS and the EKS team keeps those Packer scripts sort of collectively. Personally, I spend, I don't spend as much time with Packer. I don't spend enough time with Packer to have it integrated deeply into my build process. I'm using Composer, right? So the Composer CLI ends up being the place where I do all of my building. But I understand what you're saying. It does get complicated and I would say that I don't know that, I don't know, I just don't know. I don't know the answer to your question. And I think it has to do with the process. Miroslav, do you have something to say about that? For us, what we do now, and it's currently a prototype because we are taking over the process of image generation for real. And we have basically Ansible playbooks which set up the machine and we are using Ansible also to template all the releases that we support. And once we want to update the new Lightly, we just basically rerun the image generation and rerun some processes to get the image incorporated into the CI. But we don't have any CI on top of that just to make sure that the installation process doesn't break on all the systems that we basically support but it's definitely a good point to have. But currently, I don't have so much experience that I would see any problems there. This is just where we are heading, using Packer together with Ansible and basically creating those Packer definitions from Ansible templates. So templating. Yeah, do-do maybe ever do two-level Packer builds. Like let's say you have some base image, then you build on top of that some other image and on top of that another image because one scenario that I experienced was installing Visual Studio on Windows virtual machine takes really long. And if it becomes part of every build from like the grounds up, it becomes horrible. Very slow. We are using QCals directly that the release team of REL produces QCals and we take it as an input that you give it directly to Packer and just we just do one layer basically which is set up with those Ansible Playbooks that I talked about. So that's what we do. We don't need multiple stages. The one layer is completely enough for us, for our usage. I know the company that gave me this shirt was responsible and uses a lot of REL. It was responsible. It does a lot of that layered building for their configuration. So they expect to have a lot of a lot of configuration requirements but all of them are incremental. So, and that's where the side build that I was talking about, Kiri, is it becomes complicated because if you separate out, so if you deconstruct the machine image from the EC2 Ami to the EC2, just deregister the snapshot from the image and then re-register it, you run the risk of removing things like billing product codes that are assigned from the REL team or that kind of things of that nature. And the same thing for the Windows instances, for the Windows instances, you can decouple the registered metadata from the snapshot. And then when you re-register it, if it had metered billing associated with it, then that metered billing can be in a complicated way lost. Eventually it'll be re-established, but it's not immediate. But cool. Yeah, that's a great question. And great answer, Miroslav. I appreciate that. Thanks. Okay, I will have to run. So thanks very much again. See you. Cool, yeah. So I added in the CloudNetConfig project. That's another, that's one of those great projects I think that doesn't get enough credence. And then that whole team, this, you know, the SUSE team that works on this SUSE insulators project, they've done some really, really fun things. And these are, they include some utilities for import and export of images or modification of images based on migration, snapshot migrations. And another one of my favorites is their image utils. And they have them, thanks too much for being here. And they have that, those image utilities built so that they can take a configuration that a customer has for like a virtual machine that they're using on VMware or KVM or whatever. And they can just point to the machine image that they want it to be associated with. They will bring the snapshot up, copy the contents onto the, from one root FS to another. And then that the file systems will connect and copy. And then you'll have an image that is associated with the operating system that you expected it to be and potentially you persist metered billing in a consistent way. And it's really cool. It's all written in Python, so it's a lot of fun to read and a great, great part of our open source community. And then all of their, they have like all the things they use for their image stream, that also is published and their, the tools they had for, they have built around that are all named after, they have beer names, beer associated names. So their information tracking tool for their images to find a specific image or a regional server that serves updates like their CDN. It's named pint, you know, that information name tracker is called pint. And all of the stuff that they were, that their image validation software that they use for determining whether or not systems are running correctly on different images. It's called the IPA image proof. But it's, yeah, but it's IPA. And you've got the pint tool and IPA and those tests and take care of the images. Anyway.