 Hi, I'm Stuart. I'm a Principal Engineer at AWS and talking here about sort of Amazon Linux 2022 in Fedora and where we're sitting there and where we're going. So a few things going to basically try and cover a few topics here, sort of, who am I? What is Amazon Linux 2022 and why is it based on Fedora? And what did we change from Fedora? What's the delta between sort of when you look at Amazon Linux 2022 today and various Fedora releases and how are we thinking about making that delta sort of easier in the future and thinking about Fedora as an upstream? And what did we add? What did we actually change and what did we add that sort of isn't in Fedora and what could possibly go upstream and how are and will we work within the Fedora community? Can naturally sort of see the chat at the same time as well. So, yeah, we should be good with any sort of comments and questions along going on. So who am I? I'm a Principal Engineer at AWS working on Amazon Linux and I'm the tech lead for the future of Amazon Linux. So that includes current and future versions and past lives have sort of been very involved in the various free software ecosystems. I spent a good amount of time in the depths of firmware for open power systems. So you got that luxury of getting to the OS to boot was always seen as an adventure and debugging things, being like, well, you've got no interrupts and you've got real mode on. So that'll be fun. And before that, I spent a lot of time inside sort of the MySQL database world, like really deep in the internals of MySQL and other such related databases. And now here I am sort of working sort of at the OS layer, which has always been something I've been close to in one way or another, which is fun. And so a bit of history of what is Amazon Linux? Let's start off and go like, what is it? And what is it being? So first I thought I'd go through a bit of history of how we got to now. And so a long time ago now, like a little over a decade ago, we as an AWS, I released the first version of Amazon Linux and it was designed to be focused for the use in cloud environments being a cloud provider. And a bit later, as in 2017 kind of era, we released Amazon Linux 2 where we changed the release and support model. And this was our first long-term support version of the operating system. And current status of this is that Amazon Linux 1, so this first series of releases is in our maintenance support phase. And it's due to sort of be completely end of life in June 2023. And AL2 has life until we just recently extended it to end of June 2024. So those sort of the current versions of the operating system. And in the AL1 lifecycle, we'd release a new Amazon Linux Army at roughly every six months. And it was somewhat of a rolling release. I handwritten slide, yeah, with a pencil on an iPad. So it's really sophisticated, which is lovely. And the customer experience for Amazon Linux 1, this is the first time anyone's ever accused me of having good handwriting. Customers need to upgrade their OS every six months for having a new version of the operating system every six months. It's like, you have to go forward. Went to a lot of effort to make this as smooth as possible and offered sort of an in-place update for longer-lived instances. But there still could be major changes, right? You could still do a lot of changes in there that would require you to sit down and re-architect part of your application or do work in order to get there to the new version. And there's a lot of positive customer feedback on this and that they love sort of the freshness, right? With frequent releases, here is new package versions, new language runtime versions, keeping up to date with sort of new Linux things there. It's like real positive things. And within those six months between releases, you get security updates from young repositories, which is kind of nice. But there's a couple of interesting properties that happened in there. We have the ability to launch an Army and then do things on launch. And one of the default behaviors we had in AL1 and AL2 as well is that critical and important security updates that weren't the kernel would be applied on instance launch. And for longer-running instances, you know, yum update, either to grab security updates or to upgrade to a new major release. And increasingly commonly what we saw is customers would just launch a new Army to migrate to a new release or get security updates as well. And this rolling release model that we went from the entire lifetime of AL1 saw significant changes to the OS. There's some things which are hard to do, sort of an in-place update like switching in at systems is a good example. You know, the upstart to system detransition would be a bracing thing to do on sort of instance launch or in an in-place update. And so there's a few reasons that we looked on doing a slightly different model for Army Linux 2. Because ultimately like an OS upgrade every six months doesn't suit all use cases. And so the idea of sort of a long-term support of Amazon Linux was born. And this is what Amazon Linux 2 is. It's also seen a mix of changes throughout its lifetime. And that's both in the core Amazon Linux 2 repository and sort of our extras repositories, which is a collection of different yum repositories with alternate package versions for a number of things. All of these changes come to the various sources. Some of them are us making changes to packages. Some of them are, hey, how do we get this newer version of PHP that's packaged in Fedora and build it against Amazon Linux 2 and put it in a separate repository? And some of them are sort of unique things. Here is a new package to support some EC2 feature such as maybe EC2 instance connect or something. And there's some changes that you can't do inside an LTS release, like no matter how careful you are, right? Changing your version of libc can be pricing, changing sort of, you know, jumping a lot of system-based stuff could also be interesting. You're switching from like yum to DNF is not something we could do inside, you know, a major release that had a long-term support and stability model for our customers. So we had to ask ourselves some questions and we had to ask ourselves one big one. What's next? And to do this, we had to think some pretty serious and fundamental questions, right? We had to think of the fundamental question of like, with the rise of containerization, is a general purpose Linux distribution even still needed? Spoiler alert? Yes, it is. I think it's vitally important that we still have a general purpose Linux operating system and that be useful both on, you know, bare metal, so to speak, or within an instance of a virtual machine, as well as to be able to run inside a container and other environments. And containers are that key use case we need to think about both as a container host and what's inside it, right? And we had another OS offering we're working on and that is now out there is another open source operating system called bottle rocket. And it was the outcome of thinking heavily about running instances that host containers and operating that at scale. So it's a Linux operating system designed to do one thing, run containers, that's it. No SSH, no shell, an API driven host environment, DM Verity read-only root file system. And we're also naturally still going to support, you know, running containers on top of Amazon Linux. But we had this like nice opinionated view of like, how would you run containers at scale? And we just had these other questions were like, what about what you run in the container? How do you do that? How do you get people from Amazon Linux to to something that has sort of, you know, more new things in the Linux ecosystem? And we had to ask ourselves fundamental questions like, what did our customers like about a one? What did they like about a two? And what didn't they like? And we can also look, we looked introspectively and went. So what was painful for us in creating and maintaining Amazon Linux one and Amazon Linux two? Did any of that pain result in customer benefit? And if it didn't, then how do we avoid that pain? Right? Some pain you want to take because it is a great benefit to customers. It is not necessarily convenient to to respond to an urgent security issue, go and patch that and get it out everywhere. It can interrupt your day. And so there's a sort of amount of pain there, but there's a big customer benefit to that. And it's something that that our customers are very sensitive about is to make sure that, you know, we can timely, in a timely way, respond to problems. So we ask ourselves a bunch of these questions. And we came up with a few ideas that were sort of fundamentals that we were going to have to think about. Customers taught us they like the predictability and freshness of Amazon Linux one. And they like the long term support of AL2. And so we started to think about a model, where do we need a predictable release cadence, where we could inform people well in advance of what kind of change was coming and when. And then they could also plan when they'd adopt a new release, when they might get a new feature, and how long it would take to migrate from one operating system version to another. And so we're starting out with Amazon Linux 2022, a new major release every two years, security updates on each of these for five, and some ways to introduce sort of new functionality along the way. And a lot of things that we think about as sort of big customer benefits that customers have liked about sort of AL1 and AL2 is that they do care about sort of the availability of new versions of language run times and a bunch of leaf packages like databases. You know, I want a new version of Postgres, but I want to necessarily update my operating system at the same time. Or I want a new version of Python to run my application against, or I want a new Ruby version and things like that. And we're still going to need to have some things like Assistant Python, especially as DNF is written in Python currently, and that's what we ship in AL 2022. But we can often offer these newer versions that customers can opt into. And a lot of the time customers are pretty happy with roughly that upstream support life cycle for those packages. If you're writing code in Python all the time, you think, when is this Python version supported too? And that matches with a bunch of other ways we vend Python run times inside the cloud. And we've done this throughout sort of AL1 and AL2's life, right? You can sit down on AL1 and install several different versions of various packages, and you can do the same in AL2. And the mechanism for choosing how you do that has varied. In AL1, we did a lot more. Here we are namespacing the package, often even making us to install the multiple versions at the same time. In AL2, we did more of a yum repository approach where it's not namespacing packages, but here is a yum repo with a bunch of details in it. And so the mechanism there has varied, but the customer benefit has been the same. The mechanism for communicating, hey, when is the end of life, right? If you're going to communicate, hey, we're going to ship Python version and support it for the length of time that's upstream Python would, we need to communicate when that starts and when that ends. And that mechanism has varied as well. And with AL22, actually, we start vending metadata about support statements for packages. This is something we're going to have an interesting conversation with, because there's some applicability here to Fedora as well. When you talk about how long is a Fedora release supported for, or maybe you're building a repository for a software that's built for Fedora or Amazon Linux or any RPM-based Linux distribution, you want to communicate in a programmatic way, not just, hey, an update exists, but also this particular version will never get another update we're not looking for. And so we've done some things with a support info to say the opposite of update info, where it's not just, hey, here is an update, but is exactly the information about what is the support statement to these packages. Maybe that's the way we can talk about different life cycles for some parts of the OS. And you might have extended support periods in there where you do different types of fixes and different things are in scope. And we also look at going like, what are these quarterly releases here, 2022.1.2.3.4, etc. And our idea here was to have an opportunity to introduce new features, new packages, new instance type to support, and really think about it as kind of semantic versioning. And it's not really a change from how we've done things on AL2 previously, but it is sort of aligning to a schedule and having a communication mechanism for these changes. It's no more, am I ever going to have to get the question or repeat the answer of going like, well, what you need is an army from like November 23rd in order to use that new feature, not November 7th is the one you have. It's way easier to say something like, you need Amazon my next 2022.2, because it produces support for it's way easier for humans to understand. And this is something on a longer timeline that definitely makes some sense. So we have this sort of interesting model of what our customer needs are for support life cycles and introduction of new functionality within those versions of the operating system. But getting back to sort of a big question of like, why buy some Fedora? And this is again, it's somewhere where we had to ask ourselves a bunch of questions. And when we're thinking and sitting down and going, well, to build a successor to Amazon Linux 2, we had to ask a question. And that question was, what is an RPM based Linux distribution that's familiar to existing Amazon Linux users? Right, like, well, we love Debian. I think it's great. We also viewed like, you know, such a fundamental change to be a little bit too large to do for a new version of Amazon Linux. We have large internal and external customer bases that need to come with us and we want to have a smooth upgrade experience and update experience for them. And familiarity is good, right? It helps customers migrate and it helps that be minimal effort. And so if you think about this or it's like, what is an RPM based Linux distribution that's familiar to existing Amazon Linux users? Like, it's actually a relatively small list. There's CentOS, now CentOS streams, there's REL, there's Fedora, and there's Amazon Linux. And there's also open SUSE and SUSE are off to the side as well. But we know that there's various differences there between open SUSE and Fedora that could mean it's a larger step. And so we're thinking about, you know, what do we start from? And we could start from any of these. And we also want to ask ourselves, what is an upstream we can participate in? Participating in upstream is important for several reasons. Like it's everything from small things like, hey, we can add some packages. And thus, you know, we get to care about that in what's coming up next in that upstream. It also is useful to think about what big changes could we participate in and maybe drive some larger changes inside the operating system that would be both beneficial to our customers and the upstream Linux distributions and projects. And there's some really specific things about Fedora that align nicely with our needs and our customers' needs. And I want to bring it right back to sort of the four foundations of Fedora. And each of these is sort of a good reason why the Fedora was sort of the right choice for us to be a part of. And software freedom is naturally pretty important. It means building on top of Fedora and contributing to Fedora is a well-defined problem. Contributing to an open source project is something that we're getting really good at as an industry and that we can really easily reason about and think through. And also, if you read the text there on the four pillars of Fedora, that's right up there is saying Fedora is a completely free project that anyone can emulate or copy and hold or impart for their own purposes. And it's nice that that's sort of right up there and stated because we can look and saying like, you know, is Fedora designed to build something on top of? And the answer is yes. Right there, one of the four foundations of Fedora. And having the ability to contribute back to that project means that we can also drive changes in Fedora that both make it easier for us to build a new Amazon learning so that having to do the same work all over again. And that we can focus on empowering ourselves and others and Fedora itself to make choices and changes to suit their own purposes. So there's a bunch of crossover there, even just sort of maintaining Fedora that gets interesting. The Friends portion and foundation is also important to us. But Amazon, we have a set of leadership principles and one of which is structure the Europe's best employer, which starts off by saying leaders work every day to create a safer, more productive, hyper-performing, more diverse, and more just work environment. And as such, Fedora's strong and caring community aligns well with what we want our work environment to be in our day-to-day lives. And that's something that is quite valuable to us as well. And then we get to the other two pillars, features, where we look at what Fedora thinks about this and think we're working up with the upstreaming cases where we find opportunities for improvements or free software users benefit. And that really strikes a chord with us because we don't think along the lines of how do we get everybody to use Amazon Linux? That's not our thought process. Instead, our thought process is more, how do we make EC2 the best place to run Linux workloads? And often the answer to that is work upstream. Customer is going to pick what Linux they want to run and how they want to run it. We can offer here is an opinionated set of things in all packaged together nicely in Amazon Linux. But there's also no reason that we shouldn't have the ability to get various optimizations and integrations into every other Linux distribution so that it's all a smooth experience for customers running workloads on EC2. And so that upstream mentality is something that really also strikes a chord. And even the fourth pillar here, first, and the Fedora commitment to being first and the simple statement of we're committed to innovation also aligns with our leadership principles of invent and simplify. We also have to think about in the context of an operating system that has a longer support model where it's not all our customers want to be continuously moving to the latest everything, but they sometimes do want to peek in certain directions into the future. And there is that interesting blend of how you do that. And that's where we cut them to. What does that release lifecycle look like? What does that support model look like? Or how often do we do a new major release? What things do we backport into existing versions? Like maybe, hey, we can use a new version of PHP or Ruby or anything. I think that's an important thing we can think about. It also means that we have an upstream distribution where you're really thinking about what does the future look like? Where everyone's opinion like what are we doing forward and does everything still work together nicely? And having that development branches is really nice to be able to think about what would a future Amazon Linux be. And that's sort of the interesting thing when we look at, you know, whereas we're all high today, it's like, well, is this where Amazon Linux needs to be in a couple of years time? And that's a really interesting thing for us to have that and have that feedback look. So let's talk a little bit about, you know, what do we change? What is the difference between Fedora and Amazon Linux? And it's interesting to think about the decisions that are made in Fedora of like what packages to include, what versions of the packages to include, what options to enable, and how long to support them for are made in the context of Fedora, which is naturally different to the context of Amazon Linux in some ways. And sometimes those decisions are made at a higher level, like here is a Fedora enhancement proposal and it's a large change. And sometimes they're made on individual package maintainers level or it's like, you know, what options do I enable or not in this? And sometimes that's informed by the full breadth of packages in Fedora, which may or may not be a full set of packages that you want to ship in a downstream distribution. And with our release and support life cycle and Fedora is right, a lot of decisions are made in that context. And they're going to be made very intentionally in that context, right? The big questions of like, what currently are you going to ship? What about GCC versions, G-Lib C version, system D, what versions of open SSL or only one, what versions of RPM and DNF? You might make different decisions here when you're thinking about this OS has a new version in six months and then there's another six months afterwards of supporting the previous one versus you're starting off with five years and you need to think about what is the compatibility story like for customers during those five years. So these different release and support models of Fedora when compared to Amazon Linux can necessitate other choices too, right? Like not every feature in Fedora is one we want to support for five years. As we don't have the flexibility of Fedora is making those decisions, right? Like it's very easy for Fedora to make a decision way to release the two and go, huh, let's do it somewhere somewhere else. Well, sometimes we have a longer life cycle here, right? We'll be stuck with it for a number of years. We want to be very intentional. And in Amazon Linux, we've made some decisions that are that are different based on customer need. So there's some packages we move forward a lot more frequently than others. And so sometimes even like bundling dependencies sort of make sense in our context. And that's simply because we want to think about what dependencies to expose to customers that they might end up relying on. And when we want to change that, is that going to be breaking for them? And so this is interesting, especially when we think about sort of the GoLang and Rust ecosystems as well. And how we think about packaging those in a way that is worthwhile for Fedora, where you're trying to have one version of everything versus maybe we want to move different packages that depend on different versions of modules and crates in sort of Rust and GoLang world and move these at a slightly different pace inside the operating system. So sometimes these decisions we have to think about it could mean that we ship a packaged version that is more aligned with sort of CentOS 9 streams than Fedora. And it's fantastic to be able to sudden pull requests there. It's going to be interesting to see how that practically evolves as a collaboration base for some bits of software that you're going to probably want to keep sort of a similar version for a long time. And some of the practical considerations here and changes from looking at sort of any particular version of Fedora today is like Braille 2022, we're shipping out 515 kernel and one that's sort of as close as possible to the upstream stable branch. We really try and either grab what is stable or what are things that have headed upstream already that we can back port back and be selective about it. That's something that's really of an advantage to us. We made the choice that OpenSSL 3 and OpenSSL 3 only was the way we wanted to go. We didn't want to have to ship OpenSSL 1.1 or earlier or anything that relied on it. And that's been an interesting transition over our tech preview time as Fedora has grappled with the same thing. And for things like RPM and DNF, it actually made sense to line pretty closely with the versions that were currently in CentOS 9 streams. Is that sort of is a good place to maybe collaborate there and have a good source of stability for a period of time. And we also did a lot of changes where it came to sort of package curation. We didn't build all of Fedora or start with all of Fedora. We start with a subset of Fedora and a modified subset. And this subset is informed by a number of sort of insights into what are our customers used today on Amazon Linux 2? What are they using on Amazon Linux 1? And what might they want to use in the future? Like we know what we have on A2 and we know what important parts of that are for our customers. And we can also think about packaging something that nobody uses is just a five-year maintenance burden for us. And we can be very judicious and thinking about, well, if it's got no downloads from the young repositories, why is this here? And have we been security patching it? Because if we have, we've been doing that work that's non-differentiated and is providing no customer benefit. Not patching security vulnerabilities is certainly a life choice. And we want to be able to think about where are we going to intentionally spend our time patching things? And we should really focus that down on the things that people use. And so to take an example, until we're running in a whole bunch of places with the Bluetooth stack, we can probably avoid shipping a large amount of Bluetooth software that interfaces with Bluetooth. And that lesson over a period of time is like, if the answer to the question of is anyone actually using this is no, perhaps you should rethink about why we're shipping it and how long we're signing up to support it for and thus having to security patch it. So we're really trying to focus in on what our customer needs are as well. And so some specific choices here is we're shipping Docker and container D for container runtime, and that's what our big container servers rely on. It makes a lot of sense for us to align there and maintain that in upstream as well as in the OS itself. So we're interesting on the discussion there is like sort of how to deal with openness of self-dry and everything in there. And it's a mixture, right? Like one of the amusing things is to look at the dependency graph in Fedora as a, you know, all bells and whistles included and Amazon Linux, and you can sometimes weirdly sort of move faster and be first on some things in a smaller set of packages than you can in Fedora. And something like that is like, you know, not having open an SSL one in the distribution. And that's sort of an interesting thing. And it's even looking at sort of the, you know, PCRE version one versus version two, and how close we can get to completely getting rid of that mail 2022 versus in Fedora. And sometimes it's just like, it's less work to do it on a subset. And it's like, how do we think about this as something that we can do both sort of in Amazon Linux early on or in Fedora, and how do we make this kind of an easier process is definitely something we're thinking about into the future. And I were written close, pretty close in NL 2022 as well. Those are the last three things on the Fedora developed mailing discussion, like, oh, quick, maybe I can actually do it, which is kind of fun. We also made choices such as going with like Amazon Coretto as the JDKs that we ship and build with, as like, there is literally a whole team, like a whole bunch of people, you know, working at Amazon maintaining these JDKs and making sure that they're up to date, security patch, performance compatible, and that we'll build, you know, what we need it to. And this is what we use in production on a lot of systems, like we should definitely think about taking advantage of that team existing. And there's that idea where it's like, how do we bring the most benefit there? And Coretto is definitely one of those ways. And it's actually not too hard to get that going to a Fedora-derived OS. And one of the future questions will be like, does that make sense as an option in Fedora as well? And what are the sort of the minor random smattering of spec file changes that are needed to make it work? And sometimes that simple things like, you know, build against build dependency be, you know, Java 1.8 or Java 11, not Java 11 Open JDK, like little things, like little things is the stuff in there. It's just the, you know, the whole bunch of paper cuts that you go through to make it happen. And we also have the philosophy of where we can sustainably introduce choices that customers have asked for. And these choices don't break existing customers, like we're all for it. Right? It's about thinking about what is, what is something we can support for a length of time? How do we communicate that support lifecycle? And is that sort of a good quality package to sell off with? And sort of what is the benefit there? It's a lot of usage, then that's something that's really attractive. If it's like literally for three people, we might want to consider it, you know, really hard about how we ship it. And if we ship it at all, we also do a bunch of like package options of what we include, like, and I'm talking here sort of like, you know, maybe sometimes beacon level stuff, but sometimes it's things like, do we start off with the container image and armies with the full fat curl? Or do we go curl minimal? We took curl minimal as for AL 2022. And that's simply to have a sort of a smaller dependency graph that for the overwhelmingly vast majority of customers is perfectly adequate and perfectly fine. And this means that hopefully we end up with sort of less practical security patches that need to be applied to a large variety of systems. And that's sort of a big benefit for our customers in there. And it's always a little dance away with a DNS command to get it back. I have been meaning to for a little while now send out a pull request for doing the same thing to GNU PG2 and have a GNU PG2 minimal. Because that's also where we're going with AL 2022. And if you look at the dependency graph of the differences between GNU PG2 and GNU PG2 minimal, it's like, okay, am I really wanting to ship a container that's an extra like 10 megabytes in size, just in case someone wants to use a USB smart card with GNU PG in there when really what they're really using it for is to verify signatures on RPMs. And that is an interesting set of decisions and thoughts to go through and really examine that dependency graph and what choices can we make here that might be a bit more aggressive than sometimes can be done in Fedora, but also could be something that is reflected in Fedora and makes its way up there as well. Like, I think it'd be interesting to have that sort of GNU PG minimal discussion. And maybe the long term thing there is instead of using GNU PG at all, like just do the verification like straighten RPM itself and DNF. And so we've also done a number of choices to do things to like improve instance launch time. And so these things are perhaps things that should stay downstream and then shouldn't go upstream as default. And one of them is like, you know, pick a random example like we built Kmod without a bunch of support for compressed kernel modules, right, kernel modules are compressed in RAMFS, otherwise they're decompressed on disk. And the reason we want to do that is like, you look at the content of the unit RAMFS and go, what's dragging in all these compression libraries, and you're like, well, the ability to load a kernel module, it's not compressed in any way because it's compressed in a whole unit RAMFS. It's like, well, if we don't build it with that option be counted out, guess what, we start to save a whole bunch of space on the unit RAMFS, which you might not think of as too much. But you think about when is the unit RAMFS read from disk? When is it extracted? And where is that in the boot process? And you suddenly realize that all of those things are serial parts of the boot process, right? You're sitting there and grub loading something off disk, you're then serially, you know, unpacking the unit RAMFS and uncompressing it in order to then start booting everything in parallel. And this really like makes a natural decent difference. Like we've got a little configuration file called Drake at EC2, which is like a Drake at config that we're thinking about going, hey, what do we not need in order to boot on a on an EC2 instance, or at least the overwhelming majority of launchers there. Like some people do want to do LVM and RAID and everything in there, we can support that. And maybe it makes sense to have, here is the way that you might modify to do that. And it often makes sense to think about, is this something you actually want to boot from versus what you want for data audience in that there. So we get to make a bit of different choices there. And that's, that's sort of an easy one. That should just be a package that sort of heads up stream and sits there. And we can have the discussion of thinking like, hey, what do we think about where we have sort of the Fedora cloud images and then that they should sort of bend in a similar direction. Yeah, it's also like, what do we do and copy over. And like the Kmod stuff, you could quite easily say, like, well, try and load the shared library on demand rather than linking against them, for example, could be it as well, combine it with that. And it's one of those things that could make a lot of sense to do. And in this way, at least we have this good proof where it's like, this is a benefit to customers and instance, more time there. But I don't want to think about, you know, what, you know, a second of launch time equates to an enslaved customer cost, like that is, that is definitely something to weirdly think about. We also make some other choices in sort of what package options we enable, like, we're like system D network D and sort of Amazon EC2 net NewTills. So we can have this like really small and efficient sort of dependency tree for configuring a network. And part of the joy is there where we're thinking about the concept context of like, what is this inside EC2? And how do we make that be as small as possible? We had other decisions like, you know, we didn't want to ship GTK one or GTK two, and so we're able to go through and sitting and be calm down places that end up bringing things into the dependency graph. Like often there is like a GTK three or four alternative or it's like, you probably don't need this option for a mostly cloud environment. And it's also one of those fun things where you see some of the same activity occurring in Fedora at the same time, where I think it's, it was about the same day that the patches landed to no longer have like, you know, X MMS and GTK one come in because you were building flak was about the same day it already done the spec file work sort of randomly inside to Amazon. So there's like a lot of sort of lining up there where it's like stuff that could be like, how do we think about doing that? And other things that we wanted to, yeah, flak requiring GTK one through X MMS was a past from the past. But our customers want to be able to move to sort of newer versions of some packages on a separate cadence to the OS. So we do currently do this through namespacing packages. So you can have like multiple MariaDB or Postgres, PHP, Python, Ruby, etc. And so we do some of the base work required to support this inside the OS that isn't already there in Fedora. And I was selective about what options we build for each of these. We also think about sort of OS features, what we want to add and modify in there. And one of the things we've done with with AL 2022 is make a fundamental change in how we think about yum repositories and armies or container images. So we lock each army and container image to a specific version of the repositories. So there's this snapshot of all of the packages you could possibly install and it's tied to that army and that container image. And there's a method to move forward in that snapshot. And really what we do is called version locking because we give the OS a version, right? You have a full version number, semantic version number, 2022.0.2022.07.28 is one of them. And then we can talk about that in release notes. We can talk about it in what the name of the army is. We can give it like a container image and register that with a nice name in there. It's the version that system release has. And it also means that we move a lot of this sort of atomicity of moving forward or backwards between what packages versions you're installing right into the customer's hands. It means that you've got a way to move forward on your own timeline and a way to move backwards if there's ever a problem. So you can easily go and lock boot an army from last week or last month and you get exactly the same behavior you have there. So if there's any negative impact for some update, you've always got a way out to get there. And you've got an easy way to get out. And it's a single little thing to move forward or backwards, either container image or army or machine image there. And this is a pretty big change with Amazon Linux one. How does it change technically? It's interesting. What we do is we put the release version in the URL for the mirror list for the yum repositories. So system release has the full version number in it. And when you go and poke at the URL to where's my repositories, there it is. That's the version number four. And what this means is that when you want to move what version you pin to, you just upgrade or downgrade system release if you're within an instance. And it just means that we build new system release every time we do a new release of a package in there. It also means that we have to build a lot of new armies and a lot of new container images. But it does mean it's worked for us. But it means it's simpler for customers. You just have this one top level object that you have to care about of moving your OS forward or backwards. And it even could make you think about where it's like if I have an SLA to patch vulnerabilities within a certain number of days, as long as I cycle all of my instances to the latest army within a different, within a group of instances, guess what? Everything's up to date. You want to patch within seven days, all your instances less than seven days old and they're using latest army. Guess what? You're fine. And just do that. So yes, each system release version sets a different release first time. So everyone, you can look at the history of our system release in our 2022 and they're all different version numbers there. And it just comes in there. There's also a DNF plugin that says, hey, there's a new version to move to. And then you can do that and give information or link to release notes and the like. And no doubt the sort of different ways you want to tune this into the future to get more obvious to people in there. So it'd be interesting to get people's feedback on it practically works. It's also a very sort of different model when like a lot of the time, everyone is doing like here is new instance launches rather than, you know, I have a laptop running an OS and I'm running things backwards and forwards. But it may be something that does to think about in the context of like, could this make sense in Fedora? And what would we want to do in order to do a similar thing in there? And this, this sort of pinning is something one of the largest sort of operational lessons we've learned over the time of shipping Amazon Linux. And it is one of the things that could be quite beneficial for talking about what changes come and when, as well as having options for, oh no, something went wrong. How do I work out what went wrong while still recovering my production environment? We also added some packages. Another thing we ended up doing. So there's a whole bunch of sort of various AWS agents and services in there and not all of these in Fedora yet. But it's something we're working on and are motivated to fix and resolve. There's a bunch of different RPMs that we ship here that just need to get into a state where it can submit a minute's new packages to Fedora and maintain them there. Pretty simple. If people have some review cycles, that is also very helpful in there. And we're working internally with teams as well to get them there, which will be good. We also made some choices in sort of our tool chain and sort of what base architectures we're building from. So for the ARCH64, so ARM64 bit environments, we're building for Graviton 2 as a minimal processor. There's a bunch of nice sort of attributes that we have in Graviton 2 and that CPU architecture that gets this performance across the board. So we built as Graviton 2 as a minimal CPU type. We also came to a conclusion that sort of x8664 v2 as a middle platform makes a lot of sense. Turns out, REL was making the same decisions. So that's a nice coincidence of events. We're shipping sort of GCC11, which is not the latest version of GCC that's there, but it's also a nice stable place to start at and keep on for a while and has been for the preview lifecycle. We do make the change of building everything with auto vectorization turned on. So it's like that default that comes with GCC12. It's like, we've done that in so GCC11. That seems to be working pretty well for us. We're also really modern on our sort of go lang and Rust and LLPM tool chains. Maybe switching everything to be based on your LLPM is something you do on a release boundary. It turns out we have a release boundary, so we're actually like LLPM 14. And there's a lot of similarity with some Fedora branches here and some are a few different choices. And some of that is just planning when we're cutting a release and then we can make sort of changes coming up. So what about working with Fedora? It's been an exciting thing to start. It's one of those things where you're like early on before you announce, like should we send some patches and see what happens and if anyone notices what we might be doing. But it's also exciting to sort of work sort of more and more in that and sort of see where we grow in that regard and nurture that relationship. And some of these ways are more sort of visible than others. And this will naturally evolve and develop over time. Like right now, I look at my Fedora accounts, like there's a bunch of open and merged pull requests for like every random spec file bug fix you can think of. Some is like adding GPG signature verification to some packages that could have it, but didn't. Some of the things are moving some options to sort of a beacon. So it's easy to flip one way or the other. And the patch we carry downstream are a lot easier to continue to maintain. And there's that suite of AWS packages that wanted to ship as well. And that should start to move up as well. One of the things that we had as a really strong desire was to ship AWS CLI version 2 as part of AL 2022. And talking to the AWS CLI team were like, one of the conditions here is that like, we got to get this and all of its dependencies into a state where it could live in Fedora. And the real motivation there was like, so it's like, otherwise you're maintaining AWS CLI v1 for another five years. And so it's that level of work we have to go through and talk to you, the various new AWS CSDK teams, S20, LS and the like. And we have to think about what do we want to do there in order to get things in the state to move and get them into Fedora as well. A lot of things that we look at also involve having a Linux distribution that works well in the cloud and integrate well with cloud services. And so it's one of those things where we want to work out, like what can we do here that's sort of like maybe a little bit of work right now, sometimes a bit of an adventure, but we'll have full dividends long-term, right? Like if we get all these packages in a state where they can be accepted into Fedora, it makes it much easier to then package for Debian and look at other distributions as well. And so that's sort of a really nice thing of what we do with some of the fundamental work internally. And big comms is almost worth its own little section. I'm saving everyone's eyes, they're not copying and pasting some of these conditions because you look at it and you look at like, okay, what are some of the changes we've made and what does that look like inside some of the spec files? And you look at it and you go, well, there's got to be an easier way and a better way than having sort of nests of if Fedora, else if rel, else if Amazon, else if other and doing it in, you know, five, 10, 15 different ways. And it's especially got to be easier when you think about, well, what about when we want to do a merge or a rebase, even like within a major OS release or across them, would it even be possible to go, hey, all of these options we've set in here, what if we just said, try and apply those same options to Fedora, raw hide and see what falls out? Like, how would we even do that? And you look at it and you go, like, you know, what patches do you want to send sort of upstream and you're like, making the if then if else nested beacon stuff, like, we should just find a better way and make it cleaner and do it that way. Yeah, Fedora ALN would be interesting at something like we thought about like, how do we do this internally and think about that and know sort of what are the bits we're going to have to concentrate on to reduce our, our our lifecycle of building something. So we've been thinking about things like, okay, now that we've got something that's like, here's a release candidate of an operating system, we have a really good idea of like what things we change. And maybe some of these things should be yeah, like some nesting if L and like, sometimes you look at it and you go like, well, this is already in the spec file, how much of a complexity do we want versus to clean up everything. And so it's going to be an interesting route that we go for there. But maybe we thought about some of these things should maybe be distro level flags in a way and that way we could possibly introduce and remove sort of optional functionality from packages, even within Fedora or release boundaries, like a world where we start removing something like G2K2 or PCRE vision v1. Could we start by sort of flipping a distro level beacon, you know, disable it build time where it's an option and then rebuild the world and see what falls out. Like some of these things could be sort of interesting approach instead of just progressively go through and examine the dependency tree more and more and realize that oh, we can beacon this and then you need to do it a beacon because different levels of Fedora you're maintaining the spec file for have different options enabled and the same thing for Amazon Linux. Like there's probably a few different approaches there where you can get something that looks a lot nicer. We're also thinking maybe some of these things need to be local on a per package level. It's pure functionality within a package. It's not related to something sort of a distribution level feature. So maybe that should sort of sit in sort of RPM auto spec or similar where you could have a, you know, AMZN beacon file that sets some different options than the Fedora one, then maybe the EL1 or whatever. And maybe that's easy and that's could simply be because it's then easier to get merge and get rebase or have a downstream distro that's making different choices or not just having everything sitting in Fedora just get as things scattered around. We already do something similar where we have an AMZN changelog file and we just have a Koji plugin that goes and mergers them. We didn't want to delete all of the changelog files and inspect files. We want to, you know, acknowledge that that work has been done by others in there, but we don't want to constantly have get merge conflicts and it's sort of easy to do that at build time because the diff is like really easy to review as well and really easy to manage. There's also things that we think about where it's like a distribution specific release number and how do we think about that we add sort of a bit under the disk tag at the end of the NVR and go like this is sort of our revision of things and it's useful for human eyes to compare, you know, what is the revision that is coming from Fedora and then what have we changed on top of it? Have we changed anything in there? And that's sort of interesting thing we've been thinking about of what might be functioning around the future. We also did work on instance launch times in there and we're thinking about what can we do and push upstream into Fedora as well. We've made improvements throughout AL2's lifetime and we've done similar efforts on AL2022 and made a few choices you can only make on a major OS boundary. And some of these are choices that could easily be enabled in Fedora. Like we could have a way of thinking around like I drag a config EC2 to only think about a way of like what do we build in to the cloud images by default. Even choices on like what our cloud in in a configuration is like we don't necessarily need to run cloud local when you're running in a cloud, how do we do that by somewhere you're booting and even what SSH hopes to take key types to generate like how much entry could be you have one boot to generate a whole bunch of SSH hopes keys which aren't ever going to use and there's other things like disabled compression algorithms and K mod and the like that may or may not be suitable for various parts of Fedora. So we're looking at like what can we do here and sort of you know push in Fedora versus as downstream patches as well. Need to look into how we're doing it in the hyperscale SIG in Fedora as well to see like does this sort of suit all of our needs and how can we do it. There's also things around container image size. Customers are really keen to have as minimal dependencies as possible both in a container image and on an army. I mean it saves time and space and you know shuffling a container image around extracting on disk and as well as having to apply patches when you have to update a package inside your container or your army. If it's not there it's really easy you don't have to do a patching run after a plate and so we managed to do some of these so like you know new curl minimal going to be G minimal and a couple of other tricks have gotten us down to the point where a nail 2022 container images a decent chunk smaller than a 21 and there's even a little bit smaller than the Fedora one. I think it's because of that sort of curl minimal and going to be G minimal stuff there rather than anything else. So expect you know probably in a Fedora enhancement proposal to go with the curl minimal for going to be G minimal is definitely something we could think about in the larger Fedora context is working on and so how are we going to work with Fedora? It's like well like any other contributor there's a bunch of people working for Amazon increasingly doing things about 2022. We're also got you know a bit of a backlog of some patches that could definitely go upstream and need to sort of work our way through that and reason about what's the best way to to get those to their upstream or that we should maintain them downstream for a period of time all around so you package selection there and curation and what do we do it's always in larger things we want to think about coming from that there's even thoughts of thinking like well we've got build requires and requires should we think about test requires like there's a bunch of the dependency graph that only comes into into play when you're running a check section and you're like running the check section is nice I like doing it don't necessarily want to ship every single package that we need to have have happened so maybe that's a thing that we need to think about as one of those bigger thing we could think about to make you know our lives easier in the future as well as the dependency graph be more explicit about why something is there. We're also like likely to touch sort of more components of the average contributor right that model where it's yeah that's sort of if with tests but also like you know check replies could also mean like well here's a repository out to the side with things that exist purely to run test suites and that we can move that entirely different cadence than sort of other packages in the OS like it's it's one of those thoughts where it's like of course it has to make it to the top of someone's priority queue and and to do it as a big pain point but it might be something that that you know can start to be chipped away at and think about how it would look so you know we're conscious of the fact that you're making for requests it does actually make work for people you know a bit of a backlog myself like it's that that balance between getting something to a level to release in the operating system and how do we get that upstream and balance that time it's always it's always a fun a thing to balance and there's some components right that we rely on and probably stepping up to help maintain them in fedora or maintaining them outright as the right thing to do is that's beneficial to anyone to everyone so and so there's some tooling also that we have internally that might be useful to open source and work out if we can you know how tight is it to internal Amazon services versus you know a benefit for having that out there and we've been looking at that as well what might be useful to do because a bunch of our processes are different right we don't actually have an installer we don't anyone spite an image somewhere and run it you know cloud in it is sort of like you know that out of box experience there for us and so that sort of means that we've got you know tools to like you know write an image from scratch and that kind of stuff rather than sort of running it through an installer and that's sort of an interesting thing to think about and as well as this existing tools to do that as well but you know we look at sort of what are the lessons we've learned there what are the things we don't like about our tooling versus what is what is that outside and whether it's you know should we go to the effort of packaging that up externally should we work on a new tool should we adopt some existing ones out there as well all those are sort of good questions we can look at in there so yeah we're expanding in here the current state of the world is probably an interesting one we've recently released what we call as a release candidate zero because of course you can't bring zero and that was released the other day this is sort of another step towards a general availability lit release of Amazon Linux 2022 putting the sort of finishing touches today on what a pull request looks like to the couple of things it's needed for mock config that could be up in standard packages and hopefully try and get sort of a al 2022 added to copper or anywhere else and start to do that one of the requirements for that was actually you know I mentioned version locking before and how are we doing uh which release for so it just means we were publishing you know per version number mirror lists of where the mirrors were we didn't have sort of a latest concept but we added that yesterday in there which is useful for such environments as not only you know a coji build environment going for it all that yeah I've been meeting the message in the other spin my bad instead of being editing these slides but in in that regard like we want to think about how do we use this for building stuff in there how you might integrate that with things that you want to know the the update info feed as well that's up to date on whatever the latest is uh and what's the easiest way to integrate there so we've got that um so with that I'll say thank you um the cheeky note of saying and yes we're hiring um is a good thing to finish on there but um thanks very much don't know if we've got this sort of other uh last minute questions there we've got a couple of minutes but uh thanks for much it's good to be here for our community oh q and i tab cool that's the q and i tab it's hard to click uh where are the package sources for amazon linux um you can easily uh grab you know using yum downloader and similar um the the source rpms uh if you're asking if we have a a pile of a git repo sitting around somewhere like not yet there's a few things there with we want to sort of solve first for doing that uh to make sure we could do that with care but we do have all of the s rpms available um actually we have each of the s rpms available on every iws region so you can fetch them locally and not do cross region transfer or from a cdm when can you get amazon linux and wsl 2 um it's a good question uh is it released yet um yeah amazon linux 2022 is currently in tech preview we have a what we call a release candidate out so we anyone can launch it in any aws region there's armies there's container images so you can easily you know run a docker or podman uh anything of your favorite container runtime i can grab a container image or you can launch armies in there um uh as well ga is later this year when people have asked when is ga and i said that well it's definitely in 2022 yeah for something like wsl 2 and other and other sort of out of uh cloud environments sort of something that's in the city on a roadmap and priority list we'll have some images up but uh i don't think we have any immediate plans wsl 2 we do have a github project to request features though so that would be a great place to do it um github.com slash amazon linux uh go for there there's a amazon linux 2022 project um filing issue there for wsl images and it's really useful to know that there is customer need that's what we run on customer requests podman run mail 22 it's what i do on my fedora machine all the time oh i should we have a logo like especially one of the team members printed a coaster on it that's wonderful um but we do have some sort of uh logo and stuff around which is which is nice for the first time sort of ever well thanks everyone i'm around um for first i've asked questions and say hello uh you know fedora develop on matrix and everything and on the list uh and would love to know uh stickers yes we do have stickers um you need stickers i'll mail you some stickers if you that was a lot of stickers we did we did have summit scale um there's some different looking ones in that fat box there um that we've got as a sample so more will come and sort of go as many places as possible as well again thank you