 Welcome to the homelab show episode 89 rocky Linux interview. So this is gonna be a fun topic We're gonna talk about rocky Linux what it is and some details about it And we have a guest that's will be on here to help really bring that together for us and me and Jay are pretty excited about this We we like some of these interviews We hope to do a few more with experts in the field and I'm gonna throw it out there Our guest is definitely an expert in rocky Linux. I think that's fair to say fair to say I am for those of you watching a live show. I'm jealous because I did get a rocky Linux shirt as well So did Jay But I'm not wearing it. I should I didn't think to grab the other one. I left it at my office But nonetheless, this is still gonna be a fun interview. I want to get our sponsor mentioned real quick and that is la node Yes, they've been sponsoring a show pretty much since the beginning. We've been really happy with them This show is hosted on the node. So if you downloaded this you visited the homelab show website You will actually be pulling all that data right from a Lenovo server that we manage and maintain We really thank them for being a sponsor to show and there's a link down below if you want to get started with a node It's a great place to host a lot of the projects we talk about on here And just want to thank them again for sponsoring a show and keeping things going All right, now let's bring in our guests Greg Are there and hello, I'm doing well. Thank you. Thank you for having me See iq rocky linux. So tell me Greg. What exactly do you do at rocky linux? Um, I do all the stuff that I think nobody else wants to do. Oh, oh wow run the place less on stuff You know, I used to be like a developer and in my my knowledge used to be relevant When we started up rocky, I realized that my age has caught up with me and um, yeah, so I'm I'm doing less of the development and tech with with rocky and more of just kind of the organization dealing with legals dealing with Foundation and community etc I I think those are really really important because uh, the the best programmers aren't Necessarily the ones who create an entire distribution because it's a lot more than just talent in terms of writing good code It's all about everything around it. Um, you know, I kind of it's easy to parallel that with Linus may be good at maintaining lots of the kernel stuff But linus is not the one running any of the major distributions like, you know, we think him for his contributions He's doing exactly what he's really talented at doing but running everything else is actually It's one of those when you I look at it. It's daunting to me to go. Wow Like how do you start a whole distribution and all that that's yeah, that's a big that's a big task In our case it was um, it was actually fairly interesting Uh, I just posted a comment to a blog and the next thing I know in about a month and a half We've got 10,000 people asking how they can be part of it. It was pretty pretty surreal actually Wow, and I can't even barely get my son to clean his room Here you are getting 10,000 people to help you out. That's that's very impressive It was solving a kind of big pain point and I think that's really what what it was about If if it was right on the coattails of the announcement that centos is changing its its its focus and direction And I think if I would have done it, you know an hour before or or day before It wouldn't have it wouldn't have gotten any any interest Well, you never know you got 10,000 people. I'm sure there'd be at least a few Yeah, I think this is interesting because we talk a lot about this on channels and any linux discussion So frequently revolves around all the you know Lineage leading back to debbie and that's just the popular one out there a bunch of being that and then from you know You can spider out of a family tree of all the things related to a bunch of but when it comes to the enterprise market and Red hat just they were there first that they were the first distribution Most of us cut our teeth on if you happen to be starting in the 90s. That's where you started was on red hat um And it's really embedded Because of their support and everything else into the enterprise market lots of the tools that we use and or I should say Projects we use such as xcp and g being a favorite of mine, you know, they based them on centos So that announcement hit hard More so than people may realize to the enterprise environments that look for not the fanciest latest features But a very stable secure base by which to keep building their product and going forward on there So I think right place right time. You're definitely right about that But uh, how does your feelings on supporting all this enterprise stuff and being the way forward for a lot of it? Well, my background has been predominantly in in enterprise and to be blunt very stagnant um portion of the ecosystem which is And I don't mean that as a negative at all. There's reasons for it but in high performance computing Where I cut my teeth and spent most of my my career Uh, we we don't just like do upgrades like, you know, we've got potentially hundreds or thousands of applications That we've custom compiled for your operating system. That's running on thousands of nodes That's not something that it's like you're just going to do it, you know, a dnf upgrade on right? You have to be very careful on or you do a whole distribution update on It's be very careful We need something that has a very long life because we're not going to be doing that upgrade very often at all So from my perspective the idea of something that rolls fast or moves fast Is not something that's that's going to meet my needs I think this is a new user challenge as well We, you know, this is the homelab show but me and jay and yourself here We all have a lot of enterprise experience and we value in the enterprise as so does the people that often contract us extreme stability and not just because there's a small Minor new shiny feature that got there doesn't mean we just hey, let's rebuild the app to the new thing and just because it's there Or updates for the sake of updates. It's all about that stability security a long term running because There's a lot at stake when a server goes down or a hypervisor doesn't start up because we added some new feature That maybe we'll use in the feature But don't need right now, but we go but we wanted to compile the latest version of this because it's new Do we like new and shiny well to an extent I guess but not in some places we really just kind of want things to be status quo Yeah, I've got a funny story So I spent a long time in my career at Department of Energy at Lawrence Berkeley lab One of the projects I worked on was the yucca mountain project and the yucca mountain project was a nuclear holding facility for the for the waste that we were generating And they they had this very long-term project to kind of figure out like what's the what's the adverse effects of hosting You know all this nuclear data inside of a mountain and and what's that going to do? So lots of computation was being done and when they qualified that computational code They they literally locked the operating system down to I couldn't even upgrade ssh Like any change in the operating system would have required a reclassification or requalification On all of the applications that were that they were running on and so it got to the point where it was like, okay CVE hit CVE hit. Okay security issue. What not what not? All right at this point We just have to firewall off that system from everything else because we can't upgrade it We can't do anything on it But that's that's maybe extreme, but that's kind of typical in the enterprise space like depending on where you are in that spectrum You can't just make updates easily Yeah, you would think that they would know that because they didn't give anyone much of any notice about what was happening to center us knowing that You know their clientele or the people they use a distribution Kind of really wants advanced notice and they're the also the type that'll Be on sent to s5 for probably the next five years without telling anyone And now all of a sudden they're just going to like go to sent to us stream all of a sudden just because that's new normal Like I can't even believe they thought that's something people would gravitate to given their audience But here we are I think sento stream really solves a good area within within the ecosystem But the problem is is that it does it's not what sentos what everybody's come to love with sentos and come to rely on the sentos But it is solving a huge a huge need which is You know the community involvement closer to enterprise linux right we got fedora the fedora is kind of far away from where enterprise linux is in terms of um, you know stability and and and whatnot But sento stream is is much closer to that and this gives the community this ability to to have input into What's going into enterprise linux and and red hat? Oh, yeah, there's definitely a lot of value, but I think it it's more or less like Surprise, you know, that's going to be the new thing for everyone And it's like the people that we're already using it will be just fine on it But for those that you know, it doesn't fit their needs or they don't want to go that direction That's a little awkward. But then, you know, we have rocky linux now. So it's not so bad Now, um, kind of going back to the story a little bit You post on a forum now we have rocky linux. Let's talk about what happened in between So how do you decide you're gonna be the community leader and put all this together because you know We've all posted on forums won't be cool if there was something else and but The next step is where the the real work takes place Yeah, so I think I think j kind of touched on the the nail on the head which is You know, this this caused a major pain point Across the ecosystem as organizations were standardized on centos and all of a sudden centos just changed Rather it went end of life and the new project cento stream is kind of Continuing and getting the most of the focus Created a major pain point on the blog that was written where this was announced There was almost a thousand comments and I challenge you to find any that's positive I've read the comments and they are they they there's hilarious comments very angry comments like from all different Directions and it's one of those things where I feel like if you've been following linux, you kind of know what the comments are going to be already before you look at them Then you just get confirmation as soon as you start diving in but it's absolutely what you said And so when I when I announced on that blog as a comment Hey, I'm thinking of redoing this and if anybody wants to come and join I'm over here in this hpc slack and We went from a fairly small number of people that were already there to literally 10 000 plus Within a month and a half and I think the majority of those people tried DMing me directly Let me let me share like going from zero to 10 000 is an unmanageable problem like It was like so overwhelming And we were on the free tier of slack. So if you're familiar with the free tier of slack you get 10 000 free messages before you You buffer overflow basically and 10 000 messages when you have 10 000 people who are all active and new to a channel Goes by like that like I'll be scrolling down slack and I'd see messages come in and by the time I get there I see it. It says I got new messages. I click on it and slack's like, you know, you've exhausted the 10 000 messages You know, you pay to get to see this message. And so I was missing things as I couldn't get to it fast enough I had messages on linkedin. I had mess emails coming in and I just I couldn't keep up so one of the things that we did was We basically said, okay, let's start breaking channels out Into functional groups of where we know we need to focus on within creating an operating system So we had you know development release engineering. We had a marketing and I'm marking well community I guess more like You know socials and we created all these things merchandise Somebody came to the table and said we want to help with merchandise and so merchandise was there And all of a sudden all of these different groups started running in parallel and within each one of these groups A hierarchy started forming in terms of people that naturally assumed leadership positions and people that naturally assumed kind of Individual contributor positions and all of a sudden we had this meritocracy Just it was a almost near perfect meritocracy where nobody knew anybody Nobody had any idea who they were, you know, what their background was Most people were using avatars or aliases. So you didn't know anything about them And we started to see a a structure forming And I did my best to basically just kind of guide each one of the different groups And the next thing I know is we actually have a leadership structure in place. We have You know a group of people that are taking on certain responsibilities to get stuff done and Then we we basically then just started organizing ourselves It took us Four months to organize ourselves and to build the base infrastructure for creating an operating system It took us two months to build the operating system once we've done that and it took us a month of testing and What not before we actually released so it was a seven month total Sprint to get the operating system released and It was it was a lot of work But it was also distributed across a fantastic team of of individuals who came and wanted to be part of this All right, and I think that's important The community engagement is great And that's where you really leveraged it and fostered it to create a thing It's just um, you know, it's it's just kind of a fun story how you they went from this idea to product So to speak or project I should say it's its open source And being community different like that. That is just an awesome story It's almost like it comes full circle because of you know when linus mentioned on a message board about the linux thing And the project and now rocky linux starts as a comment in a forum post It's like maybe if I just want to start building a project or have some kind of success I just need to start posting forum posts. I guess I feel like it worked out pretty well you know Again, I think it was because there was just a huge gaping hole in terms of where the ecosystem is now in terms of operating systems and It was a pain point for for everybody um and so coming forth and and standing up and just Just saying you know if anybody wants to do this, you know, let's go let's go talk about it over here And then to see all these people want to be part of that It was definitely it was It was luck. It was Just timing and whatnot, but it was also I mean, you know Red Hat has done a lot for the community. That's been fantastic But this this is not a prime example of one of them I think this was this seemed like a very unilateral decision Didn't seem like the community had much much input into that Because if they did I think they would have had some foresight that this would be a very negative negatively responded, you know outcome Yeah, I think that's probably to be expected, but sometimes I don't know You know what happened behind closed doors at red hat because I wasn't there But sometimes I wonder these companies just come out with these ideas or this direction Oh, everyone's going to love it They're going to be right behind us because they just love it so much and this is going to be awesome But I have to have to have a feeling that they knew there would be rebuttals and there'd be some opposition to it. So Um, I feel like it's just a weird situation all the way around because it changed the Linux landscape so much I mean It's kind of one of those things where we think a distribution is going to stay the same forever But then when it's not Oh, wow, uh, nobody saw that coming. What do we do? And some people maybe embrace sent to a stream But others maybe they may not want to do that and if they don't want to do that Then what do they do and then you post a forum message and you know the answer comes out of that That's just so cool Now, how's your enterprise engagement been? I mean I look on your page and I do see Names we might familiar with like vmware on there and some other big names AWS How has the enterprise reception been to this like looking at as a replacement for some of those that base things on sentos Well, I think part of the reception that we got both from community as well as enterprise Actually came from something that we just kind of touched on a little bit, which is Um How open source projects can be pivoted in a way that's not necessarily for the community I mean sentos is by no means the first The first open source project that was pivoted by a commercial agenda, right? We don't know what the commercial agenda was but and it was it's not going to be the last So matter of fact, there there's been there's been at least two that I know of since sentos that were very fairly dramatic within the environment And but this goes all the way back to I think mongo db was the first time this kind of really was a big deal And after sentos there was elastic search, which was also end of life due to commercial redis Um, I mean, there's there's a bunch. They're they're looting me because I still need to drink more coffee But there was a bunch of them and One thing that I heard of when I was talking to two enterprises about you know, what what they're doing What are they what do they need was they need stability? They need to know that what happened with sentos Um while sentos was a fantastic operating system. This is not a slight to sentos at all It's not gonna happen again Like we need the benefits of what sentos did right but also the safeguards in terms of to make sure that this exact thing doesn't happen again And one thing that became very clear And that is I mean I'm I'm a ceo of a of a up and coming company called ciq And ciq Can't own this Right. I don't want to own it. I don't want to control it. I don't want to hold it hostage from a ciq perspective It has to be community Now what's the best way of doing that? And we we came up with some different different models and the and the teams Had various different discussions about this Um and something that a lot of people don't realize is sentos originally came out of a nonprofit 501c 3 And a 501c 3 is an actual nonprofit It's a charity The irs doesn't usually give 501c 3s anymore to open source projects or so they usually do 501c 6s, which are different but they don't It came out of that 501c 3 like a 501c 3 or a nonprofit is not necessarily the best structure to Protect an open source project as we saw with sentos and there's been there's been other examples of this as well Free note as well, which a lot of people know of oh, yeah before actually came out of a 501c 3 it was the peer directed projects There was another c i can't remember what it said for now um, but it was a nonprofit out of texas and it you know Do nonprofits really protect open source projects and i you know, I don't think so I think there's better structures that we can use and so i created the rocky enterprise software foundation I promise this is a long way around of answering your question I created the rocky enterprise offer foundation with the rest of the team to figure out like What's the best structure that we can use to protect this? and We came up with a model there that I wouldn't say it's drastically different than others, but I think there are some some fairly important distinctions like no single company can ever Can ever control this like we have rules about how many companies Or what percentage of any company can be on the board like if you have more than one third present You quorum is not made We never sell board seats. That's a surefire way of basically Having companies buy in to manipulation of your of your project Many open source projects and foundations and nonprofits do this. I don't think it's the best for the project I think that's a way of Making money for for a foundation and I don't think that's what's best for the community So we never sell board seats board seats are always provided and given and granted and voted. I should say based on their The merit of the person of the individual that's coming in to represent But this model Is what immediately attracted a number of different Big companies. So you brought up vmware um aws contacted us like shortly after I announced this this vision and what it is that we're doing and basically said you know, we want to be part of this and um They gave they offered us a huge amount of you know help in terms of of building this And then google came up and basically, you know offered us as well a huge amount of help and support and doing this Azure exactly the same thing vmware naver cloud and the list just kept continuing and growing Now we've even have sponsors that are not listed on the website because they're actually they don't want to come out as Saying that they're competing with with ibm or red hat We're doing something negative there. So we also have sponsors that are not, you know and helping the project But they're not doing it necessarily completely publicly. I hope one day that changes But at the moment We have a huge amount of support like a ridiculous amount of support And it's just been so fantastic to be part of this and just see all of this grow and see all this point together And I think it's really important what you touched on with the structure Um, we started talking a little bit before the show and I stopped myself because I also wanted to save it for the show But when I'm looking at something I want to use because we are the homelab show here So we start there But a lot of us our goals usually as the homelabbers are to get a job in enterprise where maybe you already work in enterprise And tinkering it's fun in homelab, but when you really look at something if I want to build a project I want to start um free pbx example What do you want to base the phone system on well? It matters that foundation you start your project on Going forward because you need that long term support So I need to know that that project will be available that that project will going forward You touched on elastic search that's created some issues with some of the products We use because they had that baked in and they switched it now to open Is it open search? I think it's whatever the replacement is because you're right There's there's corporate interest at play and when if there's not a good structure to hold it together as the community project We lose it as a community project and this has actually happened Like you said more I haven't had enough coffee to remember all of them, but boy the list is long And I think that's it's just really important to think about that from the long term Because especially if you fall in love with the project you're like Hey, I hope this whole distribution or this thing that's built on top of this other thing lasts for a while But if there's no good structure to hold it together, there's no path forward long term corporate interest by board seats corporate interest Will use it for their own interests and Or to some of these corporate companies, especially in the hyper scalar world They have the kind of money where they can just mess up things and go I'm done with this toy and sorry. I broke your thing community. We're going back over here Yeah, sometimes still don't know what exactly the intent was a few years later And it's like I don't know someone had an idea someone changed your mind Um community and broken for a little while Yeah, we've seen that Yep, definitely. But what's it like to build this? I mean because it's I don't know if people really understand the weight of managing a distribution, you know, it's It's not something people generally say. Hey, I want to start a distribution today as they wake up I mean, maybe people do that, but um, it's got to be such a massive undertaking to maintain this What's the build infrastructure like for this? Well, there's two questions there. I'm going to address the first one Which is first and I'll address the second but the first one is Um, it takes a masochist to do this And and here's why like if you look across the ecosystem like, you know We we have so many religions in our ecosystem, right them versus emacs, you know pearl versus python I mean the you could just go on nothing prepares you for the amount of religion to get to get stuck in the religious crusade of distribution religion and I mean, I'm I'm guilty of this just like everyone I think is to some extent and um it's it's to now all of a sudden Literally, I feel like I have a target on my back. Like I'm getting attacked left and right. I'm getting people sticking up for me I'm getting people talking and I watch some of these threads And I'm like I I see how conspiracy theories can form because I'm watching Conspiracy theories happen about me like oh, I'm going to go sell Rocky Linux to red hat next and I'm going to go and due to its structure, you know I'm the owner and as a result it's going to succumb to all of these things and I mean, it's like that's not that's not me Go talk to people that have worked with me before spreading those rumors You know do so do a little bit of diligence because you're actually talking about a real person And it kind of sucks like to have all these people talking about Me and and other people on the team and trying to discredit it just because it goes against their religion, right? There goes against their their personal preference. That's the first message. I wanted to get out there But also I wanted now to address your second question Well, I want to add one more thing. This is one thing I like about doing these interviews I think we come across because Me and jave as we engage with the community try not to be divisive at all We like all the different distributions and we Wanted to humanize a lot of this like there's people behind all these things that are going on here But you try to keep that very You know real like there's people and projects and that's what we're talking about here There's there's a lot of reality to it. It's not just some, you know It's easy when you can disassociate yourself and just see some posts or a website and go Oh, they're doing this thing or they're going to sell to this company. Um, but no, this is Real people and that's what we're having this conversation. That's part of the reason we have you on here is talking about it publicly That is awesome. I love that. Thank you for inviting me again because I really want to just you know I don't know Help it sounds so cliche, but just help people see that you know, I am a real person stop talking negative about me Please go get to know me like I'm so easy to reach. I have an open door to everybody who wants Um, I'm not 10,000 at once. There's a lot of limitations Yeah um, but uh To to the second question What does the infrastructure look like this is a really kind of deep question as well Because to go I mean when centos first started It was literally like shell scripts running on on somebody's, you know, personal server or personal work stations Building rpms in a while loop. I mean that was Plastic first versions were Then we had an initial again initial versions of centos. There was some build infrastructure put into place But it was literally done by individuals A lot of it by hand. I'd say the majority of it by hand So when you think about the security aspects of this The way you have to handle that is well, you have to have a small team And that small team has to be 100 trusted with the keys to the kingdom, right? Um, you can't have anybody who can potentially put in a trojan Do something wrong do do something malicious because it's literally getting out to every single system That's every single person that's running that so you have to really control that That exposure surface there so With centos it started off initially just with it's a small team other people asked if they can help if they can be part of this but It's kind of hard to to let other people help and be part of that core development team. So With with rocky one of the first things we all said and agreed on was We have to figure out how to do this at scale such that the community at large Can can contribute to this One of our one of our mantras that we've already kind of talked a little bit about is no single company can hold this hostage and so No keys are owned by a company. They're a secure boot shim. We started from scratch to go get from You know microsoft and the powers that be so that so that the resf the rocky enterprise software foundation Has our own secure boot shims. No company can take that away Our signing keys. I mean domain everything across the board had to be owned by the community and has to be controlled by that community So that includes the infrastructure. So we had to create infrastructure and create build You know tools and tooling and whatnot All from scratch and that took us a while to do took us four months exactly almost exactly to do And so we built all of that up now the first version rocky 8 Started with the fedora build Environment which is koji punji Modular build system and etc We started with that That system was designed not really in a cloud native way It was designed to be built on or for a single cluster. So you have a raka systems You've got a build cluster and it's kind of more designed towards that We didn't have a raka system. So we have cloud. So we wanted to build everything in the cloud So, you know, we replicated a static system in the cloud, which is not ideal, but that's how we approached it Uh, because we wanted to make sure that the output binaries of what we were creating were compatible And that the community at large can come in and contribute What we learned from that is I mean koji's fantastic But it didn't really meet our needs perfectly. So we started building a cloud native build system And we created something called peridot And we weren't sure at first exactly how this, you know, everything was going to work and uh, So we did build it, you know, initially and I know that this is not typical open source But we did build it initially kind of in a closed way just with with some developers working on a private repo Um, and then as it started to mature we started bringing in, you know, some there was some more people that that that Had visibility into it and then but but we have a rule that no release of rocky linux can ever be built using not open source software Somebody else needs to go and be able to build and replicate what we've done And again, this is for the community to make sure that we are always A stable solution because if something happens to us, I want to make sure somebody else can pick up from where we left off So everything has to be completely open transparent and documented and I'll pause right there for a second because this is This is the splitting hairs of open source that can be very aggravating where There's a lot of companies that say they're open source But they just don't give you any of the build pipeline information So they just give you a lot of source code But there's no way to to actually build the software to an easy repeatable binary and Even in devian, this has been a long time challenge building those repeatable binaries so you can say yes, this is truly the code and this is the outcome and It's easy for a single like executable to build that but an operating system is an entirely never level of that So I think it's really important that you're doing that type of work Because this is where I mean I almost want to ask sometimes Is it open source because just because I have the code I can slap the open source name on it but that build pipeline is Arguably as critical a component of it because I can't it goes through a secret machine That grinds up the swatches and outputs that trust me. It's exactly the code that went in or is it? Yeah, I've worked on a build system or trying to migrate a reverse engineer I should say a build system for a project that was abandoned And it was a really tough task and that was just one binary Yeah, that was that was a long time, but yeah, I could totally understand that being a lot of work So I I'm of the strong belief that open source software, especially gpl Software, which is which basically says that any derivatives of that source code must also be adhered to the the license of the gpl I believe that that includes binaries As a binary is a derivative of source code So from from that perspective, I don't believe binary should ever be restricted as well So when you look at that build pipeline and you look at everything that comes out Like the whole thing has to be open source. The whole thing has to be repeatable and again coming out of science And the scientific community like repeatable science is is critical If you don't have repeatable science does it is it really, you know, is it is it validated? Is it really science? So it has to be repeatable. It has to be Open in such a way that other people can get the same results. We actually had a validation point There's another linux distribution, which is not very popular Outside of china But circle linux Uses all of rocky linux build infrastructure to recreate and they have a whole another linux distribution And kudos to them, you know for for doing that and I I love that they did it for a couple reasons One is it completely validates that vision that we had which is this needs to be repeatable So other people can do this and I've also now heard of at least one company That has a derivative of rocky linux by using the exact same tooling that we do. So This this vision is is I think very important to ensure the long-term stability of the operating system And we also have other facets as well like you have to be able to We're always distributing all of the The requirements for every build every package. So every package has build requirements and in many cases Like red hat didn't always distribute all of the build requirements So it made it very hard to repeat the builds to repeat the binaries Of what's in the in the operating system because those were what we call hidden dependencies Sentos and early days of sentos We vowed that that we would never hide to hide these build requirements these these these hidden dependencies We would never hide them. We'd always make them available Um Later in sentos's life, you know, it did come up that some of these things were being hidden again After you know red hat acquired the project and and again This is possibly some writing on the wall that things were changing But we want to make sure with rocky linux that we never do that. So we put together Our our charter and everything else in such a way that Um, you know, it's going to be persisted and we've actually even talked about I think we need to host that a pdf or some output of that Um, that charter on our mirrors So every single mirror in the community will always have like this is a way of persisting it So it can never just vanish and go away Um, which is what happened with with some of these early kind of ideas with sentos and some of our early goals Um, it feels like they just kind of went away So I want to make sure that that rocky linux Is is always here to be stable for everybody and we were talking about the build system If you don't mind I'll bounce back over there really quickly. What we did Super awesome like we we basically said, okay, we got koji build system. It's kind of fine for a single system It's hard to replicate. It's very complicated. It took us four months to do well Not entirely just that also organization, but it took us a while to do So how do we make this easier? How do we simplify this process and go completely cloud native? And that was where we spent a lot of time focusing and the result of that is peridot And peridot is a completely cloud native kubernetes based build system. I mean We're not quite there yet in terms of simplicity of installing But the goal is you can helm install peridot And immediately boom you now have your your your build system and you can start throwing packages at it You can start doing get operations against it and start building You know packages for your operating system whether somebody's doing that in a in a commercial way via A vendor is is adding on to to rocky linux via peridot or Excuse me, or if somebody wants to Just go and recreate it just to make sure that you know There's there's redundancy in the community, which I think is a good thing This this this gives everyone that confidence of stability and so we release peridot and Rocky linux nine was built on peridot rocky linux eight was built on koji and we're always talking about Do we move rocky linux eight from koji to peridot? We want to make sure again in the vein of stability and nothing changing and nothing kind of rocking the boat That we don't cause any sort of disturbance in the eight track Oh, that was a funny slip But we want to make sure that we don't like interrupt that or do anything that may it may make that negative for for our users or change or break anything so You know, but but we are talking about it because it would so simplify the maintenance Um of of rocky linux version eight. Um, so maybe at some point we'll see that Um, we're also seeing other things coming out for eight as well Um, so fits so my company, um, and this is one of the cool areas where Having a having an open source project that a company doesn't control that company has to add value if they're going to do anything worthwhile So we're doing something called fits. I'm not sure if you're familiar with fists, but So we're doing fits. Um compliance validation On rocky linux eight dot six. We're almost done with that. We're getting that over to the government labs for validation here shortly Um, but to be blunt, that's a million dollar investment right there. Wow That's a massive undertaking and we're giving that away to the community like that's not even going in ciq's name That's going in the resf's name for the community. So we're giving away this this gigantic thing Um, and we're doing the exact same thing for nine as well So this is the these are important things like in my mind to make sure that the community is getting the the benefit of And this is a challenge that actually just qualifies a lot of open source So for those of you not familiar or haven't heard the term fits f. I. P. S It is a mandate of which algorithms in a very exacting way Like this is what you're going to be using and the product and you will not be using other algorithms in here Flipping on the flips comply fits compliance is usually a government contractor If you're part of dfars, you're part of any of the, you know, department of defense stuff that you do You're gonna your company's gonna have to say that the software and the things we have are compliant with that. So It's yeah, it's not it's almost like I've One in my years of being an open source advocate It's sometimes the disqualifier Why you can't use something is it may not have been fits compliant And it may be completely secure. It's not a security thing It is a compliance thing of nope. We only allow you to use this algorithm But this algorithm is actually more hardened and newer, but that doesn't matter. It's not it's not on the flips list I think I described that fairly well The one thing I may add is the flips process is also About a year it's about nine months twelve months to get done Which means if we're gonna validate a particular version like eight dot six, which is what we're doing By the time the flips is done eight dot seven and maybe even eight dot eight is is is coming is coming out So it's a constant kind of catch-up game And government agencies and a lot of corporate agencies that require that level of confidence in their crypto within their within their platform Have to make a decision like how, you know, do they want to roll that out most organizations say, okay Eight dot six was was flips qualified. We're gonna make a big leap that eight dot seven eight eight eight eight nine ten Etc whatever is all also going to be flips Validated even though it's not pedantically still flips validated because they validate the binaries That it gives them the confidence that they need to feel to feel good about that operating system or that platform Yeah, and that that's uh, yeah once you get into the enterprise market, that's a box that has to be checked quite a bit Yeah, I do you feel like you're gonna give an influx of users once that's done um, I think we already have um even uh, so Hyperion research, which is an analyst organization recently did a Or a survey to organizations across the board and they found that I think the number was 38 I can look at no, I'm not gonna look it up right now But it was near 40 percent of government organizations are already on rocky linux. Wow So that's gotta feel really great It's it's massive. I mean the uptake has just been it's just been crazy. It's been um Uh, their average across all I think was 20.4 percent of the ecosystem is already running on rocky linux Um, I mean again, it's it's really is massive. Um, I can I'll pull that up And I'll send that over to you and if you can share that out to the community. Yeah, we'll we'll throw it in the show notes Um, but yeah, it's it's massive the amount of uptake that that there's there's been and um, I think enterprise organizations are really responding well to the idea and the vision of what the the rocky enterprise offer foundation resf is set out to do which is to better join and better unite open source communities and individual contributors to uh enterprise needs Enterprise needs are not always in alignment with the community and to how how to balance that Is is is a trick, right? It's a task. So um, that's what the rocky enterprise offer foundation was was designed to do Uh, you know, we recently elected the resf board and we're about ready to have a update on the board structure coming out um, actually very soon now And um, but just to give an example, greg crowe hartman who maintains the linux stable kernel, right? Linus torbalds maintains the development kernel greg crowe hartman maintains the stable kernel Uh, he's on our board of directors. Um christa bono who um from google Uh, he was the person who was responsible for elite releasing things like kubernetes and android to the world um He's on our board of directors and i mean and and lots of contributors to the project is on a board of directors But those are two of our independence um It's it's fantastic to see how this has grown. But I think we've really we're really doing something that's important for the enterprise specifically the professional it communities I'm sure there's like a countless number of people that are very thankful for that because it Probably allowed people to go from oh my god. What do we do in the board room to? That that's the way forward right there to actually Have a solution and um, I feel like for a lot of people that was probably very relieving Especially when it was uncertain people didn't really understand the change and didn't understand why sento s dream is the new normal And all these other things I think it's great to have some kind of uh status quo something to rely on something You know a path forward that doesn't require changing your entire base to a completely unrelated distribution Which is even worse. I think Exactly. Yeah, that would be in big undertaking I mean, I I know that I think rocky linux is on a consideration for the path forward at some point with xcp and g And uh, the team over at vates there and it's because I've had it's funny when I've seen because I talked to oliver Who's the team lead they're like, well, can't you guys just switch over to debbie and he's like you don't understand How deeply embedded a hypervisor base os that's very customized is we can't just rip it out and replace It's got to be a lot of compatibility here like right the infrastructure builds around it So the name of the Apache package isn't even the same I mean every little detail every little thing you need to install is going to have Either a different name or maybe a package could be split more and one than the other combined somewhere else I mean, there's a lot of different moving pieces here and I don't I don't think a lot of people understand the gravity I think they Understand in general that this is a lot of work But I think and correct me if I'm wrong unless you do the work. You can't really know what it's like, you know I would say that for sure Yeah, till you've actually built an entire distribution. You don't know how hard it is to build a distribution It's gotten so much harder. I can validate for that. It's gotten so much more difficult um Yeah, early versions of sentos was was nowhere near as difficult as it is now Oh, wow. Yeah, I can imagine a lot more packages nowadays than we had before for sure Yeah, and securing that build pipeline not just documenting it, but also securing it like you mentioned Uh, you you can't go a week without another insert name of you know pipeline that got something injected some type of typo squatting um, it's it's a constant security threat and I deal a lot with security. So it's just like It's like dealing with the threats of the currently broken software Let alone someone slipping in a breakage in the software or some potential vulnerability. I'm like, no This is hard. We don't want a supply chain attack. Thank you very much Yeah One of the things that we wanted to do with peridot is to really ensure that that supply chain security um is is validated and transparent so you can you can really trust that um and and all the way from You know the the file that ends up on your operating system through the rpm Which which exists now through the validation of the rpm that knows it came from us and it hasn't been tampered with But then how do you know that you know how it was built? How do you know that there's any validity behind that and right now generally speaking? There's there's there's not a huge amount of validity on that so Um, that's one of our goals is to make sure that you know, there's full sbomb Validation all the way through the pipeline so anybody can go from a file that's on their system Right. Is that suspect? We don't know. Well, it's signed by the rpm It's signed by the organization who who released it and here's the environment that they build that package Let's reproduce that that sbomb and come up with you know, let's do a full validation check on that sbomb Like that doesn't exist today But that's one of the things that peridot is is Making available at this point Um, we're not quite there, but we're really close, you know, and I wish sbomb was further ahead But boy, I um, I brought it up in a security conference and the number of Shockingly security people in the it services space that hadn't even didn't even familiar with the term And i'm like, yes software bill of materials We want to know the ingredients how the sauce was made And I mean this is on the heels of log4j when I brought it up at the conference going Did you know your product was built a log4j or do you know what products were built with that dependency? And having a bill materials is huge and I think open source is way ahead of the curve in terms of advantage Of being the better way to go in the path forward. I've been a long time advocate of it It's getting proven more and more. So i'm glad you brought up sbomb like that's it's not talked about enough I'm like it's 2023 people this should be just part of what is included when you buy software Or are you software? I agree yep Yeah, very important and it's only going to be more important going forward. So All right. Well, this was a lot of fun. Do you have any more questions jay's or rat got reaching the top of the hour here I mean, I probably have a zillion more but I think uh, you know, I've got to stop at some point Yeah, maybe we'll we will have to have on again and ask a zillion more questions jay also has another podcast Dives in the linux. So there's always an opportunity to talk about uh, some more of these things, especially the security side of things so There absolutely is and maybe that's that that could be something i'll uh, i'll email you about that see if you're interested Absolutely, that sounds like a lot of fun and and this was this this conversation was a lot of fun I really appreciate you both. Thank you for inviting me. Yeah, and thank you for taking the time This was great and uh, thanks everyone for joining us. Take care and see you next episode. Thank you