 Good morning. Good afternoon. Good evening and welcome back to ask an open shift admin office hour I am Chris short executive producer of open shift TV I am joined by the one and only Andrew Sullivan Andrew longtime. No see how you doing buddy long time indeed. It's been Three weeks since our last stream together You know, I saw plenty of you. You didn't see any of me You know, I was watching your streams all during summit all during coupon. So well, thank you Yeah, and I enjoyed that it's nice to be a participant sometimes. So Yeah, I got that during coupon. It was kind of nice. Yeah Yeah, enjoyed all the streams tons and tons and tons of a content that came out of the last two weeks So if you're keeping up with all the feeds and all that other stuff around the stream, I Have added yet another stack or stack Slack teams like channel to keep an eye on which is the CNC F slack so if you're On the Kubernetes slack, there's open shifts channels. There's open shift dash users opens open shift dash dev If you're on the CNC F slack, there's open shift channels there redhead channels. I think they're You know in addition to all the other places. So we are Constantly always Never not available. I guess if you need help if you want to ask always around. Yeah There's always folks around So hello, welcome to anybody who's new or hasn't been to one of the streams before as Chris mentioned I am Andrew Sullivan one of the technical marketing managers with the cloud platforms business unit So Chris and I both focus on open shifts and in particular we bring an administrator background right we're both administrators architects prior to being at Red Hat and We we bring that perspective and really the goal of the office hour series of sessions Which the ask an open shift admin is one of those Is to give you our audience the opportunity to well ask us anything Whatever it is that's on your mind, whatever it is that you have questions about that's bothering you that you're curious about Straight ask us and you can do that at any point in time across any of the platforms that you happen to be viewing us on So if you are watching on the red hat YouTube channel and you post a question We'll get that over on twitch as well as in the open shift YouTube And kind of all together there in the circle and we'll address those here If we can't answer those questions, we're happy to follow up as well So, you know like I said both Chris and I have an administrator background You are more than welcome to ask about developer type topics But there's a pretty good chance we'll have to follow up and answer those questions afterwards So today as the title probably gave it away. We're here to talk about a CoroS So CoroS is a I'm gonna say a flavor of Red Hat Enterprise Linux But really the people who absolutely know whether or not I should be calling it a flavor or not are here with us today And that is Mark Russell on the product management team and Derek Ornelis And Derek, please correct me if I if I butchered that on the engineering team So I'll start with Mark if you don't mind introducing yourself Sure. Hi everybody Mark Russell On the product management team kind of have one foot in the business unit that deals cloud business unit that has open shift but really I report up through the rail organization because As Andrew said it is rail CoroS is a is a distribution of rail Maybe we'll set set that for another topic. We'll continue on in the moment, but Otherwise, yeah, I'm also, you know an operating system administrator type background and I've worked for multiple distributions and though this is my last one for sure and As well as you know worked on the other side of the fence consuming products and and shaking my fists at software software distributors, so I've been in a lot of different seats including support as well, so Derek Hey, so I'm Derek Ornelis. I actually wear two different hats. I do product experience for our container ecosystem inside of Red Hat That basically is me bringing Over a decade of support experience to the portfolio lifecycle management group The other hat that I wear is a rail product owner for rail CoroS so I'm kind of talked to the Clients so the engineers don't have to kind of hope there Yeah, you know, we had an office on our team who we strongly encouraged to watch office space And he watched about half of it and said I don't get this movie and just stopped My kids set one that one aged well with my kids actually not as good as Ferris Bueller, but it did pretty well So and Mark, so we already have a question for you which is coming from JP Dade. Is that vinyl behind you? it is I Like it for its inconvenience and collectability Wait, you can't play records in your car. I don't understand You can't hit shuffle on the turntable. No, you can't and that is actually one of the virtues And I don't know if you have the chat up, but I think that's Joe D'Andrea is saying hello and and also Hi Joe. Good to hear from you Yeah, I've I've I've never I'm right on that cusp of vinyl and all of that other stuff I've never taken it up as a hobby because I've got enough things to occupy my time that yeah My wife would object if I had anymore so Um, yeah, so thank you again Whoever's watching, please feel free to ask questions in the chat. I see Samuel. How does cloud pack work with open shift? I got a link Okay, um, so so at the high level cloud packs are a workload to open shift So essentially you're deploying an open shift cluster, right? It is open shift is open shift And then the IBM cloud pack, whichever one you happen to be using is deployed as a series of pods, right? standard traditional kubernetes workload on top of that So that's that's really the way that we look at it the way that we treat it And then if you happen to be an IBM cluster, essentially they bundle or customer, excuse me, they bundle everything together Yeah, so that's that's where the magic happens So for anybody who has joined us before you know that I tend to open these sessions with Just a you know five or six minutes of things that are kind of top of mind Over or since the last particular episode So it's been time for it and yeah So I had to pair this list down pretty substantially because of course with summits and uh coup con There was a lot a lot that has happened. So I don't have I don't have any of the links right here If you go to openshift.com slash blog and chris might even have a shortcut for this We've got a bunch of summary blog posts on openshift.com That cover all of the announcements all the things that happened. So I'll include those in the summary blog post So Friday mornings, we have a blog post that goes out that includes all the links as well as references to The content that we have today including links directly to the timestamps So I'll put all of those into the blog post that goes out on Friday But just know if you missed anything if you if you weren't able to watch any of the sessions from the last two weeks from summit and coup con Tons and tons and tons of great information. So I won't Reiterate or rehash that because you know, those specific presenters can do a much better job than I can Uh, so I do want to talk about a couple of things however, so I'm going to share my screen real quick here And I think I want this window I do All right, so the the first thing that I want to talk about And this is a a it came from a conversation amongst the product management team So one of our account teams asked the the open shift product management team And I happened to find this particular article that was shared as a result of that So we get asked fairly commonly about Stretching an open shift cluster across multiple sites. So One cluster that is on, you know, maybe in data center a You know with node one and node two control plane node two is in data center one and node three is in data center Hashtag or something like that right pounds on So what are the requirements of that is that supported, right? How can I do that with red hat and with open shift? and The answer to that is oftentimes very nuanced, right in that it's not unsupported But there's a lot of things you need to be very careful about And this kcs if you read through it goes through a lot of those things that you want to be aware of and what are the actual You know guidelines and requirements where it becomes unsupported So things like network latency storage latency, right availability, right? What are some of the reasons why you want to do this or don't want to do this? And the impact that it has to availability as well, right if you only have two sites You're not really getting any availability by spanning it across two sites. You're just increasing complexity, right? those types of things so great kcs One I didn't know about until just recently I wanted to share it with all of you as you're going through doing any architecture or making any decisions Because it does contain just a bunch of links to additional information that can be very useful for that Let's see Switch over make sure that there's thank you for posting the link chris. No problem. Thank you for having joined Yeah, I know it helps it helps keep me organized. Um, so if you weren't aware Upgrades from open shift 4.6 to 4.7 are now available So finally, uh open shift 4.7.6 was the or 4.7.7 Was the first version where we fully opened that up Remember we've talked about this for I don't know like a month and a half now since almost since since it was 4.7 was jade a bug yeah, so the issue was a a bug with networking and VMware hardware version 14 and later And the rel 8.3 kernel used by open shift 4.7 So the end result was or the short version is Doing vxl and offload caused some packet loss issue So if you are using a upi or ipi deployments, click that upgrade button, right? You'll be great Everything will go fine Which you'll see is that the remediation was to add a machine config that disables the offload in the kernel Nice If you have an open shift cluster that is deployed Non-integrated or a door platform equals none Bare metal ipi or upi. Excuse me. Whatever phrase you want to use to describe that on vSphere You still have the potential of experiencing this problem So the way to fix that where there there are two ways to fix that I should say So the first one first, let me say this has been updated It is now in the release notes for open shift 4.7 and I'll post this into our chat here So if you look at the release notes for 4.7 We have this bullet here about running a cluster on VMware with platform equals none Now you can see one of the recommendations we have here is to downgrade or to use virtual hardware version 13 Essentially version 13. We're not seeing that same issue. So that's one of the options that is available However, if you choose that option, remember if you're using the vSphere CSI provider that requires virtual hardware version 15 So you might have to do some balancing there I say that's knowing also knowing that you unless you have done something unnatural like you deployed with platform equals none and then manually added this the cloud provider for VMware The cns provider needs the cloud provider integration or the cloud native storage CSI provider So in theory, you shouldn't be doing that with platform equals none. So you should be safe there The other option is to basically do the same thing that the vSphere problem detector is doing and the machine config is doing Which is to apply a machine config that sets the value for you. So I'll I'll post this link inside of here so there's nothing again if you have a A bare metal or a non integrated install on vSphere And you're using hardware version 14 or later with open shift 4.7 You'll want to effectively either use VM hardware version 13 Or apply something like this as a machine config which will go in and it will disable that That offload and that will avoid the packet loss issue that we've been seeing That that led to the delay and upgrades Hopefully that made sense Makes sense to me if you have questions feel free to ask. Yep I've got to drop our special office hours Oh, there we go Keeping in line with VMware So I talked it might have been the last show I talked briefly about how and for sure it was when Catherine was on a month or so ago You can add nodes to an IPI cluster outside of the machine set paradigm Right, you can manually provision nodes. You can bring them into the cluster and You can set them with like a static IP for example All that type of stuff So Michael McNeil here published a blog post on openshift.com that walks through that process in excruciating detail He did a phenomenal job here of documenting In screenshots every step of the process that you need to go through to deploy those nodes So great resource. I find myself doing this process like I'll deploy an IPI cluster I've been doing a lot of testing lately with Using alternative or different load balancers with an on-prem IPI deployment So what I found myself doing is I deploy an IPI cluster and then I'll Manually provision statically IP nodes to act as my infrastructure nodes and then Configure the load balancer to use those statically IP nodes right so Anyways One of the ways that you can make those open shift clusters more flexible to do the things that you need to do And you can also manually create you can create a machine And I believe it will automatically provision that way as well, but don't quote me on that I need to test that is is what I'm saying Oh, the last thing that I have is Wherever I can find it So speaking of using an external load balancer with the on-prem IPI So with open shift 4.7 we g8 and fully supports this apps domain configuration So Even though this says on aws. It's actually supported with all infrastructure types The the documentation is wrong. There is a bz for that But what this does so you update the the ingress here You add an apps domain And what that does is all routes that are created now use this domain instead of the default apps dot cluster name dot domain name That's configured out of the box So if this domain name whatever that domain name happens to be If that dns address happens to reside on an external load balancer Now all of your traffic is flowing through that external load balancer. So it's one of the ways that you can use that The other one being or the other major one being to deploy a new a second ingress controller So for example using the engine x operator or the f5 operator to create a new inject ingress controller and then configuring your Routes your applications to use that Okay, that's all I got. I'm done sweet Yeah, I'm with the show so um Coro s so uh mark I'll I'll ask you and and derrick, please, you know, I'll trust you guys to to uh Answer however you feel is appropriate, but can you tell us about red hat enterprise linux coro s and I think importantly Kind of where did it come from like what what is it? You know, why is it different from rel and you know, where does it fit into the red hat portfolio? Sure, um So let's uh, I guess I can start by going back even before you know before it was a thing after red hat acquired coro s incorporated, you know the company and They had a philosophy that they'd already started a path. They'd already gone down with container linux and container linux had this had a as one of its primary goals to secure the back end of the internet the same way that mozilla and google had Helped improve security on the front end by By updating their browsers like very aggressively, right? So container linux was going to do the same thing It was going to by default update automatically Um, and it also is where ignition came from Um, which maybe we'll put off talking about ignition, but that is that is also the origin of ignition It's the origin of the operator pattern Um as well came, you know through coro s so We also had project atomic on on the rail side that used um in Used rpm os tree as an alternate Package manager you could call it a hybrid image and package manager So it it deploys images, but it does understand rpm's it does understand Yum repos. We don't make a lot of use of them, but it's there for troubleshooting and and problem problem solving And actually upstream not in any product yet, but upstream rpm os tree now Understands rail modularity. So it's it's gaining a lot of abilities to understand Other package systems, but at heart it's an image-based system Upstream is fedora coro s That's where you're going to see I think most of experimental features are going to land first in both ignition and rpm os tree They're both also used in fedora coro s And that's where you're going to see the latest features land So we took the coro s philosophy As you as you know and if and if anybody doesn't know I don't want to scare anybody off the Updates are not automatic on an open shift cluster. You do have some interaction there They're rolled out automatically once you decide them. I'm just saying it's it is in your hands so we combined a lot of their philosophy and their approach with uh with With our expertise in s e linux are the the rel binary package foundation And the battle testing that that gets out there And combined, you know combined that with provisioning stack ignition an rpm os tree so you end up with a product whose philosophy is it's a opinionated and Um distribution of rel that is built to serve the needs of open shift 4 that is You know in the same way I I think about when they talked about Uh cryo being you know only only aimed at the needs of kubernetes and and no other use cases That's the kind of the way I look at the the relationship of coro s to open shift Um, it doesn't do much standalone. We don't sell it standalone um and so To to get further into what is the what is the philosophy why rpm os tree and Why ignition actually maybe I should stop if you have any questions So I I like to talk about ignition because ignition is fascinating to me. Yes, you know, it's it's a configuration tool that runs before pit zero even starts on the system So you can do some really cool stuff inside of it. Um, I I also want to ask about and talk about like I So, you know, that's step one step two or question two is, you know, the the management philosophies or the the administrator Experience and the difference for from a rel administrator um, and you know open shift three to A coro s administrator and open shift four So to the philosophical or management approach first I think um it, uh You think about rpm os tree and and you know, if I think maybe people are aware, but it also it's a read only slash user So all the os binaries are read only. Um, which it has some nice security effects But the but the real neat thing about it is you can actually you know, you can it's it's a clear separation between what is what is What we've distributed the os vendor has distributed to the customer And anything that has been layered on top Or any changes that have been made it makes that very clear and that is an implementation of the philosophy which is we want a clear bright line between The platform that you receive that you should treat in kind of an appliance like fashion and And the user workloads and and third party and third party agents as well But there should be this distinction where a container host should be small You know, it should have a very small attack surface should have as few packages as possible um And should be built again to run containers in in an open shift cluster Um Do I I hope that I am not getting ready to open a can of bees. Oh Are they murder bees or No, no, well, I hope not so immutability And and core os can you can you talk about that? It's not not murder bees. Got it. Yeah There's something short of murder bees, but it's a but it's a perfect question It comes up a lot actually and it it's it throws people off a little bit Immutability doesn't mean You can't it's not configurable, right? It doesn't mean not it doesn't mean exactly not changeable Actually, you can kind of look at the way we distribute Um Rail core os in the same way you distribute you work you'd work with containers themselves, right? They are read only usually Images that are configured by the environmental variables, right? You don't change them You don't configure the individual containers. You have a set of configuration That is passed with this image That's our that's that's what we do except we pass that configuration or we have that configuration through the fact that etsy Is persistent so Use slash user is read only slash var and slash etsy our read write Slash var has a lot of state to it and etsy has your configuration as you would expect, but We do not support Outside of in production Managing those files manually People do it you can do it All you know to your heart's content when you're trying to figure out what's going on in a problem But in order in order to push out a change or a configuration change You want to do that through the machine config operator? and use a machine config and We should probably also mention things like the the file integrity operator, right? So if if you want to, you know, basically know or verify if something has changed on The coro s system the file integrity operator is the best way to do that The best way to know if any files change or have been added It because the there is some confusion between them with the machine config operator that it is That it is somehow watching out over all the files in etsy and that's not the case If you want that level of I need to know if anything has changed And report that's where the file integrity operator comes in it uses the advanced intrusion detection environment engine whatever the e is That I think people are familiar with it's similar to tripwire in that sense, right? So you just have a hash list of of all the files and directories on the system Okay So I'm gonna I don't want to forget about ignition So can can can you talk about ignition and Kind of what is the purpose of ignition and and You know, we talk about it a lot in the installation process, right open shift install You know you generate the ignition files if you're doing anything other than ipi you're hosting those somewhere providing them to the vm so You know, what does that do and and how does that work and Importantly, can I modify those? Can I customize those? um, and I may we won't Yeah, well, I'll start with what's the difference. So a lot of people are you know From from other distributions are familiar with cloud init, which is a post pivot, you know post init ram fs utility That uses a it seems to be like a combination of imperative and declarative configuration And it's the most widely used, you know cloud instance utility Ignition as as mentioned is a pre pivot init ram fs utility that runs a single time, right? So this is it it configures it carve it can carve up disks. It can write files Derek feel free to jump in but You know it a lot of stuff it does is is carving up disks and file and and various instances of configuration files Yeah Laying down files. Yeah, so stuff that you couldn't do post pivot with something like cloud init You know, so we get to run uh, you know very very early in the boot process before we've done anything with the root from memory we can You know like lay down encryption move the the root file system to another disk or another partition create secondary partitions and You know with the ability to define files like system d units We can lay down, you know secondary mounts or system d services that we want to run You know on the actual like provision system So so you you just opened up, you know pandora's box. I'm avoiding worm analogies now because because brett brett's given me a hard time about it So a whole pandora's box of questions for me. So things like So ignition only runs once Pre-pivot is that once per boot or once per system install and and I ask that because Like if I look at a machine config And I look at a you know in a machine config pool that has this rendered machine config I can go in and I can do things like Change the ssh keys, you know, the authorized keys and that's Defined using the ignition spec, right? so it's a So it's a subset of the ignition spec and that you know, understandably, I guess that can cause a little bit of confusion, but You know using that subset allows people to if they have an understanding of The ignition spec to easily write Those kind of configuration Define those kind of configuration changes, but the way the two function are are are vastly different under the hood So, you know, you have Ignition then you have later post install day two, whatever you want to call it We have the the management of the system via the machine config operator And it uses that same syntax, but the way that it Goes about doing that is is a bit different. It's also since we are post install Post pivot the machine if config operator can't do Things like disc manipulation file system, etc. Okay, so so essentially it's a single Syntax a single quote-unquote language the ignition spec to define configuration But ignition itself only runs once and that's before system installation effectively and then Machine config is effectively just leveraging the same syntax but applying that that configuration day two and later Yep And so for a lot of things like like Derek was saying if you needed to if you wanted to redo the raid capabilities and whatnot Things get really complicated. You have a You want to you know, you've you've moved the you've moved the node from state a to state b And now you want it to be in state c. It's simpler to just Delete and and redeploy in that state c. It's like just a simpler problem to solve as well in a cleaner way of looking at it than trying to move from state b to state c And so that is kind of part of the philosophy. It doesn't mean that You know And the reason why we have a machine config operators. We're not taking that philosophy so hardcore that any configuration change requires a redeployment We you know, we don't want to be that that cruel I did want to go back though and just say that while and we can talk about, you know, how you use mco How you use machine configs to change configuration on the host and there's some absolutely necessary use cases for that Ideally we prefer that vendors and And users adopt a more, you know Clive native approach like, you know, things should be containerized if possible You don't need to write configuration files to etsy. You can use a config map The there are ways of doing this to kind of get away from the details of the host If that makes sense It's all sense. Yeah, yeah, it does and so You mentioned, you know, we don't want to be cruel and require a node reload for every operation Are there certain things that We should or or we do want people to do a node reload And I'll before you answer I'll say Andrew tends to recommend that you configure The network interface that is on the machine network.sider Using rate pre-boot. So kernel parameters are in the live ISO and copying that over And if you want to change that network config, you reload the node But are there other things is one, is that even a good idea and two, are there other things that You should be doing that with Uh Derek will correct me if i'm wrong, but you are you are following our support policy and documentation on The networking changing the primary networking after deployment is is not really supported right now Secondary networking cards are a different story and You know, that's moving in the direction of using something like kubernetes and then the state To have a you know declarative stateless sort of Way of the deploying network manager changes, but I don't know if that that's I don't think it's going to apply At the primary but to answer your question Reslicing up the disk You know and changing the rate array settings if you wanted to Just a little plug as a four seven We can even show that a little bit of how this works later. We have soft. We have support for software Raid raid one marrying of the boot device A lot of people don't like having to pay A lot of hardware and they may have remote nodes that they can't send folks out to to swap drives This buys them a lot of time if they want and without having to throw in, you know, a few hundred dollar Whatever the going rate for the raid card that they would demand Uh would be so I just want to give props to the team for bringing that feature in in four seven Yeah, that's uh, I didn't know about that. That's a big one. Um, particularly for you know The the remote worker nodes right network edge. Mm-hmm. Yep single node open shifts in the future. I see that being a uh, a big one Yeah, um, so we so we do have a question here from apostolos And apologies as always because I'm terrible with names. Uh, just ask my children Can I change network configuration post installation by modifying in m connection files in etsy network manager system dash connections It it is It is technically possible to do, uh, you know, so we're using network manager, uh, not only are we using network manager From rel on the system. Um, but we're also using network manager to uh to configure networking and then in net ram fs phase um, so, uh For example, if you guys have done a bare metal Installation and you've used those boot parameters ip equals etc. You know network manager is writing out those nm key files If you use dash dash copy network with core west installer, uh, those those those configuration files are being copied into the installs root of s It is technically possible to modify those to make those changes. Um, it's not something that we necessarily recommend Uh, because you would have to do that manually It's not necessarily possible to do that with um the machine config operator and a machine config today Based on the architecture we have with machine config pools So we kind of expect, you know, you would probably want to make uh an overarching change to all or a subset of your nodes and As as you can imagine like say static ip network configuration settings or or the alike, uh They're node specific, uh, and the architecture doesn't Um, doesn't really cover that use case so I'll ask a question that that I get asked quite frequently which is Like and it's basically an extension of this which is you know day two. I need to modify the configuration So is it it's technically possible to ssh or maybe even debug into the node and Put it, you know, put in those network manager config files to have it configure those You know interfaces Is that I know we discourage doing that You know really favoring nm states even though nm state is also tech preview. I think in 4.7 Is that something that we support It is I don't know mark few If you want to it's not it's not something that we uh the the process itself is not something that we support now certainly, um A lot of users Have concerns about this just because you you do it doesn't necessarily it's like negate support for your cluster itself I mean it it should work. Uh, we just At the moment, there's no mechanism Uh to do that in the way that um, we're imagining that you would manage your clusters Or the nodes in your cluster today, you know in kind of a kubernetes native fashion Yeah, I and I I like that answer. I think it makes sense of Like yes, it's technically possible But there's a lot of ways that you can break something and then it's going to be a bad day for you and our support guys are You know, they're going to do their best. They always do their best to help but sometimes it can be an unpredictable Configuration so but that's said right if there's if something if something works in your lab You know again as Derek said, we're not we could break it in the future, right? Like so if something's unsupported, we don't we don't consider that a persistent interface but at the same time you haven't You know, you haven't you haven't scratched the sticker off, you know and and open the device to the point where now you've You've voided the warranty If the support case if you know if you did that two years ago, nobody's going to say aha, you know You changed your network boiled again Exactly we'll never get that cluster supported So I don't mean to say I don't mean to get I don't mean to sound like full on encouragement go for it But at some level it is your cluster and if you can make certain things work then that's More power to you I'd be careful about the networking and I would be careful. I would I would also be just You know get to get comfortable and get familiar with redeploying though. I mean, it's not that bad I'm on the workers. I know it's a little bit of a trick on the masters a little bit harder But it's not that bad on the workers and you know sort of try it. Maybe you like it You can even do things like we supported we added a feature for a customer for a couple customers that you can Do the core os install and preserve the existing partitions. So Uh, if you wanted to save the container cache, right? Like So that it doesn't so that a new node doesn't take so long to to warm up and instantiate new containers Um, because that's to pull them from the registry You can just keep that you could keep that through the install Oh, interesting. So if you're using a separate partition for I think it's var like containers. Yeah. Yeah, so you can persist that and it'll just redeploy everything else Yep Very cool. Nice. Yeah, that was what you said there of at the end of the day It's your cluster and you do have a surprising amount of leeway and the things that you could do that was something that started to click with me when catherine was on the stream and You know, she was pointing out things like well When you end up, you know, when you deploy upi or ipi you end up with basically the same cluster, right? Just because it's ipi doesn't mean you can't manually add nodes and for me that was like a light of Oh, I don't only have to use machine sets to add nodes. I can still add those just like with upi You can create a machine set and deploy nodes that way Um, so thank you for passing that on I today I learned Yeah, it's it's one of those like we make assumptions And I didn't even realize that I was making an assumption about it and she she very helpfully corrected me Um, so we have a question here from mr. Chu Mr. Chaku I don't know. Uh, what could be the best way to keep secrets on open shift a way that is scalable maintainable and inline with best practices Um, I love that word best practices. Yeah, so I don't have an authoritative answer for you here I'm going to say so secrets themselves, right can be used, right? They they get stored in ecd If you're encrypting ecd, of course, they're going to be encrypted on disk that type of stuff There's third party options, right hashy corp vaults. There's a number of them I think Is there something included with acs or something like that or am I thinking of something else or acm? I don't remember I don't know. Um, so we will probably have to follow with that And I also know that there's some work going on upstream in kubernetes with csi Using csi objects to hold secure secrets. Um, so I know there's some work going on there um So yeah, and and hello It's sk routh. Um, so thank thank you for uh, tuning into the stream. We do appreciate everybody Who does watch? So I want to I'm going to ever so slightly course change here because I want to make sure that we have time to talk about something that When I talk with partners comes up on a somewhat regular basis. I also see it come up With particularly public sector customers and that is how do I add Agents and drivers that are not containerized like, you know, my security team requires every host to have an avi agent installed on it, right? You know coro s or not. I don't care, right has to have avi on it. How do we address that? I think uh, I think that's a tricky one. I think, um, there's categories of agents. Um, and When I end up talking to customers, you know, and you get to the point where like you just you triage them. The best way to do is just start Listing them out. Um There's certain things about the way openshift 4 deploys as well by the way that can be can be a little tricky, right? They can't be the The the operating system team doesn't just to install agents and pass them the golden image over. So we believe in storing the configuration of the nodes Um, as much of it as we can in the cluster itself, right? It makes the cluster self documenting It makes the cluster of the single source of truth about itself So We think, you know, as mentioned before that it'd be better if if all if all of these if all vendors sort of moved in a more cloud native container native way so, um But getting back to the tactics of actually triaging it down. Um, sometimes you'll find a certain subset don't Aren't required because there's some function in openshift 4 that actually can provide that right like the log forwarding api maybe May meet the needs of somebody though That said we have we have uh log vendors like splunky in the certified catalog um For security agents and such it depends if they support, uh, you know, they support a kubernetes demon set Then it's you're probably not going to have a problem getting that going If if they don't have direct support and what they end up truly requiring is a host agent installed on the base os We do basically we do have something of a support problem there and In that we don't we don't allow we want to keep that separation very clean between the base os and agents so, um We're thinking we're thinking long-term about how to how to do that better. Um, and how that can be how that can be managed in a more natural way um Colonel modules are a particular or a particular Issue there and that that story is actually continuing to get better and better with the special resource operator Which is going to gain functionality and uh, and and be integrated more into open shift in the future Um, I don't it's not my roadmap to share but it's uh, but I know that there's good things coming there So as far as kernel driver type agents And then and then the third party can write their own operator that manages any other containers They might need but then they could make use of deploying that came on through special resource operator Um, yeah, I've pointed partners to how nvidia does it with with their operator for You know adding in the nvidia drivers and you know vgpu and all that other stuff As one of the ways of doing it nvidia melanox Intel others have basically taken the community Special resource operator and then copied it. Uh, basically made their fork, uh, and Um, use that as a way to distribute the kernel modules until Such time depending on who we're talking about as they get into the rail kernel. Yeah Um, so so we have a question here and I don't know if this is a question for for derrick or mark. Um so More or less, it's about rust. Um, so are there what are the reasons or are there reasons for rust using rust or rustifying? I should say Not rustifying rustifying rpmo s tree and other components of coro s Who asked that question? I mean, I can give you a general answer right like And I I answer this in chat as well The biggest reason to use rust is just the memory safety of the language But it gives you kind of out of the box it eliminates an entire swath of vulnerabilities just Right out the box. So That's my default answer to that question. Um, I'm sure there's other reasons I mean that there's it's a lot of that and we're talking about really core os layer stuff, right? Where where that safety is Is it's super really important Incredibly important. So rpmo s tree is parts of that are being rewritten And rust as well. Yeah, um, I don't really have anything else to add. I trust I trust the engineering team for Uh, what you know why they think something is necessary It's the same approach I take. I think yeah They're way smarter than me. I trust them to do what they do as best as they can do it Exactly Um, so we do have a question. I don't I don't know if this is one for you and me chris Uh, the best way to start open source projects on red hats who I mean On red hat so like Using red hat tooling for open source projects because there's open source versions of everything we do at red hat. So I'm trying to figure out the and as a developer and especially a developer on an open source project So like you can go to developer dot red hat dot com and get access to Rel entitlements, you know, all kinds of other stuff there for uh, for no charge Yeah, I mean like just look at the dev sandbox for example that if you're watching right now you see a link to in a qr code and now a link in chat Um, you get 30 days of access to an open shift cluster and you can just start from there, right? Or you could go grab a copy of fedora coro s or fedora itself as your dev environment and You know often running you go there's a number learning venues as well Highly encourage you to go check out the Uh open shift playground where I think it's an hour or 30 minutes or some sort Like where it's like a full open shift cluster all for you and off you go kind of deal. Um Those are two great ways to get started. I feel like over on learned at open shift.com. Yeah. Yeah um So the the I know we're we're down to about 10 minutes. Um, so Any any questions anything that you'll have to ask? Please don't hesitate to uh post those into chest or into chat rather Um, so so derek, I wanted to ask about butane Um, not not just a gas. It's it's a tool And I'm I'm curious about um and mark mentioned it, you know When we were having our pre meeting and all this other stuff mark mentioned it and I've heard and seen the name I don't really know what it does. Um, so if if you don't mind Uh, sure. Yeah, I guess I guess it's a little bit of history here. Um, so this is uh, so with container linux Uh and ignition Uh, we didn't really talk about this earlier when we were talking about ignition, but uh, we don't necessarily Um recommend or or expect users to write ignition configs, uh directly. So, um, we the you know, the spec is published It's not documented, but that's because we're not really recommending that You know, it's json, you know, sometimes working with json's not so great And uh, but so with with uh container linux, um They had a a tool called a config transpiler And essentially what you would what you could easily do is write some uh sugared yaml for whatever it is that you were trying to configure Uh, and then you would run the config transpiler and it would generate the ignition config for you and then you pass that and And uh, you know deploy your your notes. Um Uh, the same thing exists for fedora core west upstream. There is a fedora core west config transpiler you know, that's kind of um Come from that from that same utility Uh, it does the same thing you write some sugared yaml converted into ignition and then that's what you pass when you provision a core west node Up until now We hadn't really given users a great way to do this with a rel core west and open shift but What we're doing is uh, we're taking that fedora core west upstream project. It's been renamed to butane uh, and then we're going to pull it down into the open shift product and As I suggested that will allow customers to write some some sugared yaml Then convert that to ignition config merge that with what's uh been provided by open shift and be able to do um Some of the more advanced customization that uh users might want to do for For example, like bare metal open shift deployments. So like what we talked about with the ignition um disc Network network not so much you could still do that. Uh, you know ignition itself can do that. Um, but a lot of the The disc manipulation stuff Um Is really what the use cases aimed at at the moment so for example, if you were to take a look at um A four seven documentation As mark suggested earlier. We now have the ability to do a rate mirroring on the disc So, uh, we have this documented in the open shift Documentation to say okay. Well, here's the uh, here's the yaml that you would need It's pretty simple. Uh, use the butane tool to Transpile that into the ignition config and then you can pass that to your installation Um, I can I can give like a quick demo if you think that that might be interesting I would love to see that. Sure. Yes All right I have to say the I think the team is pretty happy that they found that there was still a Like fire related term still left Yeah, they could use So as I suggested in our documentation, um, this is our four seven documentation customizing nodes, uh Talks about mirroring discs during installation. So you would do the normal installation You kind of jump over to this documentation Uh, if you want to do the disc mirroring, uh, there is an example in here. This is, uh Uh The way that this is written right now was before the uh the rename But uh in this example, um Really all we're kind of interested in for uh mirroring is this bit right here So, uh, if you've ever used the fedora chorus config transpiler, there's a variant called fcos For four seven we added a variant called rcos Uh, and uh, it's pretty simple Really, you know, you want to do your your mirroring you say, okay Well, I want to mirror sda and sdb now this this assumes that you know, which devices you have and which ones you want to use but Uh, pretty straightforward So I have uh an example Of that right here. Um, what I can do is, uh run the config transpiler so, um In the future, we're going to be shipping the config transpiler as a binary and as a a linux rpm Today There is an upstream binary and a container image with that binary. Uh, just for this example, I'll be using the container image But it's relatively simple just run butane against the butane config and it's going to spit out a ignition config for us Of course that would do that There we go So this is quite a lot of text. Uh, hopefully you guys can see this But remember that the the sugar gamble that we've written Really straightforward just Just mirror these discs together The the ignition config that actually gets generated. It's it's it's pretty complex like the the The config transpilers is doing all of this work for us. So uh If you aren't familiar with This bit right here essentially what this says is look in this particular directory for my worker.ign file This is the one that OpenShift install generated for us Uh, and then what it's going to do is it's going to merge this into this file for us So, uh, that's what this part of the ignition config is at the top here. It's uh gzip base 64 encoded version of that file Uh, and then underneath, um We've got, uh, you know our storage directives. So okay, so we need to do something with our discs Um, we need to create some file systems on top of them and then we need to do raid on top of that And as you can see, this is all pretty complicated. Um, the config transpiler does all of this work for us Uh, you know, it it figures out the labels for the partitions. It generates. Uh, gpt you you add these et cetera Defaults the size of the partitions that we want and the file system labels And does all the work with, uh, mdadm to set up the software raid devices for us. So, uh, This is this is pretty nice. This gives us A nice shortcut to here's the really simple operation that I want to do but I don't need to understand all of the intricacies of disc manipulation to write an ignition config in json So, uh, I think that's a pretty good example of um, I mean as someone who's created I'm going to say extremely basic ignition files before Knowing the syntax and the nuances and everything else, uh, this makes me really really happy Um, so our do you have or is there a list of the types of configuration that are supported by butane? And you you said sugared yaml, which I'm not familiar with that term. I I think it means basically a specific syntax or Yeah, yeah, just just specific syntax. Just uh, just uh, just an easily consumable uh Format So I'm just yeah, I'm just kind of using that as a shortcut. So some things that butane will will will translate Are going to look this almost the same in yaml or json, right? It's just going to be the formatting But the sugar is those four lines expanding out to like 87 lines as that's where butane is doing the heavy lifting to calculate all this out and uh, Make sure it all matches. It's taken care of also knows it's taken care of the boot partition as well making sure that that's married Um, because there's no point in if you lose your brute devices. No point if you can't boot off the other drive Yes, that that would be bad So you were asking about, uh Uh The the spec I guess Yeah, well kind of like you had a very specific as mark said four lines for doing, you know, mirroring of the two boot disks Um, are there other operations or other things like that that are possible Yeah, so, uh, let's take a look at some of the examples here So the examples in the upstream uh core west butane repo are specific to to fedora core west For the most part the r-cost variant is a subset of the f-cost variant So most of these examples Should work you just need to change the the version Um The we haven't really talked about fedora core west and what it means to rel core west but It it doesn't have the same use case upstream. It's more of a like a standalone Something that you're you're encouraged and in most cases required to customize. So that's why there's more about How to interact with ignition? and use the config transpiler For those use cases for fedora course and there are for for r-cost But some really and the reason I say that is because there's some really simple examples in here. So like here's how I would define SH keys for the the default core user Here's how on fedora core west I could create another user You could do that on r-cost Ignition will allow that that's not something we Generally recommend that customers could do See thank you for skipping over password authentication So file systems and partitions there you go. So if you wanted to create, you know, if you wanted to manipulate secondary disks like stb in this example, this would be an example of how you could do that this in this example You know, it's it's taking stb Making a file system mounting that at bar. It's going to put butterf s on top of it so and just just to complete the thought process here, so excuse me the the example you showed was a short and simple rate of take the worker ignition file and Append this configuration for mirroring mirroring the disk And then that output is what you would then provide to your nodes when they're booted Yep, so if we go back here All I need to do is is save this output this this ignition config and this is what I would pass to my node. So if I for example all of my workers I wanted to raid all of my workers and the this is important distinction is they all have an sda and an stb Then you could generate this file once and and use this to provision all of your workers instead of The original worker dot ign file and it would it would include all of the changes from the worker dot ign Plus all of the changes that you need to do the rate configuration So we're we're at the top of the hour and I want to be respectful of everyone's time. Um, we we do have One last question. Yeah, so if you want to take just two or three minutes to answer that So sci Ask how do we get started with rel coro s? Don't see much in terms of documentation at least to understand and compare with rel especially rpm os tree I would answer that question kind of Counter-intuitively, I would say you don't get started with rel coro s, right? Like You get started with open shift and then you use rel coro s with open shift But I understand that some people think I need to understand the foundational layers first So rpm os tree would be the thing that you really need to understand the most I feel like Is that fair to say? Possibly as well as ignition. Um, I think obviously, yeah, you know, I Fedora core os is in a in a way a good way to get started You know, you can look at it as a standalone container host It also can be clustered and I used under okd as far as I know. Um Yep, but the upstream documentation for rpm os tree, uh, and And ignition is is pretty pretty solid And I it's uh, we can put links in but it's not too hard to Too hard to grab. Um, that would be my advice. Um There's also ways, you know, uh, it's distributed as a live iso as well So you can just take the iso file of This is four seven plus. I want to say and Four six and and just fire that up as a diskless vm on On kvm or or anything, right? And you just boot it up And I mean if you're curious about it, you know, what do you what will you do after that? I don't know. You can use the rpm command to see what's installed. Um Run some containers on yeah, and hot man's there. So yeah, it's possible So I think si might be flirting with something that we talked about in in the the meeting that we had yesterday It's sort of the pre brief for this and that is A lot of times rel admins will approach coro s and think of it the same way and Really coro s Especially in the context of open shift is meant to be more appliance like and you can inject configuration via Why did my mind completely went blank? Ignition or machine configure? Yeah So you can inject configuration before installation via ignition But after deployment, it's managed through open shift. And so you have a series of operators machine config operator file integrity operator tune d operator, right? Oh, you have all of these things inside of open shift that do the post deployment configuration The the goal being hopefully You don't have to connect to the coro s node. You don't need to ssh and do administration rather. It's done through the cluster itself I would add though that you know that you can that when you do ssh and you won't feel Completely on it, you know, not at home. It's different because of the you know, you've Mainly the package manager would probably be the first few things people run into and wonder what's What to do there, but that's pretty straightforward to get around Yes, it is it's really it's really first and foremost a component of open shift And that's kind of the way we we look at it as a as a an opinionated version of of rel for open shift Okay, well, I think we're out of time. I know we're out of time. We're four minutes over. Um, so Thank you mark. Thank you. Derek really really appreciate you spending time with us today It's been a pleasure having you on. I've learned a ton today. I hope that this has been informative to our audience Uh, so for anybody who has a question that you didn't want to ask publicly or we didn't have time to get to Please feel free to reach out to me. Uh, andrew.sullivan first name that last name at redhead.com You can also reach out to me on social media twitter practical andrew. So if you've seen me posting in the chat here It matches my twitter username Welcome to send me messages at any point in time. Uh, chris as well He now has a nice convenient easy to remember email address short at redhead.com. Yes I'll stop getting those confused now. Yeah. Yeah. Yeah. No at C short at like there's too many C's right like you can spell C in so many ways You don't necessarily know just by saying it. So yeah, uh, but short at redhead.com and uh, chris short You know all one word with two s's in it. Um twitter feel free to reach out anytime I just dropped a link to our discord. You can join that as well And dms there if you need to as well as uh, kubernetes slack, um, we're on there as well And uh, keep an eye on open ship.com slash blog Friday morning is when we'll post the blog post that has All of the follow-ups links, uh time stamps all that other stuff inside of there. So yep Thank you everyone. Thank you again, especially mark and derrick for joining us today and have a great rest of your day Thank you guys. Thanks everybody. Thanks appreciate it