 Good morning. Good afternoon. Good evening wherever you're hailing from welcome to another episode of the open shift administrator office hours Today we're talking about controlling pod resource management But the one and only Andrew Sullivan here has some follow-up from previous episodes. So why don't you start off with that there Andrew? Indeed. Well, thank you Chris and happy happy Wednesday. Happy Wednesday. Indeed. Yeah, it's it's gray. It's nasty here in Raleigh It's I've got piano movers in in our house. So apologies here. Yeah, if you hear any Just know that it's temporary and everybody's probably fine Probably unless you hear sirens in which case I might would we hear sirens from I guess we would yeah But I bet I got you beat in the temperature department sir Yes, well right now the thermometer in the other side of the room says it's 77 in this room But that is sitting on top of a on top of a computer. So it's probably getting a lot of exhaust heat Yeah, yes negative 11 Celsius 89 Fahrenheit. So yeah, it's a little chilly here Let's see my computer says it's 45 here. So yes, you you win. I think yeah, that's a win The race to the bottom best for sure All right, so this is sorry about that if I'm always banged in your ear So this is the open-shift administrator office hour So the goal of this next hour or 56 minutes 57 minutes as it were is to give you all our audience The ability to ask us questions about anything that you want relating to open shift So I have an administrator background Across many different disciplines Chris. I know you have the same as well So we'll do our best to answer any questions that you happen to have Be they administrator related or developer related just know that Well, we'll definitely have to phone a friend if it comes to the developer side of things That being said Don't hesitate at any point in time to send us a chat So yeah across twitch across YouTube across all the various platforms that we're on And apologies for the dog who seems to have escaped at the moment. No fun. Yeah, so Don't hesitate to ask any of those questions Regardless of what we're talking about today as Chris said today. We'll be talking about managing resources for applications for pods, etc But don't let that be the only topic that you want to bring up ask whatever you want Exactly. Yeah, that's what we're here for. We just use the topic to fill in for when we don't have those questions, right? so as Chris mentioned as I usually do at the start of these shows I have some follow-up some clarifications a lot of time some corrections because Andrew is often wrong Yeah, Christian. I know so this wall when I moved offices. This wall is a lot closer And the picture is Basically too big it takes up too much room. Oh, so I need to find smaller pictures now So the raccoon which is actually hanging right over here is a 24 by 24 and oh, that's kind of big Yeah, yeah, this wall is behind me. So yeah, way way So just just go with screens man. It's so it's it's way better. You can change stuff if you want it I do appreciate your attention to detail Christian that that means a lot to me Mm-hmm. Yeah So again ask questions at any point in time. So follow-ups a couple of things to talk about So first a couple of weeks ago maybe three or four weeks ago right we talked about updates I think and disconnected and a couple of times now on the stream I've showed the Cincinnati graph data mm-hmm so Cincinnati is the update service inside of OpenShift itself and The data that's that's found in that repo is effectively how OpenShift determines or OpenShift knows hey I can update to this version from this version or I can't update to this version or that type of information and If you look at it in the raw repository and I'm going to share my screen because it's easy Yes, please do talk that way So if we look at the github repo mm-hmm, and we just go to OpenShift comm and We'll search for Cincinnati We want the Cincinnati graph data. I'll push this into the chat here So if we look inside of here, the raw data is not easy to interpret Right. So for example, if I go to channels here I can look at for example the stable 4.6 and I have this big long list of all of the the versions that I'm able to update From and to inside of here. Yeah, so oftentimes we have used a Third-party site that is published by one of our product managers Rob Zomsky to kind of get an easier visualization of this Better solution still not great, right? It's on official. It's third-party, and it was still very text driven So sometimes hard to understand. Yeah, however, Rob very helpfully pointed out Yesterday that we have a new tool from the OpenShift Labs folks. Yay And I have to say I really like this thing. This this thing is phenomenal. So oh, where's the link to this thing? Yeah, I'm not all I need to know about it. I'm pushing it in here Christian don't don't don't ruin that to Apple buying duck duck go. That's not okay So When we look at this particular tool, it's really good for a couple of reasons So first and foremost this gives me an upgrade path So remember with OpenShift, I can't go directly from say 4.4 to 4.6. I have to do an intermediary 4.5 first So for example, let's say that I want to use today. I'm on stable 4.4 And maybe my cluster has 4.4 dots. We'll go with 19 I'll just choose one of these at random. There you go And I want to update to the latest version right 4.6 12 I know that that's not the latest latest 4.16 is technically out But I think it is I don't think there's any update channels for it because it's like brand brand new So essentially what I've done here is say I want to go from 4.4 19 to 4.6 12 Show me what that looks like and if my Login didn't time out here. It generates this You need to go from where you're at change your Subscription or not your subscription your channel to 4.5 update to 4.5 to 27 Change your channel to stable 4.6 and then go to 4.6 dot 12 So it walks me through step by step what I need to do and then down here below I have each one of the steps that I need to take in order to do that so Click administration go to cluster settings set the channel to stable 4.5 select version 4.5 dot 27 So it walks me through each one of those and exactly how to do that Super super helpful, right? No longer any guesswork of Okay, if I go to 4.0, you know from where I'm at 4.4 to 4.5 dot something Am I gonna have to do another 4.5 not something update to get to a version that's eligible to update to 4.6? and So takes care of all of that The other thing it does is it generates this. Um I'm not gonna say Not useful, but very difficult to read update channel graph That is Yeah, something. Yeah So we we've seen these a number of times. Um, so this is what We generate these internally and share them sometimes This is the visualization of all of that yaml data that I showed over here in the repo, right? So This is all of the edges connecting to all of the versions and what that looks like as a visualization So I find this useful for a couple of reasons so but Useful might be a stretch. Uh, so for one Over here in purple or excuse me in light blue the highest version in the channel So it's pretty easy to see. Okay The highest version I have available is 4.5 dot 30 You know if I kind of click and hold this you see how it highlights All of the source edges so I can see all the versions great. What version do I need to be at? in order to update So that's kind of useful you can sort of drag these and rearrange them if you really want to I mean feudal to me, but I just when it comes to graphs like this, I just can't Yeah, it's like it's just too much for me to like I'm glad there's the highlighting and dragging around but like I need something a little bit Yeah, I have a wide screen use it, right? So one thing to note in this view you see if I let me go back to the stable 4.5 It's actually highlighting the path that it has recommended to me So up here at the top remember it was 4.4.19 to 4.5.27 So here's my 4.4.19. You can see 4.5.27 is my update path You notice that 4.5.30 is available, but I can't go from 4.4.19 to 4.5.30 So If I had just Blindly chosen one of those versions, right? I would have had an error or it wouldn't have been available And then likewise with the 4.6 It'll do the same thing, right? It'll show me the path that it has chosen for me. So from 4.5.27 to 4.6.12 So again useful definitely encourage you to check it out if you're going through the update upgrade process, etc There is this other Show upgrade update graph by channel. This is basically exactly what you just saw that massive line chart Just for all the 4.6 versions instead of just the ones that I had selected So whether or not that's useful up to Is there individuals but it is there it is an option Yeah, um The most do with it as you please Yeah, so uh, let's see other things that I wanted to follow up on So one of our viewers killer goalie had reached out to me. I think it was two weeks ago Which was the week after we did a show on disconnected So one of the things that we ran out of time for was mirroring an olem catalog from connected to disconnected So I will point out that I believed the documentation as as it was written and when Unfortunately, we ran out of time so I didn't encounter it on on while we were broadcasting while we were streaming But there's actually a bug so The goal here is kind of twofold right if I want to mirror a catalog. So let me pick a uh, let me bring up my cluster here Go to operator hub So I have these Provider types which are effectively analogous to a catalog source So let me switch over here and do an oc get catalog source dash a And you see I have these different catalog sources here So the goal is I want to take these red hat operators and I want to mirror all of the operators that are in there over to my disconnected network Let's switch back very common use case. Yeah. Yeah, so if I just select this provider type of red hat Like these are all of the things that will get included so we do this as It's two separate things that have to happen. So one is I need to mirror the uh index the catalog index And then the other one is I need to mirror basically all of those images So the index is pretty straightforward. Um, so again, I'm going to switch back apologies for all the jumping around So if I do an oc describe catalog source Red hat operators And folks if anything isn't readable for you, let us know. Yes, please. Please. Yeah, we'll make it readable for you So one of the fields in here is the image, right? So all I need to do is effectively Mirror that image. There is a very helpful command for that So it is opm commands and we covered all of that during the previous Session so I did get this far of how to create or how to how to pull down that image The step that I didn't do was okay. I pulled down that image locally Now I just need to use podman to export it, right? And now I have it in a file that I can put onto a thumb drive or something like that and move to my disconnected network So the other side is mirroring all of those images And this is where it gets messy because in theory, I should be able to use the command oc adm catalog mirror To mirror those images Okay, so and we even have an example here of mirroring from a file Right or here here's from mirror From yeah, mirror to file so At least with 4.6.12. I will admit I haven't tried it with 4.6.6 Or uh 16 whatever version I'm on because I just deployed that yesterday evening, right? So This didn't work Effectively what happens is it mangles the output into the mirror file And it causes it to be invalid. So let's see what that looks like Oh, now I'm curious what it looks like So if I were to attempt to Here's one of them So if I were to attempt to mirror this data And let's make that easier to read by eliminating all the noise on the screen So I've got my oc adm catalog mirror my source Index image. So this is one that I pruned so that it only includes the open shift virtualization operator Nice, which by the way is 549 images totaling 45 gigabytes of data Because it pulls all versions and all the dependencies and and across all Architectures so There's a bug that's where it requires you to do this filter by os so it pulls for Power and all that other stuff as well as x86 And then in this instance, I have right and I don't want to do that, right? I want to go to a file So I'm going to say dump all this stuff to a file at Just this location And it'll think for a moment and what we're going to see is it And maybe it'll work because again, this is 4.6.16, right? It's going to dump A whole bunch of error messages into the mapping dot text file and basically instead of being file colon slash slash Location it's slash file colon slash location Which is a mangling So effectively and the fact that it's already not failed tells me that this is already fixed in 4.6.16 So effectively it wouldn't work Because this is probably going to work What I can do at this point or once this This finishes is I would be able to target zip all 40 something gigabytes And move that over and then reverse this process, right? So yeah Change this so that the file one comes first and then my disconnected registry name is second And then apply my image content source policy so Just know at least for now Even if you're not using 4.6.16 if you don't haven't deployed 4.6.16 cluster It looks like if you use the 4.6.16 oc client Judging by the not error that i'm getting at the moment It may actually work for you So killer goalie if you're out there if you're listening Please know that It's worth giving a test That being said i'm going to stop that now just in case it's trying to pull down a whole bunch of data Oh, you don't want your disk to blow up. I uh, you know, I pulled it down once It's sitting in my object store. I don't need it again on the file system Um, but just know that I did create a bz for this. Um, so they are looking into it So it it may just get closed and say fixed in the new version Maybe they'll backport it. I don't know But just be aware of that Cool, okay So the other two things, um quickly that I had to talk about are and chris You know that I like to talk about things that I see come up internally that are on behalf of customers or kind of preemptively addressing customer issues and concerns who are a very customer focused person. Yes Um, thank you. I I try uh, so the first one is the cluster network definition So let me switch back over to my cli instance And if we go to Here And yep, that's the one I want Notice how I have that clean one. That's the one without my pole secret. Good call So Inside of an install config. I have this networking stanza and inside of there. They're going to be three network definitions So I'm going to start at the bottom and work my way up So service network is the set of ip's that will be used that will be assigned to service definitions So when you create a new service object, it will be given an ip from this range Nice So the machine network sider is the public right so the the actual network inside of your environments Where at least one interface from your nodes from your machines will have an ip address So this is important and I've talked about this before this is important for things like the stn That's how it authoritatively determines which interface to use if it's missing or if it's incorrect Basically, it doesn't see an interface on that subnet. It just chooses the first interface and hopes It's good for things like proxy config. So if you're doing a cluster wide proxy This subnet is automatically added to the no proxy list So we see that a lot of times with folks who are they're deploying a proxy They don't set this correctly and the install just fails repeatedly. Yeah, I've got my proxy config correct. Well You can you know, if you don't set this correctly, you can manually add the subnets to the no proxy and then it should work as expected So that's one common reason why proxy fails And then the third one here cluster network. This is the the range of ip addresses this slash 14 With that will be used to assign a subnet of this size slash 23 by default To each node in the cluster for pods to utilize So if I have 10 nodes in my cluster each node will be assigned a slash 23 out of this 14 And pods that are instantiated on that node will be assigned an ip address on each one of those subnets right, so if I Switch back just to give us an example here I want this one and if I go to my nodes And I'm going to choose this worker node and look at our pods And what I should see here is and I'm just going to pick one of these at random When I scroll down to find my ip address see this one is 10 dot 129 If I do the same thing again, so let's go to nodes and go to a different worker And we'll choose another random pod here. See this one is 10 dot 128 So each each one is going to be on a different subnet inside of there so We hadn't basically somebody asked why are these such big ranges right Do do I really need a slash 14 and a slash 16 for these networks and the answer is maybe so the host prefix of a slash 23 is We do this because the maximum number of pods per node is 500 And a slash 23 is effectively 510 usable addresses All right, there's 512 technically, but of course first and last blah blah blah. Yeah So we need a slash 23 to be able to to maximize the number of pods right or reach that maximum number of pods per node If you have no intention of ever putting more than say 256 You know pods or 254 pods on a node. Yeah, bring that down to a slash 24. Nothing wrong with that Do I need a slash 14 here? Well, that depends on how many nodes you have in the cluster and the size of the host prefix Yep, so for example if I have 50 nodes in my cluster And I know I know I'm I'm looking over here because that's where you are and I realize that the camera's over here I should stop doing that. You should look into the lens of the camera. I know So if I have 50 nodes with a slash 23 so 510 times 50 is 25,500 No, I did not do that in my head. I did that yesterday when I responded to the question So I knew that one off the top of my head So 25,000 ip's falls right in the middle of a slash 17 so I think a slash 18 is like 16,000 ip's and a slash 17 is 32,000 ip's I mean, I can pull up my subnet calculator if you want me to So 25,000 means that I need to have at least a slash 17. So you can reduce that Same thing for the service network, right? How many services do you expect to have? How, you know, basically determines how many ip's you would need on that particular network So one of our illustrious essays also pointed out that He prefers to start at the bottom of this range. So rather than 10.128.0 do like or a How favor dot zero here over say dot 128 10.128.128 Because if you need to expand it so Maybe you need to add some more ip's If you roll over into the next subnet that's harder into the next rate if you exceed the next octets capability Right. That's that's a lot harder to do so and painful. Yes Yeah, but do you remember effectively the cluster network the service network? They don't need to be real ip space, right? They don't need to be routable ip space, right? Yeah, like please don't have them. Yeah, routable ip spaces. Exactly. Um, so I know a lot of times we have um ip space I feel like is a lot like City planning right some some cities it happens very organically, you know I'm looking at you boston and and some others some cities. It's very planned right new york city, etc so Sometimes you end up with a big patchwork of available ip space and finding a contiguous, you know 14 or 16 is hard in the non internet facing ip space. So Yeah, you don't necessarily need those No, you don't but they are there. Yeah so, uh, the last thing I have is um a question courtesy or a I guess it's a question courtesy of christian So christian and I were chatting yesterday about vSphere. He brought up vSphere data store clusters So if you're not familiar with the concept of data store clusters and vmware They're effectively the same thing is a drs cluster, right? I take multiple data stores. I put them into a cluster And when I am provisioning my virtual machines, essentially, I just select the data store cluster and whatever my capacity is and it chooses which data store to put it into Similarly, it will automatically if it's allowed to storage the motion between The different data stores to balance right different characteristics performance capacity, etc So the question was why don't they work with you know, open shift? Why don't they work with the storage prisoner? And the reason is less to do with open shift and more to do with well the storage provisioner itself So unfortunately the entry provisioner doesn't support data store clusters at all Basically, it doesn't recognize them to be the same type of object So you can't specify one and and let it do its thing I don't think the csi driver works with them either, but I have not personally verified that So if you want to still take advantage of multiple data stores You have two options. So one and the fully supported way is to go in and just add more data stores into Your vSphere config and let me post the link to that of how to add Here we go So I will post this link in here And I also bring it up on my screen Nice. Thank you. I have to try to log in of course I feel like I just logged into this I know it's never ending. Yeah I did to look at your graph. Maybe that's a different browser All right, so down here we have this this workspace definition, right? So we can add multiple of these in order to define multiple data stores and then we have One data store per storage class definition Kind of clunky, right? That means that whoever's creating the pvc has to be aware of that Spray it's not always a good thing So the storage provisioner does support the concept of Of policy based choosing and I should have thought ahead here and had this link already up and available But I happen to have a happen to have it bookmarked, which I was told that bookmarks are an antiquated concept and Really? So yeah, you just remember all the links, right? Do you just Well, I mean I have taken a different approach to bookmarks, right? Like I'm tired of them being like locked in a browser So I've put them in a different app, but they're still bookmarks, right? Like yeah. Yeah. Well same here um Anyway, so in here We have the ability to specify a policy Right, so I can define those policies as being tag-based So I could select each one of my Um data stores here. I can add a tag right and we'll call this storage Yep I'm not really gonna Worry too much about it So I can assign tags to each one of my data stores and then I can come up here and go to policies and profiles I can create a vm storage policy and then I can use one of these to In my storage definition. So I can say store provisioner Choose a data store that meets this policy essentially So maybe that's you know flash storage. Maybe that's you know, I don't know block storage whatever that happens to me Right, and that can match multiple data stores So I can kind of mimic that same functionality of having more than one data store behind masked behind one interface So the problem is well, it's not a real Data store cluster So I'm not getting any intelligent placement based off of the normal storage cluster rules It also won't move those disks according to storage drs policies not that we encourage that to begin with right And last it will choose one data store. So the policy will execute It'll say data store three. You're you're the first one to come up and it will create pvcs in that data store until it's full And then it will choose another data store and it will create pvcs in that data store until it's full So it kind of sequentially fills each data store Rather than doing them on an individual or or spreading them across all of the data stores in there So data store clusters They sort of work, but they don't really work Effectively, we don't Definitely don't recommend it. It doesn't give you the benefits that you would expect So unfortunately, I don't have a good answer for that today if data store clusters are your preferred way of choosing storage for virtual machines if somebody Wanted to do that who would they reach out to first you think us or VMware? um I would so The entry provisioner is supported by red hat okay However, the entry provisioner is deprecated right remember entry provisioners are being removed in the future. I don't remember what version it is now So essentially, you know, sure we can create an rfe maybe something like that is already there but If i'm being honest, I wouldn't expect action to be taken on it So the csi provisioner is in the cloud native what vmware calls cloud native storage right is the active and current and the recommended way of consuming that storage So csi provisioners are maintained by the storage provider. So you would open an rfe with vmware in that instance And then vmware does support right so the vmware csi provisioner is supported By vmware with open shift And you can absolutely deploy that in there. Just follow their deployment guidance So usually the big change for us for on the open shift side is making sure that the virtual machine hardware version, which is very An oxymoron right virtual hardware version. Yeah is Is at least version 15 And then you can deploy the csi provisioner in there You can take advantage of the vcenter integration You can take advantage of if you're using version 7 vsan version 7 with v sphere 7 the rwx Volumes from vsan all that other type of stuff Okay, I think that covered all the things that I wanted to bring up here beforehand and look through my tabs They all look good. So Or they all look like I've addressed them So I think we're okay Yeah, we're good. All right, um any questions No, uh, Ricky said that city planning is a great analogy. So, yeah, that's a good one Thank you. I actually came up with that right off the top of my head and wow, it seemed apt. So All right, so today's subject is controlling resource consumption And you know, we've talked about sizing before we've talked about, you know, a bunch of different aspects of what's happening inside of the cluster Um, but at the end of the day, right, we can size the clusters as much as we want um, but if The pods if the application teams aren't responsible stewards of those resources There's not a lot we can do So we can help control that through a couple of different mechanisms at the Kubernetes level So the first thing I want to point out is there is a very real and very distinct difference between The amount of resources that have been requested and the amount of resources that are in use And this plays a huge role in sizing So for example, I can create a pod that does nothing except sleep, right not consuming zero CPU time Right zero memory, but I can associate a request with that for As much CPU as you'll let me right as much memory as you'll let me And from a scheduling perspective Kubernetes will effectively reserve those resources So I could have you know, these massive nodes in my cluster That have effectively no real work happening, but from a scheduling perspective, they're full So we want to be very conscious of we want to be very aware of what those requests what those limits what those configurations are Because it can affect that Pretty significantly, right this cluster is running in azure, right? I could effectively autoscale up to dozens or hundreds of nodes that are doing effectively nothing You know costing me a lot of money and I'm not doing any work So really important to be aware of that The other thing that I'll soapbox a little bit on is Trust the metrics tools Um, so I'm doing a webinar with a partner of ours Turbonomic tomorrow, you know, they Fantastic tools for and they're one of many of our partners that do this You know for looking at and seeing and evaluating the real resource utilization associated with an application So that you can then go to that application team like look I have proof that you're not using all of these resources that you requested Let me downsize. I promise it's not going to affect your application But it allows us to be more efficient and better at what we do and more cost-effective Effectively so trust the tools build the trust with the application team as an administrator that you know Hey, we're not gonna, you know, we're not taking these resources away because You know, you're never going to get them back. But when you do need them, they will be there, right? So again that right That whole trust thing that we that keeps coming up again and again Okay, yeah So the first thing I want to talk about is kind of a couple of core things And I've already talked about them in previous episodes. So I'm not going to spend a lot of time And I just mentioned the first one which is a request So each pod can define a requested amount of cpu and ram Right, it can also define a couple of other things inside of there And this is not the page that I wanted. So I'm just going to ignore that page So That request is effectively a please guarantee me or please schedule me on a note that has at least this much resource available, right? Now the opposite of that is a limit So a limit is don't allow this pod to consume any more than this amount of resources And the two of those don't have to be equal Now Right, and they actually they both don't have to be defined either So the kind of it is advisable to define that. Yeah, very much so So the combination of those two things determines the qos class associated with our pod So if my resource request So if my request is smaller than my limit it falls into the burstable class Right if I think it's burstable Wait, if it's say that again if it's if it's under or over So let's see if a request is less than the limit Then it is a burstable qos class. Okay, right? Yeah, if there is no request or limit specified It is a best effort class And if the request and the limit are the same and I had to check my notes on here just to be sure It is a guaranteed qos class Nice, so What do these mean? So the biggest impact that these have is when it comes to resource contention Effectively so we can think of guaranteed as being the highest we can think of best effort as being the lowest Effectively it will start at the bottom Best efforts and it will begin evicting those pods from the node as resource contention occurs And you have nothing specified it's gone Exactly, right? So if you want to and you know, there are some constraints on that, right? It's not gonna like this is why you want to have a pod disruption budget So that way it doesn't go in and and eject all 40 pods associated with you know This application component and suddenly everything's down right pod disruption budget events things like that from happening So you can see that there's a number of factors That play in here, right that all go into this grand scheme or this grand plan of how do I control? How do I manage not just the resources but also my application? It's really important for application team and administration team infrastructure team to work together to ensure that we all understand those things Um, you know, it's I I know from when I was administrator I may or may not have done this before of you know, hey, I need to put this node into maintenance mode and reboot its rates Um, and accidentally taking something down or something, you know, this this can't be the emotion. That's fine I'll just turn it off for a minute and Um, you know, we went to avoid those types of scenarios. So working together All right, so requests limits, right requests is um for lack of a better term or reservation It's not really a reservation, but that's how I'm going to describe it for right now Limit being don't consume any more than this right those two combined lead to the qos policy So how do I prevent? Chris and chris's project from consuming all of my resources And you know me I'm gonna get it all you're I know you the the I'm a resource hog. Just say it exactly so I hope that you are not surprised by the page that you've been staring at for the last five minutes or so the answer is a quota So I can place a quota onto a namespace onto a project that says And it defines different aspects of resources that you are allowed to consume or not consume So you can see here. I can define a limit for cpu as well as memory I can define requests for cpu and memories to remember You are not allowed to consume more than x amount of cpu. You're not allowed to reserve more than x amount of cpu And this is just the legacy same as requests, right? Interestingly as we scroll down further in this page. This also applies to storage So I can define on a per storage class basis the amount of The the capacity so the number of gigabytes that a particular project can consume As well as the number of volumes that a particular project can consume So you're not allowed to consume more than 100 gigabytes of bronze storage and more than eight persistent volume claims Okay. Yeah, that's what I wanted to get to right like where is the class of storage and so forth It's that's where it is and then you'll note that up here I skipped over these two at the top. So these two apply across all storage classes all storage types So maybe I've got you know uh, you know, gold silver bronze cardboard plastic storage classes and individually, you are allowed 100 gigabytes from each one of those five But I can say up here at the request dot storage level Total collectively you are not allowed to consume more than 250 gigabytes across those five So maybe that's your full 100 gigabyte allotment of gold your full 100 gigabyte of allotment of silver and then 15 gigabytes each of you know bronze cardboard in those, right? Yes, so storage resources Just as important now note that this is gigabytes. It's capacity based Uh, effectively kubernetes has no concept of things like latency or IOPS Which is of course important for many storage operations. So Just be aware of that right that it is it is a thing And there are ways that you can control that depending on your storage vendor. So Refer to their provisioners their CSI provisioners because many of them have a lot of different things to help with that I know pure. I know net app. I know Dell all have the ability to define things like qs policies and IOPS limits inside of their storage class definitions. So Cool stuff The last one that I'll touch on is There is so ephemeral storage So when we talked about sizing node storage, right? Remember we talked about how slash var is where things like an empty dur gets created at And it is a finite amount of capacity So using these ephemeral storage definitions I can control how much Of that local host storage basically the ephemeral storage under any of our live containers The project is able to consume So this one is a little harder to nail down because you can say oh christ you get one terabyte of ephemeral storage You know, don't use any more than that But maybe your hosts right and by default we recommend 120 gigabytes drives on those Open shift nodes. Well, that would effectively be like 10 nodes worth of capacity that you could extinguish So it's a little bit harder depending on what your configuration is, but may still be an effective at least control mechanism So, okay, so we've talked about what the objects are, right? So request limits qos classes We've talked about how to implement quotas, etc So quotas are all great and all of that Now, but maybe I don't want you much like we defined up here You're allowed to consume, you know, this much of bronze, but this much overall Right. So can we do the same thing with cpu and memory? The answer to that is yes using a limit range And unfortunately, there's not a good definition of a limit range in here. Let me No, that would just make sense I'm gonna copy one from over here And go here and I'm just gonna Borrow the yaml editor inside of here. Yeah, why not? So we have a limit range definition where we can say effectively a couple of different things So one if you are I'm not gonna say lazy. I'm not gonna say irresponsible. I'm gonna say careless Application administrator who does not define any of these values Yourself, right? We have defaults that can be defined in here, right? If you don't if you don't create a cpu memory, right? ephemeral storage request or limit you're going to get what's defined inside of here And note that these can be defined at the container level as well as at the pod level as well as at the project level nice Additionally, you can specify the min and max available So your quota is 100 cpus and a terabyte of ram But you cannot create an individual pod that consumes more than two cpus and two gigabytes of ram Right, so kind of maybe potentially preventing you from shooting yourself in the foot Um, similarly the minimum is great the minimum and the defaults right are great for that whole scheduling thing of I always want to make sure that at least some resources Are accounted for at the scheduler level so that way it can make more intelligent decisions So Limit range is super important, right? They they help with all of these things They help prevent people from shooting themselves in the foot. Um, and unfortunately, I think we're gonna run out of time I'm gonna share a github repo because Chris, I don't think you had joined our team yet. I actually created a whole demo on this I'll share the video. I'll share the the examples that I have from back in like the open shift 4.2 days so It might need a little bit of updating but um, I'll I'll share that github repo actually. I'll do that right now Okay, thank you And I will dig up the video as well Because that one's easy to find I think you hope So Just to be clear this demo is two years old and it's still viable, right? Yeah The resource management has not changed in kubernetes significantly exactly I don't see a glance here. I don't see the video. So I'll dig that up I'll include it in the blog post that goes out with the uh show notes. So wonderful. Yes. Thank you Yeah, and if you're not aware, uh, Andrew does a very good job of getting a blog post up after Each episode that he's on. So yeah, subscribe to the blog and you'll see all the notes that you need to see from this show Yeah, in the last few weeks. I've been those have been going out at uh, 6 a.m. Friday morning 6 a.m. Eastern time Friday morning So if you're an early riser first thing Friday morning I mean drink your coffee and hear me. I'm I'm so sorry I always publish my post at like 3 in the morning. So, you know, whatever Um, so I'll I'll dig up that video. I'll share that. Um, you also have that github repo Which has a lot of examples inside of there to your point chris. These are core kubernetes concepts This is an open shift. This is kubernetes, but it's something that I have found we as administrators Really don't understand well because a lot of times we we think of it as it's an application thing, right? The devs are creating these things and assigning them with the pods Without realizing or understanding how much of an impact that has on what we're doing And that leads me to the last part of this that I want to talk about which is This is great. Hey quotas, you know phenomenal. Hey limit range is great. You know, that's that's phenomenal But each one is scoped at the project level at the namespace level Which means that for every project every namespace I create I have to have a limit range and a quota so if you're Allowing users to create their own projects without, you know And you know the the old school way would be you know, hey submit us a request and somebody will create your project and hand it over to you, right? Nowadays we often hand over an entire cluster to them and say go forth and do Yes Let me get in your way. So not i'm coming at all now We can we have the ability to Define a default that gets created with every project So how do we do that? And now I need to find One of my other windows All the windows all the windows there's I have far too many of them. Yeah So how do we do that? And we'll switch over here And clear out the noise here Because what I want to do is show this command so OCADM create bootstrap project templates with an output format of yaml And you can see I've simply directed that into a file on the file system here If I look at this File if I look at this yaml output what i'm looking at is the template That defines how to create a project And you can see a number of different things inside of here. Apologies. I'm gonna Jump up and move this window so I can see both kick both. That's really hard So my camera shakes. So yeah, enjoy it. So I can see both my code and the chat and this all simultaneously Or pseudo simultaneously So you can see in here We have a couple of variables of things like description display name. These are defined at the bottom So we see down here the parameters But we importantly have multiple objects inside of here So When I create a new project, it's going to of course create a new project using you know So when I say OC new dash project, it's going to use this information to create the project Object it's going to create a role binding object that follows these types of That has these Security permissions defined. So we see the the cluster role here and the user role that are bound to the admin user Whoever's creating it. We have our parameters down here at the bottom So if I want to for example, create a default quota I simply add it into here Right, so I just copy and pasted Very simple resource quota definition here Right. So kind is resource quota project name dash quota Right, you're allowed to have no more than 10 pods. You're not allowed to have more than four cpus eight gigabytes of ram Right so on and so forth. Yeah So I'm going to remove this one real quick. And then I also want to add A default limit range. It's going to be the exact same limit range that we just looked at a moment ago Right, so project name dash limits is going to be the name of the object Right, so I'm just adding in whatever it is that I want to be created if I want to create default users more than just the admin user quotas Right on and on and on all of these things inside of here. I can define these with the project template So we'll save that particular file And now I want to create that object. So If I look at the head here, you can see that It is a template with the name project request. So I'm going to submit this so do an oc create dash f Um project templates and I want to put this into the open shift config Project So if I do oc get templates namespace open shift config Apologies, I can't type and talk without saying what I'm typing in if I if I try to type that's actually kind of good Well, if if I try and type something that I'm not saying that I end up typing what I am what you do say Yeah, no, it's it's fine to automatize your commands because some people Can't read it or can't you know, whatever So here's the object that we just created. So now we need to tell kubernetes We need to tell open shift to use this template when we create new projects And we do that by defining it in the api server So This command here so oc gets config map named config and the api server namespace And then we're just using json to Select one one portion of that And it spits out a bunch of different things that are inside of here right that it's going to use that It's going to do with that particular api request So effectively we want to add another option to this which says use this one right when creating that new template So we will create our object And I'm going to Switch back to our yaml editor to show this one because I'm going to pipe it in line So all I'm doing here is creating a new projects So you see no namespace associated with it. So you requested a new projects And then the template to use here is project request Cool So pretty straightforward. So I want to do oc apply dash f And you know, I'll paste that in there and do my eof. So if you're not familiar with this, this is just the hear doc Yep, so hear doc cats to it sees the end of f string and then oc apply dash f with the standard input Now we should create our object inside of here Right and last but not least Now we can review our same command as before And what we should see here is Now we have this API server config project config definition. So here's our you requested a new project With our project request template So at this point what I should be able to do Is do an oc new dash project. So we'll call this the admin office hour a OH And we have a new project and I should be able to do it get Um, I don't remember if I needed to do something else to trigger that Now I'm questioning myself because I didn't see the hey, you created a new app blah, blah, blah using Yeah, yeah that why didn't that work didn't work. Okay five minute warning by the way Thank you Christian asked a really hard question in chat, but you y'all y'all can talk about it in the work channel Um But yeah Kubernetes at its core is a resource scheduler and it's built from the ground up to effectively and efficiently schedule workloads and resources Isn't that an argument for setting latency sensitive sensitivity to high and VMware? um, yes so, uh, I talked about this during the sizing uh, so at the end of the sizing stream of Over commitment is something that we especially as virtualization administrators. We love Yeah, we over commit the crap out of everything all the time and network administrators love it even more Right, it's not unusual to find especially in a high arc old, you know network architecture to see over commitment ratios of like 20 and 30 to 1 on those edge switches so Over commitment is fine But you want to do it as close to the application as possible So that means in the case of open shift in the case of kubernetes at the kubernetes layer Right the reason for that is because let's say that i'm over committing at the hypervisor level so the mware rev open stack whatever is doing the over over commitment And the hypervisor is out of a resource, right? It's it's out of memory. It's swapping A hypervisor is hurting. It's impacting the virtual machines open shift in this case So open shift doesn't know this All it knows is every alarm that it's got is saying performance is bad Something's wrong with the application. I need to autoscale And it's trying to throw more pods at the problem. It's trying to scale up more nodes ending problem And all it's doing is exasperating that out of resource condition at the hypervisor layer So you want to basically protect the resource utilization or resource availability rather at the hypervisor layer for kubernetes So this is a bit counterintuitive for us virtualization administrators because you know normally when you know Chris the open shift administrator comes and says, hey, I need, you know, 10 vms with, you know, 32 cpus and a terabyte of ram And and I laugh at him and say no So the reason why this is okay is because we have things like machine sets And the ability to dynamically scale the cluster So everybody in the stack from the application team to the open shift team to the infrastructure hypervisor team has to trust all of the tools in the stack right of Hey, when I start to run out of capacity at the Application layer, it's going to autoscale up and when the open shift layer starts to run out of capacity is going to autoscale up And we basically just have to trust that each one of those layers is going to give us the resources that we need and not give us a bunch of crap or Deny those resource requests and then impact everybody above so It's definitely a learning experience right for anybody who remembers You know, I was at net app before I used to give a whole talk five years ago on this exact same topic And it's it's still a thing. Still a thing. Yeah So, um, I know we're we're basically out of time So christian mentioned you the the api server is restarting. So if you try your Get the limit range thing again Yeah, do you see Not using a project. Yeah, I'll figure out what's going on there. I'll include that in the show notes and okay, cool If there was something I missed So thank you to eric jacob's by the way who he created a lot of these commands a while ago So yeah, he helped me greatly with how all this works. Although he probably doesn't know it because I've plagiarized So I'll follow up with that make sure that we get that covered inside of those show notes I'll also follow up with it next week just to make sure that we for anybody who doesn't see the blog post Awesome that that's included but Sounds good. Um, so yeah, thank you. Thank you everybody who has been watching. We really appreciate your time I know it's uh, sometimes not easy to devote an hour with us But it does mean a lot If you have any questions anything that we didn't address today And I'll go back and I'll review all of the chat to make sure that we catch all of those In one form or another If you didn't have time or if you didn't feel like it fit or just didn't want to ask in public Please feel free to reach out to me. Um, so I'm on Social media. You can reach me. Um twitter practical andrew. Um, I'm on linkedin and all of that other nonsense as well Uh, you can also send me an email quite simply andrew.sullivan at redhat.com Happy to take those questions. Um, love to get those questions Um, and if you don't want me to talk about anything publicly, right? I don't usually like to mention names or anything unless you're okay with it So, um, happy to also do that. But yeah, don't hesitate to reach out at any time Um, and with that, I think we're basically at the top of the hour chris Yep, time to go I will uh, catch you all later. Thank you for joining and stay tuned for a open shift commons briefing Coming up in just a second. Thanks everybody