 Good morning, good afternoon, good evening, wherever you're hailing from, welcome to another edition of Red Hat Enterprise Linux Presents. I am Chris Short, executive producer of OpenShift TV, and I am joined by the one and only Scott McBrien. Scott, how are you doing today, sir? Doing great. How's it going with you, Chris? Well, it's snowed overnight here, so I'm still waiting for spring. It's April, aren't we? April 20th, 21st. Yeah, you know, we got, like, snow, it was enough, like, further south, closer to the, like, Huron and everything, and the Ohio border, they got, like, two, three to five inches of snow, where we got one to two. And it's like, wow, come on, like, give us a break. See, here I was in the south being like, wow, it's windy today, that's unusual. Yeah, no, it's like freezing cold outside, it's 37 outside Fahrenheit, and that's just two degrees Celsius if you're watching from another country right now. And yeah, that's just cold and windy and, like, just not fun. So, yeah. That's completely about four. I know, it's, like, completely unnecessary. I don't see why we have to go through this. We should fix this. You and me, right now, let's do it on our channel. And the technology. Yeah. Perhaps there's a tuning tunable for it. Maybe, which brings us to our topic today, performance part two, tunables and tune D. Fun stuff. Yeah, so I love to do that just a couple weeks ago, we were talking about, like, all kinds of stuff in proc and. Yeah. So, although fun little directories. We thought maybe we'd revisit that and talk about how that kind of ties together with tune D, which is a daemon for managing that. So, we, how much do you know about tune D Chris, besides loving it. I mean, so, I know it has a series of profiles, and I know those profiles are very well constructed for their use cases. But sometimes they're named confusingly. Right. Like if I'm optimizing for network throughput, what does that do to IO, right, like there's some missing explanation that you kind of have to always figure out what is this workload kind of like before you set up tune D to do something. You can have tune D report, I understand, right, like just kind of get some data and try and figure itself out, I feel like, is there a function for that. Well, we do a recommendation. Yeah, that's right. A couple times and like, choose or recommend which profile thinks works the best for your workload. But yeah, you're right in effect. Like so we initially included tune D with row five, so it's been around. But as we've kind of built up the profiles, one of the things that got added to it was the ability to base profiles on other profiles. So like a lot of our profiles now are based on throughput performance profile as their base and they manipulate from there. And so there's one called network latency that includes throughput performance. But so you end up with kind of like this stacked pile of things out of your system and being able to tangle out a little bit might be interesting. Yeah, and so there's there's actually a like I'm looking at my fedora 33 host right now that I'm using in power save mode on a server because it's just me using it I don't need the highest of bandwidth and capacity and everything else going on. It runs open shift just fine and when I log into it it does my workloads just fine too. So, you know, I set up a power save profile, but somebody else is going to be like, No, no, no, no, don't do that. You need, you know, something something something, you know, whatever for, you know, causes a virtual host for example, or, you know, because it is running an open shift and used to be more latency performance or something to that effect. So there's a lot of different profiles, a lot of different ways to apply it. Yeah. Well, and I'm guessing you chose power save because you my electric one month. Yeah. My wife was like, Why is the electric bill so high? Well, I got that server. I mean, trust me, I tested them all just as I was interested about fan noise too, because at the time my office was still in the basement and if I could hear the server from across the basement. I knew something was wrong. Why is it running so hard? Yeah. So while I share the screen here, I've just got a VM that I've pulled up and I'll probably have to refresh it about halfway through. Just because these boxes are. Do you increase font size on that by any chance? Oh yeah. There we go. More better? Yeah. Okay. So, the first thing. The VM is the administrative utility for 2D. And so we can do things like list. And these are all the 2D profiles that are currently installed on this box. And that's not all the ones that we have available. No, there's more. Yes, there's more. That's the default stack. Right. And so for example, I was just mucking around with one for Microsoft SQL server, which comes from a different package. So, yeah. Let's do a list. Because it's something like 2D and SQL profile or 2D profile and SQL. I can do 2D profiles and SQL. All right. So this RPM installs yet another 2D profile. That is for Microsoft SQL server. And so now when I do the 2DM list, there's that new one that I got. And all this one is one to show that you can sometimes get 2D profiles from other places. But also we're going to take a look at this one. And I know that for row 8.4, there's actually an update to this one. So I thought it might be interesting to kind of like take a look at what it is now, what it's going to be when 8.4 releases and talk about why those changes were put into place. So, if we take a look at the RPM payload. So essentially we got a directory, one configuration file, and a main page for it. Okay. So, yeah, let's send there. Let's go see. And folks just to let you know, right, like there's, there's profiles for Postgres, Oracle, SAP HANA. Trying to see if there's any other database ones I see here. Obviously MS SQL, but yeah, like the, you can get more from other places, right? Like if your database vendor has made a 2D profile, you can install that too. And that's something I think, like ISVs don't completely buy onto is that you can actually package not just like your ISV software for Red Hat, but you can bundle in these kind of REL or Red Hat distro specific things to help optimize the performance of your app on it. By the way, so, so this is the 2D configuration, you can see that this one in the main has an included of throughput performance, which a lot of our, a lot of our 2Ds will start with throughput performance and then make adjustments off that. And essentially how it works is whatever some last sticks. So if throughput performance has a setting that conflicts with one here in the MS SQL 2D, because the MS SQL 2D was the one that was executed last pulled in all the two through performance, and then on top of that, put on the things from the MS SQL 2D. So whatever happened in the MS SQL 2D that that's what actually is persistent on the system. A lot of these like down here, excuse me, down here under the system control section. These are those prox system tables that we talked about a couple weeks ago, or actually maybe a month ago now, which are not persistent. So the other part of 2D is there's a daemon that's run by run by system D that will start at boot, and then looks at the configured profile and applies it at boot time. So that, even though we may have made these changes live and those changes are put into proxys, which is not persistent, we're at least making those same changes every time the system is booted. So, you mentioned that there were a whole bunch, and we saw that. Yeah, there's the list when we did the 2D EDM list, and those are all just stored in the user to the 2D slash there. Directory, right, so this is where you put stuff when you make your own 2Ds. And you'll see that this corresponds with the list that that we got from the 2D EDM command. So where do you want to start for a short. I don't know. Let's start. What is functions? I mean, what is that what is in that file? Question, I don't know. Good to know. Okay, so you can put script content in your 2D, and this apparently is like already written script content that you can use. So that's what what this says. Okay, cool. Like, well, what is balanced? I mean, what does balance look like? How about that? All right, well, let's see. So balance is just kind of like a generic 2D profile. And so you can see things like the governor, the CPU governor is set to conservative or power save. So we're not cranking the CPU all the way up to full power mode. That's pretty much it. Yeah. So I think for most folks, they're interested in ones they're probably based on through performance. But before we get to that, because that's going to cover a whole bunch of layer goals. I also think these two are interesting virtual host and virtual guest. Oh, yeah, absolutely. Let's look at host. Okay, so, all right, so host is included or includes the through performance. And then it updates the dirty page cash ratio. So when we have 5% of the system is dirty page cash, it kicks off a kernel threat to start writing that out to disk. So here the kernel schedule migration cost. So if you move a process from one CPU to another, you lose all the cash that was on the CPU that you're writing on before. And for things like virtual machine workloads. That's a big deal. Yeah, like losing all that cash and having to restart another CPU is difficult. They schedule this or create this cost that basically says hey, here's how expensive it is to move to another, another CPU so we try to keep things on the CPU that they're originally running on. Nice. And then this yeah force latency. Basically says something that's sleep state power basically we shouldn't do sleep state or power safe settings. Right, exactly. Because we're a hypervisor. Right. Okay. Firstly, virtual guest, which is what you'd by default run on a virtual machine. In fact, before I do this. Is your VM running. I bet it is. As it detected itself as a host or guest I mean. Yep, there you go. Remember, we, we like make a recommendation. It turns out that that recommendation is applied by default on boxes where you haven't done anything, at least on across seven and eight. So this virtual machine got the virtual guest profile applied to it. All right, so again, including through performance so we get whatever is in here. And then we're setting it up so that we keep more dirty pages and page cash than the default. Right. So we set swapping this to 30 swapping this is valued between zero and 100. Yeah, that's such a affinity for using swap space. I think the default on the system is is 40 or 50. I think it's 50, unless it's changing really. Yeah, got something right like yeah. So we're making it less likely that the virtual machine is going to swap. Right. Because swapping the disk in a VM, you don't know what this you're swapping to it could be very low performance, it could be very high performance, you don't know. Right. The system doesn't know unless it tries. And in reality, like a lot of places, let's say you were a cloud provider instance, right. This is still have swap space. Nope. So you could set that to one. And probably be be fine. Although if you don't have swap space, there's really nothing for it to swap anyway so right. I actually would probably add something to here. Just give my brothers. You smart. Yes, I probably do that. Yes, I always forget that that I used to have to do that on the freaking command line so much. Right. So many times so many years ago, elevator no. Right. This is especially interesting for virtual machine. Because you're doing all your disk IO, and you're sending it through the disk elevator for this guy, which the default for relay I believe is deadline. Let's see. Right cash. Cash type. Oops. There's something in queue. Yeah. There it is. Just do a fine for elevator. It's in here. This takes me to read ahead. No, read ahead is a setting. Okay, so this is not what I wanted. These are tunables. Scheduler. Oh, there it is. Scheduler. Good. Good either. Thank you very much. So this one. Yeah. So this is a deadline. Oh, actually, it looks like I need to change my tunable because there's no longer schedule called no up. It is called none. Right. With rel eight. And so the tunables that are in this, this queue directory are ones that are germane for the iOS scheduler. Well, actually the ones that are down here and I was scared are the ones that are germane for that scheduler. They're germane for the disk device. So things like read ahead. So after you've retrieved a piece of data from the desk, how much additional subsequent data should you read? Because if you're dealing with file IO, you grow and grab a block of a file. What's the likelihood that somewhere close in the IO queue is more blocks of that same file. So things like set the read ahead to slurp in more data, expecting that we'll be asked for that data shortly. We go back to here. Whoops. So what is the throughput one. All right, so let me finish this now. Oh, yeah, sorry. Finish your, your tuning inch. Yes. But I want to ask you that. So why would I set the elevator to none on a virtual machine? Oh, I forget. Oh my God. You had to ask me that. There is a reason and it is a reason it is a huge performance hit. I know. It depends. So for like for databases for sure it is right like, and that's this is where I'm used to setting it right if that makes sense. Yeah, my SQL servers to be specific. So you tell me. All right. So let's say that we kept the default of read ahead. Right. So we have these that the disk IO queue and I request are coming in there being put in the queue and then we're spending our time in the kernel going, oh wait, this one is close to that one. Oh, this one's about to expire. And it's doing all this reordering on the queue. But then where does that go from the virtual machine to the hypervisor. Right. It turns out the hypervisor also has a scheduler. Yeah, it's doing the same exact stuff but now across all the machines and aggregate. And so you spent all this time on the virtual machine making this order as best as you could. But when it goes to the hypervisor just gets reordered anyway. So why waste your time doing the disk IO ordering on the virtual machine which is going to be reordered as soon as it gets the hypervisor. So that's why we we set it to none and it may not be the best for every workload. It also depends on like what all the virtual machines are doing and what the hypervisor is doing there's there's a lot of things in play there. So whenever you make changes, you always want to kind of not just go. Yeah, looks good. It's like no, we're going to put the change in and then we're going to try it out. Yeah, we're going to measure some stuff. Yeah, it turns out the change doesn't do anything and we can take it out. If it does something good. We'll keep it in. If it does something and bad. We'll take it out. So you were asking about three performance. Yeah, I'm curious why that is underneath guest. Yeah, it's underneath most of our stuff honestly. Okay. Rapscallion Reeves asks, would that also affect any network shares or VF IO? Elevator maybe VF. Well, so maybe it's whatever network you have in block. Right. So if your network shares don't show up as block devices and they slash this slash block directory know things like I SCSI devices would show up as disks. So it could potentially affect those. But there's also a regular expression that you can put in that disk subsection of your tune D that identifies which disks it should apply to. So if you are in a state where you have I SCSI devices and you don't want this to apply to I SCSI devices, you can use this regular expression syntax to make changes so that avoids those I SCSI disks. In short, in our episode guide, I put in a couple of links that might be interesting. Oh crap, you have an episode guide. I know I put it together like literally 30 minutes before the show. Oh, did you send it to me and Probably not an explicitly. So there are two links to the same knowledge or to the same red hat product documentation. It's called something like the rel system monitoring and performance. I'm talking about but yeah. And so one is just that guide. And the other one is to specifically chapter three of that guide. Chapter three talks about making changes to tuning profiles and in there is like, okay, here's how you can make changes to disk. Here's how you can. Here's past the elevator. Here's how you can pass the regular expression. You can also pass other tunables like we could actually sit there read ahead in that disk section that would apply to, in my case, all disks excited limited it. But if you're regex limiting it, you can tell it what what read head you like to apply to that set of disks that are matching your regex expression. So hopefully I answered rapskians question. I think so he says projects for the wind. So it would affect ice guzzly not necessarily NFS. Yes, NFS devices typically don't show up as block disk block devices. So it naturally wouldn't, wouldn't affect those because they don't show up as that device type insist. You did. You did not share a doc with me, by the way. Oh, I am. I am a complete failure in so many things. Just sharing right now. So, let me see how all these other people have it not me. Why don't I get access to the special fancy director you have as reasons. We work in different teams. I know, I know. Okay, but I still want me for short. Thank you. I appreciate it. Okay. Cool. So you should have that now. I do. Thank you. Okay, awesome. And so we're talking about through performance. So we see a couple things we're starting to introduce the idea of architecture specific tunables. And so up here. This is for Marvell harm processors. Yeah. But we'll get more there. Here's an AMD one. And then there's actually settings later. In fact, you see right here the VM thunderex, then has a rejects that says only do this on arm, but only if you match that thunder CPU rejects so it can get fairly complicated anyway. So a couple of things in the performance one is we set the CPU governor performance meaning crank that up to 11 slurp up all the power, make it minimum performance percentage. One. Rock that thing. Yeah. Right. So, this read ahead greater or equal to. Yeah, Jesus. Okay. Wow, there's a lot going on here. So we set the read ahead. So when we say virtual guest and virtual guest includes through performance cranks up the CPU governor although on a VM that doesn't matter. But it does set the read ahead to 4096 or greater on disks attached to the VM. And then down under this control so we set the schedule granularity so every one, one million new sets or something says nanoseconds. It'll check to see whether we need to change what the schedule job is on the CPU. Right. And then down here we will check this many nanoseconds whether we should make that decision or not. And then we set the dirty page ratio higher than normal. And you may remember that in virtual guest toggle that back down to 30. And then when it's 10% of memory is dirty page cash, we'll start up a job in the background to start synchronizing out those dirty pages. And we set the swappiness to 10 lower. Nice. Yeah. So in virtual guest we then turn that back up and make it more swappy more likely to use swap space. I appreciate the documentation that's in here right like the comments are actually handy. That that is somewhat new ish. Okay. So that did not exist on real six. Right may have existed in some of the later later versions of well seven, and it exists in real eight, we're looking at an eight three box here. Yeah, so we run through performance and then on top of that we private virtual guest and then whatever it's done last six, that's our 2D profile. So what's the what's the like inheritance hierarchy there is it if it caught like obviously there's system like CTL settings you can set right and then I would assume those would override the 2D profile but if the 2D profile has another file underneath like where does do you know an excellent question it's always whatever is done last last. 2D starts up. And if you include a profile in your 2D the included profiles executed first. Okay. And then your profiles executed on top of that. So in the case of virtual guest performance is executed first. And then virtual guest. And so that's why you're able to take like the swappiness from performance and override it with a greater amount of space or affinity for space for virtual guest. Now we start talking about managing things through sys control that like there's a whole another. Yeah, angle into it. That's a wrench that I wanted to throw at you. Right. So I think that to make things simple. I like that. 2D works with both the sys control to the bulls and the slash sys to the bulls. Whereas sys control only works with the proc stuff. Got it. So I would say that 2D is the better way of managing performance settings because you're able to manage across multiple areas of performance settings. Yeah, exactly. And then you obsolete the need to know about which one happens in which order. But essentially, if sys control is executed in the old days that was done through an RC script. Yeah, yeah. And actually I don't even know how it's done now. I haven't even looked on relate. But so wherever that RC script fell for sys control compared with the 10D service starting that would determine which one was done first, or last, and then would determine how messed up your intentions were. I believe it is a unit file and system D somewhere so who knows. I could. Oh, that's an Etsy system D, I think. Or is it just system. I forget. Help me out audience. Let me just go here. Someone said something, but it's rapscaling with this fancy Google. It's fancy Googles. Oh, kidding. Is tuned the rail specific or will it work for other distros. So it's open source. Yeah, I mean it's our base right now could work on other distros I know the red hat distros all have it so fedora has it, real has it sent us stream and Linux have it. Downstream distros of those would also have it so Oracle Linux and that the newer downstream distros are starting up like home. They'll inherit it. All right, so. Okay, here we go. This I think is that the sauce right here. So the tune D service is started after system DCTL service. So CCTL will start up first, set all the parameters, then tune you will start and overwrite those parameters with whatever is in your tune D if there's conflicts. So whatever is done in tune D is what sticks if there's a conflict. So there you go. All my DBN based distros are like man, Raspberry Pi is nanos and such so really find any record of system D available or not system D tune D yeah. Yeah. Not surprising. But but these are like, you know, tiny devices that really get touched so. Well in the proxies and slash says that that's current stuff. Right so other distro uses the same Linux kernel so potentially they could use to me as well if they wanted to put in their district. All right, so I mentioned that. Oh, that we have that MS SQL one. Yes, I'm curious. So here's what's in the MS SQL one today. So it includes through performance. Basically it looks at if you're using huge page memory but through the running of things, you're not using a consistent memory space in the huge page it'll actually copy it to a different huge page to congregate it better. And then down here we're setting the memory map, like how big the virtual memory map should be. We do not use new balancing that's going to keep us from moving things between CPUs as much. And then down here we're doing some kernel scheduling stuff. Right so we're turning the scheduled latency the granularity of how often we need to wake up and check things and how often we need to make changes to the kernel scheduler. All right, so let me edit this. And I'm going to copy and paste stuff. And you'll see this there's several Linux distros that use tune D good for habit available. DBN being one of them so that means all the other underlying DBN based ones are probably going to have it available as well. Which is cool. All right so this is the new stuff that's going to be in there. So we start off at the same starting point we're going to start through performance. We're making a slight CPU change for whatever force latency is. That's probably a slash so I would imagine. Yeah, but you could see it. Whoops. Well it's going too far. There's a whole bunch more system control stuff happening in the updated profile that will drop with 8.4. So this is currently in the 8.4 beta. So we're changing the swapiness. So before we're just taking that what swapiness is 40 that was in through performance and now we're like oh wait no we should we should turn that down for databases because that's absolutely for databases. You don't want your database to swap. Trust me. We're also adding in a whole bunch of page cache information like how often we should write it which in the earlier tune D profile didn't exist. So we talked about this a little bit a while ago about how these the set of background ratio dirty ratio expire and write back sent to sex that this is essentially an effort to keep as much unwritten file rights in memory longer so that we're not doing a whole bunch of disk IO on this machine. Exactly. Because that will impact performance a lot. You're right. So the other thing is we set transparent huge pages to always I don't remember what the default is. We could check on this VM but now we are going to always use transparent huge pages in rel seven and rel six a lot of times we defaulted to not using transparent huge pages and that's starting to break loose in rel eight where we're just better allocating them and not delaying memory IO or memory requests because behind the scenes we're trying to allocate this huge page to hand back so apparently they've done some testing it between Microsoft and Red Hat and figured out that they can get away with this and actually purse performance we're increasing the memory map size so now for every process we have more available map space. This was also not in the previous 2D so here what we're doing is we're changing the read and write memory settings for network IO so the default is how much a memory buffer should be in I believe bytes of RAM and then the max is like how big you can make it and write mem is the same thing but for packets going outbound on right so it turns out that it's often the case that databases are running a standalone system and they serve as a network connected resource for things like web applications so the web app then makes a connection to the database of the network to try and retrieve that data at once so we're buffing the network buffer sizes to be able to shove more data or read more data than when we receive those connections alright and then pretty much everything down here was already in the old 2D profile I think they may have changed the migration cost let's see yeah there wasn't a migration cost setting in the old 2D so they made it they added it and it looks like the migration cost is actually lower than it was previously through throughput performance so we're saying it's not that expensive if we need to take a database thread and move it to another another CPU my guess would be that it's because the database requests that come in are not contiguous parts of the table space so there's not a lot of expense if we switch it to another CPU because we're not losing the cache there's not a lot of cache hits that happen from it right it's interesting how we use throughput performance so much it's like it's almost like why isn't that the default as opposed to balanced well balanced balanced isn't the default for a lot of stuff balance is the default I think like workstation installs for me but the old fedora profile says about balanced yeah I'll check my rel 8 physical machine real quick general non-specialized 2D profile that's helpful thanks I'll go back to my terminal here okay so my nothing special rel 8.3 laptop is running balanced by default mm-hmm and if you look at the balance profile it has stuff on like audio settings and so I think it's more for like a desktopy workstation environment yeah exactly okay so that's kind of how we build up an understanding of 2D the other one that might be interesting is network latency that one we use a lot as well at least on the server side yeah network latency is a big deal right especially if like if you do have classes of servers that do specific jobs that then hand off tasks to other things big deal yeah and if you look at what it is doing here it's actually it's called network latency but it's really network low latency mm-hmm so it includes latency performance which is low latency performance profile mm-hmm we turn off huge pages so if something asks for a huge amount of memory instead of trying to allocate a huge page and pass that off into its memory now space we just give it individual memory addresses because there's time that's taken for those to resolve those huge page allocations mm-hmm and then down here it's looking at like how often we should check the network stack for its status mm-hmm and then they also added the TCP fast open so mm-hmm I remember looking at this at some point and something like we shortcut some of the TCP connection stuff like the validation stuff to just like have something going or maybe we keep a pool of connections or something weird it was something weird that basically made receiving or sending TCP packets slightly faster so and last time we talked about how you could go under the kernel documentation and find out what the different tunables in proxess were and the description of what goes in them and maybe that would be a good exercise for that fast open to just refresh them around where it is mm-hmm your box yes do me a favor what recommend dot D recommend director in there now they're going to that directly pull D profile because I just looked at it and it was very interesting to me so obviously scroll down but yeah the it's like it picks up the system purpose somehow or the chassis type and then defines you know what the recommendation is so like that balance thing would tell you on a desktop or workstation that's the recommended thing but actually here is what caused me to be balanced right exactly if you were not like you know on a if you're on a server for example it would say hey am I a host or a guest and then tune specifically to that that I find interesting this one specifically is for yeah which is like no longer a thing for red hat um but if it sees that as a vert it'll tune it to virtual guest right which I think is cool yeah down here yeah if you just say like if you're actually on the console it'll say like this is recommended yeah in fact I think 2DADM with if you list it I don't know if it says it recommend there you go virtual guest there you have it alright that tells you what the recommendation would be but doesn't actually set it right it's not until you do a 2DADM profile and then assign the profile so I can do something like that exactly so there's logic in there to tell you right like but you can also change this for your environment and have something like Ansible come around and plop it down on the file system and say you know you have a specific kind of hardware that needs a specific kind of profile you know network latency versus you know latency performance well those two can hear each other you get what I'm saying right like so I'd go even a step further and say remember like 2 or 3 months ago we talked about system rules yeah so one of the system rules is called kernel settings and kernel settings lets you pass parameters and what it's really doing behind the scenes is looking at the running 2D profile and taking those settings that you put in your Ansible playbook and sticking them in the currently executing 2D profile so that when you restart the machine and you get the last 2D profile you're running your changes are still there from when you use the kernel setting system rule that makes sense nice yeah so you don't even have to write your own Ansible playbook to do all this anymore you can use the kernel setting system rule and it actually uses 2D for the storage method for keeping those changes across your population find that video before we get off here probably not it was just system rules right that was actually in let me just search for roles I have to spell roles right first though yeah I found it here it is it's got 10 likes got good for you 10 likes I'm a sucker for likes Chris well you know wherever you're watching please click like and subscribe if you can but yeah there's the system rules episode for those that are curious awesome so I don't know where else to go from here Chris you know I don't either I'm curious what the audience thinks right like how often are you changing syscut all settings would it make sense to make a system rule out of that right like or would it make sense to make a profile or where is that line between system role and just another 2D profile I guess might be a good question yeah so I think that there's some thought to be had there so I like the idea of using system rules because I can make changes to my playbook and then execute it across my population right so if it turns out that you know I'm reading a tuning guide and it says oh we should go from a swappiness of 1 a swappiness of 5 I can go to my playbook and make that change and then execute the playbook and the system will then execute it across the population if you have a diverse population it might be that you only want to set the swappiness on databases right now web servers so you can use the other ansible stuff through inventory to make those determinations right and then you can have some conditionals in your playbook to decide this set goes with databases and this set goes with web servers and then when you make your updates you apply them to the correct subset of your population with the system so if given my druthers that's probably how I would architect it so okay we might go on a spelunking mission here Rapscaling Reads asks are there any specifics to tuning a container host as opposed to a VM host now that is where it's like let's go look at our costs see what it's doing so I have a feeling the answer is not really probably not yeah because unlike the virtual guests where they have their own iowa subsystem and a container host you're sharing the same kernel as the host operating system so number one you don't get to change any of those things because in the container you're not doing any of those things and so I would look at something like maybe how you would tune for whatever workload your containers are generating a little box so if your containers are doing yeah go ahead no your containers are doing go ahead if your containers are doing a bunch of network protocol stuff right it's all web services maybe you would on the host make some changes to the TCP stack or the overall network stack to make it larger amounts of data that can push out I think there what you're probably more interested in on the container host is less tuning and more C groups exactly yeah so control groups you could apply that to processes and you can limit the amount of system resources that individual processes or groups of processes can consume and that way you can make sure that no container um takes more than it's fair share of stuff on the box if you have a bunch of things happening wraps gallery is the place to go look to find this is actually in the OpenShift container platform release notes so in 4.6 we did a partial tune D real-time profile um and then now in 4.7 it looks like we've got some logic if there's an invalid tune D profile that gets created the OpenShift tune D supervisor process may ignore profile updates and fail to apply the updated profile this bug fix keeps state information about tune D application success or failure so now OpenShift tune D which is a process uh recovers from the profile application failures on receiving new valid profiles so apparently there is an issue with uh tune D between versions and we fix it and there's a busy length in the 4.7 notes here so yeah there's there's definitely tuning happening in the OpenShift world for sure which means that's happening on the R-cost notes themselves which are part of OpenShift and I'm feeling that like the um msql profile it's like keeping um keeping dirty pages in memory so you're not burning a bunch of cycles writing stuff out the disk um and then also making sure that you don't move schedulers because if or sorry not schedulers move CPU uh because if your container was one CPU and had all the cached information there moving to another CPU would be um not great uh for it because you'll then have to spin up a whole bunch of more cached information I'm such a dunce um I was trying to do an OCD bug to figure out where it is in the file system but give me a second uh but I don't want to leave you hanging either Scott so oh it's cool um one other thing you might take a look at is the um at least for rel is bcc tools so bcc tools uses the ebpf in kernel virtual machine to slurp out real-time data from the running kernel um and we have things like cache hit and miss reporting data um so you can kind of see what's happening at the app layer while your applications are running that might be useful especially in a place where you have a multi-tenancy for um different container applications running my ocfoo is failing me today but yes um you're absolutely correct let's see no anyways if you do an OCD bug on a node you'll be you'll land on the file system of that node and then you should be able to then go find that 2nd profile for whatever it may be so if you're super curious go grab one of your open shift nodes and debug it real quick and you'll find it it'll be on a similar occasion like user lib 2nd um I just can't get into these nodes right now oh because they're not so fun state oh my node's upgrading duh that's why it's not working that would do it that'll do it uh short I did throw in one more link in our document so this is one of the um lab.redhat.com labs it it masquerades as a sql server lab in fact its name is sql server cstore um and it does two things one is it shows you that when you're using microsoft sql server or really a lot of different databases using column stores will save you effort when you're doing queries but the other thing that it does and this is the more like real interesting thing is it starts off by running um a cpu dist bcc tool against the msql query process fancy it's it's actually like if you look at the command that my co-worker don wrote for it it's like wow that's a big thing um but the histogram that it produces shows you like where in the um context switching like how long jobs are running on the cpu right and all of them are kind of right at this one plateau um or a lot of them are at this one segment of time and then the lab has you apply the msql tundy and one of the things that msql tundy does is change the scheduling granularity and then you rerun the same query with cpu dist and what you'll notice is that all of a sudden instead of everything being in a line right at this one time you get this much nicer um gradation of context switching because we're checking more frequently and when a query results quickly we can go ahead and put another query on cpu where if a query takes a little bit longer it might be like two or three checks and then we can switch it um and so that that's a real life example of where doing things like scheduled granularity can actually have an effect and you can visually see that with that bcc tool histogram nice very cool all right well i mean we're 1457 man we we still talked an hour we did it yes yay mission accomplished uh hang up a banner uh the yeah might yeah cluster failed to upgrade all right that'll be fun to debug later as if we didn't have enough to do crucial problems they're they're tough to have exactly so we also um let them know about our slight change in format so having we were doing a guest every couple of weeks for every episode and um i think that we're going to go to a format where we're doing a guest every other episode so next episode i'll have to find a guest to talk to us which usually is not a problem um but then after that what did we say we're going to talk about file system stuff yeah i think we're going to go through the you know this the acronyms failing me the standard linux file system stuff but file system hierarchy standard there you go that was it um and then i think that might parlay into just like a brief like introduction to s e linux of yeah things where we expect them and everything just kind of works and why is that right yeah we do things the way we did it yeah so what we're going to do or we're going to try for the next uh while is we'll have a guest one episode and then we'll do something like this more freeform uh digging down in the os for the next episode then we'll go back to a guest and then we'll do freeform excavation in the os yeah eventually we need to get like a fleet of os's to go mess with so i can make that have the os's well i mean like give me like five rail boxes what can i do now you know right like build you know maybe lay out an app right see what happens all the things we've learned and apply it that could be fun could be fun i've got a set of three that i can spin up for 30 minutes at a time but i got i got whatever i want my basement um but anyways um great show like learned a lot the inheritance all the last applied stuff somebody wants an ansible workshop i mean we could do like an intro to ansible on rel kind of thing if you want yeah i can i can ask uh show them to come over and be our guest yeah i can ask too i used to work with them you know but yeah the like the ansible channel does do stuff on our regular races too so if you haven't seen the ansible youtube go find it it's uh they're doing some stuff new stuff with it lately so yeah um the wraps getting reused if you want some ansible content we can make that happen for sure um so yeah that's it for the streaming on the air today tomorrow we start bright and early zero nine hundred uh am eastern with we're gonna be talking about storage but what happens with failed migrations what go what do you do when bump in the night does happen during your massive migration of workloads or storage in this case so yeah that'll be an interesting little adventure that eric nelson michelle palm on i will be on first thing in the morning tomorrow and it's a pretty full day so you know check out the calendar when you get a chance sign up for redhead summit it's very free uh i highly encourage you to go take advantage of that and yeah have a great day out there everybody and if i don't see you the rest of the week have a great week thanks everybody see y'all bye