 Welcome to this recording in progress. And welcome to the session. So yeah, as just introduced, it's a research control strategies with research control demo, and I am Taejun. And let's get to it. So let's start with what is research control, right? I mean, why do we even, you know, bother about it? So in Facebook, we have like this kind of edible group of people working on research control. It's people from corner applications and multiple production teams working together on this subject. And this is our mission statement, not official one, but this is what you work towards. It's world-converting full-OS resource isolation. There's a bit of word salad, so let's unpack it a little bit. So the first part, world-converting. World-converting means that, so let's say research control is self-explanatory, right? Research control means that we wanna be able to distribute resources in a controlled manner, right? Whether that's CPU, memory, or IO, we're only talking about local resources here. So to do that, right, that's our goal, right? And to do, you know, while we are doing that, we wanna stay work-conserving. And what that means is that we don't wanna lose the total amount of work to achieve that. Meaning that if you don't wanna keep the machine idle, but less than otherwise utilized to achieve resource control. And this is important because, you know, obviously we wanna use our machines to do the full list, but also because one of the big reasons why we wanna be able to control resources, resource distribution in the system is we wanna put, you know, do things like stacking and other things, putting multiple workloads on the system to improve the utilization of the system so that we can use the machines more efficiently, right? To the fullest extent. And, you know, if we have to pay, you know, if we cannot conserve the total amount of work, the capacity that the machine can do to do that, you know, that's kinda counterproductive. So we want it to be work-conservative. We don't wanna lose work to achieve resource control. And the second part is full OS. There probably is a better way to describe this, but what this means is that we don't want applications to cater to like a special requirements to be resource controlled. We don't want applications to have to use direct IO or, you know, use memory in certain way. We don't want any of that. We don't want applications to do what they have always been doing. We want the operating system to be layering resource control on top or underneath, you know, depending on how you look at it, but in a transparent way, right? So, so users or applications can do, you know, whatever they've been doing or the same. And the OS is, you know, fully responsible for implementing resource control without disturbing how they use the system. So that's what we aim to do. And we have been working on it for, I don't know, like four years now, so it has been a while. And this is the first success that we had a couple of years ago now, maybe three now, my sense of time is really worked these days, but it's kind of clearly shows what it can achieve. So both the purple and green lines are request per second. And this is from a web server running in production in Facebook. So we call this services web. And what happens is that, you know, whenever you use anything Facebook, even through the app, you know, all the traffic goes through the web servers, and then, you know, then these servers talk to the background services to show you, you know, what you want to see on Facebook or Instagram, right? So RPS means that, you know, request per second. The details don't really, you know, you don't need to worry about it too much, but this is on production workload. And this is like both machines. These are two machines, purple and green, and they are loaded fully, right? So this is what we call like a production load testing. So what you are doing is that you are redirecting user production workloads or requests toward like a certain benchmark test set of machines to fully saturate them so that we can, you know, load test in a realistic way. So this is production workload doing saturated load testing. And so these two machines are serving 650 RPS, and that's kind of the maximum these machines can do. And what's happening in the middle here is that we are starting a memory leak, right? So imagine, right? I mean, so all these machines have like a web server, right, that's the main thing it's doing. And a bunch of management and monitoring services, running on the system, right? It can be a chef, which is kind of managing the machines. It can be some Chrome job. It can be some monitoring services. So there's a bunch of these things which are just there to, you know, to configure and monitor the machine. And sometimes they might function, right? And we have a lot of these things. So let's say, so what this is emulating is that what happens if we have a bug in one of those services which are not the actual, you know, the main workload, but one of the, you know, management things and there's a memory leak there 10 megabyte per second, not too fast, you know, that happens all the time. Maybe not all the time, but once in a while we see them and can we survive that, right? That's what it's testing. And if you look at the purple line, right? This, this without any kind of resource control. And so it's just a vanilla configuration. And if you look at that line in about, it holds on for several minutes, right? I mean, 34 minutes is fine because there's some buffer in the system. And then, you know, RPS just drops to the bottom and then like the machine eventually gets rebooted and comes back up in half an hour, right? That's really not acceptable, right? Usually it doesn't take half an hour. For the test, we disabled like certain types of emediations, but it would still be down for, you know, 10, 15 minutes. And imagine that, you know, this happening across multiple machines, right? If you have a bug, which is triggered, I don't know, like certain of Friday, you know, while the JSON is out. So there are those types of bugs which are kind of latent, which gets deployed widely in the fleet and can get triggered around the same time. Those are like really scary bugs. So if, you know, this would happen on chat, on control across many machines and many machines are doing what the purple line is doing, you know, the whole service would go down, right? The face will go down and users wouldn't be able to access them. So that's something we want to avoid, absolutely, right? So, quick question. Do you, in this scenario, what are the sizes of the machines, Tejum? So these are, this is from an order machine. So these are, sorry, two gigabyte machines with Zeta SSD. So, you know, the visual is from a couple of years ago. So now our machines are bigger and our IO devices are faster, but the trends is about the same. Like number of CPUs. Oh. Or does it even matter for this? It doesn't really matter. So here, it's a single socket. I don't know. I forgot the number of ports. Sorry, too, something like that. But here, the contended resource is memory and IO. So CPU count doesn't really matter that much. Oh, we wish, I wish, you know, get to that later. So we can see with the pressure metrics that which resources are being contended. Okay. Yeah, so yeah. And we are doing the same test with the green line, but we are repeating in three times. But you can see that while the green line gives a bit, maybe 10%, you know, but, you know, it's fine, right? You can do this multiple times on multiple machines on many machines. And while the latency might go up a little bit, the whole site will stay up just fine. So this is like the first scenario that we could implement. We could show that it actually works. Yeah. And if you think about like this test, this was basically the benchmark that we were using to test resource control in the fleet. It's fairly involved to set up from our end, right? I mean, we have to set up like this production benchmark set and takes like a couple of days and because it's a production workload that there's a lot of things going on at the same time. So it's just really painful to set up. And, you know, besides all that, you know, this is not useful outside. You cannot really do that outside Facebook, right? This is not really a generic benchmark. So what we worked on is something called a resource control demo. And what it aims to do is that capturing, basically capturing that setup in a candle way, right? So it simulates the entire setup in a single application in a way which can demonstrate and simulate and demonstrate everything. And at the same time, it aims to kind of show what's going on and you can easily look at what's going on. And it comes with a lot of documentation on why certain things are working in a certain way and what they can do and how they work. So it just contains a lot of documentation about how resource control works on Linux and how it can be used. And this presentation, there's just a lot of content in it. So we cannot cover a lot of, you know, in depth, but we will go through, go over like a couple of different scenarios, important ones, and then show how they work. And then, you know, when you guys have time, if you're interested, you know, you can, later there will be a link, you can, you know, download or you can set up like a AWS instance and then try it yourself and read all the documentations and try all the different scenarios to learn about it. And so if you think about, you know, what would be needed to do all this, right? If you wanna encapsulate, right? For like production tests, like web workload and all those things in a single application, you know, it needs multiple components, right? So these are the components. The first thing is called RD-HashD and this is what emulates the web server, right? And so it has, it follows like a normal distribution to access files and hit anonymous memory and you can adjust how the memory access pattern looks. And then it calculates the hashes reading data from there. It dirties some pages, it generates some logs, right? And it regulates its own load by target RPS and also by latency, right? Just like web server, right? If you have web server, you know, you have a load coming in. So you wanna match that load, but if your latency is rising too high, right? Then you're missing your service level agreements and then, you know, your load is shared, right? I mean, you offload to some other machines. So it encapsulates that main workload behavior or web server behavior. And there's things called sysloads. So memory leak that I mentioned previous is one of the sysloads. It just means that, you know, it's something secondary we are running on the system which may or may not miss behave, right? So these are like the components that we use to simulate, you know, failure scenarios. They're called sideloads. There are things called sideloads. These are just secondary workloads which are the same as sysloads, but which are run under something called sideloader. We will, I will get to that later, what side loads are. And there's a component called RD agent. And this is the thing which, you know, runs the whole thing. This configures the whole system, verifies that all the requirements are met. And when you tell it to, you know, run certain scenarios, it will run those scenarios and give you reports, you know, what's going on, you know, who's using how much, you know, who's being how fast, how slow, all those things. And the research control demo is the key why, the, you know, the terminal app which has a lot of documentation of what's going on and what is scenarios put into it. So you can run, you know, for example, like a protection scenario to see how things actually work and read about, you know, if you want, like if you read about or all about these details and how they're supposed to work. And there's a final piece called research control bench, this which is being still worked on and it uses the same framework to emulate the entire system. So what research control demo tries to do is, you know, rather than like emulating a synthetic, like a portion of it, it tries to, you know, exercise the entire operating system and iOS tagging the whole thing with realistic workloads, right? And research control bench is on top of that to build benchmarks, like a canned benchmarks to measure research control behaviors and like IO device performance using these workloads so that you can get more comprehensive view of how the system would actually behave in production other than you're having, you know, according to this benchmark, these tasks, you know, how many IO apps and whatnot, which, you know, doesn't necessarily translate to real-world performance. All right, so this is like one of the things that are the agent test and this is shown on research control demo interface. So research control is primarily uses C-group 2 which is a corner feature, you know, which you can build like a tree structure and configure how resources should be deployed or distributed but there are like a bunch of other requirements. And like, for example, right now, unified system which can be fully controlled, which can be fully implement, which can fully implement research control is ButterFest. It's not because, you know, ButterFest is internally better for this, but, you know, it's just something we deploy. So we fixed all the priority measures in the file system. With other files systems, there are scenarios where you would have higher priority obligations waiting for low priority one. So there are all these kind of corner features and various pieces which should be configured and used in certain ways to make research control actually work. So what research control agent does is that it checks all these requirements and it will tell you, you know, if something is missing and tell you also how to fix it. In the longer term, right, I mean, so many of these pieces are already upstream and or in the process of being upstream and also not just corner, but, you know, for distributions are also adopting many of these configurations by default as their defaults. So down the line, the expectation is that, you know, if you file off research control demo on any system, you know, you're gonna be meeting most of the requirements by default. So let's see how protection scenario, right? The first slide that I explained, right? There's a web server and there's a memory leak in the in the lower priority part of the system. And let's see how well we can protect that. And let's see how that's emulated or replicated in this control demo. Yes, that's just what I just said. And I'm just gonna show screenshots. I'm gonna do at the end, I'm gonna actually run the demo and show how it actually works. But, you know, just for time purposes, I'm gonna just run screenshots for now so that, you know, we can then just have some understanding when we actually do the live demo. So it's a little bit overwhelming because there's a lot going on the screen. So let me, so this part is just kind of general overview of, you know, how the system is, what the system is doing, right? You know, configuration, you know, all the requirements are met, nothing is missed. You know, there's no side road running and all those things, you know, this is workload at 100%. And on the right side, this part shows resource usage or resource metrics for different top level slices. Just meaning that, you know, workload means that where the main workload is, the RDHD or the web server stand in. Side load is where the side load will be. Most critical is where, you know, things like secure shell are, you know, that we don't wanna lose ever for debugging. And system is where, you know, all these memory leaks would be happening and so on and so forth. So it's just, you know, showing you, you know, if you break down the system into like these bigger categories, how much resources they're using and what kind of resource pressure they're experiencing. And this portion is showing right now the RPS and latency. So the green line is RPS and it's on the, using the left Y axis. So it's, you know, 1200 is 100% load. So you can see that it started 50 seconds ago the load went up, right? And it's stabilized at 100% load, 1200 RPS. So it's just showing you that the HD is servicing at maximum capacity. And the blue line is the latency, right, right axis. And it's, you know, hovering slightly above 100 milliseconds. So the maximum, the maximum latency target that latency tolerance that it has is 100 milliseconds. And this, you know, when the screen was captured. So while ramping up, it violated a little bit but it's going to come down and stabilize around 100 seconds. So this end to end, you know, from start to request to the completion of it. And these are just some logs. And on the right side is where the documentation is. And then you can navigate the right side with the arrow keys. And then you can activate buttons to try the scenario, right? So it has like a scenario and explanations mixed together. Anyway, so here what's happening is that we just opened this page and then which kind of started the main workload, HD, at full load. So about, you know, 50 seconds in the main workload is stable now, right? And if you look at here, it's using, this is on 32 gigabyte machine. So it's using most of the memory and most of the CPU and also using some of the IOs, right? R and WPS. So the system is stable now. And the next slide, okay. The next slide, right? I just kind of scroll down to this button and press enter. And what that does is that it starts the memory leak that we've been talking about. This one is more aggressive, I think. This is not 10 megabit per second. It's way higher than that. But it's a faster memory leak, but the same scenario basically. And after that, what happened is that you can see that, you know, the green line fell, blue line was at 100 and it's now at 400 to 500, right? Your site is down, right? It's not, nobody's happy. And then you can see, you know, here, you can see this is the memory hall, the memory leak, you know, how much memory it allocated and all that. But in general, it didn't work, right? And this is, if you read the button, right? What it says is that disable all the resource control features and start memory hall. So this is simulating the condition of the purple graph from the purple line in the first graph, right? So without any resource control, if we start a memory leak, what happens? It's showing you that. And I'll ask you a question here on the previous one. I just want to understand, what is CPUP and MMP here? What do they represent? These are called PSI resource pressure metrics. And what they, so if you go to like a secret directory, there's files like IO.pressure, memory.pressure. So these are reading those files. And the way that is defined is that it tells you how much time you're, or how much CPU time you're missing because you're missing out on a resource, right? So 66% memory pressure means that your, if you had more memory, right? Your workload could be using 66% more CPU time, run 66% faster. So it basically saying that, you know, you're waiting on memory, 66, 60% of the time, only, you know, two-third of the time and you can, you are able to run only one-third of the time because at, you know, during other times that you're waiting on memory. So that's what resource pressure metrics are telling me. And I will get to that in later slides too. Okay, thank you. Okay, so, and the resource control demo, if you press G, I think there's interface has, you know, it's kind of fairly dense interface, but if you look here, this is as G, more graphs. So this is just me pressing G to see more graphs. So this is the same graph, workload, RPS and latency, but here, you know, it shows the resource pressure metrics, right, that, yes, that the shower just asked. And here is where the problem is. So before, so like a study second mark. So that's where the memory leak started, right? Before that, well, this is kind of obscured by the labels, but you can see that the green line is at the top here. And this is CPU pressure. So what that's telling you is that the green line is for the main workload. So the main workload was saturating CPU and it had like internal competition. So that's where it was limited. I mean, that's where it was saturated, right? And that's, you know, that's what you expect to see. You're pushing the machine, please, it's maximum capacity, right? So the CPU is saturated. So you are seeing CPU competition inside your main workload. After 30 seconds, right? This drops, which is, I don't know, well, let's see why that happened, right? I mean, at the same time, memory and IO graphs rise, right? And the green is the workload one, the main one that the system is supposed to be doing. So what this is telling you is that the main workload was experienced, how to experiencing significant memory and IO contention. And that's why it stopped using, stopped being able to use the CPU. And that's why your IPS dropped, right? That's what it's telling you. And the way that memory and IO pressure rolls together indicates that memory pressure is caused by, you know, waiting on IO, right? So if something is pushing your memory too hard, making it cause a lot of page faults, and then you have to wait for IO's, and that's how you ended up being slow. And the thing kind of, you know, it's kind of sad is that if you look at the purple dots, they are at the bottom, right? And this is the offender, the memory link. And it's kind of sad that this is like completely unimportant stuff on the system. This is really important things, but we are, you know, what's suffering on the system is the important stuff, right? That's why lacking resource control means, right? I mean, the system doesn't know what to prioritize. And this is another part of that graph page, there are multiple graph page, and it's showing another aspect of what's going on. And this is a little bit more intuitive, right? So if you look at it, right? This is swap, swap usage, right? So the main workload and the system, the memory link, both are, you know, using more and more swap. If you look at the memory utilization, the system that slides, the memory link is increasing, and it's pushing down the workload, the main workload, right? So that's why it's causing a lot of IO's, right? Because a lot of page efforts, which then lead to IO's, because, you know, the system doesn't know which memory to protect, so it's just trying to be fair. So the system that slides takes more and more space, workload that slides loses more and more memory, and that's where you end up, not in a good place. And simply utilization drops accordingly. IO utilization is kind of weird, right? It has a big spike, and then doesn't seem to show much, but the next pane shows about IO, what's going on with IO's. And one thing which is important here is that, that the top level, the top these two graphs are showing usage, and the bottom two graphs are showing latency, latency percentiles, you know, median 90, 90 percentile, 99 percentile. So what's going on is that because of the memory link, we are swapping out a lot, so which is pushing, you know, these producing these IO's, right? And we are also pushing the memory of the main workload to the swap and page cache too, right? So that's pushing these writes and these links. What's happening is that all these activities are overloading the SSD, so SSD is getting more than they can handle comfortably. So their latencies are spiking, so write is really bad, but if you look at just the real latency, P90 is like reaching 60 milliseconds, meaning that one out of 10 weeks you send out to the device, it's gonna take 60 milliseconds, right? And if you remember the maximum latency limit that we had for the request was 100 milliseconds, right? So if you hit two of those, you're already over the limit. And it's kind of easy to hit because each request servicing might hit multiple IO's, so it's kind of easy to hit two of these and then miss your latency. So that's how the whole system suffered. IO protection didn't work at all, which propagated back to memory pressure, which slowed down the workload and that's how we ended up with basically zero RPS. So Dejan, in this scenario, when you say memory leak, is that the application or the memory leak workload will allocate memory and not release it? Or it'll just, okay, so it's grabbing onto the memory and keeping it occupied. So that when a reclaim comes in, it really can't find anything to reclaim? Is that correct? No, I mean, so there's, it's a fairly simple memory, it's a cold memory leak. So what's happening is that it allocates memory and it just doesn't look at it anymore, right? I mean, it allocates memory, uses it once and then doesn't go back to it at all. Okay, and it doesn't release it, doesn't use it. It doesn't release it, yeah. It's just leaking memory, right? And when the memory reclaim comes in, it can do something about it and what it can do is putting this into swap, right? That's how you create more memory. But it creates IOs, right? And then if you don't control those IOs, then you start affecting the main workload. So that's where these IOs, these purple IOs are coming from. That's the system trying to put the memory, free the memory by winding out these pages into swap. I see the characteristic of your memory leak workload is that it would allocate and just hang on to it without actually using it. So the swap memory reclaim comes, okay, I can push these into the swap and then as a result, it drives up the IO characteristics, IO usage and then, okay. It's a multi, it's a kind of a domino effect. Yes, and another effect that it has is that because it's expanding in size, right? Constantly, right? And the color is trying to balance memory allocation. So it ends up pushing out the memory of the main workload to disk, right? I mean, to SSDs, right? I mean, it just drops the pages, right? And that increases the page faults of the main workload. Right? And that's why you see, if you look at this green graph, right? Let's read increasing for the main workload. So this is the main workload taking a lot of page faults because its memory footprint is getting shrunk constantly. Right. So it's kind of double-winning, right? I mean, you have competition from IO side only two because there are swap writes, but also because of memory pressure, it creates an S to you. You yourself has to do more IOs, right? Right. So yeah, so there's a really steep competition there. Thank you. All right. So now doing the same thing with the protection, right? Let me see how that works. Just to show you in the interface, going back, here it says, you know, it's red, obviously it's not good. And it's negative, you know, CPU, MIO, it means that none of these controls are turned on, right? And the inside loaders are off too. So it's just showing that the interface is showing you that reset control is off while this was running. Now doing that with C. If you look at here, you know, it doesn't have that anymore, right? So everything is green except for standby, that's fine. You get to that. Maybe not in this session, but you can read about it later. But you know, none of everything is generally green. So that's good. And this is again, about 30 seconds into the after starting the memory hall, memory leak. And you can barely see, I mean, so the memory leak started here, right? I mean, in the bottom, you can see that, you know, it did quite a bit. And IPS drops a little bit. And if you look at the latency, it, you know, it doesn't quite stay the same, right? I mean, it increases a little bit. But overall it's fine, right? I mean, it's doing most of the work that it's supposed to do. And if we look at the pressure graph, right? It's really different from before now, right? CPU, you know, about the same, a little bit lower because we are doing a little bit less RPS. But if you look at the memory and IO pressures, the system pressure is really high, right? That the purple lines, purple dots are really high. But the green one, why is a little bit waste is not going up at all, right? So now the system knows, you know, what to protect and what's important and what's not important and directing resource distribution based on that information, based on the configuration and thus it's not harming the workload to serve the, you know, system that, you know, the memory link, the less important part of the system. And the same thing here. One thing which is interesting here is that, oh yeah, this one, but that one is more interesting but this one first. So if you look at the memory lines, right? For the main workload, for the web server, it is staying flat, right? So the system is not taking any memory from the web server because it's protected. So there's memory control at work. And if you look at the Swabian utilization, the purple line, only the purple line is going up now. Meaning that, you know, that the memory link tries to expand in the system that slides in the less important part of the system. But it cannot expand in terms of memory, right? I mean, that system kind of puts cap on it. And then whenever it tries to expand, it just kind of sucks out whatever is over. That's what's going on. And if you look at IU utilization, like before, you know, it spiked and everything crashed down, right? Now it is a lot better regulated, right? I mean, there's some rise because there's more load on the system. And, but also you can see that the green lines are staying way higher than purple lines, right? So it's not being like, you know, 50% fair. What it's trying to do is that it's prioritizing the important workload of the system than the less important one. So there's IO controller working. And this is the IO graphs, right? The utilization and latencies. So if you look at the latencies, you know, you can see that, you know, it's not, you know, P90 used to be, you know, 60 milliseconds. Now it's maybe four or five milliseconds, right? So this is in a lot safer, manageable situation. So yeah, it works, right? If, you know, the system can do this, we can survive like these things, you know, how many times that we need to. And eventually, right? In the first graph, you know, we did that three times, right? And what eventually happens is that UMD, which is the user space UMDemon, and this feature has been integrated into system B, and it's for system B UMD. But here it's using like a separate standalone demon. And it monitors like resource pressures and swab disease and all those things and makes UMKIL decisions. So it just kills if something is clearly misbehaving to protect the system. And eventually UMD kicks in and kills the memory leak and everybody's happy. And with that, so that's the protection scenario. And we are starting eight minutes in. I'm debating whether we want to do live demo. Okay, let's do a live demo with this, right? My machine got disconnected in a second. Oh yeah, okay, yeah. Okay, so I already launched the thing and opened the same page. So it has been running for a while now. But you can see that the RPS is at 1,200 maximum, latency is at 80. This is just because the parameters got updated in the meantime. So this is a newer version of resource control demo. And the parameters has been tuned so that it's more in line with production workload. So in terms of like memory and IO and like resource contention responses, it shows kind of like ballpark behavior of our particular production workload that we use for benchmark. So that's why the latency target is more strict now. But anyways, you can see that, it's 100%, well, not 100%, 1,200, which is about 95% load level and latency is hovering around 80 milliseconds. And if you look at the graphs, yeah, well, this is not that interesting, right? Only CPU is being contented because we are at full load. If you look at utilization, like only the green lines are up there, right? So that's the only thing which is really using the system right now. And IO load is about 60%, right? So the system is close to full saturation, it's fully saturated, but there's some buffer in terms of IO. That's how we wanna run it because without any buffer, the system is not heavy. And if you look at IO, latency is reasonable, three milliseconds somewhere there. 90 is one millisecond, that's really good. So nothing is really wrong. I mean, everybody seems pretty happy. Yeah, these are other statistics, okay? And now I'm gonna go down and there's a bunch of explanations, but I am gonna activate this button, right? It says disable all resource control features and start memory hall. So now we are doing the exact same scenario. If I do that, I'm gonna press enter and you can see that memory, all the resource controls turned off and yeah, this is starting to fall, right? And if you look at this here, system memory is constantly increasing. This is going down and you can see those things here, right? This is memory going down for the main workload and memory going up for the memory leak, right? And CPU went down, I'm gonna go another graph, right? IPS is now zero, right? Nobody's happy and the resource pressure for IO and memory are high. And you can compress the timeframe if you wanna see a larger timeframe. You can zoom in, zoom out. If you look at the time axis, you can see larger timeframe. Anyway, so it's kind of obvious that this is not great. So let's go down and start it. And now if you look at the log, there's a log page too. So if you look at the agent log, you can see that a bunch of stuff, but it basically will tell you that what it's doing. And that was, no, it's just load. Yeah, this guy, right? So this guy was doing all this memory leak. And then when I stopped it, it got terminated. And eventually, eventually the hash will come up, but it will take some time. You know, just because it uses like a PID controller. And another thing is that if you look at the memory, memory footprint, right? I mean, it basically dropped to zero, right? So it needs to recover its entire working set. And then there's a PID controller, which regulates the RPS and is looking at the latency. Oh no, it actually died. The workload actually died. Last time we started. Yeah, I can look at the log later to see why that happened. But what happened is that the main workload actually got killed. It could have been on Oomkiller or Oomkiller. Anyways, you can see that, you know, this was not a happy ending, right? I mean, so you had a workload running at full tilt. You started the memory leak and the whole thing died, right? This is not good. There's a couple of questions in the answer box. If you want to take a look. The first one, the question is, is there any way to dynamically allocate memory to a legit workload but still protect against memory leaks coming from the workload? Yes. There are two ways to control memory. One is, one are limits. So these are in C group two terms, of course, are memory.high and memory.next. What that does is that you pick a guy in the system that you want to limit in terms of memory usage and you set the limits and they cannot, you know, go up more than, consume more than their memory amount. The problem with that approach is that that you kind of have to under commit your system, right? So you cannot use your system fully, right? Because if you set those limits, let's say there are competing users or memory in the system. And if you set the limits so that everybody's always safe, right? You have to under commit the memory otherwise, you know, because somebody's gonna be unhappy. But if you wanna increase the memory utilization by setting memory limits more relaxed, then you can get into a situation where the system is not fully protected. So this, I suppose that's what the question is asking. So that's why the primary method of control that we use and research control demo uses is called the memory.low. What it does is that it's a preferential treatment. It's like a priority for memory. So what it says is that if workload is, once this memory up to X gigabytes of memory, then it has preferential access to that memory. And above that, it has to compete with others. So that's how the system is configured. So what happens, the way that memory is configured here is that workload, the main part has 80% or 70% of memory set on its memory.low. So up to that point, it basically has like a fairly strong protection. And beyond that point, it's just competing with others. So if the rest of the system is idle, it can use the entire memory. If the rest of the system wants more memory, it competes and it can give out as the, depending on how much the other parts of the system need memory, but in a way which it prefers allocating to that main workload. So I hope that answers the question. So the consumption is not like fixed for workloads. And that's also why you so, I think this is actually a really good question. So if you look at here, you cannot see. So this is the protected scenario, right? So all the research controls are on and memory leak started. And this line didn't really change. You cannot really see that here because the change is kind of subtle. But oh, you can see it here. So the line actually was here and it went to one character. But if you look at the IOS litigation, it tells you their story. So the main workload, the web server was doing exactly the same amount of work, right? The IPS stayed the same. But when the memory leak started, IU utilization of the workload rules, right? Not by 40% or so, sorry, 40% or so, it rules. And the reason for that is because this memory utilization dropped by a little bit. And that creates more IOS because it causes more page efforts and this was being reflected here. So what this tells you is that when the system is otherwise idle, right? The main workload can take all of the system. And when there's some need, like really low priority here, but some low priority need in the system, it can give in a little bit, right? So that the system is fully utilized all the time. So hopefully, so that's, you know, work-conserving memory, memory prioritization working. Hopefully that answered the question. The second question is that you mentioned that this system uses SQL2. What are some of the characteristics differences between one and two? Okay, let's just say. So the second question is asking the difference between SQL1 and two. SQL1, the main challenges of, the main challenge of SQL1 is that it's really difficult to build relationship across different resources. So you can create like a totally separate resource hierarchy for CPU memory and IO. And that doesn't quite work well because as you just saw, right? I mean memory and IO, CPU2, they interact in a really intricate way. So you cannot really just control memory and be okay with it. And if you control just IO, something is gonna, you know, you cannot achieve like comprehensive protection for your workload unless you control all of the resources together and control their interactions. So the main reason for SQL2, the reason why SQL2 happened, right? I mean, instead of improving SQL1 incrementally is kind of to have that common hierarchy. So what happens is that now you can have a common resource domains across different resource types. So now we can control memory and IO together, memory and CPU and IO together. So that's what SQL2 allowed to do. So achieving like this kind of protection, right? Like this kind of protection, like where IPI stays at the top with SQL1, maybe possible, but it's really difficult. So there's, you know, you gotta use SQL2 to do that, to do this, to achieve this. Okay, so now we waited long enough, right? So the main workload is where we wanna be at, wanna see it at, it's fully loaded, there's latency, 80 milliseconds, right? And now just let's start the memory without disabled resource control. And yeah, I just pressed enter and you can see that system memory is increasing, right? And you see a small block and stabilization, right? So this is a little bit harsher than what's captured on the slide, just because our latency target is lower now. So this is a, the condition is a little bit harsher, but you can still see that, you know, we lose maybe 5%, maybe 4%, but I mean like very little of maximum capacity and our latency was, while our latency was up a little bit, you know, not by much, okay, that's not good. But anyways, it should be able to hold, but yeah, the tuning was fairly recent. So our IOCost, the IOControl parameters maybe a little bit too relaxed, but in general, right, there's still holding up fairly, a lot better than without IOControl. And you can see, you know, all those patterns, right? Yeah, so if you look at swab usage, right? The memory, you see it dropped a little bit, swab usage is climbing for the memory leak, CPU utilization, stabilizing, and you know, IO utilization, attenuated towards the main workload. And latency, latency is a little bit higher than I would want to see at, but you know, 5 milliseconds is not too bad. Yeah, and this is a little bit dirt here, the graph is a little bit nastier than idea, but you can still see that, you know, this is, you know, a wholly protected, mostly working environment. There's another question. Okay, if you have two processes, Steven, the memory, okay. The question is still complicated for me to parse right now. So I'm gonna get to that later. Okay, let's get to side loading. This is kind of fun. So we have like something called the TR buffer. The TR buffer stands for disaster readiness buffer. And so we have multiple data centers across the world, but let's concentrate on US because I'm in US right now. Let's say, you know, one of the data centers, like there's a wildfire in California and you know, the power went out and the data center lost power. And so then let's say that we were expecting the emergency generator to, you know, come back to back up the power and keep the data center running. But let's say, you know, somehow that failed. So when we lost power, we lost the entire data center. Because we have multiple data centers, like ideally what we wanna do is that for other regions, other data centers to take over that traffic immediately and transparently, not completely transparently, but fairly transparently so that our users don't really notice that, right? I mean, that'd be ideal. Having that redundancy actually work for you and in a transparent way. And to make that work, to achieve that, what needs to happen is that you gotta have some standby capacity, like warmed up all the time, right? You cannot have, you know, you cannot have like, you know, I had machine set aside for emergencies when those guys lose power, I'm gonna turn them on and try to, you know, install applications and start them. That's gonna take, I don't know, an hour, 40 minutes, whatever will help you, right? So you cannot really do that. So ideally what you wanna have is that you wanna have applications, all the applications warmed up fully and have some capacity on top so that when a digestor strikes, you can take over the traffic right away. So that's why we call, that's what you call digestor ratings buffer. And what that means for our workloads is that, important ones is that we tend to have some percentage, you know, that's whether that's 30% or 40% or 20% of machine capacity that we are not using all the time. And so there's opportunity, right? Can we use that, you know, DR buffer, right? Digestor ratings buffer for something else, something productive. So that's where side loading experiment started. And if you think about it, I mean, it's not just DR buffer, right? I mean, like if you have a like large enough flit, there's, you're bound to have like some scheduling inefficiencies or allocation inefficiencies. So you end up with like extra capacities in a lot of places. And the question is whether we can do that, we can utilize that for something useful without impacting what the machines are originally for. So lively thinking, right? You can sort of see that, you know, this is a little bit generalized case of the previous protection scenario, right? Protection scenario was you have memory leak, can you protect the main workload from the memory, right? Maybe we can just replace, right? I mean, substitute the memory leak with something productive, right? Maybe that's all we need to do, right? I'm gonna get to the questions a little bit. Oh, okay. No, I'm gonna do them now. So the question is, are C Group 2 and Umbi Numa aware? They aren't. I mean, not really. They don't have, they don't really have, yeah, explicit Numa awareness. C Group 2 does have in the, in C Group 1, the CPU set controller does Numa configuration things, but by and large, C Group 2, it's just looking at overall memory utilization rather than it's not specifically Numa aware. The second question is, why Rust? Oh, well, so okay. Why Rust? Yes, I didn't wanna do it in C. I didn't wanna do it in C++. And I couldn't do it in Python because that if you look at things like RD-HD, so all of these simulations or these things are fairly performance sensitive, you're trying to saturate the entire system with a multistraded application. That's actually really difficult to achieve in Python, unless you do like multi-percent. Maybe you can, but I mean, I just couldn't figure out. And I wanted to run Rust and Rust actually worked out really well for building this type of system applications. Hopefully they answered it. Okay. So yeah, okay, side loading, side loading. Okay, so this is the side loading scenario. So if you look at the top here, this is where the current page, the name of the current page is the side that intro. So it's introduction to side loading. And this is doing the naive approach where we just set up the user protection setups for CP memory and IO and start Linux kernel build in system.slice, right? I mean, so instead of memory leak, we are now just running, you know, in trying to build a kernel, right? And if that works, right? I mean, we actually have a lot of build jobs internally too. So maybe we can distribute them across these machines and then extract work that way. But that started quite a bit ago, but the compilation portion started about 20 seconds ago. If you look at the IPS line, this is mostly okay, right? I mean, so the machine is not fully loaded. I mean, machine is now about 60% loaded, which is can be user for some services, you know, when there's no digester. So the load is, you know, load level is kind of okay. But if you look at the latency, latency is higher, right? So it's not at 80 milliseconds, which is our limit, but it's substantially higher than 40 milliseconds than we were where we were before, right? We were 40 milliseconds. We are now at about 60, right? So there's more than 50% increase in terms of entry and the latency for our decads. And this is problematic. So, excuse me. So one benefit of having this kind of like a undercommittment, undercommitting machines like with DR buffers is that our latency has to improve a lot, right? When you're, you know, saturating your machine, you know, you're a lot slower doing things. But when you have, you know, that kind of headroom, you know, your case takes a lot less time. And obviously, you know, users would like that, right? I mean, they don't have to wait for things for that long. So latency rising by 50% to extract some extra work from the system. Maybe that's justifiable, but you know, it's not a clear case, right? I mean, you know, that's just kind of, you're paying a little bit too much to gain a little bit of extra work out of the system. And, you know, the trade-off might not be great depending on your application. So why is that? Why is that? Give me a second. I post my notes. Okay, there you go. Why is that? If you look at the, this is the resource pressure graphs from the same timeframe. If you look at the, so the latency is rising here at the same graph. If you look at memory and IO, right? The main workload is not experiencing anything. So that shouldn't be the cause. But here, if you look at the CPU pressure, CPU pressure is rising, right? As the system CPU, the Linux build job is experiencing CPU pressure, but also the main workload, our web workload is experiencing pressure. And, you know, that's what's causing, not drop in RPS, but increase in our latency. And why would that be the case? Why would that be the case? There are multiple reasons. One is that, you know, the CPUs are time-shared, right? So you have like, you have timeline and you're just cutting the time for different workloads. And as you, as you saturate the machine more and more, right? When something is ready to run, there's a more likelihood that something else is already on the CPU, right? Even if your party is higher than that guy, you know, depending on how the, you know, scheduling decisions are made, you might have to wait a little bit and all that. So there's a latency, which is added in terms of scheduling, right? As you saturate your CPU more and more, you end up waiting for your CPU, your turn on the CPU more. So that's one source of latency. Another thing is CPU supply source content. So if you think about CPU, it's a fairly complex thing, right? When you, when you say CPU utilization, it's not a simple thing. It's a, you know, you have, you know, the CPU logics, you have this cache, you have registers, you have, you know, your, you know, never one, never two, never three caches, and then you have memory bandwidth, and then you have, you know, your memory subsystem. And then this whole thing, like a different parts of it can get saturated, right? So when your CPU is fairly underutilized, whenever you try to use some of these resources, they're available. As you, as you saturate the CPU more and more, it's more likely that, that you're gonna experience some kind of contention somewhere in the whole stack. Meaning that as you saturate your CPU more and more, your CPU will become slower in the sense that, given the same instruction, it will take longer time to execute. And another really easy source of this is frequency scale scaling, especially with modern CPUs, right? When you are using, underutilizing your machine, the CPU, you know, has enough heat and power budget. So you do ramp up this frequency, the CPU is faster, easy instruction is running fast. As you load it more and more, and it's running out of those budgets, so it kind of scales down the frequency. So each instruction is now, so the CPU is slower now. So if you make your CPU slower, then whatever you're doing on the CPU takes longer time, so your latency increases. So these two or three factors are the main factors which play into increased latency when you really saturate your CPU. So what side loader, which is managing, which manages side loads, does is that it regulates CPU max dynamically to ensure that the system as a whole has some level of headroom. So when you add up these system level, if you look at the system level CPU utilization, it regulates these side loads so that the utilization doesn't rise above, say 80% or 90%, what have you, right? I mean, you can configure the level. So what it ensures is that the whole system as a whole has some CPU time unused so that there's a less contention in scheduling, less contention in our CPU sub-resources. And if there's a power scaling, right? I mean, it doesn't push it to the limit. And that's what side loader does. And depending on how big you set your headroom to be, you're trading off bandwidth against latency, right? I mean, if you have your headroom big, you can gain less bandwidth, but your latency impact is really low. You can, you know, you can go the other way too. Tejian, one question here with the CPUs. So would you, in this scenario, would you assign, divide the CPUs into groups and assign certain CPUs to your workload and reserve the others for side loading? That's one option. That's not what we did. That's one option. But here we are just using, so okay, so the rationale for doing what we did, the reason why we did it the way that we did it, we did it rather than splitting our CPUs, is that we want it to be as transparent as possible to the main workload, right? So from main workload's point of view, it didn't need to change anything, right? It's using the machine the same. And when it wants to ramp up, it should be able to ramp up the same. So that's the reason why we tried to, we didn't segment CPUs, then because it becomes more visible to the applications. So here we are just kind of throttling the entire system. And let's see how that works. Okay, yeah, just about the same point in time. So this is now running the same Linux build job. You can build this here. Bridge up under side loader. So it's now kind of regulating the whole thing, the CPU utilization using side loader. And you can see that it has risen a little bit, right? I mean, several milliseconds. And RPSC is about the same, but the latency has risen a little bit, but not by 50%. This is why three, four percent, a single digit percent. So that's a lot more palatable, right? That's a trade-off you can make. If you wanna like, if you ask, if you go to like a production team and ask that, if you give us 5% latency overhead, we can give you 20% more capacity, right? A lot of teams would say, yes. If you say that, I can give you 25% capacity, more capacity for 50% increase in latency, most teams we're gonna say no, right? So that's what this gives you. And if you look at pressure graphs, right? It's like the CPU graph, if you remember the previous one, both the workload and the side load one was rising together, right? So the workload one was high too. But here, while it rose some, but it didn't rose much, right? And the side load one is a lot higher. So this is kinda attenuated properly now. And if you look at utilization, CPU is probably the most interesting. Yeah, but we can go to other stuff too. But yeah, this is cool. So if you look at this side load, right? Edit up, what would that be? 70, 80% utilization, right? So we are getting 20 something percent of CPU extra. So before, oh yeah, this is cool too. So it says 40%, right? But our load was 60%. This is because when the machine can do like 100 of work, and if you let the machine do 60 of the work at the same time frame, the CPU utilization is not gonna be at 60, it's gonna be at 40 or 50. The reason for that is, the utilization to total work graph is not linear, right? It's as you go, higher in utilization, your total work is, there's a diminishing point of return and because it draws a curve. So when you load the 60% workload, total work on the machine, CPU utilization you're gonna see is about, about 40, 45, something there, depending on the CPU. And when your CPU utilization is about 80, 90%, in terms of total amount of work that the CPU can do, you're already at, I don't know, above 90% in actual utilization, right? As you push higher than 85, 90, 95%, you're not gonna extract matching amount of extra work out of the CPU. You're just gonna be slower more. So that's why it does that 40% with 60% load. And, but we can utilize about 25% of the CPU time for building the corner. So that's kind of nice. And this is actually a fairly substantial amount of CPU time. So you can do, especially across many machines, you can do a lot of work with this kind of CPU time. And one thing which is really interesting is that this grows by a little bit, the green line grows a little bit, although the CPU is doing the same amount of work. So this is showing that the CPU became slower, right? So the green line, the workload is doing the same amount of work but because we are loading the CPU more, the CPU became slower. So in terms of workload time, we are taking more time on the CPU. So that's what that shows you. And that's about it for the side loading scenario. So what this gives you is that you can gain, if you have something like a DR buffer on the committed machines, you can gain extra utilization, extra work out of these machines without sacrificing much of it. A little bit of latest, you know, three, four percent. So that's what it gives you. We don't have it deployed. We did a bunch of production, some production experiments with it, but it, because of the latency, certain percentage of latency affects how wood balancer behaves, responds to it. So we don't have it deployed in production yet. So that's something we are working on. And I'm just gonna go to this little bench. Okay. So this is some, if you have any questions about side loading, please go ahead. And I'm gonna answer that if they come up. So let's go to research control bench. So research control demo, the goal is verification and demonstration. I think it's in the name, right? So it's trying to explain how this whole thing works. It's trying to kind of let people know and this is how this is supposed to be used. Why is such a this way? And all the internal details, right? So it's because there's, I don't know, we had to learn a lot of things in the process. This is trying to share the information with live scenarios so that people can have more kind of tactile, I don't know, understanding or sense of what's going on. At the same time, because it's exercising the system in a fairly realistic way. It's a really good vehicle for verifying that the system is working as intended, right? So we actually called, I actually called quite a few bucks on the corner side and our configuration side while trying to make research control demo work reliably. And then what the list to is that once we have the framework, right? You can build a benchmark on table, right? Using all these components, you can simulate the entire system to do some kind of canned benchmark to measure certain behaviors and test certain behaviors. This became fairly important for us, especially for SSDs, because you can think about how SSDs are benchmark and you do file or you do IOPS or measure these different things and you sometimes do capture IO trades and run them and try to measure time and all that. But that doesn't really tell us how they would behave under like resource contention. So, especially memory contention, because as we saw earlier, memory contention tends to create a lot more IOs, right? I mean, both in terms of write outs and increased page forts. So under those conditions, right, read latency becomes really important with like this certain IO pattern that the corner generates for swap outs. So they can interact really badly and then we cannot really protect the workload properly. So we wanted something which would reproduce the actual jobs, right? The actual scenarios that we are interested in to benchmark devices. So there's a reason to the bench. So there's a reason, right? System behavior under resource contention tends to be really complex and it's kind of difficult to capture in simple benchmarks. So what this resource bench does is that it simulates the entire system, application and color and everything. And not like, it simulates the scenario so that we can exercise or the code path in the corner and everywhere to make sure in the devices that it actually works and we can measure how they work. And one interesting complication for this type of benchmark is for IO costs. So IO cost is the IO controller with the thing which distributes the IO resources across different parts of the system that the whole thing use. And what it does is that it has like an internal model of what the device can do. So it has like model parameters and it has like a QoS target parameters, right? I mean, quality of service parameters. So it has, in total, it has like a little bit over a dozen parameters to configure to make it work well. And obviously, if you have to configure like a dozen parameters for IO control, it's not trivial. And even different configuration works more or less but this configuration just turns out to be really difficult. So queuing the QoS parameters like that, what kind of overall throttling that we need to do to obtain to meet the latency requirements turned out to be really challenging. But what we did with IO cost tuned benchmark which is part of this control, one of the scenarios in resource control bench is that we simulate the whole thing, right? So what we are interested in is whether this device can protect, how well the device can protect at different performance level, at different throttling level. So it repeats the scenario that we ran in the live demo over and over again at different performance level, throttling level on the device and get us a lot of data and then analyze the data to eventually provide solutions, right? So if you throttled this device to this level, you can obtain that level of protection. So that's what it does. So here is an example output from one of the SSDs that we use in the flip and in production. And you know, it's just a lot but on the left side, the most important, this one is called the memory of floating factor. So access is on all these graphs is that how much we are throttling overall. So 100% is the, we are not throttling at all. So we are letting the device do whatever the device can do and we are just gradually kind of slowing down overall IOs, how many IOs we are issuing per kind and then seeing, observing the behavior. So right side, let's go right side first. So right side is the read latencies. So this is median latency. So a little bit of too much detail but I mean, what you can generally see is that as you throttle more latency goes lower, right? That's what you want to see. So you can control your latency response by how much you slow. And obviously, you know, you have a certain point, you know, there's no difference, right? The device is just doing what the device can do but as you go throttle down more and more you can improve the latency response. And you can, you know, pick a line here, pick a point here, for example, if you have a specific latency requirement. But- Can I ask you a question? Two questions, one is on the previous side you talked about isolation. What is isolation in this scope and what is throttling in this scope? Okay, so, okay, isolation, there's an isograph. Okay, isolation factor is defined as, so let's say your target RPS is 100, right? And then you start memory hall, memory hall, right? Memory interference or whatever interference, right? And you wait until that experiment is over or times out, right? And then you measure the latency, the RPS you actually observed during that time period and you divide that by your target RPS. So 100% isolation means that, you know, you were able to keep your RPS at the target level, you know, although something was going on in the system. 50% means that, you know, you lost half of your work because there was something else going on in the system. And in reverse latency impact, well, not in reverse, but in conjunction, latency impact is defined as your target latency, you know, that's the divider. And the observed latency is the, I'm getting confused. Anyways, you know, it's the observed latency over target latency, right? So if you look at this graph, right, this is showing latency impact, although it's not a great fit because the device is not the most consistent one, but you can see that as you throttle down the device more and more, you can control latency impact, you can lower your latency impact gradually. And this is the isolation factor, ISO one. The reason why, you know, it doesn't drip down, you know, drop like that, it should, is because, you know, the benchmark is tuning the size to meet this target, but anyways, this is the, should be ISO one. Yes, anyways, I think there's a little bit too much detail to go into in this session, but what it does is that let me, so, for example, I'm just gonna give you one example, give you one example. This is called MOF graph. It's a memory offloading factor graph. What it, in memory offloading factor is defined as supportable memory footprint over physical memory, right, available memory. So if you have a memory offloading factor of two, it means that your system can support twice as large memory footprint than your physical memory while not missing your latency targets because all those pages can go to IO device and IO device is fast enough to service all those requests without missing latency target, right? That's how MOF is defined. And the MOF is also, it's a latency bound bandwidth measure, right? So higher than MOF means that, you know, your IO device is doing more IOs, but also while within certain latency limits because if your device is high latency while doing those IOs, RTS wouldn't be able to hold at the target number, right? So it's a latency bound bandwidth measure. So on this device, on this particular device, because of conjunction of all these latency graphs changing, right, this is the eventual graph we get while not missing latency target, the memory footprint to that we can serve is about 1.4 or five. And then as we sort to the device, sort to, you asked about sort to you too. Sort of telling is just that the IO controller has a general understanding of how much the device can do through model parameters. And as you tell it to sort to it more and more, the device just kinda, the controller slows down how quickly IOs can be issued to the device. So at 50% throttling level is issuing, is gating request, is holding request so that it doesn't go faster than, you know, 50% of what the device can do at maximum throughput. So as you sort to more and more, the MOF drops, right? Because you get less and less bandwidth. So eventually you fit the graphs and you get this line and you can pick here, right? If you pick here, this is the point where you get maximum bandwidth, right? Maximum MOF while staying lowest in the throttling level or on the other hand, while obtaining the lowest latency, you're still getting the, you're not losing on bandwidth side, but you're obtaining, you know, the best latency you can obtain, right? So that's an example of how solutions are calculated. So this is how solutions look. It's, in particular, it's like, I don't six solutions, but this is, you know, how two of those solutions look. And this is binary solution. And, you know, it considers those factors and then spits out these numbers. And this is what, you know, these parameters, Rbps, Rcqops and Qs parameters, these are what can be entered into the corner configuration to configure your IO device so that it can be controlled. So this is one example of how a business control bench is used. And we are coming up on, it's about five minutes left. Just wanted to make sure. Sure. Yeah. I am, we're done with the slides. So we're, okay. So if you want to try out business control demo, you can visit this website. There's a GitHub and we are in the process of preparing another kind of big release. So what is released right now, version 1.0 doesn't have a business control bench in it. So we're going to be releasing another one pretty soon, hopefully in a month, which has this program bench in it. But if you go to the site, you can still try the entirety of this control demo and, you know, read everything in it, which can be interesting. I, you know, I enjoy reading it. So keep the shot and the slides will be, you know, shared too. And that's it. I'm going to put the address. If anybody has any questions, yeah, we have five minutes as Sharjah said. Can the resource control framework be used along with Kubernetes containerized workloads? I don't know. I don't think, so there were some discussions around it, but right now, resource control, resource control, the entire thing, to implement like comprehensive resource control, the requirements are fairly high. I mean, you got to be on single two, we're going to use IOCOS controller. You got to be on Barad-FS for now in this. So that doesn't, you know, quite jive with what Kubernetes is doing right now. So I don't think it works right now, but, you know, there were talks about it. And hopefully down the line, we will be able to. But right now, the system requirements are just kind of, you know, a little bit difficult to meet for Kubernetes environment if I understand correctly. That's my understanding. So the question from Maykan, I don't know whether I'm reading the name right, but I got to sit down and actually think about it. So I'm going to answer that like later, but I'm not going to try to do that right now. I just, you know, have a hard time parsing it right now. Thank you. Sir Tejan, one question. Have you played with the memory optimizer that Oracle user space application that predicts memory usage scenarios and then proactively reclaims them? If you were to use that in conjunction with your resource control, do you see you could have more benefits? Is that the memory optimization thing? I haven't used it, but is that trying to detect hard end cold memory and offload cold memory? Well, yes. I mean, it predicts the free, it uses a little bit of a predictiveness of where essentially you have watermarks, right? The memory watermarks. And then it'll kind of adjust the watermarks so that optimizer kicks in quicker so that you are not experiencing, the workloads don't experience pressure. So I can give you a link to that. Sure. And then, you know, I'm curious. So what happens if you were to use that in conjunction with your resource control? In resource control demo, it's explained in resource control demo too, but we use something called standby for sizing. So what it tries to do is that it tries to measure the actual working size rather than, you know, dropping all the cold memory. So we try to measure the actual memory consumption of needed working size. So that's what we primarily use for sizing inside Facebook. And this open source too, and it's fully explained in this resource control demo. Yeah, but that's what we primarily looked at. We haven't looked at memory optimization. Okay. So the difference I see is that your solution is very controlled solution in the sense that you have to understand your workloads, you have to set aside resources. To some extent, you have to have a characteristics of your workload. So I wondered if a generic solution like memaptivizer, I mean, it's only memory. I know you were going with other resources, but just curious thought, just I was thinking. Yeah, sure. I put it in the chat. You don't have to answer that now, but just a minute. Yeah, yeah, yeah. Just on the contribution side, because we use, even on the memory side, we use memory to low instead of memory to high or max. Configuration is fairly easy. So we basically use a single configuration for most of our machines. Because, as from, and it's tied to machine, like resource allocation, right? So when a task we cast an X amount of memory, then we can base allocation on that. So it's not quite that difficult to get that computation right for us in this. Yeah, but yeah. But I mean, sizing definitely is a difficult problem that everybody's trying to get, cool. Thank you. Right, well, I just want to thank you to Tay June and Shua for their time today. And thank you to all the participants who joined us. As a reminder, this recording will be on the Linux Foundation YouTube page later today. And a copy of the presentation slides will be added to the Linux Foundation website. We hope you're able to join us for future mentorship sessions. Have a wonderful day. Thank you so much. Bye, guys.