 Hello everyone. I'm Prudhvi. I work for Intel. My co-presenter, Suzanne, who's leading Watcher's efforts from Intel. Fortunately, she couldn't make it. So our talk here is about automating noisy neighbor detection using OpenStackWatcher. So this is the agenda of the talk. First, I'm going to talk about what motivated us to come up with this strategy. What is a noisy neighbor? What is shared resource contention? What is Intel's resource director technology? Why do we need to automate noisy neighbor detection? What OpenStackWatcher is? How can it help us achieve noisy neighbor detection? What our algorithm is? How it fits into a Watcher strategy? Then I'm going to show you a bunch of screenshots and just to show how it worked for us. And some final remarks. Okay. So what motivated us to work on this strategy? The very simple answer is resource utilization enhancement. Now that a lot of OpenStack code projects like Nova, Neutron, Cinder, they're very close to maturity, now you have an OpenStack cloud which runs fine. Everything works great. But the next question is everything works fine. But how do I enhance my resource utilization? How do I get the most out of my cloud resources which are already there? And that's the reason why a workload placement is very important. It's like a game of Tetris. You need to look at all the aspects while placing your workloads on various servers to get the most out of it. We already know a few ways to enhance resource utilization like job packing or cluster over subscription. But a lot of these methods are not very easy to implement. It completely depends on the data center, its use cases, and it has its own set of cons. So what is a noisy neighbor? So when we have more than one VM or an application on a server, a lot of VMs, they fight for resources. There is a lot of resource contention that's happening below the table. And so when you have two VMs and each VM is fighting for its own set of resources like cache, L3 cache, CPU cycles, each resource, each VM is going to affect the other VM's performance in a negative way, of course. It leads to performance degradation completely depends on the scenario. And let me just explain this with a very simple example. Say you have a server with just two VMs on it. You have a web application which uses some web services and each VM has a web service. Say one web service is very important like a payment web service for your web application. And the other web service on the other VM is just some add-on to your site like, say, image morphing. So which is not really important, but it needs a lot of computational power. And when your image morphing thing runs, it's going to try to draw all the resources towards itself, which is going to affect your very important service, which is your payment service. So this is something which we want to avoid. And this is a very simple example of a noisy neighbor. So to put in formal terms, when application performance is greatly in a negative way affected by a co-locating application, you have a situation called noisy neighbor. And noisy neighbor is the application which is overutilizing the resource and which is leading to performance degradation of a very important or a high priority VM. Let me put noisy neighbor aside for a few minutes and let me talk about what Intel Resource Director technology is. So as I've just told you, you have a lot of resource contention going on. And one of those resources is LLC, last-level cache or L3 cache. And this L3 cache has a direct, it has, it is directly proportional to the performance of your VM. So Intel RDD is a technology where you can monitor or allocate L3 cache on the go, on the fly, right on spot. And you can also monitor memory bandwidth on a per VM or a per core basis. Let's just focus on CMT, which is cache monitoring technology, because that's very relevant to our strategy over here. So using Intel's CMT technology, what you can do is you can know the amount of L3 cache utilized by a VM or by a single core or by a single app right on spot at that point of time. So using this, you can come up with some strategies when your high-priority application is getting very little of this. You can try to kill other applications. I'll just come to that in the next few slides. But yeah, CMT is what we're using. So telemetry is a very important aspect to understand how well your resources are utilized in your cloud. And a lot of times this telemetry is limited to compute or network or storage. But actually if you can get telemetry all from the silicon level, that's going to help you in understanding your cloud better. And it can help in intelligent work load placements. You can come up with strategies where you can minimize resource contention. And you can come up with strategies where there's continuous resource optimization. Okay. So by now, you know the problem here is noisy neighbors. That is something which we are trying to solve. And you know, you must have guessed by now Intel RDT is the tool which we are using to solve this. But we need some other component to fit into this jigsaw and to help us use RDT to automate noisy neighbor detection. And that is OpenStack Watcher. So for the next few minutes and in the next few slides, I'm going to explain you what OpenStack Watcher is because it's very important to understand how Watcher works, to understand the strategy. And I'm quite sure a lot of people in this room do not very well know what OpenStack Watcher is or what it does. So OpenStack Watcher is a flexible resource optimization service for your OpenStack-based crowds. It's a service used to optimize your resources in an automatic fashion. So what Watcher does is it audits your cloud against a set of predefined optimization algorithms. And based on that, it comes up with some strategies which it presents to a cloud admin which he has the power to apply directly using Watcher CLI. So, yeah. So Watcher has a set of goals and a set of strategies. So all you need, if you're an OpenStack admin, right, all you need to do is set a goal and map that goal to a strategy and just leave it and Watcher does behind-the-scenes jobs to achieve that goal using the rules defined in those strategies. So OpenStack Watcher is a part of OpenStack BigTent right now. So, yeah. So let me explain how Watcher fits into the OpenStack ecosystem. So these are your core set of OpenStack services. Hope you guys can see. OpenStack services and these are your monitoring services. You have Salameter. You can use Nukey or you can use Manaskar. And then these services, they feed off all the metrics to Watcher and Watcher based on the goal and based on the strategy you have used. It does something to optimize your cloud to reach that goal using that strategy. And the really cool thing here is we also have a pluggable analysis tool. What you can do is you can feed off all these metrics from Nukey to a third-party big data analysis tool or whatever tool it is. You can, I mean, data scientists can work on, they can do their own big data analysis. They can come up with their own stuff and using those numbers, they can again feed the information to Watcher and Watcher can pick up everything from that third-party data analysis tool and it uses that data to optimize your cloud. So if you're using CollectD, which has a RDT plugin to get these silicon-level telemetry directly from your processor and that's passed to a CollectD daemon, which again passes it to Salameter. And once it is in Salameter, Watcher just takes it from there. So this is the workflow of Watcher first that monitors your cloud, I mean via Salameter and other monitoring services. And then it analyzes, it takes aggregate flows of events and does some kind of analysis and it profiles your VMs. And based on a set of cost model, objective, and your constraints of your cloud, it comes up with some optimization schemes and those schemes are presented in the form of action plans. So action plans has a set of actions which Watcher is saying these are the actions you need to do to achieve these goals. And I mean, you can automate it or you can, a cloud admin can run it manually. I mean, I'll come to that later. And I mean, if you want whatever way it is, when you're trying to apply it, you can use Watcher's Applier component to apply those set of actions. Okay. So before I actually talk about our noisy neighbor algorithm, let me, I just want to tell you that there are a few algorithms already in place to detect noisy neighbors. Like, simple threshold algorithms where a VM crosses a threshold and you radiate as a noisy neighbor. But there are a lot of disadvantages to that. Let me just explain that. So say you have the same example which I was explaining previously. Say you have your payment web service running in a VM and say that VM is hogging a lot of L3 cash, LLC. So if you use any typical algorithm which is already in place, it's going to identify that as a noisy neighbor because it's taking too much of resources and other VMs do not have enough resources. So it's going to identify that as a noisy neighbor and it's going to try to move that or kill that or stuff like that. But that is not something which you would want if that is a very important application. So we have come up with something called job priority since, I mean, not all jobs in data centers are equal anyway. So you can pass a job priority to an instance as a part of instance metadata and watch it pick it up from there. So there are a few silicon level metrics which I was talking about which you can use to detect a noisy neighbor like L3 cash, IPC. And if you want to split IPC, you can just see number of instructions executed by a VM or the number of CPU cycles VM is taking. But as a scope of this strategy, we're going to focus on L3 cash. We already are working on algorithms. In fact, we have few algorithms ready, which are using other set of parameters, but we've still not upstreamed it. A lot of testing is being done because testing this is a real pain. It's not easy to understand all those silicon stuff. So this is our algorithm. So I'm going to talk about the algorithm which we have upstreamed. It's already an OpenStack Watcher's repo and you can look at it there. But so this is a very straightforward algorithm. What we are doing is we're going through, we're looking through all nodes in your OpenStack Cloud. And we're going through all VMs per each node. And we are identifying a victim VM and an aggressor VM. So a victim VM is a VM, typically a high priority VM that is being affected by a noisy neighbor. And an aggressor VM is a noisy neighbor, which is typically a low priority VM, which is trying to mess up your other VMs, other high priority VMs. So once we identify a noisy neighbor, we look at all the remaining nodes in your cloud and we come up with, we look at nodes which do not have any priority VMs or any priority applications. And if the node has enough resources to accommodate this noisy neighbor, we migrate them to that node. And the whole thing is again automated by Watcher. I'm going to explain how this is done in the next few slides. So to identify a VM which is being affected by a noisy neighbor, which is also your victim VM. So to do this, what we do is we go to every node and every node, we try to see all VMs in the decreasing order of their priority so that we can, we first go through all high priority VMs in the beginning. And if we see any dip in L3 cache, LLC of any VM, which, if the dip is beyond the threshold, and this threshold can be passed as a command line parameter when you're setting the strategy. So when there is any dip in your LLC, which is beyond this threshold, we mark it as a potential victim VM. We're still not sure if it is actually a victim VM because we're not sure what caused the dip. We're not sure if the dip is because other VMs are trying to take up LLC at that point of time. So we just mark it as a potential victim VM. And then what we do is we start identifying a noisy neighbor in that specific node. So we go through all the VMs in reverse order of their priority, from lower priority to higher priority. And we try to look if there is any VM which tried to take too much of L3 cache in the same period of time where our potential victim VM has lost L3 cache. So if we identify a spike which is beyond the threshold and if it falls in the same time period as your priority VMs dip in LLC usage, then we know for sure that there is something wrong with this. And this is somehow affecting your priority VM. So we mark this as a noisy neighbor. And once we mark this as a noisy neighbor, we're going to go through all nodes in your cloud, see for any nodes where there are no priority VMs and see for nodes which have enough resources to accommodate your noisy neighbor. And we migrate this VM to that node. So all this sounds complicated and a little tiresome, but an admin has to do nothing. All he needs to do is run a watcher goal and strategy. And all of this is done behind the scenes. So let me talk about two modes of watcher, which is a one-shot mode and a continuous mode. So I mean, sometimes an admin can have no confidence in what watcher does. I mean, you can't dream the admin. It's a new thing. And you just can't let watcher change all your cloud maps and your entire cloud setup. So what a one-shot mode in watcher does is an admin can request this audit to be run. And once the audit is completed, once watcher comes up with your optimization schemes based on the strategy which we have set, an action plan is created. And this action plan is presented to the admin. And this action plan might have a simple action like a VM or migrate a VM, or it might have a set of complicated actions like, move this VM here, move this VM here, move this VM there. And this is how you're going to optimize your cloud. So I mean, an action plan has a set of actions. And an admin can see what each action is. I'm going to show you in the next few slides how to do that. And so this is the one-shot mode. And once an admin is fine with all the actions, he can just apply the action plan with a single command. So this is the one-shot mode I'm talking about. And this is not always viable because an admin can't always keep running this for every 1,000 seconds or one hour or whatever the time period is. So we have a continuous mode in Watcher where all you need to do is set a goal and strategy and just leave it. So what happens is after every x amount of period of time, and this x can be set by the admin, your Watcher runs an audit on your entire cloud, and it comes up with an action plan with a bunch of actions. And it's going to automatically apply this action plan. And your VMs are migrated or killed or whatever. And then after again x amount of seconds, x amount of time, this thing is again done. So identifying a noisy neighbor is not a one audit cycle task. It takes a bunch of audits, and it takes time to understand how each application or each VM is using LLC. And based on that, over a period of time, over a period of audit cycles, your node can move from a noisy state to a noisiest state. And once it is in a noisiest state, that's what we want because your priority VM gets whatever it needs. It's not going to be choked for resources. And things run fine. So this is a plot of L3 cache with time of four VMs on a node, and this was taken before running Watcher. So if you can see instance one is actually the red one, which is a high priority VM. And you can see this is the L3 cache consumption of that, which is not really good. I mean, you have instances 2, 3, and 4, which are hogging more cache. I mean, it might be completely up to the, there is a chance that all this is happening is because instance one just needs that. But since I've done these tests, I know I've run the same amount of load and all. So this is not something that's good. And the most important aspect over here is the whole noise over here. I mean, it's all noisy. Every VM is fighting for the cache it wants. And the whole graph you can see is not clear, and it's a bit noisy. So I have some screenshots of how Watcher is run in the next few slides. So that you guys have an idea how easy it is to run Watcher and how Watcher works. So you have a goal list and a strategy list. So these are some predefined set of goals, work-out balancing. If you have too many VMs, I mean, you can balance all the VMs using this strategy. You have server consolidation, thermal optimization. And these are these strategies where you can map each node, each goal. So have a thermal optimization goal, and I can map it to a thermal optimization strategy, which is this. I thought there was something wrong. But anyway, coming back to noisy neighbor. So this is the goal we need to use, and this is the strategy that we need to use. So all you need to do is create an audit template and run an audit with a goal as noisy neighbor and strategy as noisy neighbor, and an action plan is created like this. So I've run this in a one-shot mode so that I can show you how Watcher shows the action plans in a set of actions. So if you see this, 7C53, this is the UUID of the action plan that was created. And it says this is recommended by Watcher. So now the admin has an option to go through what this is. So if you see the action list, so this is the action that was created, and you can see it is mapped to this action plan, which is 7C53, which I just showed you that was created, which was the recommended one. And now if you want to see what this action is, what's the goal of running this noisy neighbor strategy? And this is how you see what Watcher has suggested. And you see over here, the destination node is 38. Migration type is live. You have a source node, which is 39, I guess. And resource ID, which is the UUID of your instance. So Watcher is saying, move this instance from this node to that node in order to reduce your noisiness in your node. So just to run the action plan, all you need to do is watch your action plan, start with the ID, and boom. Everything's done behind the scenes. And you have your host where live migration has started. And once this live migration has been completed, this was the, I mean, one of the instances out right now. And you can see this is what the ATCache graph looks like right now. And I'm not saying this is noises. There is still a lot of noise. You still do not know if these VMs are affecting this. So this is right after one cycle, because I wanted to show you how it looks after the cycle. And over a period of time, right over a bunch of cycles, it's supposed to move towards a noises state. So this is the summary side. So I've showed you how to automate noisy neighbor detection and mitigation using OpenStack Watcher. And it's all automated. And you have Intel's RDT, which is being used for this, which can directly introduce hardware hooks to monitor and allocate per thread or, I mean, allocate resources on a per thread or a per VM basis. So this is something which we are working on right now, which is going to be released soon. So right now, this strategy, all the strategy does is look at cache and then try to change your cloud map based on that. But Intel RDT also gives you a feature called cache allocation technology. So all these things are, I think, available in the latest line of Xeon family. I think that's what's used in almost all the data centers right now. So what this cache allocation tool does is, so when you can actually allocate or restrict or reduce the amount of last several cache right on spot. So say if your high priority VM is not getting enough LSE or enough of any other silicon level resources. So I mean, using Watcher, using the strategies which we are coming up with, you can on spot allocate more LSE to that VM. And then you can reduce or migrate other bunch of noisy VMs. So that we haven't upstreamed yet. We are working on it. A lot of testing is being done on this. And so one more thing which we want to investigate is how to bring in other silicon level metrics into pictures. I mean, I have LSE. That's fine. But I want to bring in IPC. I want to bring in CPU cycles. I want to bring in other stuff and put all of them into one algorithm or a bunch of algorithms based on a set of very common scenarios in Datacenter and use all of them to automate your noisy neighbor detection and, ultimately, enhance resource utilization of your cloud. So that's it. This was automating noisy neighbor detection. So if you have any questions, I know, I see someone already standing there. I've got two questions. The first question is, what version of OpenStack are you doing this with? What's the requirement? Is there a micro version specific requirement for any of this, et cetera? So for Watcher, there is no OpenStack requirement. I mean, it's there in Ocarta. I think it's there from Newton actually a while before that. So as long as you're having your latest version of OpenStack, I think this should work. But you need to have some specific libraries to get specific versions of libraries to get these silicon metrics. And if you're using LeBert, you need LeBert 3.0.0. OK, that's cool. Yeah, and here we have used Kalecd. So that's an alternative to LeBert. And Kalecd directly pushes metrics to Cilometer. I mean, yeah. OK, second question. I did more of an observation. So in my experience, I'm an operator at Charter Communications. We've almost always found it much easier to leave the noisy neighbor where it is and move its other neighbors, the nice neighbors away. It's almost impossible to migrate a noisy neighbor successfully. So bear that in mind, people in the audience. Yeah, I mean, as I've told you, this is something which we are experimenting. And this is something which actually worked for us. I mean, at least the scenarios where we've tested it. Moving a noisy neighbor away to a node is actually that worked out better for us. We are still coming up with more strategies. We're going to upstream them soon. And I mean, obviously, feedback like this is valued. So we can come up with better strategies. And in fact, Watcher has written in a way that anyone can write a new goal and a new strategy. So if you think for your data center, that's the strategy that works. Just define a goal, define a strategy. You can use it. And if you upstream it, I mean, anyone else can use that. So that's great. So I see that Watcher has the ability to either automatically take actions or to just advise what actions to take. Are there any sort of hooks that you know of, or maybe an automated way to say, I don't want it to actually do this automated action, but I want it to advise and maybe send something out to Page Your Duty to wake somebody up to look at that advise action plan? So are there like hooks to external or any way for external things to kind of hook in to be triggered off of that? Yeah, we are actually working on that right now that something which came into our discussion yesterday. So some people said they wanted to link this with Congress or a bunch of other OpenStack projects. So one drawback of that is not everyone uses every service. So if we completely move towards making Congress APIs or something, you need to have Congress in your OpenStack and not just the basic services and Watcher. So that is one drawback, but we are still working on that. And in fact, we are trying to work on how to optimize containers and how to make external calls. We're seeing if we can write some classes where you can directly add whatever external APIs you want to call and then whatever data you get, you can directly feed it off to. Is that kind of like a plug-in for architecture? Yeah, so again, this is something which we discussed yesterday. And I mean, since a lot of, I mean, it's not just you. We've got a bunch of feedback about this. So pretty soon, we are going to come up with a blueprint and then implement that. Yeah. Thank you. Any other questions? OK, sure? OK.