 Okay. Let's start. Thank you, everyone, for coming. My name is Fatai Wehib. I'm a researcher at Ericsson Research in Stockholm, Sweden. My research is, in general, about the scalability of cloud platforms. And in that context, I've done a lot of scalability measurements with respect to OpenStack. And this talk is kind of a byproduct of all those measurements that I've done before. So the measurements that I'm presenting here, some of them were done by a number of students that also work with us at Ericsson Research. So this is how I've structured the presentation. So I'm going to start talking about, you know, what kind of deployments can you have in OpenStack. And that's kind of instrumental in trying to understand how you dimension OpenStack cloud. And then I'll continue in how one can go about dimensioning compute nodes, which is kind of straightforward. And then I'll proceed into dimensioning controller nodes, which is like the kind of the main contribution of this world. And finally, I'll give a quick summary of what I went through. So as you most probably know, OpenStack is based on distributed design. What this means is that you can run the different OpenStack services in a highly distributed setting with very little restriction. And because of this, your deployment decision can range from anywhere from an all-in-one deployment to a multi-node deployment. Now when you talk about an OpenStack deployment, you generally talk about two kinds of nodes. There are the compute nodes, which are the nodes that host and run the VMs. Then you have the services that run on these compute nodes, which are, you know, like the Nova compute service or the plug-in or the C geometry agents. And then you have controller nodes that actually don't run the VMs but that run all other services that are required to support your OpenStack cloud. Now there are different kinds of deployments. One example, which is like the maybe what anyone who tried to deploy OpenStack starts with is this all-in-one deployment, where you have both controller and compute services running in the same server. So this is something that, for example, you'd get if you're running a DevStack deployment. There are actually also production deployments where you have many of such services. So instead of only having one node, you could have multiple servers. But on each server, you have different controller and compute services. But these are not very common in production environment. What's common in small-scale deployments is this 3 plus N deployment, which is also what OpenStack recommends. So in such deployments, you have one node dedicated for network, another node for storage, and a third node for the remaining OpenStack's controlling services. Then you also would need to have N compute nodes depending on how big you want your OpenStack. And then there is this general N plus M deployment where you could have N controller nodes and M compute nodes. Then of course, you'd need to have as many compute nodes and controller nodes. You have many control nodes here because then you might want to have clustering, high availability, replication, etc. And this flexibility is what actually makes the dimension of OpenStack difficult because depending on how you decide to deploy your architecture, your OpenStack, you have to actually have a specific way of dimensioning. So let's start with how you can dimension OpenStack compute nodes. So to do this, what you need is a good knowledge of the load that you expect on your cloud. And what this basically means is that you need to have what are the flavors that you expect to run in your cloud and then how many of those flavors do you expect. So in this table, for example, we show the different OpenStack flavors and the expected number of VMs that you expect to have in the cloud that you want to run. So the column on the right gives you the expected number of VMs. Now the first thing to do when dimensioning compute nodes is to try and estimate what's the total amount of resource that you need or the total amount of resource that's going to be consumed by your VMs. And to do that, you just need to multiply the number of VMs by the amount of resource for each flavor and then sum it up for all the flavors. And in this example, we can take, for example, the amount of RAM needed and then you see that we need about 281 gigabytes of RAM in your compute nodes. Now before continuing with the calculation, one is also taking into account the overcommit ratio or that one wants to use in a deployed OpenStack cloud. So this overcommit ratio tells how much resource a compute node reports the scheduler. So if you have a hardware server, for example, with 20 cores and if you allow an overcommit ratio of two, then it means that the compute node would actually report as if it has 40 CPU cores. So, of course, you'd want to do this because it allows for having a high utilization of your cloud, which makes your efficiency larger. But then, of course, this has also a drawback if your VMs actually need those resources, then they cannot get it. But then once you have the overcommit ratio in the total, then you basically choose one server type. For example, here we choose a server which has 32 gigabytes of RAM, 412 gigabytes of disk and 20 CPU cores. And then you simply divide the total by the overcommit ratio and the hardware resource available. And then you get the numbers that you see in the column before last. And then out of this, you simply choose the largest one, which is 5.73, and then round it up to the next integer. Then what this basically gives you is that to support such a workload, you need at least six compute nodes of the specified hardware profile. One thing to keep in mind here is that by default, the OpenStack scheduler, what it does is that it tries to balance the memory load on all your compute nodes. What this means is that if your cloud is, say, utilized up to 80%, then it means that on average all of your servers are also utilized to 80%. And then if you're trying to start a large instance that actually requires more than 20% of the available capacity, then you would not be able to do that because all your servers would be full. So if that's an issue that you need to take care of, then you either need to increase the number of compute nodes that you have, or you have to actually change how OpenStack does scheduling so that instead of trying to balance the load, it tries to actually fill nodes first before moving to the next one. But that's pretty much how you can dimension compute nodes. And one can also actually extend this into how one also can dimension, for example, storage nodes or network nodes. But then when you come to dimensioning controller nodes, then that's a different story. Because the load on an OpenStack controller does not really depend on the load on the VMs, but it depends on many other things. Like what's your deployment architecture? How many control nodes do you plan to have? What are the plugins you're using? And what's the software that you use for the different plugins? How do you configure your services? How many compute nodes do you have? And how many objects, OpenStack objects do you have? How many VMs? How many volumes? Networks, routers, users, etc. Do you have? And of course it also very much depends on how your users interact with your cloud. So how frequently are users starting VMs? How frequently are they creating and turning down networks? So that kind of comes into play when you're trying to dimension OpenStack cloud, especially the OpenStack controller. So because of this, it's kind of difficult to come up with a generic dimensioning guideline on how to dimension OpenStack cloud. So what people in general do is that they just over-provision the resources that they have in your controller. And if you have resources to do, if you have sufficient resources, then this is one way, just over-commit. And then if you actually want to have a tighter control on the amount of resources that you actually allocate for your OpenStack controllers, then you have to take some more steps. Then you have, at least from my experience, you have two approaches and two options to do this. So the first one is that you can actually deploy your cloud environment and then measure the resource consumption. But you deploy it in a test environment so that you can actually test different parameters before you actually deploy it in the production environment. Or the other thing is that you try to develop a model that can be used to actually estimate how much resource would be consumed, and then you can use this model as an input for your hardware dimensioning tool. So let's start with the first one, deploying the OpenStack and then doing the measurement. Now when you talk about deploying OpenStack directly in a test environment, you can do it on a physical environment. So if you actually have access to the production environment and if you don't have any load or workload running on it, then you can deploy your OpenStack directly as you would want to run it in a really working environment, and then you can load it using homemade scripts or you can use it probably to put load into OpenStack and see how the controller performs and then do measurements. And then based on that, make a design decision on what your architecture is going to look like and then how much resource you want to reserve and allocate to your controller. The problem here is that, of course, typically you don't have access to your production environment, so you'd most probably have a smaller test environment and then you need to make decisions about how much resource you need to allocate on your production environment based on the results that you get out of this test environment. So in this case what you could do is that you can actually work with an emulated environment. So instead of actually deploying the OpenStack entirely on hardware, you could actually deploy OpenStack in a virtualized environment and then do your measurement there. This turns out to be actually moderately accurate. It's not as accurate as you'd get if you're doing it entirely in a physical environment, but it's actually very flexible. So even if you have a small setup, you'd be able to test how your controller is going to act even if you have a large system. And of course, another approach is to use simulated environment. Of course, the problem there is that there are no good simulators for OpenStack today, and what we're trying to do in the second step of this, in the second option, is to actually try to come up with a simulator kind of a formula that basically takes in the state of the cloud and then tries to estimate how much of a source it could be consumed. On the plus side, this is an extremely flexible way, so it doesn't take a lot of time to actually crunch a bunch of numbers and then get an estimate. But the problem there is that it's not as accurate as you are actually doing direct measurement. So if you're using an emulated cloud environment, then what you could do is that you could test your entire cloud in an OpenStack in OpenStack deployment. So for example, you have one test environment that runs OpenStack, and inside this OpenStack, you have a bunch of VMs, and inside these VMs, you could run your controller nodes and you could also run your controller nodes, and then you could actually look at how much resource is being consumed on the controller node, and then based on that, you can tune your configuration, how much resource is allocated to the controller node, and et cetera. You can take this a step further, and you could actually run the controller directly on a physical server, on the one, on the type where you expect to run your controller in the physical and production environment, but then run the compute and maybe even the service nodes inside the virtual machine. And this gives you better measurement and better accuracy in terms of how much resource would be consumed if things were running in a physical environment. Now, of course, when you run OpenStack inside OpenStack, then you can't really have real world workloads in there because you don't have sufficient resources, you don't have memory, you don't have physical resources or CPU resources as you would expect when you're running the cloud in a production environment. So because of that, you want to use a small and simple image to test the OpenStack. So one image that you could use is the Syros image, which gives you a fully functional Linux deployment which can run inside 48 megabytes of RAM. It can even run in a smaller amount of RAM. So that's a very small Linux distribution, or a Linux image. If Syros is too much, you can actually even boot an IPX image. It's an image that simply tries to boot from the network. And if you haven't configured any network booting, then it would simply check whether the network works or not, but then it doesn't do anything. You can even try to boot an empty image. Then, of course, when you try to boot an empty image, nothing would happen inside the VM. But from the point of view of the controller and the compute node, there is a VM running, and it is allocated the resources that it is supposed to consume. And as I said, of course, so you have the image, and then you actually would need to stress the system using, for example, Rally, or you could also write your own custom scripts as we did in our experiments. So we did, for example, one such deployment, and we did some measurements for a deployment where we have both controller and compute nodes inside OpenStack. So that's a snapshot from the Corezen dashboard. And you see that there are two nodes in the middle there, which are the controller nodes. One is a node that only runs network and storage, and you have a bunch of compute nodes. Maybe the interesting thing there is that we have a 2N plus 1 deployment, as I said. One storage and network node plus one controller node. And also we use a very small compute node. So our compute node is a 1 core, 1 gigabyte of RAM node, VM with 6 gigabyte of disk. And with this, we are able to actually deploy up to 100, 200 compute nodes and then test how the controller would perform in such a large scale. On a moderately small cloud setup, which had about 10 compute nodes. And then with such a setup, you can get these kinds of measurements. For example, the measurement that I'm showing here is that for our OpenStack deployment, where you have about 125 compute nodes and 800 VMs, where there is no interaction from the user side, how much resource is consumed. And basically what we see here is that, for example, the controller consumes the resources that you see there. So the average CPU utilization, for example, is at 171%, which basically means when the system is idle, meaning that there are no API calls to the controller, it consumes around one end or less than two CPU cores. And then you see that the amount of RAM consumed is around 5.6 gigabytes. And then you have information about also all other resources. And this approach actually allows you also to dig in deeper, so you could look at, okay, where does the CPU utilization come from? And then you see in the graph on the right that, for example, the process that consumes the maximum amount of resource from your controller is the noble conductor. It takes about half a CPU core, followed by a rabbit MQ. And then you have a bunch of other services whose CPU utilization is not that big. And of course, you can now change your deployments, I think, and then maybe you could actually look into deploying it with, for example, an additional controller node or changing the available resources that you have in there, and change the start and stop rates so that to see how the system would perform when you have users interacting with it. Then you can change things and then use this information to actually tune the amount of resources that you would need if you had a production environment of this sort. So this is the first approach. Now, if there are a number of unknowns, which means that, for example, if you're not sure how many VMs you expect or if you're not sure how frequently users are going to interact with the cloud, then it's kind of difficult to try and evaluate all the possible combinations of the different settings. So in that case, what you could do is that you try to develop a model for the resource consumption and then use this model as a simulator. So you can use it to predict what the resource consumption would be at different parameters. So in general, what this model should do is that it should basically be a function that can map the state of the cloud into the amount of resources that's consumed. So what you mean by the state of the cloud could be how much physical resource you have. It could be the number of computer nodes, storage nodes, hardware routers. If their type plays a role, then that should also be a function in here. Also, what objects do you have in the system? Like, for example, how many VMs do you have in there? How many volumes, routers, networks, tenants, et cetera? And also, how does the end-user interaction pattern look like? So how frequently do users start and stop VMs? Or how frequently do they attach or create a volume? So these things should be there as part of the state of the cloud. And then the output should be, you know, how much CPU is consumed? RAM, disk, iOS, storage, et cetera. And of course, this model, in general, when you're trying to model a real system using a formula, it's always difficult. And then you have to always simplify as much as possible. And I'll show you how we did that. So this is the model that we chose to use. So what this model says is that the resource consumption for resource R could be CPU or it could be RAM or whatever. It's given by the resource consumption, while the system is idle, idle meaning that, you know, there's no API call going through the system. Plus, the resource consumed as a result of the API is being called. So this is how the model looks like. And then, of course, you have to now refine this model a bit. You have to first choose, you know, what is R? What are the resources that you want to take into account? And then we said, okay, let's look at CPU, because the CPU is the resource that actually is exhausted faster. Then you have to choose, you know, what are the state variables that you want to take into account. Then we said, okay, so the number of VMs and the number of compute nodes seem to be interesting. So we take those into account. Then we also want to also take into account, you know, open-stack APIs that end users would make a call to in this model. And then the data at which, of course, they are being called. And then, of course, in order to make this model complete, you have to have an estimation of, you know, how much this idle resource utilization is and how much resource utilization is per API. Now, the first thing that we did was, okay, how can we, which API course do we need to take into account? Because open-stack has, you know, maybe hundreds or thousands of APIs that you can make a call to, right? And then you can't really put everything in there, because then your model would be very complicated. And actually even profiling each individual model, each individual API call, it takes time. So the first thing that we did was, okay, we tried to identify what are the API course that we want to take into account in our model. And then for that, what we did was, okay, we tried to profile, you know, a set of APIs and then tried to see, you know, for a case where you have, okay, 125 computers and 800 VMs, how much CPU time does it take to actually do these individual things? For example, how much CPU time does it take to create a VM or to create a volume or to delete a VM, et cetera? Then we found this. So what this basically tells us is that, you know, if there is a lot of call to an API and if it takes a considerable amount of resource, then you should take that API call into account. So taking this into account, we said, okay, we say, we see creating a VM takes a lot of resource and then deleting a VM takes a lot of resource. So we'll take this to as the only two API calls that we want to take into account. Of course, it's kind of important to choose which API calls one needs to take into account because this makes sense for our model, but maybe if you actually use another kind of load generator, for example, then you need to take into account other models, as I would mention later on. But then for all of us, we said, okay, we would choose these two API calls. Then the next step is rather simple. So you just need to run as much experiments as possible for different values of number of VMs and the number of compute nodes. Then you want to measure, you know, how much the resource utilization is changed with different values of efficiency. Then also do how much each API call of interest consumes when you make a call of that sort. And then finally, when you have assumption numbers, data points, then you can do a curve fitting using a number crunching library of your interest. And then basically get a function that can be used to estimate the resource consumption. So we actually did a measurement in our deployment for a specific kind of cloud configuration. We said two plus end configuration. And then these are the results that we found. For example, on the left, we see how much resource is being consumed when the system is idle, so when no one is making any API call. Then what you see is that, for example, as expected, this number increases when the number of VMs increases and also when the number of compute nodes increases. And on the right, we see how much CPU resource is, how much CPU time is consumed when you actually start a VM. And you see, and then there is some interesting data there. For example, what you see here is that to stop a VM, the amount of resource that you consume when you try to stop a VM doesn't really change much with the number of VMs and the number of compute nodes that you have. But the amount of CPU time that you consume when you actually start a VM, it increases both with the number of compute nodes and the number of VMs. And then, of course, you can do curve fitting, so you have a bunch of points which you can do curve fitting, and then you get the formula that you see there. And now you can plug in these things into the expression that I showed you before, and then you see this scary-looking formula. But it's not actually a very complex formula, so it's a function of C, V, and the lambda. And you only have one lambda here because in a stable system, the rate at which you start a VM has to be the same as the rate at which you stop a VM. Otherwise, you have constantly increasing VMs or the number of VMs is going to die out. So it has to be one. And then you get this function. This function now can be your so-called simulation environment. Then you can use this function to answer a number of questions. For example, you can ask this question, how many CPU cores are needed for the controller of a cloud that has 70 compute nodes and hosts on average some 300 VMs, and the VMs have an average lifetime of an hour. Then this formula says, okay, you need about 1.44 CPU cores. So if your cloud environment has the properties that you specified, then a cloud controller with only two CPU cores should be able to handle this workload. Or you can also do the reverse and then say, okay, how many compute nodes can a four-core controller handle if we expect to have 10 VMs per compute node and an average VM lifetime of an hour? Then you can plug in the different numbers in the formula above, and then you'd get 116. Now, it's important to note that these numbers are much smaller than what you'd expect from a real production environment, because for our evaluation, we're making this assumption that when you start a VM, you only make one API call. But if you are actually using, for example, Novaboot to start a VM, then it actually makes 11 API calls, to Keystone, to Neutron, to Glance, et cetera. So because of that, actually this number is going to be much larger than what you see here. Or this number is going to be much smaller than what you see there. So what that basically means is that if you want to actually model a system or model a system where actually when VMs are started with Novaboot, then maybe you either need to actually change the number of type and the set of APIs that are being called, or you could actually change the model slightly and then say, you know, basically what you want to do is take the amount, the rate at which a VM is started and the entire amount of resource that is consumed when you use Novaboot to actually start a VM. Yeah, and then this is kind of very flexible because now you can plug in different values of C and V, and then you can get the amount of resource that you need. And then you can even make this a bit more complex and then you can, for example, do the same for a specific resource, a specific service. So you can take, for example, RabbitMQ and then develop such a model with it and then see, you know, at which point do you need to actually cluster to be able to support more load than what RabbitMQ can support. Okay, so to summarize, I was talking about how to dimension an OpenStack cloud. It said, okay, an OpenStack deployment has compute nodes and a set of controller nodes. Dimensioning compute nodes is rather straightforward as long as you have a good feeling of how many VMs of what flavor that you want to run. Dimensioning controller nodes is not straightforward, so you have three options. One is to overprovision, or the second is to do direct measurement, deploy it in some test environment and then do measurements. Or third is to do try and develop a model and then use this model to, you know, simulate the cloud and then get an estimate of their source consumption. So that was my presentation. Thank you for your attention. So if you have any questions. CPU is 16. Why did you pick two? Are you using the perforating or you are not using the perforating? So that's kind of a random number, sometimes I'm not using it specifically. But of course, I work for Ericsson, and Ericsson, it's more important to make sure that you deliver the resource that you claim you're delivering. And the issue there is that when you have a larger amount of CPU overcommitation, what it means is that if more VMs actually need those CPU resources, then they are not going to get it, right? So maybe 16 is not actually what you want to have if you want to guarantee services, right? And yes, you're using a perforating. So when I say CPU course, it's actually taking into account. What storage are you using? Storage we're using, it's safe. For the evaluation, we're using safe, yes. Got the questions? Okay, thank you.