 Good morning. Welcome to the OpenStack Summit. My name is Federico Lugifredi. I'm a product management director in the storage business unit at Red Hat and I'm joined by two of my colleagues, Sean Cohen and Sebastian Han. One is a manager in the OpenStack team that has been managed in storage for longer that I've known storage existed and Sebastian has straddled both groups as a principal architect. So today we're going to cover very quickly very quickly because it's a packed session how quality of service works in the public cloud, in OpenStack with SEF and then how it all comes together as a single thing. It's going to be a fairly in-depth analysis. So let's get rolling before we are already late. So the first building block is how does quality of service work in a public cloud environment? Why do we care about this? If you have been following our OpenStack and given the venue, I assume that you have been. You will have noticed that OpenStack is becoming a de facto standard in the telco environment. Now every telco workload has a database dimension to it and quality of service is an essential building block to deploying a database and therefore that's why we're here. There is an additional dimension which is capacity planning and cloud environment users expect you to add capacity to the cloud before it's needed. It's not the old days of IT where you can make them wait four months. They expect it to be there yesterday and capacity planning intersects with quality of service in a remarkable way and that's why we're going to look at it from that angle. Further, public cloud has established its model for capacity-based quality of service as a de facto standard and so we want OpenStack to be able to offer the same mechanism. Users expect it. That's what they learned about in public cloud and for those of you wondering what the movie poster is about towards the end of the second Tron movie, though as the hero is about to be defeated, he has a quote that says I fight for the user. Then he falls in the sea of simulation and is decompiled and things go very bad, but for that one second he has the right thought. Now, what's the problem statement? Some workloads need deterministic storage performance. Most typical enterprise workloads that require this are databases. Jitter sensitive workloads like video streaming are another common example. These are workloads that depend on constant latency measures to perform adequately. Variants in latency is called jitter. I expect most of you are familiar with this, but just in case. A second class of problem that QoS solves is isolating workloads from the so-called noisy neighbor. In a virtual environment, this occurs when there is contention for a shared resource. In this case, our shared storage cluster. Contention starves one of the users of the needed resource and therefore it slows down, hence jitter again. Finally, operators need to run capacity planning exercises. If you want 90s flashback, the original title for this session had a very James Carville tune to it, which was that it's all about capacity planning. Stupid. But you can actually build QoS into systems that don't have QoS just by doing appropriate capacity planning. So maybe the James Carville analysis is not that far off the mark. There are two options when it comes to virtual disk storage for your instances. Instance storage, also known as ephemeral, and persistent volumes. Once attached, these virtual disks can be formatted with your operating systems tooling and behave exactly like a standard physical disk would. Key differences stem from the fact that instant storage is physically part of the host that runs your instance, whereas persistent storage is remotely connected over the network. Instant storage offers larger volume sizes and may hold the edge in absolute performance. Certainly, it is the case with the largest public cloud vendors because of its proximity to the host. But the storage type is also lost once the instance is terminated. Hence, it's a moniker of ephemeral. Permanent persistent storage is much more featureful. In particular, it offers facilities like snapshots that dominate provisioning and backup strategies in cloud infrastructure. Now, multiple volumes of either type can be attached to an instance and you can even mix types. Ephemeral volumes can be attached only at boot time, however, while persistent storage can be attached while the instance is already running. And you can add them or remove them at your heart's content. The user can do whatever they desire. The Block Device Mapping Facility in AWS allows you to build wonderfully complicated disk topologies if that's what you need and associate them with the instance at boot time. It's even possible to attach multiple volumes to an instance and build software RAID over them. Please don't do that. As a SAF guy, I have a dim view of RAID, as you may imagine. But in the public cloud, really, you need a written note from your mom giving you permission to do RAID. And if the Y is not clear, I'm happy to take questions on that layer. There are some cases where it does make sense, but not the usual reasons why you would use RAID and it's certainly not the mainstream case. Now, in June 2012, AWS began offering SSDs as a higher-performance alternative to magnetic storage. And over time introduced multiple options with different performance levels and cost. Some instance types now include an SSD-backed instance store to deliver very high random IO performance. With types like I2 and R3 being the first two that have supported trim extensions. Instance types themselves have evolved to include high IO instances. These are the I2 type aimed at delivering high IOPS from up to eight local SSD drives. While then storage instances type D2 offered the lowest price per disk throughput in EC2 and balanced cost and performance using 24 magnetic drives. In other words, there are so many options that we can't possibly go over them. So let's focus on the essentials. Oh, and by the way, some of them have burstable IOPS just to add another dimension to the problem. So let's focus on the essentials of the AWS model. Persistent volumes are what we're interested in. And persistent storage in Amazon AWS is delivered by the EBS, Elastic Block Storage Service. You can have EBS-backed instances, meaning the EBS volume is your boot volume. And EBS can be SSD or magnetic as we have. And you can add volumes at runtime. You cannot do that with ephemeral. Critically, you can choose the volume type. How many of those volumes do you have? What IO characteristics do you want? And very important, as of 2017, you can now resize live volumes. The why this is important will become apparent in a second. Now, to complete the bits of Amazon that we need to understand for this to make sense, there are two more components. CloudWatch, which is the monitoring service for IO provisioned volumes. CloudWatch delivers automatically a set of metrics over latency, bandwidth, and IOPS that are being delivered by the volume, both read and write. And there are a few more, but these are essentially the interesting ones. And CloudWatch can fire events based on the level of these metrics. The other aspect that's important is CloudFormation. CloudFormation is an automation platform which has its own syntax. Fairly JSON-y, where you script what happens on given events, and you can provision and design your cloud infrastructure using it. Now, the example here is what happens when a volume reaches 90% full or 85% full. Whatever threshold you set, that is where your operators start getting nervous. You can set up CloudWatch to fire an event when the volume is 85% full that will then trigger a CloudFormation script that you defined that carries out a resizing of the EBS volume to a larger size. Then it will run some logic that you had to write that will stretch the file system over this volume so that the new size becomes visible to the operating system. And you're done. You are no longer running out of space. And this happened automatically with live volumes as you were reading the pager message that said you're low in space. So that is a pretty cool thing, but there are more things that you can do with this platform in the area of capacity planning, which is why we're interested. Now, here is an example of a volume in EBS. General purpose SSD. These are EBS volumes, SSD backed GP2 type. They are recommended by Amazon for boot volumes for low latency applications like bursty databases that are not always running at load. While it is interesting, the key point here is that IOPS performance of the volume is directly tied to its size, the third line in the slide. Every gigabyte adds three IOPS to the volume. If you want to provision IOPS, all you do is change the size of the volume. This is a general property across EBS volumes. With storage space comes IO performance. You can attach additional twists like capacity bursts that some volume types have, or EBS optimized DC2 instances, which have an additional dedicated network pipe, and other 125 megabits, I believe, that are dedicated to storage as opposed to the other general purpose channel that accesses all other services and the network as a whole. But when it comes back to the single combined dimension of space and provisioned IOPS, that is the core of the QoS model in Amazon. With elastic volumes, which were introduced this year, one can reprovision the size, and there is a result, the provisioned IOPS, of a live volume with a running system attached to it. You can also change the volume type, altering the formula controlling the behavior of how many IOPS you get per gig. And there are some limits to what is allowed here, like GP2 type and SC1 type can become all other types, while IO1 and ST1 can only walk to the right in that list, so IO1 can become ST1 or SC1, ST1 can only become SC1. There are some other restrictions. You can expand the volume, but you cannot shrink it. You have to take the volume offline to do a shrink. Okay, so there are some limits, but this is still pretty awesome. As of two weeks ago, now you can use CloudWatch to trigger a no halt update via cloud formation for the size, the type and the provisioned IOPS. So what I just described before, you can do again, saying this volume is running at 90% of its provisioned IOPS. It's about to get slow for an event that is going to resize the volume, add more provisioned IOPS to it. That will happen transparently. This is not scale out type of reprovisioning where you kill an instance and you bring another one in. No, you're changing the existing instance. For comparison, we have a bit of Google. Persistent disks on Google work pretty much in the same way. Google Compute Engine does not have as many options as Amazon AWS, but what it does have is always excellent. Performance is predictable and scales linearly with size until it reaches instance or volume limits. Google is using the same capacity-based model AWS pioneered. When you need more disk space or better performance, simply resize your disk to add more storage space, throughput and IOPS. Volume limits here are determined by two things. There are per volume limits in size and therefore also IO characteristics that depend on the technology that is backing the volume and there are instance limits and these are defined by the network and the network egress caps on your instance and these instance limits are shared if you're using more than one volume. They apply to volumes collectively. As you can imagine, there are differences between Amazon and Google and we're not detailing all of them here, but there are some interesting bits. For example, the instance limit changes when you have a ton of virtual CPUs. For example, if you have more than 32 virtual CPUs, the instance limit is significantly higher and I believe that there is another such step around 60 or 64 virtual CPUs, which makes sense because your SSD-based capacity needs CPU to be used. Now, why do we care? Because this is how the public cloud vendors do QoS. It's simple, effective, it's understood by the users. The admin scales in a single dimension doesn't care which the user is provisioning for. More gigabytes, more IOPS, it's just more. It flattens two different scaling factors into a single dimension. The operator needs to add capacity to add IOPS anyway. That's how the technology works and here users are coming to expect this QoS format. It's what they learn about in the public cloud. So to my closing point, we can build you a cloud like the big boys do. OpenStack Cinder will be able to set QoS limits based on provisioned capacity and I'll let my colleagues show you how we're making this possible in the OpenStack community. All right, so we see how the big boys are doing it. And just a quick raise of hands. How many of you are using, not just OpenStack, use Amazon today, use Google today? All right, 25%. You're already used to actually do things this way. You're already managing your capacity, you're already managing your quality of service for storage this way. So here we are today in OpenStack Summit in Boston, which is my hometown. And we're finally discussing multi-cloud, right? So today's talk is not just about OpenStack. As you see, we're dedicated third of our talk just to the public cloud and the reason is we're really looking at multi-cloud use case and operation and now we actually set standards and not just do it on our own on OpenStack or any other public provider. So with that, let's do a quick why we're here. So Federico already asked the questions of why, right? And we did mention some of the databases. And what we've seen is not just the telcos, it's the enterprises. Every application framework in OpenStack use a database in some manner, right? And it doesn't matter if it's the Rails or Wisa, which runs like Microsoft SQL Server, means that use MongoDB, LAMP, right? So in fact, every Java framework use persistent database. And these databases, and you can see the growth from over year and year, are not standard sized. They need different quality, right? Some of them are high performance workloads. Some of them tend to burst. How do we provide them all the relevant quality of service that they need? And we look at the trends in Cinder. So this is the latest user survey from OpenStack Foundation from last month, right? So first of all, we continue to see that SEF continues to lead the charge with 50% of all OpenStack in deployment's production, right? This is huge. Just to compare LVM, which is the red, is the default, right? So a lot of people are basically changing the default moving to another open source, which is SEF. And just if you want to compare it to the propriety storage, so it drains all the way from 1% to 10, right? In terms of what we're seeing in production. How can we offer the same class of service like the big boys with OpenStack Cinder, right? With all the vendor choices we have today, including SEF, and how we do it specifically with SEF. But before we go there, let's do a quick history lesson. Our journey with quality of service in Cinder started already back in the Grisly days, something like four years ago. And later, that was in Grisly, if there was any contributor for Grisly in the room. Later in Havana, we introduced the Cinder and the Nova Quality of Service support, the front end and back end, that I'm going to talk about more internally. Around ISIS, we finally get to a lot of backfixes and stable API for quality of service. And we started to see a lot of adoption from all of the vendors, right? The storage vendors, the writing drivers, making use of the new Quality of Service APIs. Later in June, we added the support for the Quality of Service, already in the Horizon dashboard. And we basically are able today to have volume types, classes of different storage, if you will, storage tier, that maps to them, so the administrator can go and configure the static Quality of Service volume per type. This is how it looks in Horizon, which is pretty simple, right? We have the volume types, we have Quality of Service specs. Where Quality of Service specs are used to apply performance settings. You can limit read-write operations on disk. Obviously, you need to determine the Quality of Service specs based on your supported back end. And each back end basically has a unique way of exposing Quality of Service, so it's not unified or generic. Rather, the Quality of Service specs are actually associated with the volume types as we showcase here, but during the volume creation, I have to specify the volume type with the relevant performance setting that are mapped to the Quality of Service specs that are exposed in each one of the unique drivers. And the way Quality of Service works today in Cinder, we have a front end. This is pretty much how we apply the policies from the compute from Nova, right? And you can limit by throughput, so you have a total bytes per second, read per second. We're basically limiting from the instance if you want level, right? And then we have the backup specific where each vendor can actually showcase and utilize some of this new capabilities. Keep in mind, a lot of the back ends support different Quality of Service already at the back end level with different performance tiers. So in a sense, as an OpenStack operator, actually you can select a relevant vendor which brings me the specific Quality of Service class that I need, all right, and expose it to a volume type. And as Federica mentioned, different types, different storage types, different volume types to one workload, right? So you can have the database data on one volume tier, if you will, and the log on a different one, right? You actually build for a specific workload today a Quality of Service. So when we say volume types, from a workload to a space, it's maybe many volume types, right, with different characteristics based on the need. Now, if you look at the secret, right, each one of these vendors have adding unique keys because this is where you can actually add extra specs to make use of your unique features and their array level. So, for example, HP FreePower, using added latency and priority, SolidFire had the mocks and bursts, right? You saw the bursts were in Amazon, right? So both SolidFire has that capability today. NetApp is using Quality of Service policy groups for extra specs, Iwawi priority, each one of them basically exposed a different set of specs. So, here's just an example, right? So we typically expose, this is like main IOPS, max IOPS, scale means fail max, scale burst, the amount of scale of the burst IOPS of every gigabyte additional volume size. We have all these capabilities already today, right, in the code that we have today. So we're very close to what the big boys are doing already because we are able to provision the right storage performance to our volume types. But, going back to the first line here, Quality of Service values in Cinder are currently able to set in static values, right? Keep that in mind. Some examples, SolidFire driver, right? So, in Okada, this is the latest things you can do with the latest code, right? SolidFire, Okada recognized four different Quality of Service specs to a live specify setting that can scale the size of the volume. I give the example with the scale means, scale max, very similar to what AWS performed today, very similar to what Google offers today. Scale IO in Okada, Quality of Service example, max IOPS per gigabyte, right? We already got to a point where we can actually offer very, very unique max IOPS per gigabyte, max bandwidth per gigabyte, right? So we're limited, we'll be calculated by the specified volume multiplied by the volume size. This is where we are today. In Okada, really, all right. So Quality of Service, just to summarize where we are in today's world of Block Storage and OpenStack is we are able to provision and set up this unique characteristic specs, but they're all static values, right? It's not dynamic in any fashion and it's one dimension, right? What if there's a way to derive Quality of Service limits values based on the volume capabilities, capacities, rather than just the static values? And allow me to introduce our first capacity-derived IOPS. This is a new API available in the Pike release, which is right now developing. It's a new mechanism to provision IOPS. Provolium bases with the IOPS value adjusted based on the volume size, IOPS per gigabytes, allowing basically operators to cap usage of their system and define these limits based on the space usage as well as the throughput. So you can charge your customers not just how they exceed the limits, right? The max, whatever, value set up, but actually capacity as well. Just like the big guys, we're taking several dimensions and we're flattening into one. So you need more throughput, you need more IOPS, and your volume, right? We're able to provide the same thing. Important question. Would it work just for self? No, it works for QM, it works for LVM, it works for Solidify, it works for any other backend. So in a sense, you can still use your existing extra specs and all the things that we achieved all the way now by adding this new capability into the APIs. So we can actually set new standards of Quality of Service. So our goal can be, say, 1,000 gigs at 10,000 IOPS per gigabyte, right? So we're combining the points. So how does the code API looks like? It's very elegant. So in one line, you can do read IOPS second per gigabyte. So we're adding basically capacity with IOPS. So as an example, here we have on the left side read IOPS per gigabyte and write IOPS per gigabyte. So we can start with a two gigs volume at 20 IOPS, 20K IOPS, right? And then as you move to a five gig volume, you're basically going already to 50K IOPS, right? So you're getting more capacity and you're getting more performance in one API. So with that, we basically showcased how we can scale not just linearly our storage and performance, but what are the scale consideration when we should have in mind we're actually delivering quality of service in OpenStack and how can we actually use this new capacity-driven API to help with that as well as what can we do in the future of CEPH and OpenSec in general to get more closer to what the big boys are doing. And with that, I'm going to hand it over to Sebastian. Thank you. Thank you. Okay, so now we're going to dive into storage QoS and basically what we call the universal scalability model which basically gives us a proper definition of what scalability is and also provides a model, kind of a framework to help us understanding and getting a better visualization of the scalability of our platform and also how to improve it. So if we had to take a simple example, let's say you're a client, so you get one VM and you ask for one block device and then the cloud provider just kept the volume at 100 IOPS. So you ask for one volume and then you get 100 IOPS and then what about if you want more performance then you will ask for another block device and then if you need more then you will keep on asking more and more volumes. So in the end what you should be or what you expect is to get this really linear scale where if you ask for three volumes and then you get 300 IOPS. But this really boils down to the efficiency of your own system, so of the front end first and the back end because remember the things we implemented are at the front end level. So it's the hypervisor that is responsible for doing all this throttling for all the IOPS and also for the throughput that you're asking for. But in real life you won't necessarily get this really nice linear scale mostly because the system is not as efficient as you might think it is and also because you don't also have the right and proper understanding of what the system is doing when we are asking for submitting an IOPS basically. And it's really simple to determine this. If you run two benchmarks you won't likely get two times in a row the exact same results. And by benchmarks I mean you don't really need to have tons of machines and even a distributed system. You can easily do a DD on your own machine and then you won't get twice in a row the exact same result. And this is easy to understand because basically while you're performing this operation then the system is doing something and the second time you're asking for the same operation then the system is also doing something else. This is why you're not always getting this really linear scale. Instead what you get is this linear line which is not so bad but is a little bit more realistic in real life scenarios. This is basically what you would get. And this is explained by two main things. The first one is the contention. So basically the contention happens when you get more clients and some of the work cannot be paralyzed so the ultimate speedup is really limited. The currency delay is the other part where we are limited by the system itself. So the things I explained earlier the incurrency introduced by the system itself because the system is not only doing what you're asking it to do but it's also doing other things. The kernel is doing other things. So this is the more worker you add then actually the less you get in this case. But this is perfectly normal and this is just real life. This is not something that should be... Oh, okay, this is... It's not trending really well but basically what you should see is that you have a linear scale and then it goes down and the height of each volume is different. And because the unit of height for volume is the number of IOPS that you get but then the more client you add then the less performance you will get because at some point then over here you should start seeing some kind of a plateau. So you're plateauing in terms of performance because your back end can only achieve and sustain a certain number of IOPS and once you reach that point then you don't get less performance but let's say you can achieve 10,000 IOPS from your back end and then you ask for way too more block devices and even if they are capped for 100 IOPS then the back end won't be able to sustain that. And this can be easily explained mainly because of the structure of a disk. If we use disk-based clusters then disks they have to seek and even when you run a benchmark on a single drive you will never ever get this linear scale. The beginning of the device is always faster and then once you go on the edge then it slows down. So this makes really perfect sense in our scenario. So basically we get less performance for each volume in the end but the overall performance is always attained. For SSD-based clusters it's a little bit different because they don't really have to seek so not like hard drive disks they don't seek so you can really achieve on a per disk level a purely linear scale. On the opposite they... So we have less content we still have the contention of course when we reach the plateau of the performance now it's rendered nicely over here. So once you reach the top of what's the platform is capable of then obviously you will start getting less performance but you won't have this degradation effect because you ask for more volumes for with SSDs. On the opposite side the limiting factor for always these becomes the CPU so the more threads you get the more IOPS you're asking for then the more CPU constructions you're going to get from the system. So then if you had to scale what kind of scale should you choose and how should you scale properly depending on what kind of dimension you want to use if you decide to go for IOPS limits then you don't do any QoS based on the capacity of the volume then you just apply a certain QoS of a certain volume type within Cinder and what you should go for is just to get faster media in order to be able to just sustain the IOPS that the clients are asking for. On the opposite if you're aiming for volume coders then you should be just adding more and more block devices or more and more storage nodes and actually the second one is a little bit easier obviously about capacity planning even if we don't always know how many clients are we going to have and what the size will be of the volume then if we do some kind of forecasting for capacity planning then we should be able to get the right number of machines and the right number of disks obviously the latest one is a little bit less relevant for SSDs because we don't have the performance degradation all the time because of the stick of the devices in the end when you mix both scenarios, volume coders and IOPS limit you should get something like this and you even in so it's a little bit of a mix and match obviously because you have clients asking for bigger volumes so the bigger the volume is the more performance they should get or some of them just want to have maybe tiny volumes but a lot of IOPS but they just don't care about the space usage so you get this mix and match visualization and just to be sure that this is not only a theory then we also run an experiment on an open stack platform backed by a set cluster we so we ask for more and more volumes so what we can see at the very bottom is we have all the volumes and we have the aggregated IOPS and IOPS per volume so IOPS per volume obviously is at an hundred IOPS IOPS and as we can see here we really have this sub-linear scale the one we are expecting to get obviously so non-linear scale but sub-linear scale and we also see that we get the number of IOPS we asked for but at some point we are reaching that point over here the cross over there is basically because we reached the maximum capacity the maximum performance of the cluster so at this point the cluster was capable of achieving around 40 times Android IOPS and then once we reach that point we just see that the IOPS per volume start degrading so this is just a real life example and then we also run the exact same experiment on a hyper-converged infrastructure where we just it's we mainly got the same results it's just that we've been able to create more volumes when we run the hyper-converged infrastructure one of the good things is that on a distributed system theoretically we can we can get a local heat because when you get a volume then with SAF for example you have N numbers of replicas it's kind of responsible for deciding what the object should be located so if you if you collocate storage and compute on the same machine then there is a chance that you will hit the machine you're running on and then the replicas will follow on the other ones so it's just so if you want to summarize what you should be doing if you're looking at building a SAF cluster or backing up an open stack environment with the SAF cluster that we that we are providing here basically we went with OSGs with just standard 7.0 to our KRPM drives we've read back cache enabled on the drive and also on the controller we configured with REST 0 all the drives and they were backed by SSD journals we really have to do SSD journals because then we can kind of the impact of the seek of the devices but we will still get it obviously it's just to get faster acts from the clients one rule of thumb that we and it's because we are using SSD journals to just to store the SAF journal we can fairly assume that the rule is one volume per OSD and this is 100 IOPS per volume for flash as explained it's a little bit different because we can get more IOPS but then on the opposite it's more a measurement which is CPU related instead of disk related so you should be calculating calculating this with according to the gigahertz of your own CPU so now let's discuss a little bit more about the future where we are heading in terms of QOS basically so this is kind of a summarizing what we currently have as part of OpenStack is we already discussed that with Sean and Federico but we can already achieve fixed IOPS per volume this has been just released for Pyke then the second one scale IOPS oh no it's the other way around actually we could do that already for a pretty long time applying rate limiting on a volume type no matter which size of the volume but the latest use case is bursting IOPS so burst IOPS and scale IOPS we cannot really do this at the moment just because we don't have any QOS implemented on the back end we can easily do some throttling at the hypervisor level but there is no way because the hypervisor is not the storage entity there is no way to store IOPS or burst so in this case the only way to implement this will be to have QOS at the back end level so in SAF ideally we will use SSDs for this obviously with all the advantages we just discussed with SSDs so it's a kind of a reservation style system where you are able to store and calculate the number of IOPS that you can reserve for any clients and it's you're just basically counting for IOPS that haven't been consumed and they can be asked for at the time regarding SAF and how we might be implementing this I would like to really insist in the fact that it's really complex to implement QOS in a distributed system but on SAF one of the way it's not set in stones yet but we currently are actually exploring the DM clock algorithm just to achieve the QOS on the back end we just learned the first patch was just introducing a library into the SAF code like two weeks ago and our primary focus will be on the cluster activities instead of client ones in SAF there is client activities so clients are running into the cluster but there is also back end activity we are replicating data we are backfilling data we are recovering data so SAF is really self healing all the time and there is a lot of moving parts on the back end of the cluster too so what we are currently focusing on is to achieve some kind of QOS depending on what kind of IOPS we are doing on the back end of the cluster if it's recovering, if it's backfilling, if it's replication then we could apply a certain QOS with that regard but then here is the challenge SAF is a distributed system and I would say it's fairly easy there are a lot of vendors claiming to, well not claiming but having support for QOS but usually it's easier to do because they have one big array so it's just a really big and monolithic system so there is no distributed fashion involved whereas when you do distributed systems it's definitely really hard because this really means you have some kind of consensus between all the entities of the cluster, this is something that we already have but this is also as part of the QOS, this is something we will have to implement things like knowing what the hardware is capable of making sure that there is no degradation over time, we all know that there are degradation over time so it's really complex to implement and this is basically some kind of a research project at the moment so if anyone looking for a PhD or just a research project I believe you can easily go that path because it's a huge effort and with that I will just hand it over to Sean Thank you Sebastian Alright so with the few minutes we have left, let's talk about what we are and I'm going to talk about some of the open stack existing tools we have today and some of the gaps so you've seen what cool tools big guys are using for example the AWS CloudWatch that is really a combination of some of the tools we actually have in-house but we have not yet took their potential out of this use case so for monitoring telemetrys are alarming and monitoring service in open stack and today you can have a gnocchi plugin for Grafana to actually generate very cool statistics of your quality of service usage and your provisioned IOPS QM itself that we're using in the generic API has what we call the info block that's command that you can poke it's currently exposed in livered but not yet in open stack so that's another approach as a community if you're interested in taking it and bringing it into open stack we can obviously Sebastian talked about self tools the RB decline statistics socket is another area we can pull monitoring and the key interesting part for me is event triggering automation since the ability to see hey this volume has already reached the 90% IOPS it's actually going to take a slope in terms of performance heat how we utilize this not just monitoring when it happens think about the burst examples we gave earlier so database like you have this is a cloud service we have more user suddenly consuming our application workloads do you have more performance happening in real time how can we trigger a real time event to actually add more quality of service in real time so you add more capacity derived quality of service you get more throughput you get more capacity in real life so basically today we have a static picture in open stack we would like to see how we can actually change volumes types limits so today if you want to use the capacity derived quality of service we showcased earlier that is now new in Cinder if I need now to resize that volume I cannot do it online obviously Amazon has limitations they can not even shrink the volumes but we can not even resize for us it's basically you need to detach the volume right change the volume type retype it and connect it back so it's really ugly so but we are able to automate a lot of these pieces because we are already able to automate the quality of service we can automate the volume types there are things we can do today obviously there are gaps that we need as a Cinder community to see what will be the right approach to do it but this is clearly the direction we should go if you want to offer that level of service consistently across our multi-clouds this is not just open stack as I said in my opening this is to be able to manage your hybrid clouds public and private in the same manner and lastly I want to be able to reconfigure them in runtime so my workload is running right now in production I want to provide that workload seamlessly with the quality of service changes on the flyage with Christ because application workload has a life cycle we need to be able to address the life cycle including burst in real time rather than yeah let's provision a new volume let's create the static specs in advance and think what will happen when we go live before I don't know Christmas shopping craziness going it doesn't work that way we need to be much more agile in the way we provision storage so to summarize we can build you a cloud like the big boys already open stack Cinder is already able to set quality of service limits now with the addition of the based on provision capacity we're able to flatten these two scale factors into a single dimension so we simplified the capacity planning for the operator as well as allowing the operator to increase the available capacity by adding more and more distributed back end right so more nodes more IOPS fixed increasing capacity while maintaining the quality this is where we already got to this point so we have a few minutes left because we have another session kicking in so I want to invite back Sebastian Federico and maybe two questions and we will be available obviously after this session and you can follow us on Twitter as well we have two mics so please use them and if you need any assistance we can provide one of these alright any questions can you open that mic yeah we can't hear you Sebastian you mentioned earlier you had a recommendation of a rule of thumb for 100 IOPS per volume but I'm assuming that's for a specific size of volume because I mean it would depend on the scale of your cluster how many IOPS you had per volume yeah it's basically you always have to calculate how many IOPS you want to provide to infrastructures you based on that calculation if you say I want to provide 10,000 IOPS then you plan a platform accordingly obviously so it's more about like in this case it's more like hardware sizing because I think you're referring to this as the scenario right right so yeah it's more about hardware sizing so that was a pretty small cluster you were testing on where you were saying 100 IOPS okay so that wasn't like a rule of thumb you should always go by it's like that's really like you need to scale it I mean you need to start somewhere I guess and yeah so this with this like simple recommendation this is what we recommend yeah alright thank you any additional questions alright if not thank you very much enjoy the rest of the summit thank you