 So welcome to the PTL webinar series. Today we have Mark Waschenberger reviewing image services, Julian Dejoe reviewing telemetry, and John Griffith reviewing block storage. We're going to have Mark go first. Again, about 20 minutes for each speaker to review the latest for their OpenStack projects. Mark, go ahead. Okay, thanks, Margie. And thanks to the OpenStack Foundation for creating this opportunity for us to get some visibility into our project. As Margie said, I'm Mark Waschenberger, and I've been the PTL of the OpenStack Image Service, or the codename of the Hephaelius plant, throughout the Havana release cycle, and the Ice House release cycle, which we're in now. And so moving on to the next slide, I just want to, before we get started, give you a brief overview of what I'd like to talk to you about today. First, I'll give you a sense of some of the progress that we made during the Havana release cycle and cover some statistics about our contributor base through Havana. And then I should have some time to talk about the progress that we've already made in Ice House that we're about almost halfway through the Ice House development cycle and the plans for what we hope to complete. So starting with Havana progress, I'll just talk to you about some of these features. The first one that we implemented is something we're calling Image Property Protection. For those of you who are already very familiar with the plant, Image Property is a sort of free form piece of data about an image. It's basically a set of key value pairs. And the way to think about this feature is as a bit of a deployer configuration that can restrict who can read and who can write different keys for images. And restrictions are based on the role of the user that's role-based. And in addition to read and write, also we separate out who can add a property, who can remove a property, and things like that. This has a couple of important use cases. They're all sort of centered around this idea of ensuring trust. So a deployer restricts certain types of keys so they can only be edited by trusted parties. And that's really important when it comes to image sharing. You can imagine a deployer could communicate to their users. You can trust this certain bit of information about an image. No random third party could show up and just edit it even if they own the image. So that sort of creates a chain of trust there. It's also useful for certain accounting use cases. You can imagine an image property that the user themselves cannot read and can't edit so you could put something like your licensing information or different billing-related attributes. You can bed them directly on the image and trust that the user didn't just change it in order to get a lower billing rate. Taking a look at the next feature. This is something we're calling the registry API V2. I think you need a little bit of background. There's two APIs in glance. There's the Images API, which is the externally publicly-facing one. And then there's this private one called the registry. And in V1 of the Images API, all communication to the database is mediated through the registry. This caused some problems for some users in terms of, or sorry, some deployers in terms of performance bottlenecks. And so when we wrote the V2 Images API, we cut the registry out of the loop. But that also, that solution causes some problems for certain deployers who like the deployment pattern of having restriction on the access to the database. So we created this V2 registry API. It's really just a very thin RPC wrapper on top of how we talk to our database. So in the V2 Images API, we can basically imagine that we're talking to the database but now we're actually, we can actually talk to the registry V2 API instead. And so that makes it an optional, it makes it optional to maintain this registry deployment pattern. This will help us as we move forward and try to deprecate some of the things that we don't want to keep anymore. Then taking a look at the next feature. This is something that we're calling multiple locations per image. Glance generally doesn't store the image data itself. It has drivers for storing the images out in some other service. Say for example, Swift. And so we tracked the location of where that's stored. What we did during Havana was expanded the model a bit so that there can be multiple locations that we store for where image data is. So a deployer can set up replicas for certain images. This immediately offers opportunities for increased availability because if the first one fails or if it's just not immediately unavailable, Glance will skip on to the next replica. And we're also looking at features to make this improve download performance. So you can imagine Glance or even eventually a client saying, well, this location is closer to me. It's faster for me for some reason. So I'll select that one. Next, moving on to the next feature set. I mentioned those storage drivers or back-ins. We added several more of those. We added the ability for Glance to represent sender-backed images. We're still working out some of the details here. When I say sender, I should clarify that's the OpenStack block storage project. So the idea behind that is to get to this point where we have zero copies and when you say I want to boot an instance, a Nova instance from this image, you're actually sort of under the hood doing a boot from volume that's available. But we're still working out some of the details there because it slightly complicates our model. So we'll look for more from that in Icehouse and beyond. We also added support for GridFS and SheepDog stores. So if you have those, it's a bit easier now to store your images. I think we can move along to the contributor statistics. During Havana, for the Glance project itself, we had about 60 contributors spanning across about 30 companies. And we're all told it was 250 patches, fixing about 130 bugs. In terms of lines of code, we actually grew quite a bit during Havana. I think we're up to nearly 50,000 lines now. And so we added about 18,000 of those during Havana. Fortunately, in terms of the way I'm concerned about these things, most of those lines were tested. So that's good. So looking at what we've already done in Icehouse, we've already fixed 43 bugs that we were tracking. And we've implemented a bit of features already. One, in particular, we switched our notification mechanism to using Oslo messaging, which has some good internal benefits. I think it makes us perhaps a bit better with Julian's project, I think. And we've also made a lot of progress on a blueprint called Asynchronous Task, which I'll talk about a bit more here in just a moment. Looking ahead in Icehouse, we talked at the Icehouse Design Summit about what are we going to do with the Z1 Images API and how are we going to move beyond it. We came up with a schedule, which I can sort of outline here. There's a number of preparations that we need to make in Icehouse in order to make this work. If we do everything as we're planning, we should be able to mark the Images API Z1 as deprecated in Juneau. So I guess from about nine months or so when people get Juneau, the Juneau release, if they're running Z1, everything will still work, but there'll be some warnings that pop-ups say, hey, this is going to go away. And then the plan is that that should enable us to actually remove the code in the K release. Now, in terms of some Icehouse features, I mentioned the Asynchronous Task or Asynchronous Operations, and I'll drill down a bit more on that now. Normally, the Images API is all basically synchronous operations, so you upload an image and wait until you're done or things like that. And we found that that's a bit limiting for certain types of use cases where you want some processing to run in the background or something that's pretty likely to fail for a variety of reasons. You know, if it's invalid, but the data itself has some problem with it. You don't find out towards the end of the process or until the end of the process. So we've been working pretty hard on adding a framework for supporting and exposing these asynchronous operations where users can initiate them and pull and track that they're, you know, how they're progressing, and did they succeed, did they fail. And the use cases that we're targeting at first are something we're calling image import and export. And import, rather than directly uploading the data to the plan, what you would do is make it available in some form, say, storing it on your In-Your-Swift container or something like that. And you'd say, hey, clients, go create an image for me out of these bits that I have over here in this format. And that offers the opportunity for the players to configure on the back end a conversion step or even invalidation different things that players are interested in to make it so that people can bring images to their cloud. Similarly, we have the export use case, which is to store it in reverse, we could say, hey, Glantz, I thought this image stored in Glantz. It's in some employer-specific format or whatnot. And I'd like to take it and use it somewhere else. So you could say export this image, put it in my Swift container, make sure it's in a raw there format. And so that should be really useful for interoperability moving forward. On the deployer side, we've been looking at how should these operations work. And we've been targeting two different deployment mechanisms at first. One sort of maximizes convenience and probably is not so great on the performance side. This is a DevStack use case where the import and export is actually running asynchronously in a local thread to the Glantz process. But we want to be able to scale this out as well. And so we have been looking at assigning the work out to workers that we're communicating to over our PC, sort of like how things work in Nova. Moving along to the next feature, I mentioned to you this idea of having multiple locations per image. And to really get the most out of that data structure, we need to have strategies that say, you know, this circumstance pick this image location type. And so we've been working on adding those kinds of strategies, making them configurable in Glantz. This should help optimize, really optimize the download performance and have some improvements in availability as well. This isn't something that we specifically talked about, but it's sort of a use case you can imagine you could have if you wanted to store your images on F3, you could store, you could say always use the reduced redundancy copy. And then if that doesn't work, move on to the frozen copy. So that would give you really great availability, better performance, et cetera. Moving along, I really just scratched the surface of our ice house plant. There's lots of blueprints that we are discussing or we've approved. And I just wanted to take this opportunity to mention that we'll continue to talk about the blueprints and other features at our mid-cycle mini summit, which is happening later this month in the D.C. area. A bit sparse on details here, however, if you take a look at the open sector, mailing list, archives, you can get more info in case you use something that you might be interested in. So that about covers it for me. And so I'm looking to see if we have any questions. We don't seem to have any questions in the chat right now. I don't know if anybody has any questions they just want to ask. You are now unmuted. And I have the lines muted, so I just unmuted the lines if anyone wants to ask Mark any questions. And if not, that's fine. I'll be around at the end for the general question. Hi, guys. Hi, guys. Can you hear me? Yes. Dimitri, I have a question in regards to using the current solution in the ice house for storing deployment artifacts of past, which is Tollum and Murano artifacts. So my question is it will be part of the glass or it will be a separate solution? So that is a topic that has come up sort of as we've been working on ice house. And so I didn't include it here, but it's definitely something that's on our radar. And it's something that we want to discuss and figure out the details during that mid-cycle mini-summit. It definitely feels like there's a lot of overlap in terms of general use case. The image service has obviously been very focused on images so far, but it hasn't really bothered with images that much. It's been much more of a general gatekeeper in terms of users show up with this data, glance validates it and stores it in a continuously valid form and makes it available in no-but. I think there's a lot of overlap and we just need to figure out all the details for how to expand scope and what's practical in terms of the code. Thank you very much. And do you think it will be a result for ice house or for the later releases? It really depends on what strategy we go with and we haven't picked it yet. It's possible to do something kind of quick and dirty and exposed in ice house, but that really only means that there's very short developments like to get that working. I think we'll just have to figure out what we feel like is the right answer at the mid-summit. Sorry, I can't give you more detail. It does look like we have another question in the chat. Not sure that I can see the whole question, but I guess the question is about tenant and user quotas in glance. We did actually in Havana add sort of dumb image quota system where you can say across-the-board users can only have a certain amount of storage per tenant. It's not really configurable per tenant, however, it's just an across-the-board setting. Photos, we've had a bit of a rough time with in glance because it's kind of an interesting question. Do you care about how many images? Do you care about how much storage? Do you care about where it's stored? We can store across multiple back-end. We have something to offer there, but maybe not everything that people are looking for immediately. I think that's probably covered us and I don't want to run too far over, but maybe we can redirect any questions to the end. That sounds good, Marty. That sounds great. Thank you. And also, if you have other questions in the chat, don't get answered. Thank you. If we have other questions that don't get answered and you put them in the chat, I can send them out to our speakers as well. Thank you, Mark. I'm going to turn this over now to Julian who's going to review telemetry. Julian? Okay, go right ahead. I'm there. Hello, everybody. I'm Julian. I'm the PTL for Cinometer, the open-type telemetry project. So I'm going to talk about a few bits about what we did and what we're going to do as well. So first, the state of the project, as it is right now, we know open-type telemetry. We voted on that a few weeks ago. It hasn't accepted it, so we're pretty happy about it. And I'm going to analyze our first integrity. So what does that mean for us? Back to Brittany, we were only in a state where we were part of the radius that didn't have any photo on security. So we're starting with Elena. This is the case. We are now handled by the security team, by the stability team, et cetera. So it's a long time now and it's going to continue obviously for ourselves, et cetera. We also saw a lot of community growth. The graph shows that, well, I think we kind of developed a number of contributors and commits. So I think it's going to be good. So now I'm going to talk about what we did during the Avalon cycle. The first feature we implemented is the UDP transport. So if you know telemetry, you know what we used so far, the RBC mechanism to transport, or something else, that the problem that you might have with the artistic mechanism is that it can be slow. It is a state layer that initializes all the group and the rest of them. The number of use cases like the R111, which I will talk about later, where you don't necessarily need to have all these requirements. So the UDP transport is faster based on UDP and state less. It shows a different encoding, which is a message pack that is lighter than the JSON one used in RBC. The only downside is that there is no data guarantee and the messages are not signed. But it's largely good enough when you do things like learning. So this is the first thing we implemented in Avalon. Then we improved our API. So we added a few things in the API like the ability to group this on both live fields. So when you request telemetry for samples, usually you will get statistics like maximum, minimum, average, etc. But it's so interesting to be able to group these fields by other values like you would do in SQL. So we added that for a certain amount of fields in the API. We also added a few possibilities in filtering live, limiting the number of replies you get in the API when you request samples. And we extended the API a bit and provided links to all our resources when you request samples. I think you can move to the next slide, Marguish. So while working on the API, we also added a few things for alarming like the ability to send your own samples. So far the only samples we were able to send to Ceremetry were the ones where we wrote posters. You had to rely on the RBC mechanism. We extended that to the API so you can no cost your own samples both at a number actual or as a user. So this is really interesting for your user. I'm going to be able to leverage this to send their own metrics and use the statistics that it is provided by Ceremetry. It's also usable for alarming. So typically your user are going to be able to send all the metrics they want from their own applications or things like that. And they are able to add this as a source for around. So it's a little thing that is needed when you want to build custom around for it and also getting for example. The next features we added which is supported in order to do that are new storage backgrounds. So we know of two more. We have QL so far and MongoDB. We now have a Bashi HB which is based on Azure and IBM DB2. This DB2 being the new SQL one, not the SQL one which is all the same as. So it's kind of a new product off here I think. It's both working but I don't have all the features that you would think that it's not a lot. We also added the distributed STTL. So far in Ceremetry we stored a lot of data, a lot of standards that you add no way to search all these data and that's a lot of data. We now have a tool called the Ceremetry Explorer which is able to drop all the old data you don't want anymore so you can configure a global time-to-limb for your sandbox and the script will delete everything that you don't need anymore. We added also the IPV support. So this is if you run Windows Server in your Nova cloud you're going to be able to use Ceremetry Ultra-Disk Network and CPUs as statistics. You can move to the next slide I think. Yes, that's about the IPV. The next one is going to be about the new meters we added. So we added the API on points meter. So this is used to measure the number of requests that you are going to see on your endpoints like neutral, length, the vastness, etc. We also added the neutral bandwidth meter. So this one is a bit heavier. It's largely based on a feature that was added in neutral during the Avalanche development cycle. So neutral has not the ability to monitor the bandwidth used at the L3 router. And it sends notifications to Cinometer with the number of bytes and packets going in and out of this router. And Cinometer is able to monitor that and to reach these notifications and bridge new samples to monitors the number of bytes, packets, etc. for each of our tenons. So this is really useful in cases where you want to build for bandwidth. So Avalanche, you can move to the next slide. Yes, Avalanche. So Avalanche is about watching metro statistics and on trigger actions, action being whatever you want. It can be a webbook, sending an email, an SMS, whatever, based on social crossings. So these came out just in the next slide. Show the bit or it works. So the API endpoint is used live for any of our clients. And it's used by the alarm literature. So it's a demand that evaluates all the alarms each which is at the API as we want to know why. And it triggers an alarm as soon as the session is closed. For example, if you set an alarm of CPU meter as soon as the CPU usage is above 50%, it's going to trigger an alarm and send an action to do to our number of demand which is the CPU meter alarm literature. And this one is going to take action and do something like call a webbook, send an SMS, an email, whatever. All of this system is playable so you can, I mean, insert your own action set like, I don't know, send a different type of message or anything else. It's based on some of our symmetry architecture so there's nothing new we invented there. There are different types of alarms. So the first type of alarms which are the simple ones you just send something like I was saying, you do something as soon as the CPU usage is above 50% and the second type which are the combinational alarms. So it's just a logical combination of any alarm being a combinational alarm itself or a threshold alarm. So it's like the example which on the side just do something as soon as this alarm and this alarm are triggered. We also started in M&R something that we will continue during IOS which is the event storage. So all we do so far in Cilometer is retrieve notifications which are sent upon events in our open site components being Nova, Neutral, Glance, etc. We want to leverage this and to store these events for a lot of different reasons being the ability to regenerate samples and to have an audit of what and all we need to generate these samples in the first place. It's also useful to generate these samples that we didn't think about back then. When you do a sort of in-line, in-stream processing of events and generate samples on the slide you can sometimes forget to generate some samples because you didn't have the idea back then to generate them. So this is kind of an issue. It's also required for some people in some countries to have a development work on things. So to have two different sources to prove that you're going to, for example, be the orchestrator for something you have to prove but it was a real sample you got from a real event. So I think it's stored twice as a lot with that. So this is not yet complete. We only, I think, did 20% or 30% of the work that is needed to do something working but it has been starting in the last part and we had a lot of discussion during Avala and we're going to continue on that on iThouse. So I'm going to present you things we're going to do in iThouse, I think, if the roadmap doesn't change. So a few general improvements first. We're having to split the collector in two logical pieces. So this has been released a few weeks ago in iThouse 1. We know of a different demand under the notification event before we add only one at the collector and we know of a notification collector and a collector which makes the matter a bit more scalable. We are going to change the way this sounds as numbers. So far we use the RPC mechanism being a function code. We want to change that and leverage the notification system once again because it's lighter and faster. Both our main drivers, as Kurokini and MugaDV, are not really on parity in terms of features that we're going to try to improve that. We are having worked a lot of building integration testing between S2L, Mungo and CicaMaker to be sure that everything is working for a way. And there are also a couple of people wanting to work on the hardware monitoring support being for ironic or not. On the API side, there was a lot of proposals, likely the ability to build complex filtering queries. So rightfully, the ability to, for example, include a logical or in your query request for this is all that which is not possible right now in the API. You would have to do two distinct queries. So we'd like to be able to do that in only one. We are also building a new approach in relation to the API being samples, which is the same as nature but without the nature name. This has been first challenge already. So it might be in I suppose two in a couple of weeks. We would like to add things like running the rate rather than actually value for certain type of counter. Typically, if you use the bandwidth counter, it's more useful in certain cases to add the number of bytes per second to the number of bytes. We would like to add more statistic functions and the birth request number, which would be the ability to trigger a lot of samples in one query. The alarm clock should see a few new features too like improving the number of samples used to build the threshold on the ability to add chain constraints to alarms that only trigger this alarm if it's 3 a.m. over why they don't want you to trigger it. That's useful for certain cases such as autoscading in kids. The last one is about distributed polling. So we have a lot of problems in cinemas about scalability, averaging meters among a large number of computers now, something like that. It's a common question we saw in other projects like Nova, Onatron. We discussed a bit in the autoscale project to solve this solution at a larger scale. We started a project which is two, and there was another project called That Flow. To solve all of that, we need the ability to distribute the polling with you in cinemeters. So three of them are only one, demanding all the polling of all the platform in the Cinemeter Central Agents. This is not very scalable obviously. So we're working on that. I hope we will have something ready so that it takes time to build this library. We would also like to replace our custom alarm distributor we use in the Cinemeter system right now. We vote our own so far to have something working but we know it's not a perfect solution so we're trying to do better to do better for the house for this. I think that's all for the new feature we would like to add in. In that also, if you have any questions, just yell or write in the chat or if you have any questions. Thanks, Julian. The lines are open if anyone has any questions for Julian or you can put it in the chat box. Or if not, we can also go over it at the end as well. Yes. Okay. Well, thanks, Julian. So I'm going to turn this over to John regarding block storage. Let me pull that open. Thank you very much. John, are you there? I am. Go ahead when you're ready. Okay. Hi, everyone. Good morning, afternoon, wherever it might be where you're at. I'm John Griffith. I am the current PTL for the block storage project in OpenStack. I've been working on the block storage project basically since it started, but Grizzly was the first release and then Havana was our second release. So today I just wanted to go through and just sort of go through some of the highlights about Havana, sort of where we're at, what kind of the stats were. So if you take a look here, we have a number of commits, up to 843 total commits, lines of code, number of views, so on and so forth. So these numbers are all, you know, they're interesting by themselves to look at and get an idea of how active a project is and what's going on and stuff like that. So you can look at these and see that there's a pretty good, pretty good amount of activity taking place in the block storage project. But what's more interesting is if you go to the next slide, if you look at the growth from one project to the next, or one release to the next. So I went ahead and took a look at the deltas of those same numbers between Grizzly and Havana. And you know, saw some things that to me seemed like healthy growth in the project. So the number of commits, we increased by 400. You know, lines of code went up significantly. That's not always good. However, in this particular release we did a lot of things to move some code out into libraries and things like that. And we added a lot of unit testing. So a lot of that lines of code that you see there is actually unit testing. And then of course there are new drivers that came into play with this release and that always is a significant amount of LOC that gets added. What's really cool is the number of reviews, if you look at that, and the number of contributors. Those are my two that I really like to see. The number of reviews went up significantly. It went way up. And even more importantly, the number of contributors increased. So, you know, we're up positive 74 in the number of contributors for the Havana cycle. So, you know, that's great news and it's a great sign that the Block Storage project is still growing. New people are getting involved and interested in working on it and stuff like that. So that's great. Great things to see in my opinion. So if you want to go to the, yeah. So some of the things that I wanted to call out, you know, particularly in terms of the community activity and the things that have been going on inside of Cinder. One of the things that Cinder has done in terms of requirements and things like that, we have an interesting challenge because it's, you know, Block Storage. There's a lot of vendors that want to participate and there's a lot of devices out there. And keeping everything common and keeping everything working can be somewhat of a challenge to make everybody actually kind of on the same level playing field. So a while back, we kind of came up with this strategy that we were going to have minimum API requirements. So there was going to be a minimum set of API features and things in the Cinder project that every backend device that if it wanted to be, have a driver included inside of OpenStack and it wanted to be supported, it would have to meet those minimum requirements. So we started that and we came up with this scheme of basically everybody would have one release, basically to get up to date if they were out of date. And the great thing was in Havana, everybody really, all the different vendors and the maintainers and even folks that had no relation to the vendors of those drivers really got involved and got all of those drivers up to date and got those minimum requirements all implemented. So that was a really cool thing to see and it worked out really well. I think it gives a great experience for the end users and for the people that want to adopt OpenStack. And I also think it's a big attribute to, or a big testament to open source and how OpenStack can actually work. One of the other things I wanted to note was some folks that had been absent from the OpenStack world that had committed in the past had come back. Nexenta in particular came back this Havana cycle and contributed a lot and not only to their driver but they also contributed a lot in terms of discussions and roadmap planning and things like that and put forth a lot of effort and helped out a lot. So I wanted to put a special shout out to them there. In addition we had some new drivers come into play which is always a good thing. VMware got really involved and did a lot of work not only for their VMDK support but also other areas of the project as well. So the things that I really like to see from, when vendors in particular get involved and want to add a driver or add a backend, the thing that's really cool to see is when they actually also contribute to the rest of the project as well and they help out with reviews, they help out with improvements in other facets of the project. So that was a really good thing to see. And then also Dell came into the fold this release as well with their Ecologic project. So that's good to see as well. Let me go to the next one. So some of the features that we covered here in Havana that came out, I'll kind of step through some of these. I hate to just read bullet items on the slide but I want to talk a little bit about each one. Encryption, if you're not familiar, discussions around encryption and the idea of doing encrypted volumes started way back, probably at the Grizzly Summit, the summit in Portland is when we started that. So the Havana Summit. We started those talks. We had a lot of different ideas. And through a lot of hard work from the folks at Johns Hopkins and some other people, we actually got a pretty cool system in place now to handle encryption. So that's something that if you haven't looked at it, definitely take a look. It's a pretty neat feature. And I think going forward we'll do things like look at how to expose the encryption that the back-end devices actually support as well. Right now, that's more of an encryption that does even over-the-wire encryption. Something to look into, and it's pretty interesting. One of the other things that we did was we moved to using Taskflow. So Cinder Project got to be sort of a guinea pig for the Taskflow project. And it worked out pretty well. So the concept of Taskflow, if you're not familiar with it, you should check it out. But it gives you a lot of flexibility in terms of managing state transitions and doing rollbacks and things like that. So for the first step, Josh Harlow from Yahoo went ahead and did a significant amount of work to implement that for our create volume workflow. So that just continues to grow. We continue to make improvements and learn as we go. I think it's going to help in the long run significantly with some of the problems that we've had traditionally with block storage, with being stuck in certain states, whether that be error on creating or whatever, things like that. So that'll be good. You know, a while back we introduced the concept of backups. We introduced a backup service into Cinder. And those backups basically are just, currently they were, well, they were just doing, taking a block storage device and saying back it up to an object store. That's continued to mature. It's continued to get better. I think we've still got a ways to go. There's some things that we need to continue to finish there and get implemented to make it a true DR type of operation. But it's definitely growing. It's definitely getting some interest. We started off with Swift, of course, as the backup target. We've also added Cefs object store. The Tivoli Storage Manager folks from IBM actually added a driver as well. So if you're traditional tape folks, you can use that as well. So that's something that we're definitely working on and we continue to push forward. And I think it's going to be extremely useful for folks in the future. The extend volume, that started out as resize volume and it changed to extend volume. We kind of started down the path of doing resizes. There's some caveats and things, some pitfalls and stuff, with reducing the size of volume if a user has a file system on it. And we wanted to kind of think through some of those things and figure out how we wanted to control that, whether it be through a policy to allow it or whatever, or flash warning messages or something like that. But we want to do something to make sure that we don't let people just completely shoot themselves in the foot there. So for now, right now we have just the extend volume, which still is a great feature. In addition, we added rate limiting support. So there's some back ends that do quality of service and we've had that sort of functionality implemented through extra specs. We've also now added a more general rate limiting feature as well. That rate limiting typically works by rate limiting on the hypervisor, but you set that on the volume inside of Cinder. So there's some pretty neat things there, some really useful tools for people that folks have been asking for. So that's something else that you might want to take a look at too and get some more detail on. We did a lot of work to clean up the boot from volume integration. In the past, boot from volume, we made it significantly better and grizzly, in my opinion, than it was prior to that. However, we did not have support in Horizon and there were some things there that just made it still a little bit clunky. You had to go to Cinder and do a bunch of steps and then go to Nova and do a bunch of steps and you just couldn't do any of it from Horizon really. So now that is all integrated inside of Horizon and Nova, which is fantastic. So you can go directly to Nova and make the bootable volume calls and everything and you can also do that from Horizon. The implementation of Horizon came out really, really nice. I'm thrilled with that. One of the other really neat features, in my opinion, that we introduced was the ability to transfer volumes. So we implemented a method where tenant A can go ahead and say, hey, I want to give this volume, this particular volume up to another tenant. Basically what it does is it generates a key, the transfer create call in the API creates a key and then that tenant who created that can give that key to whoever they like in that open stack, in that cloud deployment. And then that tenant can do a transfer take or transfer accept and enter that key and then we will go ahead and change all the permissions and everything on that volume to basically give it to the other tenant. So that's a pretty handy feature. That's something a lot of folks were asking for. In the future we've got some more to do on that. We're going to kind of advance that whole concept and that idea and I'll talk about that a little bit later. The next thing that we added was the ability to do volume migration. So that was kind of a, we're at the first step of that and a lot of what was driving that was part of doing instance migration. Of course if you have an attached volume, what are you going to do there? So we went ahead and actually the folks from IBM, Avaché Trigger, worked really hard on the migration code and implemented the ability to go ahead and migrate a volume from one sender node or back end device to another. So that's something again, it's the first steps and we're making progress and it's a great feature. One of the other big milestones that I wanted to just kind of touch on real quick is we introduced the version two of our API. A lot of folks in the open-stack world see an API version bump and think, oh no, here we go, this is going to be bad. I think anyway, I think we really learned from some of the mistakes that we've seen in the past and some of the things that happened in the past and we really strive to make a fully backward compatible version bump. So far the feedback has been excellent. There hasn't been any significant gotchas, any significant problems or anything like that. And we added a number of new features, cleaned up quite a few things in the code. So the V2 API is something that's really great. If you haven't looked at it, if you're not using it, I highly recommend taking a look at it. And if you do run into some sort of problems or some sort of compatibility issues that we haven't caught, let us know. We'll definitely take a look at it and get to enter. So what's next? So it's kind of odd talking about this halfway through the, or not halfway, but almost halfway through the Ice House cycle. But it's actually really good because if you've ever been to a design summit it's a flurry of activity and ideas and then we try and give a project update and say what we're going to do and it's like, well, we talked about a million things and I have in no way prioritized them or scaled them down or figured out what's actually going to work. So this is actually really good. The things that we're definitely focusing on and making good progress on so far are things like a driver cert program. And the idea there is one of the things that you have in a situation like the Block Storage project is you have all these different vendor backends, all these different products but none of them are actually gated in our CI. So OpenStack has probably the greatest CI framework and everything put together that I've ever seen. It's absolutely fantastic. We do a tremendous amount of testing and gating on every patch that goes into the code base. And that's great. However, we have absolutely no automated testing or checking of all of these third-party products. And I mean, it's obvious there's a good reason why, you know, having 20 different backend storage devices all in our OpenStack infrastructure is not something we can really do. So the concept of the driver cert program is we want to, going forward, we want to actually log public information about every driver inside of Cinder. And the idea here would be we would probably start by logging a bug against every driver and saying, hey, this driver hasn't passed the cert for this release. And then there's an automated test now inside of DevStack that they can run so they can do all of this in a DevStack environment, attach their device in their lab, run the cert program. It'll run through the standard gating test that we do today for the reference implementation driver, collect the results and submit them. So basically it's just a way to get some public information that says, hey, yes, we did test this. And yes, it does work. Here's the results. It's not perfect. It's not flawless. But it's a first step. And I think it's a good direction to go in terms of getting information out there and letting people know this is what works and this is what doesn't. Or this is what's being actively maintained and this is what isn't. One of the other things that's coming online in Icehouse is replication. So if any of you were at the OpenStack Israel, you may have saw Florian's discussion about asynchronous replication in Cinder and things like that. So we're definitely not there yet. It's definitely a work in progress. The first steps right now are on-demand replication, being exposed in the API. And that's a great first step. We need to start there. So that's going to be fantastic. That's moving along really well and that should be showing up here in the I2 release. The next thing we have is local storage scheduling. So the concept here is to actually be able to, so currently you use Cinder and you say, let's say for example if you're using the reference implementation, the LVM driver, you're creating a logical volume on a Cinder node and exporting it via iSCSI to your compute node doing the attach, et cetera, et cetera. The idea here is we would like to be able to take this concept and do things like have Cinder say, hey, create me a LVM, a logical volume, but create it on the compute node that the instance is residing on or is going to reside on. So that's kind of the direction we're going with that. We've talked about this for a while and just never really dug in and got it done. And I think in Ithouse, I think we're going to get there. I'm hoping to find the cycles to work on that. I've started laying a lot of groundwork for it. So I think we'll get that in Ithouse. If not, it will be very early in the Juno release. The next thing that we have is volume retype. So one of the things in OpenStat or in Cinder right now is we rely heavily on the concept of volume types and extra specs. So those volume types and extra specs are basically used heavily for a couple of things. One is if you have a Cinder deployment that has multiple backend devices, that's the method that we use to control which backend device a request is deployed on. In addition, we also use that to do things like expose custom functionality that that backend might have. So the idea of a retype is a user can go in and create a volume of type A, for example, and then say later down the road, they may have some requirements that they need thin provisioning support. They need QoS support. They need some sort of other feature, whatever it might be, for that particular data set. What they can do at that point then is they can go ahead and do what's called a retype and that retype will go ahead and if that new type that's requested can be served by the same backend device, that's great. What we do is we just change the parameters and the characteristics of the volume and everything's good. If it can't, we'll actually provide the ability for you to go ahead and do a migrate. So that's where the migrate code that we talked about before comes in. So everything in Cinder, we kind of do in steps and we kind of build on it. So you can kind of see that with retype. So we'll go to the next slide. So that's, I blew through those pretty quick, but hopefully there's some questions. It looks like there's a few over there. I'll have to forgive my cold. I'm just going down. How is this for you? How is rate limiting with volumes implemented in Icehouse? Meaning is this an IOP, SpamWidth, or I think we have about five minutes too, so I'm sorry, go ahead, John. Sure. So currently it's a rate limiting based on IOPs. You can take a look and it's partially dependent. That generalized method that I talked about is really completely dependent upon the hypervisor at this point, so mostly LibBird. But you can look inside of LibBird. It gives you the ability to do things like rate limiting on read Iops, write Iops, et cetera, et cetera. So that's what that general implementation is right now. There are also backend devices that allow you to do things like set quality of service and do it more fine grained, or depending on how you look at it, you can call it different things. But there are some that let you do things like actually set a minimum and maximum IOP level and actually guarantee those levels. There's others that allow you to do general things like fast, medium, slow. They do things like take a look at their overall IOP capability and will do things to try and reserve that level for you. So there's a lot of options right there. The general implementation, like I said, is something you'll have to look at in terms of using the hypervisor to actually do throttling. And it's something that folks have asked for. It's something fairly new in LibBird, so that's implemented. And when you're looking at that, one of the things you have to look at is what we did is we went ahead and grouped that under a thing called QoS specs. So you actually create a QoS spec to do that and assign it to a type. And Jeff, you have a question there. Is there any QoS to ensure certain levels of performance? The answer to that is yes, depending on your backend. So some backends do that, some backends don't. Solidifier is an example of one that will allow you to do that guarantee. I believe there's some other ones. You'd have to do some research, but I don't want to get into specific vendors and what they're doing. That's not a specific sender function or a sender implementation as of today. So hopefully that works. Okay, great. Oh, no, was there another one? It looks like Rushi is answering some questions. Thank you for helping me. Thank you, Rushi. Well, we are a few minutes over. Thank you so much, John. I know we had three speakers today, so I appreciate everyone's use of time. If you have any additional questions, you can send them back to the foundation and I can get them back to Mark and Julian and John because I think there's a follow-up email that everyone receives. Once again, thank you, Mark, Julian, John, for your time today, and we'll see you next time. Thanks, everyone.