 All right. Welcome to the Cinder project update. I'm Sean McGinnis, the current PTL of Cinder. Joining me is Jay Bryant and Jing Yang, some of the longtime cores of Cinder. These are a series of updates that the foundation wanted all of the projects to provide. We've done a few of these in the past. So we've kind of tried to meld some of the updates that we've done before with some of the information that they want us to show now. So just to make sure everyone is kind of level set here, I'm assuming you probably know since you cared enough to come to this room. But Cinder is the block storage service within OpenStack for managing either external storage, LVM, what have you, to provide block storage to instances in an OpenStack cloud. You can see on here some of the little clip of our mission statement. But really, you can see from the diagram, what Cinder is is a set of APIs interface that abstracts the storage on the back end to be able to manage these different types of storage and still have that consistent cloud interface. It is the management path to storage, not the data path. So Cinder's been around since the Folsom release. Block storage as a concept in OpenStack has been around longer than that. It was originally a piece of NOVA. And then as the project grew, we kind of realized that block storage was something that had had its own issues and needed its own attention. So really, within one release, they were able to take some of that out of NOVA and it became the Cinder project. As a Volcata, we had around 215 contributors, 50, 60 companies, I think, total represented. So a lot of good involvement from a lot of different areas. And according to the latest survey poll, if you haven't seen that, the survey has a lot of great information. Cinder is one of the core services of OpenStack. It's used in around 89% of the deployments that were tracked by that. Now, you may have heard some of the changes in the schedule of how we're doing releases in OpenStack. Used to be every six months. We had a shortened cycle in Ocata to accommodate the switch of pulling out the PTG and making that a more developer-focused thing and then introducing the forum here, which is more the ability for us to reach out and actually talk to consumers of the services. So typically in the past at the summit is when we were talking about the next release. Now because of the schedule change, we're actually, if you can see there in the blue on the right-hand side, we're well into the pike development now. So hopefully that means that you guys have had a chance to actually see what was Ocata, what became Ocata, and maybe still have time to be able to talk to us and give us any feedback that we need that we can still get something going in pike if there's something that we need to address. So we'll talk a little bit about what happened in Ocata, talk a little bit more about what we see or what we're trying to work on now, and then just a little bit about some of the discussions within the group and what we see coming up beyond bike. With that, I will hand over to Jing. So in Ocata, we got a few new drivers added. They are Dell EMC Unity, Hitachi VSV Driver, Infinibox, and you see Nimble added FC support, Reducill, QNAP-ES Driver, IBM Street Driver is now open sourced. We also get a few drivers planned for pike. The generic voting groups feature was introduced in Newton. In Ocata, we wrote migration scripts to migrate existing data from the consistency groups tables to the generic voting groups tables. In order to upgrade to pike, operators need to run data migration scripts to migrate data if there is data in the existing consistency groups tables. Once upgraded to pike, all the data will be in the new generic voting groups tables. After migration, CG APIs will operate on the generic voting groups tables. Groups with the default CG snapshot type, that's a group type reserved for data migrations, that can only be operated by CG APIs. This restriction will be lifted at a later time after all the drivers supporting CGs have added that support to generic voting groups or after the CG APIs have been deprecated. We have asked driver maintainers whose drivers support in CGs to add that support to generic voting groups in pike. And most drivers have got that changed on. CG APIs still work in pike. The plan is to deprecate it in queens. The work to support active, active HRA in Cinder Voting Service has been going on for a while now in Cinder. Now Cinder has active, active HRA support in Cinder Services. It's time to add that support in the drivers. We have heard from some operators in the past that they have been using HRA proxy and that has been working fine for them. However, that is not the true active, active HRA, because in that case, each service is only managing its own volumes. And it will not manage other services volumes. As long as all services are up and running, things are working fine. But if the service fails and restarts while another service is in the middle of an operation, the restarted service could interfere with the running job. There is also potential for race conditions when there are multiple requests for the same resource happened at the same time. Cinder now has assert support rolling upgrade tech. That means Cinder now supports minimum rolling upgrade capabilities. We also have support from the Liberty Release. So if you're running a release that is before Liberty, you need to upgrade to at least Liberty first and then from Liberty to Okara or later. The next item of functionality we'll talk about is the multi-attach function. For those of you that have been waiting for this to come along, I feel more confident saying we're getting close to getting this in place. So the Cinder side of it, this is one of those functionalities that requires a lot of interaction between Nova and Cinder to get working right. And so we've merged a cleaner approach to the APIs for interacting with Cinder during Okada, which was the first step in order to start moving towards having Nova be able to interact with Cinder to then enable multi-attachment. For those of you I should explain for those of you that aren't familiar with the concept of multi-attach, that's the ability to have one volume attached to multiple instances, virtual machines at once. For a shared database that's read-only, something like that, that's the approach that we're looking to get to. So we've made good progress on the Cinder side to enable Nova to make the changes and now during Pyke we're focusing on working, we have a cross-project meeting that we regularly get together and meet Cinder and Nova people to talk about those API changes and working towards hopefully in Pyke, getting that work done. From an API changes standpoint, a lot of the changes here are in support of the other things that we've been talking about here in Okada. For instance, the ability to do reset status on groups and group snapshots. That's if there is an API change for that. Important to note that we've decided at this point to deprecate V1. We really need to get people moving to V2 and really V3 at this point. So it was decided in Okada to actually make it so that you can't use V1 without specifically adding an option to your config file that says, yes, I realize I'm still using V1, I want to do that. And we deprecated V2 even because V3 is really V2 plus micro versions. So now the default version is V3 and we would like to get everybody moved to that. So distributors, vendors that are not at that point, please be aware of that and start moving in that direction. Some other updates to the forced delete API. It's not limited to the admin anymore. So that was a change in the API. And then also we're starting to make some changes that are kind of a preview of what's gonna be coming in Pyke with filtering through the API, adding the ability to do more filtering on queries. Config option changes. This was an area that we really needed some help in the way that we're handling configs in cinder.conf. The default section had kind of become a big dumping ground. And so now instead of if you have a number of drivers would just dump their backend configuration into the default section. But then if you used enabled backends, which is the recommended solution where you have a separate section for your driver config. If you had settings in there, it could be overridden by whatever you had in default and debugging config problems was rather frustrating. So at this point now driver config options are not going to be read out of the default section. You have to have a separate config section for your backend that you then specify using the enabled by adding that backend config into your enabled backends list that is in the default section. And we've got some other discussion around that here later in the slides that we'll explain. Also change the way that the naming of the snapshots is done to include volume name dot snapshot ID instead of just snapshot dash snapshot ID. Make it a little easier to associate what snapshot, somebody's clapping, good. Good to see when we make people happy. So we've got, so that it's a little easier to associate what snapshot goes with what volume. So that's good change. And for those of you using LVM, please be aware that the way that the clear function, volume clear is being used has changed. That you need to set that to zero. The shred option is no longer available there. So please be aware of that as well. And finally, some other changes that are kind of looking forward to what we're gonna be doing and talking about here as we go through Pyke. These were things that went into Okada. We now have more capacity information coming from the drivers in the backends that we can utilize as we're going forward improving Cinder. We fixed a problem with volume extend where you could try to extend it even though there wasn't enough space on the back end that the volume was upon. We had a problem with that that it was not going through the scheduler so it wouldn't know how much space was available and it would just go, sure I'll try that and then it would fail and bad things would happen. So that was a good thing to get fixed in Okada. The Cinder all binary, which would just start all the services was not really being used that we could tell. People are using system D or you're running the processes by hand. So we decided rather than trying to maintain that we would remove that. And support was added for ZMQ messaging and for the multi backend configuration. So now as we've foreboded some of the information for Pyke we will move on and actually talk about that. So a lot of the stuff we're doing has really been carry on work. So if you've been coming to our sessions I feel kind of had to kind of say some of these same things over and over but they've been longer efforts that we've been working on. One thing related to was with multi-attach. Well when one of the delays in us getting multi-attach working across Cinder and Nova has been that once we added some of that support on the Cinder side and tried to work with Nova to actually plummet all in, realized that we had a very complicated API between Cinder and Nova and some of the assumptions made by development on the Nova side about what those APIs meant and how they were supposed to be used weren't quite right. There's kind of some legacy of how Cinder used to be part of Nova and then we didn't clearly separate them right away. So one of the things that we've had to do in order to get multi-attach actually progressing was take a step back, take a look at that API and really simplify it, make it a clean separation, make it a simple rest call of that Nova can call us. We know what to expect each time. So one of the big things we're working on now we got that attached attach API landed for the most part in Cinder in Okada. Now it's a matter of working with Nova making sure they can actually consume what we have there. There's a chance we might need to still make a tweak yet so we can't really call it completely done on the Cinder side but we are making really good progress and I'm hoping one of these times we can actually come up here and say that we have multi-attach working. And just to use this opportunity multi-attach needs a clustered file system or an application that knows how to access alignment from two different hosts at the same time. I just wanna put that out there because I keep getting misconceptions that you expose a volume up and everyone can access it. There's some caveats there. Okay. Public service announcement. Okay. Cinder client, microversion support. So we've had microversions in there. That's the whole point of our V3 API. We really want people to get off of V1. We want people to get off of V2. The base V3 is exactly the same functionality as V2. So if you're on V2 now should be a really simple matter of switching over. As we add new functionality that's all going to be through microversions of V3. So V3.27 or however you wanna refer to it. So the client libraries that are used for that and the CLI to access that functionality that gets added into those later additions to the V3 API. We're trying to make some changes to the client and make the consumption of that a little bit easier. As you talked about generic volume groups we had consistency groups. We wanted to do other grouping type of things like being able to replicate a group of volumes. So that's how we stepped back. We introduced the generic volume group concept. Got that in place. Then we added a capability of a generic volume group that is a consistent snapshot group. So that's kind of a long term plan then. That was to get rid of consistency groups. So we are getting the consistent snapshot capability into generic volume groups and we can use it through there and then now build on with replication. So tier musu is that replication of a group of volumes. We gave code names because people were getting too hung up on what does this mean, what does that mean? Okay, let's call it dessert. So we have tier musu, it's group replication. We have a replication in place right now in Cinder where you can, it's more of an all or nothing thing. If you put it on a back end that's configured for replication with the volume type it replicates everything and it's admin only. This allows the user of the cloud that knows what their application is and knows that this volume and this volume are part of this application to say this is important to me. I like this data replicated off and protected somewhere else. Yes, well generic volume groups was the building block. We needed to get that in place. So we got generic volume groups in place. We got the consistent snapshot capability in place. Now we're adding replication. So it's all built off of this concept of being able to logically group volumes. Good question. And there were some quota things around there. We didn't really have anything in place as far as those groups go. So we're adding support for that. Now this is something that they wanted consistency across groups. We'd get this question from the foundation. They have these different focus areas. So we had to kind of categorize some things. We'll see some here, say not a focus. That does not mean that we don't care about modularity that we don't care about interoperability. We definitely do care. These categories were more around being able to at least just classify where a lot of the attention is right now within the project and what we're focusing on doing. So really resiliency, user experiences has been more of the focus for us. Really, we care about all these. I just wanted to preface it with that. But you can see these are the areas that we're trying to address right now. BTG was in Atlanta a few months ago. That was where we do a lot of the developer discussion. We had a lot of really great discussions about things that we want to do there. Jay's going to talk a little bit about some of that to start. Thank you, Sean. Yeah, this was our first time we did a PTG for OpenStack and from a center standpoint, I felt it was productive. We had some really good discussions. And so this is some of the items that we want to kind of highlight to you guys that we've been talking about doesn't necessarily mean they're going to happen in Pyke, but it's what we're working towards getting done in Pyke based on those discussions and things that we're hoping to continue to work on. So as I kind of mentioned that earlier, we had cases like with extend volume where we weren't checking how much capacity was really left on a host under certain conditions like if we were doing extend and snapshot, that kind of thing, it's possible to skip the scheduler in the process of making that request. And so if you run out of space in the process, your snapshot create fails and that's not good user experience ties into the user experience focus piece of it. So we're working on looking at how do we rework that flow for things like extend volume snapshots to go through the scheduler so that we can gracefully go, oh wait, we don't have enough space before we try to even try to create the snapshot. So at the moment, what we're going to do is use the information we have in the manager to try to catch it earlier rather than waiting for it to fail on the backend, but eventually we're going to start looking at a way to put that through the scheduler so that we'll gracefully say, hey, you don't have enough space. Had a really good session. This was one advantage of the PTG was the fact that Walt and I were able to go and actually sit in on the OpenStack client discussion. Walked in there like, hey, the cinder guys are here, let's talk about your cinder stuff. That was good. I learned a lot about OpenStack client from that and we were able to work together on things like, they said, well, you guys use this manage, unmanage concept, nobody else does that. And we're like, so what do we do? And they said, hey, we like the term adopt and abandon, which seems kind of harsh to me, but it's consistent with the other projects. So watch for that if you're used to using the cinder client as you move to the OpenStack client, it's now that you are going to adopt your new volumes in and abandon the other ones, which we won't think about that. Also, we're going to make it so that the OpenStack client will talk to the cinder server that it's connected to and find out what the latest micro version supported is. So if it pings out and says, okay, I've got 3.26 running on the server, it will support the commands up to 3.26. So that will be good functionality to have. So I mentioned this a little bit earlier. We're working on making the default section of the config file work in a more user-friendly manner. Then people said, but wait a second, what if I have multiple backends that want to use the same options? I don't want to have to change it in every backend configuration section. So we're working on getting in place, and I think we're close to getting this already, getting code in that will let you have some of those items that apply to multiple backends. You can configure them in the default section, and then they will apply to all of the backends that support that config option. Capacity-based QoS, this was brought up. There are a number of backends and also enterprise customers that want to say, okay, this user has six gigabytes of storage, and I want to control their QoS based on how much storage they have. So we're gonna be enhancing the scheduler to support that. We're looking at getting that done. There's a spec out there. I'm not sure if we'll get that in pike, but we're moving in that direction. We did have general agreement that everybody was okay with putting that in. And then pulling in groups, like we've been mentioning before, the ability to get the backup service to act on a group of volumes instead of just on individual volumes. So this is where this idea that we have a generic group concept is enabling us to grow and develop how you can work with your volumes through Cinder. Had good discussion on all of those. I think we're making good progress on them in pike disclaimer. They won't necessarily all make it, but we're definitely moving forward on those items. And let's see, at this point, Xing's gonna cover the rest of the new items we had from the PTG. Go ahead. So we talked about reset state at the PTG. So right now we have a command, reset the command for volumes, and we have snapshot reset state, we have backup reset state, instead of adding a new command for every resource, the proposal was to deprecate all the existing reset state commands, and instead just use a single Cinder reset state command. This change will be in Cinder client. It takes a type, it could be a volume snapshot, backup or group, takes a state, it could be available, arrow, or any other state. It takes a resource UUID. I think the spec is already merged after the PTG. And we also talked about revert to Snapshot. We spent a lot of time talking about that, a lot of concerns. The proposal was trying to line up the design with the existing implementation of a revert to Snapshot in Manila that it takes a Snapshot ID as the input parameter. The Snapshot has to be the latest Snapshot, known by Cinder. So that raised the concerns for some people because if we're doing this, that means we could be reverting to a Snapshot. After that one, all the Snapshots taken after the specific Snapshot will be invalid. That could mean data loss. And the reason for the restriction was that we do not want to have the possibility of data loss. And we also talked about how to handle supporting, reverting a bootable volume in Nova. And so if we support this, then the problem is how do we prevent a data loss? If we are reverting, if we are booting from a volume, that is reverting. We talked a lot about this, but it all needs some changes from the Nova side. There was also concerns on how to handle reverting to a Snapshot taken before the volume extend. So if we do support that, that means we could be reverting to a Snapshot whose size is smaller than the original volume since we don't support shrink volume that could lead to data loss as well. Also we talked about how to provide a generic implementation to support backends that don't have the reverse Snapshot technology. So we talked about several options. It all depends on the Snapshot technology used by the back end as well. And another concern is how to prevent data loss if there is a crash in the service in the middle of a revert. So we spent a lot of time talking about that. We could not reach consensus at the PDG. The decision was to continue with the SPAC review. But since the PDG, we actually made a lot of progress during the SPAC review. A lot of concerns have already been addressed and the SPAC was already merged. So we talked about earlier that now Cinder has active, active HE support in Cinder services. We want to get the drivers to test for active, active HE support. We will be providing a document to instruct the driver on how to certify driver with this support. Driver maintainers should follow the document and test active, active HE in the drivers. We also talked about API microversion. After API microversion was introduced in Cinder, there were concerns that we have been bumping the API version so many times, especially when there are a lot of people working on API, it could lead to a lot of merge conflicts. So should we just bump it once before each release? We brought this up at the API working group discussions at the PDG. The decision was that we should bump the version when a bump is needed. So basically, if there's a change in the request or response of the rest of the API, then a microversion bump is needed. We should be following the API working group guidelines and we should not be break with the model. Another concern is it's hard to tell which specific API version is needed for a specific API. So one action item was to fix up the doc strings so we can have the API version information in the API documents. And we talked about deprecating the V1 and V2 APIs. The V1 one we mentioned, that was back in Juneau. And if you know our typical deprecation policy, that's been deprecated much longer than we normally allow. The biggest concern about that was even if we gave just one extra cycle beyond our normal deprecation policy and removed it, there still are a lot of clients out there written against the V1 API, or we assume there are, because we don't really know. We don't know what's actually being used out there. There's custom code written within customer environments calling these APIs and we just didn't have enough knowledge and the ability to go out and talk to any of these consumers of the V1 API. So it's been deprecated since Juneau, which is really quite a while now. We are hoping that now we can actually remove that out of there. So really, again, please upgrade, switch over to using the V3 API. We hope we don't have to keep V2 around as long as we've kept V1, but it would be very good for the Cinder code base that there's a lot of code that we could cut out of there, simplify things and usually less moving parts, the less likely things will go wrong. So I'm really hoping that we can clean some of that up. And really one of the last things at the PTG that we talked about is just internally, we've had a really strong policy towards having to run CI for any volume drivers that are part of Cinder. We still are behind that idea 100%. We need to have CI results so that when we ship something like Okada, we have some level of comfort that whatever is in the Cinder tree is actually going to work if you try to configure it and use it against your storage. So our only way to do that is with CI. So if you're not aware of that, the CI continuous integration, any time a patch is submitted against Cinder, it's up to all of the vendors that have storage devices that are supported by a driver to take that patch, stack it up, run it against their physical actual hardware, well it doesn't have to be physical, but against their system and verify that in an actual deployment it will work. And that reports back. So we know right away, the best thing I think as far as the developer goes, we know right away if we do something that inadvertently breaks somebody, but then we also get the other benefit of knowing that they're still in good shape. So one of the things that they wanted us to talk about was queendance and beyond. Like I showed on the timeline, we're right in the middle of the pike development cycle right now, we're still talking about a lot of things that we want to do in pike. For me, it really seems a little premature to talk about what we're gonna focus on in queens and what we're gonna focus on in the next release after that. So here's what I think our focus will be, pretty much what we were for pike. There are a lot of these ongoing efforts that we keep having to talk about every time, but they are things that we gotta make incremental steps towards in each release. I think we have enough on our plate right now that we probably will still be working on some of those. So I don't expect too much to really change here. We're doing things like the group replication. That core functionality in Cinder is hopefully going to make it in pike, but that doesn't mean that everyone can start using it. We need to get the support on the back end with all the drivers to actually be able to set up that replication. We have the active active HA that landed in Nokata. Like Zwing mentioned, we now need to have the drivers test and certify. And certify might be too strong a word. There's not a certification process, but we need to know that with multiple services running and multiple requests coming in, that's not going to cause an issue for any given driver. And the multi-attach piece, yeah. We're making progress. We're gonna get there eventually. Yep. So part of this is reaching out to you and hopefully encouraging you to reach out to us. You don't need anything of this here, but through the mailing list, through IRC, through the hallway as we're here after this, if we actually have some time left, we'd love to hear any feedback on the features that we've gotten out there, feedback on what kind of issues you're running into, things that it doesn't look like we're addressing, that we need to be. We need input, and that's the big benefit of having the forum and the summit here. We really want to make this about getting the developers in front of users, other developers writing other areas and making sure that we're talking and closing this loop and getting the feedback that we need. So any of those, if there's an area of your interest, let us know. So a lot of this can be found if you need some night reading material. The release notes, there's a couple of releases ago. We added a new way of capturing release notes, a tool called Reno. So before we would change code, and then we'd have to remember write that down somewhere so that we could compile our release notes. Now we actually capture that right in the code. So good and bad, it's all there, but now if you look at the release notes, it's all there. So if you'd like to, if you want information, if you can go out to this link and it's got any significant change that we've made there, hopefully we've captured it all. Like I mentioned, getting feedback, if there's some issue that you have, feel free to file a bug report. Even if it's not a bug in Cinder, if it's an issue with how you've configured something, that's a great way to get it to our attention and then we can respond on that bug and let you know. So that's the launchpad.net slash cinder. There's a link along the top for bugs, file a bug. If it's a bug, we'll try to fix it. If it's not a bug, we'll try to get some feedback to you and let you know. There's some details about Cinder if you want to get involved. If you just want to learn more about the internals of Cinder, our Wiki site has that. wiki.opensec.org, pretty much every project has a page out there. There's a Cinder page. And the last thing here, a few releases ago, I think around, I think Tokyo was the first time, we started trying to capture all of our design summit sessions and discussions and mid-cycles and things like that. Actually record audio and video of that and we've been posting it to a channel. You can actually subscribe to the YouTube channel if you look for, I think it's OpenStack dash cinder or OS dash cinder or something or follow this link. You can actually subscribe to that YouTube channel anytime that we have a discussion. We actually live stream it there so if you can attend something like a PTG, you can do that. Sometimes the sound quality's not great. We don't have a full AV crew but Walt's been a great pack on all the equipment to be able to at least get some video and audio captured and it helps us because then we can go back later and say, what did we say? So that's it. We only have a few minutes left. Thanks for listening. If you do have any immediate questions, please come up to the mic and ask. Otherwise, we'll be here all week. Thank you. One, one, two, three. Thank you. If you're looking to get involved, now's a great time. We need reviewers. We've been, there've been a lot of changes going on. Things have gotten a little quieter and if you're looking to be able to make a noticeable impact on the group, we appreciate anybody who can come help do reviews so that's why I'm gonna mention that as well. Thanks. This was great. Thank you guys. I had a question about the multi-attach story. You made a couple of references in the talk about how this has been a long time in the works and at one point, and I want you to help clarify this because this is what I heard, you are slowed down because when the people implementing that got done with the implementation in Cinder, they went and looked at what was going on in another one of the OpenStack projects and said, oh, we have to do a bunch of work over here in the way that we operate. That's what I heard. Was multi-attach slowed down at being a Cinder feature because you guys had to go do some work in Nova? It got slowed down as far as being a complete solution. So we did make change. We actually landed everything in Cinder. We had something that we thought worked for multi-attach, but for that to actually be useful, you need to be able to actually use that Nova to attach to instances and Nova needs to be involved in that process. So for a complete multi-attach solution, that does require changes in the Nova code in addition to the Cinder code. So if you're trying to use Cinder by itself, that was okay. We basically had that. But if you're trying to have a full Nova Cinder Glance OpenStack environment and be able to attach the volume to multiple instances, that was the piece that we kind of hit this roadblock and realized we needed this extra work. Yeah, good question. We're actually out of time. Is it a really quick one? I don't know. I don't know. Okay. We're in consideration with Cinder, but we chose and we moved into version three along the trend of this conference that we're trying to reduce its complexity so that we can gain further adoption. So that probably is a bigger discussion that we are over now, so we won't go into too much. Microversions does help with that to some degree. We can start to remove some things and deprecate some things that we don't want and try to simplify. And you can write code targeting a specific microversion of the API and know that that is always going to remain consistent with the way it works. That's probably all I can say right now because we are over, so thanks. Thank you.