 Hi everyone. I'm Xin Yang. I'm a technologist from the CEO office of Dell EMC Working on open-stack contributions. I'm a core reviewer in Cender and Manila. I am Jay Bryant And I'm the Cinder subject matter expert from IBM And I'm Sean McGinnis from Dell EMC and I'm the current PTL of the Cinder project Okay So we're going to go through Give an update today of the work that was done in the Newton release I'm assuming since you're here everyone probably knows but Cinder is the block storage service of open-stack. It provides the software API abstraction to be able to manage storage no matter what type of actual Storage device you're using on the back end. So what happened in Newton? We're going to go over our back-end drivers Every release we have different vendors adding support for storage devices Talk about some of the features like async messaging replication Some of these are newer things some are Kind of ongoing work that we've been probably talking about on every update for the last few summits But just things that have been an ongoing effort to get functionality added in Things like active active high-bill ability and then we're going to talk a little bit about our clients We have OS brick our Cinder client and extensions for being able to to do more direct Non-nova more standalone Cinder functionality with a client So every release we get more and more support for different back-ends They just keep coming in every release. So we've got have quite an extensive list now In the last in Newton we were able to add the list here. I won't go through all the names Some of these are just individual devices some are multiple different protocols And a few of these vendors are existing Cinder They've been involved in Cinder and just adding support for our new models So with the addition of these as of the Newton release, I believe this is updated as for our for our final Our final deadline of all the drivers These should be all of the back-ends available in Newton So hopefully if you're using if you're standing up a new cloud or you have existing infrastructure that you're using You your vendor of choice is on this list or if you're looking to deploy a greenfield deployment You can take a look here and see what your options are I do call out a few of here that are Have an asterisk next to him the the LVM driver is the official reference driver of Cinder So if you are a developer working for a storage company, that's thinking about adding a Cinder driver That is the really the best place where we direct people to take a look and see How that driver needs to be implemented? Depending on the type of storage you have there's there's some slight differences So I've highlighted the NFS driver and the sef driver here just because of the type of that Protocol technology things are done slightly different. So if your storage Is a little closer in line to those you can take a look at those drivers as well as good references to see how to implement that functionality so speaking of drivers in Newton we tried to do a little work to make sure that Not only are the drivers consistent and Have third-party CI so that we have some guarantee of at least some basic reliability functionality we added a additional interface checks for all of our drivers to make sure that you Every once in a while we ran across a driver where we added the driver, but didn't find out till later that some piece of it was missing So we have a little more Kind of compliance checking for those drivers just to make sure make sure it's automated make sure that nothing can get in Without us noticing So it's not on it's not up to us or to someone doing the review to notice that we have that as part of the checks that happen when that code is submitted and make sure that we find out right away if There's something missing We also kind of changed our policy around Drivers To give a little background For the last several releases. We've required that in order to have a driver in the cinder tree That vendor or that whoever supporting that driver needs to have third-party CI running You know what that means is anytime any patch is submitted to against cinder whether it's touching that driver code or something completely unrelated That needs to kick off a set of tests run by that vendor against the actual storage device and Make sure that everything passes We do event, you know once in a while catch cases where something that seems completely unrelated Makes a change that ends up breaking a driver So third-party CI is a way to make sure that we catch that right away and don't let that in So with the policy of requiring that what we had done in the past is if a vendor is Unable or unwilling to run third-party CI anymore. We used to remove them from the cinder tree pretty much immediately and There's a set of tags at each project an open stack Says or states to declares that they follow and and one of those takes is follows deprecation And and the deprecation policy for open stack is that any functionality that exists in a project needs to be marked as deprecated for one release Before it can be actually removed in the next release so You know that these aren't necessarily a feature of cinder, but you know drivers are a really core piece of the cinder service So just removing a driver immediately for not being able to follow the CI policy Actually wasn't following the intent of that tag saying that we follow our deprecation policy We would just take it out of there and you know if you were an end user and running a cloud on one of these drivers and You weren't aware of this and you ran the upgrade of the packages for the distro You know you could be in a bad spot where suddenly your drivers no longer there So what we did to address that we you know we we don't want to back off on a requirement of for running third-party CI So we still need some way to enforce that well So to make it a little easier transition and being able to follow that deprecation policy We now just set a flag on that driver That that marks it as an unsupported driver and maybe it's supported by the vendor But as far as the cinder community goes the open source community To us that's an unsupported driver We have no way in knowing as a community whether that actually works or not because we don't have the vendor hardware So we now mark it as unsupported. There's a new option that needs to go in cinder.com That you need to explicitly set To enable being able to run an unsupported driver because we want to make it very clear So that it can't be easily overlooked and then you just run into that problem the next upgrade so we flag a driver is unsupported you Need to make that change in cinder.com and this does make it a little easier if that vendor had some issues and we marked them as unsupported and Now they're able to get back involved in the community and start running their ci We can just change that flag and makes it a lot easier rather than having a whole New driver submission or having to revert all those changes So it ends up being a little cleaner a little nicer on the end user definitely if you're using those back ends and Makes us makes it so we still have a way of enforcing our policies So next I will pass off to Jing So when a user performs a volume operation such as create a volume and and if the operation fails The user will get the status but in order to find out why it fails User will have to dig through the logs, which is not convenient So the purpose for this user message feature is to provide the user with detailed failure messages If the operation fails and then present it in the horizon in Newton We have added ability to list show delete messages The message list can be filtered by resource you your ID resource type event ID and so on And we also have our internal API that you can use to generate a message and save it to the database There's no more work needs to be done to add more messages and make them available in horizon So by default the scheduler will always choose a backend with the highest weight in Newton now we have a new option called the Scheduler weight handler by default that this option is set to ordered Host weight handler it will always choose the host on top with the top with the highest weight but if you change it to Stochastic host weight handler. It's going to make a random selection And there's another change in the scheduler that will affect drivers that supports think theme provisioning support and thick provision support in the same pool previously if the driver supports both same provisioning and sick provisioning support and If the user wants to create a sick long the scheduler will use the logic for same provisioning to evaluate Which could potentially lead to over provisioning so this fix is to add a check in a scheduler to To check the volume extra steps One extra specs if the provisioning type is set to thick then the scheduler will use the Free capacity to to see if there is enough space to create a volume So in newton, we have not done any update in replication API Just we just to have a few more drivers that have added the support. They are Caminario netapps CDOT and solid file drivers So we added a generic volume groups construct in newton So what how is that different from the consistency groups construct that is already in sender? Consistency groups in sender currently only support consistent group snapshot so when we are trying to extend that to support replication group we found that there are limitations because Backends have different requirements some back end can support consistent group snapshot Well other back ends they want to be able to replicate a group of volumes While not maintaining the right order consistency So we couldn't use the consistency group construct to support both And also if a application has multiple volumes It will be convenient if we can group them all together so that user can manage manage them together So that's the reason that's the reason why we added generic volume groups And eventually consistent groups will be migrated to use the generic volume groups similar to volume types and extra specs we added group types and group specs To describe a group only admin can create delete and modify a group type Only admin can set key or unset a key for group specs and view the details of the group specs We now have the ability to create delete update show and list groups in order to create a group you will have to specify a group type And also one group can support multiple voting types So other than the group type you also need to specify the voting types That are supported in this group when you create a group if you want to add add existing volumes to a group or remove the remove volumes from the group then you use the group update command To delete a group if the group is not empty you will have to specify the delete volumes flag So that will delete the group as well as all the volumes in that group If you just want to delete the group itself, but keep the volumes around then you'll have to use the group update first To remove those volumes from the group then you can use the group delete to delete the empty group We also added APIs and CIS to create delete show and list the group snapshots So the group snapshot can either be a consistent group snapshot or they can be just a group of Snapshots that does not have consistency the driver That supports consistency groups currently can add that support using group snapshots By using group types. You can differentiate between different types of group snapshots We we can also create a group from source group or create group from a group snapshot As I mentioned earlier, we are going to migrate consistent groups to generic voting groups So it's going to take a while for that transition to be completed Because this change involves some data migrations We'll need to migrate data from one table to another and eventually Drop in the old table So in order to support rolling upgrades We will have to it's going to take at least I think three releases for this to complete We will provide migration scripts for that. So this is a working progress and we're hoping to get this done in Okada We also have planned to implement a replication group Based on generic voting groups. That's also working progress but because Okada is a short very short release cycle, so We're not expecting that to be done in Okada In center, you can manage volumes and manage snapshots but in order to run that command you will have to provide a Identifier for the volume and snapshot Now to get that identifier, you will have to go to the back end to find out. So now we have this New API to list all the manageable volumes and all the manageable snapshots So and if you list them that then you can take a look and decide what are the volumes and snapshots you want to Bring into under Cinder's control in Utah we deprecated the ability to Define the back end storage in the default section in Cinder.conf If you still use this option, then you'll get a warning the recommended the recommended way is to define them in the back end specific section in the Cinder.conf So we also added the ability to retype between encrypted volume types and non-encrypted volume types and we also Have castelan key manager interface now Castelan is Implemented by the Barbican team is a generic key manager interface right now only the Barbican driver is the Recommended I think production ready driver for for castelan We also added API to update backup a backup's name and description And also now we can delete multiple volume metadata keys using one request We added the e-tax in the API course. So this is a performance enhancement We also we can also query volume Query volumes filtered by Glance metadata key value pairs To use this option you will have to use add Glance metadata to query volume filter option In order to enable that We have also removed XML API That was already a deprecated back in Mitaka Now I'll hand over to Jay. Thank you Shane Okay, so the quota related changes this is a little bit different than some of the other items This was actually an item that was a bug that came in when we added the ability to do higher hierarchical quotas nested quotas in Projects so previously if you were using non-nested quotas if a admin wanted to come in and modify those quotas For the projects they'd be able to do that But when we went to the nested model We had problems that if the admin was not a member of the project They were no longer able to modify quotas for nested projects. So Fixed up the code to check if it's an admin user and then allow them to modify quotas for all the projects on that That system as had previously functioned So not so much a new function, but getting nested quotas working more the way that Quotas had previously worked and that's the wrong button We'll just do this there we go Technical difficulties So that the rest of the things I'm going to talk about here are Items that are continued work in progress Items that we've been working on over multiple releases give you a little bit of an update on how that's going Been doing a lot of work with the active active HA code and trying to get the the functionality of the the volume process Setup so that we're able to have You know multiple volume backends running that you can fail over From a control node perspective so that you know if you've got redundancy on your back end storage Obviously, that's good, but we also need control plane redundancy as we're moving into more enterprise type setups So there's the addition of a new cluster entity where you can indicate which volume back end which volume processes Are clustered together as a a failover unit for high availability? And then adding the commands that go with that and able to to be able to see you know What is the current state of? Those those clustered entities, you know what what what option what backends are clustered together And and that sort of thing so that you can manage your high availability highly available environment There we found in the process Gorka's been doing great work on this lots of API race conditions As far as you know in in consistency groups and when doing migration or retyping extending volumes So there you know analysis and changes that went in to help avoid having race conditions in that code Had to add in the ability to do you know remote locks between between the two clusters So with twos locks added that and then start and stop coordination of the volume service and Gorka's got a great Blog post out there that we have linked on on how you can test Active active and give it a try and if you find issues Please let us know so that we can continue to improve and develop that Multi-attach we actually were just talking about this this morning If you're a fan of cinder and Nova, you know, this is something we've been working on for a long time and so we're we're Been working on improving the APIs to be ready on the cinder side so that we can can Attach one volume to two multiple instances Have them access the same volume On the cinder side, we've got a lot of the plumbing there to do it We're cleaning up the plumbing so that the that it's easier to follow the code path And there's less likely to be issues the challenges on the Nova side Depending on what kind of volume connector you have we have to figure out okay some of them There is for each instance that you attach the volume you have a new connection from Nova standpoint So when you're done with that connection you disconnect it and you know you're done with it And you don't have to worry about it But some of the the back ends support shared connections So you have if you have it attached twice you only see one connection on the host But really it's being used by multiple instances and then it gets complicated from a Nova standpoint to know whether When we disconnect well is that the last person using it or not? So we're working around We have a number of proposals We're working with the Nova team to try to find a solution there that will will work and allow us to do Multi-attached so all of these patches are out there under review We're gonna continue talking about them here at this summit and hopefully hopefully That's something that will we'll go in here in In Okada Rolling upgrades So again, this kind of goes along with all the work to be highly available and working well in an enterprise environment Need to deal with the fact that if you want to keep your Cinder environment up and running while you're upgrading to the next release you need to then Insulate the the volume process and all of cinder from changes that are happening to the database in the back end as the upgrade is happening So there's been a a slow process of moving all of our our different Bits of code all the different objects to a versioned object And what that lets us do is take data and and package it up with the functions That that data is used by and then that insulates it from schema changes on the back end database So These are the the portions of code there. We got updated And we will continue to work through that so that we can get to a full support of rolling upgrades added the new version 3.0 of the RPC API, but it's still backwards compatible with with the 2.x versions So this we're really pretty close to having it all done But like with the active active HA set up There's a lot of testing that needs to be done to make sure that it's really really working and that we're not going to push anything out that That could lead to issues in the future Brick so we continue for those of you that aren't familiar with brick This has been an ongoing project to create a library that instead of having attach code in Nova and Cinder We've created the OS brick project, which is a library that then can be accessed By both Nova and Cinder and we only have to change attach code in one location so you know over the last couple of releases we've been moving to that environment and Now in this release we added more connectors so greater support GPFS now has its own connector so each of these connectors Have support in in brick for any of their special options that they need When they they connect up to a host and so these were added So that they could could support their individual functionality In this release and then also we used move to using privset for privileged operations This was to catch up with Nova and and use the same security solution there for this code and That's something to be aware of if you were running on master at the time as the switch happened you could see some Unexpected changes depending on your environment most people this should just be smooth and Not notice the difference, but it's something to be aware of if after your upgrade you see different functionality in your commands that run as root and Then in the python cinder client just for your awareness. We've really kind of talked about most of this before It was just the addition of the commands to support doing the cluster commands for HA active active support the commands for accessing the message user messages for async jobs and Make you aware that the cinder endpoints command has been deprecated So that that keystone related command is no longer is going to be be removed here and then the new Commands for group types group and group snapshots. So With you with the latest cinder client. We got all that rolled up and pushed out and finally Sean mentioned this a little bit earlier the work we've been doing with the standalone python Brick cinder client the ability to use cinder from from an ironic node not to have to have Nova there in the mix added the ability to actually do a volume attach and detach there so Getting closer to to sharing functionality for cinder on ironic nodes And at this point, we'll hand it back over to Sean to talk more about what's coming in Okada Thanks, so as you've probably heard at least hopefully at this summit The schedule is changing slightly for releases. You know, we've had the six-month cadence forever and As we try to restructure how some of our design activities happen That means that the Okada cycle Really only goes until the end of February And now counting holidays things like that We we really don't have a lot of time in the Okada cycle because of that. I Decided for this release. We had enough kind of long-term feature implementation activities going on You know, we have a few bugs outstanding there so Since it really wouldn't have been enough time to to really jump on a big new feature We're trying to make the Okada release for cinder be mostly about bug fix and stability Going through what issues we have out there. How can we fix them? How can we make things better? so For Okada, there really won't be too much new it'll mostly be You know just a better product better more solid service at the end of Okada That doesn't mean that we're not looking at Both finishing the futures that we've already been working on things like active active high availability This has been a long-term process to try to get all of the those changes in place We hope to finish that in Okada rolling upgrades the rolling upgrades are Pretty much complete the only reason we still have any hesitancy to just say yeah this rolling upgrades are good Just we want to improve our automated testing around that make sure that we have everything covered but everything's in place and As we mentioned before the multi-touch activity with Nova changing You know that whole effort just how it Really highlighted some of the issues we had interacting between Nova and cinder We're really hoping that we can do something for that in Okada You know maybe we might end up being one of those that we talk about again at the next summit But we want to iron out how that interaction works and really come up with a solid API solid plan for What's going to work well between Nova and cinder so that we can actually implement this multi-touch support? with a sane API so well Here's a related links, but but on the topic of Upcoming things beyond Okada. I just wanted to say we are discussing that we are seeing What other functionality we need to start thinking about for the pike release and beyond that? We have some ideas as a cinder team But definitely this is a great time great opportunity if if there's some functionality in cinder missing that you need for some to meet your use case Please talk to the team Go on IRC send to the mailing list Track me down We'd love to get any feedback from operators to players. What kind of issues are you running into a cinder? What do you need in that service to be able to to give a good solid storage API for your cloud? so with that I'll ask Jing and Jay come back up on stage and we'll open it up for any Q&A we got it all covered Thank you Yeah, so so the question was if we have a driver with That doesn't currently have consistency group support. What should that driver vendor plan to do? Yeah, I should just use the the new CLI the new API is actually that there is a driver interface for that Just implement the new driver interface Yeah So you can you can you know In your driver, you can use different group types where you can say you can report the consistent group snapshot you could do true and you can You can use that to add the support So you're in your driver you can just Call your special API that create the consistency group snapshot and you can actually support both so You can depend on the type. It's either Consistent or you know the just a group of volumes and groups of snapshots that are not consistent You can support both depending on the type that actually I don't know if we have new vendors or vendors in the room that are considering adding drivers, but You know given how quickly the list is growing. I bet you there's a decent chance of it So we still have the you know something that hasn't changed Here in Newton is you know, we have the list of Expected functionality that every driver has to implement to be included in Cinder. We have not added additional function to that since gosh Probably ice house now You know, but but one of the questions I get internally from all my IBM driver people is What do we need to add? And really, you know, not every back-end supports consistency groups or the other functionality And it's really up to the vendors as to when these new functions are made available in Cinder When they want to implement it and and that's you know, we've got a matrix out right now If you search for it that shows the least how we understand the current state of Drivers and what functions they support, but that's definitely an area that we can Continue to improve on but I would I would ask back to you You know if you add consistency groups for your drivers make sure that that's that's documented and indicated so that we know what each driver Is supporting other questions That's a good question. Thank you. Well, thank you. Thanks very much