 Good afternoon, everyone. Welcome to the session titled Manila share does not simply move and protect itself. Oh wait, it does I'm Rodrigo Barbieri. I'm a core developer in the project Manila and I'm a driver developer for retouch data systems Hi, and I'm Gotham. I work for NetApp. I'm an open stack contributor. I'm also the API working group players on for Manila Today the storage is one of the most important pieces that comprises any cloud which in turn must provide indispensable properties like 24 7 availability It's a major concern that the hardware and software That comprises any cloud is able to offer always on availability So when you have this hardware and software that needs to offer Availability, you would like to test them over and over again to make sure that these Disaster recovery mechanisms are failsafe and they're and they're most importantly reliable So one of the easy one of the things that we'd like to provide is the ease of performing switchovers and switchbacks failovers and failbacks any number of times, right? and this determines That this determines how You know resilient your cloud is in the face of any failures and how amenability is to any maintenance and lifecycle operations In several cases Administrators have to develop and perform maintenance on several complex scripts in order to to maintain Protect recover and move data in order to minimize downtime And so when you have a cloud that's running multiple storage vendors, right? So and you you you know that this vendor is running Snap mirror and this vendor can do recovery point and this this is how you move data for this vendor And this is how you move data for this other vendor But what happens when you want to move data from this vendor storage to this other vendor storage, right? And there's also this concept of Oops, sorry. I'm a little messed up with my slides today Okay, so there's also this concept of being uh, I mean managing all of these without having to worry about all of these Intricacies that go with specific storage systems. So many at times cloud administrators are probably going to switch to slower And more inefficient ways of talking to all of their cows with the same language So in heterogeneous clouds, there is they are running Storage from different vendors these scripts become even more complex because they have to Maintain and provide interoperability between those different storage vendors Okay, next one of the solutions is using manila and Motive vendor storage abstraction that works So for many releases Cloud operators have tried and experienced the simplicity that open stack monologue brings to their clouds, right? The the self-service storage abstraction that that abstracts away all of the complexities that that you have to deal with Individual storage vendors, right? Many many your clouds are just one storage vendor. That's all right But that's still dealing with having to write apis api calls to these storage vendors And the service architecture that manila has is something that cloud operators are already familiar with That it has this it has an api. It has a scheduler. It has multiple managers. It talks to a database It talks to a message through a message queue and so on So what manila can bring to you and what manila has been bringing to you is reducing all of this back end Storage complexity and delivering all of these features that you still care about through a consistent and a restful api Manila allows making use of several vendor specific features by mapping these features and their back ends to share types It also allows the creation of shared file systems Access rules creating snapshots and deleting snapshots and many other operations So now let's switch gears and discuss an important feature that we added in manila in the recent releases We're calling this share replication So there are several approaches to increasing data availability in your clouds Right. So in the face of hardware software or even site failures So mirroring provides a mechanism that You can facilitate this data availability across multiple availability zones And that's the concept that manila is trying to bring to open stacks for shared file systems Failure is definitely lurking in your data centers, whether it is, you know, loss of power Failure because of loss of cooling or there's there's been a calamity. There's been network connectivity loss Or there is just an operator error at the end of the day So one of the biggest features that this feature can bring to your clouds is data protection for the specific use case Of disaster recovery. Of course, there are other use cases for replication that we'd like to solve Right. So just you might be mirroring your data for other use cases It's just creating these data stores that would allow you to to have continuous Application development by using your mirrors or you could be maintaining you could be load balancing your reads on your replicas, I mean on your source shares by by directing some of the reads to your replicas right and there's also the the concept of Reducing any performance latencies by totally turning off the reads on your source share having only your output Directed to your source share and reading off the replicas for any test and depth sort of use cases So replication in manila is designed to solve these use cases And it has three different flavors that we expose through the rest api dr readable and writable dr stands for disaster recovery It's for solely the purpose of disaster recovery. Therefore your replicas are managed by manila safely until after A disaster has occurred that is they are not exported. No one sees them You know they exist, but they're not exported until after the disaster has occurred readable on the other hand right we we when you create a replica we export the replica as well So you can go ahead and mount the replica And you can start reading from the other replica But of course write operations are disallowed on any secondary replicas that are there And writable is meant to allow simultaneous writes, right? You you have a uh, I mean you can think of this as a Clustered share where you you probably want to write from multiple clients and you want to synchronously replicate between them And manila makes that possible as well Today we have driver support for two of the three kinds of these replication flavors that we have We support dr and readable today in code and for this first party driver support for readable type of replication Yes, it's limited in this in the sense of being driven by manila as such and we have plans of improving it One other use case that we currently don't support is this is the one with share servers and we have plans of adding that very soon So one of the most attractive Attractive features that share application in manila specifically brings to you is that this is enabled on Your tenants for your tenants, right? So these this feature is visible to your tenants So users can actually go ahead and manage their replicas as well as they're managing their shares currently So this allows for users to write applications that are aware of their disaster Protection, sorry data protection strategies and so on Of course, you don't need to consult the administrator We're taking away one one more level of interaction over there to talk to your own replicas and so on and you can Test your disaster recovery strategy by fail performing any number of failovers and failbacks So we we create this ring of replicas and you can always fail over and fail back And you can you can test this even before you actually go hit that disaster someday Manila also ensures that both your data and associated snapshots of a given share Are replicated to the destinations and you can definitely always continuously monitor the health of these replicas as you see them So when working with share application in manila one of the most important design Considerations that we kept in mind was the fact that we'd have to interoperate with the The concept of availability zones across your open stack ecosystem Right. So availability zones in manila are still tenant visible And they can they can manage and control their shared file systems alongside their compute their network and block storage To help cloud administrators, however gain one other level of granularity in determining What backends can they configure for replication? Right. You can have 10 different kinds of storage systems in your in your cloud Right, and you would want to expose the goodness of this storage systems And you know that this storage system can and you're going to be configuring the storage system to replicate with this other Storage system and so on. So we introduce the concept of a replication domain So it's a configuration parameter, which you could you could basically have I mean you can configure two different storage systems to be part of a replication domain That just means that they're going to replicate data between each other And that if if storage systems can't replicate To each other, they're not going to be part of the same replication domain. Of course, these replication domains are not tenant visible So let me take you through an example use case for share replication I'll be using the replication support that we added to the manila ui plugin in newton And we will begin with this architecture. So we have a cloud. We have two availability zones Right on this slide. I've called them az1 and az2. But in my example, they're going to be called barcelona and mudrid and We're going to let's say we have a net app back end and az1 and another net app back end and az2 and we We know they're going to be set up for mirroring. So We're going to configure them in the same replication domain And our use case for the manila share in this example is to run an enterprises logging strategy, right? So it's we're we're going to be using the elk stack elastic search logs log station kebana to show you this. So Let's begin See that. Yes, you can All right, so let's log into the horizon ui and create a share. This has been sped up a little bit and We're going to pick up a share type that can support replication And we're going to create our share in this availability zone and It's a configured availability zone called barcelona And let's go ahead and create this share And let's allow access to this share to the client that we're going to mount the share act Right. So it's as easy as saying, okay add an access rule. It's an ip access rule This client can do reads and writes on this access rule. That's my client ip And let's go back to the share and pick up its export location And Let's go into our client which is This one here and let's mount this share It's me mounted to this mount point called logs. This is because I'm going to store my logs in this mount point It's mounted right now. Let me go ahead create a couple of folders in this mount point I'm going to call it a a folder for logs and a folder for indices Right. So I'm going to pipe all my logs into this logs folder and I'm going to maintain my elastic search indices in the index folder So I'm going to just run a script to pull all these logs from my ci because I have no application that can generate a volume of logs like this That's my ci systems logs being piped and I'm going to start running Logs let's take a look at the configuration. That's my Mount point and it's configured to the ci logs folder and my output is going to elastic search It's running off the same machine on that port So let's go back. Let's look take a look at what elastic where elastic search is putting its indices again. It's called data node, I suppose or path to our data. Yeah, there it is. So I'm going to Point my log folder index folder at that. So that's where the elastic search is going to put all of its indices So let's go ahead and start elastic search And start start stashing our logs Since we've redefined this and we're just going to use log stash off the command line like that and when you do Start stashing your logs, you're going to see the output. I have not piped the output So you can see that brilliantly go up and down there All right, so let's go into the ui that we have configured for this. It's called kibana And let's just perform some simple queries on this, right? So Looking at logs for the last seven days and let's just look at all the errors that could have happened Well, that's our ci. So obviously there are a lot of errors that we intentionally caused and Let's make make a little more You know a little more complex of a query We're just saying give me all the no valid host errors Where all the marla scheduler failed to find a valid host about 33 hits not bad That means we have to write more tests Okay, and then we go back to our ui at this point and let's create a replica It's as simple as going back to the ui and saying all right create me a replica here My share is in Barcelona. So let's create a replica in Madrid At this point i'm going to pause because i'm going to talk about what we are going to do after this replica is created Right, so this replica starts out and it's in out of sync status It's going to start syncing from the source share at this point. We probably have about Gigabytes of some logs that that are that are getting piped and stuff and it's once it starts syncing It's going to continue on some sort of a cron schedule That's what that's all back and dependent marla is not handling that but marla is letting you know that this This syncing process is in sync or out of sync out of sync meaning there's something that could have gone wrong And you know it's not being updated and we we regularly poll for the health of these replicas To simulate a disaster i'm going to bring down my data center Not by doing anything this thing actually by just pausing the nfs service on the Source back end. So let me go in right, let's wait for the replica to go in sync and let me go into the system manager and disable the nfs service This is a disaster My logs have stopped stashing everything is stopped and then kibana tells me oops can't connect to elastic search I'm down right so logs have all gone end of the world Let's go ahead and promote our replica. It's as simple as saying set this replica as active and There we are we start doing it and marla goes sets this share into a replication change You can't do anything to this share while it's in the replication change Let's pick up the new export location that gets generated And that's log stash puking all over my screen Let's go back and unmount the old replica. Let's do this a little gracefully And let's go ahead and mount the new share. Of course, all of this will be scripted in your environments. I'm just typing them out for ease of use And let's look into this mount folder. Of course, I'm not going to validate each and every thing is there But of course, we're going to start stashing the logs again. We'll see how that works We have our logs. We have our index folder so All right And then we will go back and we will start the log stash Thing again. So we have we know we know our source is still writing. So we we can just go ahead and stash it so pointer of the config file log stash picks up where it stopped And There we go again At this point, let's pull up the user interface Boom it's back. So indexes are all fine and healthy. Let's just search for something These are all the times that monola actually created a share 522 25 times All right, so that was How simple it is to use share application through the ui Oh, sorry Uh, let's talk a bit about share migration now Share migration is a feature that allows an administrator to migrate a share from a storage pool to another storage pool so In which case is uh, this share migration coming hand. Well, whenever you're uh, the administrator is optimizing a hardware Tagging in the flying perform taking down for maintenance. Uh, optimizing loading balancing The the share migration feature can migrate any share contained within to another another storage um The the user may also, uh, may also, uh It's the previous one. Okay. The user may may May also want to the the share to be closer to his location for beggar performance. So that's that's another use case and Beggar yet share migration allows The administrator to migrate a share from one vendor storage hardware to another Storage hardware from a different vendor So with with share migration comes its own benefits and challenges and Whenever The administrator is trying to perform a share migration. Uh, it may be necessary to change certain properties that are Are available in manila such as driver modes share network share type and availability zones And the in in order to migrate or share to your destination of choice and the share migration api allows The administrator to change these properties when when performing a migration Also share migration May a performing a migration may disconnect some users That are connected to to the share And that's why we designed this feature to be To have a two phase approach one phase is the one that copies files and the other one that's We call cut over phase. It deletes the source shares and and Uh Performed export location updates is a network paths updates this Possibly disconnecting the clients Two flavors of migration are supported one is the driver assisted migration where the vendors can expose their advantages such as more more efficient ways of migrating shares And a host assisted migration that is able to provide To provide The capability of migrating shares from a vendor to another vendor So this is what an administrator would see when clicking the migrate share button And here we could see that the first field to input is the destination pool field And then we have four parameters. The first one is to force a host assisted migration Which is best suited when It is migrating from Sorry migrating across different vendors And the other three parameters Are used for the driver assisted migration in order for a vendor to perform A share that sorry a migration that remains writable during the whole migration Also the capability to preserve all File metadata during migration And also to do so non-disruptively and We also see two drop-down lists in order to select New share networks and new share types that may need to be changed in order to migrate your share to your destination of choice So when the the The administrator clicks starts it starts migration It will start the first phase in this phase the share remains accessible and It may it may be accessible only in read-only mode depending on whether the The driver supports a writable migration or the host assisted flavor is used um This this phase is also cancelable, which means that the administrator can go and Decide to cancel this migration because it may take a very long time hours or days depending on the amount of data and he can also Obtain the progress of migration so he can have An idea of how long it's going to take to complete And then after the the data the data cop phase is complete We proceed to the cog over phase This phase is expected to be disrupted But it may not be if if a vendor is able to To implement a non-disruptive migration And then depending on the circumstances like same storage pool or same back end or different availability zones Uh This this phase separation in order is Implemented like this in order to Provide the convenience to the administrators to choose the the best moment to to trigger this phase In order to minimize downtime Uh, so what magic is that that can uh migrate shares between different storage vendors? Well, it's no magic It's just uh, um automatic a data service a dedicated data service That works like an automatized administrator. It mounts the source and destination shares and copies data over It's best it offers best performance when Deployed a standalone on a single single server So it does not disrupt the the other the other services that are running the drivers or api or scheduler Now, uh, we are going to demonstrate A migration between two different storage vendors one that is the generic back end using the generic driver provided in manila And the other one is the itachi h&s back end So let's log into our open stack dashboard As ended a administrator and let's see which shares we have We can see that we have a share created in the generic back end It already contains an access rule which provides access to a guest virtual machine Let's copy the export location And mount it on our guest virtual machine So we can see that we already have 800 megabytes Uh file in there. Let's create a demo file just to demonstrate that the share is writable At this moment So let's trigger a migration Can you please pause? Oh, thank you. So let's feel here here the destination destination, uh pool String that's where we want to migrate the share to the h&s back end Here we are going to select the forced host assisted migration because we are doing a migration between different storage vendors And they select the other ones because those are only for driver assisted Now we are going to deselect the share network set as empty And uh, we are going to choose the share type Compatible with the destination This is in order to change the driver mode Okay, so we have just triggered a migration. We can see that the status changes to migrating Let's try to obtain its progress Since migration has just started it shows It's at zero percent While it's migrating, let's do something else. Let's try To see if the contents are regular Are there preserved great Let's try to create another dummy file. We will see that it's not possible because For the host assisted migration, uh writable migrations are not supported and then the share becomes writable Sorry writable. Let's try to obtain the progress again. Now we can see that it's at 100 percent So let's trigger Migration complete but before that let's amount our share gracefully Okay, let's trigger complete migration All right, we can see now that it shows that the share is in the h&s back end And we can see that the export location has changed and it has migrated the access rules as well So we do not have to migrate them now manually Let's copy the export location and mount the share again So we can see that the contents are there the dummy file that we created has been migrated We can create another dummy file here. So the share Has become writable after the migration Now let's do something Let's do another experiment. Let's try migrating back this this share to the generic one So let's go back to the list of shares and click migrate share We're going to input the the pull of the generic back end Again, select first it hosts assisted this selected the other ones But now we have to choose a share network because the driver mode of the generic back end requires using a share network And we will also change the share type to be compatible Okay, we successfully started the migration Let's try to obtain progress Right now is that should be at 0% And for the interest of time this video has been sped up at this moment So we will be taking progress again and see that right now it's at 100% Let's trigger migration complete them Okay, so it says that the share the migration has been completed and the share is now back in the generic back end With new export location the same access rules Let's mount it again And we can see that the data has been transferred back along with the file that we created in the hns back end So we have all files there with all their content That's it for the share migration awesome so Now that we've gotten you acquainted with The work that we've done so far that let me present to you a prototype of something That that's going to be part of a future release non-destructive data motion So manila is poised to expose the goodness of backing storage in terms of Providing non-destructive operations to the tenants of your cloud So in this example, we'll be using a manila share to run some live work workloads And while those workloads are running, we're going to be moving that share from one storage pool to another storage pool The workload that we will be running interestingly is your open stack instances That is nova compute nodes. We will do this by using the manila share as the nfs backing store for cinder And so let's begin Let me go ahead and use the cli It's create a manila share. We're calling it the boot volumes share And uh, let me just go ahead and see that the share Is created. Yes, it's available. That's where it's created. It's on Some host uh with that Huge string out there Let's go ahead and allow access to a subnet the private subnet on this resource And then let's go ahead and grab the export location for that share There's the export location So if you're familiar with using an nfs backend for cinder, what you would do is grab this I mean configure cinder to run with the same backend and use this share as your in your nfs shares configuration So let's go into cinder. There it is my net app Boxed the same backend that i'm running for manila and i'm going to go into the shares config file and Paste this particular export path So what's going to happen is when we restart cinder services We expect that cinder picks up this as the backing store This manila share Takes a second or two for the driver to start up in cinder And we can verify that by using cinder get pools Get me all the pools that are available for cinder. I just have one my manila share. It's 200 gigabytes Uh, there it is. It's the same export location that we grab just to prove it to you and Let's let's check whether our volume service is up and running and everything hail and Hali So there is our volume service. It's enabled and it's up Now let's go ahead and boot an ova instance to a volume on this It's backed by the cinder pool that we just have so we're going to boot to a block device That's the only cinder backend I have so the block device going to be created on my manila share Let's go ahead and watch it being created. Of course. I sped this part up. That's your instance going active All right, so let's switch to this split screen interface where I'm going to list my uh This thing grab the ip of my an ova instance and log into my nova instance on two separate windows. You'll see why so Just connect to the nova instance As you can see I'm a big fan of clearing the screens. Let's do that Let's get back Into our controller and go ahead and list our share again Let's see where it is currently. Of course in your scenario You're you're going to be performing some sort of a balancing of your storage system and so on So you know where you want to go, but I just want to see this is where it is right now It's on an aggregate called agar 2 which is a manila pool. Let me go list the list of available pools So that's the pool that the share currently is on. I want to move it to another pool And let me pick the first one That's my destination and as This thing we'll we'll go we'll go ahead and start some workload on our No instance a simple dd command writing some Random data into a file And let's monitor that Process as it's happening. So we're just going to list see some sort of the right statistics and and stuff So there it is. It's writing something And let's go back and start a migration as easy as it it is to do it with the cli Sorry, the ui we've made it pretty easy on the cli as well So it's as simple as saying one. Let's start migrating my share To this back end right take it to this pool. I already know what pool I am on so take it to this pool So let's go ahead and watch the progress as it does as you can see while we started the migration nothing is happening to the Boot volume of nova that nova is writing constant data to everything is fine Let's go ahead. Oh, sorry. I skipped the part. We completed the migration It finished migrating copying all of your data. There wasn't much data. So it's going to be really fast so We've started the cut over phase at this point. We expect our nova instance to crash I should have practiced this It's not crashing All right, and we've moved to a different storage aggregate. What's happening? All right, so as you can see the nova instance is unaffected And your monolush is shares moved to a different aggregate All right Okay, so that's wrap wraps up the main content of our presentation before we open up for questions. Let's uh, Let's tell you guys about What's coming in our roadmap? so We're going to transition we plan to transition in the the replication and migration apis Out of the experimental phase. They are currently fully functional experimental apis and uh, this is planned to take place in Okata That is also a spec and code proposed for the share backup feature Which will use the data service to provide a common mechanism of Backup functionality to different storage vendors in a similar way that migration does But we will also optionally allow storage vendors to to To provide their special capabilities and advantages Also planned in Okata And the one that we just had a design session today The data service jobs table, uh, we will allow administrators to monitor What jobs are running in manila? We also plan on having a feature called share modify Which will be a user facing api to change several of manila properties such as protocol share type Share network availability zones and uh, this feature Depending on the property change it may or may not use the the migration feature to to move the share between pools Yeah So we're also looking to tie up the loose ends and share application We are going to fully support the share service use case Hopefully in the Okata release and are we going to improve the first party driver support that we have added We we we want uh users to be able to experience share application without having You know in any proprietary back end on the on this thing Of course you can do that today and you can also replicate remote systems Except it's driven by manila and that's the part that we want to decouple from manila and We are going to be uh, we're also working for the past couple of releases on a On establishing a concept of a group of shares Which upon which some tenants can operate upon so you could say, okay Group my group these shares and then take a snapshot together or take a consistent snapshot together Migrate this group together or replicate this group together and so on so we'll be continuing that work as well So that said we'd like to open for questions You have uh mics Okay, there's one mic Yes You can call that was really cool. Um, I Wondering if you truly support multiple back ends though are only the capability to do that with netapp today, right? So this This isn't something that's in the code right now So even netapp can't do it right now not not with manila at least So there are many many vendors that can actually live, uh, I mean migrate Nondisruptively from their own storage to their own storage, right? So we'd like to expose this functionality Through our api so that administrators if they know that they're just rebalancing between the same The same vendor and stuff they can possibly perform this migration Nondisruptively So this is just one of the prototypes and this is code that we'd like to commit in the future releases Yep, no problem anyone else. Yes It's too far. Okay. I can repeat your question Do we support migrating between different protocols? Not yet It's uh, my the migration allows changing the properties that are mentioned that there are share type share network Consequently driver mode and availability zones But the destination pool has has to support the same share type as the source So the the share type is preserved when it is migrated Can you explain why you had to manually complete the migration? Uh, the the the purpose of manually completing the migration is so the administrator can have The best precision in order to Minimize downtime so the administrator can leave overnight copying data Or if he has a database application running it can Pick the right moment to to trigger complete migration, which should be a very quick operation And minimize downtime to his application to his clients that are using the share That's true And while non-disruptive operations can actually complete on their own right So technically you could kick off the migration and just expect it to complete without with this level of transparency We wanted that simplicity to be in the api to allow for a two phase So administrators know they're performing a non-disruptive migration, but they'd still have control of when to kick off the Complete they could script it if they wanted Yes, so you could possibly write something over manila just use the same api to Outside of manila. Yes, you can write a script that washes the get progress output and based on these status you can trigger No Yeah, that's a that's a that's a great suggestion. We did consider that but we have so many different flavors of What this back end engine that we that we've built is going to use that we wanted this two phase You know api It should be really simple to write those two curl commands in one script Or you should use the python manila client. Sorry any other questions All right. Thank you folks. Thank you for joining us. So that's rodrigo and this is gotham And