 Hi guys, I'm Ben Swartzlander, the PTL of the Manila project. I'm here to do the project update for Manila for the Ocata Pike releases. I'm going to preface this presentation by saying you guys are more than welcome to interrupt me with questions. This will be a lot more fun for me if it's a discussion. But if you guys are shy and you don't want to ask any questions, that's fine. I'll just talk. But if you do ask any questions, please start using the microphone. So let's start off with what is Manila, for those that don't know. Manila is the shared file systems as a service project. The simplest way to describe it to people who are familiar with OpenStack is it's like Cinder but for NFS, but that's a gross oversimplification. It's similar insofar as it allows individual tenants in a cloud to self-service provision storage without the involvement of an administrator. And so it fits into the OpenStack ecosystem very well for that reason. But it's different than Cinder because it's shared file systems, which are significantly different than a block storage. I'd say the main use cases where it would make sense to use something like Manila as opposed to Cinder is if you need storage, it's accessible to multiple instances. Block storage is inherently limited to a single instance. So there's a lot of use cases around sharing the storage across multiple compute instances where Manila makes sense. File storage is inherently more elastic than block storage. Block storage has a pretty fixed size. It's hard to make it grow or shrink. Sometimes you can't make it shrink. File systems are very easy to resize bigger and smaller. And so you don't always have to guess how much storage you need up front if you're using shared file systems. And probably the most important use case is there's a huge base of existing applications that are built around shared file systems that already have NFS knowledge. And having something like Manila is essential to making those applications cloud ready. So there's a lot of interest in Manila from more traditional applications that use NFS. So this is an update on Okada and Pyke, but I'll start off with a brief history for people that haven't been involved with the project. The idea for Manila goes all the way back to the Folsom Design Summit and conversations between me and Rob Esker. At the time of Folsom, Nova Volume was just being split out as a separate project from Nova. The idea for Cinder was coming around. So we saw that there was block storage as well supported in OpenStack, object storage was well supported. There wasn't anything for shared file systems. And that's where the idea for Manila started off. The code started off as patches to Cinder. We had hoped at the time that Cinder would become the storage project for OpenStack and it would handle both blocks and shared file systems. But that was not to be, and it turned out that was probably a good thing because having two separate teams has enabled us to tackle problems more efficiently than if we had been dealing with both the block stuff and the shared file system stuff in the same project. So we officially forked from Cinder in August 2013. Manila became an official OpenStack project in the Juneau release. Since then we've had about 3,700 commits from 270 individuals from 57 organizations according to Stackalytics. So a lot of contributions. The project has come a long way since it started. We're up to 22 drivers, which is not quite where Cinder is, but it still shows significant involvement from storage vendors. And in addition to NFS, we support five other shared file system protocols, including SIFS, Cluster, Cef, HDFS, and just most recently, MapRFS was added. So we'll talk about features from Okata. Okata was kind of a short release, but we still got a decent amount of work done during the Okata release. The first thing I'll mention is we changed the concept of consistency groups to be shared groups, and I just saw that there's a misspelling on there. Sorry about that. The concept of consistency groups was added in the Liberty release, and it was a relatively inflexible feature that was requested by some users. And it turned out that after we added it, we realized that not even everyone agreed on what a consistency group was supposed to be. And different people had different ideas about what consistency groups should be, and the implementation was a huge mess. And so fortunately, it was an experimental feature, and we were able to just rip it out and replace it with something that we're much more happy with, which is share groups. It's a generic grouping concept that you can use for any purpose. You can still do things like create consistent snapshots with a share group, but the idea is there will be a lot more capabilities that will be added over time with share groups, and it's a fundamentally more flexible grouping concept. And that was added in Okata. We added mountable snapshots. So the use case here is suppose you have a share with a bunch of files on it, and you accidentally are a minus RF star in the wrong directory and you're a little bunch of stuff away, or you overwrite a file. In any case, somehow your data gets corrupted. So you say, well, crap, I gotta fix this. Fortunately, if you have a snapshot, Minola's always had the capability to just create a new share from your snapshot. You can go mount that, get your data out. Unfortunately, some back ends can't create new shares from snapshots very efficiently, and they involve copying a bunch of data around. And if all you wanted to do was just mount it, grab some data out, and then unmount it and be done, a mountable snapshot would do exactly what you wanted. So we added that capability in Okata so that back ends that can implement that efficiently have an alternate way to recover data from a snapshot. The revert to snapshot feature is similar. The use case here is imagine that you have a data set where you want to start with from a known set of data, run some experiments on it, modify the data, run some application that's gonna manipulate the data and do some things, but then when you're done, you just wanna throw that all the way, go back to where you were and start again. And maybe you wanna do this over and over again. It's a pretty common use case if you're just experimenting. So again, in Minola, if you have a snapshot, you can create new shares from the snapshot and do this kind of experimental workflow. But again, on many back ends, it's an expensive operation to create a whole new share if all you're gonna do is just blow it away when you're done and go back to what you had at the snapshot. So the revert to snapshot feature was added to address this use case. It's essentially a way that you can just time warp your share back to the last snapshot that you took. And that's another optional feature that some drivers started supporting in Okata. Snapshot migration was added. This is a really interesting feature. So migration was added also back in the Liberty release. And at the time, our main focus for the migration feature was to be able to migrate from anywhere to anywhere and to have it be a totally widely supported feature that wasn't limited to only certain back ends. So we implemented what we call the fallback migration path, which is all about being able to migrate your data from any place to any other place. Unfortunately, having that amount of flexibility means that you have to make some compromises. So the fallback migration path can only guarantee the contents of your files, the file names, and the directory structure. But we can't guarantee to preserve possibly vendor-specific metadata or any sort of file metadata bits beyond the standard POSIX metadata. And so what ended up happening is users said, well, it's nice to be able to migrate. But I need to know that when I migrate, I'm getting a bit exact copy of my data. And that absolutely everything is exactly the same after the migration as it was before the migration. We heard that very loud and clear from users. So we implemented a sort of a driver-assisted migration path where if the source and the destination of the migration are on two different storage controllers that are compatible such that the driver knows how to migrate the data for Manila, that it can do that and leave it up to the driver to handle the migration. And in that way, we can ensure that there's absolutely no data loss. So that work was done prior to Okada. And then in Okada what was added was the ability to also preserve your snapshots because once the storage or once the driver is involved in managing the data migration, bringing along the snapshots is not that much extra work. And so Manila now supports that mode of snapshot migration as well. More about Okada. So another change related to migration, we implemented a two-phase migration API. Migration was originally conceived as a one-phase operation. You say migrate and it goes to the other place and it takes as long as it takes. And in the period between when you start the migration, when you end the migration, you kind of don't know when it's going to get done. So this creates a problem. Imagine you have a lot of data like you got a 50 terabyte share that you want to migrate from one place to another. Even with like 10 gig ethernet, that's going to take a couple of hours, more likely a couple of days to migrate that amount of data. So what you'd really like to be able to do is to tell the system, start migrating my data and let me keep using it while that's happening. So in the fallback migration path, you can only have read only access to your data while a migration is in progress. With the driver system migration, it is possible to have a writable access to your share while a migration is in progress. So that if it's going to take a week, you can keep using your data right up until that last point when there's inevitably going to be a disruption when the migration completes because your share is going to stop being here and start being there. So what the two-phase migration API allows you to do is to decide when that's going to happen. So it gets almost all the way done and then it waits for you to say, okay, now cut over. So you can schedule that to happen in the middle of the night or during some downtime or whenever is convenient for you as a user. So during Okada, we had three new drivers added. There's a constant flow of new drivers in every release. We're always welcoming new drivers. We got the Dell EMC VMAX driver in Okada, QNAP and the MapRFS driver which includes the protocol support for MapRFS as well. And the last thing I'll mention about Okada is we did some work on TOOs integration. For those who don't know, TOOs is a library that supports distributed locking and other distributed computing primitives. These are important for the ability to run multiple copies of a service across multiple nodes and have them stay in sync with each other and avoid various race conditions and other mutual exclusion problems. So during Okada, we got the TOOs library integrated and there's work on going for TOOs that I'll talk about in a little bit. So Pyke, we've been working on Pyke for a couple of months now. I would say the main feature that I'll mention for Pyke is IPv6 support. And before I confuse anybody, Manila has always supported IPv6 for things like access to the REST API, communication between the various services, communication between the share service and storage controllers. But it turned out there was a hole in the actual share access. When you go to mount the share from a compute instance, we only supported IPv4. And so during the Okada release, we recognized this problem and decided that we needed to add IPv6 support. It unfortunately didn't get done during Okada largely because testing issues. The code was more or less ready. It turns out that the bits to support IPv6 with NFS were only added upstream in the last couple of years. And the stack that we were using to test with Ubuntu didn't have the right bits for testing. So we made the decision to use a Red Hat based distro for all of our IPv6 related testing. And this feature will hopefully go into Pyke. It's a high priority feature for us, IPv6. Solometer integration. Presumably you guys all know what Solometer is. It's a time series data collection service telemetry as a service. A lot of OpenStack projects have integration with Solometer. Manila hasn't really done it. Yet, so we're getting around to it now. We'll be able to provide various telemetry like number of shares, total shares consumed and other things that will be interesting from a Manila perspective. The other feature on here, shared usage monitoring is related. So this is actually, this came up just in the last couple of weeks as an important feature for Pyke. So as I mentioned before, shares can be very elastic in terms of size. And one of the advantages of shared file systems is that you don't need to necessarily know how big your share needs to be at the beginning, like you do with the volume. You can say, hey, give me a 100 terabyte share, give me a petabyte share. And then it's usually easy for most backends to provide as long as it's empty and then once you start filling it up then you find out how much data you actually need. So but the problem is is deployers who want to support that mode of using shared file systems don't have a way to bill for how much you actually do use. So if someone says, give me a petabyte share and you say, sure. If they actually use that whole petabyte, that's a lot of bytes and you want to be able to bill them for the amount that they're actually using. So in Pyke we're working on a feature to allow the share manager to find out through the driver how much space is being used in each share and report that through the same interface two kilometers so that you could export that information to a billing application or other sort of storage management application and do the right thing with it. On the quotas side, there's a few new features related to quotas. So quotas on replicas, share replication was a feature that was added several releases ago. And when we added it, it turned out we forgot to put a quota on the number of replicas you can create. So it turns out that today you could actually request thousands and thousands of replicas and Manila would try to create them, which is not a good thing. Group quotas, again, share groups is a new feature. There needs to be a quota on it just to prevent people from creating millions of them and causing a denial of service attack. The more interesting one is per share type quotas. So the theory here is if I'm an administrator and I have a really nice fast expensive storage and a bunch of cheap crummy storage. I would like to be able to set my quotas such that a user could maybe only get 100 gigabytes of the really fast super good storage because I don't want him eating that all up. Then I could do that through the quota system by creating a share type for super fast storage and giving the user 100 gigabyte quota on that. But then I could have a slow share type that maybe is really cheap, cheap storage that I have a lot of and I don't really care how much users use. And so I could give you a higher quota on that and that way you could have different quotas for different classes of storage. So this is a new feature coming in pike. User messages is a feature that the idea came from Cinder. A lot of our ideas come from Cinder. There's significant cross-pollination between the two teams. The idea here is there are some operations that happen in Manila where you request the operation, the request succeeds, but then later the operation fails. This happens in Cinder too. And there's no way for the server to tell the client what the error was because the request already succeeded and the operation that was triggered by that request was the thing that failed. So the idea here is to create a mechanism for when things fail asynchronously to be able to store a message so the user can find out what happened. And ultimately the goal here is so that the UI can tell the user, your operation failed and here's why, and hopefully avoid situations where the user has to go file trouble tickets with the administrator or go bug the administrator if it's something that the user could solve on his own. So this is a useful thing to have just from a usability perspective. So I alluded before to some features that were experimental when they were added. Some of them have been around for a while and we're pretty happy with them and we're ready to mark them non-experimental. In particular, the migration API after the most recent changes in the Ocata release, we're very happy with it. There's nothing to stop us from officially supporting this API going forward and marking it non-experimental. Also, the replicated shares API is something that was been added a while ago and hasn't really undergone any changes since it was added. And while there's more work to be done on the replication side, we don't anticipate it affecting the APIs. So we're looking to see if we can mark those features non-experimental in the pike release. And lastly, there's more work on the active-active deployment scenario. So as I alluded to, we have the two support in there. The idea here is, deployers, especially distros like Red Hat, like to be able to scale out all the services. So that you can get extra scalability by having multiple copies of services and extra reliability. So in case something goes down, you have another copy of the service running. We're very interested in supporting this in Manila. And so there's work ongoing during the pike release to support that feature. And beyond pike, there aren't any official plans. The tradition with Manila is we typically carry over unfinished work from release to release. And we plan about running release ahead. So we feel like we have enough plan for pike. And we'll see how the rest of the release goes. Does anyone have any questions for me? I knew you guys would be shy. Thank you. Can you say a little bit more about the active-active work that's going on? I looked at some point a little bit about trying to do that with NFS. And it didn't look like the protocol was really amenable to doing that. With NFS, so when I say active-active, I'm particularly referring to the services themselves. So Manila API, Manila Scheduler, Manila Share. So there are individual drivers and individual storage controllers that have single points of failure or don't have single points of failure, depending on the storage architecture behind them. So some vendors already have storage controllers that don't have a single point of failure. And if you're using one of those, your main concern is just the availability of the Manila service, right? If the Manila service goes down, I can't delete shares, I can't snapshot shares, I can't create shares, that's a problem. But you still have access to your data, right? Any shares that already exist are fine. There are some first party drivers like the LVM driver, the generic driver that are open source and the infrastructure for those are on the Manila node or on an OVA VM. And those do have single points of failure that we would like to address. But that's a separate concern. So the question is, are we gonna do what Cinder does and require drivers to confirm they support active-active? Actually wasn't aware that Cinder had gone that route. It's a good idea. We'll have to think about if we wanna do the same thing. To be honest, I'm not sure why that would be necessary. Maybe it's a QA concern. You just wanna know that someone has tested it. So we may wanna require that a vendor certifies that it's been tested before we allow that configuration to work. Yeah, that's a good idea. I like it. Thank you. I hope these aren't dumb. One is Tenant Networks and Virtual Private Networking in VXLan. Is there any talk at all about support of that? The ability to plug in the storage to Tenant Network? Yeah, so that support actually existed from day one with varying caveats. So Manila is actually almost as old as Neutron. And for a long time, Nova Network was more dominant than Neutron. And so from the beginning, Manila supported creation of shares where the export IP for the share was actually right there on the same Tenant Network. Initially the design was, that was the only way it would work. It turned out that was really hard to set up in most circumstances if you had actual hardware because plumbing the Nova Network VLAN through to the storage controller such that it would appear to be on the same network as the client. Okay, so the API supports it, but the appliance might not? Well, what ended up happening is we ended up creating a different mode back in the Kilo release so that drivers can either support the concept of share servers, which is the mechanism by which we provide shares on the Tenant Network that has existed from the beginning. Or they can just provide no share servers, which means everyone is going to access the share through some common IP of storage controller and the routing, and you're going to go through a router to get to it. So since we did that split, I would say most of the focus has been on the simpler case where people just want to set up their storage and access it through a common IP and not have anything on the Tenant Network. There has been work to make connecting real storage hardware to virtual neutron networks work better. In fact, there was- That's probably what I'm waiting for there. Yes, there was work during, I'm trying to remember if it was Newton or Ocata, I'm embarrassed, I can't remember. But there was work added to our Neutron Network plugin to do port binding such that if you have a provider network in Neutron that's connected to your storage controller, you can actually get share servers created by Manila to appear right on the Tenant Network. So they're right on the same L2 domain. You don't have to go through a router to talk to them. So that exists. Okay, just probably not on my net up. Okay, the second one had to do with the NFS version four and Kerberos and LDAP. I didn't really see anything about that in the user documentation. If I want to have my Tenant Network with its own LDAP server or something in user group mappings and so on. Yeah, so again, if you don't use share servers, the more simple configuration then everyone gets whatever the administrator set up. But if you do use share servers, there's something you can create called a security service and unfortunately support for security services varies from driver to driver. It's not consistent. But you can set up Kerberos domain and other Kerberos authentication information in the security service that's attached to the share network. And then if you create shares on that share network, the storage controller will configure Kerberos the way you asked. Okay, so you could have different tenants with different configs? Yes, yes, if you're using the share servers mode of the driver. And again, that's a big giant switch we added back in the Keelow release that most people don't use because it's complicated. I mean, there's a lot more concepts that the administrator and the tenant have to understand to do that. But obviously, some people do want that, so it is supported. But the testing there is not where it needs to be. So if you find bugs, we would like to fix them. One of the things that I was missing on the list that you plan to implement, or it's in the planning, is the integration with the OpenStack client. Actually, on running a large OpenStack deployment, we're trying to use this to basically consolidate on the OpenStack client. So we're trying to convince them, rather than using NOVA and CINERA, they use OpenStack client. Now we introduce Manila and they basically have to either use Docker or use special nodes or install the client themselves in order to use Manila. So is that anything that? Yeah, we've been interested in integrating with OpenStack client from the beginning, exactly how that works has been sort of a question, the answer to which has changed over time. So there's some limitations in OpenStack client around microversions and other things that we use in Manila that have prevented us from doing the integration in the past. And furthermore, we're very interested in being able to run Manila in a stand alone way, much like CINER guys like to be able to run CINER in a stand alone, because there are applications where you don't want all of OpenStack, you just want CINERA or Manila or the two services. So we don't see a world where the Manila client ever goes away entirely. But we obviously wanted to integrate with OpenStack clients such that users who just want one client can get that. It hasn't been a high priority, but it's something that we've been wanting to do as soon as all those dependencies are straightened out. Okay. It's just sort of a back burner thing. Thank you. I have a question about the project. The project will support a low base AC control with a hierarchical sharing. Hierarchical what? Sharing because of, for example, for a grant sheet maybe the project manager can access the fire employee part. I mean the hierarchical access with a sharing. Are you talking about hierarchical shares? Yeah, right, hierarchical sharing. So I don't, I'm trying to imagine what the use case is here, but I don't think that we support it. So the model behind Manila is like CINER, it's a self-service, so typically if you want storage you create a share as big as you need it to be and you mount it at the root and you own the whole thing. So protocols like NFS will allow you to mount sub trees and you can do things like that. But again, that's just an NFS detail. From Manila's perspective, it's just one share. There's no hierarchy that we're aware of. So if you think there's something missing, I'd like to understand what's missing. Because I think that you could probably, file systems are directory trees that are inherently hierarchical. For the attributes of any file, any file you could be shared, just apply some policies to add into this file. For example, my file can be shared by my, can be shared to my boss, but I could not be sure. Okay, I think you're talking about file access control. Yeah, right. Okay. Yeah, Manila does not get involved with the individual file access control. That's a data protocol thing for NFS and SIFs and other protocols. Okay. So NFS has its own way of doing access control on directories and on files. And once you have a share, you can chmod and challenge files as you want. If it's a SIF share, you can set owners and set all the Windows permission bits. Yeah, right. So that's an in-band activity between the client and the storage controller that the shared file system protocol supports. Manila's not involved with any of that. Okay, thank you. It just works. Okay, thanks. Thank you guys. Thank you.