 Thanks everybody for coming. My name is Bob Calloway. I'm chief architect and senior manager of technical marketing for all things OpenStack at NetApp. And today I'd like to talk to you a little bit about OpenStack Manila, which is a recently incubated project. We'll cover today if my clicker will work. There we go. Quickly give you an overview of the project. Go through a couple different use cases that are relevant in terms of where we see the fit both within OpenStack as well as more generally within IT infrastructure. I always love to do demos because if I'm sitting in your chair it's like if PowerPoint charts are great but demos are better. So we'll go through a quick couple minute demo just to kind of give you a sense of what the user experience is like with Manila. Talk a little bit about the architecture, some of the design points that are relevant. I briefly mentioned a couple of the drivers that exist. I certainly don't have enough time in 40 minutes to go over all of the vendors and all of the drivers, but I do want to just give you a sense of what's out there today and what's coming. We've also been doing some work at NetApp around automation for deployment and management of Manila. So we'll briefly talk on Puppet Manila, which is a new module that's now in Stackforge. Talk a little bit about what we did in the junior release, what we're going to do in the key of the release looking forward, and then a quick blurb on how to get engaged in the platform. So hopefully by now you guys have heard of Manila, but if not, one of the easiest ways to describe it if you're familiar with OpenStack is Manila is for shared file systems what Cinder is for block storage. So it's a self-service provisioning and management framework for dispensing shared file systems as a core unit alongside block and object within an OpenStack cloud. The use cases would be someone would come in and say, well, let's say I've spun up some additional VMs for a new marketing project. There's already an existing file share that other marketing VMs have access to, and I simply just need to enable new access to these new VMs that have launched to handle that burst and work that have come in. Or perhaps I'm spinning up a new workload, say a new R&D type project. I need new VMs and there's a need for shared storage across those VMs. So the Manila API which you'll see in the demo is very much create a share, delete a share, take a snapshot of a share, very basic vendor agnostic set of principles in terms of how you interact with the service. And we feel that shared file systems have had a long legacy of success as a key tool within IT infrastructure over the past 15, 20 years or even longer than that. And so in a cloud specific context, we're seeing a lot of purpose built services emerge, things like databases as a service if you're familiar with the Trove project. A lot of Oracle systems and MySQL systems in the world run on top of NFS as the backing store. So thinking about how would we integrate some of those best practices and proven architectures into a cloud context, shared file systems has really been missing. If you think about OpenStack in the context of, it's very much a mirror of Amazon. So if you look at Amazon Web Services and their full catalog of services, there's pretty much a one-for-one map for everything that's in OpenStack core. But there's no Amazon service that dispenses NFS or SIFS on demand. But we think that that's generally applicable and that's why we started the Manila project. The other minor thing is that if you believe what analysts tell you, which generally they're right within a few standard deviations, two-thirds of all storage in the world is sold for file-based use cases. What does this really mean? There's a lot of management and provisioning and overhead and lifecycle maintenance that occurs for these artifacts. So adding a sense of automation, repeatability, consistent behavior, all within a context of a cloud operating system makes a lot of sense in terms of a project. So kind of diving into a couple different use cases, certainly the most natural is seeing shared file systems alongside block and file and other storage artifacts within OpenStack. This just kind of gives you a logical diagram showing you all the core OpenStack services with Manila, the shared file system service, kind of sitting inside of that architecture, interacting with other services like Cinder, Nova, Neutron, as it builds up provisions, infrastructure provisions, these artifacts. Manila sits alongside that. So one of the major aims for Manila is that anytime I would deploy an OpenStack cloud, I should see that that file share service sit alongside block and object. But given that we talked to a lot of customers around network attached storage and shared file systems, there's another predominant use case that comes to the mind of our customers quite a bit, which is perhaps I've used NFS or I've used SIFs for a variety of different projects and I have existing business processes or existing kind of homegrown technologies to provision shared file systems and understand who owns what, how do I do quotas, how do I do size management, etc. What I'd really like to do is take that piece of software, that set of Pearl scripts that one guy in the company wrote, understands and actually operates. I'd really like to move that towards more of an open standard model such that I can plug in different vendor technologies behind it. I get a community from the OpenStack ecosystem around that such that one guy doesn't support the solution inside my organization. So while we envision Manila to be obviously a core part of many OpenStack deployments, we also recognize that there are customers that leverage this modular nature of OpenStack to deploy things standalone, so we have customers that deploy Cinder just by itself if they're perhaps not ready to avail all of what OpenStack brings to the table and we expect that Manila will be used in very much that same pattern. So getting to the aforementioned demo, we'll quickly go over here to my wonderful recording. What we're going to do is kind of back to the use case chart that existed before. We'll look at two use cases. We'll look at just one given the interest of time, which is we'll go into Horizon. You'll see here on the left side of the menu, and my apologies for this not being the full screen, but on the left here you'll see just alongside instances and volumes, you'll see shares now in Horizon if you install the Horizon plugin for Manila. So it's a first class citizen per se. You see the tabs here around shares, snapshots, share networks. So the demo what we'll do is we'll actually provision a new share and then we'll go into a couple of existing Nova instances and we'll add access to those. So we'll see that there's already two shares there. We'll quickly look at the list of instances and we see there's four Ubuntu instances just for the sake of consistency with the setup chart, their name 167 and 8. So we'll go to quickly the share networks and there's a concept that I'll go through a little bit later that associates the actual network topology that a tenant of OpenStack has defined. You need to make sure that when you provision a shared file system that it's accessible over that neutron specific network. My apologies for that. So that's one of the interesting technical challenges with Manila is that we have to make sure that shared file systems are accessible over the network and so we have to make sure that we're plumbed into the correct networking topology. And Neutron supports a variety of different Layer 2 protocols, VXLAN, VLAN, GRE, etc. So Manila has to figure out how to inter-operate with those different frameworks in a way that's seamless to the end consumer. They just simply say I want a share of this given size in a way the system just needs to figure it out. How to provision that in a fully multi-tenant environment. So just for the sake of argument we've got those four guests. They all exist on the Neutron network named Private. And so what we'll do now is one other minor detail with the demo is the actual NFS export location is stored as metadata on the individual share. And when you think about this in the context of the comparison to block storage if you've ever used Cinder there's an operation called Attach Volume to an existing instance. With the concept of a shared file system there's not a natural certainly there's a I'd like to virtually attach this share to a VM but there's no way to raise a PCI interrupt on a VM and say ah there's a new shared file system that you need to go mount to set up ahead of time. Whereas with block storage you can raise that PCI interrupt from the hypervisor and oh I have a new block device I know what I should do with it. So there's a set of challenges around mount automation figuring out when these shares are created, how do I actually attach them to a new VM in a completely automated fashion. We're having to come up with a variety of different approaches to address that particular issue. So we'll walk through, we'll go back and we'll create a share in Horizon so again it looks and feels just like you were creating a block volume. You specify the name, the storage protocol you want to use which is either NFS or SIFS today. We are adding some new protocols in the future. You specify the size and then the actual share network which again is kind of the mapping between the neutron network to the individual share. So we go off, we issue that request into the REST API through Horizon, that share will get created and provisioned behind the scenes and it will be marked as available. And then what we can do is we can drive down into the actual metadata for this new share and set the access rules such that we know the IP addresses of the two VMs that we wanted to grant access to. So we'll specify these two addresses via their IP, we'll click add and then what we'll do is we'll actually log into the VMs, I'll prove that you can actually mount these shares via that export string that I showed a little bit earlier. So as soon as the, there we go. So we'll go ahead and we'll quickly log in over the neutron network into guest number four. So we'll specify log in stuff and then we'll go back and we'll get the mount string. So again what you would typically do if you're mounting an NFS share to a Linux host, right? You need to call mount and pass it a string. So we'll go back, we'll get the actual mount string, copy and paste it out of Horizon. You can also call via the CLI or via the API to get this information as well and if you were going to script against this. So it's all available programmatically. So we'll then specify the mount point is this new directory that we've created and then we'll just change into that directory. We'll write a file to prove that we can actually do datapath.io on it. You can see here the information from the DF command showing that it's a one gig share and basically unused. We'll then write a file just to say hello from guest number six. We'll then quickly switch over to the other guests that we added the access rule to just to prove that again it's a shared file system. Multiple clients should have access to it at the same time. So we'll create a mount point. We'll mount the share and my apologies for not knowing to call pseudo. You live and you learn. So you'll mount this up and then we'll change into the directory and we should see that guest six file exist. It does exist. We'll then just copy it over into a file called guest eight. We'll change the content real quick with my favorite editor VI. Sorry to any Emacs folks in the room. That can be a controversial thing. So you can now see that we've written that data. And the last thing that I want to show here is we do have full integration with in terms of the quota system and that is actually visualized within Horizon so you can see the numbers of shares, the actual total amount of storage consumed as well as the number of shared networks. So all those can actually be limited via quotas and all have the same type of API controls around them. So if you're familiar with going into policy.json and actually setting, I wanted administrators to be able to change these types of things where I only want privileged users or you can actually set up the access control in the same way that you would do any other open stack service. So switching back to PowerPoint. Sure. So is it possible to use Manila as a standalone service if I only have Keystone, Neutron and Manila and then on Neutron I create a flat provider network? Yeah, any client that can access that network should have access to the share. So it doesn't currently Manila requires Neutron. That's something that we're actually moving towards a plugin model in the future such that if you don't deploy Neutron as you hear a lot of people saying, we want to eventually right now maybe not. So we want to be able to support both modes because again, thinking back to that kind of standalone use case that I mentioned at the outset, we want to make sure that we're not requiring Neutron. It's certainly an option that we'll support moving forward. So the VM has to see the share. Yes. Do you have a time volume in case of feedback? Yeah, so it's an interesting technical challenge because Mount requires a privileged user as you guys saw. And second, so there's a security thing. Do you really want Manila to log into your instance with root permissions and do that? How do you raise a notification to say a new share exists? So do we put a notification on a queue and let you read off the queue? Do we integrate with cloud in it? I mean, there's a variety of different approaches. Some people are comfortable with giving SSH keys to open stack and having you just automate it. And other people are absolutely not, right? So we're not going to impose one solution to that problem. We'll probably just have a variety of different frameworks that will support. So I'm not sure. So Cinder actually doesn't allow you to have multiple hosts attached to a single volume quite yet. And even if you do, you're attaching multiple hosts to a block volume. So the upper layer applications has to manage access to that shared medium, whereas a file system actually has locking built-in for such a thing. So happy to answer that question after the fact given I only have 40 minutes. So if you don't mind, we can take that offline. Yes. So I wanted to quickly go through kind of one of the key objects within Manila. So obviously a share is kind of the big deal, kind of like a volume would be for Cinder. So again, the user says, I want this size, this protocol, and this kind of share type. So just as Cinder has a concept of a volume type to be able to have an administrator define either a gold-silver bronze or a workload-aligned label, you can do the same thing within Manila and have those capabilities reported back from individual drivers into the scheduler framework. As you saw, we have access rules notion as well. So what actual clients can access the share? You can specify that by IP with the CIDR suffix. So if you wanted an entire subnet to have access versus an individual host, you can handle either model. Let me pause you to the next chart. So you can do IP level roles. And then as I mentioned, the share network is the association. And an individual share can only be a member of a single neutron network at any given time. We do have the concept of a security service. So thanks for the segue to where you can actually connect into an active directory system or an LDAP system and assign user-based relationships between an individual share such that, again, if I have a heavily Microsoft infrastructure, a lot of clients in that share access is done through AD rules, we have to be able to integrate with that and support that. So that's done through an object called the security service. Just like with Cinder, we have the concept of taking a snapshot. So this is a single point-in-time operation to say, save the state of what this share looks like. And that's actually persisted after the fact. Those snapshots are read-only. However, you can use them as a source content for a new share. So just like with Cinder, I can take a snapshot of a volume and then create a new volume from snapshot. Again, Manila and Cinder, from an API perspective, look very, very similar. So if that helps with how you think about this. Again, similar to Cinder, we have the concept of supporting multiple back-ends. So multiple vendors, different drivers can hook into the scheduler. And then individual vendors obviously code to drivers to affect the implementation of the Manila API. And so at a very high architectural view, we require a SQL database. We have an API Python process running off of the WSGI framework. The scheduler exists as well as a separate process. Those processes talk to one another over AMQP. And then you have one or more share processes that map to individual back-ends, plus an additional process that kind of acts as the manager of those individual shares. And it's the share processes that actually deal with talking to the individual storage subsystem. So if you're talking to Nova and Cinder in the case of our reference implementation, it's a share process that's doing that. In the case where you're using the NetApp specific driver for Clustered On Tap, that's the process that's actually handling that communication. Another thing to point out is that Manila does not sit in the data path between client and server in this case. It's simply a provisioning and management framework. So it'll create things until the client and the server about each other. But after that, it gets out of the way. So it's not like there's a new performance implication of this existing with the Manila processes in general. As I mentioned at the outset, the introduction of the network kind of intersection with Manila sets up a very interesting set of kind of use cases, some of which where a customer perhaps hasn't adopted Neutron at all. You've got just flat layer 3 networking everywhere. And the back end storage actually doesn't support any elegant layer 2 segmentation. So Manila has to be able to support that direct mapping between a share on an IP network. And that comes up a lot where if I have an existing service that isn't VlanAware or VxlanAware, it simply just assumes there's IP access from everyone. Manila needs to be able to support that. On the flip side, you've got somebody who's heavily invested in, let's say, the technology like Vxlan. And their back end storage may have support for VLANs. So when Neutron comes up, you have to figure out how to bridge these two potentially disparate network technologies together. So what we've been working on are kind of, technically I guess you could call them network bridges, but things that would actually bridge the layer 2 protocols together such that you can get IP traffic to flow to support connectivity. So while I won't go into all the details of this table, given the amount of time that I have, I did just want to point out that there's a variety of different use cases that Manila needs to adapt to. Some of these we kind of support out of the box, some of which require calling out to Neutron and future innovation, to be frank, that we're kind of figuring out what are the best models as Neutron matures and evolves along with the rest of OpenStack. So I do want to talk a little bit about some of the drivers that we have available. So I've talked about just like most other OpenStack projects, we have a reference driver, a reference implementation built completely on open source software. What it actually does to supply shares, it will actually spin up a NOVA instance under a private service tenant that actually hosts, by default it's in a stock Ubuntu image with an NFS server and a Samba server installed. And then when we get an API request to go create a 5GIG SIF share, what we'll actually do is we'll create a 5GIG Cinder volume, attach it to that NOVA service instance, and then export Samba. So we'll lay down a file system and export that through the VM. So while we don't envision a lot of people would actually use that in production, it's a way to at least get started with the service and understand kind of what the implications are, test it out and move forward. NetApp, obviously they pay my bills, so I have to mention that we have a driver with Manila, specifically with our cluster data on tap platform. We do, to be clear, Manila is not a NetApp only thing as a fully integrated and incubated project. We have a variety of different vendors that are integrated. So EMC has a driver, RedHack has a driver, IBM has a driver that's up on Garrett at the moment in review, HP is working on a driver and there's more to come. So if you join our OpenStack meetings, you'll see a lot of different vendors coming in trying to learn more about the service, how do they write different drivers. So it's a very active and growing community to itself. To give you an example of what a vendor driver kind of looks like behind the scenes, we'll use the NetApp one as an example. So the individual driver process that we showed from the architecture will then call out to the NetApp storage to do provisioning operations. So we'll create a new NFS export or a new SIFS export. It will then create a new, what's called a data, in NetApp speak, it's called a data lift or a data logical interface. It'll put that on the right network segment so it'll apply the right VLAN tag if you've got that specified in Neutron. And then it will actually call out to the storage controller to set up the IP rules so that that actually is enforced not at the hypervisor but at the storage system itself. So it's enforced at the last mile to be as secure as possible. So that's just to give you a flair of how kind of a production grade enterprise class storage system would integrate with Manila. That's what it looks like at a high level. From an automation perspective, installing new services can often by hand, if you've ever gone through and downloaded code from Git and tried to do it by yourself, it can sometimes be a bit challenging. And even thinking forward to when Manila becomes part of OpenStack Core, we've got to have support to integrate with the different distribution vendors. And a lot of them have chosen things like Puppet and Chef or Ansible, technologies like that to build their installers off of. So we wanted to make sure that we support those frameworks in order to make Manila successful and accessible to end customers. So one of the things that we've been working on on my team at NetApp is actually writing a Puppet module for Manila. So if you go to StackForge now, there's a new project called Puppet Manila that's there. You can pull down the code. Since it's on StackForge, if you have updates or changes or bugs, it's all done through Garrett, so it's a normal standard process with an OpenStack to get engaged and get involved. And just to give you a sense, if you've never seen a Puppet manifest, this is what they actually look like under the covers. Apologies that it's perhaps a little small to read, but you specify the connectivity to the AMQP server, the SQL database, as well as particular artifacts around what you want the Keystone credentials to be. If you want to install the scheduler service as well as the share service, then you can define driver-specific fields to say, oh, if I'm connecting to a NetApp system, here's the IP address, here are the credentials that you use, et cetera. Puppet figures out, oh, I'm actually running on a Nubuntu host. I need to go pull these packages from a Nubuntu cloud archive. Or if I'm running on Red Hat, it'll go say, ah, there's an RDO repo that has these Manila RPMs that exist. I'll go and download them and install the dependencies and figure it all out from there. So, you know, Puppet gives you that framework to kind of apply or at least declare your intent and let Puppet figure out the details of how it actually goes and affects what you want it to do. In the Juno release, we were very busy, and I know we have several of the core members in the room as well. So if you have particular questions on some of the contributions or items in here, feel free to raise your hands and guys chime in if you want. So the big milestone was obviously getting to incubated state, and that happened at the TC meeting at the end of August, so we're really happy that that's now official and behind us. We did see a transition for the core Manila modules. They're now not on Stackforge anymore. They're now under the OpenStack project on GitHub. So in case you were looking at this before, you've got to note that the URLs actually have changed. We released Puppet Manila, as I said. We've done a lot around making it easier for people to come on board and only use Manila but also develop in Manila, so we've done a lot of work around developer and admin docs. We've added this new concept of a share server to try to formalize kind of this behind the scenes, whether it's a Nova VM or whether it's an artifact within the actual vendor-specific implementation that kind of corresponds to a container that manages the access for multiple shares within a tenant. So this concept of a share server now exists and has full API support as an object. And that helps to kind of formalize some of the new use cases that we see moving forward. Along with any maturing product, we're always adding new test cases, adding stuff to gate to improve quality. We've ripped out a lot of code that kind of was one-off as we were growing the project and tried to standardize with what's in Oslo. So that's been a significant improvement. We've also had some feedback from some early users around, hey, this API is great, but I'd really like the ability to filter this down so that I don't collect everything at the outset and have to filter through it on my code. So being able to kind of better specify some search options and a variety of other things, we've improved upon. To give you a sense of where we're going in the Keele release as well as moving forward, I do want to call out briefly that we have design sessions with Manila starting tomorrow, Ben. Tomorrow afternoon. Tomorrow afternoon in the hotel across the street. So there's now, with it being a fully incubated project, we have kind of our own track and our own sessions. So if you're interested to learn more about where we're going, feel free to come and join us. Some of the things we're going to be working on in Keele are around adding additional capability into the scheduler. So Cinder recently added the concept of resource pools. We think that's a really attractive idea to take into Manila. So we'll look at that contribution to figure out how to integrate that concept into our service. We're also going to be continuing to add automation support. So we've done Puppet now. We'll be looking at Chef and some of the other popular frameworks to add hooks into. In terms of the kind of the day-to-day writing code, we're actually looking at the third-party CI framework that's now emerging and figuring out how do we integrate that into our processes around Garrett and the upstream CI system. And we obviously want to become a successful project and a core part of OpenStack, so making sure that we're hitting all the milestones that are required to become a fully core system, a core module within the community is important. Around the networking stuff, we're also, as I mentioned, we're working on this network bridge to be able to handle different technologies in a seamless fashion and also removing the dependency on Neutron as a core item for us moving forward. Adding capability around bringing existing shares under the management realm of Manila. So I've got a shared file system on a NetApp controller or an EMC controller. I want Manila to manage its lifecycle and its access rather than some homegrown system. So the ability to kind of start to manage or stop managing a share is something that we'll be adding. A replication framework to be able to create a share such that I know that behind the scenes is actually replicated to meet some RPO or RTO objective doing backup, working on the automated mount stuff. These are all areas that we need to improve upon and obviously it's a call to you guys now in the community. This is an area of interest. Certainly we'd love for you to get involved in any of these areas. And as time goes on, obviously the lights go out or back on. We'll start to think about how do we integrate at the triple O level, figuring out is there a place and we think there is for shared file systems actually at the undercloud. So how can we use NFS to avail live migrations of VMs or to set up backends for provisioning cinder volumes, for example. So we think there's some interesting use cases around kind of the undercloud as well as introducing more of a formal concept around quality of service for individual shares. That's something that we'll improve upon later. And of course things change. It's hard to tell what OpenStack will look like with content tailing off. Because things will come up, things will change. So just to give you a sense of where we're going. So I do want to leave a little bit of time for questions at the end. But I do want to also kind of give you a pointer in terms of where to get started. So if you guys are developers and you're probably familiar with DevStack by now, if you're not, you should be. If you want kind of a hint of how to get DevStack configured with Manila working, I'd point you to a blog post that one of the guys Greg on my team did that kind of walks you through how to point at the Manila repos, set up all the local.conf variables to make sure that everything works out of the cover with the horizon plugin so that you can get started. So feel free to visit our blog. In terms of getting access to Manila in a pre-built form, we've been working with distribution partners like Red Hat, like SUSE to get builds of Manila done. We actually do have at the moment an RPM built off of the RDO distribution as well as the open SUSE distribution as part of their Juno releases which both recently went out the door. We haven't integrated quite yet with their installer frameworks. We've got integration with PacStack actually as a pending review with the Red Hat folks on Garrett. We're also working with the SUSE folks and the Ubuntu folks as well. So all of that will come here in time. PacStack is probably the easiest way to pull the bits down and get started but we realize that people want to use the distribution. In terms of other ways to engage with us if you have questions, concerns, hey this doesn't work right, obviously this is OpenStack so IRC is a great way to get a hold of us. We're in OpenStack-Manila on FreeNode. We have weekly meetings just like every other OpenStack project at three o'clock in the afternoon UTC on Thursdays and just to point out a couple that the PTL is back there. Valeri is along the wall. He's also core. If somebody else is Manila core in the room that I don't recognize, I apologize. But you know, these are the guys that are here this week. If you have questions or concerns, now's the time to ask as well as to engage us later. Any other questions? What's the room between share networks? You cannot see one share if we use another share network even being the same neutral network or network. So for a share network there's a one-to-one association for a given neutron network and neutron subnet to a share network. So I can create as many shares as I want on that share network. But what that does is it allows us to know when we call out to neutron and we ask for an IP address for the share. And then the subnet that it needs to exist on and what segment effectively from a layer 2 perspective. Correct. Yes. Or instance ID or something like that. Yeah. So that's a question that I think we've raised and I think we've... We're working on it is the PTL's answer. We're working on it. In terms of the context of a heat template, I'm not going to know what the IP was when I launched a stack. In that instance, whatever its ID is, I need that to be referenced too. IPs are the common way that that's done for shared file systems. That's why we started with that. But yeah, that's adding in some of the cloud polish to the use cases. Yes. No, not yet. So that's something that we're working on as a community but I don't know that the code is actually ready for use that you can plug in. I know heat also has a framework for incubated projects that have to contribute in a particular way and so we're working on that and one of the guys on my team is actually working on that as I speak. I haven't seen a demo of it working yet. Any other questions? I know it's really hot and it's really crowded but I want to make sure you guys get the answers. A guest agent. Yeah, so that's another approach in terms of the mount automation. Something like a cloud init may be present in a lot of VMs that get spun up on open stack so figuring out is there a way that we could hook into something like a demon that already is running within a guest. Obviously that won't work for windows so we'll have to figure out an approach for windows guests as well but there's a variety of different approaches to solve it a guest agent would be one. You're right but there still has to be a notification framework to either the agent has to pull to say what shares exist and that I have access to based on my IP or my instance ID or if you want that to be completely on demand then how do we actually subscribe via that guest agent to some type of do or a pub sub network to figure that out and there's security around that and so there's a whole slew of problems if it's an easy problem we would have solved it by now. Exactly so that's why we recognize that we want to go towards a pluggable model that allows you to select whatever you want or nothing if you want some external automation framework like a puppet or a chef. If it knows I'm going to go and create a share, fine then it will know the credentials and it can connect it in. So we have a couple of POCs where it's been shown to work. There's actually a session right after this one that we'll talk about a use case with SAP where we're actually provisioning VLANs for an SAP workload with Manila. So it's certainly been done. I know a lot of our Neutron code has been a bit in flux in terms of figuring out how do we make it more of a pluggable framework but Ben you want to comment on that? Yeah it's on the back in storage to what you support. Some products don't support any things like NetApp supports just VLANs today. Figuring out these models do we want to deploy an open V-switch bridge to do this work? Do we want to deploy a VM to do that work? There's a whole slew of approaches that we would need to consider. Sure. To make a driver for Manila? No. You can support just Layer 2 as well if you've done a flat network from a provider perspective with Neutron. But then you'll have all of your shares available on the same Layer 3 network and that may not meet the use case. But for a lot of environments that's how it exists today and that's fine. But in a cloud specific context where I want each tenant to have its own set of VLANs and I want complete isolation and shares to be offset and there would be no Layer 2 access to anything else then you have to start to move towards that segmented model. Anything else? There's also surveys here so please give us some feedback in terms of the content what you liked, what you didn't like. I would really appreciate hearing from you and appreciate the attendance and have a great summit.