 Welcome everybody. My name is Clayton O'Neill. I work at Charter Communications. I'm a principal engineer on the OpenStack team there. Today I want to talk to you about the work that we've done with getting Nova and Neutron to work inside of a Docker container. We've found that those are two of the two we've done. Those have been the hardest ones to get working. So I want to share with you how we're doing that. So a quick overview I want to be covering today. First, we're going to talk about some of the crazy command line flags you need to be able to get these things to start at all whenever you're running them inside of Docker. Second thing we're going to talk about is if you're using the Neutron L3 agent inside of a container and you're using HA routers, how do we prevent failover events that we don't plan for? And third, I want to talk how do we make Neutron's use of network namespaces play nicely with Docker? And if we have time, we'll talk about how that particular topic can help us solve another issue that we had with sender. So at the last summit, a co-worker and I gave a talk titled, Deploying OpenStack Using Docker in Production. And that we talked about why we use Docker, how we deploy OpenStack Using Docker, and also a lot of the pain points that we'd run into at that point. So that video is on YouTube. If you're interested, that's kind of a general overview of what we're doing. Another thing you might be interested in is we've actually published a puppet module that we use for deploying OpenStack services inside of Docker that works with the upstream puppet OpenStack module. So you may find that useful even if you're not using puppet. Like I found going to, say, COLA, very useful to find out how they managed to run services inside of Docker, you may find something useful there. So just a quick review of what we're doing with Docker at Charter. We first started deploying OpenStack services in our production environments in July of last year. Since then, we've managed to get all of our control plane and compute services running inside of Docker. We're using SAP and SolidFire for sender volume back in, so that comes up whenever we start talking about Dockerizing Nova and with sender. And we're using mostly VXL and tenant networks today with HA routers, which has some impact on how we get neutron to work inside of Docker. Last we're using Docker 1.12 at our production environments, and this is going to come up later. That's kind of an important point. The last thing I want to get out of the way up front, I'm going to use var run and slash run completely interchangeably during this talk. On most operating systems, var runs just a sim link. Documentation usually says var run. A lot of things use run, so just know that they're the same thing. So getting started with the first part of this. So a lot of the difficulty with getting a lot of open stack services to run inside of Docker at all is just figuring out what the magic command line flags are that you need. And specifically, this is true with Nova and Neutron. So I'm going to start off with kind of showing you some examples of services that we run in our production environments and how we start those. So these are a little gross and scary. I'm not expecting you to take it all in at once. This is the Docker run command line we use whenever we start the Neutron open V-switch agent. This is actually slightly simplified. I removed a couple of things that weren't actually really important. So this is kind of horrible. And then whenever you look at Nova Compute, it's actually a little bit worse. Now I'm not going to go through every option here. That would be really boring. I am going to go through a couple of kind of high level themes that I felt were kind of interesting and why those are important. So the first one here is dash dash net host. Very familiar with Docker and you've done very much with complex service. You might have seen this before. Normally Docker is going to start a container walled off into its own network name space. And this effectively is going to tell Docker to skip that step. Now this does have a side effect of actually making networking inside of the container a little bit faster. And we actually do this for all of our open stack services as a result. It makes things a little simpler. But for Nova and Neutron, most of those services actually need this. That's because they are going to interact directly with the host networking stack. So they need to make changes to the network. The next one here is dash hash privilege flag. This is effectively allowing running root processes inside of a Docker container. Now this is needed because Nova and Neutron can't do what they need to do if you're running them inside a Docker container with the default permissions. Now we still run Nova and Neutron as unprivileged users inside of these containers. If they need to do something that requires root access, they're still going to have to do that via the rootwrap library which is going to invoke sudo. Now this is a bit of a sledgehammer. I walked in at the end of the last talk and the presenter before me was talking about capabilities. Since we've started putting services into Docker, they've added the ability to do fine-grained capabilities using dash dash cap add and cap drop. Now this does require a little bit more work because you have to figure out what your minimal privilege set is. And it is going to make those command lines a whole lot longer. This is not work we've gotten to yet. But the bulk of those command lines that you saw up there are actually these dash v commands. And this is what Docker calls data volumes. You'll also hear them called volume mounts or sometimes volume bind mounts. These are similar to bind mounts if you're familiar with them in Linux. Now for us these fall into two different usage categories. The first is, we want to store as little state as possible inside of containers. And one of the ways that we achieve that is by actually storing that state on the host. So sometimes we will share a directory effectively from the container to the host to store that state outside of that, keep our containers more disposable. And then the other scenario is sharing resources from the host into the container. Now this slide has some of the more interesting Docker volumes that we're sharing into our open vSwitch containers. The first example here is an example of sharing resources from the host into the container. It's just Etsy Neutron. We're saying Etsy Neutron on the host. That's the first part. It should be made available at Etsy Neutron inside of the container. The RO on the end there says that's a read-only mapping. There's no reason for anything inside of the Neutron container to be making changes to the Neutron config file. The next example here is the opposite pattern where we don't want to store our log files inside of the container. We want to store them on the host. There are other ways to do them log file management, but this has the benefit that it preserves the ability to actually look at these files easily and tail those files without doing anything special. The next one here is a little more interesting though, run open vSwitch. Now this directory contains a couple of different Unix domain sockets. Those sockets are used by OBS and OBSDB to expose an API to any of the CLI tools or any libraries that know how to speak that protocol. This is needed obviously so that the open vSwitch agent can do its job at all. Lastly, I have a couple of other directories here that are kind of that storing state outside of the container pattern. These are directories that there's a lot of miscellaneous things in here. Things like DHCP lease files and generate config files, PID files, things along those lines that we don't want to lose just because we restarted or had to delete and create a new container. I'm going to simplify the format here a little bit because generally we're mapping from the same directory on the host and inside of the container. This is a list of the more interesting directories that we're sharing from the host for Nova Compute. We have Etsy-SEF here. This is so that it has access SEF keys and this is mounted read-only because Nova doesn't need to change SEF keys. We also have Etsy-ISCSI here. This is because we're using SolidFire and expose the storage over ISCSI. So those ISCSI commands that Nova Compute is going to be running to mount that storage need to be able to see their config files out in Etsy-SCSI. And we also need slash dev here also for that. And this is because Nova Compute wants to verify those block devices are in place and look the way that it thinks it is before it asks LibVert to attach them to the instance. In order for live migration to work, at least in some versions, Nova Compute wants to be able to SSH to another host, the destination host in order to verify a couple of things. So we need to share the host keys that we've already pre-populated on the host in the container. This is another one of the situations where there's no reason that the Nova Compute should ever need to change it. The next one we saw run OpenVSwitch. We saw on the previous one, Nova Compute needs to be able to plug things into OpenVSwitch so that instances will have connectivity. And the last one here is probably the most important one. This is very similar to the OpenVSwitch scenario that we talked about a little bit before. LibVert exposes its APIs with the Unix domain socket that it puts in this directory. And this is needed so Nova can do all the things that you want Nova to do with LibVert. Now, this is not the last time we're going to talk about command line flags, but I'm going to try and keep it to a minimum from here on out. Now, for Nova, most of getting it to work inside of Docturer is actually just figuring out how to start the container with the right flags. But Neutron was a little bit more involved. One feature we were really eager to start using when we upgraded to Liberty was HA routers. This addressed a pain point that we had for a while. And what that allows you to do is have automatic failover between network nodes if you have a problem on one for virtual routers. Now, not giving up anything whenever we moved Neutron into Docter was a real big priority for us. We didn't want any regression in functionality. And when we started looking at this, we realized that there were some interactions between HA routers and Docter that weren't going to work right out of the box the way that we wanted them to. So just real quick, let's talk about how HA routers work for us a little bit of background. The short version is that Keeple-IVD provides most of the functionality provided by HA routers. Now this is used for failure detection, but it's also used for managing the IP addresses that are associated with that virtual router. The L3 agent is responsible for starting up a Keeple-IVD instance on two different network nodes. And the Keeple-IVD instances on each of those network nodes are going to send heartbeats back and forth. If one of them is in standby and he doesn't hear from the primary on a regular basis, he's going to assume that primary has gone away, something bad's happened, and he's going to take over those IP addresses. The other failover scenario that we see pretty regularly is if you shut down Keeple-IVD cleanly, it will actually, and it's the primary, it will actually release those IP addresses and notify its counterpart on the other node. So those are the two failure scenarios we've seen most often. Now that IP failover isn't instantaneous, there is some time involved to move it over. It's a short period of time, but there is a data path interruption. And you can also end up dropping traffic because the NAT mappings associated with floating IPs aren't replicated over to the other nodes. So this does have impact, but it is a huge improvement over the manual processes that we had in place. We've been very happy with HA routers overall. But this gets us to the actual problem with HA routers and Docker. Because that L3 agent, or because the Keeple-IVD process is started by L3 agent, it's going to run inside of the container that we're running L3 agent in by default. That means if we have to restart that L3 agent for any reason, whenever it shuts down that container and cleans everything up, it's also going to kill the Keeple-IVD process. And we just talked about how this isn't an impactless problem. We're going to end up with a failover event just because we needed to restart L3 agent, and that's going to be impacting to our customers. Now this leads to some kind of bad behavior. I'll give you an example. Say that you need to make a change to a L3 agent config option. Maybe you're just turning on debug logging to troubleshoot something. In order to deploy that, you're going to have to restart all your L3 agents. That means that as you deploy that, each one of your HA routers is going to experience at least one failover, and some of those will see two failovers as it fails over and then fails back. This was going to be a regression that we weren't looking for in moving to Docker, and we weren't really looking to move into Docker and then have things actually get worse. That wasn't part of the plan. Another thing I want to notice is that the DHSP agent has a similar issue. DNS mask is also going to run inside of that container by default. Now there's a lot of redundancy around a DNS mask. You're usually running at least two nodes, but you can also have DNS service impacted depending on how you have things configured. But clearly this isn't as bad as a virtual router failover event, but we can solve both of these problems in a similar way. Now in order to get around this problem, what we want to do is we want to separate keep alive D and DNS mask out from the agents that start them. And to do that, we run keep alive D and DNS mask. Each of those processes in a separate container. In order to do that, we had to be able to convince L3 agent and DHCP agent to start those containers since they're the only part of the process that knows when they need to be started or when they need to be shut down. Now by doing this, we can avoid those keep alive D and DNS mask failures that we've already talked about, but there's a couple of changes to actually get this to work. The first step was to be able to start Docker containers from inside of the L3 agent and DHCP agent Docker containers. And there were kind of two pieces to that. First we need to be able to talk to Docker engine from inside of this container. This is pretty straightforward. Docker exposes its API similar to OVS agent and libvert by exposing it over Unix domain socket by default. Now that socket is var run Docker.soc. This is a little different than what we saw with the OVS and libvert examples. It's a file directly in there, but we can actually share that file directly in there. It doesn't have to just be a directory that we share. Next we had to add the Docker client so that we could actually talk to the API. And that was pretty straightforward. We actually just added the package into our Neutron image that we were already using for our testing. Now one thing to note about that is that the API between the client and the server is versioned. It tends to change every minor release for Docker. So if you're going from Docker 1.11 to 1.12 or 1.12 to 1.13 one day, that API will generally change. But the engine does typically support multiple versions of the API. And so this means that you will occasionally have to update the client inside of that image, but not every time you do a Docker upgrade. Lastly I want to point out that the Neutron user inside of this container cannot access the Docker engine. Only root or a member of the Docker group can access the Docker engine by default, and that's the only ones that have access to that socket. And that means that Neutron itself only has access to Docker by executing commands through via rootwrapped and sudo. Next part was we had to figure out, well, how do we get L3 agent and DHCP agent to actually start Docker containers, because that's not really built in. Our solution for this was pretty simple, not too gross. We wrote some scripts that can pretend to be keepLiveD that we are DNS mask, and we put those out in user local bin in our images. And then whenever L3 agent or DHCP agent wants to start one of those processes, user local bin is first in the path. So these processes are able to intercept the invocations of those processes. And then what they do is very simple. They look at the command line that's been passed in, they try to determine what the router ID or the network ID that's being associated with it, and then they start a new container for those processes, passing the command line through to the Docker run command. The container names that we used for this are really simple. We're just doing keepLiveD-routerID, DNS mask-networkID. So we have one per process. We did have to update the rootwrap filters for Neutron. This is so that the rootwrap library would realize that we want to invoke these things in user local bin. It does have the user bin paths hard-coded there, so we were able to just extend that. And then we also needed to add dash-pid-host to the invocations of these containers. The reason for that is that L3 agent, DHCP agent determine whether or not their child processor continued to run by reading the PID files that they've written, and then checking to see if those processes are still around. But at fault, those containers are only going to have visibility of other processes inside of the container, and we needed them to be able to see outside of that boundary. So that was one more thing that was necessary. So the last thing we needed before we could really say that this was going to be a stable thing that we could use in production was Docker 1.12. Now until Docker 1.12, whenever you restarted Docker engine, all containers on the host would be restarted also. This is something we could work around in a lot of other circumstances, but since we've put all this work into making Keep Alive D be a long-running process, it felt like it was going to be kind of a waste of time to have Docker upgrades and restarts and such like that, just, you know, kind of throw that to the side. Now, Docker 1.2 upgrade went pretty smoothly. However, as I mentioned before, we were mounting just that socket and var run into those containers. So the first time we restarted Docker engine in one of our dev environments where we first deployed this, the good news was is that all of our containers continued to run, which is actually after running Docker for a while kind of miraculous to see. The bad news was is that none of our L3 agents or DHCP agents could create new networks or new DHCP agents. The reason was is that Docker engine, whenever it starts up, it deletes any old socket that's there and it creates a new one. But inside of those containers, they still had the old socket that was not connected to anything anymore. So we had to, we'd spent a little bit of time brainstorming around this and came up with a fairly straightforward solution. Docker engine has the ability to actually have multiple, to listen on multiple sockets. We configured it to listen on a second socket in the inventively named var run docker-sock, docker-dot-sock. And then we changed our tooling to share this directory into those containers. And then we updated the wrapper scripts to actually use this socket inside of the containers. Now this works because whenever it deletes that old socket and recreates the new one, it's going to be visible in the directory instead of just having this statically mounted socket into the container. So the final outcome here is that this works. This is an excerpt from Docker PS on one of our network nodes that's hosting virtual routers. I realize it's a bit of an eye chart. It's probably really hard to read in the back. If you could read this, you would see that it says we're running Neutron L3 agent, we're running Metadata agent, and the open vSwitch agent on this node all inside of Docker. But then you also see a partial list of the keep alive D processes running. For every router that's, well, before it gets cut off. So this has worked pretty well for us. I want to share this approach. I'm not aware of anybody else that's kind of had a solution to this particular problem, so hopefully it'll be useful to someone. So one last thing that I wanted to cover, one last major thing. Last thing I want to talk about is how to get Docker to play nicely with network namespaces. Now network namespaces are kind of an esoteric topic, but they can have some confusing interactions with Docker that are really not obvious. I want to break this up into three parts. I'm going to talk briefly about what are network namespaces. But more importantly, we're going to talk about how they work, because the interaction between Docker and network namespaces is really not obvious, and I found a lot of this to be really unintuitive. And lastly, we're going to bring this back around to how does this actually interact with Docker. So if you've used network namespaces very much, you probably realize that IPNet and S is the primary CLI tool for interacting with network namespaces. This is the first couple of paragraphs of the man page where they attempt to explain it. It's not very layman-friendly. The first paragraph here says simply, a network namespace is logically another copy of the network stack with its own routes, firewall rules, and network devices. This is not too hard to follow. What it means is that network namespaces allow you to partition the networking on a host into multiple areas or namespaces. And they're not going to interact unless, other than in ways that have been explicitly specified. And this allows doing things with neutrons, such as per-tenant private networks that may or may not have overlapping address space, without the kernel getting confused about which packets belong to which tenant. If you have a networking background, you're probably familiar with this as VRF or virtual routing and forwarding. The second paragraph here is also not too bad. What it's saying is that by default, everything runs in a single network namespace. If you want to do anything special in network namespaces, you're going to have to ask for it. The kernel's not going to spring it on you. In our case here today, neutron is the thing that's going to be talking about network namespaces, and specifically DHCP and virtual router services, it's going to be using it for enforcing isolation between networks. And the last paragraph is the one that's most relevant about this particular topic. One thing we can get out of this paragraph pretty easily is that var run net and s is an essential part of doing anything with network namespaces. So we're going to need to make sure that our containers have access to this directory in order to be able to run any namespace commands. And we can do that pretty easily using the dash v flag that we've already talked about earlier with Docker volumes. But it turns out that that's not going to be quite enough. So we're going to talk about why. The primary thing this paragraph is trying to say, though, is something about how network namespaces are implemented. It seems to assume that you already know that. What it's trying to say is something about how namespaces are created and persisted. And that's the part that's really relevant. So what happens if you run ip net and s add test and you want to create a new namespace? So the short version of this is that what's doing the bulk of the work in creating that namespace inside of ip net and s is actually really simple. There's a system call called unshare. And it's going to call this with the clone new net flag. Now, this clone new net flag is going to take the, it's going to put ip net and s in an empty brand new network namespace that's not shared with anything. And it's also going to change this link out in proc self ns net. Now that points to whatever namespace the process is currently in. So it's going to point it to a new one. Now, this is a sim link to kind of a magic file that represents the network namespace. It's not a real file. You can see what that looks like down at the bottom. Maybe the red might be hard to read. So using, one of the things it's telling us here is that the namespace's lifetime is determined based on something using that file. And there's a couple of ways in which that file can be used. One is that you can have a process running in that namespace. That counts. Or you can have something that has that file open directly or indirectly. But if nothing's using the namespace, it'll automatically be reaped by the kernel. It's going to disappear. Everything inside of it is going to go by the wayside. So the question is how does ip net ns keep this namespace from disappearing whenever it exits? There has to be a mechanism for persisting this. This is the part where I think the author of ip net ns is actually pretty clever. You know, any reference to that file will keep the file alive. Now if you run strace ip net ns test, you'll see down towards the bottom of the output these four commands that I've got up here. And this is pretty straightforward. The first two there are creating a file and then closing it immediately. This is just going to create a zero-length file. The next one is creating that namespace using the unshare syscall that we've already talked about. The last one here is kind of weird though. It's a mount syscall that is mounting that magic file from proc over the new file that it just created, this new empty file. So this mount line is really the key to how namespaces are persisted. This is going to create an alias from that magic namespace file out in proc that is going to disappear when ip net ns exits. And it's going to bind it over this file that it just, this empty file that it just created. Now this means that whenever the ip net ns process exits, the proc entry is going to disappear. So it won't be referencing the namespace there anymore. But this empty file there is still going to be bind mounting that file and that magic namespace file. And so the namespace is going to persist. And then at that point we can start sharing it across other processes on the system. Now that's all great, but none of you signed up to come to a network namespace talk. So what you probably actually want to know is what does this have to do with Docker? Because we kind of gotten off on a side topic here. What this has to do with Docker is that Docker volumes interact with file system mounts in ways that are a little strange, I think probably unintuitive for most people. We just talked about how network namespaces are actually represented as file system mounts. And we talked earlier about how we're going to need to share var run net ns into these containers for namespaces to work. So we have these file system mounts on one hand and then we have this thing that we know that they may not interact with very well or not ways that we really understand. But we need to make this work for our L3 agent and our DSP agents to be able to run inside of Docker at all. So the question is is that if you set this up and you expose that directory into the container, what is going to happen? And the behavior is not what I think most people would expect. The first part here is pretty intuitive. Whenever you start the container, any mount points underneath that directory will show up inside of the container. But if you mount anything new inside of the container, it's not going to show up on the host. If you mount anything on the host new, it's not going to show up inside of the container and unmounts have the same asymmetrical sort of behavior. Now this is because whenever you start up that container, effectively those mount points are copied into the container, but nothing is going to be keeping them in sync by default. And this is critically important if you're using namespaces because namespaces effectively are file system mounts. So you're going to get an inconsistent view. And I want to talk a little bit about what that's actually going to look like because we learned this the hard way. So I ran a quick test to kind of show what this might look like. What I have here is I created a namespace called test1 on the host. And then I started a Docker container in the background that mounts var run net and s into the container and just sleeps for an hour. Then I created a second container called test2. Now the question at this point is what does the var run net and s look like on the host and then inside of the container? So on the host side of things you see var run net and s looks pretty reasonable. We've got two files here, one for each namespace. And if you look inside of proc mounts you see that we have two mount points there and it's using this weird NSFS thing which is actually only in the newer kernels that used to be represented some other way. And this stands for namespace file system. So we see our two mount points. This all looks good. This is what we want. But if you look inside of the container it's not as happy of a picture. You see that test1 which we created before the container started, it looks good. It's a normal file. Everybody can read it. And if you look inside of proc mounts you see there's no mount point there. This is just a normal file as far as the container's concerns sitting inside of this directory. Now anytime you see this things have gone horribly wrong. There's a lot of different ways you can end up in this scenario. I'll give you one quick example of the one that we found most recently. It took us a while to find this one. We run neutron net and s cleanup out of cron every 30 minutes. We go and clean up any namespaces that nobody's using. Now that process, whenever it starts up it catalogs all the namespaces on the system by cataloging this directory. So anything inside of this directory it says that's a namespace. Now if it finds anything that's not a namespace in this directory it's going to die during the catalog phase which means it will never clean anything up. So I'm embarrassed to tell you how long this was broken in our production environments before we actually discovered that it was a problem until it took more than 30 minutes to catalog the network namespaces we never noticed. But at that point you had once overlapping and they started contending for this directory in ways that were really unintuitive. And the fix was easy. We deleted the file that was in that directory. This can fail in a lot of ways. This is actually the most innocuous sort of failure that we saw. Anytime you're trying to share this between containers you add namespaces here and delete them there. By default you're going to get this really weird behavior that doesn't do what you want and it's really hard to puzzle out what's going on. So how do we fix this? The answer is actually pretty straightforward. It uses flags to Docker volume mounts. Underneath the covers this is using a kernel feature called shared subtree flags. Now these flags have a lot of flexibility that affect how propagation happens between the container and the host. And in Docker 1.10 they added support for this. That was actually one of the first releases where Kola had the ability to run the neutron container separately instead of in one giant container. And actually thanks to those guys they were actually the ones that told me about the shared flag. I have a link here to the kernel documentation. If you have some problems with insomnia it's very complete but not thrilling reading. Oddly enough the best documentation I've found on this is actually not on Docker's website but I think it will be eventually. The GitHub it just hasn't been published yet. And it's on the service create command for some reason which I don't even know what that does. It's not something I've ever used. But the documentation there is actually pretty good. There's a number of different flags besides shared that control different directionality and private behavior and things along those lines. So if you're interested that might be a good reference for you. But the short version of this is that anytime you're using network namespaces you do need to mount that run.ns directory into the container. So make sure that you use this shared flag. That shared flag is one of the many flags there but what it's going to do is it's going to give you bidirectional behavior. The thing that I thought was how it was going to work out of the box and that if you mount something inside of the container it'll show up on the host and vice versa and the same behavior for unmounts. Now this shared flag can actually be used anytime you're doing file system mounts where you want them to show up inside of the container or on the host and you're doing them from the other side. Not just network namespaces. So that kind of leads into the bonus topic. How are network namespaces related to sender? So you might have a better idea now now that you know that we're talking about file system mounts, not weird network stuff. So specifically the way we ran into this is with the NFS backend for sender. In our smaller environments we sometimes use the NFS backend for sender because we don't want to spin up a SEF cluster to test something real quick. Now the NFS backend for sender doesn't want to let you just run just put the file system mount in SEFS tab. It wants to actually run the mount command itself. Now if you're going to be running sender volume and Nova Compute inside of containers they're both going to run that mount command and the problem you run into is specifically with Nova Compute. Nova Compute is going to mount that share. It's going to create the file that's there the file will already be created by sender volume and then it's going to ask Libvert please attach this file to the instance and Libvert's going to say I can't see that file I'm not inside of your container. So we need to figure out how to make that mount point visible outside of the container automatically. Now this is basically the same solution that we had for network namespaces. You can see a couple of commands here down at the bottom. And you can see this actually looks a little bit more complex than the sender scenario. The reason being for that is that for the shared flags or any of these shared subtree flags to work you need two things to be true. You need to have a file system mount point that you're going to set the flags on. You can't do it on arbitrary directories. But then you also have to have that flag set on the host side of things. Now you may be thinking well how did it work with network namespaces? So for network namespaces var run or run net and s is actually a file system on its own outside of the box. Operating systems already set that up for us. So that's just happy chance. For the shared flag though nobody's, I don't think any of us have ever actually set that on var run net and s. And that's because IP net and s is actually setting that shared flag on the file system. Every single time it does anything with that file system. This was identified as a problem and I think the solution was just solve the problem ourselves. So the first command here is actually going to solve our first problem where we need var lib sender mount to be a file system in its own. There's a couple of ways to solve this problem. I chose a kind of gross way of just bind mounting the file system on top of itself. This works. It's gross. Then the other next command is to run mount with this dash dash make shared flag. And it does exactly what you think. It's going to set the shared flag on that file system. Now once you've done that you can then start sender volume and nova compute with the dash v run net and s, run net and s colon shared flag. And at that point whenever nova compute actually mounts that file system inside of there and asks libvert to attach that file to the instance, libvert's going to be able to see that file. This was kind of the solution to us for our dev environment. This isn't something we're using in production but it's worked okay. And that's everything I have today. If you want to get in touch with me feel free to send me an email reach out to me on Twitter, IRC. I'm going to be around for the rest of today and tomorrow so if you can catch me this week. Otherwise I think we have about five minutes for questions and I don't know if we have a mic. Okay, sounds good. Thank you.