 Right. Good afternoon, everyone. Thank you for coming out. We're here to talk today about Aconda, which is a new open source, production ready open source network virtualization platform. I am Jonathan LeCour. I am the vice president of Cloud at Dreamhost. Dreamhost is a service provider. We've been in the web hosting and cloud services business for about 16 years now. Have over 400,000 customers. And we've got some really great cloud products built on OpenStack and Ceph. My background is engineering. I've been programming since I was 15 years old. I've been using Python since 1.4, which is a really long time. And I'm a member of the Python Software Foundation. I've been using OpenStack and been involved in OpenStack since the Diablo days. And this is my cohort, Ryan. Hey, everyone. I'm Ryan. I also work at Dreamhost, where I've been for about three years now. I'm a veteran of Python software development. I've got some experience with Python service-based startups in the past. And I work on Aconda at Dreamhost. So let's talk about what we're going to actually be doing today. We've got a lot to cover, so I want to give you guys a little bit of road maps, where we are. So first, I'm actually going to talk about not Aconda. I'm going to talk about some context, some history around why Aconda was born, and why we needed it, and how we developed it. I'm going to introduce the technology a bit. And then Ryan's actually going to dive very deep into what makes Aconda tick, and how the actual platform works. And then we'll talk about operations as well. So we actually use Aconda in production today in our OpenStack Cloud to provide all the Layer 3 plus services in our compute platform. So we have a lot of experience operating it. So we want to share some of that as well. And then we'll do a little bit of a retrospective. Talk about some of the decisions we made, both good and bad and why. And then we'll talk about the exciting future that lies ahead of Aconda. There's some really cool news that came out yesterday about it. So we'll start off with some emoji here. Did anyone know there was a little baby's head in emoji? I thought that was pretty funny, so I tossed it in there. We'll talk about the birth of Aconda. So that starts by talking about something other than Aconda, Dream Compute. Dream Compute is Dreamhost OpenStack Public Cloud. It's been a long journey to get it out. We are in public beta as of last week. So you can go to our website and sign up right now and consume our OpenStack Cloud. It's really cool. It is built on top of OpenStack, as I mentioned, and on top of Ceph, which we use for block storage. Ceph was actually born at Dreamhost. It was technology created by one of our co-founders, Sage Weil. We developed it in-house for some time and incubated it. And then we spun out a company called Ink Tank that you may have heard of and sold them to Red Hat earlier this year, I guess. So it was a very cool and interesting story there, and it's been an interesting segue into Aconda for us. Similarly, we developed Aconda in-house to provide some of the networking for Dream Compute. So it's KVM hypervisor. You guys all have seen. I'm not gonna bring up Horizon and spin up a VM. I think you guys know how it all works. The interesting thing about Dream Compute is we had some design tenants that were, some would say a little bit aggressive, but very interesting and very principled at the beginning of the project. We really, really wanted to provide every tenant in our public cloud with complete L2 isolation. We wanted to do that in a scalable way and in a way that would last for many years. So we didn't wanna use a lot of the traditional ways to do that. We really wanted to have virtual L2. So we also know as a smaller provider, we don't have an unlimited bank of IPv4 addresses and we also can count and know that there aren't that many of them. So the future is IPv6 and we wanted IPv6 capability natively from the beginning. And this was, mind you, three years ago. This was before Neutron existed. So there was no open source solution to provide us with what we wanted. We went out to a vendor and we got L2. We went to a company called NYSERA. They're now VMware and they have NSX. It's a great product. Great guys, we love working with them. And it was really working well for us but we needed Layer 3. We needed something to provide all the routing and other services to our tenants in Dream Compute. First thing we did, being Dreamhost, founded on open source software, built on Linux Apache, MySQL, so on. We wanted an open source solution and we didn't really find anything that fit the bill. There were some things out there but many of them were being controlled by vendors who didn't necessarily understand open source or cloud, didn't understand the model and didn't support IPv6, weren't interested in supporting IPv6. So we just thought, how hard can it be? Let's do this on our own, right? Words of folly that everyone has uttered, every engineer has uttered in their time. So that born, that birthed a conda. And when we first started out, the idea was we were gonna run OpenBSD in a service VM inside of each tenant, plug it into their V-switch, use packet filter, PF to provide all the services that we need, build a little API and it would be really easy. And turns out it's really hard, really, really hard. As we went forward, we realized we had made some missteps, we iterated and iterated, tested and tested and came to kind of the model that we're at today. So we determined that OpenBSD was not really well suited for our use case. So actually if you had looked at the talk description, I think it still mentions OpenBSD as the foundation. And it was for a long time and we got very far with it. It's a great OAS, we love OpenBSD, but it wasn't quite perfect for our task. So there was some community resistance in OpenBSD to virtualization. When you go into the OpenBSD kind of mailing list and mention that you're doing things that avert environment, it's not as accepted as it is in say Linux. We had some issues with drivers and some network throughput and under a virtualized NKVM, booting an OpenBSD image was taking three to five minutes which is a very long time for our opinion. And also we needed hot plugging to do some other features that we were interested in and it wasn't quite working well for us in OpenBSD so we were having to reboot routers and service VMs whenever we wanted to do certain things like add networks. And everything else is Linux, so naturally we decided to switch to Linux. Now at the time we already had a condo working and it was just a matter of spending a brief period of time and by spending a brief period of time, I mean locking Ryan in a room for a few weeks and moving over. So we switched from OpenBSD to Linux, from packet filter to IP tables and we got a much larger community, a lot of support, a lot of knowledge we already had in the house around IP tables, we got better performance and now we can build and reboot service appliances, service VMs in about 45 seconds which is a much more acceptable kind of time period. And we also got hot plugging support and Ryan will address why that's interesting in a little bit. So as we start slicing down here, let's look at the architecture for a condo and kind of how it works at a high level. This is the current architecture and there are boxes in here that I'm not gonna touch on right now. But the idea basically is that a condo has this service VM that runs inside the tenant, as I mentioned and it can provide a variety of virtual services. You could call this NFV if you want to, that's fine. I kind of prefer to think of it as service virtualization, network service virtualization. So we've got routing, additional services and development and then we plug into the OpenStack APIs, some extensions to Neutron and we have a back end right now on top of our VMware NSX L2. We realize that other people may not want to use VMware, they may want to use something else. So we're gonna work on pluggable back ends there. We'll talk about that more in a bit as well. And the thing that kind of is the hub in the whole model is this orchestration platform called the rug. That used to stand for a thing but we'll get into what that stands for now. So before we jump into depth on the technology, let's talk a little bit about whether or not this was all worth it. At the end of the day, Dream Compute, it works really well because of Aconda and we're very excited about that. It was not an easy decision. We actually have done bake-offs between Aconda and some other technologies that are out there these days and it still is beating everything else for our use case. That's not to say the other ones aren't good, they're great. And IPv6 support came right there out of the gate for us and that was very important to us. So now if you sign up for Dream Compute, you spin up a virtual machine, it will get an IPv6 address and an IPv4 address. Everybody gets them, it's included in it. And it's really open, it's not sort of open or we throw the source code over the wall every once in a while, it's up on GitHub. You can see every commit we make, you can contribute yourselves. And also it's just Linux, right? This is a beautiful thing. When it's just Linux, you can run anything you want on these service VMs. So it opens the whole world to all these additional network functions that we can add to the platform. And stability, it's working today, it's in production. Customers are using it every day. So big wins. Now Ryan is actually gonna talk about in depth how Aconda actually works. Thanks, John. So Aconda is primarily composed of two major components. Like John mentioned, there's the service VM itself, which is a VM running Linux and a variety of services. And then there's also the rug, which is the service that orchestrates, monitors, and manages these service VMs. So let's start out by talking about the appliance. So we build these appliance images with a piece of software called Veewee. Veewee isn't the only option, but it's the one that we've sort of built our tooling around. And so the Aconda appliance code really consists of the different pieces of software, the Python software that we've written, code for generating IP tables and DNS mask and bird configuration. And then Veewee is what we use to build the image and we then upload that in the glance. So some of the pieces of software running on the actual appliance, IP tables, we use that for tenant address translation, floating IP address translation, DNS mask, we're using for tenant network DHCP. In our cluster, we actually use config drive, so we don't make use of this, but it does work. Anytime somebody goes into a Neutron subnet and sets up static routes, custom DNS servers, that's all managed by DNS mask. John mentioned V6 support for all the instances in a tenant network. So we do upstream routing and address advertisement with bird. We use BGP and RADV. And then we also have some Python services running on these service VMs. So the first one is a Python proxy for Nova metadata. So if a customer spins up an instance with Nova, that instance is gonna look for metadata at a well-known address. We have on the router a piece of Python software that is running and is listening for those requests, proxies them over the management network so that it can speak with Nova and metadata can work for those instances. And then additionally, this is probably the bulk of the code that encompasses the appliance is a Python service that the rug speaks to. And so this does a couple of things. First of all, it's an HTTP endpoint that the rug can speak to that provides an alive check. So periodically, for every tenant's router, the VM will just ask, are you alive? Are you there? So that it can determine if it needs to be rebooted, if it's unresponsive, those sorts of things. And then also, there's the configuration push so that as users create networks, subnets, ports, plug things in, create VMs. In reaction to that, we are pushing across configuration updates that change settings on the router and reload services. Oh, I missed a slide. So this is sort of what that configuration push looks like. It's a JSON data structure. We've really condensed this down because I want to make sure that people can read it. But if you've worked with Neutron at all, this should probably look pretty familiar. There's a lot of information here about a network and its subnets, the sitter for the subnets, the interfaces and their addresses, floating IPs. So this is what gets pushed across. And as a result of this configuration push, we reconfigure different services and restart them. For the addresses, that might mean for specific interfaces, we add an address or remove an address. For floating IPs, it might mean that we change the IP tables rules that we're netting properly, restart IP tables or reload rather, setting the fault gateway, that sort of thing. For reference, this API is part of the reason that Ryan was so quickly able to change the underlying platform. We went from OpenBSD to Linux, and we were using this higher level API to say, here's what we generally want, and I don't care how you do it. I don't care that it's DNS mask, bird, IP tables, Linux, OpenBSD, whatever. So that's a big win here, why we're not just pushing across IP tables rules and things like that. I think implementation is pluggable too. So I mean, for instance, when we move from packet filter to IP tables, none of this had to change. We just had to write a new plugin that understood what it needed to do with IP tables. So that brings us to the rug, which we think really ties the room together. The three of you apparently who has seen Big Lebowski, I was expecting more of a chuckle, come on guys. I think we were debating the other day, I was like, does the rug actually stand for anything? We thought it stood for a router update generator, maybe? But not really, I think it just really ties the room together. It's the orchestration, it brings it all together. At one point, I think we had an internal service, we named the dude, but that... We still have a service called the dude as well, but it's not part of a condo. It doesn't actually mean anything, we just wanted to call it the dude. Lebowski's a part of every network virtualization platform that's quality, in my opinion. So the rug does orchestration and monitoring of the service VMs. So let's talk a little bit about the architecture of the rug. So it's a complicated piece of software, but we can probably condense it into three pieces. I'll start with the middle one, which is the event processing. So some of you have worked with Neutron know when you update ports, perform various actions in Neutron. Notifications and RPC messages are sent across generally your AMQP message bus. So the rug has a process which listens to the notifications that we care about and then performs actions based on those. And not just from Neutron, from Nova as well. Nova as well, yeah, but primarily Neutron. Right. So the event processing listens for these. Some of those examples might be a router interface create or a port create end. We actually dog food this ourselves a little bit. I'll talk about this later, but so we actually can send custom messages to the rug to tell it to do things. And that flows through this as well. So all the events come in through this notification system and they're sort of coalesced down into actions. So some of these are pretty obvious. If a router is created, then that's sort of a create. We've got read, update, delete, crud sort of oriented actions that this gets coalesced into. And as we receive these, we put them into a queue that is processed by a Python multiprocessing queue with a configurable number of workers. As those events are added into the queue, the second component of the rug, which is the state machine consumes them, looks at the current state of a tenant VM and then decides what to do next and transitions through these states. So if a create comes through, depending on the state of a router, the next action might be to boot a virtual machine through Nova. And then finally, health monitoring is another area where we sort of dog food this event processing. So there's a background process that runs as part of the rug and every minute or however many minutes you configure it to do so, it pushes an event in which is basically ask every router if it's alive, if it's responsive, if not, we need to take some action. So this is the state machine and I know you can't read it and I didn't intend for you to be able to read it because it's very complicated, I can't even read it. There are 10 states there and there are a lot of different transitions. It's very sophisticated. So I'm going to go through a couple of the potential use cases here so you guys can sort of understand what's gonna happen when different actions occur. So some of the ones we need to know about, there's a create VM state which does exactly what you would think it would do. There's a configuration state which is once a service VM is actually booted we need to push the configuration that we mentioned earlier, restart services. Periodically you may want to destroy one of these service VMs. So those are sort of the actions you would take to manage your virtual machine. There's another category which is collecting stats and usage. So you can pass messages into the rug which are, give me information about bandwidth consumption for this given tenants router. And so down the line I imagine we'll probably have some type of integration with salometer. Right, this would be great for say if you were running a large public cloud and you wanted to, I don't know, track bandwidth on a pertinent basis for billing purposes. And then the live check which is something I mentioned earlier. So here's an example flow through the state machine. All of the state machine transitions start at this calc action state which calculates the next thing to do based on the state. So this actually happens when a new customer is signed up for a cloud, a dream compute account. A router is created for them automatically that ends up turning into an event that we capture from Neutron. And as a result of that the rug has logic where it realizes okay, there isn't a VM for this person we need to go ahead and create it. So we transition to create VM. From there we transition into a state where we're checking to see if it has finished booting yet. And so you notice that I've actually got arrows between check boot and calc action here. So this is actually a process that continually goes back and forth. We say have we checked if we booted, if not we yield back and some more work can be done. Maybe a moment later, this event arrives in the queue again. Has it booted yet? And maybe about a minute later a VM has come up and we've actually booted it. We need to configure it. So that pushes the configuration that I mentioned earlier. We get a response from the API that says yes it worked, all the services restarted. And at that point we go back to calc action and we have a service VM that is live and ready to do its work. Now across time as people plug in networks, boot VMs, we're going to do something very similar to this only this time we won't be creating the VM we'll be going straight to config. So here's another example. This is the health monitoring that I mentioned earlier. Again we go into calc action and this time we're receiving an event which is pull the router. We go to the alive state and in this circumstance it looks like we're trying to reach the VM over the HTTP interface. It's non-responsive. We've waited maybe a minute or two and we decided okay let's kick it out, let's just make another one. So you can sort of see the direction that this is going here. We're going to stop it, create a new VM, check the boot and we sort of continue that cycle. So let's talk about a couple other interesting features that the rug has. One of those is network hot plugging. So what that means is that as customers add or remove networks to their tenants, we can add them with no downtime, no required router reboots. And so that makes use of the Nova interface attach and detachment calls. So what happens is at configuration time in the state machine, there's actually a bit of code that asks, well router you're alive and I can reach you. What interfaces do you have and do they match what I think you should have? And in the circumstance where they don't, well that means that we need to hot attach a network or hot detach a network. So we do so and then we have a process where effectively we wait for Nova in the background to attach that interface and for it to show up on the live service VM. At that point we continue onwards with our configuration process. So I think you mentioned this is one of the things that BSD didn't do for us. BSD, OpenBSD has purported support for hot plugging. The problem is that the device drivers also need to support that and none of the ethernet drivers actually support it. So that was interesting. We could not get it working reliably with OpenBSD in a virtualized environment. So yeah, this was a huge win by moving to Linux. And I think that and some of the other stuff that you mentioned surrounding performance was part of the reason for our move. So another interesting feature. You've got this automated system that is managing these service VMs in the background and it behaves pretty well. Largely you don't have to care that it's working. But sometimes things go wrong. So you want to be able to track what's going on when things are failing. So the rug has an interface, it's a CLI which gives you insight into the state of all of these service VMs within your entire cluster. Excuse me, other states. And you can sort of see which ones are sort of in an error state. So if for instance the rug tries to boot a VM, maybe it fails and it retries a configurable number of times. Eventually what it will do is it will back off and it will say, okay well I just can't boot this router. Maybe Nova is having an issue, a hypervisor is down. Maybe there's a bug in the code that somehow is preventing the VM from coming up and being configured. Probably not, there's never a bug in the code. No, no bug's in the code. So this is the sort of thing that saves you from overwhelming Nova and just continually trying to boot things when they just will boot. We had a brief period where we had reboot loops as a problem and that got solved pretty quickly. You don't want that happening knocking over Neutron and Nova by just constantly rebooting a VM. Or a thousand of them in this case. So another interesting thing that the rug does for you. John mentioned that we use NVP. So NVP by default. NSX now, rebrand. NSX, yeah. I'm used to calling it NVP. But so they have this feature where for the Neutron routers is a thread I think that monitors the state of the router and then updates the database periodically. So for our implementation we actually have that disabled and the rug itself actually manages that based on the state of the state machine. So as for each tenant the rug is sort of monitoring the position in the state machine of each router. It's actually calling out to a Neutron API and saying, okay, here is the state of this router. It's currently in build state because I'm building it. That's nice if a customer actually wants to be able to use the APIs or go into Horizon and say, hey, what's up, what's up with my router? So they can actually go in and they can see, oh, my router is, I've just signed up. It looks like my router is still in the process of being built. So that's sort of a high level bit of detail about how a con actually is implemented and how it works. So let's talk about how we actually run this in production and what it means in practice. So I mentioned earlier that we have a CI process where we use VWI and we build our service VMs. We have this code on GitHub internally on our Jenkins CI. We have a process where we monitor the GitHub repo and anytime we pull in a pull request and merge it we basically, we build a new VWI image. We can gate against those and test them. If we're satisfied with them, we upload them into Glance so that they can be used. First in our staging environment then in production, but yeah, either one. Ideally, yeah. No, sometimes straight to the production, right? So once you've got this image in Glance, there's a configuration option for the rug where you basically tell it the UUID of which image that you want to use. And so from there, the rug does what it does. It actively monitors the tenants, looks for missing ones, creates configures and keeps them alive. So the rug also has a CLI called rug CTL, which we can use for sort of managing and getting insight into the tenants. So there's a handful of commands that we care about, but I'm just gonna go over a couple of the really important ones. Rug CTL browse is an interactive interface for getting a list of the routers, determining what state they're in, whether they're building. It shows details about the actual Nova VM so that you can see when it was created, what image it's using. You can sort of page through these on a case-by-case basis. You can say, well, this one is misbehaving. The customer's reported issues and there's something weird. Why don't we go ahead and restart it and see if that fixes things. Rug CTL debug is also a really useful one for us for development purposes. So you can imagine that in an environment where you're spinning up all these VMs, you're managing state, things are changing in your restarting services, you're trying to debug something that's going on with one of these and the rug is managing it for you. There's a pretty good chance that while you're working on it, it's gonna cut you off at the knees. Before we had this, it was very, very frustrating to work on this because you would go in and investigate and look at a log file and then suddenly your SSH connection is gone and it's because the rug destroyed your VM. So this is a mechanism for basically saying, okay, I'm going to take a look at this VM for a short period. Go ahead and just don't work on it and then at a later time, once you're happy, you can say, all right, it's yours again. Handle it for me. And then finally, there's a command for manually rebuilding. So we actually use this for some of our testing. Periodically as we make changes and we wanna stage them or sometimes we have select customers who will say, you know, you told us about this thing that you wanted or we fixed this bug and do you wanna be a first adopter and try it? So this allows us to on a tenant by tenant basis, basically say, well, here's the special image that we've built that's beta. Go ahead and reload it for this individual tenant so that they can play with it and we can sort of do some testing. We also use this kind of framework in order to do rolling upgrades of tenant routers as well so that we don't, you know, DDoS Nova by basically saying, do all the routers right now, please. It just, you know, we wanna do them a little bit at a time and kind of roll them through. But yeah, rug CTL is great. Someone asked me the other day if there was a user interface and I said no, not yet, but that would be a pretty cool thing to add to Horizon for admins. So I think I'm gonna hand it back to John for a little bit of a retro. Thanks, Ryan. That was really good deep dive. So let's do a retrospective here about some of the problems we faced when we went through this process with the conda. So remember I, you know, and I, my discussion, we started this project a couple of years ago and at the time there was no such thing as IPv6 support in Neutron. So we had to make some modifications ourselves and as many of you know, we're still in the process with Neutron of getting IPv6 fully working and it's really getting, they're making great progress but we're kind of early adopters and so we had to go through some of that pain. We still have some differences in deltas between our version of Neutron that we have to run and mainline Neutron. So we're working on getting that all upstream, moving to everything to make it more vanilla and I think probably the biggest challenge that we faced is the fact that state machines are hard and distributed processing. It's a really difficult problem. The appliance itself is really pretty straightforward. You know, having a service VM and wiring it up to your layer two and to the tenant itself is pretty easy to do. You can do it manually and you can automate that pretty well but if you wanna manage these things at scale and push out configurations, monitor them for their health and make sure everything transitions through the right states without getting into reboot loops and DDoSing, Nova and Neutron, you really have to have a very sophisticated system and the rug took a lot of development time so that was a really big challenge. Another big lesson that we learned is the best way to achieve stability as quickly as possible is to just, you know, automated in an automated fashion beat the hell out of it, right? We spent some time spinning up thousands and thousands of VMs a day and destroying them in an automated fashion, monitoring for errors, creating networks, destroying them, just absolutely destroying and demolishing the cloud to really exercise the rug and exercise a conda and that turned out to be the biggest win by far so we're big believers in that and a conda's gone through a lot of hardening as a result so we're really happy with where it is and another big challenge is as a small team, you know, keeping track of OpenStack as it's exploded with, you know, thousands of contributors and hundreds of companies involved is a difficult task and keeping up with mainline OpenStack is not always easy to do but we're excited about that and especially excited because of the future so earlier I mentioned how DreamHost incubated Ceph, the technology and then created a company called Ink Tank and really helped push it forward into the community and make it an awesome thing that we're all using and that turned out really well it worked well for the Ceph community because additional features, capabilities, bug fixes and support, it worked well for customers, people who are consuming Ceph because they can now get support and could get help making it run and scale so it was really great for everyone involved, contributors, community and so on so obviously you can see the writing on the wall here we've actually launched Aconda as a separate business we've spun it out and got a CEO on board who's here in the room, his name is Henrik, wave Henrik we're really excited about this and it's kind of like, you know, that's why I use the little baby icon for Aconda it's our little baby, we're setting it free and we want everyone else to use it as well we really want the community to take advantage of it as an open source project, right we're not taking this thing closed source it's a real open source project, a community that we want to start building so we'd love for you to get involved to help us build kind of the future here and some of the things that we have on the roadmap as I mentioned earlier, additional services, right these are just Linux service VMs a lot of the NFV talk right now that's going on is talking about all the different functions you can virtualize, network services that you can run in these little Linux VMs and wire them into your networks kind of the sky's the limit here but we're going to start with load balancing and firewall because we feel like integrated with Neutron's APIs that would be a really good spot for us, Dreamhost so as a customer of Aconda we're going to ask for that and help contribute to that put the code out in open source for everybody and we also know that not everyone in the world is going to be running VMware NSX and that's fine you know, we like it a lot we think it works really well for our L2 have a good relationship with them but we also know that other people might want something so we're going to be working on other L2 backends working with the open daylight community working with just going down to physical talked a little bit about VXLAN and some other options there so we'd love to get people involved on that front and then we want to add some kind of enterprise features to the rug itself some more HA kind of operation modes scale out we've also talked about taking advantage of the hot plugging features that we've worked on now to do things like having a pool of pre-booted unconfigured routers or appliances that we can then hot plug into the customer networks to reduce the time to availability for new customers or for additional services we've done some early testing on that and that works really well too so we're really excited go check out the code go to aconda.io to get more information it's linked to the Dreamhost GitHub for it eventually that'll move over to Aconda and both Dreamhost and Aconda are hiring so if you're interested in that as well let me know and wherever you are if you want to get involved in the project please do get in touch we are, you know, it's all under Apache 2 license bring your code, bring your bugs, bring your ideas, we love it so and that's it, thank you very much I think we've got about three or four minutes for questions maybe so if you have a question find your way to a microphone we've got one over here on your left I'm wondering how you implement the floating IPs do you do the one-to-one NATs or are you just routing those directly in can you talk a little bit about those details yeah, so for the floating IPs we do one-to-one like source and destination at NATs with IP tables even on the IPv6 side so we don't really have many people that do v6 floating IPs yet, right this is the thing we want to do right now the way that works and this is an area of improvement every VM gets a v6 address, it just gets one and you can't assign additional ones so obviously if you want to do things like failover using a floating IP on v6 you can't do that yet but we are going to work on that for sure and if I understand this right I mean this is basically an alternative to Neutron is that accurate? no, it absolutely is not it integrates cleanly with Neutron so Neutron has a pluggable backend as you know and as requests come into Neutron that's the API that is exposed to the world that's the API that people use to assign floating IPs to do anything and we just listen to the messages that come out of Neutron and then make things so, right we make it happen on the service VM so you're an implementation of Neutron routers is a better way to say it and other services and features we integrate with Neutron I think that would be the best way to put it thanks so these service VMs are visible to the tenants or is that hidden from them? so they are not they are actually they are under ownership of the Neutron service tenant so that way customers don't have to worry about that for quota and they are so they're not going to shoot themselves in the foot now they can interact with the service VMs in the sense that they can issue Nova or Neutron requests, right and those Neutron requests will get consumed by the rug and then cause things to happen with a service VM and they can also see their router state in horizon or through Neutron through the API so it's not completely out of their reach but the API itself listens on a management network that the tenant does not have access to so did you have to change any horizon or anything to make this VMs show up in that network? no changes to horizon we do have some changes in both Nova and Neutron we're trying to get as much of that upstream as possible and try to get away from having to have custom versions of that so that's kind of part of the next couple months of the development's gonna be upstreaming everything, all the things yeah and initially this started out as just like for us a big, big fork of quantum for the changes and over time, like you said we've slowly just been bringing things in so the repository might still be called quantum so for a very long time it was called quantum we'll have to make sure we talk to someone in the know on that, make sure we don't get sued let's see, I don't know who is next we'll go with you I just had two quick questions where does the rug run and like do you have any scalability information when you do DHCP, is it Slack or sorry, when you do V6 is it Slack or DHCP V6? okay, so second one first, Slack first one, so the rug runs inside of what we call our cockpit so it runs kind of alongside the other open stack services and in terms of scalability we are not even breaking a sweat right now and but we do think that there is opportunity from our scale out, right now it's based on multiprocessing and multi-threading so it can scale up a lot on a single node and we have failover but we don't have full on HA or scale out horizontal yet so that's in progress in the plans and break a sweat is thousands of service VMs? thousands and thousands, yes so one interesting thing is that actually in terms of scalability and performance issues so the rug itself, it's really just multi-processed Python code and much like Horizon I guess it's only going to be as slow as the endpoints that you talk to the real problem is not the rug I'll just put it that way we tend to knock over Nova Neutron yes if I understand correctly the data plane that you've added is that you've got an extra hop now to the service VM how does that affect performance for your tenants? I actually don't know if I have a good answer for that the throughput's really good that's one thing I can say in terms of how it would be versus other options I don't know if I have a good answer maybe we can talk afterwards thank you so you mentioned that you're planning to upstream most of them but there's an effort with Neutron to actually get service VM as an incubation project sure so I guess we're going to participate and roll this out yeah so big difference between Tacker and Acondas that Acondas working so we'd love to work with Tacker obviously but it's an early stage kind of thing so we wanted to get out there, get it upstream or get it open source and then engage and talk and see if we can work together I mean obviously it'd be good to be you know the same thing so yeah definitely interested obviously we want to go the way the community goes perfect it's 240 and we're done thanks guys