 Okay, can everyone hear me okay? Hey, good morning everyone. Thanks for coming. I'm Adam Johnson. I'm from Mido Kura and Mido net project and Today we're gonna talk about Mido net go through a one-to-one And cover the the architecture if we have time. I'll attempt a demo as well So before we get started just curious how many of you have actually tried Mido net so far Just raise your hand. All right a decent number great And I'll kind of go over how you can actually get started and try Mido net if you haven't if you're interested You can see the agenda there. I won't go through it because I'll Start with this so First of all, you know, why do we need why do we need Mido net or net virtual networks in general? You know, this is pretty basic stuff, but you know, basically there are requirements, you know Infrastructure as a service especially open stack has requirements like multi-tenancy More agility on the networking side open stack in general has allowed, you know, people to spin up VMs and storage in a matter of seconds or minutes but a lot of the People running in production still have kind of very crude networking that requires Manual intervention whenever you need to make network changes so you're either forced into very very simple networking models where you don't have advanced networking services or Your provisioning time is just very slow. You know, we talked to several people who run These environments in production with a you know, very basic networking models and you know They're provisioning times are lengthened by two to ten times Just because of networking changes that have to occur in production so Moving networking into software certainly can fix that just as it has for storage and compute So that's what this is all about is just making things faster making it more scalable and multi-tenant of course So before I get into how Mido net works This is kind of a slew of features that meet and that covers today. It's I'm probably missing some stuff But in general we're doing layer two through four Services and some other services around neutron. So our our goal was to Essentially provide all of the networking services that neutron needs in a fully distributed system So to do that we've embedded all of these services into Mido net itself And I think it's it's it's very cool. So so essentially we've As of the latest version that's that's in RC right now. We've we actually have Pretty much everything you don't have to run any of the Python agents of neutron anymore And you just run a single media net agent on every compute hosts So, you know the last pieces we had to do were basically distributed metadata. So even metadata DHCP And then all the normal networking stuff like layer two layer three switching and distributed routing firewall like security groups NAT Everything is covered in in media net There are some other advanced Kind of use cases that we also cover You may have heard of courier. It's was covered in the keynote in a few other sessions this week so we've been contributing to that project with the with the intent of connecting media net into Docker and in other you know related projects as well We also can connect into physical Networks using VTEP. I'll cover that briefly and then of course, you know the the main one we're talking about today is Open-stack and neutron integration So just a little bit of a background on media net media net started from Mito Kura the company We actually started Mito Kura back in 2010. We started here in Tokyo and It was originally a proprietary project our original version of Mito net we called middle man it's kind of a Japanese plan words and It was written in Python We hacked it up really quickly it was Essentially connecting to open v-switch in user land And we had some extensions And then you know it was providing basically layer three on layer three with encapsulation using GRE We decided to rewrite it Quite some time ago into Java for the 1.0 release and the reason we did that was mainly because we were As we were adding a lot more functionality and doing a lot more distributed stuff. It was you know becoming very difficult to maintain quality and There was just a lack of Tooling that we needed for kind of distributed debugging and things like this So we decided to rewrite in Java and very quickly after that we started introducing Scala I think most of the code now is in Scala if you go to our github page It says we're a Scala project We actually open source Mito net back in November 2014 in the Paris open-sack summit So it's been almost a year now We decided to open source Mito net because You know we were just seeing Demand from the market for an open source solution We saw a lot of fragmentation in neutron with a lot of different proprietary plugins. There's like 20 some plugins in neutron doing various things And it felt like you know there was just a need for this to you know try to get people to avoid You know reinventing the wheel and just provide something that people can use whether they want to pay for it or not so they you know Just to kind of solve a lot of the problems that we were seeing around the OBS project for production workloads and Also get into a little bit of meeting at 5.0 and actually a lot of the architecture is that I'll talk about is based on the 5.0 We actually jump from 1.9 to 5.0 because of the open-sack versioning changes So this is actually our fifth release. So we decided to kind of match Versioning schemes that way as well. So that's why there's a weird jump So You can go to metonet.org. There's a lot of information on there. That's the the home page of the community And I'll talk about the community a little bit later as well So getting into the architecture so Metonets architecture is fairly straightforward and simple This is like the 10,000 foot view here and I'll zoom in to get far greater detail in the next slide But I just wanted to show you the the concept of a virtual network with overlays So if you see the bottom of this diagram, this is the physical infrastructure And it is very simple on the left hand side where it says internet these boxes are just x86 boxes The little metonet logo there denotes that this is the metonets agent or meto man Running on those boxes. It's talking to the internet using BGP multi-homing Then on the bottom middle you see this network state database This is actually just running some minimal metonet software to do some Commands like VTAP and things like this and it's mainly a patchy zoo keeper So this is our topology storage, which I'll get into as well So this there's three of these because they run in a quorum. You can run some odd number of them Defending on how much you want to scale and the points that I want to make here is that this is not a Controller it's not doing any computational flows here. We do the computational flows at the edge So that would be on the left-hand side or the right-hand side So the left-hand side being the gateway nodes and the right-hand side being the compute hosts where the VMs or containers are So on the right-hand side the the metonet agent is also installed on those That's the same agent that runs on the gateway nodes and then on the top you can see the logical topology So in the logical topology, it's very familiar to neutron Those concepts pretty much map identically the only one that might not map On this in this diagram. It doesn't show it. We have a provider router Which which is proposed in neutron and I'll talk about that later So kind of zooming in a little more you can see more details of of the components here So really the only components of metonet in an install are the metonet agent or metoman the network state database Which is zookeeper and we have some cluster Things in in 5.0 which we're introducing and then the metonet plug-in in neutron and We have metonet API and metonet CLI These are just talking to the network state database So that's pretty much it. There's not much there's not much to it. It's very straightforward. It's very simple There's very few moving pieces But this basically allows you to have a fully distributed network so zooming into the compute hosts on the compute hosts we have the metonet agent which is talking directly to the Open vSwitch Datapath it's using a netlink channel to talk down to that So we're actually replacing the the open vSwitch user space agent with the metonet agent And the reason we did that was because we are adding a lot more higher layer networking functions into that So instead of you know having another agent that talks to user space OBS We just decided to talk directly to the kernel. It was simpler for us to do and a little bit more efficient and You can see the VMs on this host. We'll just have normal interfaces So if you do if config you'll see a bunch of tap interfaces metonet will plug into those to to to have control of the network and the metonet agent also connects down to the Network state database. You can see a little peek of that at the bottom So this is where the metonet agents get all of their states. So this is getting all of the high-level states And they'll cache that information locally as well So you don't have to always go off-box to get that the topology And you can view the metonet agent as kind of a network simulator So it's not like a virtual appliance or anything like this So when we show the logical topology those you know logical devices don't exist in any certain physical location They're just Computated on a flow-by-flow basis at the ingress of the network So how do we get packets from one place to another? It's just overlays. So we're using tunneling and encapsulation so Between two Compute hosts if you're sending VM east-west traffic, it's gonna set up a tunnel between two two boxes And then encapsulate that we can use the x-ion or GRE If the kernel supports other things like a geneve or or other types of encapsulation formats It's fairly easy, you know to add those into metonet as well And then if we're going outside to the external networks like the internet it would be going out these BGP gateways We show two but you can actually run as many as you like the metonet agents are They don't have any hard state. We're not using any namespaces to do any of the services So we don't have to worry about Storing state and and having to synchronize those Across each other we do synchronize some states between each other which I'll get into for stateful networking But in general you could run three or five gateway nodes And the return traffic is allowed to come back through any of those And since they're just talking to the network state database for the topology they can figure out what to do with that traffic so the network state database is Is kind of the you know very simple Heart of metonet, so we use a patchy zookeeper And the reason we use a patchy zookeeper is because it's you know, it's first of all It's a distributed database, so it's it's very easy to run And and we're not storing a lot of data in here. It's just very high-level network topology So this is very similar to the neutron model. Our model differs slightly, but we can map the neutron model into metonet Very easily and the other reason we use zookeeper is because it has a watcher Kind of public pub sub mechanism that we take advantage of And this allows the metonet agents to get topology updates. So if you were to Change the security or brewell or do a live migrate zookeeper knows which agents are actively processing Flows for that topology and it will send updates to those agents saying your topology has changed You need to redo your flows. So this is a really Essential feature of zookeeper that we use In the in the 5.0 version we did a lot of re-architecting to make metonet a lot more modular So we we actually have abstracted out zookeeper from From the rest of metonet so you could actually program into what we what we call zoom So we actually program proto buffs which are then converted into zookeeper So it's possible we could replace zookeeper with some other database like ecd in the future There's been some some talk of that And then how we integrate with open stack We actually have two ways to plug into open stack today The the main method right now is the monolithic plug-in to neutron So you'd install our metonet plug-in. This is written in python. The metonet plug-in is is in open stack project and essentially all it's doing is taking the neutron API calls and converting them into Metonet API API calls, which are fairly similar. Is it just a little bit different? It's writing that into zookeeper And then neutron is also writing into mysql So it's making two copies of that so you can interact with metonet via You know horizon horizons neutron functions, or you can use neutron CLI or you can use metonet CLI or or API if you the thing to note is if you make changes to the Neutrons so the metonet CLI those are not seen in neutron So if you create a router in in metonet through the metonet CLI it will not be seen in neutron But it will exist so that's you know most people are interacting via neutron itself That's the intent is user facing API is neutron But if you need to do some advanced functions Like setting up a dynamic routing which is not part of neutron You could use the metonet CLI or API to do that. So it's more admin admin functions So so we Have a concept of active active gateways, which we call sometimes we call L3 gateway or BGP gateway And this is very cool actually because it's exactly the same software that runs on the compute host That's running on the gateway nodes those gateway nodes are connecting to some upstream physical router or switch That speaks BGP It is using eBGP and it's essentially advertising the floating IP pool. That's that you've set up So this is handling the the external network All of those hosts are advertising the same pool. So they're all multi-homing and the so this allows The physical network for the inbound traffic to use equal cost multi-path to balance across all of those gateways And then we do the same thing going out. So you have a nice Balance of traffic there So just kind of a quick recap of the of the architecture as you can see, you know We have the compute hosts which are Identical to the gateway hosts except we're not running VMs in there you could but it's not not ideal But we have the network state database And then we have the the API and the plug-in. It's very very straightforward And then again zooming back up 10,000 feet you can now see the logical topology And you can see the the v ports on the logical topology are you know connected down to the physical infrastructure That's the only part where the logical topology touches the physical topology so in in media net you'll have virtual ports you can bind those ports to interfaces in Linux seen in any interface so it could be a physical nick or it could be a tap interface With our plug in an open stack Libvert is doing the vif plugging for you automatically and for the gateway nodes. Basically you just bind a port on the admin or provider router to the To a to a nick in in the in the x86 boxes acting as the layer 3 gateway So zooming in a little bit more to go through some of the How some of these features actually work? So we'll cover a few of these so distributed L2 switching so on the top of this you see the virtual Virtual topology or logical topology on the bottom left you see the physical topology And then the right is kind of explaining what's going on essentially all of the state is stored in zookeeper. We are storing The host IP to virtual Mac mapping so whenever you spin up a VM in open stack We are storing the virtual Mac of the v port And we're that's associated with the host and IP that's associated with that. This would be the tunnel IP So so when you set up meat on it When you add a physical compute host you're adding it to a tunnel zone And you're basically saying I want to put ETH one on on this tunnel zone So we know which physical nick to send the traffic to for tunneling So since this is stored in zookeeper when you are doing a Ping for example from VM 1 to VM 2 which is on the same neutron network. We don't actually have to do a physical ARP We already know that In the logical topology so we can just assume that this we I mean we already know where to go So we just kind of fake an ARP And we program the flow directly And then just send it across so there's actually no traffic sent off the box other than the The tunnel and the the encapsulated traffic itself and When you do make a change to that again They're subscribing to that change in zookeeper So if you were to do a live migrate of VM 2 to another hosts that mapping will update and the The host where VM 1 is living would would get that update so it would know to make a change to the To the flows so to invalidate all of the flows related to that Resimulate and add new flows Now if we want to connect this same neutron network or virtual switch, that's the same thing basically to a Physical hardware without going through some routing like like say you have a Database that can't be virtualized. So you want to connect that to a vlan We can use vtap to do this hardware vtap. So hardware layered to gateways. So there are some switches capable of Allowing us to program them and send them this host to virtual Mac IP mapping Those these are the Broadcom Trident to chipset switches basically and we're using OBS DB To to program those switches. So it's fairly standard now Most of the switch manufacturers have this capability and essentially What you can do is you can create a port on the virtual switch That's a that's a VXLine Gateway ports or vtap ports and you essentially are pointing that to a port or vlan on a hardware switch and Then meet on at the this is where the cluster API comes in so On this on the zookeeper nodes. We also are running some code which acts which is acting as the cluster API This is actually doing the OBS DB calls to the switch. So it will set that up. It will then also send the The physical host IP to Mac mapping to that switch. So when you're doing a A ping for example from VM one to some physical hardware that tunnel will go directly to the hardware vtap and not go through the the software gateways anymore, so So you can see basically with this distributed networking model We can do a lot of interesting stuff and as you have you can have multiple These multiple multiple layer two gateways spread across the physical network Going to multiple different Vlands all with the same neutron network So logically on the right-hand side, you see it's very simplistic on the right-hand side You know it could be you know spread across the data center all going over layer three And really it's very simple. So the the network admins don't have to do any changes to the physical Network as long as there's IP connectivity. So it's it's quite nice So going up the network stack to L3 routing We have as I said this concept of BGP routing Dynamic routing so we in in meat on it. We have a concept of this provider router. So this is not Something you'd find in neutron yet. There's some blueprints that's been around for a while Trying to add this capability but essentially Right above the the tenant routers in meat on it if you look in meat on that You'll see the tenant routers are connected to a provider router This is done. This is done automatically when you set the external network in in open stack So the provider router you have one of these typically although it's it's not built-in you can you can do a lot of different Models, but this is very standard and this provider router will have multiple up links usually these up links are Separate physical boxes so two or three physical boxes you bind the the v ports on this provider router to Physical mix across three three different boxes. These will then peer to some switch upstream physical switch using BGP And then you'll you'll have the connect connectivity out So this is very neat So we can also handle security groups and firewalls of service So in neutron, you know, you're defining the security groups, you know, very straightforward And we we actually have a slightly different model within meat on it. It's a little bit more low layer And we but we can convert the security groups automatically into what we call chains and rules. So this is very much mapped after kind of Security groups, sorry after IP tables in Linux. So It's not exactly the same, but it's very similar. So this allows us to have You know chains and in a chain you have rules and we can have jump rules to jump to other chains So it gives us a lot more control. We can do a lot more interesting things. We have access to all of the layer two through four Fields and we can do things like anti spoofing rules. We can do wild guards and ranges And you can apply these chains to any of the logical Topology so this could be ports or it could be on routers pre and post routing You can do it on on in and out filters on the bridges as well So it's very flexible where you put things and it can be dynamically updated or removed at any time The the very cool thing here is that since this is also distributed it means that the the the Gateway nodes and the compute hosts are able to do these firewall function. So Depending on where the traffic is coming in so if it's from the VM, it's handled locally So the security groups will be handled locally if it's coming from the external network It's going to be handled by the gateway nodes. So if you're if you have a VM that's Blocking traffic from a particular source that's attacking it That traffic will be blocked on the gateways themselves not on the private network I say it will never go through the private network as opposed to If you're using OBS plug-in right now That's going to go all the way to the compute hosts and then run on IP tables locally and dropped So so it still traverses the private network and affects all of the VMs on that hosts. So this reduces a lot of unnecessary traffic across the across the network So just to kind of see an example of what a security group would look like and meet on it You know, we have a chain and on this chain. We have some drop rules and accept rules And you can also have jump rules to other security groups. So this is kind of the idea It's not really meant for users to program it But if you have some advanced functionality that you want to create it's pretty flexible So you could you could add features to meet on it by doing this We can also do NAT Stateful and stateless which is which is also very cool. So It's basically the same idea here Meet on that agent since it's just simulating it's able to do essentially all of these functions and And the tricky part about NAT for for us was there is some state that has to be exchanged between the agents So in our first implementation, we were using Cassandra. So we were storing the stateful connection information in Cassandra so that the agents can both have those and the the main use case here is that The gateway nodes since we're running active gateway nodes They all need to have access to all of the stateful connection tracking information So the return flows, you know can can go across any of the gateways Going through Cassandra was kind of problematic because it requires a ton of calls off-box to Cassandra This can't be cash because it's updated frequently And it was pretty poor at performance. So recently we introduced Stateful connection transfer between the agents so now the agents are actually doing a kind of a peer-to-peer Communication between each other. So when you set up the gateway nodes, you add those gateway nodes to a port group And that port group is basically the the group that is getting sent the connection tracking information So as a packet will come out of the VM as a flow is set up it will send the connection tracking information out of band to the To the to the gateway nodes the port group and then we can also optionally send that Connection tracking information to Cassandra as a backup. So there are some edge cases where If you're not storing that in Cassandra, it will fail So those would be for example, if you restart a meet an agent, it's going to lose that state So it may not get the it may not know what to do with it So if you're running Cassandra as a backup if it doesn't have the state for that flow that's returning It will be able to go off-box and call that from Cassandra This is the other use cases for live migrate. So if live migrate happens, it's gonna have to do the same thing We still so even if you're using Cassandra, that's out of band and it so it doesn't actually affect the the traffic or performance So Getting into the community a little bit It has been about 11 months since we open sourced It's been growing steadily. We have a metrics website You can you can go to and and see some of the stats, but we have about 77 developers 4,800 commits 32,000 downloads from about a thousand a thousand IPs and we have 20 supporting companies We don't have a lot of external contributors though. We recently we have a large company who's Putting a full-time developers on internet. So our goal is to have not a single vendor project We really want this to be a community-led project So, you know, if any of you guys are interested in contributing definitely come talk to us We have a lot of infrastructure and Ways to get involved. So we have a slack chat channel That's you know, you can always hop in there and if you have any questions or want to get involved It's a great place to go to talk to us. The mailing lists are pretty active as well We also have a lot of code walkthroughs on YouTube. So if you go to the meat on that org There's a link Mito TV and you can find some of those walkthroughs as well And if they're if they're kind of outdated you can also ping us in slack And we were happy to hop on a hangout and walk through some code with you guys if you're interested in contributing And then we're using the similar infrastructure as open stacks. We use Garrett hub We're using Jenkins for CI It's really quite really quite similar So you can see some screenshots of all of these things and docs And and help so so we have a pretty good wiki. We're having weekly Meetings in slack the the meet and a plug-in is actually an IRC, but the other meetings are right now in slack You can find the logs of those on the wiki Our docs are fairly good as well And we are using JIRA for for issue tracking our docs. We actually introduced Japanese Language documentation just recently as well. So for the Japanese people in audience You know, please check it out. And if you find mistakes, you know, let us know and we're happy to have your contributions So if you want to get started, it's very Very straightforward. We have a couple ways. There's actually a lot of ways to get started But the easiest is just to run this quick start Essentially just run this on a fresh Ubuntu 14.04 box Probably eight gigs is better if you're gonna spin up VMs because this is gonna install OpenSack and metonet and configure it all for you. It takes about 20 to 30 minutes. I ran it last night It took about 20 minutes But I had a good connection. So the the stable one is this this top one But if you want to try the latest 5.0 version, which is in kind of release candidate stage Which I did last night it worked You can run that as well And I'll try to get into a demo as well. I think we have some time Little I'll maybe a little bit. So just to kind of sum up These things you can read that I'll kind of skip through because I want to show a little bit of metonet See if this actually works Okay, so when you run the quick start you can you can go to a horizon. It's gonna be set up for you The there were basically not setting a BGP gateway. We're just doing a static static routes for the you know It's just an all-in-one. So it's fairly straightforward But if you want to kind of see the internals of metonet, we have this mean at CLI So you can just run metonet CLI so you have to log in as root or you or you have to you know set up a definition file and then you can basically Start Checking it out and it has tab complete So it's pretty nice, but you can say list routers so you can see all the routers. I can do things like List the routes of the routers It's kind of hard one-handed. It's a router router zero and also I'm on shoddy pretty Bad Wi-Fi, but I'm using mosh. So it should be okay So I can list the routes like this. You can also see the This security groups. So if I do just a list chains Or list chain you can see all the chains. So you can see in the It's a bunch of you you IDs, but we try to make we in maps to to open sack you ID So it's easier. So OS SG and then this you you ID This is this is the same you a you ID that's in neutron So you can easily find the security group that maps to a chain And rules in metonet so you can see the egress in the outbound We also have the anti spoofing rules that we put on every port So you can then do a For example chain Chain see what's an interesting one chain three list Rules it's low oops So you can see the rules. It's very low-level In here as well. You can also obviously you can create rules. You can modify them delete them and See if the tab complete will show us anything good. See there's all there's a bunch of stuff You can do there you can view the tunnel zones. We also in 5.0 have some tech previews of L2 service chaining L2 insertion mirroring We also have some some troubleshooting tools like a trace virtual trace route So that we're actually have a demo at 1245 to go through that. See I'll share that information as well Is there any questions? Yeah, so the question was if you want to use hardware offloading with the for for VXLan for example The excellent yeah, so we do support VXLan offloading So the if the nick supports VXL offloading that works fine You don't see a huge performance gain for 10 gigabit networking, but for 40 It's it's recommended to do that. Yeah, we have customers running that There shouldn't be we're running out the We're running on the standard tunnel ports So it should it should work, but it depends on the nick hardware So it depends on if the nick hardware assumes a certain port We may have to change the port or change the configuration of the the driver But it just depends We've done testing on emulax and melanox cards so far and they seem to work pretty well. Yeah For For for like a production we recommend setting up three a minimum But if you like this this is setting up one in the all-in one, it's just setting up one It's not highly available, but it can run Yeah, so if you set up three nodes you can handle You can handle one failure if one of those fails you can still read but you can't write So if you want two failures, you'd set up five. Yeah Any other questions? Yeah, sorry So so for no net use cases we can support it, but it requires some manual config so basically You would just you need to add the routes So if you want to just do static static routing you can add the routes in the media topology that could be scripted And we have some customers and users who do that So that whenever they create a new tenant they automatically create the tenant a router and network and add the routes So it's just routed through. Yeah, it's pretty straightforward though Yeah, I think the latest one nine and further you can actually stop running Cassandra and it's it will still work Yeah, but those edge cases will not work where I mentioned before or live migrate or restarting an agent for the for the nap Stateful nap. Yeah, but it works this this 50 quick start doesn't actually install Cassandra Yeah, so it's a little bit lighter weight Any other questions? Okay, so stop by our booth if you have any questions or want to get involved and if you want to see some cool troubleshooting tools At the marketplace theater at 1245 today. We have a demo of some of that stuff. So I recommend checking that out as well Thanks a lot everyone