 All righty, good afternoon. Hope everybody had an enjoyable lunch. So the stock is inside the architecture of Neutron. I apologize for the AV display problems we had. My name is Mark McClain. I am a core team member of the Neutron team. I was also the PTL for the Havana and the Grizzly Cycles and serve on the technical committee. I work as an architect at Yahoo. So one of the things when we start talking about the architecture of Neutron, we first of all need to take a look at why Neutron was created. One of the things is when you compare it to Nova Network, because Nova Network gives you a fairly static set of topologies which you can create. So one of the motivations for creating Neutron was to enable the ability to create rich topologies, including multi-tier networks. Also recreate real-world topologies so that you could have overlapping IP addresses within the deployment. Also, to make Neutron more extensible, when you compare to Nova Network, Nova Network has a very static set of deployment options and it's really also to make it technology-agnostic too. So Nova Network allows you to use VLANs, but you can't use VXLAN, overlay networks, GERI tunnels, similar stuff. And lastly, support advanced services. Within Nova, the only advanced service they really supported was CloudPipe, which allowed you to VPN into your private tenant. But in Neutron, what we want to do is we want to add support for load balancing VPN and firewalls. So when we take a look at why we created Neutron, so those are kind of the motivations. We need to take a look also a couple of basics just to kind of level set. So when we start thinking about the OpenStack API, and people will see this over and over, is when you take a look at the user, what does the user see when they use OpenStack? They see a compute API. They see a network API. They see a storage API. And these APIs are agnostic behind what's the actual back end. And the user is unaware that maybe the compute API is backed by KVM, the ML2 plugin, or SEF, if they're storage. When we start talking in terms of terminology and abstractions that we use within the Neutron community, within Nova, you'll see the orange boxes at the top are VMs. Each of those VMs has a virtual interface for VIF. That's how the instance is plugged in. Within the network side on Neutron, we have virtual ports. Virtual ports is the same as you would find on a real world switch. It's what you connect the VIF in basically if you were to have a virtual cable. The L2 network is a single-shared broadcast domain. And on top of that, we have virtual subnets, which is an L3 construct for address ranges. So now that we've kind of level set, let's talk really about the architecture. We had a couple design goals when we set out to start to make Neutron. We wanted a unified API. We wanted something that had a small core. The main reason is if you develop a very lengthy core, the consensus, or very large core, the consensus process can take time. Also, not all the networking vendors support both open source and proprietary support the same feature set. So if you have a small core, what that does is ensures that the users can reliably expect certain resources to exist. Also makes it easier for the vendors to contribute and develop their plugins. Speaking of which, we wanted a pluggable open architecture. We didn't want to make design choices that limited vendors and what they could deploy, what they could design, how the community can innovate. And so the innovation is really that vehicle for doing that is being extensible and through extensions. We'll talk a little bit about that. Now, I'd be remiss if I was giving an open stack talk and did not show this diagram. I think somewhere's getting a royalty for this thing. But if we really zoom in on the right-hand side, you'll see the Neutron server and the components within that. But really, if we start even more simpler, let's just start the simple case with the Neutron server. Well, typically what you'll find in the basic deployments to the Neutron server will be backed by some kind of relational database. You'll find this in the open source versions. You'll also find this in many of the same versions and many of the proprietary versions as well. Along with that, that Neutron server talks to a message queue. The message queue is used for exchanging messages with other Neutron agents, not for exchanging messages say with Nova or any other components of open stack. And that message queue helps you talk to the agents. Typically in a lot of deployments, you will find an L2 agent. This L2 agent is responsible for wiring up... Oops, one ahead, sorry. The L2 agent is responsible for wiring up the ports and the devices and connecting them into the shared broadcast domain. Another agent you will commonly see is the DHCP agent. This allows you to configure the host. Now, if you're using a deployment where, say, you're using config drive, this is an option where you may not have a DHCP agent, but in some deployments you may want to use both depending on which kinds of images you've built. Lastly, we have the L3 agent. The L3 agent is responsible for giving connectivity outside of the isolated tenant deployment into the data center or actually into the public internet. Now, if you notice with the basic abstractions, I'm only showing you one L2 agent, one L3 agent, one DHCP agent, but when we're running these things in the real world, you're really going to have multiple copies of these things. You're gonna be running multiple L2 agents. Typically the L2 agents are going to reside on the hypervisor. And then lastly, I touched a little bit on it, is advanced services. Many of the advanced services agents have special agents, whether you're doing load balancing or VPN or firewall. So when we take a look and let's dive in a little bit, because that box that says Neutron Server, it's like, okay, great, that does a lot of stuff, but what exactly is it? So when we dive in, really the Neutron Server is really three parts. It's a REST API service, an RPC service, and a plugin. So we take a look at those. The REST RPC service is an HTTP Python whiskey application. That's standard whiskey layer. The server that it's written in right now typically runs on the Ventlet server, but being a whiskey application, if you want to, when deploying it, you can actually take that application out and run it within a different container. Being a Ventlet base, your container does have to work with a Ventlet. And the reason we use a Ventlet is I'll touch a little bit on it, is with the RPC service. The customary port that you'll find this REST service running on is 96.96. And what the REST service does is it exposes logical resources. Now, I said earlier the core of Neutron is very small. So we have networks, subnets, and ports. That's it. That is the core V2 API for Neutron. And then lastly, the REST service is responsible for serialization and deserialization, request, response, handling. Handling all the aspects of the HTTP protocol. Typically what you'll find in a lot of deployments is the preferred serialization is JSON. The secondary one is XML. And one of the challenges you will find is not all the extensions play well with XML. So typically from the team, we generally prefer people to use JSON serialization where possible. So if we dive in a little bit into the RPC service. So the RPC service is basically speaks of our AMQP bus using Rabbit, QPID, whatever your preferred server is, using Oslo messaging modules. Those modules are based upon a ventlet which allows you to do concurrency within the process. One of the other benefits of using RPC service is it enables bidirectional agent communication. It also enables the agents to talk to each other without actually knowing where they're located within the data center. You don't have to do any discovery or sharing of IP addresses. The only thing you need to do when configuring is ensuring that the RPC service is ensured that the RPC service is connected to the messaging bus. The interesting thing about the RPC service is it is optional. We remember we wanted a pluggable, open, extensible architecture. So some of the different plugin implementations may not have an RPC service because they may not need agents. They may not need to connect to them. But it is an optional feature that we do support. And then lastly, digging into the bottom layer is the plugin. So like all Python, like all OpenStack projects, it's written in Python. The key thing for a plugin is you can only have one active plugin at a time. It's kind of like if you have a car, you can only have one engine, you know, choose wisely. So we take a look at that and then the active plugin implements it minimum the V2 API calls of networking, subnets and ports. And optionally, some of these plugins can do a number of different things. They can have optional database access. They can have optional extension support. And typically when you're configuring it, you're gonna take a look at configuring a core plugin. And so in this case, the open source reference implementation we ship is the ML2 plugin. The ML2 plugin, so on the part of the right, it basically shows you a snippet of a config file. And the ML2 plugin is really a subclass of something called NeutronDB plugin V2. What that is is that subclass provides a lot of the framework for doing a database manipulation within Neutron. It enables plugin implementers to basically get logical resource saving, serialization into the database, caching, and some of those layers for free. But ultimately all plugins are a subclass. The parent is Neutron plugin base V2. And if you look on the right, you'll notice if you know anything about Python is it's all abstract. There is actually no implementation in the actual parent class. And that's by design. One of the things we wanted to do when designing Neutron is to ensure that implementers had their options of design patterns they wanted to follow. So for instance you could use the plugin base. If you didn't want to use the built-in database support and you wanted to directly proxy into say another SDN controller, you could write a very thin proxy that stored nothing was completely stateless and immediately just ship the calls over and retranslate them on the way back. So we have that plugin base but most of the operations you'll find in there is we only require 12 operations in a Neutron plugin. And they're basically create, read, update and delete operations for ports, subnets and networks. So it's very repetitive code. But in and of itself for some private deployments that's interesting, that gives you exactly what you need for virtualization but for other deployments you're running maybe you need tenant isolation, maybe you need higher level constructs. We support plugin extensions. The way to think about a plugin extension is it adds a logical resource to the API. So that API could be like a router or you could add security groups which is a common feature you would find in Nova. Interesting enough not all vendors support security groups because in some cases you have deployments where you want them and some deployments where you don't. Additionally the extensions are discovered at server startup. As a user of a cloud if you want to you can make a REST call on v2.0 slash extensions in the server will send you back a list of all the active extensions in the server. Some of the common extensions you'll see are binding DHCP L3 which is routing provider networks which enables you to bridge existing networks in your data center into your cloud deployment. Quota support, security groups. Other interesting extensions are allowed addresses so if you want to restrict which IP addresses a MAC address may use for speaking you can do that extra routes and metering. There's a couple different plug-in paradigms so we can talk about when we talk about a plug-in we can talk about a monolithic plug-in. So when you think that what it is is it implements the entire plug-in interface and typically what you'll see within Neutron if you dig into the code is you'll see they really fall into two types which is a proxy plug-in which proxies onto another service which actually manages the orchestration of the data plan or direct control. The direct control one that you will find within Neutron is really just one that we use for the reference architecture to either manipulate the Linux bridges or to manipulate the open V-switch. But one of the things you can also do is we decided that implementing that big plug-in you repeat a lot of calls. You're gonna repeat a lot of code. Maybe you need code for generating IP address managed for generating an IP address for generating a MAC address and really all you want to do is provide connectivity to different segments. So one of the things the team worked on over the past 12 months was rolling out Modular Layer 2. So what we've now taken is that plug-in interface and added drivers to it. We have both, there's two kinds of drivers, mechanism and type drivers. Types represent segments. So you have like a type driver for say a flat network where you want connectivity directly into an existing network. Maybe you have a type driver for VXLAN or any kind of other tunneling. And then there's a mechanism driver which actually handles the implementation depending on which vendor or which solution you've chosen on the back end. So if we were to take a look at say a simple or a standard L2 agent, the L2 agents typically are gonna run on a hypervisor but this is not always the case. There are solutions where the L2 agent doesn't run on a hypervisor or if you're using a central SDN controller, the controller speaks directly to the V-switch on the hypervisor. Depending on how you argue in your theory of logic and abstraction, some would argue that is still agent based but I'm not gonna go into that. Typically the agents will communicate via server with the server via RPC. Again, we want a bi-directional communication. It gives you a well-known channel to talk through. What the agent will typically do is watch and notify the controller when there's devices that are added or removed from the hypervisor host. When that device is detected, it contacts the central server and then it wires a new device ensuring that it's plugged into a proper network segment and if you're applying security groups that those rules are applied properly. So if we dive into the OVS agent, OVS is Open V-switch. It's an open source virtual switch that you can find on almost any distro everywhere. One of the things that people ask is what does it actually do? How do we get isolation? And what the switch does and what the agent does is it takes a look at the different tenets and ensures in networks and ensures that the ports within the switch are logically grouped together but those mappings within the switch don't always translate to mappings external. Also, if you're running VLANs or overlays, it's going to properly tunnel the traffic and plug in those tunnels into the tenant's group. As part of the processing, look like I said, it's going to detect and search for devices that have been added and removed. One of the things the neutron team did this past six months was working on making this agent more robust and work faster. So that's a little bit on the L2 agent and before it proceeds to the L3 agents, one of the technologies we decided to use and one of the problem spaces we had is overlapping IP addresses. The Linux IP stack is monolithic and so you cannot recycle IP addresses so we had to use Linux network namespaces. And basically, it's an isolated copy of the network stack. Each namespace gets its own private loop back. The scope is limited to the namespace. So if I wanted to reuse 10.0.0.2 in namespace A and then use the same address in 10.0.0.2 and B, it's still going to work. The interesting thing though is if you take a look at it, the namespaces in less eth0 in A or B are actually connected to a physical interface. Those namespaces do not have connectivity to the outside world. So typically what you'll see in virtual deployments is if you need connectivity, you either have to connect the device into an internal bridge on the box. Typically VIFs are used and just think about it as a virtual patch cable if you're not used to it. But you do need explicit configuration to connect those. Some of the L3 agents in Neutron will take care of this for you. If you're interested in you haven't seen namespaces before, it's the IP route to Utils. It's found in most of the modern distributions. If you're using a few of the older ones, the kernel support has been there for a long time but the user land utilities have not. The other benefit of using namespaces is you can spawn processes within a namespace and their view of the network stack is only the view of the namespace. So if you wanted to run multiple DHCP servers that were isolated, you could do that within a namespace. You can also run multiple forwarding. So when we talk about L3 agents, typically an L3 agent's gonna run on a network node. In our reference architecture, a network node is all the network traffic goes through it. I know some of you are immediately thinking, hey, this is a single point of failure. This is a choke point in my network. It doesn't scale. Yes, you're right if you run only one of them. But for some people, your traffic needs are enough that you can run one and be okay. Some people will run more than one. And it uses namespaces to provide isolation for the tenant so that traffic from 10 to A and 10 to B do not mix. And also the network agent, the metadata agent is enabled. So if you're not using CloudNet, you can still get access to Nova's metadata services. So like I said, you can run more than one. In this case, now we're running two. But you can space those out and you can have support for scheduling those out. And if you notice, each of the hypervisors talks. And so how is the L3 agent implemented? What the L3 agent does in the reference architecture is it manages the collection of namespaces. They each have isolated IP stacks. We enable forwarding. So it is a simple forwarding device. But it is static-based routing. So you can put static routes in there, but currently the current implementation does not use dynamic routing. So if you're interested in looking for something that uses, say, BGP or OSPF, that support's not a neutron right now. That's something that lives at a higher level there. But you can also write tools to push static routes in if you wanted to use, say, a BGP speaker external. As well as the metadata proxy. So if you look over the two routing instances for tenant A and B, what you'll notice is typically in a neutron deployment, if you were to dive into the namespace, as you will see, a QG prefix, which is the gateway prefix. Typically that is the gateway device. It typically has an external IP address in your deployment and is connected either in BRX or, alternatively, you can also connect in with the provider network. If you're wondering where the queue comes from, that's really a historical artifact that neutron began its life as a project called Quantum. So for compatibility reasons, we've kept Q around and it shows up in random places. And some of us have been involved in the project for a while every now and then, still say Quantum. So with those, you have the gateway, which is QG, and then you have an internal interface which faces the tenant network, which is QR. So the QR is what typically what you would put in the gateway setting of any one of your instances. If you're using floating IPs, what the L3 agent will do is it will put the external IP on QG2 and then write NAT rules to do bidirectional NAT into the floating IP to route traffic to the proper host. If you're not using floating IP and you wanna have an instance, what we provide out of the box is source NAT so that it's typical just like you'd find in your home router at home. You plug it into your cable modem and everything gets automatically NATed. So with that traffic and affording, the one thing to take note of is you'll notice it says that we only enable IPv4. This is one of the shortcomings that we're currently aware of. We have a group of people working on adding support for IPv6. The other interesting thing about how the L3 agent works is you remember it works via an RPC interface. So just because the reference implementation is done this way, there are other folks in the community who have implemented other alternative L3 agents that implement the L3 RPC interface. So from the viewpoint of Neutron, it can't tell the difference that it's not talking to one of its own. One of the problems we always have in the cloud is how do we configure things. One of the options is Config Drive. Some folks also want more dynamic configuration. And one of those is DATP. It's been around forever. It's a well-known understood protocol. And the way we do it is the RPC-based notifications drive a handler which manages DNS mask. And we typically spin up a DNS mask instance for each isolated tenant network segment. Again, typically DHCP will run on the network node. And so we get the isolation support. And if you wanna have multiple copies for HA, you can spin up multiple copies of DHCP because naturally the DHCP protocol is automatically HA compliant because a proper DHCP client will take, will accept several offers and select the best one. So from an HA perspective, we don't have to worry about active passive when you're monitoring your operations. You just have to make sure that at least your two DHCP nodes that are running the agent, at least one of the two is up. The other side is how do we get public keys and how do we get keys into the instance? And one of the options is using metadata. It's similar to what you would find in Amazon. But one of the problems you have is when you have an isolated network segment, that segment can no longer talk to the Nova Metadata Service. Also one of the other things is the Nova Metadata Service is originally, as it was created, did not support overlapping IPs. So if you had two tenants and they said, hey, please give me another metadata request and I'm 1001, Nova did not know which one you were. So one of the things that the Neutron team did as we partnered with the Nova team made a few tweaks to Nova and now that we can support overlapping IPs within Nova Metadata instances. Nova, the metadata proxy, if you have a routed network, is usually typically embedded into the router namespace. In non-routed networks, what we do is we use a static route to redirect traffic into the DHCP namespace where we're running a copy of the instance. The only downside with doing that is your DHCP client actually has to request static routes because we can't send you options in which you don't request. So if we were to take a look at how configuration and how the metadata proxy works, so typically config drive is, Eric Claudinette's going to just call curl, that should be 169, not 254, 169, 254, slash open stack latest meta JSON. And so as you see with the red dot, the VM has made that call. That call is then passed to the metadata proxy. One of the first things the metadata proxy does is adds a little bit of context to the HCP request. It also does a little bit of sanitization of the HCP request to ensure that there's no improper headers to ensure that the X forwarded for header is correct, that the router ID is correct because putting the router ID is important because the next step is we have to get out of our network namespace. We're stuck in a jail, how do we get out of there? Well, one of the cool and clever tricks that we did as far as architecture goes is the Unix domain socket actually allows you to talk to processes on the same machine so you can tunnel out of a namespace and talk to another process that's local to the same machine. So we use a Unix domain socket with HTTP still and it talks to a Nova metadata agent. The first thing the Nova metadata agent does is it takes the router ID, it takes the IP address and goes and looks up the instance ID from the database in Neutron. When the request hits the metadata agent network, this is now on your management network. This is not running on tenant traffic so that you can keep and maintain a secure management network. And once that request is looked up and we've added information to the header, the request then goes over the Nova metadata service. The Nova metadata service, when it's configured to answer multi-tenant request, we'll look for the instance ID. Also, it's optionally you have the ability to add a shared sign token so that both ends know that you actually are able to make request to it. So let's boot a VM and take a little bit, take a look at what's going into architecture. So when we call Nova boot, what Nova does is if you just, up here I called it, I didn't provide any networks, created something called My Instance. And what Nova's going to do is it has a plug-in where you can configure to call Neutron. And Nova's going to schedule the instance and then when it goes to spin up the instance, it's going to say, hey, which networks do I need to connect to? Nova's going to go ask Neutron, do you have one network available? And if the answer's yes, then Nova will automatically create a port for you on that network. If you, alternatively, you can create ports in Neutron and pass them at boot time as well if you want to be a little bit more specific. Behind the scenes, what goes on when we go to create the port is Nova's creating it. So if you see at the top, I know it's kind of hard to see, but it's a dump of running Neutron port lists and you'll see that the port's been created, a fixed IP has been allocated and a MAC address has been assigned. And then also simultaneously when the port request is entered back to Nova, Neutron underneath the hood sends out an RPC message to the DHCP server and creates a DHCP instance if one doesn't already exist. So in the second box, you'll see where it says pseudo IP net NS list. That's where we're creating two network namespaces. On this particular network node, I have both my L3 agent and my DHCP agent running together. And what you'll notice is we have a DHCP namespace and we have a router namespace. So the DHCP agent will create it. And then one of the things it does next is it will set up, add the host entries, which you'll see is the DNS mask and fig file for everybody's ever played around with DNS mask. And then also at the same time, the tap device is created by Nova network. So we see where it says tap. And then what's going to happen is as new and ice houses, Nova will now wait for this tap device to actually be wired up. The good thing about this now is this coordination again, this is where the Nova teams and the neutron teams got together, said how can we make this experience better for booting? And so it's a callback procedure. So Nova will actually defer booting the instance until neutron has informed it that the device has been wired appropriately and been set up. So the port's been wired. As you'll see, we have the tap device. Oops. And then we get a successful boot. So that's kind of the process for booting. The couple of other agents which you'll see from time to time and they get implemented a couple of different ways. Our load balancers of service. Typically what it is is this is a service plugin. So if you remember back where at the beginning where I said we had a rest RPC, we had a rest layer, RPC layer and a plugin, service plugins are actually loaded alongside of the core plugin. For the load balancing as a service, it's a driver based plugin. Benefit of that is we have an open source implementation and several vendors have implemented proprietary solutions which use that leverage that driver. In some cases the proprietary solutions will talk directly to another manager that manages load balancing instances. In other cases it will talk to an agent with a driver. That agent is very similar to how the L2 agent works. The open source implementation requires name spaces again so we can provide isolation. And like I said, others will interact with other systems. VPN as a service works a little bit differently. It's a service based plugin. Again, we have drivers, the agent actually co-resides with the L3 agent. So if you enable VPN, we'll create a router, we'll create a metadata proxy for that instance and add the VPN driver which typically configures open swan. So that way you can have full routing within your tenants network. And then firewall as a service is an edge firewall. This is an experimental service. So it's similar like edge, it's basically you're creating rules at the edge of your firewall. Again, service based plugin, agents with drivers communicates. And if you notice it's similar in construct to the VPN as a service. Now there are a couple differences to be aware of is that we wanted an open architecture. So there are going to be different design decisions depending on where you're using a proprietary solution or open source solution or even differences amongst some of the vendors will have multiple items in their product line with different design decisions. Some of those will sync immediately with the backend. Some of those will proxy through the database. Some of them, like I said, will have L2 agents. Some will not have L2 agents. Some will implement everything into one big massive agent. Again, different design decisions. And a lot of those decisions also play into what your choice as an operator has, but that's a good thing. Again, the extensions are optional. So just because if it's not part of the core API you can't always expect it to be there. So in summary, what we were looking for is we wanted a unified API. We wanted that API to be easily implementable which resulted in a small core. But we also wanted a pluggable open architecture which is why we are a plugin architecture. We also wanted to even make it easier for those who didn't want to implement a full plugin. So we added drivers. The good thing about Neutron is it has multiple vendor support. So these are currently right now, we have 19 different vendors within the tree. There's also vendors who ship code outside of Neutron as well and I expect we'll add a lot more during the Juno timeframe. And it's extensible. So if you wanna go create a new resource, maybe you wanna create IDS as a service, you can go create an IDS extension. You can, and extensions don't have to be part of the Neutron tree immediately. So you can create it, test it, build consensus community about what it looks like and then bring it back to the Neutron community and say, how do we standardize this? Or in some cases what works out a lot better is you start working with consensus to build consensus within the community. So if you're looking for more information in terms of docs and how you can docs and explains a little bit more of the items in detail, it's the cloud administrator guideline guide as well as the network 2.0 API guide. That will include the core resources, the API guide also includes the extensions and typically JSON samples. So, questions? Yes, can you go to the mic? Hey Ma. Yes. So you said Neutron is small core, right? So it is basically port, networks and subnet. But even for this small core, Horizon does not have all the, it doesn't support all of these resources in the sense that I cannot today go create V port from Horizon and then when I'm instantiating a VM, I cannot attach that VM to the particular port from Horizon. So where do you see Horizon supporting all of these things? Part of that would be a little bit of coordination between the Horizon and Neutron teams to make that supportable. Right now Horizon enables you to choose which network unboot, but ports is something we should make, currently it's best available through the CLI, which I know for some folks is a little unwieldy. Okay. So yeah, in a multi-tenant environment, how do you really troubleshoot because you've got all different DSCP services, QRouter services running their own namespace, but they are essentially associated with a certain tenant. So how do you correlate and troubleshoot the scenarios? So the question is, how do you troubleshoot with the different namespaces? One of the ways currently right now is you have to go find the router namespace. We use the same UID as the actual object, so that way you can see where it's scheduled, you look at the host, and you can actually, as an operator, go in and log in into the namespace to make sure that DCP is functioning, and you also have all the same tools you can do traffic capturing. So, and we have unfortunately hit our time limit. So I apologize for the late start with the AV troubles, but thank you.