 Let us part some bugs here. This must be very networking folks out in the room, okay, I am sorry, okay, that is good one, yes, that is one, there are few more, can you repeat that for me, yes, that is wrong, correct. So, what was your point here, right, it is a broadcast address, so you cannot use a broadcast address for a host and here you have a 257, okay. So, here are the answers, so you have a subnet, you got some duplicates and the other one which is pointed out correctly is there is an incorrect subnet up here and you cannot use the network address and you cannot use the broadcast address, so those are some of the, well, available on the OSSNA website also, if you want the slides in the original format, this is the link, so here is my, here is the agenda, I will start with some introduction, how the container networking evolved over the time, what is the history of CNI and go over the CNI, how it works and what is the, and then let us walk through the CNI spec, what are the options available from the CNI repositories, we have an SDK and what are the CNI, reference CNI plugins available and then let us try to build a simple CNI runtime and a CNI plugin and if you have any questions, you do not have to wait till the end, please feel free to interrupt. So, with that, let me introduce myself, so I am Murali Paluru, I have been working with Rancher for the past one and a half year as a principal software engineer, I call myself a plumber because I take care of building the pipes for the containers in our product, so just the way Mario, you know, goes inside one end of the pipe and then he shows up magically on the other end of the pipe, I make sure the data sent by one container goes through the pipe safely and reaches the exact destination where it is supposed to reach, so that is what I do at Rancher. What I have done in the past few months is I built CNI plugins at Rancher, I have built IP cycle overlay plugin, I have built VX LAN plugin, I have implemented network policies and before joining Rancher, I was at Gigamon, Cisco and I worked at a fail startup. If you want to reach out to me, these are some of my contact information, so how did the container networking evolve? So, long time ago, a computer was invented and the computers were able to do that, we had virtual machines and they were able to talk to each other and later the awesome containers were created and we have now containers running on a host, so when Docker first came up with a bunch of easy to use tools which are made running containers, very easy and it came up with a lot of other good things like registry, images, blah, blah, blah, you all know. So here, I wanted to talk about the networking works on a single host, so when Docker starts, it starts up a Docker zero bridge, it assigns a cedar for all the containers on this host, the default is 172.1700 slash 16, when you start, you have weeds plugged into the bridge and the other end of the weed is plugged into the container and a nice IP address is assigned to the container. So this way, the containers can talk to each other on a single host. So, you know, here, one important question I want to highlight so that we can get back to this question is, who is responsible for setting up networking here? So when a container is started, Docker daemon is responsible here to set up the networking for the containers. So it's responsible for figuring out what is the IP address for this container and, you know, programming it correctly and taking care of all the other stuff. So if I have to explain this Docker networking in a little easier way, what we can do is we can give acronyms to the IP addresses. That's what humans are good at, trying to, you know, not work with numbers. So let's say we call Amy, Bob, or Deb, for the different container IPs. So here, you know, Bob can talk to Deb. He can just say, hey, Deb, I have some data for you. I have some message for you. Here you go. You can just pass it on. Now, what happens if we add more servers or, you know, you can throw in more nodes into the picture? So the default Docker settings when you start up is it starts with the same cedar and all the containers here, they get the same, they get the IP addresses from the same subnet. So what's happening? So you have Bob, you have Deb on both these hosts. So if you have more hosts, you know, there are a lot more Bob's. So if Bob wants to talk to Deb, what happens? You know, it's a complete chaos. It's total confusion, which, you know, Deb I'm talking to. So there's a problem. You know, when you have a lot of containers and a lot of servers in the picture, you have a big cluster, how can containers talk? So we have a problem. A lot of people, you know, jumped in and tried to solve this problem. So one of the solutions a lot of people used was they used something called as publish ports. So with that, what happened? You use the IP address of the server of the node and you assign one port of it and you map it to the service running into your container. So this way, the containers on a different host can talk to the other container using the host IP address. You know, to make it a little easier, you know, the first host, let's call it, you know, building alpha, the second host call it building bravo. So now if Bob wants to talk to somebody else, he will use a new paradigm now. He'll say, hey, calling from the first floor of building one. And I want to talk to the person of in the second floor of the building to something like that. So now containers able to talk. So this is one of the solutions. This is another solution which Rancher has used. So what we did was, you know, we gave a secondary IP address for the containers. So we picked a unique subnet for the entire cluster. So let Docker do whatever it's doing currently. But what we will do is, we pick one subnet and we use it to assign IP addresses for all the containers across the different hosts. So that way, you have unique names. So now when containers want to talk to other containers on a different host, instead of using the Docker names, they use the new names that they have. So it's like, you have Ram, I want to talk to Mira, you can use the new names. So that is how we solve the problems. And Kubernetes took a different route. So what they did is, hey, I don't want to start with the default Docker settings. Let me have my own custom settings. So what Kubernetes does is it starts Docker, or in fact, here, it's not exactly Docker, it uses a different bridge. So it uses a different bridge and it assigns a unique subnet for every host. So it's per host subnet. So that way, when containers start, by default, they have a unique IP address. So, you know, you can come up with your own acronyms and the containers have unique names and they can talk to each other. So like this, a lot of companies, a lot of creative people jumped in and tried to solve problems. And they all wanted to provide this solution to everybody else. So, you know, there's a company called Vee. They all came up with their own networking plugins. And there are other SDN solutions also. So they all wanted to plug into network. So how do I do that? So, you know, just the way you have different standards for your, you know, electrical outlets, there were like a bunch of standards available in the industry. So one of the standard was introduced by Docker. It's called CNN. So the way it, you know, was released was, you know, Docker was trying to solve this problem. They came across this company, I believe, Socket Plane, and they had a networking solution. And Docker liked them and they acquired them and integrated into their product. So that's how the Docker guys came up with the CNN specification. But then there was another company called CoroS. They were building a different container runtime called Rocket. And those guys, they saw there's a need to specify a specification so that other runtimes can also work. So they introduced CNI as part of their app C efforts. So these are a high level comparison between the two specifications. So you have all the similar stuff working in both of these plugins. So all the companies they have implemented, or they have, you know, made sure their plugins work with both these interfaces, like the Calico plugin works with CNM as well as CNI, we works with it Cisco products work and all the other companies products they work with both the specifications. The other one of the important disadvantages or one of the subtle disadvantages of using CNI is, you know, a lot of people use Docker inspect and the network settings section in the JSON output, you won't be able to find out what is a container IP address. So that's one of the problems of using CNI. So some time when Kubernetes wanted to adopt one of the standards, they were they looked at CNM, and then they looked at CNI. So after doing a lot of research, what they have realized is CNM doesn't work for their model. And they have blogged about their reasons for choosing CNI. And it's a very popular blog. So a couple of reasons that I can point out why Kubernetes went with CNI, but not CNM is CNM needs a lib, a kv store, a key value store. And they Docker had a certain way of doing things. And Kubernetes already was using HCD or console as their key value store backend. And if they wanted to run CNM plugins, they had to either, you know, write a wrapper on top of the existing HCD store, or you have to run a different instance of HCD, which Kubernetes didn't like. So they went ahead with CNI because they felt CNI was much more simple, and it fit their use case much better. So that's, that's how CNI got more popular. And then a lot of other networking projects started adopting CNI. And finally, in May of this year, CNCF cloud native computing foundation adopted the CNI project. Let's go through the high level overview of CNI. So what does the network need to do? What are its responsibilities? Now, the first thing is, it has to take care of the plugin. It has to take care of setting up the network for the container. That's the first thing. And to do that, the plugin needs certain information. And most of the time, you know, this information is not available in the plugin itself. It needs help from somebody else. So it talks to IPAM plugin to figure out what IP needs to be given to the container, what are the routes, what is the DNS configuration that I need to do. All the extra information that's needed is provided by IPAM. So these are the main responsibilities of a CNI network plugin. So let's get started. So to run a container, usually you pick a runtime. You know, it could be Docker, it could be Rocket or something else. So the first step in using CNI is you need to set up the network configuration. You know, you have to define what network, what kind of plugins you want to use. So it's a simple JSON file. So you set up the CNI config file. That's the first step you do. The next step is, you know, you start a container. One important point I wanted to bring up here is, if you are using Docker as a runtime, if, you know, a container is, or if the plugin is a CNI plugin, the container is started with dash dash net equal to none. And that's the reason why the information in the Docker inspect doesn't show the network information. The reason for that is, earlier in one of the slides, we saw that when a container starts in Docker, Docker is responsible for setting up the networking. But you want a CNI plugin here to take care of networking. So you have to tell Docker not to interfere with you or not to interfere with your CNI plugin. For that to happen, we use this flag to make sure Docker is responsible for just starting the container and it doesn't set up any networking. And then the CNI plugin can take over and it can do its magic. So the next thing the runtime does after starting the container is it reads the CNI network configuration from the disk and figures out what CNI plugins it needs to invoke. And it passes the CNI network configuration along with some other information and it invokes the network plugin. So one important point that I want to bring up here is, in the Docker's world, in the CNM world, a plugin usually runs as a demon. And here in CNI world, here the CNI plugin runs as a binary. So that means every time a container starts, and the plugin is invoked. So it's not a long running demon. You cannot store any state in your runtime. So you have to depend on some external store or external database to store your runtime information for your CNI plugin. So after the CNI plugin is invoked, what it does is this is an optional step. Some of the network plugins can do their job on their own, but sometimes they cannot. They need help from an IPAM plugin. So what they do is they pick out the IPAM config information from the network configuration and pass it on to the CNI IPAM. So they do not pass on the whole network config, just the relevant information that's needed for the IPAM. And then this is also a binary invocation. There is no demon running here. And then CNI IPAM plugin figures out the information. It can talk to some external server or it can talk to a local file or it can talk to a DB and figure out the IP and all the other information that's needed to start the networks, start the networking for the container. And it returns the information back to the network plugin. And after that, the network plugin uses the network config as well as the IPAM information and it sets up the networking for the container. So this means if for example, if it is a bridge plugin or if it is a weed plugin, what the plugin does is it creates a weed, it takes one end of the weed and puts it into the container. And then it brings up that interface, configures the IP address, configures any routes and all the other extra things that are needed. And after setting up the networking, the CNI network plugin gives back the information to the runtime, whatever has happened, what is the IP address assigned for the container and it passes on the information to the runtime. And then runtime can make a decision, if there is a failure, what should I do and things like that. So that's a high level overview of the CNI. So now let's walk through the specification. In the previous slides, we saw the first step that we do is we set up the CNI network configuration or the network config file. So simple of a simple network configuration. It's in JSON and these are some of the fields. So one of the variables are like the CNI version. What kind of version is the runtime supporting? And what kind of the version the plugin is supporting? So you can figure out what are the capabilities and other stuff by the version and other API calls. And here, this line up here, it specifies what kind of a plugin are we using. So the important point to note here is the type of the plugin, the bridge, is the actual binary name of your plugin. So let's say if you are some XYZ company and then you built a magical tool which you want to release to the community as XYZ-magic. So you specify type XYZ-magic. And this is the IPAM section that I was talking about which is specific to the IPAM plugin. So here also there's a type and the type specifies the actual binary name that is invoked by the runtime or by the network plugin. And have all the other information needed for the specific plugin, IPAM plugin. And during the step of the runtime invoking the CNI plugin, this is what happens. The runtime passes the config in the std in and it also sets up few of the environment variables. So this is how the runtime communicates with the network plugin. And if you look at this, this is the same config file that we have set up in the previous step and the environment variables set up while invoking the CNI network plugin binary. So the important variables here are the container ID. This has to be unique across your installation. Usually it could be a UUID or it could be the Docker ID. And netNS which is the network namespace of the container. Where should we set up the networking for the container? That's what is specified by this variable name. And what is the interface name that needs to be created inside the container? And the CNI command specifies whether we are adding a container or deleting a container. So the process is same whether you are adding or deleting a container. The only difference is the command changes. If it's an add, you specify add. If it's a delete, it's just specify and delete. CNI path specifies where do I find my CNI byte so that I can execute them. And if you have your custom arguments which are not part of the specification, you can pass them using the CNI arguments. And this is the step where the network plugin invokes the IPAM. You can see only the IPAM section is passed from the CNI plug-in, network plug-in, the CNI IPAM plug-in. This is the subsection of the network configuration file. And you pass on the same environment variables. Once CNI IPAM plug-in figures out the information, this is how it communicates back to the network plug-in. You can see the different options available here. You can specify an array of IP addresses. If you have an IPv4 address or an IPv6 address, what are the different routes that needs to be programmed? What is the gateway IP address? And what are the DNS server information? And the different search strings which go into the resolve.conf and things like that. In case there is an error, this is the output... Okay, sorry. One important thing. This JSON information, when it is being written from the IPAM to the network plug-in, this is printed out on the STD out. So the JSON information is printed on the STD out. And the written code, if it is a zero, it signifies it's a successful call. If it is a non-zero, then something failed. So in case of a failure, the code is non-zero and the result, the JSON result is printed out on the STD out. It looks something like this. You have the error code, you have the message and the details. And once CNI network plug-in has set up the networking stuff inside the container, this is how it passes on the information back to the runtime. You can see the different sections of the JSON file. Most of the information is from the IPAM plug-in and then you can add a little more information here and pass it on to the runtime. So this is an error code of zero if it is a success and an error code of non-zero, a code of non-zero to specify an error. So the CNI repositories, one of them has a library and it's an SDK, probably I would say, an SDK to build CNI network plug-ins or CNI IPAM plug-ins using Go. Because Go is more popular in the microservices world so we have ready to use SDK or a library in the upstream CNI repository. And also it has an example tool to test your CNI plug-in that you are building. Earlier the CNI authors have released a couple of reference plug-ins so that people can use them as is in their products or they can customize them on top of the existing plug-ins. So these were part of the CNI repository but in the recent release they have been refactored out and put it in a separate plug-in. So different repository. So you can find these plug-ins in github.com slash plug-ins. So the different plug-ins which are available here are the bridge, loopback, IPvlan, Macvlan. So these are the networking plug-ins and you also have host local and DHCP IPan plug-in. So what these are very useful if you are trying to build your plug-in you know you can use them as is. So the host local what it does is you give the configuration in the network configuration a subnet you know that you want to allocate for your containers and this plug-in takes care of allocating IP addresses keeping track of which container got a certain IP and then once you delete a container it releases the IP address. So the host local is useful in that and the DHCP plug-in is similar to the DHCP client running on a virtual machine. So if you want to get IP address from your DHCP server for a container you could use the DHCP IPan plug-in and the bridge plug-in is one of the most popular plug-ins used by a lot of other folks. So what this guy does is it creates a bridge on the host and when the containers start when containers start they create v-pairs you put one end of the v-pair in the container and the other one on the bridge. So that's how these containers can talk to each other. There are other Metav CNI plug-ins what these do is like for example the tuning plug-in is used to enable sysctl settings on the network interface of the containers. So these do not fall either in the network or the IPAM category so that's why they are called the Metav plug-ins. So these plug-ins you know you can use you can fork these plug-ins and you can build on top of it. I wanted to show you the simple CNI interface so there are like couple of important calls add network and delete network. So the container runtime calls into your plug-in using these two APIs. So when a container starts you you get invoked on the add network you get the network configuration and if you have any runtime information you get that in this call and with the new release of the CNI what they have added here is you can add it's called add network list. So what this does is if you want earlier you could invoke one plug-in of CNI but now let's say if you want to chain plug-ins I want to run the bridge plug-in first and then I want to call the tuning and then I want to call a port mapping plug-in. So if you want to change them you have an option now so you can specify that in your network configuration file. So when a container starts it goes through these different stages and when you delete the container the plug-ins are invoked in the reverse order. So these options are available with the new release of the CNI. So now I just want to show you how to run CNI runtime and CNI plug-in. So before I do that you guys have any questions? Okay I'm sorry sorry I couldn't hear you. So I think it depends on the runtime. So some of the run times currently they do not support multiple plug-ins but it is possible theoretically it is possible. So you just drop in two network configuration files in the config directory and then if your runtime understands them then you can set up multiple networks using multiple network plug-ins. So it should yeah so it depends on your runtime. That's too tiny. Is this visible? Okay so this is one of my demo servers that I'm using. So before I jump on to this I want to show you some of the files that I have created to run the demo. So I will upload these on GitHub by end of day today. I'll put in more notes and and commit them to the repository. So here I have a vagrant file to run the different virtual machines on your host so that we get the similar configuration and for one of the simple demos I have the network configuration file. The step as we saw earlier is you create a config file and you drop it in the config directory. So here you can see I am using simple CNI plugin. So that's the type so the actual binary name for this plugin is going to be the same and here I'm also using a simple ipam plugin. So that's the one which I'm using for the demo right now. So if you look at these binaries how do they look like? So this is the simple CNI plugin. Okay this of course there's a little bit of code here but I'll go through this quickly. Well before I yeah I can go through this. So here what I'm doing is I get my network config from STD. So I'm trying to read the network config into my variable and I'm figuring out what is my name of the network and what is the ipam plugin that I'm using. So that you know I know that the CNI the simple CNI plugin needs help from the ipam. So I need to figure out what is the ipam plugin. So I'm figuring out the ipam plugin name here and I'm invoking the ipam plugin. So let's just jump to the ipam plugin and see what is it doing. Cool a complete hack very simple. So you just take the JSON output that is expected by the CNI plugin. So we have the CNI version, we have the IP address information and whatever is the IP version IP address version whether it's four or six. So I'm just echoing it on to the STD out and the return code is zero. So it's a totally simple hack. So here if you are building your own ipam plugin what you would do is you would use the container ID and then you can query your database and figure out what IP address you need to assign it assigned to the container. All the interesting stuff happens here. And once the ipam plugin result is available what we do is we figure out what is the container IP address from the ipam plugin result and we figure out what is a bridge that that is used to connect this container to. So we figure this out information from the network config. So here if you see we had this information in the config file. So we specify I want to use a demo bridge zero to connect my container. Here what we do is we use the information available from our environment which is the CNI container. Here I'm just trying to use a small trick. So I truncate the container ID, build a short ID from that and I am creating a v-th link in one end of it. I'm naming it as v-th so that this is the end that goes on to the host and the pure name is container dash whatever and I'm setting the host side of the v-th into the I'm plugging the v-th end into the CNI bridge here. That's what I'm doing. The next step I'm bringing up the interface. After that I am moving the other end of the v-th inside the container and after that bring up the interface inside the container and then program the IP address. So I do an IP address add and use the container IP that we got from the earlier step. So this is an example of a very simple CNI plugin. So it depends on the CNI plugin that's being used. So here for this demo purpose I created it manually so you do it as part of the setup but in your CNI plugin what you could do is when you are running when you're involved for the first time here you could add code saying that does this bridge exist? If it doesn't let me create that and you need to give an IP address to that. What you could do is you can use the gateway IP address that you got from the IPAM and use that to program the bridge. So that's that's what is being done in the reference plugin, the reference bridge plugin. This is this is exactly like how it is done in Go in the reference plugin. This is exactly the same step. Obviously you'll see the steps in the Go using the Go libraries. So let's set it up and then see oh let me show you the simulated container runtime. So here what I'm doing is I am executing the plugins. So when I'm executing the plugins I need to pass this information right? So I am exporting the container ID, the network name space, the interface name. So here of course I've hardcoded it to E0. You could use whatever you want and I am reading the network config from the config path and of course this is also in another hack. I'm just assuming there's going to be only one file here and figure out the plugin name. So here I'm figuring it out from the type so so that I can invoke it and here is the line where I'm invoking the plugin and I'm getting the plugin result and I'm checking for the error code if it is successful or not. So this is a very simple container runtime. So if you are a runtime author you could use a pattern something like this and this this particular script takes two arguments whether you are adding a container or deleting a container. So if you are adding a container the the the hack that I'm using is I start a Docker. I'm starting it with net equal to one. I figure out the process ID and from the process ID I figure out the network name space and then I'm linking the network name space so use the network name space name in the IP commands. Earlier Docker used to do that but they stopped doing this so that's why I have to do this manually and I exec the plugin with the container ID and the network name. So that's the simple container runtime. Let's see it in action. So we have the demo host and inside this I have created a few VMs and one of the VMs is one of the VM. So is it good? Okay so we have the simple demo files. So the first thing that I need to do is drop my network config file in the config directory. So the default config directory is slash ETC slash ETC CNI net dot D. So that's a default location and then the next one is you connect the IPAM plugin to OPT CNI bin that's a location for the plugins and the other one is CNI. Okay so now we have installed the config file and the plugin files and now I go to scripts and then simulate container runtime and I'm doing an ad container has started and let's just quickly check Docker exec dash IT bash IP address. So that's the IP address we got from the IPAM plugin that we just wrote. So of course we just hardcoded it. So yep that ends my presentation. So if you have questions we can probably take it outside because we are out of time. One question I can quickly take. Yeah so I'm simulating a runtime. How do you author a runtime? If you want to write your own runtime. So CNI plugin means you need a runtime that accepts the CNI spec and you need a plugin who can do that. So I'm trying to specify and give examples for both authors. Thank you.