 Now, we're going to get the next talk by Roman on networking in Swiss Army for Go, a round of applause for him. OK. Hi. My name is Roman Moore. I'm working for Red Hat Soft Engineer. And it's great to see so many people around. As always, the small rooms are pretty full. And I want to tell you today a little bit about Go and how to solve networking problems with it. And also, in general, why Go is such a nice language to bridge the gap between providing rest services and still having the possibility to work pretty low level and to know it at the same time. And basically, I want to go with you through three main points in my talk. First, I want to talk a little bit why Go is so nice for infrastructure service writing. Second thing is I want to introduce you a networking example or a task I had to do to solve. And finally, I want to show you how we can solve that purely with Go. First, let's start with why Go is so nice for writing web services. And one thing is pretty obvious. Compared to languages like C, you can just, for instance, annotate your structs and just marshal the data. And if you run it, you're already done with creating a JSON. That's not something which is so trivial with some other low level languages, like for instance C. Also, providing rest services is pretty simple. Here you see an example where we are creating a rest service with a slash user path. It can take XML or JSON. And there is an even a sub path for it where you can specify the user ID to fetch it. And it takes some parameters. And then you just register the find user callback for looking up the user based on the ID. And what you see here is the Go RESTful framework, which is, for instance, used inside Kubernetes. So far it's good, but there are, of course, other languages like Java or Python Ruby. They all have nice web service frameworks. And there is definitely nothing wrong with using them instead of Go. Might even be better because of, especially with travel because of the library support or so on. But what Go also provides you is working on the low level details on your notes. Here, for instance, that's something you would see if you, for instance, try to use pcap extensions for Python. And when you don't have the pcap DevLibraries installed, you will get a message like this if you want to pull in your dependencies. GCC will try to compile an extension for Python, and it will just fail because you don't have the DevLibraries installed. And how do you solve it normally? You pull in GCC, pcap DevLibraries, and then you can finally add your application. That's a process normally you then have to kind of package the results to bring it to a node. And that's not really necessary with Go because there you can extremely easy access all the low-level details of notes. And to show you how easy it is to work low-level with Go, I came up with a networking example I had to work on some time ago. It's basically based around VM to host networking. So we wanted to attach virtual machines to host networks, which were already existing, without the need to modify the host network too much and integrate the virtual machines in the host network. For instance, you had an external DHCP server. It should get the DHCPs, the IPs from there and so on. We didn't want to make assumptions like that the network already consists of a bridge or an ordinary interface. And make VTAPs would have been nice to use there since it gives you some nice reduction to the delays of the network traffic compared to bridges. And finally, since I'm working in the communities area with virtualization add-ons there, we wanted to have the possibility to ship our code as a single binary without dependencies so that we can use the pretty common pattern of init containers, for instance, in the community. So you just put your binary in a container and then you have to place it somewhere in the host so that management application there or a demon there can find your binary to use the plugin. And therefore you don't want dynamic dependencies for your code. If we now look a little bit more detailed on the task, what you normally see when you do virtual machine management is you have a bridge on the node. You have your interface attached to the bridge and then you have two machines here, in this case two, this VM and this VM, they both have their own TAP devices. They're all attached to the bridge. And if now let's say the VM on TAP one wants to ping the VM with the dot three suffix, it can just reach it. And if it wants to ping the node, which has the dot two address and the two, that's also no problem. Now with make VTAPs, it's a little bit different. Their advantage is that you can directly attach make VTAPs to the ethernet bridge, so to the ethernet device. You don't have to prepare the node in any way for that. And in principle, that works already. It can talk to every other node, but it can't talk to the node it's currently running on. That's the limitation of make VTAPs. So the virtual machine on make VTAP one interface can without any issues reach the other machine. And it can also reach every other node in the cluster, but it can't reach the node it's currently running on. So this ping would fail with the dot two prefix. One solution for that is adding extra routes. So you can say that a traffic coming from that, from the dot four suffix IP should be routed over a make VLAN on the node, which is also part of the same network. And since this is on the node, finally we can ping the node. And that is actually pretty complex to solve if you wanna make the generic because you need a route for every virtual machine you create. And if you don't wanna make any assumptions about the host and if you don't wanna make any assumptions about how the virtual machines gets their P, I mean, I accepted that it's done by DHCP, it's pretty complex to find a solution to get all the pieces together. What you can do is if the make VTAP one device sends a DHCP request, you can steal it with traffic control from the ethernet zero device, move it to a bridge where you then can, which is in promise creative mode, there you can analyze the traffic and see what's going on. And once you're done with analyzing, you have to re-inject via the make VLAN to the packet so that it finally gets to the cluster or that the answer gets back to the make VTAP one VM. And that's what I'm trying to show you now with Go. And we will start with wish we're on a net leak. That's a net leak implementation for Go. It's based on the Docker networking library originally, but was since then greatly enhanced. And in general, all rivalries I'm presenting here, I'm using them every day, but I never contributed there. So all the thanks go to the, all the cool developers out there which we are creating those libraries. And we will start with something pretty simple here. I'm just importing the net leak library and let's just list all links on the node. And it's pretty simple. And I'm just printing out the dump objects again in this here. I have a local network. That's my Ethernet, my Wi-Fi, some virtual bridges. And here I have a Sniffer X device, which we'll usually add on. So pretty simple to get it already. That's also pretty simple with the core library of Go, but also nice with net links. We can also have a look on the addresses attached. In this case, with link by name, I'm just fetching the low interface and fetching the addresses assigned to the interface. And then just let's print the interfaces. In this case, you will see, okay, the low interface has the typical localhost interface, a localhost address. All the creating interfaces is pretty nice. Let's suppose we wanna create a McVeland device. You just create a McVeland struct also with net link, give it a name. You tell it to which parent interface you wanna connect it. In this case, my Ethernet device has the net link too and I've just hardcoded it here. And you can even say in which mode the McVeland should run. And then you just call link add with that. And in this case, I'm then finally fetching it and printing out. And here we go. And here we see the McVeland device. It was created and printed out. And it was down at the stage when I fetched it again. Now we already have the first part of our solution, a small part, but now we've already seen how we can create McVeland devices. And so this part is now solved for our puzzle. Next thing is, the next topic we wanna do is we wanna now wanna create a sniffer bridge or bridge where we wanna introspect traffic and we wanna use traffic control filters to move the packets from the Ethernet device to the sniffer syrup bridge. One could also try to use IP tables, but that would not help much because McVeland and McVTAP interfaces or traffic is so slow and the kernel stack, it never reaches the IP tables. So you really have to use traffic control. Also, if you do a TC, yeah, otherwise it will just not work. It will just ignore any traffic and you have no influence on it. That's why you need to use traffic control. So the first part is we're just creating a bridge. Again, you just create a bridge struct, give the bridge a name, add the bridge, set the bridge to up, link set up, set it to promise creative mode, otherwise we won't see much when we're sniffing the traffic there. And then let's just print from IPA, the output, so that we see the bridge. And if I'm going down here now, you see the sniffer syrup bridge was created and it's in promise creative mode. And so, we are one step farther. Now we already have the bridge too, we want it to respect the traffic. So let's steal the packets next. First, to steal the packet, we have to create a ingress queue disk. That's something you need in traffic control so that you can then finally attach your filters to. There are different ingress queue disks, but I'm just using standard one, adding it to the interface with the link index two. And then when we've created the queue disk, we create a U32 filter for traffic control. We tell it that we're interested in the Ethernet packet frame in the value at offset 20. And the 0x44 value, 0x44 is the port 68 and that's the answer from the DHCP server we're interested in. And we then create a filter where we tell the filter on which parent interface it should be located. That's again my Ethernet zero interface and that we are interested in Ethernet frames and then where is the selector now? Oh yeah. And that the filter should contain the selector we prepared. And finally, we also wanna tell it that the action it should take if the filter matches is mirroring the traffic. So we just create a netlink.newMirrorAction and tell it to which device we wanna mirror the traffic. In this case, I had a device called foo. And I'm just taking the index from the interface here. And then finally, filter add, the filter is there and we'll output the resulting filter to see if it really was created. And there we go. We now see on, where is it? On index two, that's the device index. A redirect to device foo was created and we're stealing the package. We're even seeing that. And that also does its merit action. So we're farther. We got one step farther. We have now stolen the packet. Next part is introspecting it. And there, GoPacket from Google helps you a lot. That's a really nice library for introspecting network traffic. And you normally start by creating a, by opening a pcap connection to the interface so that you can save on the traffic. And then you tell GoPacket what kind of traffic you are trying to insuspect on by, by creating a new packet source. So we can try to run it. And then we see exactly what we didn't want to see and import from C code. So how we can we go around that? And the answer is pretty simple. There is, for instance, from any layer raw packet which implements the packet reader from Go standard library. I think that's not the correct name. I forgot the actual name of how it's called. But this raw packet allows you to listen on packets. You just give it the interface and you tell it that you're now trying to listen on ethernet, internet, ethernet frames. And that's important here. Here you can do something run pretty easily. For instance, if you are really listening on a normal ethernet device, you obviously see ethernet frames. But if you're using, for instance, VLANs and you're creating the filters which we had before. Then you probably have to recalculate the offset because you're not really on an ethernet frame and the offsets for instance, finding out the port would be definitely wrong. And after we've created the raw connection to the interface, we then just have to implement the read packet data method where we are waiting for packets to read. First, we're creating a data buffer which has the size of the maximum transfer unit of the interface. We read the data into that buffer and then we are filling in some capture info like when we captured it, how much we captured, how big the packet is at all. In this case, the reader always reads the whole packet so it's the same value. And we tell the capture info which interface it was and then we return the data. And actually there you don't see much except that it's now compiling and no C library was linked in. It didn't really get to the point to creating a more useful example here. And once we have that, we can use this new packet source from before and we'll listen for packets there like this and then have a for loop over these packets which are coming in there. And then we can just take the packet and try to convert it to different layers when we're working on it. In this case, if I want to see if it's on the UDP layer, I just try to convert it to the UDP layer via the layer method and if something comes back, it's a UDP layer, it's a UDP packet, otherwise we can just continue, we're not interested in it. Then we convert it to UDP packet, do some checks and calculations on it because we want to resend it later and that's necessary, otherwise we can get them all from package. And we can also go farther and say we're now only interested in packets with port 68 since the DHCP server is listening there. And yeah, so there is a lot more to go packet but that's in general on how you can introspect your traffic there and we can use it in our example to extract the IP address and make address of the DHCP which is very valuable information we're interested in. And so now the next part is that we also need routes, now we have the IP and now we need to create a route. And to create the route, we need to create a destination IP that's the IP of the VM, we need to create the gateway to get the address for the via part of the route. And in this case, since I wanted to create the demo in this, the route in this code snippet, I'm also adding a device to the interface where I want an address to the interface where I want to create the route, otherwise I can't create the route. So I am giving the interface where I want to add the route, the address of the gateway and it can simply be assigned to the interface by address. Then I'm making sure that the McVillain is in upstate and finally create the route. I give it the index of the McVillain interface. I give it the destination IP, I give it the this gateway ID and I add the route. And if we now run that, we will see that the right route was added. We are now routing from the VM IP over the McVillain with that IP to the host. And if we go back to our example, we're now on Safari again, we have that part also added. Now we're almost done. What we're now missing is we need to write back the packet after we've read it. And therefore we can again use the raw packet package and just implement the right packet data. And then we can directly write the package we've read from a GoPacket with that handler to a different interface again. And so we have that part of it. And that's pretty much the talk. Here is an overview of useful libraries I've been talking about now and also about libraries which I didn't have the time to talk about mostly. The Netlink library, the GoPacket and draw packet for packet introspection, then for IP tables you have from CoreOS IP tables. There is an interesting difference to the other packages. This one requires the IP tables to own the host. So here I didn't completely fulfill the goal of my talk but it's still a very useful packet. Also if you wanna send and receive DHCPX or whatever, you can use the DHCP library mentioned here. Yeah, that's pretty much it. So questions. What, the testing strategy in that case? So the question was what would be my testing strategy for such scenarios? I'm basically just creating isolated bridges, attaching make-we-lens, make-we-taps and all that and just sending the packets and seeing if it comes. I mean, you need privileges so it's a little bit harder to test than if you're running unprivileged code but works pretty well. Does that answer the question? Yeah, sure. Just writing test scripts with or creating the interfaces and yeah. So and for instance, the Netlink library here has some pretty nice example on how you can write the test because they have a lot of tests. So it's very nice place to start looking when you wanna automate it. And I mean, we're using parts of that what I've presented here, for instance, in Cupert. And in Cupert, we have all the end-to-end tests from the user, so your provision of all Kubernetes class to installing our add-on and then we're creating all the objects and then we're checking the host if that really happens. So I mean, you have to test it on all layers starting from directly single nodes with unit tests basically up to the host here. Okay, looks like that's it. Thank you.