 Hello everybody and welcome again to the OpenShift Commons briefings today. We're having a reprise back in May of last year, I think it was. Alvin Creakey of Kinfolk was here demoing a very early version of their integration for traffic control with Weave Scopes from Weaveworks and he's back again. They've made it an official plug-in and he's going to demo it and show some of the new stuff you can do with it. So there are a few folks from Weave also joining us and this format for this today will be Alvin will give his talk and his demo. You can ask questions in the chat and we'll try and answer them and then any of the good questions we'll ask out loud so they get reported the answers and then we'll open up for Q&A at the end. So without any further ado, Alvin please take it away from Alvin. Thank you and welcome. So I will talk about the traffic control in plug-in and how it can be used to test applications with web applications but I will start to present myself. I am Alvin and I contributed to Rocket and at the moment I'm working on Weave Scopes on eBPF. A few years ago I worked on traffic control already but for different use cases it was for multimedia application in cars. So the demo I will do today is we're using that traffic control ID but for web application running in microservice. I work at Kinfolk, it's a company that I co-founded in Berlin and we work on foundational Linux technology. We lack low-level software, we lack Rocket system, Linux with Weave Scope. That's some of the software we work on and you can find more about Kinfolk and GitHub, Twitter, etc. So today as you said it's a second time I do this talk but last time when I demoed traffic control it was in a different state. It was very much an off-off concept but now it's properly integrated into Weave Scope. So I will show Weave Scope with some demo using an application called the Sorgshop. It's a web application developed by two companies, Container Solutions and Weaveworks and it's user microservice architecture. I will show how to install that or how to install Scope on OpenShift and what's the architecture and then I will show the plugin, the traffic control plugin, how to use it and what it serves for and I will show another plugin of Weave Scope. And at the end I will show some guides that Weaveworks has on their website using Catacluda. So Weaveworks, I will start with showing directly so let me create my slide and show you. I will show you my setup so I have my laptop with virtual machine running OpenShift. That's the OpenShift console where I have the default namespace that I created in OpenShift and I have two new namespace that I've created at the bottom, Sorgshop that's the demo application and Weave Scope where I installed Scope. Now I'm showing you Scope. So that's a view of all the containers running on my cluster and here I am on the port view. So the port view is a Kubernetes port containing one or more container image and what's interesting here is I can see the communication between the port. For example here I have the front-end problem and it's communicating with a catalog that means at the moment there is one socket communicating between them. I have some demo effect that I will come back to that a bit after. And that's WeaveScope, sorry that's WeaveScope. That's a web shop where people can buy socks and I have a different page. I will quickly go through them. The home page and the catalog page and each of those use a different container. So when I'm clicking on Firefox here, Firefox is talking to the front-end container but behind the scene the front-end container is talking to different containers. So let's go back here. Since I click a bit on the web page, the front-end established communication with other containers and I see I have a visual way to see the architecture of my application. So it's I think easier to read that way than to have just a list of containers like that. Another thing I like is we can have a more high-level view. So here it's a list of pod but we can see a bit more high-level. You see the Kubernetes deployment. That's not a big difference here because I have only one replica of each pod but on the larger cluster that's more useful to have a bigger view and then I can see the communication that happens at the moment. But I also have the low-level view see here I have all the containers. That can be quite a lot of them and I can have a view per process as well which will take a bit of time on my laptop to load. Here it is. So that's with scope. Let me go back to the slides for now. So our high-level scope on OpenShift. The first step I did was to create a project on OpenShift or Kubernetes namespace that I call with scope. I used the following command to create that with the OC command line tool with OC new project with scope. And then I gave it some privilege because I want the container running with scope requires privilege to be able to list all the process on the node to list all the connections happening. It cannot just live in its own container but it needs to have access to everything to be able to inspect what's happening and report them. So I need to give the privilege capability to the with scope project and I do that with that OC command. And then I want with scope to have access to the Kubernetes API so that it can inspect, it can ask Kubernetes what are the list of ports, the list of services and everything. So I do that as well with OADM command and I give the white of with white access to the Kubernetes API. Once I prepared this Kubernetes namespace, I installed scope. So for that I get the ML file. I will quickly show you how this looks like. So on the Weaveworks web page on scope there is a different way of installing it. But what we want here is to install it off Kubernetes. So we can get this ML file which is described the different components used with scope. With that, I can just install it using, for example, the OC create. Scope has several components. There is one component called scope app. That's what Firefox in my case is talking to. So let's provide the global view of all the containers to the user. But it gets this information from the scope agent and scope agent is a demand set. That means it's a demand running one time on every node. And that's what will fetch the list of process, the list of connection, the list of parts, et cetera, on each node. And then the scope agent is also called the power, by the way. It's sending a report to the app. It's actually a JSON file that I send through HTTP. So regularly the agent send the information to the app. And that's how it works. Now, I will go to the main topic of the talk. It's a traffic control plugin on demo it. So let me go back to the pod view. And I will show you that I added something in the cart here. And if you see, if I refresh the page, it's a bit slow on my laptop, but it's not that bad. And it seems to work fine. But actually, there is some UI problem that are difficult to see for developers who are usually good internet connection. But when there is a connectivity problem, it's more difficult to see from a developer what's happening. So I will show you right now in the front end. I see it's connected to the catalog. I wonder what happens if I add a lot of latency to the catalog container. So here, I click on the catalog pod. I see it as a list of containers. Actually, only one is interesting. Now it says one catalog container. And then the traffic control plugin that is installed on scope, by the way, you can see at the bottom. I don't know if you can read. That's a list of plugins that are installed. It added some controls on the UI to change the latency. If I click on this, that's the demo effect. That's not supposed to show that. But it's supposed to add two-second latency on every packet going through that catalog container. Okay, I will try again. Let me try another time. Okay, I will check if it works now. So to check if it works, I will show you the debugging console of Firefox on the network page. I will reload and I will see how much time I spend on each request. So I see most of the requests happen quickly. In any case, less than a few seconds. But here, as you can see, there is a couple of requests which take more than eight seconds. And that's on the catalog page. The reason for that is that when I fetch the page, the home page here, you need to ask different containers to populate the UI. And some containers are fast because I didn't change the latency. But some containers like the catalog added two-second latency, as you can see here. I'm the plugin. So that's why when you delay every packet by two seconds, it makes it a lot slower. So what you can do with that is when you refresh, you can see the shopping card appear to be empty. That's a bug in the UI. That's not supposed to be like that. It should at least show that something is still loading. So there's not a good user feedback to debug this kind of issues on a website. It's useful to add some latency on different containers to see what's happening. This way, you can debug a website and see what's happening. So I have the source of that page here, that HTML page. And to get the list of elements in the card, it's actually done in a asynchronous JavaScript code. And that's why it was not showing up directly when the HTML page was loaded, but it appeared later. So I can show it again. And then I can, from the UI, remove the traffic control. I have some demo effect, which make it not working sometime. And I will try to do it on another container. For example, the user container is responsible for getting the information about the user that you see at the top here. Here, I'm logged as a user. So if I fetch the user container in the pod, I want to make it slower. When I refresh this page, at the top, you see there is no information here. We can wonder whether it's a good UI or if we can display a loading message or something like that. Okay, that's it for the demo. Let me go back to my slides. To install the plugin of SCOP, they are not part of the main SCOP project. They are on different repositories. Here, you can see the GitHub repository. It's on with works slash SCOP traffic control. And here, you can fetch the HTML file that you can use to install on Kubernetes. And then you can install directly with OC or QPCTL command. And how it works. The plugins are a demo set, although that means that they run in one copy on each node. And then the adjunct on each node will ask each plugin if they have additional metrics to give or if they have additional information to give. In the case of the traffic control plugin, the plugin will tell the adjunct that it wants to have more control on the UI. That is a button to slow down the add more latency or reduce the latency or something like that. So the way it works is HTTP communication on the unique socket. On the adjunct, we ask regularly to the plugin what it wants to display on what metric it has. As I said, it talks on a unique socket. So each plugin should create a unique socket in that directory. And then the SCOP adjunct will look in the directory and talk to whoever is there. That means the directory is a shared bind mount between the SCOP adjunct and the plugins. There are two protocols. The plugin protocol has two interfaces. One to report additional metrics and one to add more UI buttons to have more control. If you want to develop a SCOP plugin, that's the way you can do that. You can follow the documentation. I will show you right looks back here. On the reverse website, there is documentation how to, how works the protocol between the plugin on SCOP. And there is some JSON extract to explain how it works. And there is also some go code showing that. But there is also some existing plugin. So if you go to the GitHub organization, we will have plugins. There is the traffic control plugin that I'm talking about. The Http statistic plugin that I will mention after. And some other plugins. So you can look at them to look how it's done and implement your own plugins. Okay. So now we'll talk how the traffic control plugin actually works. I will explain things about traffic control. So traffic control, the first use case was not about testing like that. But it's usually for, for example, for web servers to guarantee that each client has a fair share of the bandwidth or to reserve bandwidth to a specific application or for routers to avoid the workloads. And how it works on Linux. Linux has what is called a queuing discipline. And it's a network scheduling algorithm which will decide on a network interface which packet to emit next and when to send it. And that's configurable on Linux by default. Network interface always has a queue disk. So it has a default one and you can change it to different ones. As an example, there is one called stochastic fairness queuing. That's just an example. This one will do a run robin on a different class of connection on some one packet at a time on each different queue. Okay. So I'll talk a little bit about queue disk, but how does it work for testing? There is a queue disk called a network emulator or net em. It exists since a long time ago in Linux since Linux 2.2. And that's a network scheduler that you can install on a network interface. And then you can configure different properties like the bandwidth, the latency or to say I want some packet loss or I want to correct packet. So there is a lots of different parameters you can play with. In this demo, I only play with latency. Like I want to add two second latency or a bit less to each packet. And that's what I was using for. But I don't want to change the latency of everything on my laptop because otherwise I've done the video call or everything. So I want to configure the network scheduler only on specific containers. On each containers as their own network interface. So if you can configure the queueing discipline on a specific network interface, then that's okay. Only the thing in that container will be affected by the change of configuration. So in this case, that's what happened. The traffic control plugin of scope actually configure the network interface of specific containers, but not everything. That's it for the explanation about traffic control. What if I have done, I think there is time to talk about the different plugin, still a plugin in scope called HTTP statistics. That would be great. Sorry, say again. That would be wonderful if you could do that as well. Okay. So let me go back to code. I will close this. So I have a terminal here and I have a child script. If you see correctly, that's loop. That's connecting to this URL and so repeatedly connecting, fetching a webpage. While I have that running in a background, I will go back to scope actually and I will inspect the frontend container. So let's see frontend container. I should have inside the list of process. So here I have a node process that is the web server actually on other process. If I click on the node process, the HTTP statistic plugin added additional information in that view. So this view with this graph is not in scope. The thing with that plugin is added by the statistic plugin. So here I see there is about three HTTP requests per second and about the same of HTTP response. And I see that's HTTP 200 response. That means that everything is okay. Actually, if I look it correctly. So let's stop that. And I have another test. That's a different URL. As you can see it's right. So URL doesn't refer to anything. It should return a 404 error. When I run this loop, it doesn't load the file, but the server should give a 404 error. So let's go back here. And after, so that's not completely un-started news. It required a few seconds, but hopefully after a while, it should display the number of HTTP requests per second. And this time the response is no longer 202, but it's 404. If you have two requests, you will see the different graphs on the same view here. So that's the HTTP statistics that I installed. I can show you the view here at the bottom. Here you see the list of ports, but I can filter on different namespace. I can see all the namespace, but maybe that's a lot because there are some containers for some ports from OpenShift, some from with Scope, some from Stockshop. If I filter on with Scope, then you have four containers. That is the agent, the application on the two plugins. Okay. So the HTTP plugins, how does it work? Sorry, that's not what I wanted. Okay. So to install the HTTP statistic plugin, it's pretty similar to the traffic control one. You go to the web page to get the ML file, the Kubernetes ML file, and then install it. That will just create a demand set. So one pod on each node that will give the information. So how does it actually work? How does it give the statistics? That's a bit more complicated. It uses eBPF programs. So for that, I need to talk a bit about the kernel, what happened when packets are being sent between process. So there is a kernel function called skb-copy-datagram-writer, which is called every time we copy a buffer to also get to a specific process. And what the HTTP statistic plugin has is eBPF program attached via CAPROB to this kernel function. That means that every time this function in the kernel is executed, it will trigger the execution of the eBPF program. And the eBPF program will have a look at the buffer to see what's inside the packet. And if it looks like a HTTP request, that means it has the keyword get, or put, or delete, or something like that, or if it looks like a HTTP response, then it will update a variable, hashmap, actually. On the hashmap, it's a hash table, so that's a key value store. Between the key, that's the process. On the contours, which is how many HTTP requests there is, how many HTTP response. And the eBPF hashmap is a variable which is shared between the kernel in the eBPF program and user space in the HTTP plugin. And regularly, the HTTP plugin will have a look at that hashmap to know how many requests and response there are, and then it will be able to send that to the agent. And to install that eBPF program, the plugin does that. For that, it has C function, which is compiled on the fly by IOHazard PCC to a eBPF by code, and it's uploaded in the kernel, and it's installed with the kprob on that function. That's all for the main talk. I wanted to mention some of the Katakoda guides that we've worked on on their website, so. And I'm also going to unmute Ilya, who has joined us if he wants to add anything in. Okay. He just needs to unmute himself. Ilya, do you want to say something? Should I continue? Continue. Yeah, Karen. Karen, Alden, please. So on the Weaveworks website, there are some guides, and I will show you how to, it's useful to give software like a Weavescope a try without having to installing or having to install a Kubernetes cluster or a friendship cluster. So if you don't have a VM or if you don't have a cluster, and you just want to give it a quick try, you can go to that web page, and I will show you how it looks like by trying the first one. And it's a guide with explanation how to set up Weavescope in WeaveCloud. And if I click on the links, I have to connect it to the WeaveCloud, but I prepared that before the talk. So I can switch it. Yeah, it's already prepared. No, it actually doesn't work. Okay, so. Because it's live? Yeah, it's live. And I guess after one hour, it disconnects if I don't use it or I don't know exactly. Okay, so I will stop my screen sharing. While he's doing that, the other folks who are on the call, if any of you have any questions, just raise your hand in the chat and I'll unmute you and you can ask them. I know quite a couple of you I recognize is having large production deployments of OpenShift, and I'm curious to hear how you manage monitoring and doing this sort of traffic control work currently, and if any of you are already using Weavescope. Can I continue if not? Yes, please. So I click on the link, and now it seems to work. I can, after just a couple of things, I can, I have a Kubernetes cluster, I can type some kubectl commands, and I can deploy things by just clicking, it deploys the SockShop demo, and it's already running. And I can have a look at this page to see it's already running. It's faster than my laptop. And I can install scope by clicking, I can type, here I want, and it's running, and now I should be able to see that in WeaveCloud. So that's the very similar view to what I showed you before, but this one is not running on my laptop, it's running on WeaveCloud, and I can see the list of all my containers running through that cataclysm code. So I will not show further. Ilya, is there anything that you would like to add to the conversation here, from your perspective, on what's coming with Weavescope, any future features that we should watch out for? Oops, Ilya, we're not hearing you like, and to turn on the media and not the audio. We have recently released Weavescope 1.0, and the features of 1.0 include external service recommendations, such as if you're after stalking to something like S3 or DynamoDB in Amazon Web Services, you would be able to see those quite clearly in Weavescope. Other things we've been working on are to do with WeaveCloud, and we have additional services in WeaveCloud, which include Weave Flux, which is a continuous delivery tool for Kubernetes, and it should be possible to make it work with OpenShift. We'd be looking to hear about your use cases for continuous delivery, and if something like Weave Flux could help you, I'd be happy to talk to you about that. Additionally, we also have Weave Cortex, which is a Prometheus as a service, which is part of WeaveCloud, and I'd love to talk to people about this too. We're looking to integrate all of the three products together in the very near future, and you start seeing features which cross the troubleshooting, distributed systems, debugging, that scope does, and time series that Prometheus does as well as continuous delivery tech stuff. Perfect. We'll definitely get you on to do a talk and a demo of Flux and some of the other parts and pieces. I'm going to leave it open a little bit longer here to see if any of the folks here. I'm very curious if anyone who's on listing in right now or listing later, if you're already using Weave Scope, if you tested the traffic control plugin on your production environments already, so I'd love to hear from you either the email or the commons mailing list, and that would be great to get your feedback on the plugin and any insights you have to share with Kinfolk and Weave on that effort. So please do if you're using in production or thinking about it or have a POC, the WeaveCloud Catechona. Stuff looks great as a way of testing it, so I'm really pleased to see that, and hopefully we can all take advantage of that. So thank you very much. We will be very close to Albin soon. March 28th will be the OpenShift Commons gathering, will be happening in Berlin, and we'll be able to have Alex Richardson is going to be speaking there. So if you're interested in coming and meeting some of the Weave folks and some of the Kinfolk we will make sure that they're there and available. That's March 28th and to find out more information about that, that's commons.openshift.org slash gathering, and you can find all the details and register for that if you'd like. It's the day before KootCon, so if you're coming to KootCon, come join us at the gathering and we'll hopefully connect you with all the Kinfolk and Weave folks that you need. So with that, is there anything Albin or Ilya you'd like to add on top of that? Oh, and I'll be at Fosdown too, so good. We'll see everybody in the chaos that is that wonderful event in Brussels. It's one of my favorite ones, and there are two rooms at Fosdown this year that are dedicated to Kubernetes, so there'll be lots of talks there if you're coming and that's in two weeks time. It's at the fourth and fifth, I believe, of February. And that's in Fosdown. I will give a talk about traffic control as well as Fosdown, but that's in a testing dev room, that will be a very short 10-minute talk. So I will be both at Fosdown on KootCon. The wonderful thing about Fosdown is you're giving a 10-minute talk, it'll take me 40 minutes to figure out what room it's in and where it is, but I will see if I can get myself there. All right, thanks again Albin and Ilya for joining us, and if any of you have questions, please do reach out either directly to the KIN folks, Weave folks, or reach out to me and I'll connect you with them. All right, thanks again, Ilya, and take care and have a wonderful evening. You too. All right, take care guys, thank you.