 Hello everyone and welcome to the CNCF on-demand webinar introducing MISU the Kubernetes API traffic explorer for Kubernetes Armofel, and I'm an upline co-founder and leading our customer experience things and I'm honored to be here today and share How our open-source project help developers devops SREs testers And basically anyone who runs their API in a Kubernetes environment troubleshoot their APIs So here we go Quick agenda after today You will know these three things One there is a better and easier way to debug your APIs in Kubernetes How to deploy and use MISU on any Kubernetes cluster right away Utilize MISU's feature To help you troubleshoot and actually capture issues in real-time I'll start by explaining the pain and a high level and I will do a quick overview of MISU to get everybody in sync Don't worry. If you've never developed a single code to Kubernetes API, you're still good to go. No, you can try that Then we'll jump into a hands-on how to demo I'll show you how you get from zero to on zero Getting all the API's tracked with MISU and how you can actually even do an on-demand tracing for a split simple scenario Summary and takeaways and a couple of frequently asked questions that got our MISU users and customers Intrigued so we'll answer them here and give you the next steps And with that let's dive into what happened So first of all, if you are a backend developer This slide is for you. We feel your pain Kubernetes developers are troubleshooting in the blind and it's time-consuming not to mention frustrating the business logic is distributed across many hundreds maybe thousands of services with high complexity and Exponentially more API traffic than we used to before Developers have very limited access to that traffic The best analogy that I can give you Think of debugging your website without no Access to dev tools in your browser or any type of logs of your website browser or brother Fun, huh? How do we get to that? The actual reason for this API jungle It's because we decided to go on Microservices and making sure each microservice is decoupled to gain all the beneficial of technology agnostic Scalability is of use et cetera In this picture you can see we had the monolith on the left end and we had three parts We had the external APIs. Those are the github Twitter Google Facebook that used to do the single sign-on log-ins, right? Or for example the stripe for payments, right? They stay the same You just call them you get a response great. No problem. You can even do that with a curl command however your code base all those Interfaces that you expose are now internal API's they represent a significant portion of the business logic that Replaces what used to be a public interface in the monolith? so Remember you just used to just do an instance of a class and then dot and auto complete to get access to that now You actually need to call an API function There are exponential more API's to debug the behavior just moving from something that runs locally in the same memory to network Serving multiple business cases. Sometimes it's not just me calling that code Many other microservices we gaining the ability of the reuse we can use that however now the APIs are more complicated harder to debug covered under new variety of new protocols for example Your PC and rest you can have both of them even Called over the network meaning hey TCP. Sometimes the server is not responding. You got 500 and encrypted So with that how you debug? So let's have a look on what we have today. So today we have The APMs Right, so the APMs are there for quite a while. The problem is microservices represent a revolution as opposed to just a simple evolution Where it relates to the infrastructure? It requires a new ecosystem to support it Here is a look into what API people what the API people have today If you have just APM, it's like me telling you you have access to the CPU percentage 10 years ago, it's not enough Existing solutions focus on limited information that tells very little of the story So if we look at like distributed telemetry and tracing They do give us the ability to understand like the response and the response code and the path However engineers need access to more information In order to travel to the API's and they usually need it now over several microservices communication including the API payloads between them and maybe Into the message queue for example on the or the third party as well and it is of course now So what do we do? Let me introduce MISU What MISU allows you to do is see the Kubernetes traffic It's a simple but super powerful tool that I'll give you an analogy. It's like reinventing wire shark for Kubernetes It can run on any cluster no matter what size Instantly meaning you don't need to do any code change no deploy no SDK no changing your infrastructure Implementing a sidecar or something like that. You don't need to give me your coupe CTL and Wherever that coupe CTL points to start debugging You will see all microservices communication at real time Filter it according to your needs and see all the calls and the responses It works with all modern protocol services service meshes as well It allows you to see the rest and your PC the Kafka and the rabbit MQ the red is the linker D the Istio and Encrypted data whatever it's empty less or TLS So How about we'll just take MISU for a spin. All right, so let's jump into the demo What I want to show you is how you can download MISU Deploy it into your Kubernetes cluster and start debugging the application super fast so what you see here is we've socked just a demo application it's installed on my Kubernetes cluster here those are the pods and Here I'm going to actually start using MISU. So the first thing we want to do is download MISU How can you do that? You go to Google and if you'll type Kubernetes traffic viewer, you will see the Github repo You will probably see the up nine blog about it and they get MISU.io page Which either one will get you the command that you need you just scroll here Copy the command that you need and just paste it into your CMD line What will happen? It will just download MISU. It's a file around like 43 megabytes And that's verify that we actually have the file and it's got the right permissions. That's great Which version of MISU? Are we using right now? MISU version will test 29.0 today is March 22. So that's the version that we have today Okay, good. Now what I want to do. I want to actually start playing with MISU So what I need to do is just normally do MISU. Ah, sorry for that Let's make sure that I had the permission that I need, right? All that is double checks that what MISU is doing you can see everything is good to go and with that Let's activate MISU. MISU tap dash and the sock shop Or if you wanted to do everything is just dash capital A Enter and MISU is basically now deploying everything into the Kubernetes cluster You can see the pods are deployed here and there is a demo set per node And this is the API service that you can see here. Let me just do that Here we go Now we have MISU on the one side my application on the other side Let's bring up the Network config and do a small refresh and you can see I have all the traffic data here Similar to that, but let me show you something else. Let's do a login. That is actually Not gonna work. So if I do log in that You can see I know myself here the 401 that we saw here. Yeah, we can see it here from my computer to front end But however, we can actually see what happened from the front end to the user, right? so I can click on that and Just click response and see the request response. I can see that. Oh, not found for example If I'll do something different, let's say I'll use my username and I'll just add a couple of characters to the password Do the login? Now suddenly we can see Same behavior, right? But now it's unauthorized. Very interesting, right? Cool. Okay, let's start playing a little bit, right done with that. Let's give you the right password and right now We don't need that anymore What I want to do is first and foremost, I want to debug a specific case. What I want to do is I want to see What happened when I try to buy something that is over $100 I'm gonna show you add it to my card and Let's reduce the quantity to that proceed to checkout. I Got 100 so I want to kind of Figure out where the problem is we can see if we open the DevTools, let's just do that again We can see that we got the 406. So we got the 406 from here. Let's trace it What we can see is first of all, it's an HTTP request, right? So let's go and HTTP Edit the filter here. Great. Now what I want to do is okay. Let's see who is actually involved with that So let's do that again. Here we go We Looking at This so front and cold user sock shock, right? We have the user now After the user it calls orders interesting and get already 406 So we see Reducted we're gonna touch that in a second and we can see it's not acceptable and we can actually see the code Not the code the exception that that code through and what was the problem here? Okay, I want to debug that one level further First of all, I want to see who plays with whom here. So let's look at the service map Look at the service map and this is a service map inferred from traffic So we can already see what we have here. We have front-end calling carts front-end calling user Front-end calling orders orders is calling payments Hmm interesting. Let's create a Mizzou Tapping regex that will kind of capture that so what we're going to do going back to my screen control C and As you can see control C just Removes Mizzou completely from the cluster. No trace back. No problem And what I want to do is I want to have a regular expression that catches like front-end orders and payments And remember the reduct. I want to see all the data No reduction because I want to debug everything again now this time we're tapping only four pods and Let's give it a second. Here we go. And now let's do the same thing. So let's clear the cart okay, and add Holy again a two cart and You know just for the sake of it, let's clear the let's clear the filter and Item in cart clear the filter proceed to check out. Here we go Stop perfect. Now. We basically have the trace of the application. So look we've got My browser calling front-end front-end calling customers Literally showing me the request and the response. Remember, we had that reducted now We can actually see the address and everything that we need here great after customers after the user what we got We are having orders doing a post With all the data to payments great I'll move myself here and we can actually see the entire request and the response remember here. We only see the orders Uh, what is the response? Oh, no Payment is false payment declined. Why is it 200 then? Huh? Maybe I'll call my third party Or maybe we'll call the service owner and kind of check with them. Okay, great payments then we see orders still checking things with the user and then Front-end service get from orders 406 which is making sense you you ask for that and Orders now responds with not acceptable Giving you giving the front-end gateway the information they need and the message to bring to the customer as well, right? So now front-end Answers the user which is me Hey, that was your request That is the response. So I I personally will not add the exception there, but this is the error that the user got Did I have to change my code? No, nothing by the way on top of that Showing an open opi open API spec for that and you can just go here Look at the front-end and maybe send that to someone else Here is the post to the cart can grab it from here and then do it with the curl command or something like that whatever you want and That's how you debug with Mizzou with literally less than couple of minutes Going back Cleaning Mizzou just control C Again, I use the no-reduct of Course you can have a profile that says what are you reducting and what not? There is set of policies and I'll touch that in the Q&A. So with that going back to our presentation Okay, switching back to the slides So let's talk about couple of frequently asked questions that we got from our customers How do I top my entire cluster which might be useful if you have like a dev environment multiple namespaces? You just want to tap everything start from there How can I redact remove the items from observed traffic? Which again, you know, especially if it's production, but also like in prep production Maybe you don't want to share the authorization keys of the API keys that the individuals are using How would you be able to redact that from data? Are you running as a sidecar? I explained that Which permissions does Mizzou require to run and and what is the overhead? So you would be a bit surprised with the answer. So let's start with How do I tap my entire cluster? So super simple Mizzou tap dash capital a just take everything and that's it You can have dash n and the namespace just as I showed you that's also a possibility to do Can I redact or remove items from my observed traffic? So out of the wall out of the box Mizzou does reduce the value such as token Authentication password it does that on the header level and the body of the request and response So it kind of gives you a clear thing. We literally took a whole document or article in our documentation to kind of define what are the rules and Explaining how you can remove or add to that list So for example, if you have your own special header or key or value Then you can definitely add that as a one-time to your client and share it with people Or just in a central as way in the Mizzou pro just define it one time and then nobody will be able to do that And that if you're using the open source dash dash no dash reduct We just open everything for you if you need to debug something and you don't want to redact at all You can even make that default Are you a sidecar? No next no, sorry Actually, we started as a sidecar. The problem is that you are doubling the resources. It was super super hard What we've converted to is a demon set per node that is actually have a pod that obeys the regular expression or the namespace and One more pod for the web server that kind of shows you all the traffic and everything So it's very lean, but definitely a mean tool and with a control C. You just clear everything Let's go into the permission so permissions in Kubernetes are super hard to explain We again created an article just for that describing all the permissions And of course you can do the Mizzou check pre-install and then check after the installation on a high level Okay, as a rule of thumb if you have coupe CTL if you're using coupe CTL to manage your cluster Deploying removing deployments changing port forwarding, etc. You are most likely ready to go with all the main features of Mizzou For empty LS and TLS. We do require some Linux capabilities for the eBPF technology some of them are here actually and It's fine. If you're not feeling comfortable with that you don't have to and we still show you the unencrypted traffic If you're using for example link linker D, then there is a seamless integration. You can do that same goes for Istio And the last what is the overhead of the capturing so? hard to say because it depends on a lot of parameters such as your node Specifications such as the amount of traffic that you're going to do like request per seconds and even the size of those requests From what we've seen It's none to very very low amount of resources that are actually being consumed by Mizzou you can also define what Mizzou will use and Mizzou runs in lower priority than normal applications, which means By default we prefer your application to work rather than us seeing the data Of course, you can switch that automatically. You can just say I want to run in higher priorities For example as a standard debug tool for everything you prefer to see that let's say you have a Loop that goes over and over causing CPU. I want to see what started that maybe From an API then that would be great. So Most like again operation super low even the encrypted data and the eBPF overhead is Very minimal and you can try that I will definitely try that before I'll go into some edge cases of you know, like a APIs that are doing 50 megabit megabyte per second and their overwhelming Monitoring or normal systems Next steps So you get Mizzou from getmizzou.io or from the github we where you can just build it yourself Documentation getmizzou.io slash docs our Mizzou community where you can just come and share Problems thoughts questions. It's super fun to to hear from what people are using Mizzou to as I said Somebody came and just asked us to compile Mizzou for Rapsberry Pi because he had Kubernetes of Rapsberry Pies Crazy amazing One more thing next week. We're gonna have another on-demand from the on-demand webinar from the up-name family Done by no other than Tom Ackhurst the founder and maintainer of Warmok He's going to explain how we took Warmok into the Kubernetes microservices world. Here is the link and Thank you very much. I Hope this was useful for you guys and would like to see you in the Slack community Have a good day or rest of the day