 My name is Mark. I'm a console support engineer at HashiCorp helping customers with their service mesh. Today I'll be talking about how to troubleshoot Envoy in Kubernetes using KSNF. When using distributed systems like Kubernetes and Envoy, you'll encounter a few running pieces when trying to troubleshoot. From my experience with customers or from what you heard today, this can go from networking to configuration or security. All of these can take time, especially when you're trying to troubleshoot on a time crunch. And this is where KSNF has worked in my favor. KSNF is a cube-cuttle plugin that allows you to capture packets on the wire or network in your Kubernetes pods. On execution, KSNF will open up the Wireshark user interface for you to capture packets in real time. So you'll need to install Wireshark locally before using KSNF. And if anyone is aware or isn't aware, there's a few components on the Wireshark user interface. So let's look at the KSNF command line, which starts off with cube-cuttle, SNF as the action, your target pod, the namespace the pod is in, and dash P so that KSNF has access to your pods to SNF. And this is what it will look like in a Wireshark. And here's a diagram showing that on execution, a KSNF pod is created with a privileged container and that container will access your pod's container. And on the right just shows KSNF locally opening Wireshark. So when we have Envoy in the mix now, we're going to use the dash C flag to target that container, and this is what it will look like. And there's not much difference in this diagram except that the Envoy container is now being targeted by a privileged container. So now that we've gone through the command line for KSNF, let's try to troubleshoot Wireshark Envoy on Wireshark. It's important to know that Wireshark won't explicitly tell you what's wrong or how to fix it, it'll simply gather the data for you to analyze. So you'll need some knowledge or a front knowledge of general and Kubernetes networking. So whenever I troubleshoot Envoy on the wire, I want to make sure that I know the difference between normal behavior and abnormal behavior. So let's say I'm trying to configure mutual TLS on Envoy containers and I need to see what a healthy TLS handshake looks like. So I run KSNF, Wireshark opens, and I can try to sip through logs and I can see a healthy TLS handshake at the top right. And if I want to go a step further to see that my downstream is targeting the correct upstream, in Wireshark I can click on the client hello packet, click on the TLS drop down, and check the server name, which is correct here. And so this is a normal behavior. What about abnormal behavior? Wireshark has fatal alerts to show you what that looks like. So in this scenario, it's certificate expired. And if I want to go a bit further, I can go into the Envoy admin panel, go into the search section, and then try to see what else is happening there. So we troubleshoot it from the Wireshark part. Let's try to look at the Envoy logs and then try to open up Wireshark with KSNF. So whenever I help out customers or my own projects, 503s come up, as you may know, and those errors can vary. So in this scenario, I see a certificate verified filled. So what does that look like in Wireshark? So if I sift through the logs, I can find a fatal alert such as certificate unknown. Also, I can see that there's a pattern of reset packets, which in my experience usually correlates to 503s, which is helpful the next time. And what about 504s, which can bubble up to a upstream request timeout error? So if I really want to troubleshoot this, I want to compare the downstream side car and the upstream Envoy side car and see what happens there. And from this example, I see reset and TCP retransmission package, which is indicative of network congestion. This can also indicate that I need to tune my request timeout in Envoy, which is a part of the configuration aspect that for me can take time. Here's a few other ways that you can use KSNF and Kubernetes and Envoy troubleshooting. And this is also a link to the case in the repo, if anyone isn't aware of it yet. But I hope that my examples I provide help you understand that KSNF can be used to troubleshoot networking, configuration, and security issues. Thanks.