 Hello, and welcome to Kubernetes Ingress 101, Why and How. We're thrilled to talk about it. Hello there. Welcome back to another 101 Essentials Track presentation for the Open Source Summit. This morning, our talk is going to be on Kubernetes Ingress. Presenting will be Theodora Chang and Vanessa Wilburn, both from IBM. Following the presentation, there will be some time for a Q&A. With that, let me hand it over to Theodora and Vanessa. Hello, and welcome to Kubernetes Ingress 101, Why and How. We're thrilled to talk to you about this topic, but before we get started, I wanted to point out one thing. Although this topic is 101, it is not Kubernetes 101, it's Ingress 101. So we expect you to already have some basic knowledge of things like clusters, pods, worker nodes, and so forth, some of the basic aspects of how Kubernetes works. There's been some great sessions this week, and hopefully you got to see those. So if we go on to our next slide, it is a quick introduction of myself and Theodora. We have myself, a product manager here in IBM. I'm a program director. Been enjoying the cloud world for five years, specifically containers. Kubernetes and OpenShift. A lot of my job is guiding product design and development to build great products. And Theodora, who is kindly presenting with me, is a software developer in the IBM Cloud Ingress Team. She focuses on microservices and DevOps to maintain high availability and continuous integration. She's a Golang engineer on IBM Cloud. So if we go on to our next slide, we'll give a quick look at what we are up to today. So I'll start with why Ingress. I'll even give you a little bit of a story of how Ingress fits into the overall Kubernetes networking landscape. We have tons of questions about Ingress versus NodePort. So I'll do a quick tour into that. And then probably what you're most interested in is the common use cases and annotations we see with our customers at our company. Then Dora's gonna take it over and talk about the community controller, what it is, some of the advantages of using one, and then no presentation is done until we have a live demo. And she'll be showing configurations for security and reliability, some of those key pieces we see, and eventually troubleshooting Ingress. So let's go on to the next slide. So this diagram here is just a 10,000 foot look at the overall connectivity for Kubernetes. I'm just using the IBM Cloud setup just for you to kind of get a feel of what this is like. I'm not gonna get into the details of every piece of this, but for today I'm going to be honing in on that blue box in the middle that shows the ALB or in CUBE turns the Ingress controller. As you can see, it lives on a worker node and handles client requests. In this example, this one is handling public internet requests. Well, let's dig into this a little bit more on the next slide so you can get a feel of how all of the CUBE networking works. And I'll just work left to right across here, getting to ultimately today's topic. So once again, overview in the CUBE networking world, there is worker to worker communication. And that's where all worker nodes must be able to communicate with each other on the private network. In many cases, communication must be permitted across multiple private VLANs to allow workers on different VLANs and in different zones to connect with each other. So that's worker to worker. Next one up to the right is worker to master and user to master communication. So your worker nodes and your authorized cluster users can communicate with the Kubernetes master securely over public network with TLS or over a private network through private service endpoint. So that's the workers talking to the master or your users. The third one in the CUBE networking world is worker communication to other public cloud services or on-prem networks. So this allows your worker nodes to securely communicate with other cloud services such as a container registry and or to an on-premises network. And finally, that fourth column over there, the one today is about external communication to apps that run on worker nodes. And this is Ingress, right? This is gonna be our topic for today. And this is where you allow public or private requests into the cluster as well to requests out of the cluster to that public network. So that was just a little feel of how all of those work together from that diagram. So always think it helps to start on the next slide with a bit of a definition. So this one comes from the CUBE.io docs and it says Ingress is a network service that provides HTTP and HTTPS routes from outside the cluster to services within the cluster. Gives services externally reachable URLs, load balances traffic, terminates SSL and TLSL and offers name-based virtual hosting. So all of those aspects we're gonna dive into and Dora will be demonstrating in just a moment. So let's go on to the next slide. A quick tour of terminology will make Ingress a little less confusing and I'll show a diagram in just a moment also to help. So we'll go through three pieces here. The first one is the Ingress resource. This is the YAML file you create to connect your apps with the Ingress system. In technical terms, an Ingress resource is a Kubernetes resource that defines rules for how to route incoming requests for your apps. It also specifies paths to your app service including subdomain. So that's the YAML file, that's one component. The next piece is the Ingress controller and it's also known as the application load balancer or ALB and this is the external load balancer that listens for incoming HTTPS or TCP requests. And then that ALB forwards requests to appropriate app pod according to rules in the Ingress resource. So that's sort of the two main ones. Now we have this third optional one here and it's a load balancer for a multiple zone. So if you're concerned about high availability, HA, you use multi-zone clusters and ones of that of course include a multi-zone load balancer to go with it. And so this might be something like Cloud Flare or similar service that puts IP addresses of your ALBs behind the same subdomain and then enables health checks on those IP addresses. So those are our components, the ALB, this is the IBM term and other cloud vendors also use the term ALB for the word Ingress controller which is doing the actual routing requests, sorry, actual work of routing requests from clients to apps. And the load balancer service that works with the ALB and actually make sure your ALB has an IP address and network connectivity. All right, let's go on to our next slide because I promised a diagram and now that you have those terms in a definition, I thought it's helpful for us to kind of look through this and see how all of this works for how a request gets to an app with Ingress. So number one there, that cloud, a user sends a request to your app by accessing your app's URL. This URL is the Ingress subdomain for your cluster appended with Ingress resource path for your exposed app, such as shown in that cloud icon. So that's the mycluster.containers, so on and so forth. Now let's look down to the number two in the lower left. A DNS system service resolves the subdomain in the URL to the portable public IP address of the load balancer that exposes the ALB. Once again, that's your Ingress controller of your cluster. You'll also see that becomes 169.soo and so forth. So three, they're in the middle. Based on that resolved IP address, the client sends the request to the Kubernetes load balancer service that exposes the ALB. So now we're moving to the right, a number four there within the worker node, the load balancer service, in IBM's case, that's Cloudflare, routes the request to the ALB or Ingress controller. Then in step five, the ALB checks if a routing rule for my app path in the cluster exists. If a matching rule is found, the request is proxied according to the rules that you defined in that Ingress YAML file to the pod where the app is deployed. The source IP address of the worker node, I'm sorry, let me, the source IP address of the package is changed to the IP address of the worker node where the app pod runs. And if you have multiple app instances that are deployed in the cluster, the ALB load balances the requests between the app pods. You see we have two nodes in this case, odd one and pun two. Finally, step six, we start seeing arrows going back in the, oh, I'm sorry, yeah, in the other direction is when the app returns a response packet that uses the IP address of the worker node where the ALB that forwarded the client request exists. The ALB then sends a response packet back to the client. So that's how all those steps get to, requests get your apps. Let's go on to the next slide. I wanna take a little bit of a detour here from all these definitions and instead talk about how node port and Ingress are compared. And you can see, certainly you're like, well, there's so many Kubernetes services, which one do I pick to an exposed app? And as you can see, Ingress gives you the most flexibility and strong security. So TLS, custom routing rules, multiple apps. That's why we're here talking to you today. It gives you the most flexibility. Yes, you can use some of the other service types, but Ingress really gives you the most for the bang for the buck. Now you might be saying, well, why is cluster IP in this table? It's just a blank column. Well, no, it doesn't give you any of the externally accessed items, but it's included here for this particular scenario. So let's walk through that. The cluster IP service provides an in cluster IP address that is accessible by other pods and services inside the cluster only. So no external IP address is created for the app. So that's very basic, simple thing there. All right, well, let's go ahead and step to our next slide where we can talk about the common use cases. And I'll walk through several of these so you can get a feel of how it all works. The first thing I wanna hone in there on the left is you can see that custom is a key idea here for Ingress. So I wanna dig into a couple of these. On the customization you can customize routing rules and SSL termination for multiple apps as well as customize tuning for multiple apps. Both of those are very critical as each app has its own unique flavors and needs. On the bottom right, I did wanna point out of course Ingress is for exposing apps. So this gives you flexibility to expose those apps inside or outside your cluster to the public, right? So not all of your apps necessarily want a hundred percent public facing. You might have some parts of the workload that's not appropriate. And then of course, being able to expose apps to a provided network. I have a little example up there in the middle about subdomains. I love this one because subdomains are very helpful when you're setting up maybe development and staging environments. So when you're deploying an app, perhaps just do a dev environment for a small team. Everyone can go to dev.example.net for example. Or when you're ready to go to staging, then you have another subdomain that your users or testers could hit on stage. One last thing is I mentioned TLS and SSL a couple of times. Ingress is where you're able to set up your own TLS certificates if you bring your own or if your cloud vendor provides them. This is how you set those up. On top of TLS certificates, which is something you actually create in, I'm sorry, as you set up with the Ingress controllers in cloud services like my IBM one, you can also add authentication to your apps with an authentication service like App ID or an Ingress integration with that. Another aspect of securing your workload is certificate management. So you might wanna explore some thing that will handle things like explorations, renewals and audits. So although Ingress does quite a bit of exposing your apps and helping you control access for request routing, it doesn't do everything. So let's move on to some helpful annotation groups in the next slide. These annotations show some of the typical ways you can customize your Ingress to make it work for you. You put these annotations in your Ingress resource YAML file. Watch out, these can get complex fast, but Dora will be showing in just a few minutes how to simplify these annotations into groups and make these make sense. So let's walk through a few of these. Of course, on the left, we have timeouts for connection. So whether that's keep alive, upstream events, maybe you need some buffers for your particular application. That's one of the things you can do with Ingress. As we've been talking along, you can enforce HTTPS, TLS or maybe just specific ports is another helpful set of annotations. This third one is kind of interesting, being able to limit requests and connections to your apps and this is all about performance and security. So let me take a moment to give you an example. So in a client request body size, this is one of the ones we look at, you may want a client to be able to upload large files. Okay, if you increasing the maximum request body size might impact the performance of your ALB and because this is gonna cause the connection may stay open for longer than you need. So that's where you tune that. And the final one there is just one of my favorite easy to understand ones that you can do with Ingress is, vanilla 404 pages don't look like much, it's just sort of a white page with some black text but you could redirect HTTP errors like 404 or 500 whatever to more fancy custom pages. Maybe you want an image on it. I think in IBM cloud, we have like some robots. You could include specific troubleshooting information and so forth. So just a few of annotation groups. And my last slide for me and the next one is talking a little bit about private load balancing. Ingress is your sweet spot for this and this is allowing you to allow having requests from outside the cluster come in but not putting it to the general public. So this is where you might wanna test access, look at request routing or just start to add more configurations to how you're setting up your networking. With that on our next slide, I'm handing it over to Dora and she is gonna jump in. Dora, do you wanna take it away? Thank you Vanessa. Now that you know why using in Ingress is important, I'm gonna go into a little more detail on what Ingress is and how to use it. So first I'll be going over what the community Ingress controller is. Then I'll be talking about the advantages of using a cloud managed Ingress controller. Then I'll be doing a live demo on configuring your Ingress for security and reliability and show you some troubleshooting tips as well. So what is the community Ingress controller? So before we dive into what the community Ingress controller is, I wanna do a quick review on what people usually mean when talking about Ingress. There are two main components when talking about Ingress. One, the Ingress resource, which is a Kubernetes API object where you can make configurations. And two, the Ingress controller that interprets the Ingress resource and routes any incoming HTTP traffic based on those interpretations. So let's say for example, that you work at an airlines company and you have two microservices that you want to expose the API to, services A and B. And you want to expose service A on any requests that come in on airlines.com slash flights and service B would respond to any requests that come in on airlines.com slash flight status. You have put those configurations in your Ingress resource so that every time your Ingress controller got a request, it would direct it to the corresponding service based on that path. So you might be wondering, what is the community Ingress controller and why does it matter which one I use? So while the Ingress resource API object comes out of the box as Kubernetes, the Ingress controller that interprets and implements the Ingress resource needs to be installed. So that's the user which one to use since there are multiple controllers to choose from. With the community Ingress controller specifically, its code base is open source and developed by the Kubernetes community and is based on Nginx. To see the code and learn more about it, I've linked the URL here below. So what are the advantages of a cloud managed Ingress controller? Setting up an Ingress controller can seem daunting. So using a cloud managed version of Ingress controller can make things a lot easier. Different cloud platforms provide varying levels of the support out of the box, but I wanted to just quickly touch on four things that we manage on the IBM cloud platform and is critical in general for anyone in setting up an Ingress controller, regardless of whether you plan to use a managed version or set it up yourself. So one, a cloud managed Ingress controller can mean having out of the box image management. This means that the responsibility of building a container image from the Ingress controller source code can shift to the cloud provider along with managing any vulnerability scans. Two, a cloud managed Ingress controller can mean having domain certificates managed by the cloud provider. The certificate is important for securing requests from a client with TLS. This means that the responsibility of creating a certificate and renewing it before it expires can shift away from the user. Three, a cloud managed Ingress controller provides a load balancer with a publicly accessible IP. Since pods and services in a cluster only have IPs that can be accessed within the cluster, this is vital to allow users to access your service. Four, a cloud managed Ingress controller can mean having your DNS records managed by the cloud provider. So after you have a publicly exposed IP to your services, you want a DNS service like Cloudflare to point a user to a readable domain name like airlines.com to it. So now I wanna talk about configuring for security and reliability. And to do that, I wanna go through three different parts. First, I'll show you how to set up a basic Ingress resource. Then I'll show some configurations to make your application more secure. And then lastly, I'll be showing you how to make some configurations for reliability. So what is entailed in setting up an Ingress resource? This includes three things. Starting on the right, first, I'll deploy an application that creates replicated containerized pods of your application. Then I'll expose those pods with a service which will identify a set of those pods using a label and allow your Ingress controller to load balance between any of your application's pods. Then we'll create an Ingress resource that maps a client request, matching your host and path to your corresponding service. In this instance, we'll be mapping any request that comes in on the flights path to any pods that correspond to service A. So I'll share my screen in a moment. Okay, so first I'll be showing you how to set up an Ingress resource. The first thing you'll want to do is create a deployment file for your application. So you can see here that I have a YAML file with a kind deployment. It has a label flights and I configured an image that uses the image that I created for my application. And you also see that I configured three replicas which means that I'll have three pods of my application. So I'll apply that YAML file to my cluster. And once I do that, then I can see all my relevant pods by listing the pods on the label I created with them. You can see here I'm getting my pods on the label flights and by doing Y I can see all the IPs related to it. So here I have three pods and I have three different IP addresses but those are still only accessed inside the cluster. So next I want to create a service for my application pods. Since pods can die, this is really important so that you have a stable service IP that your controller can find all your application pods for. So you can see here that I'm targeting port 80, I'm naming my service flight service and I'm exposing the deployment that I created earlier named flights. So now that it's exposed, I can describe that service and see two very important things. One, I can see that I have a cluster IP now which is an IP that can be accessed anywhere from within the cluster. And two, I'll see all these endpoints which correlate to these pods of the applications that I created earlier. Lastly, I'm gonna create an Ingress resource that maps any request that comes in on the flights path to my flight service. So you can see here that I have an Ingress YAML and I named it Airline Ingress. I use this Ingress class to let my Ingress controller know that I want this Ingress use in my Ingress controller and I specify a host. This host points to my load balancer IP and path that points to my flight service. So once I apply that YAML to my cluster, then I can, then I can see that I can curl against that path. So now I can describe that Ingress and I can see a couple of things here. So I have all the configurations that I specified before which has the host and the flights, the class I specified and I have all these events that are normal which means that my Ingress controller was able to pick up that Ingress resource successfully. So now when I curl against that path, I can see all my flights on that endpoint. So here I'm curling and I'm getting a response that I have flights that go from Austin Dallas, Austin, New York and finally, I also log which pod I made that request on. So going back to my slides, I want to talk now about how to configure for security and reliability. So now that I've set up a basic Ingress resource, I might start thinking about my security configurations. As you're exposing your API to the public, you'll always want to ensure those client requests coming in are secure. So I'll show some basic configurations on how to set up TLS termination so that your controller secures the connection from the client request to your load balancer using TLS. Then I'll show you how to force all incoming HTTP requests to use HTTPS. I'm going to show my screen again really quickly. Okay, so to enable TLS termination via HTTPS, a certificate is required. So now I'm securing my Ingress by specifying a secret that contains a TLS private key and secret. And by referencing the secret in my Ingress, I'm telling my Ingress controller to secure the client connections to the load balancer using TLS. So I already have a secret which has a certificate and private key in it. And it's located in my default namespace like my Ingress resource. And you can see also that I've already set this secret as my default SSL certificate and my Ingress controller deployment. So here you can see that I have a default SSL certificate and it's from the default namespace under the secret name that I just showed you. And what this does is if I don't configure TLS specifically in my Ingress resource, then the Ingress controller will default use this secret. It is recommended still that if you want to configure TLS for your Ingress resource, they should still specify it explicitly in your Ingress resource. Okay, let me show you how to do that really quickly. So now if I apply my Ingress resource, then I can curl on HTTPS against it. And if I do dash B, then I can see that it's using TLS termination against it. So you can see here that I'm using TLS and that is checking against the certificate that I specified earlier and I get back at 200. So what if the user doesn't use HTTPS but you wanna force them to? You can do this by redirecting them to HTTPS. And this is enabled out of the box of the Ingress controller if any TLS configurations are specified but it can also be enforced directly using an annotation. So let me specify the annotation. And you can see here that's SSL redirect and it's specified to true. So I will update my airlines Ingress file. And now if I try to curl against HTTPS, then you'll be able to see that I get a redirect. So you see here that I get a through way permanent redirect, but if I go and I'll use HTTPS, do TLS termination. So let me flip back to my slides really quickly. And I'm going to talk next about how to configure for reliability. So one thing that you can do if you wanna make your service experience to your client a little more reliable is to set retries on specific failures from within the Ingress controller. For example, let's say I got a 503 back from one of my upstream server pods but I don't wanna return that to the user on the first error. I might set a configuration called proxy next upstream to try making a request to multiple upstreams or pods before sending that error back to the user. It's important to note here though that this only works if nothing has already been sent back to a client yet. So if the request times out while sending back a response, it's impossible to fix because this is a configuration on the server level. It affects our request instead of on a specific path. I'll be making the current configuration in a config map. I'll be showing you this on a path that I purposely returned 503s from and you'll see how the Ingress controller low balances between my different pods as it gets back the failures. So let me share my screen really quickly. So to retry on failures, first I'm going to expose that path I told you about where I always returned back a 503 and this path is called bad flights. And I'll be configuring the retries through a config map that was passing as a flag when creating the Ingress controller. This config map like an Ingress resource also does configurations on the Ingress controller. So let me show you how we configured that config map to be used first. This is the same config map I showed you before where we specified that default secret but here you can see that I'm specifying a config map that a user can use to configure things and it's in the cube system namespace. So let me edit that config map and you'll see here that I already have two configurations that I've said before called proxy next upstream and proxy next upstream tries. So you'll see that on the proxy next upstream what this is doing is for every 503 it gets back it will try to make a request three times. So here I'm gonna change it to four just to show that I'm configuring it and then I will show you how now I'll be making four requests across all the applications. So first I'm going to log all the logs from my application and I can do that by doing log follow on the label flights return a couple of times so you know where the new logs come in from. And now I'm gonna curl on the bad flights path. I think I needed to update my Ingress file. So now I'm getting back a 503 and you can see here that I try to hit four different pods as I'm getting back a 503. So I'm going to switch back to my slides now and talk a little bit more about how to troubleshoot things if you ever hit any errors. So I'm gonna be talking about two different things. One, where to investigate when you're troubleshooting and two, how to troubleshoot HTTP status errors. So the three key areas to investigate when troubleshooting are one, your Ingress resource. You can describe it to see your events and endpoints. Two, your Ingress controller logs. And three, you can check the configuration file that the Ingress controller generates. So let me switch back to my screen so you can see that. And I'm gonna be showing you how to do all those things that I just mentioned. So first I want to describe my Ingress resource. This can oftentimes provide really valuable clues when something goes wrong. So here I'm just doing a quick check that everything I configure show up when I describe. Yes, my TLS configurations configured, my flights to my services configured, and everything looks normal in the events. And then the next thing that I wanna show you is the logs where you can see what could be going wrong. And this is a really helpful place to see where your Ingress controller could be telling you is happening. So I know that the logs here seem really overwhelming. So I'll be going over that in a bit on how to actually read them. And lastly, I wanna show you how to look at your comp file. So this is a configuration file that the Ingress controller generates to tell the end-to-next server where to direct your request. And you can do that by using this debug tool that the user, that the community created for you to debug with. And here I'm doing it on the deployment that my Ingress controller lives on and the host that I want to troubleshoot with. So here you can see that I have bad flights and there's a namespace that I set before the Ingress name, the service port. And what I'm doing here is I'm just doing a quick check to see if everything that I think should be configured is. Obviously this is our last resort thing as this does take some experience with end-to-next configurations to understand. But it is a really valuable tool to understand why your Ingress controller might not be behaving the way you expect it to. So to go back to my slides, I now wanna talk about how to troubleshoot your HTTP status errors. And two key areas that you want to look when troubleshooting is, one, did the Ingress controller know where to direct my request? And two, did my request reach my application? If the Ingress controller didn't know where to direct my request, it could be because the configuration is wrong. And if the request didn't reach my application, it could be that my pod was down or that the application isn't listening on that path or port. So I just wanna go over a quick example of an HTTP status error that we see is really common, 404. So I do wanna point out some key fields in these logs and these log formats are determined in that config map I showed you from before. And you can always add or remove from what you find helpful. These are end-to-next specific. So if you Google any of these terms, end-to-next and logs, you'll be able to understand better what they mean. So on the left here, we're seeing a 404. Where the controller did not know where to direct that request. And you can tell that by seeing that on the upstream address that on the left, that there are no endpoints. So the Ingress controller didn't know where to direct it. The upstream status is negative one. So it never got back a response from the upstream pods. And the request time is zero. So once the Ingress controller got it, it immediately sent it back to the client because it didn't know where to go. And on the right, you can see an example of where you get a 404, but it never reached my application. And the server was, and this reason was because the server was not listening on my path. So here, what I did to get this log was I logged on a path where my application was not listening on that path. And so it immediately sent back a 404. So the Ingress controller did know where to direct that request, but it, my application did not know what to do with it. You can see here with the upstream address that there's IPs for it. There's a status that's coming back from my upstream and there's an upstream response time. And all these things mean that it did show up in my application. And if I was logging those requests, I would be able to see that there and better able to understand why that's happening. Another example that we see that's really common that people struggle with is 502s. And the reason that this happened in the specific scenario was that my services target port was not the port my upstream server was listening on. So you can see here in the two factors that I told you about before that my Ingress controller did know where to direct my request, but my upstream never received that request. So you can see that here from the upstream address that I have IPs of the Ingress controller knew where to send that request to, I get an upstream status of 502, and but I get an upstream response time of zero, which means it never fully, my upstream never processed that request. And this one is a little bit trickier and it does take a little finesse to understanding them more, but I'm hoping that with these clues that I've given you, you can better be able to explore the world of Ingress and be able to configure things and troubleshoot them in a way that you can confidently move forward in that. So that's all I had today. If you wanna try this out and receive more information, go to this short URL. And I wanted to say on behalf of Vanessa and I, thank you for your time and we hope that you were able to learn something along the way. Thank you. Yeah, thank you very much and good luck. Okay, bye-bye. Oh, hey, it sounds like we are getting started for our Q and A. We've had a few that we already responded to that were just smaller questions. One thing we did wanna point out to you is we're writing a blog series. And so in that blog series, it's starting this afternoon will be the first one. So it's just sort of a recap of what you heard here. I know a lot of folks love hearing the sort of like seeing the text version of a verbal one. So we don't have that link up yet because we wanna do the session first. If you search on my name and medium, so it's my blogs on medium, you'll find it and we'll have a series of those coming out throughout the months and so you'll see that there. You can also probably find, I'll get a link maybe on LinkedIn later this afternoon. Dora, oh and I see we have, will you be able to share the deck? Yes, we will need to get that uploaded. I believe there is a way that it'll be linked from this page. So we'll get that uploaded as well so you can see that. Dora, did you see anything else we needed to respond to? I don't think so right now. For any of the similar questions, we can answer them in the chat. We'll continue sorting through those. So if you have any questions, we're happy to continue to answer them. And I'm checking, I see that someone was having trouble hitting that link. Checking that right now. Is it gonna be one of those days? It does look like the URL shortener is not working right now. We will post something back into the Q&A later to get you the direct link. As far as the GitHub link to try and solve all the errors, that'll be part of our blog series. So if you follow Vanessa's medium account, we'll be linking it there. And I'm trying to pull it up while we're looking. I see someone also asked about if JSON is supported in the Ingress Controller. I haven't personally tested creating an Ingress Resource using JSON, but it does seem like you can through the REST API at least. That's just not something I personally tried in using the KubeControl client in the CLI. Okay, hopefully that link is now available for you to try it out. Any other questions? So it was what would the possible next steps be? That link that I just posted under the IBM Biz Ingress, maybe I can just put it as a question instead. Try here. It might be more visible for folks instead of in the thread of view. I think, did we get all the questions? It looks like it. I did see someone ask if Ingress uses TLS 1.3. The Ingress Controller that I referenced in this presentation does use 1.3. And also, I really appreciate that Jell enjoyed the demo in this session. And we saw a lot of very nice words. We appreciate it. Kubernetes is our passion. And so we really enjoy sharing this talk with others, especially because we know Ingress is not easy. And any way to get a leg up is really essential. Thank you so much for your time, everyone. Yeah, thank you. Thank you. We really appreciate it. All right, well, let's wrap it here, I think, yeah. Okay. All right. Bye, all. Hope you enjoy the rest of the conference. Oh, I should once again say there are some very good sessions I posted in I think LinkedIn, a list of my favorite ones. Everything across the board from AI to 101 and Kubernetes talks, please go check those out. Those look good too. Enjoy the conference.