 Hi, I'm Don Shank, Director of Developer Experience with Red Hat Developers. In this video series, we'll examine how the Istio service mesh works with Kubernetes and your microservices. In this particular video, I'll demonstrate how Istio pool ejection allows you to remove and manage non-functioning pods from your Kubernetes-managed systems. For this demonstration, I'll be using the Red Hat Developer's Istio tutorial, which can be found at this URL. If you wish to follow along or run this demo yourself, the instructions can be found here. When you are managing your microservices with Kubernetes, such as is done with OpenShift, your capacity is auto-scaled up and down based on demand. Because microservices run in pods, you may have several microservice instances running in containers at a single endpoint, with Kubernetes handling the routing and load balancing. This is great. This is how it's supposed to be. Everything is good. As you know, microservices are small and ephemeral. The birth and death of a particular instance of a microservice in a pod is expected, and OpenShift and Kubernetes handle it quite well. Again, this is how it's supposed to be. And again, everything is good. But what happens when one particular microservice instance, a container, goes bad, either by crashing the 503 errors or more insidiously, taking too long to respond? That is, it wasn't auto-scaled out of existence. It failed or became slow all on its own. Did you try again? Do you reroute? Who defines taking too long and should we wait and try again later? How much later? When did this tiny microservice stuff all of a sudden get so complicated? Again, Istio comes to the rescue. Let's take a look at how the circuit breaker pattern with pool ejection works in Istio. Istio detects faulty instances or outliers. In the Istio lexicon, this is known as outlier detection. The strategy is to first detect an outlier container and then make it unavailable for a pre-configured duration or what's called a sleep window. While the container is in the sleep window, it is excluded from any routing or load bouncing. I've already started Minishift and have all the pods up and running so we can jump right into this demo. Here's a curl loop running against our demo system, and it's using both versions, one and two, of our recommendation service. Using out-of-the-box Kubernetes routing, which is round robin, the versions are being used for every other call. To enforce pool ejection, you first need to ensure you have a route rule in place. Let's use a 50-50 split of traffic. In addition, we'll use a command to increase the number of our containers in version two to two of them, two pods. Here's the command to scale to two pods. Now that's been scaled, let's take a look at the contents of the route rule and see that the traffic is in fact going to be split between the two pods. Now let's create that route rule and check the results. Looking at a loop of curl commands running against our demonstration system, we can see the route rule in effect. Even though we split 50-50 between the two microservices, because version two has two pods instead of one, you're going to see kind of a 67-33 split. Two-thirds are going to go to version two, and one third will go to version one. Okay, let's have some fun and introduce some misbehavior into one of our pods in order to get some errors. So to do that, here's what we're going to do. We're going to get a name of the pod, and then we're going to exec into it to get to the command line. And then we're going to run a curl against an endpoint that we created already in this demo program to cause the pod to misbehave. So first we get the command to get the list of the pods. So there are all the version two pods. Now we need to exec into it, which is this command. And now that I'm inside of it, I can curl localhost 8080 slash misbehave. And now any request just to root of the URI is going to return a 503 error. So that's done. Now we can go over to the other to the loop and see the results. Okay, we're going to start a loop and see what happens. And here you can see some 503 errors popping up from the misbehaving pod. That's not something you want in a production system. So what do we do? Well, with Istio's pull ejection, we can configure a simple circuit breaker and that will remove the misbehaving pod for a specified length of time. Okay, let's open the YAML file that has this pull ejection that I take a look at. So if you look near the bottom, you can see that once there's an error, we're going to spend 15 seconds in a sleep window. In other words, you're going to take that pod and remove it for 15 seconds before allowing it back into the pool. So let's go ahead and create that and take a look and see what happens. As we watch the loop of curl commands, we can see that the 503 errors will go away. And then if we wait long enough, we'll see it appear again once and then go away. This is pull ejection with the idea that your code is either self-healing or your monitoring system will remove the pod and add a new one to take its place. Combining Istio pull ejection with monitoring, you can start to build a framework where faulty containers are removed and replaced automatically, reducing or eliminating downtime and a dreaded pager. Since I mentioned monitoring, my next video will cover some of the monitoring and tracing available with Istio. In the meantime, you can visit the URL mentioned at the beginning of this video to run the Red Hat developer's Istio tutorial on your own and you can visit the URL shown here to download our free ebook. Thanks.