 All right, good morning everyone, Buenos Dias. I don't know about you, but Adam and I talk about Envoy and technologies built on top of Envoy to people all the time. And for the past two years, most of that talking has been to, you know, two-dimensional people on the other side of a Zoom meeting. And so to see this many people here in three-dimensional form is pretty exciting for us. So we're really glad you're here. We hope you're going to get some really useful information, perhaps a spark of inspiration out of this workshop. Envoy Proxy. And specifically, we're going to be talking about it in the context of a hands-on workshop that's going to give you the opportunity to get your hands dirty just a bit to get a taste of what using raw Envoy Proxy is like. So if you're interested in following along with the hands-on exercise, you know, get your laptop out. If you have Chrome installed on your laptop, we've had great results from Chrome with this particular workshop. Occasionally there can be issues with other browsers. So yeah, so absolutely get that fired up and we'll give you some instructions for getting connected to that in just a bit. Before we start into the actual hands-on workshop part, what we want to do is just lay a little bit of groundwork for who we are and what is Envoy. So my name is Jim Barton. I am a field engineer with solo.io. I've been here for a couple of years now, Adam. Adam Sayah. I'm a field engineer working with Jim and the team a couple of years too. We'll be taking questions here. Yep. So if you have questions over the course of this session, what we'd encourage you to do, given the size of the audience and the fact that a large percentage of it is remote, what we'd like to do is encourage you to get onto the session chat. You can do that by following this bit.ly link up here, bit.ly slash Envoy-chat. And we have one of our solo colleagues here somewhere, Alex. Over here, stage right. Hiding in the corner. Yep, in the corner. We keep him back there in the corner. And he will be happy to answer your questions. Just go to the Q&A tab. And I think that can work as kind of a quasi-chat mechanism if you have issues. We hope that you won't. But just in case Alex will be there to answer your questions. We also, there's a lot of content in the next 90 minutes. And so we hope to have some time for interactive questions. But let's save those for the end if you don't mind. Yeah. I also mentioned that if you guys have questions right now, like, I mean, during after this workshop, we are still available. We're going to be available hanging out outside the alley here or at our booths. Or, you know, you guys can find us on Slack. We're always happy to answer any questions. All right. So let's get started. So just a couple of slides to level set. So back in the mid-2010s, Lyft had something of a problem. They envisioned a future that involved collections of microservices that would be operating their organization rather than a set of static monoliths. You've heard this story before, right? And so you're probably living this story, some of you. And so they were looking at API gateway technology, reverse proxy technology to allow them to publish those services out to their consumers, both internal and external. And what they found as they looked around the API gateway landscape of that time was that most of them weren't really designed for the kind of dynamic environment with ephemeral compute that they faced at that time. And so they made the fateful decision that a lot of you have no doubt made, you know, do we build or do we buy? And so in their case, they decided to build. And so they built an internal technology. And with that technology, it went really well. By 2016, they had this thing called Envoy. They donated it to the CNCF in 2017. It gained popularity very rapidly. And in 2018, it graduated from as a CNCF project. So a lot of success very quickly. And since that time, its adoption has really exploded across the enterprise computing universe. So, you know, why is that? Why is it that Envoy has become so popular? Well, number one, it was built to be extraordinarily fast and scalable. It's implemented in C++. I know that we use it, we talk to customers every day who are managing hundreds of millions of transactions per day in mission critical applications all on top of fleets of Envoy proxies. And so very fast, very scalable. It's also very comprehensive. It works at both L4 and L7 being able to manage requests with those protocols. It's really built from the ground up to work in a dynamic services environment. So an environment like Kubernetes where services can be ephemeral, they're going up, they're going down, new things are being added. It's not a single static environment with a collection of, you know, 10 or 12 software monoliths behind it. It's actually designed for a dynamic environment. It's also built from the ground up to be dynamically configurable. So one of the things that Adam will talk about a bit later as we get deeper into the workshop is the fact that you can use a, it's not just, you know, traditional API gateway technology, traditional proxy technology is based on a static configuration file, right? So if you change that file, you want to change a route or something like that, often what you have to do is bounce your API gateway in order for those changes to take effect. Envoy was designed from day one to be unique in that it works with a dynamic control plane. So in other words, you can have a control plane that serves up configuration as situations change on the ground, as new policies are added, as new services come in or go away, you can have those policies served up dynamically to your proxy without having to bounce anything. So it's designed to be that way from day one. It's also designed to be a very extensible environment. So Envoy was built from day one to work on a filter chain architecture. So if you go to the Envoy proxy.io website you'll see an entire library of filters that you can use to apply various policies and routing techniques to your request. And we'll be looking at a number of those today as we go through this exercise. All right, it's also very observable. So out of the box you have integrations with things like Prometheus and Grafana, which we'll show you today. There's also an extensive collection of metrics that are published from Envoy. So why Envoy proxy for API infrastructure? There are clearly a number of alternatives that are out there, open source alternatives like Envoy that have been out there for years. Why Envoy? For me, it really boils down to four reasons. Number one, and I think the most important one from my standpoint is the large, vibrant community that supports and sponsors Envoy proxy. So you look at the list there on the screen that comes from Envoy proxy.io, just a sample of the companies who have adopted a remote Envoy proxy. We've already talked about the fact that it's extensible, it's dynamically configurable, but also I would say most importantly it is something that's battle tested. So Envoy's been chosen as the foundation for a lot of the cloud native application networking technologies that are growing very quickly right now. So technologies like Istio are based on Envoy proxy as the sidecar that manages traffic going in and out of your service pods. Also multiple API gateway technologies like GlueEdge from Solo and others are built on Envoy proxy as well. So at this point it is definitely battle tested. It is something that you can put into production environments and when it's properly configured it will work as expected. So what we're going to do today is a hands on workshop. If you'd like to follow along that would be great. I think you'll get a lot out of that process. If you want to just watch we'll actually be doing everything that's going to be part of the workshop up on the big screen here. So you should be able to follow along in that fashion as well. But what I'd like you to do if you are planning to follow along go ahead and navigate to that bit.ly link that's up there in red on the screen right now. bit.ly slash Envoy dash workshop that's going to take you to the Instruct platform. I don't know how many of you have used Instruct in the past. It's a really really effective platform for these kinds of large scale interactive sessions. In fact, I think there's some Instruct people in the audience just want to give you a shout out there. It's very solid technology. We use it for a lot of our workshops and big events like this. Envoy dash workshop. There will be a provision compute environment for you there. If you have a browser and internet connection you should be able to connect to it. We're going to go through five exercises today. We're going to start off with just some Envoy basics. Adam is going to cover a lot of the core principles of Envoy and give you a couple of exercises to sort of reinforce those. We're going to talk about HTTP connection manager which is one of the foundational filters in Envoy that manages routing of traffic to services. We're going to talk a bit about security and then we're going to talk about observability which when you're talking about managing collections of microservices observability becomes a critical component. Adam is going to discuss that and as well we're going to walk you through a live debugging exercise so hopefully all the principles you're going to learn in the next few minutes we're going to be able to take those and then solve a real common problem that you see when you're working with Envoy proxy and Adam is going to walk us through that as well. Now as a bonus for the first time anywhere in the world we're going to be offering a certification based on the results of this workshop it's an Envoy Foundation certification this is something we've done quite a bit with other open source technologies like Istio and EBPF where we offer certifications at different levels this is the first one of these we've done with Envoy we have a foundational certification that's available today feel free to take the exam we'll post a link to that at the end and we'll actually talk more about that as we go forward alright so hopefully those of you who are following along have had a chance to follow this link perhaps you've seen a screen like this at this point you shouldn't need to enter an email address or anything like that you should just be able to click the button to add this to your study room once you've done that you should see an interface that looks something like this so click through the my exclusives button and then you'll see the intro to Envoy proxy instruct track and simply click on the view track button there and that should take you to a screen something like this one and at this point we'll actually start working alongside you so I'm going to go ahead and start this track up alright do not watch the video that's on there that's me blabbering on for about 20 minutes about what is Envoy proxy instead you can listen to me blabber on in person so you'll see there's a there's a spinning green circle down there this basically means that the environment is being provisioned so definitely go ahead and get that process started it takes a couple of minutes for the environment to spin up and then once that's done we'll actually be ready to to get started with the interactive bits ok alright so as I said while we're waiting for everyone's environment to spin up we do have an exam we'll offer at the end of this we've been doing certifications now at solo.io for over a year on a number of popular open source technologies Istio and EBPF being two notable examples it so it'll be a 12 question exam requires greater than 80% in order to in order to pass you can retake the exam as often as you like however you have to use a different email address for every retake of the exam it's just a Google form and we need a unique email address to sort those and if you pass the test the badge will be issued within a few weeks by our very efficient marketing department and it is suitable for framing or putting on your LinkedIn profile whichever you prefer alright also if you are as passionate about application networking as we are at solo if you're interested in API gateway and service mesh technology we are always hiring and we would love to chat with you again as Adam said you can chat with us here after the session or we'll also be available in the booth for a while this afternoon at booth s10 in pavilion one I believe anyway one of the pavilions s10 first one yeah alright so let's see if we are ready to roll here we are ready so hopefully you're all looking at a screen similar to this with a little green start button and so press that start button and away we go I'll try to zoom in a bit I don't know if you guys see the terminal I think you're alright you guys in the back are you guys able to see the terminal here just a thumbs up awesome awesome thank you alright alright so thanks Jim for this amazing introduction to why actually we need Envoy during this first lab is mainly we're going to cover the basics of like just Envoy components and just technology itself I guess Envoy everyone is if you guys came here there's two possibilities either you are new to Envoy technology and want to just understand the basics of Envoy itself which is great or there's also maybe you are using Envoy technology that's embedded into for example Istio or Buedge or any other technology and it's kind of like abstracting the Envoy complexity from you and it's good to take this workshop to basically understand the basics and be able to debug and see basically the magic happening and how things are getting transformed into complex Envoy configuration so again the main the main goal of this workshop the one one foundation here is to give you the tools to understand basic Envoy configuration that will probably help you to kind of debug an environment or just have pointers to be able to ask the right questions and figure out instead of just being completely abstracted from your data plane alright so this workshop is based on we tried to simplify enough so it's going to be just like a Docker stack it's going to have a couple components one demo application one at Grafana and Prometheus installed already we're going to have Envoy and we're going to have like an X-Alt server we're going to use later during this workshop but something to notice here is that we are using static configuration and Jim mentioned it earlier either we configure it using dynamic configuration or we configure it using a static file for the simplicity today just to understand the basics we're going to be using static files but everything done here can be done first on communities obviously and also can be done using control plane like Istio and like glue edge and other do alright time to start so again you guys can follow up follow with me here like doing the same thing I'm doing on your laptop you can just follow along I think this environment is available for you for a couple hours so don't be in rush if at the end you're not able to finish you still have a couple hours to wrap up your work there and obviously take the exam alright I think we're ready to start here awesome just also a way to communicate between every single lab we're going to give couple minutes for you guys to be ready so I'll ask you guys just do a thumbs up just for me to know that you guys ready to continue so if you guys ready to start just please a thumbs up awesome awesome audience alright let's get rolling here first step is to just start the stack we're not going to talk about envoy configuration at this point it's mainly to get your demo stack example running and to do this what we're going to we're going to do here is we're going to set up the workshop so just go in the right directory and then again we're going to use a docker compose here so if you guys want to take a look at the docker compose configuration super straight forward we'll have envoy itself and it's going to be mounting some files and we don't have to get into details but it's basically mounting the basic configuration we're going to be modifying every time and restart right then it has an upstream service which is the demo service for us just to test the connection all the way to your back end and then we have an auth server that one going to be used for authentication later in another lab and we have the primates plus graphina stack for monitoring and also we're going to see that in the next lab first step here just starting the stack that's pretty much it for that we're going to do a docker compose up and it's probably going to take a couple seconds here to pull all the images and start your your stack there's sometimes maybe if you guys have trouble with internet I mean either use your data or keep refreshing if you refresh the page you won't lose your session it's fine you know if you see something stuck refresh the page try not do it often but if you really see that refresh would be helpful please do okay so now creating the services we see that everything is done here so we have our services running do a docker ps and we see a couple components here ready for us to demo so one thing the first step here what we did is that we create a couple components with Envoy itself we used a basic configuration having only the admin interface now what's the admin interface it's one of it's basically the way to interact with Envoy from an administrative standpoint where you can see the configuration if you want to see the admin interface configuration click on Envoy config there on top if you click there you'll see if you click on the file here that is the bootstrap configuration we use for Envoy now and we see to configure the admin interface we just have to set up a port and an address and that's it now the admin interface is one of the really important component for you to understand if you want to debug Envoy now the admin interface itself sometimes is masked like for example Istio and other technologies we don't do it in glue edge but there's some technologies that mask the admin interface from a user because it's risky using an admin interface you can drain listeners you can fail health checks you can really interact with Envoy itself in a dynamic manner that can be hurtful to your system so I would really advise to understand how the admin interface work but not expose it externally something internal to just check what's going on so for that there's a couple things to really be aware in the admin interface and we're gonna check them later but one that I like looking at every time I start the admin interface on Envoy is the config dump the config dump is basically the state of Envoy at this time especially if using dynamic configuration, configuration pushed by a server you want to be able to just go to Envoy and see what Envoy is seeing, what's the configuration Envoy is having so config dump is one of the most important things you're gonna see the admin interface another thing that I would advise taking a look at is clusters and listeners but we are not there yet since we haven't created any listener or cluster but we're gonna see that in a second alright so again, admin interface super important, we're gonna be using it a couple of times in the in the first labs and then we're gonna try to switch to more like a Prometheus and Grafana stack for metrics later that reminds me that also the link to stats is really important too you're gonna see that here either stats or stats Prometheus that just formatted for Prometheus but it will allow you to give you an idea of what's going on why a request is failing and things like that we're gonna see that later so admin interface is there installed so now next step what's one of the most important components in Envoy? I would say a listener alright, a listener in Envoy I'm gonna just switch to this a listener in Envoy is basically a way for us to open a port, simplified easily a way for us to open a port in Envoy and be able to interact, send request there, okay, and Envoy will intercept the request there and do something with it either reroute it or terminate it and so on so an Envoy listener is basically a network address open on your Envoy let's create one listener for Envoy here to do this, what we're gonna do is just use git checkout only have all the configuration it's gonna get messy if we have to do it manually so we use git to kind of switch configuration from a lab to another in this case if you go in the admin Envoy configuration we see here that we have only the admin interface configuration now let's do a git checkout of lab one two do this if we go back to the Envoy configuration we see a little bit more content here and what's important to understand here is that we added one static listener that's opening a port 8000 on Envoy now a listener always needs a filter and it's a good time for us to introduce this second important concept in Envoy is a filter so we started with the listener is the one taking the connection and is the one we talk to as a client then there's a filter a filter is the one processing that connection and that request there's two types of filters there's the TCP filters we deal with an L4 type level where we can just do a pass through TCP or there's an L7 type filter that will be able to attach to the HTTP connection manager we're going to be talking about later in this example we're just using a simple filter called direct response operating at the L4 level we see here it's a TCP it's a network it's a network filter and what it does here is to return hell word to any connection on that specific port so once we got this configuration we'll have to restart Envoy quick just to pick up this and to do this we're going to do restarts alright there you go do a restart here that's good now Envoy is picking up the config again if you want to check the config dump you should be able to see the listener added to the Envoy configuration itself not only in the static file and if you do it's just a curl here we get hell word back right so at this stage we have only a listener and we have a filter it's taking communication and returning hell word alright now time to ok so I want to talk about something before something really important Envoy and Jim mentioned earlier is be able to observe what's going on and Envoy has a ton of metrics really valuable to understand what's happening so during this workshop we're going to be pointing to some metrics that we think are variable but we invite you to check all the list to see exactly what you're looking for so one thing to do here if you go back to the admin interface before if you guys clicked on listener there were nothing here if you click this time we'll see that we added one listener listening on 00008000 ok and this is a really important debug tip here if you see Envoy not receiving any configuration just like this a connection failure for example the server is not there at all first thing to verify the server is created you need to check first if basically the door is open right now we talked about listeners we talked about filters a third really important concept in Envoy is Envoy clusters a cluster is a grouping of addresses to point to a certain destination so you'll think about the cluster as a representation of a backend a backend service for example either it can be your local community service or it can be pointing to an external service somehow but basically just a collection of IPs that Envoy can use to interact with the backend system in this case earlier we just terminated the request right there but this time we're going to route traffic to a backend to do this we're going to check out this time Lab 1.2 Lab 1.3 and if you go back to the Envoy configuration this time something is added here which is a cluster a cluster definition is pretty straightforward here we're just pointing to a DNS address AppStream8080 AppStream again is a reminder just a demo application we installed with our Docker Compose stack okay so here we have a backend service now that we have this name and this cluster called AppStream defined let's define the routing to that and I mentioned earlier using TCP filters you can do a TCP proxy routing all TCP to a backend in this case we're using the TCP proxy filter instead of the direct response filter we had previously right so here we're doing a pass-through all the way back to the backend and if you see we're just referencing the cluster AppStream that we defined which is our demo application so now that we have the configuration again let's restart Envoy to pick up this and let's make a call here we are going to make a call to our backend system there you go so here we're getting hello back, hello back is coming from our backend system raw TCP going through Envoy to our backend system that return hello back so that's really an important concept to understand here the concept of clusters because ultimately if you're dealing with Istio or you're dealing with any technology behind the scene a control plane is transforming a set of like Kubernetes service for example to a set of APIs at Envoy in the sense so if you see that your Envoy is not able to interact with let's say a sidecar Envoy sidecar not able to interact with a certain service first thing to do is to verify if the cluster is defined there does Envoy know about that specific service so something to keep in mind alright just I want to show you certain diagrams if you want to just look at it you can just expand this on the side and here is kind of the example that what we did during the lab one what we did we have a client talking to a listener that we just opened on the port 8000 is going to talk to a set of filters it can be one or many it's a filter chain so it can chain multiple filters to do multiple things one on top of the other and then route to your back-end service alright so and one last important thing here again if you have a cluster defined you need you can check the IPs that actually the endpoint is Envoy is trying to use if you go back to the admin interface just go back here with the arrow and this time going under clusters so we can find it here up top second one alright clusters now we see that we have an upstream cluster defined but it's really important here we see the endpoint also we see this 172 1805 defined this is our IP that we're trying to reach this is our back-end service something really important to notice like a even if you have the cluster defined try to check the cluster path in the admin interface or the metric try to see if the endpoints are defined ok have the upstream cluster but do I have an endpoint do I have an IP defined that talks to this service ok I have an IP here again check if the health check is passing is this back-end endpoint healthy right this is important to note alright that was the fundamentals of couple components really important in Envoy so again thumbs up if you guys ready to continue sweet yeah ok cp0.9 news just for curl we're dealing with raw TCP and somehow cur doesn't handle it well that's it if when we're going to switch to the HTTP connection manager we won't deal with this situation anymore alright awesome back to you man very good thank you Adam so that was a great introduction to the basics and the core concepts of Envoy proxy if you haven't already go ahead and hit the green check button at the end of the first exercise that will hopefully say everything is correct and will load up the next challenge which is we're going to talk about HTTP connection manager so Adam worked with a couple of examples with TCP and he also in some of his examples used direct responses straight from the proxy there was not even a back-end service that was handling the response that's not the most common scenario with our customers what we see using Envoy proxy very commonly what we see is people who are using HTTP connection manager to be able to process requests at layer 7 to be able to apply let's say transformations to apply policies based on headers and that sort of thing and so you need the L7 access in order to do that so Envoy has L7 capability and the core of that is something called HTTP connection manager and so that's what we're going to explore in this second exercise so what we're going to do here let's go ahead and move into the correct directory and check out Lab 2-1 and so I just want to show you first graphically what we're going to be doing this time so it's a little different right we still have a client we still have a listener at HTTP 8000 now we're going to be going through the HTTP connection manager is the only filter in our filter chain it's going to route to our upstream service which is simply going to give us back a hello world example sorry hello world response so let's take a look at let's take a look at the configuration so here is what we've added here so you can see the HTTP connection manager filter specified and here the real key bits right here in this section you can see we have identified a virtual host here that we're calling upstream it is listening on domain hello.onvoyproxy.io so if it sees that in the host that's going to be like passing the first gate for this filter to fire you can see we also have a route defined so we're basically saying any request that has a prefix begins with hello we want to capture that request and we want to take that request and then we want to route it to this cluster called upstream which is basically the little hello world app that's written in python that's running behind the scenes okay so that's that's pretty much all we're doing for this first example so let's go back let's go back here we've already applied that config now what do we need to do because we're not using a dynamic control plane we're just using a static configuration file every time we make a change like this we have to restart onvoy to pick up the new config as we said before Adam's going to talk a bit about control planes more in the end and you'll get some sense of the value that they can add here so you'll notice now our current command is we're specifying requirement before because this is not TCP this is an HTTP connection you can see it's going to local host 8000 with a path of hello you can also see we're specifying the host header hello.onvoyproxy.io and when we do that we get a hello world back just as expected now just to show you kind of a counter example let's say we didn't have the host proxy specified properly what do you expect in that case the filter is not going to match and you're going to see a what? 404 excellent well you didn't actually see it but if we do this then you'll actually see there's a 404 if we don't specify the host header correctly okay alright now as we've said before really a core bit of value from onvoy is observability and metrics and so not surprisingly you're going to see that with this example as well so Adam showed you just briefly the admin interface and so you can go in here and hit the stats interface and you can actually see this whole raft of metrics here of course you can also we can just access this via curl and then just filter out the bits that we want to see and so in this case you can see here is a metric that indicates you know there's been one request that's been completed successfully with this since onvoy has restarted if we go off and let's say we run the we run it again right we run the curl command again the correct one and we and then we take a look at that metric again not surprisingly you can see that metric has been updated to two from one so so the definitely as you as you get deeper into onvoy if you haven't been there before we definitely encourage you to spend some time encouraging understanding this vast metrics library that onvoy publishes okay there's also a Prometheus instance as Adam talked about that is that's included with this example we're not going to go into this deeply right now Adam's going to cover that in lab four on observability but you can see if we wanted to we could go take a look at this same metric you know graft out using using our little Prometheus setup that we have in this example as well okay so let's let's run our check on that and wait oh I said haha let's not run our check on that because we're not done yet alright I got to the bottom there I thought it was at the end of the lab alright so we're going to go through one other example with HTTP connection manager so what we showed you first was just a very simple you know hey here's a request route it to this back-end server you know it works everyone's happy now we're going to deal with something that's a slightly more real world situation which is let's say our back-end service is unreliable right in fact let's say it's really unreliable in our case we're going to simulate a service that actually fails 80% of the time hopefully you don't have any of those but if you do you can you can appreciate what we're going to do here so what we're going to do if you look at it from a graphical standpoint it's pretty much exactly like the previous example except on the connection from the upstream cluster to the upstream service we're going to modify the Envoy configuration to add in some retry capabilities this is going to be very basic there's a whole a library of enhancements you can provide but we're just going to show you the basic configuration of how to do these retries okay so let's let's dig into that so first of all what I want to show you is the way we're going to trigger this unreliable behavior so we have a little Python server back there that's operating as our upstream and so our Python server pretty stupidly if it sees the header unreliable set to true then it's going to fail 80% of the time hopefully your app developers you work with are a little smarter than that so let's go ahead and try this out so if we run this curl here with unreliable set to true you'll see that 80% of the time you're going to get back a 500 internal server error now some of you will actually see that request succeed if you do simply retry it will most likely fail on the next occurrence alright so what can we what can we do about that well let's take a look at how we have how we've modified the configuration here let me get a little more screen real estate and so we haven't done anything yet because we haven't we haven't brought in the new configuration so let's let's restart our envoy proxy I think you missed the checkout did I not do the checkout yep alright so where was that must be back up here somewhere somewhere there it is yep skipped right over that alright so you want to be sure before you do that you want to check out lab 2-2 which will be the retry part of this lab alright so once you switch over to that now you go back to the envoy config and you'll see there is a change and the change is right here so within the context of this HTTP connection manager config we have added this retry policy right here you're noticing we're only retrying on certain conditions so envoy has a number of conditions that you can specify you can also aggregate them into a list but there are only certain connections that we're going to retry on in this case we're only going to retry on 500 class internal server errors and then we're also going to limit the number of retries so we're only going to allow 10 retries if we see more failures than that then we're going to punt there's a whole host of other things you can specify here as well but again this is just a basic a basic configuration to show you the foundational idea alright so now if we restart envoy we should see some good things happen so note this time we called it in exactly the same way we did before we passed in the unreliable flag but this time what we see is that is that it returns true it returns a 200 ok now the question is what happened there right did our retry mechanism actually work and so to confirm that what we're going to do is we're going to run we're going to run this command right here to take a look at the logs of our upstream service ok now if you look very carefully what you can see this last on this last time that we executed the service you can see the first two times actually it looks like it was just these these two here maybe just do like some spaces and we run the curl yeah I think that's a good idea so let's put some space in here you can see it more clearly if we run this again alright so you can see the request returned true and that time of course it worked the first time alright well one out of every five times that's going to happen so let's try again ok here we go and so you can see what happened on the last time the first try we actually got it on the first try and then the second try we should go to Vegas next week and try our luck so you can see the first attempt failed with the 500 but then completely without client knowledge or the Envoy proxy intervened and said hey I've got this retry configuration I'm going to retry that and on the second occurrence it succeeded and so it returned a successful successful 200 code to the downstream client ok hopefully that makes perfect sense now you will not be surprised excuse me you'll not be surprised to learn that there are also metrics associated with these retry capabilities as well so if we take a look at that um here's a set of metrics that give us a lot of useful information about what we're seeing you can see first of all at the bottom of that list we see that there's a total of 5 requests that have been issued to this upstream since the Envoy since the Envoy proxy was restarted you can also see that of those of those 5 2 of them 2 of them succeeded or sorry 3 that's right we ran it right we we ran it the first time then we re ran it twice to show the logs and so there are 3 of those that succeeded just as expected you can also see that there were a number of retry so right here you can see the count of the number of 500 responses that we got and the number of retry so each one of these elements this HDP connection manager filter is emitting metrics that are associated with all of these capabilities and you can see we're seeing those here both in the aggregate as well as broken down by particular error codes that are being returned alright so that's just an introduction to some of the things you can do with HDP connection manager we're going to be using it as we go forward in the next exercise as well so if you haven't already hit the check button at the bottom of your screen and let's go forward to the next lab I think we can do a quick check who is ready for the next one thank you Adam alright cool great and so we're going to talk a little bit about securing traffic using Envoy and we're going to discuss two separate topics here one is really really basic something that every one of you no doubt has probably done which is you're going to set up TLS configuration to encrypt the communication channel between the downstream client and the proxy so we're going to take a look at that and then we're also going to investigate a second Envoy filter which is the X off Z filter so we're going to add in a little custom authorization service into the mix and we're going to configure Envoy to properly authorize our request before we actually go off and hit the upstream service so we'll show you how to do that with Envoy as well okay so let's go to the correct directory let's check out the configuration for this exercise and so here is what we're going to do in graphic form we're again going to have a downstream client we're going to be talking to a different listener this time we're going to be talking to an HTTPS listener that's listening on a different port 8443 this communication channel is going to be encrypted and then it's going to delegate similarly to what we did before in this first example through HTTP connection manager which is going to route to the upstream cluster and then ultimately the service that backs that so the way we're going to do that is we're going to configure something called the downstream TLS context and we will provide some certificate information in context of that let's look at that over here so you can see one change we made was simply changing from port 8000 to 8443 we didn't have to do that it's simply a convention based on the traditional values for HTTP versus HTTPS port values and then you can see we've added this bit of configuration this transport socket stands into our envoy config everything else is the same so you can see you can see we're using this downstream TLS context we're actually configuring this to require a client certificate to be presented and then we're specifying the certificate chain that we're going to be using and that's really all there is to it we're using self-signed certs in this lab exercise so that will change things just a bit from what you'd see in the real world but you'll definitely get the idea okay so because we've changed configuration we again need to restart envoy and now what we're going to do let's test the same let's test what we had before and just confirm that that no longer works so if we try to hit HTTP listener at port 8000 not surprisingly that's going to fail because that listener is no longer active let's say we just correct the port number but we don't correct the protocol to HTTPS again you can see that's going to fail because the listener is expecting HTTPS and it's not getting that and then finally let's get everything right now you get back from curl you get a warning hey this is a self-signed certificate are you sure you're okay with this and of course we are so we're going to add in the minus K flag to basically say ignore that warning and now you can see we are have a properly secured connection to our service to our proxy that's delegating to the service and returning the value we all expect okay so that is a very quick tour of configuring TLS in envoy proxy all right now let's take a look at something that's a little more complex and this is the more probably the most complex flow we're going to use in this in this exercise and what we're doing is we're adding another element to our filter chain here and the element we're going to add is the X to Z the authorization filter so this is a standard envoy filter that you can you know pick up open source into your envoy instance and what that's going to do is it's going to give you the ability before you actually process the request you'll be able to delegate first to an off cluster and the off cluster will simply take your request and based on whatever criteria you specify will give you a thumbs up or a thumbs down this request is acceptable or this request isn't acceptable by default you'll get back a 403 if it is unacceptable and if it is acceptable it's simply then going to delegate on to the second step in the process which is to your upstream cluster which is exactly the same one that we've been using before okay I just want to mention something quick if you guys want to take a look at the code all this example we have it on public github so you can take a look at the X off code that we use it's super basic it's all right here if you drill down into this you can tell Adam and I spent we spent a ton of time working on this we were actually thinking about closed sourcing this code but instead we'll put it out there for you this is our sophisticated off service it's just a simple python server that looks for hey is there a header called authorization in the request and does it have the value of workshop if it does up if it doesn't thumbs down okay simple stuff and our upstream service is equally sophisticated as this alright okay so let's check out the let's check out the new configuration here and what you're going to see when you look at the differences notice what we've done here in the context in the context of our filters we have added in this X off Z filter and you can see what it's going to do is it is going to delegate to this cluster called off right we were very creative with our naming conventions either and you can see we've also added now an off cluster with the location where it can be reached so that that will be the first step in that process is for the request comes in the X off Z filter is going to pick that up it's going to send it off to the off service all service is going to give a thumbs up or a thumbs down if it gives a thumbs down will return a 403 forbidden back to the client if it gives a thumbs up then we will delegate the request off to our upstream server and allow it to do whatever processing it wants to do okay so again for the 6th time let's restart on void and then let's test out a couple of options here to see if this is working as expected so at first we're going to test this using exactly the same call sequence we did before and what do you see so we did not supply the authorization header we don't supply the authorization header what happens we get back the default 403 forbidden response so that's working as expected let's try something slightly different let's say we provide a bad authentication key is that going to work yep just like expected it gives us back a 403 forbidden code and then finally let's do the let's offer the correct header with the correct value and now you can see we get back a 200 okay so it's working just as expected all right so that's that's the basics of how to use the out of the box eXtAuthZ filter that comes with Envoy all right now you're not going to be surprised to learn that like every other filter that comes with Envoy the eXtAuthZ filter publishes a nice collection of metrics that you can use to understand what it's actually doing so let's take a look at that take a look at some of the metrics and then let's see if we can interpret what's going on there so what you'll see is that here here at the top you'll notice that the upstream for the eXtAuthZ that delegates to upstream there was one request that was okay right that was the last one but then there were two requests here that were denied okay so that's all good you can see also similarly here at the ingress level we also so we got metrics that were published at the individual upstream level we also get metrics that are published at the ingress level so if we had multiple services running here that were fronted by the same filter we would see this broken down by different upstreams and then aggregated here into this HTTP ingress set of metrics okay so you can see that again we have two that were denied from the ingress point and one that was okay you can also see at the bottom at the listener level you can also see a breakdown by response code so you can see right there was one that was a 200 class response and two that were 403 class responses and a total number of requests that were completed for three so you can see how this produces a whole lot of metrics that are really useful for you observing what's going on in the system Adam is going to talk about that in just a minute in more detail okay so at this point we are done with lab 3 hope that was helpful in understanding some of the on voice security mechanisms now we're going to hit the check button and this time it's going to say everything is good and so that's going to move us over to lab 4 and I'll turn it over to Adam are we good with a thumbs up to move on to lab 4 awesome you guys been killing it today that's great I'll be asking you a quick question here if something goes wrong in your environments what's the first thing you have to look at metrics logs things like that and we saw during this workshop that Envoy just produces a ton of data that is really useful to debug and see what's going on right so during this lab what we're going to do we're just going to take a look at the truth is already configured to basically have an aggregate all what we saw previously so if you go back to if you could start this workshop and we go under Grafana Grafana here you can just you know use I don't know if you guys are familiar with the Grafana basically the default credentials admin admin so use admin admin you're going to be able to log in here just skip this you don't have to to put in your password and we're going to take a look at what we have here just try to take the 15 last seconds I'm going to be useful for us just to see what's going on and also in term of upstreams try to take the upstream one which is our fine like the demo application right so here this is just an aggregate right using Grafana if you guys are familiar with you can always create your own panels use all the metrics we used to have like define you know create your alerting if your system is down so on so something super important is if something is going down check your metrics that's the first thing to do like why I mean what's happening there it's going to get you like some key data there and then after that obviously you're going to check something else which is locked and we're going to talk about this in a second so here we have just a collection of some metrics that are important that can be used as a framework or you guys can do your own create your own dashboard we saw that we have all the night because we added the alt filter in the third lab we saw that we have couple of the night there that because of that for regarding the upstream the cluster upstream cluster membership we see that everything is healthy everything has been good for the couple past minutes 15 seconds 15 minutes sorry we see the success rate is is good the only time when something didn't happen well is because we did this unreliable header and so on so connection timeout we see there was no connection timeout which is which is awesome we see some 400 mainly because of our 403 XR denied there and and so on so really valuable to understand the core concept of like dealing with metrics if you debug your environment if you set up Envoy you need to set up your metrics either directly with Envoy or if you're using like something I control plane and so on if using like like an Istio or glue edge so on to always have some sort of automatically pushing injecting this metric collection to your primates sometimes it even comes with a primates server with you all right so that was super quick for just Grafana we're going to go back to this like in couple seconds for a debug exercise but another thing that is I think even more important is logs right something goes down something doesn't work okay metrics will alert we're going to receive a 3am notification that everyone loves then we're going to have to check logs right so let's see what we can do to add logs to Envoy to do this we're going to use something called access logs so access logs are really useful piece of data that is produced by Envoy on each request if Envoy intercept the request and something happened either good or bad like if everything's successful you're going to have a log saying all good 200 no problems there now if something goes south what do you have to do check also the access logs and check to see what error which error code was was provided and so on so let's just set up the access logs quick here to do this let's check out the lab 4-2 and oh sorry I have to go in the Envoy folder check out lab 4-2 and let's take a look at the Envoy configuration this time to add access logs we added that on our tp connection manager it's basically just one line ok if you add the one line type thing it's just using default format that I'm providing here in the lab that you guys can check that one here I think it's better for me to squeeze it this way but you can configure your access logs like you want you can add specific data like you can print specific headers that you care about headers coming from your client and things like that you can print what's the client IP address for example things that are very important which upstream where the traffic should be going and what kind of answer you guys got from the upstream so all this is super valuable to have in your access logs now one key thing if I have to advise for only one thing to watch in your access logs is the response flags ok response flags and envoys are super super important if something goes if everything works fine you don't have any response flag gonna be a dash everything's fine now if everything is not working fine you're gonna have a response flag which is a two letter or more code and this code will tell you why at least give you a good hint on why this request failed you're gonna grab that response flag go on google go on the envoy documentation match that to what's the core what's why actually the explanation around that specific response flag and from there it's gonna give you the good hint to continue your debugging journey ok so again response flags so let's just restart the configuration here to pick up our access logs config and now let's you know make a couple calls and see how things are printed let's do this alright so I just made five requests here to my backend again but this time I have access logs on so if I go to if I check the logs of envoy this time I would see my access logs and this is if you guys see the five last lines here this is access logs that can be collected right you guys can collect that using your log collection metric you know you have fielding and so on aggregating this data sending it to any anywhere you guys persisting your logs and you guys can query on it and so on but this is really important here where you see that using the basic access log format we see that we're making a call on slash hello we got a 200 it was a 11 htp11 connection we see basically which host was using we use and which fstream ip we try to target something important here is that the response flag is not provided is not implemented we should see a dash awesome dash means good alright so this is basically a quick intro to monitoring an envoy this is super important to understand and if you have to remember one thing from that response flags alright thumbs up if you guys are ready for the last lab which is a super super small debugging exercise but just to kind of what we try to do during this workshop is just to you know take you through this journey of understanding the basics understanding metrics and understanding and now we try to try to put everything together in a quick exercise to see if something doesn't work well how to debug it alright click on check once ready and let's start debugging the fun part of a software engineer life okay it's going to take a couple seconds because I have to set up because Adam's script is doing something evil behind the scenes that we're not going to tell you about yet so let's mess up your environment alright let's start um great so I'm going to go back to the folder I've always been using for this workshop and now that just makes some calls right I mean expecting everything works well no we have problems connecting to envoy envoy is returning something but I don't know what it is he's saying that looks really bad no healthy upstream well this time upstream doesn't mean our cluster upstream just means we can do anything with your request okay well this is not a fun time to you know with our system so what can we do first thing again as I mentioned um just kind of an extended version of this debug exercise first thing to do if something doesn't happen I know that at least envoy is returning something for me in my mind if envoy is returning something I was able to connect to envoy so listener is there I'm moving the listener possibility from my equation now if I have the listener defined and taking traffic and something is happening so either something is happening within envoy or something is happening on the backend side of it so now let's figure out if it's between envoy or in the backend what you're going to do again check the logs we have axe logs defined already so this time uh oh there's a code there yeah we have a bad thing going on we should not see any code and we see u h so if you see that I'm sending a link here to a documentation so hopefully it doesn't mess up everything but I mean it's super super small I'll see if you guys go on the envoy documentation uh now that we have u h we can check what it means oh sorry there you go it give us a bit more explanation at least you know this is super basic one right you guys but you guys can also see the more complicated ones and that's the one that's really really important hint for us so no healthy upstream hosts in upstream cluster in addition to a 503 response code okay well no healthy upstream hosts so there is something with endpoints like there is something going on that I can connect to my backend I'm gonna go back here and take a look at well I'm gonna start taking a look at Grafana first to see what's happening I'm gonna check my well 15 minutes should be enough and I'm gonna take a look at the upstream one well this is super maxed out so I can see everything oh okay so I'm looking at the upstream one I can scroll down here I'm looking at the upstream cluster and something is telling me that the cluster membership just deep down right the number of healthy members just crashed no healthy members we can't connect to the backend so let's also take a look at the admin interface if you guys remember when we talked about clusters we said when we have clusters we have clusters definition plus endpoints which is where we can connect to and if we look at the upstream one nothing is there there's no IPs so basically Envoy doesn't know about your backend it knows by definition that I need to talk to something called upstream but I don't have the address I just have the name, no address kind of situation so everything pointing to the fact that Envoy doesn't have a way to connect to my backend and that can be because of mainly one thing the service is not there or the health check is failing and health check all endpoints are failing in this case we have only one IP so when Envoy evicts this host there's no way to connect to it at all so let me just quick check the list of services I have running oh this is my backend service super basic but I think what's important to note here during this quick debugging journey is that Envoy will give you some hints to take a look somewhere else first I'm going to tell you hey there's no healthy hosts I'm having a trouble there connecting to something you're going to go check your response flag it tells you a little bit more maybe there is no endpoints and then you're going to go check your and usually I mean if you can connect to the admin interface that's fine if not you're going to connect to Grafana check if there is data there if there is endpoints if there is configuration we saw in Grafana there is like a cluster membership dip so we saw that something happened which is bad from there you can have a hint on what's the next step in our scenario we just see that there is no backend service well today is just to start back our upstream and that's all if we make a call again amazing right so super quick debug exercise go you through the things yeah that was that was so tough one right oh man that kept me from sleeping all night exactly yeah all right awesome I want to just start talking about a really important concept here we saw today the configuration of Envoy and we can open it again and see how it looks like right today we did something super simple we just like had an HCM matching on a host directing traffic to an upstream backend service and having X Auth and TLS on it that's super straightforward but we see that it's super complicated configuration and honestly we had on file and dealing with this it's no go you guys can do that it's and that's one service imagine about like you know 200 so this is why Envoy and we mentioned it earlier Envoy has a mechanism to push data through what we call an XDS server okay which is basically your control plane server it can be like just a server serving Envoy configuration Envoy connects to fetch the configuration automatically updates and be able to route traffic right so it's really important to have a control plane and we know there is a lot of advantages to have you know to use a control plane it's pretty straightforward if you want to use Envoy for a simple exposing of one service it's fine to use a file I would say now if you are doing you know enterprise networking and application exposure and stuff like that you need you care about a couple things and I'll just you know just a summary here of things that you really care about is mainly I want to simplify the interaction I want to have a way to just create I mean I'm a user what matter for me just I want to I want to tell the system hey expose this service called upstream through like either a cloud native way of doing it using you know CRDs in communities and I create my CR automatically control plane will pick up the configuration and modify my data plane which is Envoy to do this so it's really important that simplify the way to simplify configuration and as part of the simplification another point is mainly configuration validation right we were doing modifications here I mean we helped you guys using git to kind of check how the configurations imagine if you guys had to write this with YAML and mess up one tab right and restart and somehow just push your configuration that would be a mess bringing all your infrastructure down for like real real silly mistake so having configuration validation is a key component that control planes provide you're going to have a CR pushed this CR going to be validated by either you know Kubernetes schemas in the CRs or you're going to have like a backend type validation that will take this configuration you provided compared to whatever exists if you see that we're defining another host on top of a host that's already defined for example this is a problem I don't want this I don't want to do that so I'm going to turn this configuration to the client saying hey this already exists figure out your problem again also you know canary upgrades and so on if you have a decoupling between your multiple versions of Envoy with a control plane you can always you know let's say it's rare with Envoy to change stuff but let's say tomorrow they migrate to the new version of Envoy like we did between v2 and v3 if you got familiar with Envoy a bit it was a change of API so if you have a control plane that understand this change automatically convert an old configuration to a new version of an Envoy API it will make a wastewater upgrade to your system instead of like modifying every single file you know a thousand times so with this being said I really invite you to look at a control plane for your Envoy okay so it will avoid you the headache of going through manual change of files and restarts and so on we have an open source solution called glue edge that is a simplified control plane of on top of Envoy we can just create CRs on communities and that transform your CR into an Envoy configuration basically all we did today that can be done through CRs alright so I'm gonna give back the mic to Jim so you can give us a bit more probably a closing statement absolutely so yeah we have had some questions come in over the chat and I think Alex is answering those now so we appreciate that alright if you would like to continue this conversation so Adam and I will be available immediately after this we'll also have a number of people who understand Envoy much much better than I do at our booth S10 and we'll be happy to talk to you about more details about Envoy there and you can win a Millennium Falcon Lego set and all that good stuff alright now some of you will be interested in this if you would like to take the exam to get your Envoy Foundations badge there are some additional instructions here again you need 80% you can take the test multiple times but you'll need a different email address it's just a Google form the questions are just multiple choice and they're going to be drawn directly from the material that we've presented during this workshop so if you would like do you think we can zoom in the link there or select what's that? just to zoom in a bit oh I see I see yeah so this is a this is a program that that we've been running for a while at Sola with a number of open source technologies EBPF being the most notable ones but if you see the link there at the bottom or you can scan the QR code the link to get to the Google form that is the exam is bit.ly-envoy-exam again bit.ly-envoy-exam and so you can you can do that you'll also have access if you want to continue to play around with your environments and maybe explore some things that you didn't get to do deeply as you would like in the course of the discussion here you can actually restart the track if you would like to and you have access to it for probably another two and a half hours or so and certainly if you want to get access to it later just contact us at Sola we can set you up to be able to do it on your own okay and again did I mention we're hiring so yep you're passionate about application networking we would love to talk with you about about working at Sola okay I think we have yeah and I want to really mention something that was like a one-on-one basic debugging foundation and Envoy has a lot more to offer at different levels and we're planning on doing something more intermediate and something more expert in the future so yeah please just check out our website or reach us to us I'll tell you if there's any update there but yeah we hope to see each one of you in our next workshop Envoy workshop we have about five minutes left if there's anyone who has any questions you'd like to ask at this time there are microphones set up in the aisles feel free to ask us most anything all right go ahead can we get the mics live up the front mic live here okay try again hi thanks for your great workshop I have a question or something that was always bothering me was it possible to actually see the request coming into Envoy and debug them because this was always something that didn't manage to do that go ahead so when you see them you mean like I just want to understand like what do you mean by see them because you can obviously through metrics you can always see multiple filters sometimes expose different metrics right so if it's a filter chain you can always see that the listener got the connection and then you can go to every single filter and see if it's just intercepted the connection now if you're looking into like specific request and how it's flying into the filter chain definitely logs well access logs it's something simple but you can always turn on debug logs sorry on Envoy itself Envoy every time it intercept a request it's pushed like a request ID right so using that request ID that you're trying to track you're going to be able to track it down all the way down to basically your back end so I would say logs is the way to do it let me offer one other piece of advice too there is a filter in the Envoy library called the TAP filter and the TAP filter basically allows you at any point to tap into the request chain and to see what the state of the request is at that moment so that might be one that I think to address your specific question the TAP filter might be something that would help you certainly thank you alright anyone else alright let's put a wrap on it thank you so much for being here as I said at the opening we really appreciate talking to three dimensional people and we look forward to seeing you at the booth and seeing you around the show have a great conference thank you everyone