 All right, we're going to get started with a lot to cover. Thank you so much for coming. Thank you for your time. My name is Amir Abbas. I'm a product manager at Google. I focus on application modernization, service mesh, also part of the Istio steering committee for all the Istio folks out there. I'll be at the Istio booth later. I've been with Google for about six years. And now I focus on reliability and multi-cluster architecture patterns. Steve? And I'm Steve McGee. I'm a reliability advocate, which is a title I just made up, part of DevRel. But I used to be an SRE for a long time. So I sort of feel like I am just an SRE. Is anyone here an SRE? Raise your hand. Yeah, my people, right on. So I worked on a bunch of services like Android and YouTube, Cloud for a while, and then I left. I left Google. I became a customer. So I moved to California. I helped this company move from on-prem, monolith, big knock, all this stuff onto the cloud. And it was difficult. Let me tell you, it was very difficult. I learned a lot in that year. So I actually came back to Google to basically help more companies go through that process, because I don't think it's easy to go through it alone. So I'm hoping we can show you guys a couple of ideas here today to help you build more reliable services and just kind of make the internet more reliable as a whole. But first up, a question. You can answer this in your head. You can raise your hand, whatever. The question is, can you build four or nine services on top of three nines infrastructure? Can you? Is it possible? So my favorite thing about this question is not what the answer is. It's the fact that every time I ask a room full of people, I get half yes and half no. Not necessarily left and right, but like half and half. I actually asked that question at SREcon here in Amsterdam last year. Still, half yes, half no. So it's not a straightforward answer. So this question is kind of weird. So to help understand what I mean by all this and why I think the answer is yes, I'm hoping that by the end of the talk today, you will agree with me if you don't already. But to help understand that, we're going to use a couple kind of symbols and analogies, I guess. So first up, we have these pyramids. This is a little abstract, but bear with me. The way that we have done reliability in the past for a very long time was like the pyramids in Giza. Big, giant thing, very stable base. The thing at the top just kind of inherits the reliability from the stuff below it. And this makes perfect sense. This is how buildings work. Humans just kind of understand this. If you want something to be stable, the stuff underneath it should be more stable. So if you think about this, if we have like a 3 9s application, we want like a 3 1⁄2 9s database, and we want like a 4 9 network under that. Something like that, right? The trick is when we move to cloud, it flips around and nobody tells you that before you get there. So for example, I've had many customers come to me and say like, hey, Google guy, your VMs have 2 1⁄2 9s. What's the deal? Like how come your VMs have so few 9s? How can I build a 4 9 service on top of 2 1⁄2 9 VMs? And the answer is this. It's the pyramid, right? So your application needs to increase its reliability as you go up the stack. And how in the world do we do that? Think about RAID, right? So we have a bunch of disks. They all have like a 99% availability because disks fail all the time. But if you stick them all together and put a bunch of software and maybe some hardware and some complexity around it, you get a SAN. And your SAN is way more reliable, right? Like how did you do that? How did you make something more than the sum of its parts? The answer is software. We did software to it, right? So distributed systems are a thing that allow us to do this at greater and greater scales. Pretty great. Here's just kind of breakdown of, I don't know if you can read the text, but essentially in the old days we had our data center and we had our networks and things. On the cloud, we think about, you have IS on the bottom, you build your platform on how you're gonna use the infrastructure and you put your app on top of that. Does that make sense? So I wanna make sure that we point out that the way we're doing this is through software, right? So the software actually increases the availability of the systems underneath. And I think the important part is that once you move to the cloud, you have to change the way you think. If you're in the cloud and you're thinking in the right hand side and the right side up pyramid, it's not gonna work as well. So the first thing we have to do is change the way of thinking and then we have to build reliable applications which we're gonna show you some frameworks and stuff that Steve's gonna get into. That's right. So let's define a few terms first. We're using this word archetype. It comes from a paper. You can just think of this. Maybe also it's just a pattern but an archetype is essentially just an abstracted architecture, right? And specifically we're talking about reliability focused archetypes. So we're looking through a lens of reliability here. There's many lenses you could use to decide your architectures. This is one of them. The architecture is what you do next. So once you have the basic idea of what you wanna build, now we have to fill it in with real services that products that you can buy, projects that you can download, things like that. Then you need to write code, right? You have to actually fill it with services, your own system. And this is something that's gonna change all the time. You're constantly upgrading your code and writing new things. The other term we use here is footprint. So you're not just gonna deploy your code once to like a production. You have to put it in many places, right? You're gonna have many regions and zones and instances and clusters and things like that. And your footprint is gonna change as well. It might be because your business goes into a new market. It might be because you couldn't get quota in that one place and now you got quota in the other place. Like there's many reasons why your footprint might change. And finally, you're gonna be tracking your SLOs constantly. So the pattern that we have here is you have a really tight loop at the bottom. Your SLOs are constantly being measured, making sure things are working well. You have a slightly less tight loop above that, which is the changes that we see every day or week, however often you deploy and change things. Your architecture, you will change every once in a while, slightly broader loop. Your archetype should really stay the same as long as you kind of understand what it is that you're getting out of it. If you change your archetype, it's gonna change your architecture, it's gonna change your code, it's gonna change your SLOs. So there's like a pattern arriving here. So let's learn about this in another way as well, like graphically. Yeah, so let's talk about what we're actually trying to do. So we're trying to build applications for our consumers, for our customers. And since I'm a product manager, we'll start with the products. So at the very outset, you have something called a product, like that's what you're trying to sell, right? And they have product managers that are building these products. For example, let's say you wanna sell some hipster fashion lifestyle product. Within that, you have applications, and your applications could be a marketing application, or in this case, like an online application, a product can have multiple applications, right? So in this case, let's just say we have an online store or something where we're trying to sell this fashion accessories. And then within that, an application is composed of one or more services, right? So now it's starting to maybe look a little familiar. We're all at KubeCon. We know what a Kubernetes service is. This is very similar to a Kubernetes service, but just think more canonical, think more universal, right? This doesn't have to run in Kubernetes, but it's sort of the same concept. So in this case, for example, I need to have some sort of a web frontend, maybe a product catalog, maybe a checkout service. A bunch of these services collectively form this application. And then these services have workloads, right? The workloads are the actual tasks and functions that are running within them. You can think of those as deployments or pods that are running there. So this is what we're ultimately trying to build. But we're trying to build this on top of a platform, right? So the way we define a platform or a simple definition of a platform would be a set of capabilities where you abstract these capabilities away from your developers, right? You provide them through some form or fashion. Notice that we did not put any product names here, right? So for example, your platform could have virtualization as a capability or container orchestration or serverless or CI CD observability, right? Or you may not have CI CD right now. You may have to open a ticket and then do a manual rollout. That's perfectly fine, right? You sort of iterate on your platform. What do we call that when we do it that way? Oh, paperwork as a platform. Has anybody ever heard that term? I think it's great. That's a perfectly, right? It still counts, it's still a platform. It's still a platform, right? So, and we're continuously iterating it. So in a way, you can think of a platform, or at least we think of platforms as also products. A platform is sort of your internal product for your internal consumers, developers, application owners, product managers that have capabilities. You can even market that platform internally. You can name it. So at Google we call our platform Borg. You can name your Cylon. Any Battlestar Galactica fans out there? Yeah, some nods. Thanks. All right. Where archetypes sort of fit in is this idea that how do we connect these services to the platform? And this is sort of where the archetype's been. This is kind of a new term and we're kind of rolling this out and see if this lands on you. So an archetype is an application or a service concept. A service belongs to an archetype. For instance, we can say that this platform called Cylon supports three archetypes. We're gonna get into what these are. Global archetype, active passive region, active passive zone. Hopefully they're starting to make sense what they are. Now you can place your services on top of these archetypes, right? So in my online store, I have many, many, many services, but I want the front end to run everywhere. It's a global HA web service. I want everybody in the world to be able to buy the clothes that I'm selling. But my product catalog might be regionally significant or geographically, because every region has different products, let's say. So I'm gonna run that as active passive zone so I still need some redundancy. We think archetypes is great because it's a unified language that you can speak with your developers. So if you're a platform admin, you're talking to your application owners, your product managers, you're not talking about products. You're not talking about Kubernetes and service mesh and get ops and those types of things. You're just talking about capabilities and specifically in terms of reliability. So now you have these two products. You have an internal product called your platform and then you have this external product, which is your applications. Where reliability comes in is the goal of how we measure the health or where we want this product to do what we want it to do. So you can think of it as this sort of a dotted line and the way we measure it through SLOs. And we're gonna get into SLOs here shortly. Notice that the line, so if you're above this line, you're healthy. If you're below this line, you're unhealthy. Notice that we did not draw this line below it. We don't want 100% uptime. Think of it as like water. These like boats on water, right? Yeah, so we always want to allow for a little bit of error, right? And that's perfect line. This is the whole idea of building four, nine services on top of three, nine. Risks are the ways your applications or your products can break. There are two types of risks, known and unknown. And you can think of risks are forces that are trying to push these things below water. They're trying to sink them. The resilience are mitigations or countermeasures you put in to defend against those risks, specifically known risks, right? As unknown becomes known, then you start doing that. And that's sort of the life of, and I'm sorry if I'm not mistaken, right? That's right. So an SRE team provides resilience to a platform. The platform becomes more robust and it can handle these risks. Yeah, so visually you can think that risks are trying to push these things down. In some cases, they're further down. You're breaking your SLOs or something. Your errors are going up. And then your resilience is trying to push these things back up. And then you're measuring these two reliability of these two products independently of one another. So your platform might be going through some sort of a rebuild. You're moving from VMs to Kubernetes or what have you. But your applications are perfectly fine. So Steve's gonna talk about five application archetypes. Yeah, this is the meat, right? So let's get into it. So this all comes from a white paper was written by two Googlers named Anna and Brad. Their names are up there. The link is up there. These will all be in the slides. This is one example. We'll get into this in just a second. But just so you know, this comes from a paper which has way more in it. Like you should totally read the whole paper. Okay, the first one you've all seen before, like this is our old friend, Hot Cold. So if you have like a system and someone's like, we need this to be slightly more reliable. The first answer is let's make two of them, right? Have one beyond and the other one be like ready. And we'll get to it if we need it. That's all this is. Specifically with what we're doing is we're putting one in one zone and one in the other zone. And the reason why is because a zone is the basic like unit of failure domain, right? Is the smallest failure domain in cloud. So if you know something's gonna fail, it's gonna fail on a zone. So this is why we split them in half. We have a few things you can kind of measure here. I'm just gonna point out a couple of them. One is like, what can this be robust against? And specifically we're saying it's robust against zone failure. And that doesn't necessarily mean like the whole zone just like evaporates. It can mean like a service in a zone fails, whether it's your own service or maybe a managed service that you depend on, but the granularity is within a zone. This is not robust against a region failure. Again, regional services or like think of natural disasters like a hurricane will take out an entire region at a time or a flood or an earthquake or something like that. So if you're in some sort of like regulated industry, you have a lot of contingencies required where you say we have to be in multiple regions because floods happen, right? And so this will not defend against that. I think we get that. This is a good model for something like Cots off the shelf software or anything where you pay by the license, right? Cause only one is available at a time. Okay, this one's pretty straightforward. Next one is multi-zonal. So this is like hot, hot, hot. So if you need to move from the last one to this one, you're gonna have to make some changes. So you can't really just lift and shift into this. In fact, often what we see is someone kind of like thinks let's just do this. They lift and shift onto this model and like it doesn't work because the sessions were held like in memory, not in a database and it all breaks, right? Everyone has seen this before. So yeah, some changes are gonna be made if you move to this model, but you're gonna get more out of it, right? Now we're able to handle the zone failure. We still can't handle the region failure, but in this case, the zone failure is way less fail ops. Like there's fewer things we have to do and the cost is a little bit better. Instead of doubling what we're spending, it's actually just 1.5X and it has to do with the fact that we're using three zones here. And we can get into the math if you'd like in the back later. This is great for just kind of standard web services. Okay, the next one is active passive region. So we're kind of combining the last two. So we take the three active active actives and then we do a backup version of that. This is getting more expensive, right? We're spending a lot more on our capacity but we can now handle a new risk that we couldn't handle before. We are now capable of handling a regional failure. In this case, you may have to still do some stuff when you're fail ops. You might have to change the load balancer. In this case, we're changing the DNS entry. And you may have to promote the read replica in the surviving zone to be like a primary database. Sure. But now we can survive both zone and region failures. This one's funny because it's almost exactly like the last one. I'm gonna back up and see if you can spot the difference. What changed between this one and that one? It's right at the very top. You have two host entries. And basically what you're saying is your customers either go to one side or the other side. And this is actually pretty common again in regulated industries when we say these customers, all their data has to live over there and can never leave. And these customers have to live over there and they can never leave. So the interesting thing about this model is that if one region fails, the other region is not affected. The services that are in the region that fails, they are failed, they're down, but the other customers are unaffected because they're isolated. So you can imagine this might be useful in some instances and not others. The last one here is kind of the big fancy one. This is for global consumer services. So this is like the Google model. This is like Google ads and Gmail and search and all this kind of stuff. So this is when we're using global anycast load balancer at the top. We're using spanner or cockroach at the bottom. And this is great because you can handle zone and regional failures. And the fail ops in this case is absolutely nothing. Like you just keep drinking coffee. Like that's your plan and it's great. So zones, they go down, no big deal. Region goes down, no big deal. You have to have the capacity in place, sure, but you can handle it. Pretty great. So that was like a lightning round of archetypes. What do we do with them? So first thing is that an archetype is used by a service. A service cannot occupy two archetypes. It has to kind of pick one. So it has to do with its footprint. Okay, that's cool. An application can have many services. This is what microservices are all about, right? So in this case, you can actually have some of your services be one archetype and some of them be another archetype. Okay, that sounds crazy, Steve. Like, why would you possibly do that? What we're trying to get at here is graceful degradation. We want to allow some services to be run in a cheaper way because we don't care about them. If they fail, no big deal, right? This is how we get graceful degradation. So we get the application is still running. It's not perfect, but it's still running. It is not down, right? Sometimes you hear things about soft dependencies, right? So if we have a system that has a soft dependency in letter D here, we can allow D to fail. And our fail ups in this case is do nothing. Let it fail, right? Move on. It's great. You don't have to answer the pager. Okay, how do we measure them? SLOs, this is gonna be a quick reminder if you haven't seen them before. Yeah, so we're gonna now take some of these archetypes and actually build clusters out of them. Because you probably see them and you say, well, this kind of is common sense. You're putting stuff in more clusters. We're gonna end that with talking about very discrete archetypes as well as what types of applications that are there. But before we build architectures and Kubernetes and all that stuff, let's just quick recap. An SLI is a service level indicator. It is a quantitative way to measure the health of any system. We're using some sort of a metric. So metrics would be like latency throughput availability. I think most people know this. There's a lot of SREs here. An SLO is service level objective. And that is where you take an SLI and you create a goal around it. And this is what measures the health of a service or anything. So for example, 99% of all get calls should complete in less than 100 millisecond in a given day. That's your SLO. Collectively, it's uncontroversial. You're trying to achieve that goal as both the platform team and a product team. Here's another way of looking at it. Your SLIs are constantly changing, right? So sometimes your applications are healthy. Sometimes they're not. And your SLO is that dotted red line. That's that surface of the water. And you want to make sure that the SLI stays above that. Rarely we talk about aggregate or cumulative SLIs or SLOs, I should say. So here's an example. Here's a typical microservice chain. Service A talks to B, talks to C, talks to D. So in order for this request to pass, all four services must be up for the life of that request. Not one of them could go down. So these are hard dependencies. The numbers below them are just examples. Let's just call them availability SLOs. So individually, they're 99.9% available. But if we want to calculate the SLO of this entire system, we just multiply the SLOs together. And when we multiply them together, you'll notice something that the SLO actually goes down a little bit, right? So you would think maybe, you know, common sense would say like, oh, the SLO of this is three nines. If they're all three nines, it should just be three nines. What's the problem? Right, but that's not the case. So sometimes Steve and I just call this the bad math, right? Because it's going in the wrong direction. The number here is not that important. We just want to look at the trend. Is it going down or up? That's what we want to focus on. 99.6 is the best you can do. That's your best case scenario, barring nothing else. You can always do worse, obviously. The other model is the distributed model, right? A lot of people kind of know this. So this is the same service where previously it was four separate services. This is the same service service A, but now we have four instances running. So these could be the four pods or four VMs. And now the formula is a little bit different. So now if we want to calculate the SLO of this, it's one minus the failure rates multiplied together. So one minus 0.001 raised to the four. This is great. Let's just do this. 11, 9. 11, 9s. Does anybody want 11, 9s? You guys don't believe that, right? Four pods everywhere, you get 11, 9s. No, so we know that's not right. So if we have any like probability and statistics majors in the house, like what are we doing here? We have four independent events. That's how you get the math to work. But if you have systems that are not independent, they share some failure domain. This math does not work this way, right? So how do we get independence between these events? We have to span failure domains. So in this case, these pods cannot be in the same cluster because a cluster is a failure domain. They cannot be in the same zone, right? Because the zone is also a failure domain. They also can't be in the same region or the same planet, right? In which case- Or the same universe. Yeah, we thought maybe multiverse possibly, but we haven't invented a load balancer that spans multiverses yet. So we can't really do that yet. So someday we'll get to 11, 9s, but not until we get that. If Marvel has taught us anything, multiverses exist. But actually we found out that you can cross multiverses, so it's shared failure. So let's get back on track. So again, we're looking at the trends here. If we do math, this is what we call the good math. It's always in green. Greens are good. So we're going up this way. We're going down the other way. And we can't ignore both of them. So now let's start taking those archetypes and actually build architectures. Remember, we started with archetypes. Now we're gonna build architectures for our platforms. And the reason we talked about SLOs is because we wanna talk about reliability and we wanna do it measurably. So we're gonna give you a bonus archetype. This was 2.1. And by the way, the numbers up there were what was in the paper. You don't have to memorize them or whatever. So this is where typically people start, right? So you just are getting started in your Kubernetes journey. You deploy a Kubernetes cluster. You throw some services on there. You use ingress. Maybe you have a backend database. So it has a load balancer, a Kubernetes cluster with a service and a database. The numbers that you see underneath each are the SLA numbers that at least Google Cloud publishes externally. We're gonna just use them as SLOs, SLA service level agreement. It's the commercialized version of an SLO. Why not? Again, we're just interested in the trends. And then these are the SLA numbers of Google's managed services. So not of the service A, but of the load balancer, Kubernetes cluster and the Cloud SQL. So we know that this is the bad math. So these are all connected together. And in order for this request to pass, the load balancer must be up the cluster and the service as well as the Cloud SQL. So we just multiply them together and you get less than two and a half nines. Bummer. That's why we didn't even put that archetype up there in terms of reliability. But this is just to show you that the math there and we have a baseline, we have a bottom line. An important thing to notice, this is the best you can do in this architecture. You can definitely do worse than this. You can operate this poorly and get like zero nines. So this is the best you can do. You're less than two and a half nines. Okay, now let's start building it on the five archetypes that we showed you. So this is what the architecture would look like. By the way, we're using GKE and Google stuff. You don't have to, but this is just because we work at Google and legal told us to do it. And the fact that we wanted to use the SLA. So here we have a active passive zone. The couple of things that we notice here, and we're gonna go through this very quickly, is we have two Kubernetes clusters in one region in two separate zones. So we've separated out the failure domains here. Same service running on both clusters. So we need some sort of a way to connect this beyond the scope of this talk. But let's just say we figured that part out. And then we have two SQL servers or two SQL databases as well. Again, in separate zones. Now we have this good math happening in the middle for these subsystems, for the Kubernetes system as well as the SQL. So we've done that math right below them in green. But then we still need the entire thing to be up. And then we've done that math at the bottom. And you can see that just going from one cluster to two cluster, we're headed in the right direction. We're almost at four nines. So please start here. And this should work for a lot of applications. Next one was active, active, active. It doesn't look that different. Now we have three Kubernetes clusters in the same region, but in three separate zones where we can survive one or more zone failures within that. And there's no fail ops for this one. So your service must be able to handle this idea of distributed Nes. Cloud SQL doesn't change. And then again, we've done the good math for the three clusters for the two SQL servers. And then if we connect them together, we're at four nines. We're headed in the right direction. Everything is moving up. Let's go into multi regions. And this is a good sort of progression, right? Where you start maybe in a single region and then something either bad happens or you've identified that there could be a risk to a region or you've pushed a bad config to a load balancer one time. And you said, we don't want to have that happen again. And then you start moving to multi regional architectures. So now we have two regions. So it's basically the, it's almost like times two of the previous one. So you have a blue region up top with the three clusters in three different zones, a green region at the bottom, three clusters in three different zones. Now we have two regional load balancers. So you have two ways to get things right, right? In case something happens. We have a Cloud SQL pair in region A, Cloud SQL pair in region B, and then DNS is doing its thing. Lots of parallel math happening. Now we have load balancers and six clusters and four SQLs. There's a lot of complexity. Yeah, and like we're not claiming that you will actually get those nines. This is just like showing you the direction of travel. So now we have, I don't even know, like 18 nines or something. So looking good. The next one is actually exactly the same. The only difference here was this idea that we're sending half of the users to region A, half of the users to region B, they may be able to fail over. That's beyond the scope. And we've also changed the database into a more sort of, I don't know, Cloud native database. In this case, we're using Spanner, but you could use Cockroach or something like that too. But otherwise, it's exactly the same. Again, we're headed in the right direction. And then the last one was global. And this was this multi-regional, no-fail-ops, using a global database layer and a global load balancing layer. It actually got simple, right? So we have one global load balancer, front-ending a bunch of clusters. In this case, we're just doing it in two regions. So six clusters total. Three clusters in region A in three different zones, three clusters in region B. And then a global database in the back. And then we have this. So yeah, I just noticed that, I don't know, did anybody notice something weird happened here in terms of an SLO? I don't went down, right? Embarrassing. That's not cool. So like, well, let's just go back. So like, we went back, this multi-region sort of a thing. We have, I don't know, so many nines here. And then we have less here. What's happening here? Yeah, so the reason why this is okay is like it's still a lot of nines. Like four nines is kind of a lot. There's way less to screw up here, right? So there's fewer chances to mess up. Remember, this is the best case scenario. You can definitely get, you can do worse than this when you build this out. So simplicity matters a lot when it comes to operations and making changes. So, you know, take these numbers with a grain of salt. There's a lot more going on here than just what's on this one screen. Yeah, now previously we talked about this idea that a service can belong to one archetype and we've just showed you five different architectures. And then maybe you're thinking like, well, do I have to build five platforms, one for each type of service? Well, that's not the case. As a matter of fact, the astute amongst you might have noticed that the further we move along, the other archetype sort of gets subsumed into the one. So here's the same archetype as the last diagram. So this is the global archetype. And we're showing an application that let's say has three services, A, B and C. And they're running in three different archetypes. So your service A, that could be your front end. That's running as a global archetype. Your service B, that could be your product catalog. That's running as a regional or like a multi-zonal archetype. And then your service C, that could be your rating service or some sort of an email service that gets sent out when you order something. That's just running it as an active passive sort of a way. So what we're trying to show here at the end of the day is you have this continuum. If you want to get more reliability, you have to spend more effort. Think of that either as money or just time or technical debt or whatever. And this is an exponential curve. So the further you go on here, the steeper and steeper it gets. What we're trying to present here today is just some discrete points along that line. So you don't have to just invent everything from scratch every single time. You have some known good starting points. And we're calling them archetypes. They come from the paper. You can call them patterns. You can call them points. You can call them whatever you want. But at least you're starting somewhere. And you can present these to your dev teams, right? You can say like, pick one of the five. Like you don't have to learn all of Kubernetes and all of Istio and all the things. Just pick one of these things. And they'll go like, oh, the middle one. And you're like, great, we're done, right? We can move on. The best part about this is then if they're wrong, you just graduate them up or down, right? Pretty straightforward. So if you think about the types of services that will go along here, they just map to these points. So your cots probably doesn't need to be all the way at the top right. It's probably not going to work anyway. But you can give them the options, right? Don't forget about the effort. If you really want that fifth nine, look how big that E is, right? And look how far to the right it goes. Like not a lot. It's like a lot of up and not a lot of right. This is the problem with exponential curves, is it gets harder and harder to hill climb. So don't forget that. Make sure that when people say, I want five nines, you show them this graph and say like, are you sure you want to spend all the money? Another thing you can do with this graph is if someone says I want 100%, you just point at them and say, here you go, it will be infinite money, right? That's how limits work, right? And I think, oh, we have some conclusions, good. So start with archetypes. That's a good, if you remember anything from this talk, remember that, just start with these things. It's a good place to start before you start drawing your architectures. Don't forget that your applications have many services in them, right? We do microservices for a reason, let's use them. We want them to degrade gracefully. And finally, what we're trying to get here is we want to have resilient teams, that's us humans, right? We want to build robust platforms that can handle these risks because we want more reliable services and products, right? We want the internet to be more reliable. That's the state we want to get it in. This is how we do it. Don't forget to give us feedback. I think we have five minutes for questions. If you guys want to ask us some questions, we'll both be, you'll be at the Google booth from two to four, I think. I'll be at the Istio booth from like 130 to 330 and Google booth from four to six. I don't know if you guys will remember that or just come on up after the talk. Otherwise, thank you so much for your time.