 Hello, everyone. Welcome to the session today. Today's topic, it is building multi-tenant routing and scaling with Envoy. I'm the speaker in Impen. I'm from the Amazon Web Service. Happy to be here to talk about the topic with you guys today. So a little bit self-introduction about myself. I'm the senior engineer from AWS based in Seattle. I'm the founding engineer of the AWS AppRounder. I participated in building the still small product from 0 to 1 and we did in the 2021. The product I'm working activities closely in the AWS is the including AppRounder, the Amazon ECS, Elastic Container Service, the AWS Fugate, which is focused on the serverless containers compute and Elastic Beanstalk. So my expertise I will focus on the cloud native containers and the serverless compute and open source. So I'm also pretty passionate in the open source community. I'm the founder and the maintainer of the cloud native and the serverless meetup community. You can find me in the GitHub and the Twitter. I'm pretty active there. So I want to show a little bit the motivation to have this talk today. I want to show the journey from the end user point of view by using NWI. I want to also give back to the community to show the lessons learned and experience by using NWI as a builder and hope it brings value to the community and help the product growth. And I also want to use this opportunity to thanks and appreciate all the support from the NWI community no matter it is internal or external. By the way, Amazon has an internal interest group for the NWI proxy with like around 160 people. I'm the founder and maintainer of that channel. I also want to send to the platform and I want to use this platform for the future discussion and communication. I'm a fan of KubeCon for years. I'm really happy and excited to be here to talk with you guys for the topic. A little bit of the introduction for AppRounder. As I mentioned before, so AWS AppRounder's GA in the 2021 is a fully managed service that makes it easy for builders to quickly deploy container web service and APIs at a scale with less and zero infrastructure experience required. Here is our official website. Feel free to check it out. So from the user point of view, the AppRounder provides a managed experience for customers app hosting, request the routing, load balancing and auto scaling, networking, the ingress, egress, both observability, CI CD deployment and the safe deployment and more. So customers just need to have their GitHub source code or the ECR image, private or public and deploy the image of source code connection to the AppRounder. AppRounder will directly make it up and running for your web app and or API service. And AppRounder going to output outcome and public URL for clients to access. AppRounder is a multi region supported and it is mainly focused on API and a web server. It accepts the HTTP request and it supports a concurrent request. And it also focus on the long time running servers which is today's world most popular and the typical traditional workloads. To build the AppRounder, to support the management user experience for AppRounder, including the request routing and load balancing, what's the responsibility of the request router? You may ask. So AppRounder needs a request router on the backend to help online multi-tenant request routing in different AWS regions. And it provides the URL for customer to access and online surrounding application instance for their containers. And then we need a request router to be in the middle between the URL and application instances to do the routing, to do load balancing and to do the traffic splitting. Product requirement. I like this because this is the motivation. Multi regions, region-expansable. Of course, we need to make sure we can build new regions all the time with the solution, with the existing architecture. You need to be multi-tenant. We are serving hundreds and millions customers as a platform and a solution. So we need the request router to be multi-tenant available. Auto scaling, we need to be scalable and we need to fail because in the future we're gonna face more and more traffic and workloads. Safe deployment. Of course, it is one of the basic requirements for the customer product. Observable. It needs to be manageable and provides necessary insights. So with the product requirements, the technical challenges will be obvious. They need to be high throughputs, high performance, high availability, high reliability. Think about your website. You don't want it to be down for a couple of minutes or seconds even. Extensibility. Of course, we need to build more and more features on top of it. Security. Large number of concurrence connections. As we mentioned, we accept the concurrent requests, which means for the request router, there may be a large number of connections happen in the same time. Observability. It's different with the product level of observability. We need the request router also provides observability and visibility for builders to understand what's going on in the middle and what's going on for the application activity and workloads. So we think about the on-way-based request router. On-way-based on open source edge and service proxy designed for native cloud applications. It provides features like auto process architecture. It is pretty lightweight and portable. HTTP layer 7 routing. GRPC ready and easy for builders to build the microservice around it. Basically in class observability. Provides necessary visibility for the system and service on the line. Microservice friendly. And it also provides managing the open source version available by the AWS app mesh. So we choose on-way as our request router base. I want to talk more a little bit the auto process architecture on-way provided. So on-way works with any programming language. Friendly, we can write the application in Go, in Java, in C++, and any other languages. On-way can bridge the gap between all of them and its behaviors identical. Regardless of the applications programming language or the operating system they are running on. Wiper owner, we are using protobuf and we write the service in C++, in Java, in golden. They can iterate with on-way with no gap and it break out all the boundaries of programming languages. The auto process architecture is beneficial as it gives the consistent see across the languages and application stacks. You can get an independent life cycle. Talking about building the on-way base request router. So first, the building blocks. So on-way supports the bootstrap configuration which is the key to make on-way up and running. Here we use the dynamic resources because the application instance is changed all the time. And looking at the architecture this is a very typical on-way usage architecture you can find in any way. You can, you should have a control plan to pass through all the configurations you need for on-way runtime. We will have a log and a metric system to collect and aggregate all the data and insights from the on-way runtime activity. You should have a UN streamer which is stream out all the outputs of the on-way activity and workloads. Talking about on-way itself. There are four building blocks for the on-way itself, internal. So first one is listener. Listener is the name network location. For example, the IP address plus the port. So on-way receives the request through the listeners. Roles. Roles is the configurations in the HTTP Connect Manager filter. Matching incoming requests like URI and handlers and define where the traffic is sent. Clusters. Cluster is a group of upstream hosts that accept the traffic for a route. List of hosts or IP addresses on which service is listening. Endpoints, which is a service application instance and the upstream IPs. We will talk about them one by one. First, the listeners. Listener is the component which is listening all the external HTTP requests and will operate on the page payloads. Received request and wrote to the next component block. Roles. So on-way will be the CRUX of load balancing and layer seven request proxying layer. It maintain a map of the service URL to the application instance IPs and wrote incoming requests to proper endpoints. It also mapping a domain from the service URL for example, halloween here to the upstream endpoints. Clusters. You can treat on-way clusters as the application service. For here, each service version going to be assigned an on-way cluster and the configurations will be updated through the on-way cluster discovery service. The short name is CDS. Endpoint. The upstream endpoints updated will the on-way endpoint discovery service. The short name is EDS and it supports the dynamic and auto-scaling and also covers health check which is part of it is the on-house endpoints reaper. I want to talk about the middle layer between the listen and the loads. It called HTTP connection manager. Show name is HCM. HCM, it is a network layer filter. It translates real bytes into HTTP. It handles access logging, request ID generation, tracing, handler manipulation, retry policy, timeout setting, traffic weights, and road waiters. For HCM, the road used the information from the incoming requests. For example, the host or authority handers and matches it to an upstream cluster through the virtual host or routing rules. Activity filters use the road configuration that contains all the road table for the HTTP routing. An set of handers can be specified to match the request. The router checks the request handers against all the specified handers in the road config. Some match includes the range, present, string, inward match. HCM filters also support the query parameters matching, the TRS context matching, and the GRPC routing matching for routing. And it always supports traffic splitting to different roles within the same virtual host. We can split the traffic between the two or more, upstream clusters. The traffic split happens on runtime, percentage, or weighted clusters. So, ANWI supports two types of the health check, active and passive. With active health check, ANWI periodically sends a request to the endpoint to check its status. With the passive health check, ANWI monitors how the endpoint responds to connections. You can use either of them depending on your case. Load balancing. Load balancing is very important because it is a way to distribute traffic between the multiple endpoints in a single service and a single upstream cluster. The reason to distribute traffic across multiple endpoints is because we want to make the best use of the resource. It helps the cost-effective for the product oil for your applications. Retrides. ANWI allow the retries and allow the setting the retry policy at the virtual host level at the HTTP Connect Manager and also the road level based on the conditions. For example, if there's a 429 status codes return from the endpoint, then we can set the retries. Then retries will happen stiminously on the upstream of ANWI. We can set the retries on the status code. We can set the retries on the priority level depending on your use case. Circuit breaker. This pattern prevents additional failures managing access to failing services. It allows us to fail quickly and apply back pressure downstream as soon as possible. It is best practice. We can configure breaker threshold for each road priority separately and globally. Time mode. ANWI supports multiple configure robot time modes depending on your scenarios. For example, there's a request time mode specified some amount of time ANWI waits for the entire request to be received. There are also idle time modes represent when a downstream or upstream connection gets terminated if there are no active streams. Rate limiter. ANWI supports the global and local rate limit to the upstream endpoints. We finished ANWI itself's building blocks. Let's step back and see the old architecture, the control plan component. Generally, you can view the customized control plan for ANWI runtime. It will responsible for service cluster discovery, CDS, and service endpoint discovery, EDS. And it was also responsible to passing all the route or other configurations necessary for ANWI runtime activity to the ANWI. It can be built as the GRB server listening on a defined port. ANWI needs to be updated with latest service information at runtime. So ANWI discover dynamic resources by curing the management servers using XDS which is full name is the discovery service APIs. The dynamic resource I'm in here including the clusters and endpoints. ANWI request resources with subscriptions by initially GRPC stream from ANWI client to the management server. Talking a little bit more about cluster and endpoint discovery, cluster going to be configured on ANWI through the CDS discovery service API and endpoint going to be updated through the EDS discovery service API. All of these APIs are open source. An app runner use the CDS API dynamically update clusters and the EDS to update endpoints per cluster. Location metrics is the other component which is important to collection all the insights and activity data. ANWI can configure exposed metrics locally and the static sync that received the metrics immediately from ANWI and do all the collection aggregation. For the metrics such as 2XX, 5XX request account can even truly forward it to logs and metrics service. This can be built as a GRPC service listening on defined port and all the collected metrics and logs can be routed to the popular cloud native or Adobe solutions. For example, the cloud watch which is for metrics and logging and then managing the service for permissions for monitoring and open telemetry or graphona or x-ray. The last component is the event streamer. It is responsible for reporting service status and health check results for the ANWI runtime. It also can be built as the GRPC service listening on defined port and all these health check and status results will be used in the supporting service like a safety deployment, blue-green deployment or automatic scaling, load balancing and request routing. So we finish all the functionalities of ANWI related ecosystem. Let's talk about how to make ANWI based system in production readiness. So besides the functional readiness, we need to make sure the capacity management is under control. It includes the CPU and the memory consumption because it is high performance, high throughput. You want to make sure there's nothing which is out of the expectation. We also need to make sure the performance and the scalability for the ANWI system. And at the same time the security, for example, DDOS, how you prevent the death DDOS which is in the place. And also, ANWI operational readiness items including the monitors and lamps. To ramp up, so the ANWI brings values in building a very solid request routing with the many features. High throughput and performance requests, routing and load balancing, portability, extensibility, reliability and availability. Of course, it provides very, very good observability. Easy for builders managing the online system and extends the system to build new things on top of it. With all the ANWI related topic finished, I also want to introduce a little bit on the Apronur's public roadmap. Apronur as a product has a public roadmap in the GitHub. We receive and collect customer requests and also we interact with customer directly, actively on the roadmap to show our plan, show what we are working on. There are also a couple of items which is related to request routing and poxing in the Apronur. For example, the private service support, additional ex forwarded hander support and WAF support and configuration timeout support. We are happy to receive your feedback and feel free to check it out and let us know how we can do better. This is all the things I want to talk about here. So thank you again to be in the session and also I will leave the rest of the time for the Q&A. Thanks.