 Hello everybody. Today we're going to be talking to you about service proxies, service meshes and API gateways. To the cloud native novice they all seem to do similar things, but as always the devil is in the details. If you're only tool as a hammer then every problem looks like a nail. Hopefully by the end of this session you're going to come away with a better understanding of the different tools available to you. The ecosystems around them and what kind of problems they're optimised for solving. We'll be giving you a high level introduction into the rise of microservices and the challenges that they bring. Then we're going to take you on a journey through the fundamental concepts and approaches to service proxies, service meshes and API gateways. This discussion is going to get quite deep but try and follow along and obviously you can catch up with the recording afterwards. My name is Armit Somali, I'm from Tyche, an open source API gateway and management platform. I've been here for six years now, I started out as a consulting engineer for our prospects and customers and then worked in product leadership and now in R&D where my team and I dig into the various technologies, understand what they mean for our users and the opportunities it provides for our customers and then I pester the product teams to help shape their strategic road maps. Hello, so I'm from the product team, my name is Carol, I work as a senior product manager at Tyche. What I do is take on this rambling user requirements and then feedbacks into our product road map to deliver values to our API platform owners and API developer users. I work in the platform and ops area and I'm responsible for Tyche's communities operators and ham charts. So let me begin by going back in time to understand why we need these technologies. In the past, most of us had only a handful of applications and then with the evidence of Microsoft's architecture, the number of services started to grow. The Microsoft's architecture brings a lot of flexibility and efficiency to the way it does and ops works. However, it also suffers some of the difficulties when we need to deal with the increasingly complex network communications. The increase in the number of services brings an increased number of services to service communications. However, network communications is always unreliable text time and they are vulnerable to attack. These days, it is pretty normal to have tens, hundreds or thousands of apps. We have replicas of these apps running on different nodes in one or more clusters. Those clusters can spread all over the world covering multiple regions. So basically, what were previously pretty simple internal application interactions is becoming a complex web of remote procedural calls that are transmitted over the network. Kubernetes solves a lot of networking challenges for us. For instance, Kubernetes service allows us to reach the correct application wherever it may be using a static service name. Ingress allows us to write external calls to the right applications based on the request header or path. But we need more. We want a load balance request across all the replicas of an application. We want to do mutual TLS. We want to have distributed tracing. We want to have rate limits to request and guard against web app security vulnerabilities. Over time, Service Proxy, API Gateway and Service Match were developed to solve these cross-cutting concerns. They all solved it without requiring applications to be rewritten. Let me start by talking about Service Proxy. Service Proxy is a networking component that can intercept traffic to and from an application. It evaluates a client's request and applies changes to it. These changes are configured and executed within the Service Proxy. There is no need to modify the application itself. Rather than connecting to the microservice application directly, clients now have to connect to the proxy. We commonly refer to the actual transmission and processing of data packets within the network as the data plan. So Service Proxy is a data plan component. Service Proxy are general purpose, meaning you can apply different kinds of checks or modify the traffic however you like. Say terminating TLS, executing authentication and authorization checks, rate limiting or dynamic routing based on system metrics. So all for it is a CNCF graduated Service Proxy. They illustrate the number of ways it can be deployed. In this diagram, the proxy is placed at the edge as an ingress or egress reverse proxy. Basically ingress reverse proxy will pick up external calls, send them to the credit service based on request header and URL. It can also make sure the client is authenticated. It can enforce SS control policies by command of patterns like ODOF. It can rate limit the request here to avoid overloading the backend service. The service proxy can also be used as an internal load balancer. So all service to service connections are routed through this internal load balancer allowing us to manage routing rules, authentication, authorization, monitoring from a single location. All for it can also be used as a cycle proxy. So cycle in Kubernetes world refer to the additional containers in the same port that extend and enhance the main container. Here all the request and response to and from the service are routed by the proxy. The proxies form a mesh network here. When service A wish to communicate with service C, it will first go through service A proxy to service C proxy and then reach service C. You may ask, what is the benefit of having a cycle proxy? Some of the reasons could be different servers require different proxy setup. So it is impossible or impractical to share the common one between them. Also this design can reduce the blast radius if one of the proxy goes down. If we are sharing a common proxy between all the servers, then when the proxy go down, all servers behind that proxy will be impact. However, if the proxies are now distributed, how could we configure all of them effectively? Service mesh has an answer to that. So service mesh is a dedicated infrastructure layer for managing service to service communication within a microservice architecture. The service proxy is usually deployed as a cycle proxy next to the application to route and process all traffic to and from that application. So here, who configures the cycle proxy around a certain request to service B? How are load balancing, circuit breaking and time out set? How to set the blue-green or canary deployments? And also system-wide authentication or authorizations? The service mesh control plane handles all of the above. The control plane takes a set of these isolated, stateless cycle proxies into a distributed system. Istio and Lingardee are two CNCF-graduated service mesh implementations. They both have similar architecture as just described. Service proxies are injected to the application to cycle automatically for all servers in the network. Istio used on-for-proxy as the data plane proxy while Lingardee developed the Lingardee to proxy and Rust. And also different products would have different philosophies and offer different sets of control plane features. The main features of the service mesh would include first dynamic service discovery in a dynamically changing container environment. It needs to register and de-register service endpoints as application instance goes up and down. Secondly, security. MTRS is a very powerful feature of service mesh. A service mesh provides MTRS by encryting and authenticating communication between this microservice using digital certificates. Each microservice has a unique certificate to ensure secured and authenticated interactions. Also, the service mesh also provides traffic management like routing, throttling, load balancing and progressive delivery. These L4 to L7 networking operations were previously implemented in the application developers. Networking code is so hard to write and maintain. Now, service mesh allows these concerns to be completely separate from the business logic. Service mesh also allows us to have visibility of all the traffic in the network. Here is an example of the QLE dashboard showing how traffic flows through Istio in real time without the need to code or configure your application. From this dashboard, you can track number of requests per second that the service has processed as well as success and failure weight. This is handy if you need to make sure your application do meet its SLO or SLA. Now, let's talk about the benefits of service mesh. Service mesh automates the best practice for observability, security and reliability. Service mesh provides central observability over all network services, reducing troubleshooting time. This may be the first reason users use service mesh. Service mesh automates log aggregation and telemetry collection for DevOps and SROG teams to monitor the service performance. These aggregated data help us to discover issues and trends. Security. Service mesh automates MDRS for all services to service communications and certificates rotation to prevent many in the middle attacks and achieve zero trust security. Fine grained traffic governance is possible with different service level authentication methods. This lets you create role-based access control for each service and restrict their communication. And also resilience. Service mesh automatically handles load balancing, retries, timeouts and circuit breaking, making the application more resilient to service failures. It will improve the overall user experience. This is something you should consider before implementing a service mesh though, especially from the operational perspective. Before Kubernetes 1.28, when a port starts up, the application container cannot guarantee that the cycle is ready. It becomes problematic if you need that cycle to say make network connections to the outside world. In Kubernetes 1.28, it introduced the cycle containers as the first class citizen in Kubernetes to natively solve these container-raised conditions and other problems. Cycle container support is now in alpha. Second, even if the data plan process is really thin, using the cycle deployment model still means adding an extra hope to your request and needing more resources to run. Linkety creates their own lightweight proxy and rust to reduce these performance impact. Thirdly, how much memory or CPU should you allocate to the sidecars? In practice, platform owners may offer proficient since these calculations are difficult. This can affect resources and costs. These issues are addressed differently by different providers. You could run a POC or talk to the maintainers here to learn more. I would like to talk about one more new approach from Istio here too. Solo and Google came up with a new architecture for Istio called the Istio Ambient Mesh to address these issues. Here, there is no more cycle process in this new architecture. It does this by separating the data plan process into two layers. A secure overlay layer called Z-tunnel takes care of the alpha networking task such as establishing MTLS and TCP telemetry. The Z-tunnel is deployed as the daemon set on every node. A second layer known as the Waypoint Proxy is then deployed to handle more advanced L7 processing like routing requests, load balancing for the ingestion, retries, timeouts. The Waypoint Proxy can be configured per namespace or per service account. It can be scaled and deployed independently with your application. The Waypoint Proxy is like a special gateway cluster for an application. You can choose to use just the Z-tunnel for core functions like sewer trust security, or you can also adapt the L7 processing. The claimed benefits of this process is it is a simpler application lifecycle because we don't have the cycle container running. Also, L7 processing can be skipped when it is unnecessary. Z-tunnel alone needs less CPU and memory to run. Waypoint Proxy is like regular ports that can be automatically scaled depending based on the demand with the application. It also helps with saving resources and using them more efficiently. Here is an example of how the data flows. When there is no L7 proxies required, service communications go through Z-tunnel on each host machine. When L7 proxies needed, EC2 control pan configures the Z-tunnel in the cluster to pass all the traffic also to the Waypoint Proxy to ensure the policies are always enforced. Don't you think the use of Waypoint Proxy looks very familiar to how a lot of our customers are using API gateway? That waypoint proxy does actually look quite similar. An API gateway is a tool that aggregates application APIs, making them available all in one place. It allows organisations to move key functions or cross-cutting concerns such as authentication and authorization, or limiting the number of requests between applications to a centrally managed data plane. An API gateway functions as a common interface for often external API consumers. In this section, I'm going to be talking about the typical key capabilities for API gateways, but I'll be focusing on the differentiators from service mesh traffic. We're then going to cover some typical deployment topologies and then introduce the concept of API management as not only a control plane for API gateways, but perhaps more importantly as a business enabler. In terms of L4 security, the edge of a system is where your users first interact with your applications. It's also typically the point where hackers first encounter your systems, and most enterprises are going to have multiple layers of security at the edge of their stack, such as a CDN, a WAF, or a dedicated DMZ. But for many smaller organisations, the API gateway may be the first and perhaps the only line of defence. This is why many API gateways include these lower level capabilities all the way down to L4. At this low level, they include TLS termination, IP allow or denialists, TTP routing, and even basic WAF capabilities that have been built in or integrations. You could, of course, use a service proxy for this use case too. But L7 is really where the API gateway shines. At L7, a distinguishing security feature of the gateway is the ability to interoperate with external authentication authorization service using, say, standards-based patterns like OAuth 2 and OpenID Connect. So what does that actually mean, though? The API gateway can read an authorization header, introspect tokens, validate JWTs, extract claims such as scopes and map those to validate against internal security policies and then route the traffic whilst propagating identity to the appropriate upstreams. Each API staff has their own security requirements, and not all gateways will have native support for your required API staff. For rest, this might include the ability to protect an API route by method or path, and more advanced gateways will be able to validate requests and responses by an open API contract or schema. For GraphQL, there will be the ability to enable or disable different queries, mutations or subscriptions, and even apply query depth limiting or granular field-based permissions. For GRPC-based services, an API gateway would need to be able to handle requests, authorization for services and service methods. Finally, for asynchronous APIs, one might need to apply rate limits based upon the number of events or the amount of data and enable authorization for what topics can be subscribed to. A common requirement for API gateways is the need to be able to optimize endpoints. So in terms of requests and response composition, most transactions will involve calls to multiple microservices, and sometimes they need to happen in a predetermined sequence or concurrently. API gateways can simplify chained or batch calls to expose a single endpoint or return a single aggregated response to the API client. So this results in improved performance, such as reduced latency, and it makes it really simple for the end consumers or developers to consume your services. In terms of API transformation, the API gateway can provide a public interface for a client which is completely different from the disparate set of polyglot API upstreams. This is known as protocol transformation or mediation. It allows you to keep a consistent public-facing API but swap out the implementation of the underlying microservices with no ill effect for your end consumers. Perhaps some of your APIs are legacy services that you don't have the desire or the resources to modify in order to make them fit your current business requirements. Microservice applications can expose all kinds of different formats over time, and this may or may not be intentional. When consumers want to consume GraphQL but your underlying microservices use other formats, such as JSON, Protobuf, Apache Avro or MQP, then the API gateway is the tool to perform that protocol transformation for you, all with no code. Then there's the use case for migrating from the monolith to microservices. So when you're breaking up the monolith, you need to ensure continued access to clients through the API gateway whilst you progressively spit out that monolith over time. I'm now going to talk about some of the typical API gateway architectures or deployment topologies. And because we're at KubeCon, I'll be presenting these examples in a Kubernetes context. But before I do, I just need to caveat that API gateways are not just for Kubernetes. Different vendors have different offerings and typically gateways can be made available as hardware appliances, deployed to virtual machines, and even available as a pure SaaS if you're not looking to manage infrastructure. In the shared gateway model, a single centralized gateway or cluster of gateways is deployed to a common namespace, such as Ingress. And the gateways may be deployed as a demon set, one per node or as a deployment which can scale as required. The API gateway is then used to serve many services, potentially across many namespaces. And it's up to the teams how to group their namespaces, and we often find that they're grouped by application domain or team. This is the most common deployment topology when the API gateway is being used for API productisation or composition. And it provides a single point of entry that hides the implementation of the underlying microservices, and it enables you to create rate limits and quotas and even expose tailored API products for different audiences. It even allows you to apply, say, the back end for front end pattern, BFF, by exposing different subsets of individual microservices to better support different API consumers, such as web browsers, IoT or mobile devices. Now, the gateway per service, this looks very... It's quite similar to the Ambient Mesh Waypoint proxy that Carol mentioned earlier. One of the challenges with the shared gateway approach is that the Ingress gateway is often difficult to tune for specific workloads. And inbound traffic to one service might overwhelm the gateway and negatively impact consumers of another service. Another challenge is that different workloads might require different levels of security and governance, say, PCI or hyper traffic, might need to be isolated from regular, publicly available web traffic. So, to solve these problems, it makes sense to deploy clusters of gateways dedicated to specific services. A further benefit of this approach is that the gateways can also independently scale and file. And we find that our users typically mix and match the shared gateway approach with the gateway per service approach. Sharding allows different gateway instances to selectively load different sets of APIs. So, despite having a common control plane. So, let's assume, for simplicity's sake, that we've got two zones in our network, the DMZ and the internal zone. We can deploy a gateway cluster to the DMZ and another to the internal zone, and then shard the gateways appropriately. So, it's possible to dynamically make routes to all services available to the gateway in the internal namespace, but perhaps restrict the APIs and services made available within the DMZ. As a kind of a side, you might have heard about the Kubernetes gateway API that has just been announced as GA. And for those of you who have missed it, it's definitely worth looking for the recording on YouTube because it's a work of completeness. I just thought I'd just give it two minutes introduction. The gateway API is a collaborative project that's concerned with the development of a common API to model networking inside Kubernetes. Now, if you're familiar with the older Ingress API, you can think of the gateway API as a kind of more expressive next-gen version of that. Rather than configuring Ingress using various annotations, we've got a set of strongly typed Kubernetes resources to help us. And there are three main kinds that you'd need to be aware of. The gateway class, gateway and HTTP route. And starting with gateway class, this is much like object-oriented programming. It describes the kind of controller that's responsible for a given gateway. And it could be regarded as a template for gateway deployments. And the gateway class describes an instance of an API gateway. It binds listeners to a set of IP addresses. And then finally, the HTTP route, as the name implies, provides a way to route HTTP requests. And this includes the ability to match requests by, say, hostname, path, or header, and even a query parameter. You can also define backends here to tell the gateway which backend to route traffic to. So the thing that's really powerful with this gateway API is that it's been designed to be role-oriented where the API is separated by the responsibility of the different users of the system. The infrastructure provider can be concerned with the gateway class, and then the cluster operators would interact with gateway objects, and app developers would focus on the HTTP route objects. You'd also want to be aware of the Gamma project, and that extends the use case for the gateway API beyond just north-south ingress traffic, and extends its use case for east-west service mesh use cases. So be sure to check out Flynn from Boynt's talk on Gamma tomorrow. I can keep listing and talking about the various features and functionalities of API gateways from, say, cores to caching, service discovery, mocking, circuit breaking, but we're going to be here for the next several hours, and you guys are all going to get bored. So I'm just going to zoom out a little and kind of provide a snapshot into more of a kind of product thinking business perspective. An API gateway alone won't solve these use cases I've got on the screen right now, but coupled with API management, they can bring significant business value. Now, in terms of development workflows, it's been a long, long time since I was a developer, but when I was, I'd often have to integrate with different services, and those services would be accessible via, say, RabbitMQ exchange, or arrest service, and the documentation would be provided via, say, a PDF or a wiki. Now, I'd need documentation in various formats, and then I'd need to translate that into an API client. Now, do I test against the real service? Do I mock that service? Do I test against the sandbox? With modern API gateways, I can deploy them locally on my laptop or within a docker container, and coupled with something like an open API specification, I can load that into the gateway and have the gateway automatically mock the response provided within that specification. So, what's more, the gateway can automatically validate payloads so that I can write tests that actually call the mock service on the gateway. Now, for more complex or dynamic mocks, I can create virtual functions to better simulate the real world, shaving hours of development time and assisting me with automated testing. I guess what I'm trying to say here is that the API gateway isn't just a thing that you go and deploy to a Kubernetes cluster and just leave it there. It's something that you can have at development time and it's useful for developers as they're developing their applications or integrations. Now, in terms of discovery, subscriptions and documentation, when I deploy my microservice, the chances are that folks need to be able to discover and consume it. Now, for an API to be discoverable, I need it to be published to some kind of developer portal. Backstage anybody? I mean, it's no good just making that API available to the developer portal, though. The API needs to provide a golden path to aid on boarding. The technical documentation needs to remain in sync with the actual service or the APIs that are deployed as the services kind of evolve over time. And API consumers need to be able to self-serve by requesting access to the API and somehow obtain credentials. Version management is also extremely important for APIs. Who will be impacted when, if we decide that we want to deprecate an endpoint or an entire API? How do we convince our consumers to upgrade their clients? And what if they just say no? So how many different versions of an API do we need to maintain and keep available? Now, developers integrate with a specific version of your API contract. So whilst it's very nice, it's also often a luxury to be able to just introduce breaking changes or enable progressive delivery. Not all services can be delivered in this way. And sometimes a more manual or managed approach is required. And then in terms of API analytics, developers, they're typically focused on the performance of their individual microservices. But given that a single endpoint in an API gateway may end up making multiple microservice calls, how do we get visibility of the overall performance of our API products? If a product team offer an SLA to their customers, how do we prove that we're meeting those SLAs and what are the financial penalties if we miss them? How do we better understand the adoption of the APIs that we publish, and how do we optimise the, from the time that a consumer requests, say, an API key through to the time that they make the first API call? Perhaps that's a success metric that we need to monitor and track. So, from the API consumer perspective, how do I understand how I'm consuming that API and how can I get requests or event logs and how do I debug my integrations without being forced to contact the developers, the development team support service? Thank you very much, Ahmed. So, in the final two minutes, I want to sum it up what we have discussed. I know we have a lot of information to you. So, although service proxy, service mesh and API gateways are all components for aiding network communications in Microsoft's architecture, they serve very different purpose and have different distinct features to help in particular use case. Service proxy is a general purpose proxy. It is commonly used as a load balancer. Service mesh is particularly useful for platform engineers, DevOps and SREs when they want to apply policies for security and governance purpose. It's also enabled them to get detailed insight into how the Microsoft is performing. API gateway, on the other hand, speaks more to API developers and API product owners. API is a crucial business exit that enables innovation and goes far beyond ingress. Think about open banking, stripe, GitHub, Trello and HubSpot. APIs for these business have been prioritized and used strategically to grow significant adoption and usage of the platforms. When you're trying to decide between service proxy, service mesh and API gateway, it's not a matter of choosing one or the other. Let's think about what outcome are you hoping to achieve? I think it's necessary to have zero trust security in your network. Service mesh can totally automate that for you. Would you like to consider adopting an API-first approach in development? Then API gateway is a tool that's here to help you out. Do you want both? Yes, why not? Making decisions like this can be really tough. I hope we have given you some good insights into the key differences between them in this talk. Thank you very much for your attention today. Please do lift it back while I talk. Thank you. Have we got time for questions? If anyone wants to ask questions at all? No. If anyone does want to ask questions, come and grab us after. We're here for you. Thank you.