 Hello everyone. Thank you for coming to our talk. We're really amazed that so many have showed up here this late in the day and that's really nice. We'll promise you three things today, two, maybe three. First of all, we will give you a presentation without no YAML. I know that's almost not allowed here, but that's what we will do. Also, we will give you a demo, fingers crossed. That's mostly visual, a little bit script maybe, but mostly visual. So that should also be good here in the end of the day. And maybe we will also show you a tool that will be useful for your needs. We'll see. So my name is Michael. I'm from, come from to Denmark and here is my colleague Martin. And we will talk to you about how we have built a self-service platform or self-service solution for our developers so that they can manage all their networking needs through a self-service solution. So Martin, what kind of company is to Denmark? Well, to Denmark is a governmental owned commercial broadcasting station that was founded in the late 80s with public service providers. So we provide gains with news and entertainment. For many years, our main products have been like a range of linear broadcasting channels, where the main channel was actually the most viewed linear channel in Denmark last year. But besides that, we also have some digital products. We have TV2DK, the web page website that mainly contains news, along with some free mobile apps for news, weather forecasting, sports and TV program guide. And then we have our streaming service, TV2 Play, which is a subscription based streaming service. And it's one of Denmark's largest with more than a million subscribers. And actually for quite some years, we've had the streaming service doing both live TV on demand content. And we also do some interesting things in terms of dynamic power channels when they're big sports events. We do some linear ad replacements, so basically targeted ads on linear flow TV. So we do some quite exciting things. And throughout the years, our digital products have gained a lot more focus and popularity due to the decrease in viewers on linear TV. So TV2 as a company, we are like around 1500 employees in total, zooming in on the digital department. We are a bit more than 200 employees in 20 different teams spread across many different roles and responsibilities. We have more than 10 products, counting both our public products, but also internal products. So we do some internal tooling for editors to publish some prioritized content on our public products. So looking a bit on the cloud journey we've had so far. So for the past four or five years or so, our development teams have been building their own cloud platforms. This means that today we have many similar platforms, but with some difference in details. So we have many different snowflake platforms. We've basically been using traditional inquires. Most developers have been using traditional inquires to expose applications. Some teams are using Istio for more advanced networking. And throughout the last couple of years, we've seen an increased need for network features external to Kubernetes, such as web application firewalls, DDoS protection, network filtering and so on. So last year we established a platform team, tried to set out, build a common Kubernetes platform and multi-ten platform that with time will support all of our developers. The main focus is like most other companies to reduce developer cognitive load to try to abstract away some of the infrastructure that most developers don't really care about and provide a paved path for running applications in cloud. One of the main principles we are aiming towards is to try to use the Kubernetes API for everything. That's basically so whether we want to deploy an application or configure our network provision in a database, we would use the Kubernetes API but we would of course expose it with some kind of abstraction on top. But we don't want to take away the freedom and autonomy that that our developers have today. So we will use the GitOps to enable cell service. So one of the challenges we've seen throughout the years is basically gluing things together from the Kubernetes world with your cloud provider. And we've been over a couple of different solutions to try to solve that. Basically we provision infrastructure with Terraform like many others are doing and then we've been using custom scripts and templates to basically configure our Kubernetes resources with the right IDs and odds from cloud resources. Then we did a bit of experimentation with a tool called Catographer, which is basically a Kubernetes controller that you can use to build template and supply chains. It didn't quite fulfill the needs we had so we set out to build a custom controller using a framework called MetaController. Try to glue things together with that. It's a framework that makes it really simple to get started writing your own controllers but with the simplicity also came up with some limitations that again didn't quite fulfill the needs we had. So in the end we set out to build our own full blown Kubernetes controller from scratch using a cube builder and at the same time combined where we were building the platform and we switched the focus a bit to network configuration as that's a crucial part of the multi-tenant platform we're building. So yeah we set out to build a Kubernetes controller implementing the gateway API so we could use that for configuring our network. So how many here knows gateway API? So approximately a third I guess. How many here are using an implementation of gateway API today? A handful, right? So looking at the ingress it was initially designed for more simple traffic routing designed many years ago. It is extensible but it's only extensible via custom annotation as it's a network model that only consists of one Kubernetes resource and because it's only consists of one resource it also combines multiple organizational roles and responsibilities into this one resource whereas gateway API is a networking model that consists of several Kubernetes resources making it a bit more flexible. We already see a lot of other CNCF tools that support gateway API such as external DNS. We expect a lot of other tools to support it in the future so we also see it as a bit of like the new standard and a more future-proof solution. On top of that giving that well it consists of multiple resources it's also role-oriented so as you can see on the illustration here I mean you can have infrastructure providers specifying one part of the network configuration you can have cluster operators doing something else and the application developers only actually have to care about routing to the application. It's also quite expressive so natively you have support for header-based routing, traffic waiting and so on and it's also an API that's easily extensible with your own custom resources. So these models or well the current implementations of these two models there's not really that good support for network features external to Kubernetes and that's something we tried to solve so maybe Michael can tell you a bit more about how we utilize the gateway API. Yes so what we did was that we used the gateway API as a sole interface to our platform. When building a platform the API choice is really really essential because that will be I mean what you present to all your users and we use the gateway API for all roles in our platform so not only infrastructure providers but also tooling so the gateway API becomes the main integration point for all tooling tooling that we use them now and tooling that we will use in in the future so that's one of the powers of a common API and of course most importantly it also becomes an API that we present to our developers and with all the configuration now in defined through the gateway API all we need for our platform is just somehow a mechanism to provision an end-to-end data path and this is what a gateway controller does that is simply provisioning this this data path now here we are we're stressing that it's an end-to-end path and why do we do that that is because we really want this data path to be from customer all the way to business logic because if it isn't end-to-end then it's really not a self-service solution if someone else needs to go in and hear and define and part of the data path it's not really a self-service solution we think so that's why we stress that is an end-to-end path now in our case because we have so many products and teams that have different needs are evolving at different speeds and so on we want to be in more control of the this actual data path we want to introduce features and and the things as we go along that that suit the needs of our individual teams and this is where we didn't find a solution that fitted our needs so that's why we decided that we would build our own controller that would I mean kind of scratch our itch and do what we want to do so we can fulfill the needs of our product teams when we did this made this choice that we would develop our own controller we also made the choice that hey we do not want to do all the heavy lifting ourselves that's actually already other controllers that can do a lot of the heavy lifting so we decided that this controller should should I mean is to respond to gateway API definitions that would be kind of the things that drive what it does and this controller should then generate other Kubernetes resources so for example we could use cross-plane to talk to the the cloud API that we want where we want to provision our resources so this means that our controller just needs to to create for example cross-plane resources and then the cross-plane controller for for example an AWS cloud would pick up those resources and implement whatever our controller has generated likewise we also need a data path inside Kubernetes we need something there or we may need something there in the same way we could let our controller create let's say deployments we could also let our controller create through the gateway API a data path that can be implemented by another controller for example we could use Istio which is also a API gateway implementing controller which implement a data path already so we could also create a data path in this way this is just an example using cross-plane and using Istio our controller is not I mean this is not hard coded in the controller that we're using cross-plane and Istio because in fact we also wanted to make this data path definition generic and reusable ideally we want to be able to share these blueprints with you so that we can have a controller that that implements the gateway API and then through some shareable generic blueprints uh pluggable blueprints I think we can share be able to implement a data path that has a specific characteristics a specific design so that's an important thing of our design that is we we separate the controller from the actual data path implementation so now we're talking about these data paths so what kind of features are we talking about and why is it so important that we have control of this our teams have different requirements and needs so and some teams might just be happy with a simple data path as shown here a cloud loop answer and then forwarding that to a Kubernetes resource service and then they are quite happy we have other teams that use Istio or a service mesh so in their case they would need an increased gateways as part of the data path so in that case we need to implement that for for them other teams might have even more needs I mean some may have an opinion on what traffic should be locked how should it be locked where should it be locked to it could also be that they have an opinion on what should be the network attachment of this load balancer I mean should it be exposed on the internet should it be only on the internal network for example it could also be that they have an opinion on who should have access to this load balancer it could be that they want to to limit access through an IP side arrange or maybe maybe through ID open ID connect for example such things we also need to to cater for it could also be that they want to do more advanced processing of the network traffic some of the features we typically associate with API gateways not to be confused with gateway API so this is a mechanism that can process data in many different ways which cloud providers also support it could also be that they want to use a serverless version of load balancing because that has a different load different cost profile and so on there's actually quite many features that our teams our products are actually using now and there are features here that we want to add in the future as a part of the platform team and which we might want to introduce without the developers having to know we would like them to use these golden path and then we will be able to control what are the the features for example security features that we add to this to this golden path when we set out to design this controller we made one key design principle which is I mean the overarching design principle for for the controller and that is that the data paths or the blueprints for the data paths that should be based on templating not code that was really an essential thing for us to do because this is what allows us to evolve these data paths without having to having to change the controller imagine that we want to change the Kubernetes resources that goes into a data path and we needed to recompile our controller build a new container image and do a new release and and all that just to do a tweak on on the Kubernetes resource that that that the data path is is built out of that's really not ideal and and separating the blueprints from the controller is also a foundation for making the blueprints shareable and generic and finally because these blueprints just contains templates for Kubernetes resources which could be cross playing or Istio or anything else it also makes a gateway controller that we are presenting here cloud agnostic because you could just because we're just using we're building data paths for AWS but if you have a need for building data paths for Asia or Google then you could definitely do that I should also say if you look at this picture here this is really the mental model of of our controller think of it as a generic controller responding to gateway API definitions and creating Kubernetes resources on when it implements the data path so if you're thinking does this work for me in my scenario then the question would the answer would be if you're thinking of data paths that can be implemented with Kubernetes resources then most likely it will fit I mean you could be on Celium you could be on Asia you could mean whatever platform that you have underneath if you can present represent your data path through Kubernetes resources then most likely this controller will suit your needs okay having decided on templating we acknowledge that templating there's only so many things you can do with templating I mean it's not an ideal expressive language it's not as expressive as code or at least it's it's difficult to express the same things with templating so we acknowledge that having decided on templating why it's a great benefit of this controller it's most likely us it's also its greatest liability so this triangle here is supposed to illustrate that this controller operates within three main constraints and one of them being being templating the two others are the inputs and the outputs we are responding to gateway API resources they are already a settled design or mostly a settled design we have to abide by that standard when we cannot deviate from that because then we wouldn't be an API gateway compliant controller so we need to abide by that similarly the API or the gateway sorry the communist resources we create as on an output side they are also I mean they could be cross-bain resources or other gateway API resources there's also a fixed schema for that so the same way we cannot deviate from that so we also need to create schema compliant resources there so these are the three main constraints that our operator worked within and it's actually major constraints because going for example from the gateway API to the appropriate cloud definition of the cloud resources that we want to create that is not a wanting conversion I mean you can't go from say a gateway API gateway definition and say hey I'll just convert this through a template into a load balancer resource that's not how it works typically it will take five ten different cloud resources to implement a gateway so it's one too many here it's the same going the other way setting the proper status condition statuses and condition on gateway API resources that's not just a question about copying a single field from some resource that's typically also something that is is yeah a little bit more complicated than than that so these are main constraints we want to highlight them here we think it's also part of the strengths using templates but it also means there are yeah things that can be difficult so to help with this we introduce or yeah I have an example here that also shows where templating is is difficult imagine you have a gateway a gateway definition that says hey I will allow route attachments from from any namespace that has the label foo bar and we then we also have three namespaces that each define an hcp route and say I want to connect to or associate myself with this this gateway now the two first namespaces has the required label while the third does not so in this case it's quite obvious with that when we implement this we should ignore the third route if we imagine that the data path here also contains templates for creating a certificate for example then the question arises what host names should be in those certificates again the obvious answer is that host names from the two first acp routes and the third host name should be should be ignored I mean this is easy for us but expressing this was template with templating is a bit difficult so to to make this more easy to make this actually workable in the controller we introduce something that we call normalization and what that is is that the controller will react to gateway api definitions it will then encode with whatever we can do in code it will process those gateway api resources and do the hard things in code create something we call a normalized model a model that contains pre-computed data you can say that's easier for templating to to take and express in in the actual data path now this normalization here will then implement for example the filtering that we saw on that would have been necessary in the example that we saw on the previous slide another thing we did that was that because we want these data path blueprints to be generic this means that they cannot contain any contextual configuration because then they wouldn't be generic if we didn't do that so what we did here is that we used this gateway api enhancement proposals 713 which is about how you attach additional configuration to the to the standard standardized gateway api resources so this means that we can attach additional things additional policies or actually any configuration to these gateway api resources so in the example that we have here we have a developer that has expressed that they want to have a gateway created and we also have an infrastructure provider that has created a gateway class and a gateway class is kind of a named gateway implementation uh together with that the infrastructure provider has also uh configured that gateway class to use a specific gateway class blueprint and the blue box here the gateway class blueprint that's a crd that we have created as part of this controller it's not part of the gateway api the green boxes are standardized uh gateway api resources but the blue box is our generic blueprint that expresses or defines how is the architecture of this given given given data path and this is where the templates are contained again these are without contextual configuration so it's not implementable what we have here it's only the architecture you can say the blueprint that's in this in this blue crd here and what we're missing to make this implementable is that we miss yeah and these are the generic so so what we're missing to make this implementable is this contextual configuration so for example we have some environment specific configuration that is needed it could be vpc or subnet ideas that is information that we need to be able to implement the blueprint that says something about the load balancing for example additionally since this is a platform for a platform uh the infrastructure provider might might also have some contextual configuration that's needed uh for for the application developer here which you call a tenant so this could for example be tagging rules it could be that the infrastructure provider says whenever this tenant creates a load balancer i want it to be tagged in a specific way so that i can for example do cross distribution and finally the application developer might also have some contextual configuration that they want to attach so it could be for example the docking configuration it could be the open id configuration which would have access to the load balancer and and so on so there are these three categories of contextual configuration that we want to associate with these resources here and to do that we created two more crds we created something we call a gateway class config it's a crd that contains the contextual configuration that is needed for example the vpc and subnet ideas and then we use this gap 713 mechanism to attach that crd to the gateway class so we're basically saying whenever you want to implement a data path from this gateway from this gateway class and the associated blueprint then this is the environment specific context configuration that you should also need in in this and the same way with the tenant specific configuration here now we're not associating it with the gateway class but we're associating it with the namespace of the tenant so this becomes the default of that when we create resources or gateways for for that tenant again this is owned by the infrastructure provider the tenant cannot change this it's the infrastructure provider that are associated with the tenant's gateway or a namespace when they create the namespace and finally we have this gateway config crd which is a crd that the application developers of the tenants can create with a whatever contextual configuration that they have that they want to associate with the gateway so now this is the logical view of how we have designed and configured our a data path remember this picture because this while this is the abstract view this is what we will be showing in a demo with actual actual gateway resources actual gateway classes actual blueprints and and the gateway class config resources and and so on so Martin would you describe the the demo use case we will present yeah sure so we are describing this a full example use case that's also mentioned in the gateway api documentation so imagine we have a infrastructure team or a sorry team that defines a gateway specifying the configuration for the gateway using this host name then we have one development team the full site team who deploys their full site service and basically specifies an htp route specifying the routing to that application utilizing the gateway another team the full store team would then have two different versions of their full store service and they specify routing to that service using the same gateway with 90% going to version one and 10% going to version two so we'll try to demo this and hopefully that works mic will just set it up and we we try to visualize this using a small tool we made that visualizes a gateway api resources so we start with an empty cluster where we only have Istio and crossplane installed so what you see here is we only have a gateway class an Istio gateway class deployed to the cluster so we as a platform team now apply a blueprint blueprint and two gateway classes defining an internal and public data path and as you see here we have the three different resources the gateway class blueprint uses AWS crossplane and Istio to specify the data path this is this is just templates for now we apply the configs to set the environment specific values so the internal gateway class we have specified to use the private subnets and using public subnets for the public gateway class as well as the different tagging then we set up a another gateway class config for a tenant name space and as you can see here it it's not really attached anywhere because well there's no gateway api resources in that namespace so if we have a sorry DevOps persona specifying a gateway and applying that to the data namespace we see that it will use the gateway class config that we just applied to the namespace and next up we will whitelist our local cider so specify an acl allowing us to basically reach the services that will use this gateway and as we can see here there's a gateway config attached to that gateway that whitelists our own IP address now the developers can actually deploy their services along with HTTP routes specifying the routing to the application using the gateway that we applied and as you can see I hope it's not too small the full site service and the two different versions of the two full store services applied we have the HTTP routes and everything is connected the controller is not running yet we're deploying that now meaning that this is just set up when the controller is deployed it will pick up the resources and it will actually create the Kubernetes resources that are defined in the blueprint with the combination of the different conflicts so configured with the values that specified in the different conflicts and shortly we should see some more resources being provisioned hopefully and creating all the different resources takes a while so we may not have time to show you that there's actually we're able to curl the services but if there's time we'll we'll show you that it's basically the load balancer takes approximately three minutes to provision as you can see here we now have an issue a child gateway provisioned that utilizing the issue gateway class we also have some child htp route resources so the gateway controller handled all the different resources that we applied so as you can see here the different resources yeah so the bottom resource here is the load balancer and it's in false it's not ready yet because it's it's being provisioned so while we wait for that we will just go back to the presentation and move on to some of the learnings we've had so yeah basically gateway api is not really a trivial api to implement turns out that I mean that there's a lot of code to be done approximately two-thirds of the code base is is implementing the gateway api and the the last third is templating which still seems like a good choice it's it's it's really a balance but but it seems like a good choice for us and yeah we also had a couple of other learners and one of those I would highlight is that well the normalization process is also like it can easily become a garbage can because if you have a use case that doesn't fit into the templating then you would just fix it with code and yeah so so the code base could easily evolve and end up being a garbage can I don't know should we do you think it's still ready no not ready yet right so basically that's it the project is open source there's a qr code for it it's called bifrost gateway controller and it's present in the t2-os s github repository this is our first open source project it's also our first kubicon talk we're really excited to be here so feedback and contributions are more than welcome it could be fun to see some other blueprints with other use cases other cloud providers so on so yeah reach out and yeah yeah just one thing maybe jumping in for a question I would I would expect that a obvious question would be how does this blueprint look like what is and we decided not to show any yaml that's why we haven't shown it to you but you can think of it as I mean it's code templating so if you can think of it or you can think of it as a mix between a helm template and a crossplane composition so it's really about go templating but where you where you can reference between resource because a data pass you cannot instantiate a data pass like it was a helm chart because sometimes you need to create a resource and wait for some of the status fields to be updated and then you can pass on some of those status fields into the next resource that's why we we also have some of the crossplane composition patching mechanism if you know that into this into this design okay are there any questions there much at at the end seems to be no questions yeah that's a question great so um with gateway API not being released yet how are you guys gonna you put this in production how do you feel comfortable evolving this project when the core component isn't released yet I think we have a lot of confidence in the gateway API not really changing in the error that we have used here and if it changes I mean we are moving fast and breaking things that's no problem I also think we heard in in this coupon here that it's a plan to move gateway API to a GA this year so I mean we're not really worried about that I mean and also we're not using it at production yet we plan to use it in production by the end of the year depending on how things are and it is ready now I think yeah so those of you who didn't leave yet should be able to see that we can actually curl the two services