 Hey, welcome everyone to our CNCF, where we now how to build an internal developer platform on Kubernetes using backstage GitOps infrastructures code. Let me share the slides quickly. So share the screen, okay. So shortly to me, my name is Engindiri and I'm working at Pulumi as a customer experience architect. I do cloud transformation and enablement and I really love to continuous everything, deploy, delivery, you name it. I put also my socials there if you would like to follow me. So I'm on X, LinkedIn and so on and so on. So feel free to drop a follow. And I'm joined in this talk by Kingdom and I will pass to you, Kingdom, that you can introduce yourself. Hello everyone, I'm Kingdom. I'm an open source support engineer at Weaveworks and I'm a flux maintainer and I'm happy to be here today. We're gonna see some really interesting crossover between Pulumi and flux and kick it back to you. Yeah, absolutely, you're right. That's going to be awesome. So again, for everybody who watch this, I give you a short introduction on Pulumi. We will give you a short introduction on Flux and a short introduction on backstage before we head over to the demo and see every parts in action working. This is going to be awesome. So shortly for Pulumi, with Pulumi, you have the option and we have here this three pillars to have build, deploy and manage. You can, everything is under control with Pulumi on the build part. You can use your programming language, your generic programming language, for example, Python or Go. You can reuse your IDEs of your choice. You can use your package manager, which you already use for your language. So there is no change for you. So you just continue, you import the library in your IDE for the, and now we come to the middle part, the providers, you can choose from a huge variety of providers. So if you're going to create a deployment for AWS, for example, and you want to do this in Python, you just import the PyPy package into your IDE of your choice and yeah, start to program your infrastructure easy as it is. Then on the right side, of course, Pulumi embeds perfectly fine with every Git system available outside of every source control system. Also, Pulumi is not inverse if it not dictates you which CI CD approach you have to take. So you can reuse your existing pipelines. You can build new pipelines out of this. So there is a Pulumi CLI, there is a Pulumi Kubernetes operator and you can tailor Pulumi perfectly fine for your workflow you have. And of course, we have also, when you use the Pulumi cloud offering, for example, you can connect also your identity manager so you don't need to create some local users. But again, this is on the Pulumi cloud side. Just short again to say, okay, what are the core features, as I mentioned before, any language? We offer state management, we offer secret management. This is something you get included, but with every approach with the batteries included but replaceable, you can switch the Pulumi secrets management to, for example, vault. If you're using our vault, you can use also the AWS secret manager. So it's really up what you have on your side. You can run previews. So this is very helpful to see if any changes are there which could maybe affect your infrastructure. It could affect maybe your running service because some of the changes are maybe more inverse and needs delete and recreation of the resource. You will get the feedback and then you can embed it in your pull request flow and decide then if you want to go with it or maybe you need to ensure that the services are still running with maybe a more bigger architecture. CICD integration, I just mentioned that you can integrate it with every CICD system. You have on your side web hooks. That's really, really awesome. You have the option to connect different web hooks. We recently launched also the Microsoft Teams integration customer asked for this and Slack integration is already there. Generic web hook is also present. So that's really cool. REST API, so if you have the situation, you need to communicate with the Pulumi cloud offering, for example, and you can just connect this. We will see this now, for example, in Backstage, the Pulumi Backstage plugin is communicating with the REST API. Automation API is an advanced topic where you can programmatically call Pulumi so you don't need to have, for example, the Pulumi CLI commands running up or down or destroy. You can just use the Pulumi automation API and which is also very useful dashboards and reports. This is something which makes sense if you have shared responsibility in the team then you can use this functionality. Again, just to mention this, Pulumi is completely open source. So there is no need to put some credit card details into the Pulumi cloud offering. The Pulumi cloud offering has a free tier. You can just use this. So Pulumi cloud taking care of your states and that's it. So from this part, let me quickly deep dive or what means deep dive a little bit, give you an overview and high level overview of how the Pulumi architecture is built up. So first, this picture really you will see in several of our presentations and every time I see the picture I really promise myself that I want to change this, that I want to update this. So I have to put really a note somewhere that I need to update this slide but it will explain the architecture. So everything is around our CLI. You have to think about Pulumi as a multi, as a microservice architecture where all the services runs on your local machine. So Pulumi use heavily the GRPC approach to communicate. So if I start with our language shows, so I start for example to choose one language. Let's see in our example here we use go and I say Pulumi up. The CLI will detect, ah, we're going to use go. It will start the go language host and then communicate with the CLI to say, okay, what resources, what commands we are actually doing here. Yeah, CLI, that's our communicator here and the main core engine. And then we start to find out what kind of resources we are creating. So for example, in my code, in my go code, I want to use AWS, I want to start an EKS cluster. For example, I want to provision as free bucket. Pulumi detects this that we are asking here for the AWS provider. It will download the AWS provider if it's not already present on your machine and then we'll start to communicate via GRPC for create, update and delete. So for example, I started a fresh program defining an S3 bucket. Pulumi will see this and say, okay, and we come here to the last point to the state file as we'll check the state file and say, is there already an S3 bucket with this name present? If not, we have here create situation, go to the provider, does the create, come back with the results like the URL, like the ID and everything we get from the cloud provider will store the state in the state management. As you can see, we have the Pulumi cloud service. We have also the option. If you say, hey, I want to run this on my local S3 bucket, artifactory, you name it, you can all run them and take care of your state or use the Pulumi cloud service. It will detect any changes, give feedback in a create. Of course, everything is new. If I start up the second time of Pulumi up and maybe I changed something, it will then check and say, okay, did I deleted the resource? It will call the delete on the provider, wait that the provider did all the deletes, feedback the result, go to the state file, also remove the resource from the state file and give us all the information. If it's an update, you can imagine that Pulumi is doing also the updates and put the changed values into the state file and give me as a user the feedback. This is following properties which got changed. Whoops, I put this a little bit too quick. So let's head back to this one. As I mentioned, state file, you can handle on the cloud or you can handle on your own. So next view is our programming model to say, okay, how does a Pulumi program look like or a Pulumi project? So everything starts with the project. A project is actually, and then comes the Pulumi program is where our Pulumi YAML file lies in. The Pulumi YAML file defines which runtime we are using. If it's a Go or TypeScript, there's a runtime flag and this is the definition of your Pulumi program. We have also the concept of so-called stacks. Stacks you can think about different instances of your program. Most people, most customers use, for example, the stages they have of their infrastructure as a stack. So they say, okay, I have a dev QA and prop and then you can imagine also that some of the properties you maybe have are different from stack to stack. I always say, for example, on a dev environment, you maybe have a Kubernetes environment with no GPU supported nodes because it's maybe not necessary or maybe the nodes are not dimensioned. For example, on a prod system because you want to save money or you want to just test a different thing and the closer you get to prod, maybe you say, okay, I ramp up the infrastructure. This is everything what you can do inside the stacks. You can overwrite the values you defined or you can keep the values the same. The building blocks for our Pulumi program are the so-called resources. The resources are really like the lowest, we call it always a Lego piece to say, okay, this is for example, my S3 bucket, this is my EKS cluster and then you start to put them together and say, okay, for a Kubernetes cluster, this is the amount of nodes I would like have and this is maybe an OIC enabled and so on and so on. Every resource has an input but also has an output field. So the moment the resource is created, for example, you can get the output and now comes the magic. You can start to stitch them together and say, okay, the output of one is the input for the next resource and now comes the clever part also of Pulumi. Pulumi, let me switch here. Pulumi can detect when you create your program dependencies to each other. So when we create our Pulumi program and we don't have any dependencies, Pulumi will run the maximum parallelization and give you the quickest provisioning time because it just doesn't need to wait for anything. If we have a dependency structure with each other because for example, I want to deploy a Kubernetes help chart but the Kubernetes not only the API but also the nodes maybe need to be up and running. I can take care via the input output chain that Pulumi detects and when Pulumi creates the duck it builds the graph and say, okay, this is the dependency. This is the tree I need to fulfill. Then first the Kubernetes cluster gets created maybe depending on the cloud provider also the nodes get created, gets attached and then I can use the installation of a help chart because now everything is up and running. So here we have a dedicated chain. Last but not least, you can also create outputs from our project and this is something I will show also in the demo. So we can use, we can set inputs to a project. We can get outputs and use the output from one Pulumi project as an input for the next Pulumi project. This is called stack references and this is very powerful and very interesting. Whereas potential use cases, one of the use cases could be we have a separation of concerns inside the company. So for example, the networking team is taking care of creating the network and tell the people who are consuming for example, a subnet and so on just as an output the ID of the subnet. So when I create now my application or my infrastructure on top of the networking I just reference to the subnet and use this for example or I can separate also with different velocities. So maybe the networking layer doesn't change so much. So I take every part out of the less quicker changing parts and run them and things which have maybe a higher velocity because it's application based or it's development team working on this can be an own Pulumi stack. We see this in the demo and it's one of my features which are really powerful and then we come into the situation what we call it Pulumi micro stacks to say, okay, we go away from this monolithic infrastructure definitions where it takes like 20 to more minutes depending on the count of resources you have to roll out any changes to smaller packages to smaller stacks which give you a quicker feedback and you don't need to wait so long. And of course when something happens the blast radius is also not that deep because you maybe not accidentally delete your S3 buckets because there's three buckets belong to a different stack where you don't have any rights. Okay, that's a lot and I could speak for ages for this but I would like to go to our second component and pass here also the mic, the virtual mic and to Kingdon and say, please Kingdon explain us a little bit about flux. Okay, perfect. Let me get my screen share going here. All right. Hopefully my slides work, can see everything. All right, so this is going to be the very brief, very easy for beginners flux in two minutes. The main concept here is the user experience should always be simple for devs. So the developers are just pushing their changes to get the desired state is what we're talking about and they go automatically to Kubernetes. So this is very simple. So there are a couple of variable, potential variable things in here, the developer pushing changes to get. Get is just a version store here. So we, here's also the possibility that something goes wrong. We'd like to know when everything is fine with a notification of some kind. So this is all part of GitOps and developer pushing the changes to get is only part of the story. So there's also this concept of reconciling and drift. So if you've created some infrastructure and the infrastructure is changed in place, it can drift from what is defined as a desired state. So GitOps will put it back. That's part of GitOps. So this is distinguished from regular imperative CI CD experience that many people have already been familiar with for many years by the fact that we're using declarative configuration artifacts, which are generally idempotent. They can be applied as many times as we want without changing the semantics at all. They should be reconciled continuously according to GitOps. So that's from the GitOps working group. And so back to the easy version for our visual learners. Like I said, this should be very simple. The user is pushing to get and it goes to Kubernetes via Flux. But there are some ways that it's a bit more complicated than that in technical terms that make it better. So what we're talking about is on Kubernetes, we have the ability to subscribe from a customization to a Git repository. These are technical terms for moving parts in Flux, but the customization is actually doing the applying. The Git repository is doing the fetching. So these both run on an interval. And as we said, Flux is hosted on Kubernetes, which means we can negotiate that subscription automatically. But what we can't do is set up a subscription automatically with our webhook on the internet. So GitHub or GitLab or wherever we're hosting our source code probably has the capability to send us a notification right away. All of the major providers have this capability and you set up a receiver so that reconciles the Git repository right away, which then notifies the customization. It's time to pull and apply the changes to Kubernetes. So the experience though should be very easy and like this, very fast for the user. And you should feel right away, you push your changes, they're on the cluster. So that requires a little bit of extra configuration in some cases, but on Kubernetes we can negotiate subscriptions automatically. So some places it's needed, some places it's not. And if you'd like to know more about Flux, we're gonna see, but also there are public dev meetings. You can come ask us questions in various forums. Flux has developed under the open in the CNCF and we have a public roadmap and a transparent RFC process. So if you wanna participate in our development, also we offer support so you can contact me if you'd like to follow up on any of these support questions that you might have after you try what we're gonna show you. So one more thing I wanted to explain about Flux is the bootstrap process. The typical bootstrap process, which is different than what we're going to see today. So like this is GitOps and Flux is doing GitOps. So how does Flux itself actually get installed on the cluster or is that where it goes? Does it go in Git? Is that a job that we can delegate perhaps? We'll see. The answer is yes, of course. It does go in Git, typically that's the default, but it is up to you. You can change the way that you install or use Flux. There is no one way. Flux is also called the GitOps toolkit because it's flexible, you can build it. So copying Flux into Git, that is an optional decision that you can make. It's typically what we do if you're using bootstrap according to the manual, but so real quick, what is bootstrap? In case you've never seen Flux before, the definition of the Flux controllers are called components. They go into a file and then the other file that's created here in this directory structure is the business-ended Flux, the config. So we'll see more of that as we go along, but there are a few other things that bootstrap takes care of for us. And the security issue of how do we, so this is probably a private repository where we wanna put our infrastructure. So we have to negotiate a way for that to come into the cluster. So authentication and we'll need right access, but only temporarily since we're writing to the cluster. If we were doing bootstrap, those are all things that we would need, but like I said, that is an option. We can use an operator to install Flux or we can use one of many solutions that provide Flux. There are infrastructure vendors as well as cloud providers who have their own Flux distributions. So Flux itself also has several different ways that you can install GitList Flux. This is one of them. And OCI repository, I hope the sketch is on as a way that people start distributing their own software a little bit more. But so rather than dwell on that too long, let's go back to Angin. Can you take over the screen share and let's see the demo? Okay, I just used your time to charge my headphones because I heard them that they're beeping. So let's switch to them. Okay, so you can hear me perfectly? Yes. Okay, perfect, fine. So then thanks for the introduction of Flux. And you will see also in our demo some of the things we saw on the slides. For example, the OCI artifact, I really love this and I really try to use this as much as possible. For example, our Pulumi operators also the HelmChart as OCI artifact available that it's actually currently the only way to get this. So that makes things very, very easy, especially with our customers because sometimes they have an artifact to rear harbor, they can use this as a proxy and just have it internally always available. Okay, so I will activate the share again and switch back. So, okay, now we come to the last part. Kingdon, you can see this, the screen. Yes, backstage, I see. Okay, is the little window disturbing here on the right side? Maybe it's... No, I don't see it at all. Ah, okay, then it's just me who see this. Okay, so backstage, what is backstage? It's one of the pieces here in our platform tool. And yeah, backstage is similar to all of the products and projects we see here an open source tool. It was created by Spotify and then later, oh, you cannot see this very well with my slides, it was later donated to the CNCF. So it's probably currently, is it a graduated or a sandbox project? I have to look this up, but it's around this to the maturity level of the CNCF. So that's awesome. So you can use backstage inside your company. You don't need to be concerned about any license changing and so on. There is the CNCF behind this. There are also office hours you can participate and so on, but I think that's not in the scope currently. What does it offer us? So if we have here a front end, we can use backstage as a front end here as a communication with our customers, with our internal customers to say, okay, we offer them a way to interact with our platform to be tunneled to have a selection of golden paths to say, okay, I can choose from a golden path which is maybe the agreed way on provisioning on deploying stuff, on creating stuff because it has a scaffolding functionality. So I can create my scaffold of a project, of a infrastructure, what we're going to see now. And then I can just amend it with additional information from the teams. Maybe they have some requirements they want to deploy it in a specific region. We can gather the information we can deploy this. Second part is also we have another way for visibility. We have a tool where we can see what is already in place. So imagine you're in a bigger company and you start typically working on some enterprise service like the typical LDAC lookup, there's an AD directory, whatever with all the people working in the company. And then every team, I worked in a place where every team created the hundreds iteration of this kind of service, reading out the directory of the company. Now you can see inside backstage what is already there. There's a powerful search functionality. And then I can say, hey, there's already a team working and I can see the name of the team who belongs to, maybe it's part of a bigger project I can reach out or maybe they published also the API as an open API specification inside backstage. And I can already start to consume this. So very, very powerful. And this leads here to this Venn diagram to say, okay, we gain on speed. We have now operation and self-service, everything at scale and now comes the funny part and chaos control because we know there is now one way to provision to work with the infrastructure. So there is now the 15 iteration of fragmentation. Everybody has their own way to provision maybe an EKS cluster. So we reduce chaos control and guess what it's in the middle. Of course here, we see the crossing points of this is backstage paired with Pulumi and paired with Flux. And I will take a screenshot with of this because I love this very much, this picture. So, okay, continue. So we just talked about this. So the benefits, of course, yeah, we can now roll out compliance and best practices to our internal community, to our developers, for example. So we don't need to take care about if how stuff is created. We know there's always this way and it has already the best practices in place. We have the central localization of the whole software we're going to manage. And of course we have here a software ecosystem. I just talked about the collaboration possibilities we get with backstage. Part of this what we see now, and this is also an interesting thing because I used the power of backstage, the customization and the extensibility I used here in the demonstration because we installed here not only backstage and doing some configuration to talk with GitHub and so on. That's one part, but we've worked. The company, Kingdom is working. They have a backstage plugin. The company I'm working for, Pulumi has also a backstage plugin. So we just amended the backstage installation with both of the plugins to give you the, to give here the demonstration and you when you try this out, the most visibility of everything. So you see the, what Kingdom just mentioned, you see the Git repository, you see the helm releases, you see it's really amazing and you don't need to leave here. Maybe you don't even have access to the Kubernetes cluster, but now you have a place where you have the possibility to see what is installed and could even trigger, do some basic day two operations or do at all operations what you could maybe not do before restarting a pod, for example, maybe you have to call your operations team and ask them, hey, can you restart the pod? It's behaving strangely. You can do this now on your own. Okay, talk much about backstage. It's an important part of this whole webinar. Let's head over to our demo. So asking the slot machine and say, okay, what are we going to use? Yes, we're going to use go. We use AWS and we're going to use an EKS as part of the infrastructure. Yes, of course, that was not a random thing. It's completely random. It's completely random. It's not staged and so on. So I'm sure if I could trigger this, but we're going to stick now with go AWS and EKS. Again, I give you a little bit more detail about the architecture because it's a little bit more sophisticated than the slot machine is telling us. So first of all, this is our very, very basic infrastructure. So we have our GitOps platform and behind this GitOps platform, this little avatar on the top is our platform team. They own the GitOps platform and they're like, okay, how are we going to set up this? Again, please, this is an opinionated solution. Every GitOps platform may look different depending on the company, depending on all the circumstance. And this is the power also, Kingdom mentioned, you can adapt everything on your needs. So this GitOps platform consists of one Pulumi stack. We just had the presentation about different possibilities with MicroStacks. So we created here a Pulumi program. I called it GitOps infrastructure, which defines one part of the GitOps platform. And this is the EKS because we said we want to use Pulumi, we want to use Kubernetes and the power of Kubernetes, Kingdom had also the five pillars of GitOps. And we use Kubernetes as a control plane here. So I provisioned an EKS, of course, BNCs and all this stuff, everything, VPCs, everything is also behind this. I just dropped this, otherwise the picture would explode. So first part, second part, the second stack, and this is a separation again, it's an architectural decision we made here to say, okay, then we have the backstage installation. We don't let, for example, backstage run as part of a container and so on with maybe our Kubernetes cluster, just in case to say, okay, we want the backstages maybe always available, we want to connect other systems to it and so on. So we just separated this one and I installed backstage here using a Fargate, for example, a serverless approach. So I create a Docker image out of it, a container image with my backstage installation with all the customization I did, installation of the Weaverx plugin and everything, upload the container image to ECR, configure Fargate, configure managed database here, Postgres database from AWS and everything gets delivered via the application load balancer of AWS. So this is the backstage installation separated from each other. Next point is we're going to install Flux. Flux is the ignition here in this case. So I installed, and again, this is an optimization, we had a discussion before we did the talk with Kingdon about, hey, there are different ways and so on, there are also downsides on some of the installation. You have to see with what you can live here. So in this case, I installed Flux using Pulumi, using the Pulumi resource called Helm, the Helm resource and I just installed Flux but start then to say, okay, now everything else will be managed by Flux. So I create the custom resource of Flux called Helm release. So they coincidentally have the same name. The Helm release custom resource is already in the Flux world. So it will be delivered via Pulumi and also saved in the Git repository. And so yes, of course, you could also do then some kind of, hey, I managed myself using myself but here in this case, the Pulumi operator itself which is an OCI artifact will then be delivered via Flux and not via Pulumi but it's defined inside Pulumi crazy. Next thing, I mentioned this before that the backstage installation, installation of the Pulumi plug-in Kubernetes and WeWorks plug-in, that's fine. Everything points to a GitHub organization. So I created a GitHub organization, everything is wired up with a GitHub application set up so backstage can communicate completely with our organization can do creation of pull requests of creation of repositories everything what we maybe need. And now comes the last point when we then release our GitOps platform to our self-service GitOps platform to our development team. The development team just communicates now with the backstage, with the front end. So we have a unique unified interface with our customers and then does everything they are allowed and they are agreed on to do provisioning, seeing, communicating. Okay, that's the infrastructure. So let me show you now the pieces in action. So I let my screen stop the sharing and I start closing my windows and share the first repository. So now this happens what I was afraid which going to happen. I don't see, here it's okay. Always technical difficulties. You can't have a live talk. It's, yeah, you're not supposed to do a live talk. Okay, so this is the organization I talked So there are three repositories. One repository contains our, where is it? Yeah, contains our Pulumi code for managing the infrastructure and so on. We will go, I will share later also my IDE for this one. So no worry, we'll see a little bit more details. This is one of the repositories. The second repository, I clicked the wrongly here defines now the template we're going to create. So that's very important to say the backstage. This is the scaffolding I created. So we will have a look into this one. So, yeah, this is backstage terminology. So I say to him, hey, get me some input of the developers. They can choose from some things here. In this case, we're going to create a static website. So this is all the informations I gather. I need the stage and so on. Then I'm going to render all the informations. Everything gets rendered into the system. And last but not least, we also see here the pull requests. So we would like still that platform team, maybe we don't have the confidence or the maturity or it's not part of our release process because of whatever regulation there should be still a human looking on top of this. So we create a pull request, people get a notification. You can now attach GitHub actions or get workflows which maybe also do some automating testing to give you some feedback and we will get the output of the URL. So that's one of the repositories. And last but not least because we would like to point flux to unrepository. There is this Pulumi infrastructure repository. So everything what gets provisioned will create a pull request in this repository. I created a customized folder. We're going to use the customization resource of flux to point to this folder and say, hey, when there is a change of the share of this Git repository, please have a look into customization folder. I did not put a customization yamble inside of this to organize. I'm like, I don't know how much there's inside and I don't want to work now with adding stuff now we're on the resource tech. So you can define flux like this to say, look into this and then create on top on yourself a customization virtual customization yamble but everything which lands here will then automatically picked up from flux deployed on the Git repository and then the Pulumi operator kicks in because it listens to the specific CRDs of the type of program. We will see this in the code and then starts to provision the stuff. Okay, without any further thing let me share the IDE. Now we come again to the point where did I put this stupid IDE in? That's not the one. Okay, so you mentioned Kingdom that you could see this very well. So I take your word on this one. So this is the first deck on our architecture diagram the GitOps infra part. So everything written in Go. So you can look up the definitions and so on but to give you a short insight and everybody who worked with AWS and EKS knows that I cannot start now a fresh one because it will take depending on the load on the region up to 10, 15, whatever minutes to get the things up and running. So EKS can be sometimes very slow. So we create our VPC we create everything we need for the networking Then we see one of the features already here we export because we want to consume some of the networking informations in our other stacks. So I'm not going to redefine the VPC I'm going to reuse the VPC but I export some of the informations already here. So another stack can consume this. So for example, the route table and the VPC ID. The next part is I define here my Kubernetes cluster so I can set up the informations and accordingly what I need. If I'm coming now into the situation and say hey, this should not be hard coded this should be separated. I can put this in the config file so there's a Pulumi config file where I can define now everything which should be configurable. I can take this out here in this case it's for demonstration purpose. So it's here, it's very hard coded and any changes should trigger has to be changed in code here. Then we finished everything. This is the state where I say, okay I'm going to export also the Kube config this is sometimes very helpful if you want to run for example K9S or KubeCuttle commands. You can get the Kube config file from your cloud provider. What I'm going to do because everything should be programmatically I'm already starting to provision a new so-called Pulumi provider. In this case it's the Kubernetes provider and I enabled server set apply because there are some things I will mention later which is due to the opinionation of the Helm chart I had to patch some stuff. So I created here now the provider. One thing you will see here now is the backstage label. Part of the Kubernetes plugin of backstage is that you have to annotate every resource every Kubernetes resource with this label. So now you get the idea why I want to use patching and so on because not every Helm chart thought about this that we're going to use backstage and you're not able because of the opinionation to set in the pod template spec and so on this label. So that's the thing. But it's very important because that's the way backstage know how to look up the stuff. So here I gave it a Kubernetes ID. This is later configured also in backstage. If we don't do that, we won't see it on our dashboard. That's what you're saying, right? Yes, yes, the lookup of the Kubernetes plugin works with this approach. It's interesting and makes some forces us to do some customization here with the values you see. Now I start to deploy the flux. So I'm using the OCI artifact here. Just push it because it's already, Pouloubi uses the latest version of the Helm SDK underneath so it can handle OCI artifacts also. And now you see, I have to look up now the values file and check where are the labels set up and so on. So this was very painful because I have to go through all the stuff and set the labels. Yes, if you're in a production environment, you already maybe set stuff up because you want to have the service monitor and so on and so on. In this case, I had to go through this. Now comes the part what I mentioned also. Some of the stuff were not able to configure. I should do a pull request later into the flux community provider and also the Pouloubi operator because there's also this situation. And I patched now the deployment. So you see what I talked about dependency. I put now a hard dependency and said, hey, please execute this part of the code only when the flux part here, the flux release is done because now I want to do some patching. So for this, there's a resource in Pouloubi called deployment patch. And now I go through the deployment patch and I just set the labels, for example, on the deployment because I would like to see the deployments in backstage, very painful. You don't need this part when you're Helm chart, when you write your own Helm chart, for example, and you have full control to this third party Helm charts either you fix it on your side or you do it on upstream. In this case, I did it on my side. Now we come to the next part. I define the Helm repository resource and I tell now flux, listen, you're going to deploy now the Pouloubi operator. So I define the Helm repository here. Again, I use the type OCI because I also want to deploy the Pouloubi operator via the OCI artifact. And again, because we would like to see in the Weaverx plugin, we would like to see also the information. Of course, we have to annotate also this resource. We have to set the label and then we use the Helm release, everything's done. We come to the next point. We now create a Helm release programmatically, same thing, annotating the labels. And you can see now here, I also do some additional stuff here. In this case, I say, hey, Helm normally don't do the CRD parts. It's not part of it, but the people from Flux or the community around Flux created also a nice approach to take care to overcome the shortcomings with the CRDs. So here in this case, for example, I say, hey, when there is a new Helm version, I roll out a new version of the operator and there is a new CRDs. Please replace them. That's very, very cool to overcome this stuff. That's one of the most important features of Helm controller for sure, is the ability to upgrade CRDs. And for anyone who's not familiar, intimately with Helm, you should understand if you're using Helm to install an operator that has a CRD, it could be problematic. So this is a solution for that based on Flux. Yeah, absolutely nice. And I love this, that it's already thought about this because otherwise you start to separate the stuff and you say, okay, first deploy the CRDs. We are a GitHub raw YAML to the repository with the specific version, attack and so on. And when this is rolled out, then please then do the Helm release. I mean, you can imagine this makes things life very complicated. Now I had to use also some advanced topics of Flux also here, so I deployed everything. Now comes the situation that the Pulumi operator is the same didn't thought about that people going to use backstage. So I'm in charge of this. So I'm going to take a pull request on this but for the demonstration, I didn't want to change this. So I'm using here now the post renderer. This is also very, very cool. This is part of the Helm inbuilt into Helm. So you can create post renderers. The cool thing with Flux is that they already have some post renderers, for example, customize. So in this case, I say, hey, use customize and use strategic merging. So I could define all the missing parts on the deployment and just say now do a strategic merge of the stuff and it's not going to break. So I just created now here and the missing things I say, hey, you will find that deployment and the name of the deployment. So if you find the deployment with your name Pulumi operator, then please add this to this and then the spec template at a label here and also a matching label. So then you're completely fine. You have also the option to use the patch j since 6,902, whatever that means where you can use op path and value. And I think it's probably nice for some cases where a strategic merge is not possible because it's not deterministic or whatever. I'm not so deep into the law of the whole JSON stuff but you could use also this one. Just to add some color, the two different patch types they still sort of notionally exist but in a modern Flux and in modern customize they both been replaced with patches. So there's one directive you can use now called patches. For anyone who's not intimately familiar with Flux this stuff all probably looks completely foreign but if you are a Flux user, hopefully this looks familiar if not uncanny. It's like he's been saying another one of the most important features of Flux. Yeah. And yes, that's probably due to my because I worked started with customize so this was part of the customer so I felt welcome. I will check the patch one. So yes, you have the option to use a post renderer which I found very, very useful. So in this case I overcome a shortcoming but I think maybe it's for your company where you are obliged to label the team, the cost center and whatever. So you have now the way with Flux to do this and then I do some magic for example the Pulumi access token using the Pulumi secret engine underneath and I just set the Pulumi part as a secret and you can see the seer. It's completely not visible. I can commit this. It's not possible to decompress this or decrypt this and that's it. So that's the part to deploy. Last but not least, we also have to say to Flux, hey and don't forget there is a repository to look for any changes where all the stuff for Pulumi lies in. So again, I define a git repository. I label the stuff because I want to see the stuff and here's the magic. I point now to the Pulumi infrastructure repository which we saw before which is containing now every provisioned stuff. You can go much more crazy with this. You can separate them. You can say, okay, everybody has their own repository. You can go a mono repo approach. This is just an idea. In this case I said, okay, everything is in a bucket style. It's inside. I don't care. So I'm not sure if it's just me but there's something happening to your microphone. It seems like if you have a backup microphone you might want to switch. Okay, that's probably that the battery is dying. Is it the microphone? I'm not sure. You're still completely intelligible but once in a while it goes into like a weird vibrating sound. I'm not sure if it's. Okay, is it better now? It seems to have stopped. Yeah, I'm not sure. That's probably my, I mentioned I used the time when you presented to charge them but yeah, they won't hold anymore. So, okay, I mentioned this one. So I create a Git repository, plain Git repository pointing to a Git repository. Backstage is, no, what I say with backstage. This is a public repository. So I don't need to set any credentials. I could set some credentials, SSH keys in this case. This is a public one. So I don't need to say to Flux say you will see them because they are publicly available. And now I create another, not confusing with the customization approach you normally know with the customization. Yeah, no, this is a resource called customization. I'm sure that you can explain this a little bit better than me, the fine details but this is a customization resource where we point now the source ref to the Git repository. And now I also set the path. I say, hey, please look into this folder. You will find Kubernetes manifest. There is no customization YAML, which is client side and is not deployable to a Kubernetes cluster where we can do some patching and all this stuff because here I did not want to take care about YAML and adding stuff dynamically. As I said, I throw the stuff in and good guy Flux has this functionality to say, I got you when in the folder there is no customization YAML, I can still handle this because I will take care to create a virtual one. Ooh, that's it. So it's deployed last part service accounts that for example, our backstage user can see the resources and so on. So as a good security guy, I of course gave the backstage service account the cluster admin roles. Yeah, you need to nerf it down for your use case. I just shot with the highest thing on this one and gave him the cluster admin and yeah, that's it. So reading, creating a token for this backstage user because that's the way we configure backstage using the service account token. So the API URL, the service account and the certificate are the stuff we're going to use to communicate with the Kubernetes API. So that's everything prepared. You can see also I exposed the endpoint. Cool, that's a big thing. So now we come to the last part before we see moving picture from the kingdom. The backstage deployment similar approach just we use different resource. Again, we get our VPC ID because we say we're going to create new subnets on the same VPC. So I just created new subnets availability zone. So this is for everybody who is into the cloud architecture, very interesting. I created also a load balance and this case HTTP because there is no easy inbuilt way from AWS to give us an URL with HTTPS. HTTPS, I mean, some cloud provider offer this here not and I just didn't want to point some of my domains on this for just the purpose. So it's HTTP, don't do this in prod. Created the listener. Now we define the Postgres instance. So we create an irrational database of the engine is Postgres. And again, here just username and password. This is not a public one. So that's fine. So create the instance. It's not public accessible, but in the same subnet. So our target can communicate with this one. I created an ECR repository to upload the dog image give it a life cycle policy. So I don't have hundreds of images lying around doing the I am magic of what the Fargate instance can do because it's assume policy, I assume a role. So this is everything set up. I just go over this because it's not that interesting. Then we come to the part where we create our dog image. So as you can see, I checked out backstage and this is now goes to the way backstage is working. You need to clone it. You need to change the code pieces and so on. I think there are enough videos to check on backstage. If you're interested, just short you just create a new backstage app with the CLI inside the folder and then you point the Pulumi Dogger provider to it and say Pulumi, please build now a dog image out of it. So every time there is a change in the folder because I do some customization, Pulumi detects this one creates a new image uploads to ECR. That's the reason I have here the life cycle policy to reduce the amount of stuff. And then I do a Fargate task definition, put in here some informations inside what we need and so on and so on. Network security groups to nerf it more down to say here in this case, I again opened it quite widely to say, okay, ingress, egress is quite open, but of course you could also tune this down and say, okay, the Fargate instance should only expose the port of backstage. There is no anything else to do this. That's fine. Last but not least defining the service, the Fargate service launch type, Fargate and so on and so on pointing the load balancer, poof, everything done. So these are the two ways to run the stuff. If I want now to deploy this, I can say now pull me up, I can say pull me preview and it will run and I'm sure in this case, my AWS token is expired. So let's see, I'm doing a pull me up. I did not change anything. So normally I'm not expecting any changes. Minus Y minus F is just jump over some, how do you say this, asking, telling us, hey, do you really want to do this and so on? So here in this case, I know what I'm going to do and you see here, I get no changes. There's nothing is changed. The image is still the same. There is no change on this one. If I go the same for the pull me Github's infrastructure, it should also not give us any changes so I can be very confident here to run the stuff. So let's finish this one. Again, if I would start something new, it will give me the full list of changes, creation and so on. And then I can decide on this one, but as you can see here, there is no changes and yeah, everything is set up. And as you see, anything which is secret is automatically hidden by pull me. You can also set to secret manually. So you never, you have all the means in the hand to not leak any information in your state file. So pull me as an auto-detect and you can still set by hand and say, hey, the cube config file is a secret. So please to secret. Okay, that's it. So I pass now to Kingdom for Kiria's some action. Okay, so we're gonna try it. We're gonna try the user experience. It's a lot of work, I noticed it. It's a bit, let's hope the demo goes with us. All right, so I know that we've got Git repository and we've got a, you gave me the link for the dashboard. Yes. I'm trying to find our history here. Okay, so we've got our backstage dashboard which appears to be up and running. That's pretty cool. And we can click around a little bit and see, let's look at flux first. Okay, so we see flux, see some details about flux here. And we, I know we can go in to see the Kubernetes definition of flux and what resources here. So we have pods, these are all of our flux pods. Looks good. Okay, and we've got some flux resources. If we want, we can interact with them here. That's great. All right, so what should I be clicking on now? And you can show the Pulumi operator also, if you want, because that's, you see also the dashboard. When you scroll up a little bit, yes, just a second. So perfect. And then you can see also the Pulumi. So we aggregate the informations of the Pulumi cloud here. So you have all the informations by hand and can see the stuff. So what you can do now, you have now the role as a developer, you got told to create a new component, a new static website. That's one of our paths. So you go to home. Okay. And you go to create. Just have to make sure that everyone can see the create button. I'm moving us around the screen here so that we're not in the way. Yeah. And now you can see now, before we click on this, you can see this is one of the golden pairs of the templates. When you click on the icon of the GitHub of the OctaCat, you can, on the left side. Look on the GitHub here. Yeah, on the GitHub icon, you can see the code also of the template. That's very cool because now we have a transparent, a transparent way I can participate. So Kingdom can say now, hey, I don't like this. Oh, hey, there's something missing. Create a pull request, tell the platform team that there is something missing because you as a dev team maybe know it better what is missing. So that's one of the thing. So there is a definition. It's all wired up as we said. And yeah, that's, I think I introduced this before. And if you go back, you can now say choose and are guided through some topics. Do you want to use an HTML tag? Oh, well, that's a great idea. Yeah, so this is a website, right? Yeah. I guess I need my head tag here. Been a while since I've written an HTML document in full. And I'm just going to put, well, we'll come back and we'll put a link to the site. Okay, so this is actually, this is supposed to be the text here. So, all right, there's our HTML content. I didn't leave any closing tags out, right? I think that's fine. That should be fine. Looks pretty good. We'll see if it doesn't work. And the input to other things depends now. You can get more information out of your people. So this is just my thing. So the select stack, you can also hard court and say everything which gets created via backstage is always development. And then you do the propagation of stages. Maybe on a different approach. Here, we want something big. People can see, that's better. Hope that it's working. Okay. And the stuff, so that's fine. Okay. We mentioned we don't do instantly delivery. We want also to have a pull request. So, Kingdon opened the pull request. Here, we can see the same typo that was in our template. That's okay, we can fix it later if we want. Yeah. And here's the component and the program. Okay. The component is important. So that's the reason I created a folder with the name of this project because backstage is always looking for the catalog info.yaml. So I want that it appears in backstage and backstage is configured to scan the whole organization for this specific yaml file automatically. So I get out to discovery of all my stuff via the catalog info yaml separate. I mean, I have separation, but still aggregation at the end. So you do what a good developer always do. You approve your own pull request. And then merge it straight away. Who's created it. So you can assume it's goody. So, and now I share quickly my screen because we would like to see some action on my side. So, okay. So here I'm now connected via K9S and Q config. Yes, of course you could do it. So I'm now the platform engineer again. And I can see now under CRDs. Let's look if our program, yes, our program. No, it's not delivered currently. So maybe you can trigger the, because Flux has an interval how often it checks because maybe you don't want to stress your API so often with stuff like, hey, keep on checking the stuff and so on. So to trigger the story, you maybe can let me look. I'm going back to the Flux UI here and source I think is what I want to synchronize. It says it's 57 seconds ago. Yeah, I use the Flux reconcile command also. So you can do this on everything. And let's hope that it's working now. As I said, if the demo gods are not hating us. Okay. So you didn't call it. The NCF demo, I think. Okay. So what is the problem here? So he says, yeah. Okay. There's malformed Yammel. Okay. You know what? I bet it was that HTML tag. You told me what in there. It's the HTML tag. So what you could do. Just go to the pool to the repository. You just approved the pull request and fixed the part there. So. Okay. I'm going to share my screen for this part. So now we are in the Github's world and we set up this repository. So we can fix. Okay. So that just went in here. It's in customize. Customize. Yep. And then I would go. Yes. Into the content field. What does he say? Okay. I'm going to share my screen for this part so everyone can see. What does he say? He says line 44 is malformed. Yep. I can see what it is already. I think we need to make sure these are all consistently indented. Yeah. There might be something we can do in our. Backstage template to make that work. So let's just put some spacing in here. And then we should have a valid Yammel. This is very cool. Yeah. This one goes on me. I, but it's good, you know, we test stuff. So it's always good. Everybody has this colleague who always breaks stuff. I had this every time, you know, this one guy who is really able to break everything you should create. I'm not sure if you can push. Okay. You can push. It said I could do it. So I didn't create any branch policies because normally that's not the thing. Okay. So that's completely fine. I'm just going to sync this so that we're. Like I said, if you have a receiver, you won't have to do that, but we haven't configured web hook receivers. So hopefully we can see. Okay, it's there. It works. So you can now click on the dashboard and the polumi console. And can see also the rollout. So you see there's already an update happening. So, um, you have to go. To, um, yeah, I don't know why the first one failed, but you should get. Um, the URL. So under overview or you can copy it from there. Oh, there's a. Oh, okay. That's where you meant to find it. Great. Yeah. I mean, we could also create it so that the output is also visible in, um, in backstage, but yeah. So, and then of course what I, what I want to do now is go back and fix my, uh, link so that it actually links back to this page. Where. Where did our, um, I mean the wrong repo, aren't I? Um, which repo you're looking for. So. This one, I guess. Yeah. We're going to fix the link. This is bad link now we don't. Ah, okay. Yeah. Yeah. Of course. Yeah. Of course. Yeah. You can now. Okay. You, you absolutely understood this now why it would maybe make sense to give every developers now, um, the link influx to their own repository. Um, for example, you could now here make a pull request to additional git repositories. And then point just to the git repository, what, what the development team is, uh, is, uh, owning and then they can change the infrastructure on their own. So. If you, you changed it already. Yeah, I just changed it. I pushed the change and we'll probably don't see it yet. Yeah, it, uh, you can see in the Pulumi, uh, activity monitor if, um. If the change is already propagated to, um, so when you click on death again to the stack. To the stack on death and activity. And it's not so it looks like you have to trigger flux again. The flux, um. Repository refreshed. I have no interval intervals, not too long. Yeah. That's always, uh, when I do a demo and you're probably too, it's like, uh, in real life, you don't wait like this two minutes. It doesn't pain you, but in a thing is like, geez, something didn't work. So let's hope that. This is working. The link works now. That was fast and easy. Yeah. So it's a simple demo. Now we have a static page and so on. And of course you can now blow this up and say, Hey, it's not a customized. It's a helm release. I'm pointing to it's a multi repository approach. I go, but I think you see now, um, the way to handle this and, uh, work with. Okay. Then. Let me put the slideshow back. So play from not from start. That was here. I've never used backstage before. That was delightful. Thanks. That's, uh, that's cool. That's, uh, I miss. Let me share my screen. Okay. And we say play from current slide. Okay. So we saw the demo. Now. Wrap up. This we do. Uh, I mean, we saw now. And we talked about this a way that we created now a platform with a highly self-serviced approach in this case. Yes, it's a static page and so on. There is much room for improvement, but the engine, the automation works. You can. Get stuff. You get information. I mean, what is missing here would be now to say, I need a Prometheus, I need that YDC and so on. That's all things you can now create, creating a service in AWS. So kingdom would not look into some Git locks events, but would look up. In, in, in a Prometheus owner, Grafana and stuff. If you want to know more, here are some follow up links. So the link to backstage, the link to weave works, pull me with all the documentation, direct links to the flux CD. And of course the link to the demo repository. It's here in the bottom. And if you have an AWS account, you can easily just start, pull me up and you should get the infrastructure up and running. If you want to get backstage up and running, you have to see that you add the production file to it. Currently everything, if you run it like this, it will not use Fargate and everything because it's the local stuff doesn't use this. It uses an SQL light. So feel free to adapt this then or reach out for us and we can help you also to get the backstage up and running. But with this said, that's it. Thanks so much for joining us everyone in the audience and thank you and game for the wonderful presentation. Yeah, thank you too for joining and looking forward for some additional sessions, especially with the subscriber pattern and the receivers. So this is something we should definitely see more in action. So where should people go if they want to address us with questions? Well, let me think. You can come to the flux channel on the CNCF Slack. That's one place. Pull me is also community slack. So hit us up. I'm also in the CNCF Slack and in the flux Slack. So yeah. Yeah, all right, great. Thanks everyone. See ya.