 It's the last talk of the day, so thank you so much for joining. I hope you're not too tired. I'll try to entertain you and make it as fun as possible. My name is Joaquin Valiz. I work for Microsoft. I'm a software engineer. I live in Austin, Texas. This is my social networks. If you want to connect or if you have any questions after the talk, we'll put it through a chat. So yeah, so let's just start with a couple of clusters. I do want to do a special thanks to my friend out of Tundra. We were supposed to present this talk together, but we gave Mary some time to come. So here I am. So let's do a quick introduction about how we run application employment in the past so many years, right? It all started with punch cards. Anybody here used to do punch cards? Not actually, but anybody in the audience? No. Okay. So back in the day, a programmer would have a set of punch cards and they would feed it into a machine. And the machine would do the processing and then it would give you back an output. Obviously that was so efficient and we have gone a long way from that. So from there, we started seeing modern operating systems. You know, Linux, command line interfaces, and things got a lot better. We started seeing Docker and version machines, doing things with containers, and things got much better. After that, we started seeing the container orchestration such as Kubernetes, and things got much better. After that, we started seeing things that will allow you to automate the deployment of applications in Kubernetes, such as Argo and Flux, and things got so much better. After that, we don't know what's going on. There's these things about OpenAI, HHMT, this is the future, some say that it is, some say that it's not. Who knows, maybe in the next few years we'll just sort of find out. Okay, so for today's presentation, I would like to focus on GitOps. So we'll be talking about GitOps operators such as Argo and Flux, but then we'll be using Argo, but it's also going to be going to Flux. So let's talk about deployment to Kubernetes using GitOps. So let's understand the journey to GitOps. So even before we started seeing GitOps, everything started doing insanity by applying the Amnus record cluster. So we'll be using CtlApply. And we'll set up many deployments, we'll set up scripts. And obviously, this was not efficient. As you can, I'm sure some of us can relate. This was prone to human errors and inconsistencies. It would not scale very well. It was just so time consuming and it was resource intensive. So it was just too much, especially if you're running enterprise applications. This is not the way to go. And then there was this open about the ICD tool such as Jenkins. Even today, a lot of enterprises are using Jenkins to deploy applications. In some cases it works, in some cases it doesn't really work very well. But this actually didn't prove the inconsistencies and speed of reliability compared to doing things manually. But it's still a push model. So this involves pushing YAML directly through a cluster. Setting up complex pipelines, having these organizations manage all these workflows and it's just in some cases a huge mess. And then again, if you want to roll back and you want to see history and have an inventory of what happened it's just too difficult. And then we have GitOps which make things a lot easier. There's really great operators such as Argo and Flux that they got created in a way that you can declare your implementation and your infrastructure and then what you see is what you get. So that being said, it enhanced the collaboration, the versioning. It allows you to do rollbacks a lot easier. So things got a lot better. Like I was saying, it's also easier to do audits, do traces, and make sure that you're following compliance. However, oh, and one more thing, GitOps has been growing a lot that now there is an organization called OpenGitOps which is pretty cool because they in a way work together to standardize the GitOps principles. And it's like a community-driven approach and it's great. I mean, if you haven't checked them out I put the website on the bottom. And you check out OpenGitOps. So even with GitOps, there's always challenges, right? One of the challenges is, well, how do you automate the deployments? And as you're doing this automation, well, how do you scale? And then as you're scaling, well, yeah, you need observability and then as you deploy observability stacks, you need to automate that and then there's more scaling and it gets complicated. Instead, one of the things that I would like to talk to you today is how cluster API and GitOps allows us to fix this. So if you're not familiar with cluster API, it's a project that allows you to declare your clusters in a management cluster. And with the help of some providers, you can deploy clusters to other platforms such as Azure or AWS, Google, vCluster, etc. And in today's demo, I'll be showing you how we do that using cluster API with GitOps. So, and you might be thinking, well, why does it have to do with the Minions? So I like to think of cluster API as, you know, the Minions grew being the manager and the Minions are the worker clusters that are provisioned using cluster API. It will make more sense in a little bit. Oh, and by the way, in this demo, oh, let me go back. In this demo, no Minions were hurt. I just want to make sure that. Okay. So the text stack that will be used today, first of all, we're using AKS, which is Azure to set up a management cluster and it will be the main control plane for setting up all of our infrastructure and also the worker clusters. Oops. We're using Argo as the GitOps reconciliator. And then for, so we didn't want to set up everything in localhost, we actually wanted to use proper DNS name. So we're doing the trifecta to set up the next Ingress Controller, external DNS for managing the DNS entries and also start manager to make sure that we're running proper certificates. Also for our observability stack, we're setting up Prometheus to collect metrics. We're setting up Thanos to collect the metrics among all the clusters and also we're setting up Grafana to make sure that we can observe. And then Qverno, Qverno is actually a pretty cool project that I just got familiar with not too long ago. Basically Qverno it's a policy management operator. I like to think about it as a is this then that for Kubernetes. So we're using Qverno to automate the secrets when, I'll show you in a little bit, a new AKS cluster using cluster API. Well, how do we manage that using Argos? So we automate it using Qverno. And also, of course, cluster API which I've been talking about already and also Vclusters. Anybody familiar here with Vcluster? Okay, cool. So Vcluster is a nice project that allows you to run virtual clusters inside your Kubernetes control plane. So for us, it's kind of like the cherry on the top. You know, originally we were not intended to demo this, but we thought it would be cool to also show this capabilities. So in addition to deploying some clusters using cluster API, we'll also provision some virtual clusters inside of our management cluster. So I'll show you in a little bit. So this is the proposed solution. So we have a manager group and that's in our management AKS cluster which holds all the infrastructure needed to deploy the other clusters. So our ephemeral clusters will be the minions and they already get deployed through cluster API. Right? We also, like I was mentioning before, we have an ingress for our DNS. We're setting up a central Thanos that will be receiving metrics from everywhere. And then each cluster will have Prometheus set up with remote write that will write back to Thanos through an endpoint. Okay. And then we have Argo controlling and managing all of the application sets and deploying all of our infrastructure. Okay. So let's do a demo. So the first thing that I would like to show you is the repo. Which is actually I'm going to show you with this code just to make it a little easier. So I put the link on the slides and you're more than welcome to take a look. So it's pretty straightforward. Actually if you follow the instructions you'll be able to set up this infrastructure for yourself as well. It was coded to be set up in AKS but if you're using AWS you can do that as well but you have to tweak it a little bit. But the first thing that you would do is you create that management cluster which is the group cluster and then you set up some variables you set up your version of AKS you set up the DNS zone that you want to use. In my case I have the domain Q101.dev and then you set up like what's your cluster this is more specific for AKS but what's your resource group name etc. And then you create your cluster and then once the cluster is created you bootstrap it with Argo. With Argo we already have defined a getups folder called management so whenever you set up Argo point it to this folder it will automatically bootstrap and install all the infrastructure components that are needed and I'll show that in one second actually and then once we install Argo the next step is to bootstrap the management cluster with cluster API I wish we could have done this using getups we were having such a hard time with it I know there's a cluster API operator out there that claims to help you out doing this with getups we had a hard time so we're actually installing this manually unfortunately but once you do that then you can deploy your clusters via pool requests so the first thing I'm going to show you is how I can deploy a new ephemeral cluster using cluster API so let me pull that video so right now as you can see I'm doing getup cluster I already have one Minion cluster created and you can see that it's provisioned so in here if I go under my getups folder and then I open my Capsi application set you can see that I added these two lines for Minion 2 and Minion 3 and then I'm going to create a pool request I'm going to commit these these lines and then manage and then create the pool request so pool request created and it's kind of silly I'm pulling my own pool request but it's just a demo so it's okay on my other demo I pushed straight to main so this one definitely will be a little better okay so now that they got merged if I run kubectl get clusters you will see that automatically cluster API picked up via Argo that new CRD and it's going to start provisioning my clusters so now if I go into the portal for Azure you can see now that NAKS cluster was created so this typically takes like 5 minutes so that's why I decided to do a recording because I don't want to make you wait 5 minutes but as you can see after 4 minutes Minion 2 got provisioned and then Minion 3 still provisioning should be done anytime soon okay got provisioned and now if I go to Argo you can see that Minion 3 and Minion 2 showed a little bit got deployed and then also through Grafana you can see that these I guess the Grafana is collecting metrics through Thanos and you can see that we have metrics for our Minion 2 and Minion 3 clusters I didn't mention this but one of the applications that get deployed through Argo is the IMDB app so IMDB is just a sample web app that runs an in-memory database and all it does is just collects metrics and sends it back to Thanos so whenever Thanos detects that there's Thanos receives the metrics coming from a Minion 2 and Minion 3 cluster it will automatically populate that on a dashboard and also these endpoints are available so if you want to try it out you can go to dash.Q101.dev we set up Grafana to be read only so anybody can play around with it and you can see here that we have the 3 cluster like clusters and then we have a virtual cluster called Minion 1 as well the other thing I wanted to show you is here in Argo we divide everything by projects so if you go for example here and open up the let's see Infra you can see all the infrastructure applications are deployed here you can see everything you can see cluster you can see Grafana you can see Ingress, Kibarno Prometheus and also this is public too it's set up using read only so if you go to ArgoCd.Q101.dev you can take a look and you can play with it as you wish let's see what else oh yeah so other project that I have in addition to Infra you can see so workload so these are the apps that got deployed on each cluster so you can see that we have Minion 1 IMDB, Minion 1, Prometheus and then the same for the other clusters Minion 2 and Minion 3 so if I deploy Minion 4 Minion 5 all this gets automatically deployed and you can see it okay so let's go back to the slides okay so some tricks or lessons learned here so for Argo we use projects to segregate responsibilities we try to use best practices we don't want to put everything on the default project so it makes it a lot easier especially to organize things just like I was showing you right now you don't want to put everything in one project and then it's just too messy to see things and like I was saying these endpoints are public probably run them for another week or so after the conference if you want to take a look or just mess around with it server side-apply is your friend I actually learned about server side-apply not too long ago and it's pretty cool when you want to deploy large CRDs sometimes it fails but Argo recently released this thing called server side-apply and it just works so I highly recommend also cluster API they have their own add-on that allows you to deploy help charts but we decided not to use that we want to make things simple and we wanted to use Argo instead also as I was mentioning before with cluster API we actually installed that manually on the management cluster there's this thing called Cappy operator that claims to do things a lot better for installing cluster API we had some trouble using it so hopefully in the near future it could be better for us to use and leverage and some of the takeaways with this demo and this experiment multi-cluster is very hard to implement however if you have the right tools you can team that complexity of multi-cluster and open source is your friend so using open source tools it really helps a lot especially given that there's a big community out there like for example when we were using Kivarno we were running into some issues and we just reached out to the maintainers of Kivarno and they helped us out and it was really awesome something that we're not using for this demo but we think it will be very useful in the future if you split the configuration for cluster type so like right now all of the apps are the same apps are getting deployed into all the clusters but wouldn't it be nice if you can say okay you know if I'm gonna deploy this type of clusters I want these apps in these clusters or if I want to deploy an observability stack I want it to be in this type of cluster so having that division and split configuration among clusters we thought it would be pretty cool and we didn't so maybe in another talk but and where do we go from here so in addition to what I just said we think it would be pretty cool if you can set up a data plane such as a video that allows you to forward traffic among clusters or set up different you know like if you want to set up different loads with different clusters I think setting up a service mesh could be pretty beneficial and I think that's it do you have any questions yeah sure it's down there oh yeah and there's Michael representation by the way the first question that I have is like once that you create all these clusters how can you interconnect between each other so is there a way like a cluster API manage all these things or you have to use an external tooling for interconnect those minimum clusters like if I want to connect to the cluster directly no the clusters that you created with the cluster API is there a way to interconnect them between each other like a open a connection between oh yeah yeah so right now we're not doing that but you can do that and that's why I was mentioning you know we set up like a service mesh could definitely help you right now all we're doing is just set up let me go back to the to the original so right now all we're doing is we set up Prometheus with the remote write and then we give it the endpoint of the ingress and that's how we have to talk to the the management cluster now one way is yeah you can set up endpoints but again these are ephemeral clusters so these are meant to be short lived yeah so I don't know if that answers your question no the thing that I ask you that is because I'm working with an FEO project it's kind of a new project targeting for 5G deployments definitely the idea is to express like your deployment in this base manner so we're using cluster API to manage all this cluster creation but yeah definitely in 5G we have that situation where you want to express like how you see in apps like every single component are going to be deployed and you need to specify like also consider like the clusters and all these things so we are facing basically that like probably one of the take-off ways that you mentioned with nephew we are trying to address that particular thing so but we haven't realized that we don't have like a solution where we can interconnect I mean if you want to reach out to me we can definitely talk about it thank you any questions? ok well thank you very much