 Thank you all for coming. So I'm going to be talking about how to automate with Terraform. And in the demo, I'm going to be showing Flux. And I'm going to be using a kind cluster. And then I'll be deploying an EKS cluster and configuring it. So I'm going to show how to do all that with Flux. So a bit of an intro. My name is Prankaravi. I go by Pinky. And I'm a developer experience engineer at Weaverx. I've been there for about a year and a half. And before that, I was actually at a company called State Farm, where I was on a team called the delivery engineering team. And we helped implement get-ups at the company. So we were using Flux for our Kubernetes platform. And so that's how I got introduced to Flux and then the term, the team at Weaverx. And actually, I took the migration workshop. And that's how I met them. And then I also like forcing my dogs to represent Flux. It's fun. They don't mind it. So no abuse. They're happy dogs. And then if you're not familiar with Weaverx, we're a company that's founded in open source. We are also the company that coined the term get-ups. And we have a lot of projects, actually. The most popular being Flux and Flagger. And then we have a bunch of others, including Weave get-ups, the UI that we have for Flux. And then also the Weave get-ups Terraform controller, which is the one we're going to be talking about today mostly. But just in case you are new to Flux, I'm going to just give an introduction to what Flux is. Flux is a get-centric package manager for your applications. So the idea is that, and it's not just get, by the way, that you can use as a source. But I'll mention that, I think, in a second. So, and it's also just really meant for, it was made for Kubernetes. It was created with all the things that you're probably already using in mind. So it's made to be something that's very drag and drop. And it's also very, because it's modular, it's extensible. You can tailor your experience with it. And you can kind of do whatever you want with it. You can start really small. You can get really more in depth with it and do a lot more. So in short, basically there's a few things we like to mention. So it provides get-ups for both apps and infrastructure. And also, you can use Flagger to then do also progressive delivery as well. And the idea is you just set it up and you push to get. And Flux does everything for you. It takes care of everything. You don't even have to worry. And like I mentioned earlier, since it was created with Kubernetes in mind and everything that you all are probably already using, it works with your existing tools. So you really just install it under your cluster. And it also does multi-tenancy. And as we like to say multi-everything, it can do soft multi-tenancy, which is what my experience is more in with Flux multi-tenancy. But it can also do multi-cluster, multi-tenancy. And yeah. So you can also set up alerts and notifications. And so that way, let's say you do a get push. You can immediately be notified that a change has been made. And then as an end user, I like to say that users trust Flux. We had a really good experience with it. And from my experience, the community is really, really nice and helpful from both sides. And so it's really like we really welcome anyone to come join our CNCF Slack and reach out and have a chat with us. We really do love participation. And so the benefits of Flux are that it reduces developer burden. Like I said, it's extensible. So you really can just change it to however you want. And it is out of the box support for customized in Helm. And like I mentioned earlier, it's designed for Kubernetes. The idea is that you don't have to do any more manual steps. You get away from the cube control problem of having to make sure your versions are up to date and in sync with each other. And also you're not as likely to be at work or working on a weekend trying to fix the deployment that you manually did on Friday. I don't know if anyone else has ever had that situation, but I definitely have. So a brief overview of the architecture of Flux. Flux is a set of Kubernetes controllers. If you're not familiar with Kubernetes controllers, they are the things that control basically the lifecycle of objects in Kubernetes. So updating it, creating it, deleting it, et cetera. So the controllers are as such. So there's the source controller, which pulls artifacts from your source, such as Git, Rebos, S3 buckets, image registries, OCI registries, Helm registry, Helm repositories. So it does a lot of things. So it's not just Git. And then the customized controller. So it's named such because it is using customization. So what that means is if you're not familiar with customization files, are there things that you can do with overlays? You can specify certain files that you want it to apply. And so it's looking for a customized file in customization.yaml within the folder that you tell it to apply from. But if it's not there, it's going to just recursively find all the yaml's that are there and then basically create its own customization.yaml in its brain and apply those. So it'll apply all the manifests that are specified there. And then the Helm controller actually manages deployment of the Helm charts. And it is using the true Helm API. So you can still use the Helm CLI to interact with Helm charts afterwards. And then the notification controller, it's pretty cool. It handles inbound and outbound events. So it can do things like notify you in Slack, like I said earlier, if a change has been made or if drift is detected. But it can also listen for a webhook from GitHub or whatever to see that, oh, there is a change. So the way that these controllers work is that they're running on a sync interval. We call it a reconciliation when it does run. And the idea is, so like if I set it for every 10 minutes, then every 10 minutes, the source controller is going to go check my source and see if there's any new manifestable. But with the notification controller set to listen to it, you can actually say this webhook is triggering the source controller to then go pull it. So you don't have to wait the whole 10 minutes if you're unlucky. And then the image controllers work together to update YAMLs whenever a new image is available. So you can set it up using Sember to like say arrange to say if it's minor changes, then go and pull in the change and actually push it to get. And that way it'll immediately deploy it as well. Okay, so I just put this together. I just really like pointing out reasons that I like it. There's one feature that I like to mention here, and I'm actually going to mention it more when I talk about the Terraform controller, but Dependthon is a feature that's really cool because if you have things that are back to back, then you can actually say like I want this, let's say a database. I want this database to be stood up before my application runs Ertl error. So there's things like that. But most of these I think I'll hit, I've already either hit or I'll hit in this demo. So what is the Terraform controller? The Terraform controller is really cool because now it basically, GitOps is not just for Kubernetes with Flux. So now you can actually use Flux to deploy whatever you can Terraform. So it manages all Terraform, any Terraform resources. And it's, like I said, it's not limited to Kubernetes resources. And these are some links you can go to. There's the GitHub link as well as like if you, like the controller, the docs are there as well. And they're really detailed of how you can get started with certain features as well. So it's really cool. Benefits of the Terraform controller is that, now you have the ability to do all GitOps automation for, sorry, I put this here so I can see the time. But yeah, so you can, GitOps for whatever you already are doing or for any new resources. There is a cool feature that you can actually set it to do manual plan and applies because most people who are familiar with Terraform, you probably wanna see your output like your plan before you wanna apply it. Maybe, it depends, right? So if you're used to that flow, then you can still do that with the Terraform controller as well. And then let's say you're not ready to actually commit to letting Flux deploy your Terraform. You can still set it up to listen to your Terraform resources. And it actually checks live state. So that way, let's say for some reason, someone went in and deleted something, it'll actually, you can set it up to just notify you. It doesn't have to actually go in and automatically apply it for you. But you can still set it up to let you know, just so that you're aware of any drift detection that happens. And so then, so the features, I wanna mention for the Terraform controller and it's really a growing project. So it's always being, new features are always being added to it as well. But like I said, you can set it up to do manual and auto approvals. And what you can do with that is you can set it up to output your plan, to output your plan as a config map and in human readable form. So you can actually see it with the pluses and minuses that you're all familiar with, the Terraform. And then you just change the, you can change the, I'll show you in a second, you can change like the flag that it'll give you in the code and then push it to get. So it'll give you like a shot and you just put it in there and say like, yep, this is great. I saw it, I like it and make it happen. So then you just push that to get and it'll apply. And then like I said, drift detection. It also accepts a list of config maps and secrets as variables. And so this one, I don't want to confuse anyone cause I've seen some confusion around it, but by default the state file is stored as a secret in Kubernetes with the Terraform controller. But that is just the default. And honestly, that's obviously not like a very good practice for bigger workloads because a Kubernetes secret would not be enough to hold some of those bigger state files. So this is just like if you were just playing around with it and you didn't want to set it. But in this example that I'm going to show, I'll actually be using Terraform cloud as the backend. But you could use like, if you're already using S3 buckets, you can keep using that. You can set it up to do whatever you want as the backend. It's pretty easy to set. Also it has health checks, so you can set those up as well to notify you and stuff like that. Also it does, okay. So destroy resources on deletion is not default behavior, which is probably better to be safe, but if you do want it to be the typical behavior just like with prune, you would just set a flag to make sure that things get destroyed. And then you can write outputs to a secret, which I'll show. And one thing that's really cool recently is that, recently it's been a little bit, but is that it actually runs concurrent jobs now. So they added runner pods to it. So it's able to handle, they did a test with I think 1500 modules and it was very fast. So with that, because there's runner pods, you can actually customize it. So you can customize the image you wanna use. You can set the number of runner pods you want it to do. You can do things like that. So it also can use just like Flux, it can use OCI artifacts as a source. And there is the ability to force unlock state if you need. And then it integrates with Terraform Cloud and Terraform Enterprise as well, which I'm gonna show, so yeah. So I don't like playing around with the live demo gods. So I deployed everything yesterday and I'm gonna like walk through how I did it, especially because don't wanna play around with the internet just in case. So, oh, that's really small, I think. So is that better? Yeah, okay, okay. Well, you're in the farthest back, so I believe you. Okay, so the way that I set this up, so to get started with Flux, the easiest way and the most recommended way, the way I like best is to bootstrap it. So using the Flux CLI, you can run a Flux Bootstrap command. It looks like, I think I still have it pasted. Yeah, it looks like this. So it's just, that might be too small too. Yeah, sorry. So it's this bootstrap command and I'm just saying, create it under my group and then, yeah, so then it's saying create this repo. Create this repo. In this case, my repo already existed. So bootstrap can actually bootstrap into existing repos or if it doesn't exist, it will actually create it for you. So it requires your GitHub token or Gitlaptop, whatever you're bootstrapping into. And then that path is actually saying, this is where I want the Flux namespace files to go. So I ran that command and then if I go in here, so if I was bootstrapping it new, it would have created these three middle files. So the GOTK, well, I'll start at the customization.yaml. So it would have only had these two and so this, if you're not familiar with customization.yaml, this is what it looks like. And so it's, in this case, it's saying just apply these four files. So if I left one out, it would just not apply it. And then it creates this GOTK components.yaml, which is the one with all of the components that make up Flux. So it's got the namespace for Flux system. It's got all the controllers in here, any resources that it needs. And then the sync is how you actually set up Flux. So you have a source that's telling the source of controller to listen to this URL here and it's telling it every minute to go and pull from the main branch. And then you have a customization which is telling it to go apply what's in this cluster's kind folder path. And this is actually applying every 10 minutes. And so, and like you can see it, so this prune true command is the one that was talking about earlier, referring. So if I actually delete things, it will prune them for me. Okay, so then, sorry, last time I tried to find. Okay, so that's how Flux was set up, right? And then how did I install the Terraform controller? It's through a Helm chart. So if I go in here, you can see a Helm repo and the Helm release. So this is telling the source controller, hey, go listen to this Helm repo every hour. And then this Helm release is actually the one that says, this is how I want the actual Helm chart to be deployed. So it says, you can also put values in here. And yeah, this is also saying every 10 minutes, go apply it. So that's how that was stood up. And then I'm also using, in this, I'm using Weave GetOps Enterprise. Weave GetOps, there's an open source version of it too, but I was taking advantage of this fun feature. Sorry, should have logged in before. Okay, but yeah, there's a couple of things. One is that there's the Terraform folder in here, the section in here that shows you your Terraform resources. So I thought that was cool. But the way I was actually setting up my Terraform was through using templates, which is a feature in Weave GetOps Enterprise. And the idea is I have this YAML in here. Let me, sorry, let me find it, templates. Okay, yeah, so, sorry, I have this template in here that's a kind of GetOps template. And what it's doing is it's creating, it's creating this template that's telling, that's putting all these things in here, like the vars and everything. And it's all templated out. So I have variables, let me find one. Sorry, let me keep scrolling. Yeah, so like I have variables in here that are then gonna be inputted through here. So the idea with this is because it's able to set up, like let's say I want 10 different EKS clusters, I could just copy paste and then change them every time I want to, like every single reference to it. But in this case, I can just hit use this template and change the cluster name. And I can create as many as I want. And this will actually put back, create a pull request back in GitHub. And so, or I can preview it and copy it and paste it in, which is actually what I did in this case just because I was trying to put it in a specific location. But yeah, so in here, if I go, then it actually created this file. One second. It created this file. So what this is doing is, so very much like I showed with the customize and the helm release, it's gonna be a kind terraform. So this is gonna create an object, a terraform object within Kubernetes. And that's what the terraform controller is looking for. So in here, if I scroll down, so this is that line that I was mentioning earlier, the flag where you can basically change it to be, if you just remove the word auto and literally just put empty strings, then it will actually do a manual apply. It won't automatically apply it. And then when you're looking up the terraform object in Kubernetes, like if you do a, so if you do this, give it a second. So in here, instead of like no drift, it would say plan out and then it'll have like a plan finished and then it'll say like commit back this and it'll give you a quote and then you just take that and you put it in here and you push it and then it'll apply that specific version. So pretty simple. And then this is necessary as is, so this whole section is the part that I had to set up to use terraform cloud. If you look at the docs, it's all like specified in there, how to create the secret and everything. And then this is to set up the backend for terraform cloud. If you're using a separate backend, that syntax would be different, but also still very simple and right in here. And then there's a, let me scroll down, sorry. Okay, so this is telling it to go look at this path. This is where the terraform modules that it wants to apply are located. And then I have these variables that I've passed in as well as there's this line down here that says write outputs to secret. So this is actually saying that whatever this module is outputting, write it to a secret called TFC, we go demo core outputs. So that was just the core. So there's this core module terraform object, sorry. And then there's this secondary terraform object in here called config. And this one is actually right here. It says depends on the core. So the thing is, I don't, how many people are actually using like terraform cloud or terraform enterprise by any chance? Oh, one. You're using which one? Okay, terraform cloud, okay. So if you're familiar with terraform cloud, then you know like workspaces, there's no way in either even an enterprise to actually say like this workspace depends on this one. And so with this depends on, you can actually force basically the workspaces to be dependent on each other. So it kind of solves that issue. We had that in my experience, like you can't set that up. So, but if you weren't using workspaces in terraform cloud, it'll still do the same thing. So the idea is that it's gonna make sure that that terraform object is fully healthy, like whatever it was managing, sorry. Whatever it was managing is fully healthy before it'll actually run this one. So it'll make sure that it's fully planned and applied and finished. And yeah, and then also in here, you can see I take the vars from. So in this outputs, I'll show you in a second, but I'm outputting the subnet IDs. So that's what this one requires. And so if I go in here, if I go to that location it was pointing to, sorry, terraform, and I go to cluster templates. So I can go to core and it's basically just standing up the EKS cluster, that's what this one is doing. And it's pointing to this file path. So this actually has the more specific modules, but in here this one is just setting it up. And then my outputs, like I mentioned earlier, is actually outputting my IDs. So this is what the config is then going to use in order to get its subnet IDs. And then this config is then taking the EKS cluster that was created and it's going to create a node group, it's going to create the auth, the identity and stuff like that. So it's just initializing it. It's adding the ability for me to actually access things too. I wouldn't be able to access it if that user wasn't created. So if I go into the terraform cloud, so this is what it created. So the nice thing about this is if you are using terraform cloud it actually creates the workspace for you. So it creates the workspace for you and it will then plan and apply in here. And so a few things I wanted to mention here. Sorry, just got a wrap up morning. So the core is, like I said, the config waited for the core and some cool features in here is that you can actually set it up so that you can set up variable sets in terraform cloud. And so that's what I did in order to pass it to AWS credentials and you can specify that it can apply to all the workspaces in the org or just like certain ones. And in this case, like you can even say sensitive and I can never actually output that secret, which is really cool. So that's a really neat feature. And then if I go back to clusters, so this is the one that was created, wego demo. And if I click on it, you can see that not only was it created, but also that node group that was specified in there was created yesterday as well. So yeah, so you can see it was created yesterday by that, by this, these runs. Yeah, so I think that's everything. Is there any questions? Maybe not much time, but. Oh yeah, sorry. So I see that we are storing the output of the configuration in a secret. And I want to understand why do I need to do that because I can always get to the output by. Yeah, so I'm doing it in this case, just so that I can say in the second one that, hey, go grab the variables from the secret. So that's just like a trick I was using with the objects. Cause you know, like in YAML, you could just say vars from a secret. So that's why I was doing it in this case, just for accessibility. There's other ways to do it, but this is a very simple way if you have like a depends on module to be able to do it, yeah. And other thing I noticed that on your Terraform CR, you were referring to your GitOps template, which is your enterprise. So how does that work? Because one's open source and other ones is enterprise, right? Is that just working for your demo or is this something that we can do as well? Sorry, I'm not sure I'm fully following. So if you go back to your Terraform CR, yeah, right here. What weren't you referring to your? No, so I think I understand what you're saying. The template was just used to create this file, but like if you, you don't have to do that. You could just create this file from scratch. Like if it says Terraform object, the Terraform controller will listen to it. I was just showing it off because I think it's really cool to be able to just create multiple at once. Yeah, no, but you don't need to use that. Yeah, it's really, it'll work without the templates. Oh, and then like one thing too is that in here it actually shows your resources. But if you were, like you can still see applications. So there's an open source version of this UI and it'll still show you your applications and sources. The only thing Terraform is kind of like, it shows the Terraform objects in the enterprise version, which is kind of cool, but yeah. Thank you. And then if you are interested, we do have this QR code will take you to any other like happenings within the like flux world here at open source summit and get up, Scott. You had a question? Yeah. Is this limited to just TFC or TFE or can I use like S3 backends? Yeah, yeah. The TfState. Like I said, it stores it technically back end by default as a secret, but there it's really easy to set up. Instead of saying, I think I have it commented out in here actually or I thought I did. But instead of this, you would just say like, I think it's back end config. Yeah, back end. And it's pretty easy. You can just set it up to point to whatever. Yeah, we were just doing this because this actually will also create your Terraform cloud but workspace, but you could do it the other way and it would still work with TFC. It just wouldn't create it.