 You know whenever I work with Kubernetes like I really can't remember how to write my manifest, so I definitely need some of that co-pilot thing So hello, my name is Richard. Most people just call me carding This is really my second time attending a user group Singapore meet-up. I mostly attend the Philippines meet-up So I've never talked in front of a very large audience like this before so One of the things I really ask for you to do after the after the after the event is let us know Through the organizers as well if we're you know, we're hitting the right level in the talks as well So it can help us calibrate because you know, I really don't know like the kind of levels We're we need to be talking about in Singapore so What I do is I'm a prototyping engineer for AWS in ASEAN we work with a lot We work with a lot of tech so pretty much kind of the breadth of things instead of really the depth We touch on a lot of technologies and one of the things we really see very very often is Kubernetes for some reason containers serverless and all that just right Today I pretty much just like to talk about a few things particularly in GitOps so With a short time I have I'd like to give a brief bit backgrounder on Kubernetes Just to make sure that everybody's on you know, kind of the same footing on what Kubernetes is Is and what does et cetera what what you how you do stuff? What you can do with it and so on and so forth and then we'll jump right into What is GitOps like what is the argument for it? What why do we want or why are we interested in doing stuff like GitOps? And then maybe we'll touch into some of some bit of live coding just to see what you know How things actually work and so on and so forth sounds good. Yeah, cool So who among you work with Kubernetes on a regular basis. Oh wow pretty cool pretty cool amount Good. How many of you regret it? Oh Oh There we go. Cool. Thanks for being honest now. I like to wail on Kubernetes, but It's kidding aside. It's a very very good tool. It's a great tool if you if you use that then, you know For the right reasons, but again the question is what are the right reasons, right? And I'm not gonna go into that kind of worms Just to give everybody an overview of what Kubernetes is Cates is a container orchestration tool So what you do with Kubernetes is that you tell it that I want to I want you to run These and this containers maybe three of them five of them and so on and so forth and Kubernetes does its best to maintain that for you So if you asked it, hey, I want to I want five copies of a container It will do it's absolute best to make sure that there's exactly five of the browning So if one of them crashes or for some reason there's now 10 of those containers What Kubernetes will do is it'll make sure that you have just a five, right? So if one of them crashes, it's pretty much an automatic health check and all that, right? Kubernetes is also an abstraction. It's a pretty big abstraction And this is something that a lot of people don't understand or don't comprehend fully The one of I think one of the biggest draws for Kubernetes is that it abstracts everything mostly everything About your infrastructure and your platform, right? So if you're running on top of Kubernetes and you're deploying on Kubernetes You really don't have to think about are you running on AWS? Are you running on EC to on Fargate on a fleet of raspberry pies? Or are you on an on-premises kind of location? You really don't have to think about that for the most part Right, so the idea behind it is that it's supposedly Agnostic that you can bring Kubernetes anywhere and then it functions pretty much as you'd expected to do so No matter where you are Right, and then Kubernetes is also made up of objects This is something you have to understand if you're working if you if you're working with Kubernetes One of the kind of concepts that you have to grok your head around is that Kubernetes is actually made up of objects Anything running inside a Kubernetes cluster is actually an object Kubernetes thinks of it as an object So that can be a pod that can be a deployment that can be a service that can be a node That can be a namespace and so on and so forth absolutely anything You deploy on Kubernetes inside a Kubernetes cluster is an object and The nice thing about it is that the way Kubernetes thinks about this is that every single one of those objects You can represent in something we call a manifest. So mostly that's written in YAML An example of a pod would be right there. So every single object inside Kubernetes Hopefully I'm not overextending that can be represented in a manifest file, right? Finally Kubernetes can be declared to so you can use Kubernetes in a declarative fashion In fact, I'd actually argue that it's probably for your in your best interest to work with Kubernetes declaratively What that what do I mean by that when I say declaratively as opposed to imperatively Most engineers most coders kind of think automatically around the imperative point where you where if you want something done You tell the computer you tell you tell the you tell the system How to do something if you're programming that's normally how you do stuff, right? You tell it exactly what to do how to do it in a set of set by set instructions Algorithmically, right? declaratively, you tell it what you want done and then The Kubernetes pretty much figures out for you what needs to happen to ensure that happens, right? So my favorite example pretty much easy to understand is that imperative would be asking somebody to buy your pizza Declaretively would be just kind of implying that they I need a pizza I kind of hoping that they buy a pizza for you, right? Something like that. You can use Kubernetes both ways So you can be imperative in Kubernetes. You can be declarative about Kubernetes. The nice thing about Being declarative with Kubernetes is that you don't have to think about what exactly needs to happen when something needs to happen So for example, if I want to deploy again that five copies of a pod that five copies of a container On Kubernetes, I can just tell it. Hey, I want five copies of container and Then Kubernetes will need a little figure out for you exactly what needs to happen. Like do you have enough resources for it? Where do you deploy it? Right, how do you distribute it? Do you already have like some copies running and I just need to add into that or do I need to actually remove some of those pods? You don't have to think about that because Kubernetes will do its best again To maintain that kind of sort of state, right? You want five. I'll make sure you have five, right? And I'll do whatever I can to make sure that happens. That's that's pretty much how the declarative fashion works with Kubernetes so Now putting all of those things together all of those concepts together You really can Mention a Kubernetes cluster With just a collection of manifest files. Remember every single thing in Kubernetes is an object Every single object is the manifest file in yaml format and everything you can just you know Declareatively tell Kubernetes that hey, this is exactly how I want my cluster to be This is exactly what I want inside my cluster, right? So if you think about it you actually can Maintain an entire Kubernetes clusters. We're just having like a plethora a good stack of yaml manifest files, right? Theoretically if you have a Kubernetes cluster and you have all of those yaml files that represent everything in your cluster You can go to any other Kubernetes cluster Put those exact same manifest files and voila you've got a copy of your Kubernetes cluster now with exact same thing Right same things running right because that's just how it works Now if you think about that going one step further My absolute mad lad of a colleague actually asked me this a few months ago. Why don't we? apply CICD practices to those manifests Right everything in a Kubernetes cluster can be represented in a flat code file and We can use that flat those flat code files to describe exactly what needs to happen in Kubernetes cluster Why don't we pipe that through the same automated CICD tooling that we already use in application code? Right, and that's pretty much the long-winded gist of what get ops is That's kind of like my long-winded introduction to it now For my next magic trick, I'll try to Guess what your change management process is and no matter what you what change management process you use in your companies If you do use a change management process, hopefully It it will always gravitate I think at least I see at least form experience, right? It will always gravitate to a few fundamental steps The first one being that you have a managed product somewhere, right? You will have Some artifacts some material a system a solution maybe a code base somewhere. It can be in a git repository It can be in an you know an s3 bucket, right? Whatever that is you have something managed Your change management process will dictate that somebody will have to introduce some change. Maybe it will be a bug fix Maybe it will be a new feature Maybe it'll be just be some mean yell chore. I just need to you know remove regression testing for example, and so on and so forth Hopefully your change management process also has a way to review changes, right? So people can collaborate and talk about and discuss and I'm pretty much review exactly everything that needs to change if all of those changes gets approved Then those builds are built those deployments are deployed those changes are incorporated into your system All right, and then you're back to step one That's pretty much your change management process. Hopefully right get flow. However, you're doing it monorepo style Repo style doesn't really matter Pretty much. I'm thinking all of the change management processes will look like that now If you think about it How much benefit are you getting from that? Like if you're applying that to your own applications, right for your own engineering teams who are maintaining code maintaining products maintaining applications a Lot of it is quality of life, right? A lot of it's making just making things easier automated deployments come on, right? That's you know, I'll take I'll take 10 of those any day, right? So why can't we use it? To manage infrastructure as well We've all heard of infrastructure as code. We've all heard about cloud formation the CDK Terraform, etc. Why can't we use the same kind of tooling to automate our deployments and measure deployments for our infrastructure? and our for our resources right and that's really kind of what get ops is so I There's you know ask 10 people you get 10 different answers I kind of like this answer from the get labs documentation They call get abs at they call get ops as an operational framework It's kind of a subset of DevOps that applies the best practices for application development Onto infrastructure deployment and infrastructure automation. So pretty much the same things that you probably already are familiar with You can apply to your infrastructure particularly in kubernetes At least for this session In an automated fashion the same kind of expectations you have code reviews automated deployments Pull requests, maybe it's on and so forth, right? The way I kind of like to think about get ops, there's really three things one is you'll need some form of infrastructure scope You'll need some form of change mechanism. It's you kind of think of it as a trigger what triggers What needs to deploy right and eventually you will need an automated deployment format or a mechanism to deploy Incubinities easy enough Infrastructure is code is automatic. You already have your benefits just like I described right all of those channel files It's pretty much your IAC and then your change mechanism can be whatever your team is already accustomed to If you're already using pull requests for example over github or merge requests over on get lab up to you The really big question then is what would be your automated deployment mechanism if you were doing think if you were doing this on Kubernetes what would make you know, what would allow you to do automated deployments and that's really where The flux project comes in So if you're not familiar flux is a project under the CNCF umbrella It's what it's pretty much managed by the CNCF and it allows you that's that's pretty much the question It tries to answer the problem tries to answer how do you manage automated deployments? pretty much on Just like how you expected them, you know any other system, right? And It's pretty easy One thing I really like about it. It's pretty easy to hook up to an existing cluster So if you already have an existing cluster, it doesn't even have to be on EKS for example We're talking generic Kubernetes here, right? It can be an EKS. It can be on any other platform It can be on your local for example if you want to hook up a cluster With flux tooling to allow, you know automated deployments on it You pretty much just had to run that Command right there. So you'll need the flux CLI tool. You can download that from the website pretty easy And then you'll need to give it credentials To whatever your platform is for your repository. So in this example for example This is GitHub. So you'll if you if you're working off GitHub You'll need to create a personal access token and then give that somehow to flux and then Off you go, right? Probably better to just show you what happens. Yeah Okay So I've prepared a Terminals font size. Okay. Yeah, so I prefer Prepared an EKS cluster. So if you're familiar with Kubernetes Whoops, let me just log in. Oh, okay. Sorry. Sorry terminal. Okay Perfect Terminal using a wc li if you're familiar with Kubernetes you can just use the kubes control kubes ctl tool to kind of ask Your cluster things right so for example here. I'm asking it for namespaces So I'm just showing you that I have a ready Kubernetes cluster for use Right. This is the pretty much Most generic Kubernetes cluster you'll see so if you prepare a an EKS cluster Using the official tools. That's what you get So a pretty bare bones Kubernetes cluster now I already have I already have a personal access token ready But I'll just show you where you can go to do that. So if you're on github, for example You can go to your settings then over on the very bottom that you have your developer settings there And then over here you can just create your personal access tokens So you have to create a personal access token. Let me just zoom that in for you so you have to create a personal access token with you'll have to create a personal access token with permissions to manipulate your repositories Because that's what flux is going to be doing Once you create a personal access token like that you'll you'll I won't generate the token anymore You'll get some you'll get a pretty long string and then what you have to do at least if you're doing this on the command line is you have to store that in Your github token environment variable Right. So once you put that in there export like that That's what flux will that's what flux will be using to bootstrap your cluster, right? So now Every every single piece should be okay now. We have a Kubernetes cluster. We have an existing Kubernetes cluster We have a personal access token to whatever we want to use in this case. We're using github We just need to bootstrap it. So in this case, let's try that I Was the command again So it's flux Bootstrap I have to tell that that you I want you to bootstrap yourself for github, for example and Then the name of my the owner Can be a github organization or a github user I'm going to be using a github organization and then I need to give it a repository name because what it will do Is it will create a repository under my control? Right. So let's try test github's repo I need to give it a path so that inside that repository what will be the folder structure for where it's going to be putting manifests I can just say clusters flux flux cluster whatever Right and hopefully this works I've had pretty good luck with a live demo gods this month. So hopefully I Keep that sweet going. So as you can see what's happening is that flux is talking to my Kubernetes cluster while on the side is also talking to my github account So what it's doing is it's creating a repository on my github account It's populating that with a few manifests the manifests being everything that flux needs to run inside my account my cluster So that flux will work Right. Okay. Good. So that that was okay. So once you see that all components are healthy at the at the end That's you're good to go. So now Let me just show you what happened, right? There's a there's a few things that happened here. So I went So I called my cluster test github's repo, right? So it actually went inside my github account in this case I my github organization and it created a repository for me inside that is just a few manifests So if you're familiar with Kubernetes, you'll you'll recognize this right away But it created a few manifests in that repository for me. So that doesn't really tell you a lot now The main difference now is that here remember the Kubernetes cluster. I was showing you earlier One of the things it did was it created a new namespace for me So there's now a flux system namespace inside where all of the components the flux needs is running inside of all right, so Pretty much two things it created the repository with the manifests and then it created all of the flux components inside the Kubernetes Right, but that's not really the fun part. The fun part now is that because of those two things My cluster now is watching That repository any change should happen in that repository Will be reflected onto my Kubernetes cluster. So let's give that a try Let me just clone this repository down to my local All right, then open that into an ID and this is what we've got So this is exactly what I was showing you right, so we've already got like Entities inside so let's try Deploying let's try deploying something new into our cluster. So what I'm going to do is I'm just going to write up that and that same engine x pod that The previous guys were trying to do huh, so let's try that engine next pod and hopefully I remember the So this is a pod a metadata metadata Let's give it a name my engine next I suppose I Need to give it a spec. I need to give it which containers make up the pod The name of the pod is my engine x like that The image is going to be engine x What's the latest version 1.2 and 3.1 Something like that Just use the latest idea container port Oh, sorry, and then not really needed, but let's give it a container port Like that Okay, so this will deploy a pod but instead of But instead of the standard, you know could see to the L apply f whatever right giving it the ammo file What we're going to be doing is I'm going to be committing this to my get GitHub repository, right? So This will be introduced new engine x pod like that Then I'll push that in So, you know if you think about it since everything now is managed within your GitHub repository I'm being really really fast about everything here, but in reality You really should be going through the whole change management process, right? This should be a pull request This should be a review everybody, you know the reviewers need to be you know talk about the changes that needs to be made That needs to happen and so on and so forth So if I refresh this down, I've updated I updated that let's see if That's actually deployed I Kind of think we were you're already hitting an error now Okay, let's see if that actually deployed It did not because that I know I know why because we didn't We didn't give it a namespace Let's just have that deployed like that Yeah, yeah, there is there is actually I'm actually trying to think of how to get to this so Cube Hard watch yeah, hold on repository. I have that So get customization I have that to CDL logs Whoops. Oh, sorry. Sorry flux logs Okay. Oh Weird that should have worked See I told you I used up all my luck with this month Let's just see if this will fix things up. Yeah Yeah, hopefully it's just a good push But that's pretty much the idea of it So what's happening now is that inside the inside the cluster? These are these are the kind of the low-level changes that are happening, right? I Showed you that in the cluster. What's changed is that the flux bootstrap process Created that namespace for us. So it now has a flux system namespace Inside that namespace is running a few controllers. So for example These are all that that's happening inside that flux system namespace. So I should be in the right thing. So yep So there's a few services There's a few pods that are pretty much just managing the flux system inside a cluster and then if there's a few Objects there that are watching my get repository. So The source controller over here This this guy is taking care of everything It's watching all of the sources like get repositories and all that and then the customized controller over here That is that is taking care of everything that needs to sync onto my cluster So if it sees that something has updated with a source the customized controller then takes that and then Applies that under the cluster. All right Let's see if that's now work. Yeah Still no, okay I'll give that a thought but that's pretty much all you had to do. So There's probably something wrong with my manifests. So that's how you do the bootstrap thing and This is exactly what's happening like I just showed I just described. So when you deploy The flux system or the flux flux project onto your Kubernetes cluster What's happening is that between Your cluster and any other source be that a get hub repository or get lab repository You are essentially deploying Pretty much two fundamental things one is a source. So in this case, this is a git repository source And a customization object. So two objects, right? Again, the flux source object it takes care of watching for changes on the source and then the Customization object then takes care of a syncing whatever those changes are Back down into the code back down into the cluster, right? So if you actually take a look back into Here oops These these are the actually two important things that the flux Bootstrap deployed. So one is the git repository source and then one is the customization Object, right? So I think if ever you try to do this for your own systems for your own clusters What you have to what I personally change every every time you do this is the intervals over here So by default flux will look at your git repository or your source by default every minute And then over here it will sync any changes. It collects every 10 minutes That's under the customization object now for personally, I think the customization for 10 minutes is a bit too Lacks so I kind of narrow that down a bit and there's really no there's really no Non-benefit to kind of narrowing that down. I normally bring that down to maybe two minutes or something like that So the thing you have to understand here is that What happens when a change is affected so if your git repository changes some code files or manifest files changes What happens is that the customization object? gets the manifest files from that git repository and runs a coupe coupe CTL apply on your cluster So what happens is that if you manually change stuff on your cluster? Those changes will all be over overrated as well Most likely because what happens is that the customization will just run coupe CTL apply or whatever is in the git repository So now your git repository becomes your single source of truth Right, whatever is inside your git repository is What's supposed to be running in the cluster? another benefit of this kind of approach is that for I Guess for a lot of reasons you can kind of narrow down exactly who has Credentials to directly talk to your cluster who has the permissions to actually run coupe CTL Blank something on your cluster Right, I've seen I've seen some groups who've gone away with no giving nobody access to the cluster at all And just making sure that everything is right there And then if they do need to give a you know direct access maybe for hot-fix debugging or investigation then that's when they Give temporary credentials, but because if you do this then there's you know There's almost zero reason for anybody to actually deploy directly on to your cluster So you can kind of pretty much outsource that kind of permission down to the deployment system to pretty much your flux system pretty much Right, so that's really what's what's what's happening under the hood so that also means that if you think about it you can kind of Explode this, you know, you don't have to have your cluster just watch One repository in a more practical sense in re you know in a more real-life situation Your teams will probably be managing multiple repositories that make up, you know multiple products So you can actually set up your cluster to have multiple Instances of those customization sources so you can have your cluster have like Well set up multiple watch multiple repositories for example And then whenever those repositories change your Kubernetes clusters just adjusts and changes for you What do you have to do to make that happen pretty much this all you have to do is just create more of these things more get repositories sources or whatever sources they are and customization sources if you can see The object at the object manifests are pretty simple. They're just you know, where where's the source, right? Where's the repository and then what is the path to the manifests inside that repository? And then flux pretty much does everything for you and manages all of that for you If it works Okay, so definitely that's something that's something you can do and something I see happen as well Some quick quick fire things of what else you can do with flux if you use helm charts It's compatible with helm helm is kind of like a templating language a package manager kind of thing for Kubernetes as well So you can use that with flux as well. You can set it with notifications So whenever something happens to your cluster you can it can throw an automatic Notification to your slack to your discord Or even to your raw Notification endpoint in Jason format if you'd like so you can throw it to elastic search for example Automatic updates you can have it watch. It doesn't have to watch a github repository It doesn't have to watch a git repository for that matter You can have it watch an arbitrary source can be an S3 bucket can be an image tag in an image repository if the tag changes to Format that you specify then update the repositories to match the image for example and so on and so forth and then one thing I really like doing is I want I Switch flux to using a pub sub pattern instead of a polling pattern So if you remember me telling you about the intervals it actually pulls Every minute every five minutes you can switch it back so that it's instead a subscriber format So it only sinks when you tell it to so you can have your CICD Frameworks automatically tell it via an API call for example to sink because something's changed and so on and so forth right So where can you go to learn more just in case you want to take a look at everything? I've been telling you so some of the stuff that I you probably will work with if you're working with Kubernetes are there You'll definitely need to CTL definitely to talk your Kubernetes cluster If you don't know EKS CTL is the official tool for managing infrastructure on Amazon EKS So that's right there as well if you want to create an EKS cluster. It's an ECS EKS CTL create cluster Flux is right there. That's what I've been showing you this entire session then some other things Coups CTX and Coup NS are very very useful tools for switching between Kubernetes clusters If you work with Kubernetes these two tools are really a godsend particular for me and then if you want to work with Kubernetes Play around with it without having to deploy those, you know huge Deployments mini-cube is pretty much a single node deployment on your local so you can run a single node Kubernetes cluster on your local using mini-cube and then try out whatever tools you learn from there We have a dedicated just in case you didn't know we have a dedicated Web page for containers and containerization technology Kubernetes included so that's right there as well pretty easy URL to remember Something I like much more than that is we actually have a dedicated workshops portal as well So we have quite a few containers EKS ECS all of those things kind of workshops over at that workshop portal It's just I neglected to for I neglected to put the URL workshops that EWS Sorry for that. I should have put that there. So pretty easy URL to remember anyway and That's it. Sorry for the demo not working. I'll try to figure out how that happened any questions Yes, sir Two questions. How does box compared to August CD? For example, I actually haven't tried that so I can't I can't really compare The second one is like say someone makes a mistake. Yeah But unlike yours You have something running. It's working and then someone makes mistake like chooses an image tag that doesn't exist or something Yeah Well, um fluxes Flux is pretty dumb in that sense that it doesn't know about those things It doesn't care to be honest, right all it cares about is deploying So like what like what you saw earlier if something fails in the deployment pretty much is just doesn't deploy Right now. What happens is that I think the more important question would be what happens if you want to do a rollback? For example, something, you know, we deployed something wrong. It actually deployed successfully But something's wrong with it and we want to roll back, right? Flux for them for the most part also doesn't do rollbacks particularly because Flux philosophy is that single source of truth get repository or your sources, right? So if you want to do a rollback, you'll need to do a rollback on the git repository itself and then have that take that back Well, you can't set it up to have a those kinds of deployment there are on the documentation site There's kind of like a collection of blueprints canary included a B Blue-green and so on and so forth what you can do it does have some monitoring with Prometheus for example So you can't have it work like that Just not on its own. I'm assuming Yes, sir. Did you want to ask something? Oh Sorry I'm sorry, I didn't get that question I Actually haven't tried so I can't I also can't say but I'll take your word for it and say yes. Thank you. Yes, sir Yes, so you can you can adjust you can deploy new things in your cluster and Proactively tell Flux not to touch it So If I remember correctly just have to append your object with a label with that's very specific label So that it's kind of opt out that flux doesn't touch touch this whatever As far as I know yes, so there is kind of an escape hatch if you need it and you can You can tell Flux to pause or suspend syncing for example for emergency reasons if you need it So you can do that as well question What happens with complex modifications in Flux, for example, if Includes Does include for example resources Oh, and if I add resources manually and then The Sky Okay Okay, good question. So two things one is The thing you have to understand with Flux at under the hood the they're really fundamental thing It's doing is it's really just calling kubectl apply kubectl apply right with your manifest in the git repository So I guess a good rule of thumb it would be if this doesn't you know if kubectl applying All of these manifest files in one go it's not gonna work then it's probably not gonna work with Flux as well second thing Flux is pretty much it's a bit intelligent in the sense that it knows like a Fundamental organs or things like for example if you if you do a get a get Repository change where and you put a namespace and then a pod that belongs in that namespace Flux will be able to do that, but I think that's really not because of Flux, but that's really because of kubectl Right, so I guess rule of thumb if it doesn't if it will not work with kubectl One time big time then it's probably not gonna work with Flux Yes, that's the that's kind of like the short version of it. Yes I'm sorry, I think so. Yes You could yeah Very good question Particularly because what one of the really good things I like about Flux is because all of the logic that goes into the action of deploying and managing your cluster is in one place Right. It's just in your cluster, but if you do for example if you have that That last thing I was talking about like you have multiple repositories and all of them You have to kind of point towards the same cluster to deploy and so on and so forth, right? It becomes a little bit tedious than you think so I think that's kind of like, you know, one of the one of the kind of Dealmakers for me. I kind of like coming from that other end. I kind of like coming from the cluster Sam Can you give an example who's using it? Ah, well, I don't really have something top of my mind, but I I remember the Flux website having There we go. So they have they have a pretty good list over there Ah, well, well, I do I do know some but I don't think I can share. I'm sorry This will be the last question Yes, sir As I understand it's integration with like github or githlap and so on So that's providing a like Let's say deeper integration. So, for example, when I first merged of my full request I actually want to see what happened So in your example, it was not able to deploy in my port. I want to know about it Is there a nice Yes, so Flux does come with a few events that happen within the framework, right? So one of the things you can actually do as I share is that notifications But one thing you can also do for example is you can have it report back to your repository of how your deployment went So for example, you can set flux up to be part of your github checks as part of your pull request approvals Right, so that's part of it as well. You can hook it up like that Sorry, I know we're out of time So if you have any more questions feel free to connect with me on those Social media links and I'll be happy to answer them for you. Thank you for your time