 All right, as you can see we're having a little technical difficulties, but I'll We'll work around them. So my name is Christian Hernandez. I'm part of the Red Hat Clap Platforms BU My social handle GitHub Kubernetes Slack is Christian H814. So you can hit me up there In either one of these avenues. I'm usually pretty responsive So today I'm going to be talking about GitOps the holy grail of DevOps. So I'm going to be going first over kind of just defining what DevOps is a little bit just to make sure we're all in the same page And then I'll kind of go over the history of GitOps and how it all fits in. I Also had a demo planned but since I'm not using my laptop I guess we can use that demo time for questions and answers or I couldn't show you on the outside if you like so So first of all, what is DevOps right so What DevOps is Really is like a it's a methodology and it's practices, right? So it's It's not a department, right? So I had a long time ago when We first started doing DevOps and in my previous company there was a guy that would say oh the these two departments will never work together Right, it's not one in that I had to like explain. No, no, it's it's not a department. It's it's a it's a way to work, right? Dev and ops work really closely together, right? Before there was a there was an idea of you know throw it over the fence, right? You know, I developed something and you know Packaged it up throw it over the fence is the ops guys problem to deploy manage do all that, right? So now this is more of a methodology of working together being able to Solve problems and no longer throw throw the throw the code over the fence as it were And it's meant really to shorten the development life cycle, right? So you there was a lot of talks today I just listen to a talk on tecton About CI CD, right development life cycle, you know why that's important for fast delivery and really DevOps is a way to Deliver those things rapidly, right? It's also meant to have a higher quality software, right so when when everyone is working together the the issues that come up can be fixed quicker is what the idea is really and What I like to what what I actually seen out there is that the application stack now is seen as a whole Responsibility right so no longer is like, you know again I do the analogy of throwing the code over the fence is no longer is like, okay You know now it's their problem now It's the the whole stack as a whole is seen as like an atomic unit right from the From the development all the way down to the infrastructure component is all seen as a component so So when I talk about DevOps is I'm kind of using these Methodology you can agree or disagree or whatever, but just in the context of this talk This is how I'm gonna view DevOps And really really how did we get here, right? So there there's there was the buzzword of DevOps has been around for a while but really Really it's really come into fruition in the past. I would say five years or so and And really really how did we get here, right? So I was debating on where where to start this How did we get here, right? I was gonna maybe maybe start of You know In the times where we're rocking rocking servers, but really this is all started where virtualization, right? is really when we started getting into the the idea of a rapid development So with virtualization came with the idea of like snapshots and cloning and With that you'd have really like a template of what What the infrastructure looked like for your application, right? You would you would build a server with all its components and libraries and you would Clone those right like this is my application foo and that's the server for application foo And this is how I'm gonna stamp them out And there was a lot of tools for for this automation, right? Tools like a puppet chef later on ansible. I know I've used chef exclusively in my my former former life and Those tools were were made to automate a lot of those right especially with virtualization and some of those cloud instances right like AWS You can really have this in like this idea of infrastructure as code, right? You can define what your infrastructure looks like programmatically, right? The kind of challenges that I really came up with that Is that you still get the idea of of a one-to-one relationship, right? So you really get the idea of okay. I have I have a server and I'm cloning it now But that whole server is still defined as the application, right as part of the application This service it's it's eventually it still turns into into a pet, right? You have you heard that that phrase pets versus cattle You still have that idea of like this VM is still a pet because it's still specific to the application, right? so all of these tool sets Gave birth to what we call an SRE now so SRE You know before I said that DevOps isn't a A Department if it was the SRE would be they would be part of that department, right? Or they were at least spearhead those that department if it was one But these these are specialized people that know the application so when I said before the application stack as an atomic unit These people know that entire stack, right? So they're really an extension of dev and extension of operations they They really Take care of the application once it's up and running they they submit patches. They they they monitor They do all of those things that need to be done for that application stack. So And that's kind of like the the progression Then after virtualization came this thing called Docker, right? So Big buzz right of Docker before What docker really did is it took the concept of LXC's Linux containers and simplified it, right? They wrote an interface to be able to Like a package and build these containers I remember when I first thought Docker out like many of you I was like blown away. I was like, oh now This this changes the game and actually I think it really did You're able to To build these containers Readily and easily and actually they wrote a really nice workflow for developing Applications and packaging goes up right because they give you an idea of bundling your your application everything, right? All the dependencies everything needed to run the application is built into this this this atomic unit, right? we talked about I talked about atomic unit a little bit before and It's really Containers is really nothing but a It isolates processes right it isolates By processes and namespaces inside the kernel, right? So no longer do you need? The entire operating system in order to run your application You just need the bits that you need to run the application and with the isolation inside Colonel namespaces or in C groups and all the all that nice stuff that you get with Docker You're able to now run you get you actually get multi-tendency now, so you can run multiple Applications without having to worry about oh, yeah, I need this version of the library compiled versus this version And now you're able to run those two different versions right inside side the Inside the operating system, so they're immutable right so which which is really cool because Before with with VMs and puppet and you know ansible you get the problem with drift right so You get the problem with drift until like you run that that sync that makes everything What then the state that you want it right so here with containers they don't really change right you say I want this there since since with tagging you say I want to this version this shot to run That doesn't really change right they did so they're they're They're immutable from that fact the You're able to run it the way you run it runs on your laptop is the way it runs on a server, right? So that's that's I think that's very very powerful, especially with dev devops, especially with that idea of throwing things over the fence Even if you're still doing that right now, you know exactly what you're throwing over the fence It runs the same so you know the only thing really changes is the environment underneath right so So right problem solved right all the problems are solved right so now when you build an application It's easy right docker made a good interface docker build run right my app version one and With the same interface you can run it right docker build docker run easy command line interface And now you can actually see how You can start plugging that in into okay now that I built and I run this container I could run it anywhere physical virtual environment private public cloud all that good stuff And then you can also see how you can plug that in Dressy CI CD pipeline Right, so you have you know developer and then you have a source core repository And then you have some sort of CI CD engine Jenkins or tecton And that builds the container and then you can then deploy that container, so it's starting to come together right so But Application stacks are rarely that simple right so you know even the most simple Application comes at least with the front end back in and a database right maybe have a caching layer So they're more complex than just a few containers right so you usually have a You know usually have a few servers running a few application stacks over a dozen servers, right? So now you think okay cool. I can do okay. I could build the front end back end database cache messaging whatever you need for the application You I built that the same way you can think right and then when you run it then you have to you know kind of think okay well I Run my friend it will not need two front ends right because you know you want your ha so I have two front ends and I to make Sure I link it to the back end well now I have two back ends right because I want that ha so I want to make sure I link that to my database and caching system I want to make sure My caching systems linked to the DB and you just kind of just have to keep all that in mind right so this This is one application stack So now you have that one application stacking you have a way to link it and okay Well then but now when you want to that's fine if you want to deploy it to like you know, maybe five servers, right? Maybe even ten servers But what if you have dozens of applications over across a hundred hundreds of servers? Then then it becomes maybe not that manageable right so as Y'all may have guessed the answer to that question mark right there is Is It's Kubernetes. Sorry these slides were out of order. It's Kubernetes, right? So Kubernetes came in and Really answered the the following the following slide right so You need more than just containers So just containers itself is not solving anything right you need scheduling security Health management persistence, you know all that stuff right so you need more than just Running containers and this is where Kubernetes comes in place right so Kubernetes takes care of all of that for you right so I have a container. I want to Be able to deploy that container the application stack Across you know hundreds of servers and I want to be able to link them all together I want to be able to do health checks. I want to be able to do Rollouts deployments right all that stuff Kubernetes takes care of for you Alright cool now that we have Kubernetes That solved a lot of that that problem itself right so now Kubernetes really Just to from a from a high level Is it really a declarative way to describe your infrastructure or deployment application deployment right so? It's in Yamal and Jason right so really anyone can read it like right even the ops guys can read it It's it's you know you may have hundreds of hundreds of lines of Yamal, but at least you know you can read it whereas You know it before when using tools like puppet and chef and even ansible to an extent You really have to learn the language or learn. It's like a coding scheme more than in It's not as there's a learning curve there right there's still a small learning curve with Yamal and Jason But it's you know you can easily read it Yes Okay, yeah, let me take this off here Is that better Maybe it's this thing is that better? There's a still hello better a little bit. All right, I'll try not to move Yeah, so A Kubernetes really What it really does is monitor state right so you it takes the desired state and the current state and it reconciles it so It'll be things like I want you know three instances of my application running and it sees to it'll spin up another container to meet That right so it's it really has it it has a control loop that basically monitor state your declarative state Versus a running state reconcile high-level Right is that easy? It's not just for containers anymore right so with the With the introduction of CRD CRD's custom resource definitions You can really program anything Into it right so you can if you know a little go we could pretty much program anything into a CRD and and CRD custom resource definition, so it's just real quick from a high-level CRD is a way to program things into Kubernetes without having to write it directly into Into the the tree right so it's it's basically like an extension of Kubernetes right so you can say so like there's things like pod Deployments you can write databases right and whatever databases is is whatever you define it as And you can further get further automation with operators right so this is Just from a high-level operators is writing operational knowledge into your CRD's so if there's a there's a talking operators I'd recommend you guys to go check that out so it's it's really It's really cool technology to be able to automate a lot of this stuff for you so So just my thought Kubernetes high high high level it abstracts the underlying infrastructure So I attended a talk long time ago, so This this will tell you how long ago it was it was a Kelsey high tower, but it was back when he was working at Coro s That's how long the talk ago was And just talking about Kubernetes v1 right so 1.0 He I think he explained it said something very elegant that how would you design a system if I took your SSH keys away, right? So as an admin as an operation as a developer if I took your SSH keys away How would you design a system and really that's what Kubernetes is it abstracts the underlying infrastructure So that way you can interact with the infrastructure and the the the VMs the the instances really just become a method to run containers And there's really no need need for you to SSH in right? That's the idea. So I Think that explains it really well. So if I imagine I took your SSH keys away so Kubernetes is right so How does get ops fit all in all in all this right so now that I went to through the through that whole history and that whole Thing about Kubernetes. What is get ops right so get ops very similar dev ops is a Is a series of methodologies and practices, right? The idea is is that use get as a canonical source of truth For everything in your infrastructure. So with Kubernetes Before you you had like I said before everything's defining YAML and JSON. We'll Just store that YAML and JSON in get the whole thing the idea with get ops is even the the Platform itself all even the Kubernetes itself all that export all that and put it in get and that becomes your canonical source of truth so any changes or Actually, let me backtrack a little bit the entire system described declaratively right so verses so you're talking about a Series of facts versus a series of instructions, right? So that that's it's a small a small Distinction right you're thinking about like ansible puppet Chef those are a set of instructions versus get ops the idea is this is a set of facts, right? So get out get ops Says like this is how the in my infrastructure Looks like everything right so any changes you want whether you're an operations guy developer whatever I Need to do a pull request, right? So I want to change the scale of my application pull request. I want to add another node in the infrastructure That's a pull request. So everything is declared Inside of get so that's that's the idea with get ops, right? Software can be used to reconcile differences, right? So this this doesn't have to be anything special There are tools out there right but you can actually use things like puppet chef Ansible to reconcile those changes, right? You can use this shell script But there are tools out there With two the biggest one is a flux CD and Argo CD. I'll be talking mostly about Argo CD But you can use software to reconcile those changes and Some of these tools like Argo and and flux which I'll talk about a little bit Uses CRDs, right? So it takes advantage of the control loop that's built inside Kubernetes to be able to reconcile those differences for you, so And I don't know if I mentioned before but changes can be automatically applied, right? So with that with that process, right the automatic changes, right? So some of the the benefits of using get ops is that since you're storing everything and get all of a sudden you get this convenient audit trail Right, you you know exactly what changes were made when and by whom so you get the compliance and stability of Leveraging your your your source control management that you already have in place, right? So you get reliability as well Consistency and standardization right so since you're using get ops You know that this is the the way to do it right in your environment Whether you're in dev staging production, whatever the way to get things through is through through get through pull requests, right? A lot of people call that operations by pull requests ops by pull requests So and you get that standardization, you know across across your Across your environment, right? So generally makes life easier, right? So It's it's as easy as all you need to change is submit that pull request Which makes life easier It speeds up recovery time, right? So I read they read this article about the guys that we works That someone accidentally deleted their infrastructure Like, you know, it sounds weird, but like stuff like that happens. I know I've done blunders in my career as well But someone actually didn't know what they were doing. They actually deleted their the entire infrastructure And they brought it back in 15 minutes Just by applying the get ops practices, right so And get ops really becomes a way to manage your Kubernetes infrastructure, right? Whether you're running Kubernetes or OpenShift it becomes a way To manage that system. It's how do you manage your Kubernetes system? A Lot of people say I just use get ops and that's how I manage my Kubernetes system there So so if I'm storing everything in a repo, right, how's that repo laid out, right? So Some of the things and some of these things I've gotten by attending some talks Mostly by the we've works guys But these are some of the things that pilfered from those talks is that only use one repo, right? So like you'll have a repo called foo. You won't have foo.dash stage as one repo foo.dash prod another repo foo.dash Right dev and another repo you'd have just one repo And you'll use branches, right? So you use a branch To determine what environment you're in or what stage you're in However, you're gonna you're gonna design that system You'll use one branch Use protected branches so protected branches important right in case of accidental change accidental deletion Use code review, you know all those best practices that you should use and get we use and get use it The same and get ops is the same idea, right? Leverage customize right so There's a there's a few ways To do things like overrides, right? So for example if you want to do like a like an a b deployment blue-green employment Or you want to change scale, right? So you have multiple clusters in one cluster you want five instances running and the other one you want I don't know maybe three or four If you're using the same production repo, how do you override that so you do overrides and Think with things by using a customize, right? So you can change things like environment variables Scale different clusters if you're managing multiple clusters with get ops you use customize Question that came up a lot was so what I do with my secrets, right? So in Kubernetes I'm using secrets database connections Things like that right usernames passwords. I don't want to store that and get Bitnami has something called sealed secrets That you can use is basically you're storing you're actually are storing your your secrets in get but they're encrypted and they're decrypted on the platform, right? That there's also other tools that you can use for that there's things like vault right ansible has also a way to store Encrypto secrets, so you just need a management system to Store your secrets, but you yes, the answer the question is yes, you do store your secrets you just need some sort of management system and Encrypting the secrets is probably the best practice. Yes question Yeah, so you can so the idea is to keep track of what's going on in that in that app In that cluster itself or in that deployment itself so you would have one Main repository and different branches will then say I have a branch called foo, and I have sorry I have a Repel call foo and have a branch called prod for example have another branch called dev That's that's the that's the idea behind that you have one deployment would be a deployment of of An app like not the whole application stack it could be it actually depends right it depends how you how you're designing your Your application, but I see it as you'd have foo is one component of One component of you know like microservices right because you want to track those differently. Yes You could use master branch. I don't normally use master branch. I use it. I Normally have branches Depicting on where I am in the in the environment. I mean you can you can use your master as your production. That's fine But I usually just branch them out differently So so again like I said before there is there's two I guess before I said you can use any any tool to reconcile those Two of the biggest tool sets is flux CD Flux CD was developed by the guys that we've worked and then they donated it to the CNCF right so it's a CNCF project Argo was actually developed by the guys over to intuit And That's That's really the tool. So red hat has been kind of just observing what's going on in get ops Get ops has actually been around for a while It was really pioneered by the guys over at we've works Right now was kind of monitoring, you know, what's going on how that evolved but eventually red hat is a settle on going to Argo We actually have an operator on Argo. So if you go to operator hub You'll see there's a community version of Argo Up there. I'm not sure are we do you know if we're including Argo in an open shift eventually For So yeah, so what we are We are going to include it in in In open shift eventually I'm not I guess I'm not not 100% sure when but we are Going towards Argo. So As soon as we decided we're going to Argo this thing happened a number 14th We've works and into it join forces And they created something called Argo flux, right is what they called it I guess it's just a codename for now. I would have called it Fargo, but that's just me AWS also did a joint announcement saying that they're getting involved in the project as well So they're building a community around something they call the get ops engine, right? I Intended the first meeting what they really really what get ops engine is is that okay? Well, we have the two different tools that do a lot of the same thing, right? They may they may have different end user experience, but at its core is doing a lot of the same thing So really they wanted to Take any pre-existing functionality. That's the same and just kind of that's not duplicate code It's really what what what their what their idea was It's not intended at least for now to be like a like a framework or anything like that It's it's really what they wanted to do is the wanted to create a library, right? So it's essentially right get ops engine. All right get ops engine. We have Let's not duplicate code. We have all the similar code Let's make one code base and then from that code base then we'll branch off to our respective tools And that's what first step that's what they want to do now The idea eventually is to have some of these one tool to rule them all sort of idea It's still in its infancy so the idea is is It might change so they're still talking about how that's gonna look like how it's you know gonna evolve from there Who knows so things are subject to change but Yeah, this is so new it's still evolving right as I was building this presentation They came out with this, you know after red hat decided. Okay. We're gonna do Argo. No We've worked send into a said we're joining forces. So we're still continuing monitoring that so I Had a demo so I had a demo actually to take up the rest of the time so My laptop isn't working And it's all in my laptop. So Is there any questions Yes Yeah, so the yeah, so the question was is about a get workflow. This is a is a get workflows With the different branches the same right as you would in the development the answer is yes Right, so you would make a change to your dev environment by doing a poor questing your dev And you would have some sort of process To promote that right so you'll have the dev branch then maybe deploy to the dev environment It'll sync that up. You'll see that at works fine. There's some sort of Approval process that goes through that whatever that may be and then that gets merged into let's say staging right and then it'll continue on from That so yes, so you would treat it like you would treat your code. Yes question So he's talking about tests right so if I make a pull request, how do I test the poor request? So ideally you would have a testing environment that you would see those changes so the The Some of the things that you get when you do an application deployment You also get when doing get ops as well, right? So if you for example, you say I want to scale my my infrastructure I want to add another note for example, that would be a pull request That would go into your dev environment and if that goes horribly wrong You can always a revert back because you just use the same the same process and get You leverage get to do you like your rollbacks or or anything like that For your tests your test could be anything really whatever you Put into Jenkins is what you test. It's mostly you're testing the application stack So you would just still test the application stack to see that doll. That's fine But you would do some tests as well for the infrastructure Some of these tools do some testing for you a little bit and We'll do some of the some of the rollbacks. I did have a demo for that, but Technical difficulties, but that answer your question at least a little bit Okay, okay There any other questions Yes Yeah, so the secrets as I yeah, so here you would essentially encrypt them right so you would have Let's say a config map with a username and password. Let's just say for example You would encrypt that with some key that's already stored on the platform and then you would you would store the the garbled up, you know Config map in get so when so when the sync happens when that tool sinks it It'll actually decrypt it and put it on the platform for you. So you would store your secrets Encrypted on on get somewhere that would that would be on the platform So you would it would already be on the platform So yeah, so that you'd have to have some sort of workflow That's why bit Nami a sealed secrets is good. You have to use some some tool that would manage that for you That would make a lot easier Yes question Also, the question was is the seal secrets a CRD of Argo city. No, it's a sealed secrets is its own project By a bit Nami so that would be it would be a CRD in but it's it's not it's not like on Argo or flux There are plugins and for flux for sealed secrets. Just not an Argo cool. Is there any other you have eight minutes left? I can leave that you guys go a little early More questions Okay, all right. Well, thank you very much