 So well, first just to introduce ourselves in some housekeeping items, my name is Dan Garfield. I'm a co founder and chief open source officer at Codefresh and the three of us are chairs on the GitOps, the open the GitOps working group and open GitOps, Scott. Exactly. Yeah, Scott Rigby. I'm developer experience team at Weaveworks and I'm a flux and helm maintainer and also, you know, help with this all around awesome person. Yeah, thanks. You're welcome. I'm Chris Short. I'm from AWS senior dev advocate over there. I work with these guys just some housekeeping items. We have two tracks. This GitOps gone first time ever. And you're here for it. So the keynote is going to be we're going to talk Christian is going to do an awesome keynote. And then after that, that's when we break out our second breakout room is upstairs in the Joaquin Rodrigo room one level up. Also reminder, no food and drink in here. No, no. Keep your mask on and there's plenty of stuff out there to drink. There'll be food later. The bathrooms are somewhere. I don't know. We'll find out. We'll find out sort of I had to be shuttle to the catering room just to put on my GitOps con t shirt. Yeah, all good. All good. Excellent. Chris, you are okay to take off your mask if you're comfortable as speakers, but I mean, I kind of want to get you did just tell everybody to keep their masks on. So I'm kind of I'm leading by example. So we're going to be kicking off and we're going to be talking about GitOps as a journey. And as we go through this talk, what we're going to reiterate are some of the GitOps principles and some of the output of what the GitOps working group and open GitOps have come up with. And this has been a ton of work from a ton of people over a very long time. And one of the outputs is this event that you're currently at. So hopefully the output is excellent. I know that we have an excellent number of speakers lined up for you. And we do have two tracks as was mentioned. One thing I didn't mention, I forgot to mention, we had some people that had some travel problems. It happens, right? Like people miss flights and stuff. So some sessions will be virtual. And the schedule doesn't necessarily reflect that. We're sorry. Everything happened last minute, I swear. Thursday and Friday I remember sending emails to CNCF about speakers not being able to travel here. You don't have to worry about anything. It's still in the same place. It's still going to be good. Perfect. Thank you. So first off, there are four GitOps principles. How many people are familiar with the principles? I've read these before. Open devops.dev. About half of you make sense. How many people feel like today they are practicing GitOps? About half. Maybe third. So everybody else is looking to get into GitOps, hopefully. And we do have a lot of speakers who are going to be talking about this. There are four principles that have been put together. And we're going to go through these one by one first. But they're really simple in their language. They get more interesting as you get into implementation. So we really think of GitOps as a continuum, as a journey of best practices and principles. And a lot of people, they feel like, hey, I'm doing GitOps because I'm doing this part. Or I feel like I could improve my GitOps because I can go to this next level. And so we think that this is not just a journey, but it's a destination. And there's a lot to be done here. So as you start your journey, you're probably starting the declarative side. And the declarative principle is essentially infrastructure as code. Yeah, exactly. Pretty much except for it's not just infrastructure. Infrastructure apps, policies, anything required to get your system of whatever kind you're managing. Let's just say it's a Kubernetes cluster because we're a KubeCon or any other type of systems, fully operational from zero. And while we describe what shouldn't be in Git per se or in your desired state store, all of your rules must be there. So if you have external databases or other things like this that have ETLs, et cetera, all of those need to be, should be in an ideal state in all declarative, everything declared. So this is not big news for Kubernetes people. But with Kubernetes primitives, but with everything else, it's not really quite there across the board as a community. Right. It's not cutting dry, right? All your state, your metadata, the whole nine yards needs to be in a store of some sort. It could be Git, it could be S3, S3-compatible, whatever you want. Yeah. And the reason for that is you want to be able to reproduce your environment, right? And you also want to be able to recover from a disaster scenario. Yeah. This one, declarative, is pretty universal at this point. I mean, most people understand they should be using declarative infrastructure. They understand the idempotency is super important. And that's pretty clear. So when we talk about... Intra, but not necessarily apps. That's the thing. And also not necessarily policies. Yes. For intranet, not necessarily apps. That's right. So when we talk about the system, like Scott said, there's a specific definition of this that we look at. And it really does include your entire stack, not just the infrastructure, also the applications, also the policy, all of those elements. That doesn't necessarily mean it has to be owned by one team either, right? Like multiple teams can be contributing to all that. It's not necessarily just you out there doing GitOps. It could be your entire org, right? So don't forget that. GitOps is for everyone. So now that you've got declarative, you've got your configuration declarative, the next step on the journey. And so a lot of people, if you've adopted Terraform at this point, you're probably at number one. If you've adopted Customize or Helm, you're at level one here. Though you may still need to do additional work to bring in those additional policies and everything you need. Basically everything you need to operate control access and bootstrap these systems. Level two over here is versioned and immutable. And this one trips people up because they're like, oh, that just means I'm using Git. And it's actually a little bit deeper than that. First of all, we don't use Git in the name of the principle, though it maps to Git as the source of truth. We don't use Git because the key is that it's versioned and immutable. And you can be using Git in a non-immutable or versioned way, for example. Yeah, so special precautions need to be taken even if you're using Git. But that's where the Git in GitOps, the name GitOps comes from. I forget who said it right now and I'll have to give attribution later. But someone said that it's the worst name for the best set of processes. It's catchy is what it is. And so Git is the primary, it's probably the most popular way of having an immutable state store with a full audit trail. But it's not the only thing you can use. You can use object storage buckets generally. And we're just talking on principles, not specific tools, right? You can use OCI. You can use other helm repositories or also versioned and immutable as well. And you need to make sure that the storage systems are set up to be that way. And if you do that, then you've got that principle covered. Yeah, and your references need to also be that way. We were talking with some folks yesterday who are using floating tags like a latest version in their manifest. Well, that manifest is no longer supporting a versioned system because if you do a new version, so it deploys and you need to do a rollback because you're using a latest tag on that image, you can't do a rollback by just changing the configuration any longer. So by reverting the configuration, you now have to introduce new configuration to introduce an older version. So it's not just about using Git. It's about using it in the right way so that it's going to support the kind of things you want to do down the road. So again, these principles, when you read the words, very simple. But as you start to dig into it, it becomes clear there's actually a little bit more to the story. So coming up on the third principle, pulled automatically, your desired state is always pulled. It's not only event driven. So up until this point, you might just be doing great DevOps hygiene with a simple CI CD pipeline and you'd be in business. But at this point, we have a divergence on that. Right. And don't get caught up on push versus pull. That's not the story here. The big thing is that if a state change happens, a reconcile loop occurs, or five minutes, whatever. Whatever your desired trigger action is, that's how you run it. And the system pulls all the configuration in, parses it, and then spits out whatever you told it to. Right, exactly. Essentially, this is the only place I think the push pull part matters is that your system does need to be able to access your desired state in your state store rather than the other way around being sent. Your system can still be sent to a ping that says, oh, pull me now. But it really is required to be able to access your state store rather than giving all of your system credentials to your CI system or whatever other system. Yeah, and this is pretty common as people make the transition from kind of a traditional deployment flow to GitOps. This pull mechanism, as a term, becomes important because you may, as Scott said, you may have a trigger that triggers reconciliation, but you shouldn't be reliant on that trigger. And this goes to the resilience of your system and its ability to always figure out what the source of truth is saying about how the state should be. Yeah, and here's one reason. Let's say you have drift because something goes wrong, not because you've made a change in your desired state that you say, I want there to be 28 replicas or I want there to be 38,000 apps instead of 37,000, whatever. You don't want that to be the only way that, you don't want that event that commit to your desired state to be the only thing that triggers your reconciliation because if something goes wrong in the cluster, something goes wrong with networking, let's say you have an entire system failure, who knows. You want the system itself to repair itself continuously just like Kubernetes as well with Kubernetes primitives. And so that should apply to the rest of your stack. And this is why I say GitOps is an implementation of DevOps, right? Like it takes it a step further. It gives you some guidance, some guardrails, and a lot of implementation details that, and a DevOps work is kind of like a dream come true because, oh, are we doing DevOps or not? We don't know. If you're doing GitOps, you are, for sure. Yeah, yeah. And these, I mean, we're going to get into the process of how these principles came about. So number four is continuously reconciled. So this is where we get into that drift detection idea. And drift is a really simple idea. Drift just means that the actual state of your system has diverged for any reason from the desired state that was previously set. So for example, if somebody were to log in manually to your AWS console and make a change, that would be drift because you now have something within your state that was not declared and did not go through this system. And when you think about the interface for how you move changes through a system, when you do GitOps, it means that they only essentially have to operate on Git. And so suddenly you don't have to have as many people accessing those systems in a manual way. And so much downtime happens because people access and make changes manually. And it doesn't go through change control. They fat finger a change. And it's also a security issue to have people making changes that way. Yeah. I work at AWS. I understand the console is big. There's a lot of products at AWS. It's fun. It's fun. We're making it better day by day, trying our best. But having your entire team access the console is not exactly the most secure environment. So you want to hand out those credentials very carefully because the real system that matters is your reconciler, your GitOps controller. That's the one that you want to have the full control. That's the only one other than some kind of break glass procedure. That's right. And this continuously reconciled thing goes to a kind of deeper concept. There's a concept within engineering called control theory. And there are open loop systems and closed loop systems. And you can imagine if we were going to chop down a tree, we would have an open system would be a machine that just swings an axe. It just swings the axe. It doesn't know if the tree is there. It doesn't know if it's hitting the tree. It doesn't know if it's making progress. It doesn't know if something has fallen in the way. It's just told, hey, swing that axe. But in a closed loop system, you can essentially say my desired state is that the tree is chopped down. And now the system says, OK, am I connecting with the tree? Do I need to adjust the angle of the axe? Am I making progress on it? Has someone moved in the way so I shouldn't be swinging? So there's this feedback mechanism that goes into that. And that's kind of where this idea of continuously reconciliation comes from, is that as a software agent, a get-offs operator, if you will, there needs to be some intelligence to understand what is happening in the system so it knows what to apply. So how does chopping wood relate to software? Here's an example. What if you've got, you want 10 replicas but you've only got eight. Is it because the system is working to create those 10 or is it because it's drifting away from that? And this is one of the reasons why, OK, we are at KubeCon. Get-offs specifically does not mention anything about Kubernetes and the principles themselves. But it's not a surprise that Kubernetes is the most popular platform that runs get-offs because in some ways get-offs is a natural extension of Kubernetes as Brennan Bern said recently. The system already handles those events and conditions very well and there's a really good setup for that. So building on top of that system in this context, like a tree, I guess in this sense, it would be, your controllers need to be intelligent enough to know what to respond to based on the feedback that they're getting from the actual state of the system. So that is one thing that the get-offs principle say is in principle four that your systems, excuse me, that software agents are continuously monitoring the actual state of your system and comparing against your desired state and then working to bring them to resolution, to reconciliation through, if you dive into the glossary, through specific actions and those actions can be notifying you if things are wrong. Those actions can be other kinds of policies or specific gates and they can also be, well, if these conditions are true then proceed otherwise, proceed with caution, sort of like how crash-leaf back-off works. Right. This is why I say, shoot, I just lost the phrase. But that's a good quote. Get-offs moves at the speed of the Kubernetes audit log, right? Like, it's really trying to parse that log and figure out if it needs to do something or if this is just normal operation. Yeah. And so when we talk about continuous reconciliation, let's imagine a scenario. Let's say that you had a set of Kubernetes manifests and you set up a cron job to kubectl apply those manifests constantly every five minutes. Right. Would we then be doing get-offs? Well, let's see, we have the source of truth from Git. Okay, so we've got checkbox number one. We're on our way. We've got it declarative. Sorry, that's checkboxes one and two. The desired state is not actually known by the system because it's just a cron job. So it actually, you know, if someone were to change those things in production, that would be overwritten, but not because the system was aware that it needed to be. So we're kind of missing on that one. And then on this reconciliation, there's no real sense of, well, what if I can't reach the server? What do I do in that case? Or what if there aren't enough available nodes for this deployment? I've applied the configuration, but I don't necessarily have any feedback mechanisms. So I'm missing some of those components. Right. So you wouldn't be not doing get-offs in that sense. It would just be operating in a normal loop or an open loop. And it may not be, you know, retrieving all the benefits that you could get if you dig into the supporting documents for the principles. So as we look at kind of a system overall, you can see basically you have your desired state, right? You have your actual state. You have this reconciler. You have your versions. And so you can see how when you introduce a change, your desired state has changed. That's the source of truth. Your reconciler sees that the desired state has changed. It sees that it diverges from the actual state and moves to the system. It attempts to apply the desired state. If there's an issue, an error, maybe it needs to send a message to you. Maybe it needs to wake you up. Maybe it needs to wait longer. Maybe there are security policies that aren't being met. Any of those things from a declarative standpoint is going to change the behavior of how this reconciler is going to act. And within open GitOps, we don't necessarily choose a winner and loser. We don't say, hey, you have to use Flux. Or you have to use Argo CD. You have to use Flee. You have to use whatever. We say, look, these are the best practices. These are the systems. There's going to be great implementations using a variety of tools. But you can see how this flow, it really puts the power in the hands of the developer to be effective with their time. And from a release management team, you always have a clear idea of, what do I do if something goes wrong? I can revert my desired state, and it's going to fix my actual state. And so that puts you in the position where we know that when people adopt GitOps, they deploy more frequently. They have fewer regressions. Their mean time to repair goes down. And ultimately, that's the goal. The goal of doing GitOps isn't to check a box and feel good about yourself. The goal of GitOps is to improve those critical DevOps research and assessment metrics, your Dora metrics, if you will, and drive those times down, drive those error rates down. And that's the goal of all of this. Who's right at Accelerate from GeneGam, or not GeneGam? I forgot her name. We've got three. We've got a few people. Accelerate sits on my desk. And I literally open it up every week and like, hey, this matters to you? I have not read it. You've not read it? No, I haven't. It's a great book. Read the audio book. It's even better because you get to hear Nicole Forrest-Gren say, winky face emoji. I love that. It's kind of nice. Yeah. So if we map these onto best practices, and I think this is something that, William, was this William Catton? Yeah, this William Catton. Kind of inspired this. Exactly. This is an update from William's graphic. So if you think about someone, if you're doing infrastructure as code, this maps really well to the first principle. You've got a declarative component. Now, you need the configuration as code. And obviously, you need DevOps and DevSecOps and those kinds of things. And as we move through our journey, infrastructure as code is taken care of in kind of principle one. You do need to bring it into principle two to have it implemented and have it done in a versioned way, in a mutable way. And then you, as you kind of move through this, you can see how it maps to a lot of practices that you're probably doing today. Yeah, exactly. And the thing is, GitOps didn't come from nowhere, right? It's in the versioned, in the 1.0 version of the GitOps principles. If you go to, you know, if you go to open-gitOps, GitHub repo and check out the documents repo, you can see the principles there. Oh, also on the open-gitOps.dev, right? It'll say that, you know, this is a set of practices that really, it builds on established practices over the last 20, however many years. Yeah, it's a lot of lessons learned. Yeah, exactly. So, you know, if someone says, well, GitOps isn't new, that's definitely true. You want to, we want to help folks as a group. And it's not just the three of us, of course. There's a bunch of people in the GitOps working group and that are participating in the open-gitOps project. We want to help bring what the specific things that GitOps provides that builds on top of these other practices to a wider community. Yeah. We often say that Henry Ford didn't invent the car, but he did make it accessible to everybody. And that's the goal of open-gitOps. It's to make these things that are the best practices, that we know work, make those things accessible and easily consumable. So, now that we've gone through the four principles, I asked you at the beginning of this talk to raise your hand if you felt like you were kind of doing GitOps all the way. So, let me ask the question differently now. How many of you, raise your hand if you feel like you've implemented all four of these principles fully within your organization and your stack? It's a very big diff. It's a smaller percentage. So, we're all on this journey, right? So, for those of you that have implemented all of these things all the way, congratulations, well done. For the rest of us, mere mortals that are still working our way through the journey, we feel like regardless of where you're at in that journey, whether you feel like you're at the end or maybe you're in the midway, we feel like the program ahead of us is going to be really impactful for you. And we've picked talks that we think are going to do that. So, these are the GitOps principles version one. They are at OpenGitOps.dev. So, if you need to refer to them, we have an excellent glossary that we put together that discusses the definitions of what a system is, what drift is, all of these different kinds of ideas that go into the principles. Yeah, please click the links. This was like the result of six months of debate between representatives of tons of different orgs around the world. We had over 60 companies, 96 different interested parties, a ton of co-authors, many of them are here today. And so, this is not something that the three of us made up. This is something that I think we have a lot of those meetings recorded. So, there's probably over like 40 hours of meetings and a lot of back and forth that went into each of these words very, very carefully chosen to make it as clear and simple and as accessible as possible. And if it's not, it's version. We can update it. That's the best part. Yeah, that's right. Get us the source of truth on this. Yes. So, the question is what's next? Like I said, this is a journey. And there is really, within this movement of GitOps, there is a huge frontier of implementation to be done. We've seen a lot of success of people implementing GitOps in Kubernetes. We're still seeing these principles start to come into traditional infrastructure. Is my S3 bucket? Am I managing that in a GitOps friendly way? What about my networking? What about my secrets? How am I handling that? And so, as far as kind of these frequent topics, we describe this as the breadth of GitOps. How wide does this thing go? And there are a number of open discussions in the Open GitOps repo. It allows you to jump in and help out with these and bring your wisdom as well, because this isn't something that a high tower council is going to decree down. But handling secrets is a constant topic. We're hearing people ask all the time, should I be using sealed secrets? Should I be using SOPs? Is Vault okay? And a lot of the stumbling blocks that people get into with GitOps comes around secrets. And technically, within those four principles, if you were putting secrets in plain text, you would be compliant, technically, with these principles. You would also have a breach, but yeah. Yeah, stop writing that down. Don't do that. I see you back there. Quote. Dan Garfield said. Hey, if they're not secret, no problem. No longer secrets. Exactly. Yeah, but this is the place to please participate with the group if you haven't, the GitOps working group and the Open GitOps project. The entry point is currently GitOps.com slash open dash GitOps is the org and the project repo is the landing spot for all of these things. The documents repo holds the... Outputs. Right, exactly. The principles and glossaries so far, but other documents that are coming. White papers and other things, best practices documents that have already been in progress for a while but are not yet up. Please join that and then join the discussion. So most of these are discussions. Some of these are just in notes. So we can send you a link to this deck so you can click on these things. These are just a selection of some of the topics that are coming. And one about part way down, I just want to mention because it's super important, I think, and it just happened is there's also a CNCF Environmental Sustainability Working Group. So for anyone here that is remotely interested in surviving past a certain point on the planet, join that please. And one of the things that we're looking at is, specifically for alignment, is how does and can GitOps impact that? For example, GitOps can allow you to turn off your IT. There's a lot of... And also if done poorly, you're not doing things unnecessarily and probably impacted negatively. So these are just, you know, again, some of the topics, please join in. And yeah, and I guess we're about ready for the... We're getting close, yes. So this is within the breadth of GitOps. These are the frontiers. This is where the exploration is happening. So we're hoping maybe next time we come up here, maybe there'll be some additional principles. Maybe there'll be some additional guidance. Maybe we'll have additional reference architectures and best practices. And we invite you as a community to participate in that. Come own that with the community because we're part of a movement. We're explorers. And it's not just mining the past, but we are also explorers of the future, right? So the breadth of GitOps is very wide and that the depth goes to how deep you go within your system. So many people, they're not doing GitOps maybe for their applications, but maybe they're not doing it for the clusters. Maybe they're not doing it for the access policies. Maybe they're not doing it for their database schemas. Maybe they're not doing it for their networking. So how deep within that stack can we go with GitOps? And there's work being done to expand the frontiers of GitOps both from a breadth perspective but also from a depth perspective because if we can bring everything within the entire stack, at some point, you know, someone has to physically move a server into a rack. Right. And I don't know if we're ever going to get that part. Totally GitOps maybe. But we will. We will. So these are the frontiers and the speakers that we've selected and... Sorry, real quick. Before you move on to that, I should say that Chris is going to give at the end of the day, the closing talk is going to be, it's going to go much more in depth than how you can get involved. So this is just more of a yes you can and then Chris will get into how. Exactly. The biggest thing is that all of these aren't solved problems yet. All those discussions are open still. We haven't totally figured out everything that we can do with GitOps yet. We haven't figured out all the implementations of GitOps. That is totally up to all of us. So please get involved and I'll tell you more about that at the end. So stick around to the end. We're all part of this GitOps journey together. So the speakers and the talks that we looked at today, they focus on not just the areas of success and well-worn paths, but also on these areas of exploration. And so there are a number of talks that address how to do secrets really well and success stories there. There are a number of talks that look at deeper into the stack, how deep they can go with their GitOps. So we hope you're going to enjoy that program and we thank you for coming and are just thrilled to have you. We hope the day is super successful. Don't forget to tweet about it. Hashtag that. GitOpsCon. We're going to have a great day. It's going to be awesome. Thank you for joining us for the opening session. We're going to introduce our round of applause because six months ago, seven months ago, we actually asked this person as a committee, can you go and make GitOpsCon Europe happen? And he jumped in, worked with Lindsay, organized all that stuff. His name is Christian Hernandez. Please join us. Give him a round of applause. Without him, we wouldn't be standing here. Well done, sir.