 Let's get into the big topic of open source, something that we actually have in mind. This is so awesome. We are an open culture that is in the community. What is that process that a developer or, let's say... How's the Kubernetes ecosystem really doing that? Everybody, sorry for starting the stream a few minutes late. I'm a chatterbox and got a little bit distracted by talking with... Today's guest. So I'm going to introduce you to today's guest. And I hope I don't butcher this name. I'm so bad. It's Vanya Pernat. Close to it. It's fine. Anyway, whom I will let introduce himself better than I ever could. He is a fellow Red Hatter and author of a book that we're going to be talking about today called Getting GitOps. And I can tell you from the previous conversation that ran a little bit long, just a very interesting and fascinating person to get to know. So Vanya, I'll let you introduce yourself and completely correct my terrible inability to say your name. And then tell us about yourself. Tell us about your book. Thank you so much. Yeah, well, I'm also a chatterbox, so that's not the program here. Thank you very much for inviting me. And so yeah, well, my name is Vanya Pernat. I'm coming from Germany. I'm living close to the Alps or southern Germany somewhere. And I joined Red Hat 15 years ago closely. So it's 14 points something still. And I joined Red Hat as a solution architect for middleware right after the JBOS acquisition several years ago. And my background is doing consulting, doing training, doing everything, being a developer and whatever you can think about. And at Red Hat, I was a solution architect at the beginning. And especially at the beginning, being a solution architect means that you're doing more or less everything. So from doing consulting, training, sales, pre sales and writing documentations and everything what you can think about. But I'm happy to be here. It's a long journey already. And I really enjoy staying at Red Hat. Because last year, my management came back to me asking me, well, you were blogging so much during your work and for work, of course. So I expect you to write a book about what you were writing or what you were blogging about. Well, and this is what I did then. So I thought, damn, writing a book. Well, I love writing. It's not the case. But writing a complete book. No, not necessarily. So this was not really what I wanted to do. But well, anyway, my management supported me in this way. And also from time to time, I got some kick in my back, right? And yeah, so I'm very happy that after a year of writing, everything is done. So which means you can now download the book. It's published by Red Hat developers. And I'm very happy that this is there. And no, I think that's it first. Yeah, and so that's what I say. And a year is actually, I feel very fast for writing a book. So I think that you actually managed to do that really pretty quickly. So that's impressive. Well, there was already half a year of blogging. Yeah, don't forget that. Okay. All right. Well, 18 months then. So we'll say 18 months of working on it. And that's really cool. So you've been at a Red Hat way longer than me. As I'm at a little more than a year and a half now. So you were writing your book the entire time I've been at Red Hat. For perspective. And, but yeah, so, but in general, right? Obviously we're here because we're both have, I'll call it a passion for GitOps, right? I know that I don't know how much of the stream you watched before. Some folks who've seen the stream will know this already. I spent 11 years in quality engineering and like CICD. So GitOps was like, you know, bread and butter at that time. And then now as an SRE, we still use GitOps. And then we actually also use GitOps for fun things like incident management response, some, you know, things like playbooks and running certain types of scripts that we want to have always maintained to, you know, collect certain types of like logs and events and so forth from our open shipped environment during during an incident. So I've seen, I've seen a wide variety of GitOps applications. I got to confess I haven't read your book yet. I'm sorry. I was on vacation and then I was just it did not happen for me yet. But I would love to hear a little bit more about like your overall like history with GitOps and like some of the things that you, I know you talked about being able to show us a bit about some of what you do with GitOps gated day to day. So if you could like just segue us through that, I think that would be great. I'll interrupt you to ask questions. If questions come into the chat from either YouTube or Twitch or wherever we restream this to, I will interrupt you and tell you what the audience wants to know. So, but yeah, let's just, let's just hop straight into that. Okay. Thank you. The problem is always the same. So when you start talking and then sometimes you think, well, am I still on track of what you were asked and things like this? So, okay. Okay. Thank you so much. So, well, my passion that the in my, in my past was always software development. Right. So I started when I was 12 years old or something like this with my good old Kodoba Amiga and of course also the C64. And then later, I think with 20 roundabout ish, I came into professional development. So where I was on university and well, a friend of mine asked me, so he owns a rental car company and he asked me to write the compete software there. Of course I did. Right. And so the biggest problem I always had was doing delivery of what I did. Right. So when I started as a Java developer in roundabout 2000, 99 2000, it was always the same. So I went over to customers, I went over to work from home implemented the stuff. And then I put everything on the disk and or USB stick or whatever. And then I had to go over to my, to my customer to install everything. Right. So this was my software delivery automation. Okay. So I have talked in previous like places. I've talked it on the stream before about being old enough to have had to deploy to production over FTP, not even SFTP. FTP. So that is a whole next level of pain. At least. I can do stories about this. I can do from one location. I, my comparison had it go good. I even had no software version control system in one of my projects. So we created a bowl and be right. The name of classes of the files we were creating on a piece of paper, put all those pieces of papers in the bowl. And when somebody wanted to check out a class name. So we had to search this in the bowl. And then we said, okay, I checked out this class. And everybody said, well, okay, great. Cool. That's where I worked over these days. I worked in a team of five people. So that was really cool. And then we had to check back in and then we had to merge the code. That was really interesting. I can tell you. Oh my gosh. I think we could spend the entire rest of the hour on this bowl system. And I might ask you back. Just to reminisce. Because I think that's what, my favorite thing to talk about is the weirdness. I mean, I think that's what I'm talking about. Because I think that's what my favorite thing to talk about is the weird things we had to do before the technology caught up with the needs. And that is the quintessential weird thing I've done because there was no technical solution story. So like, I want to put a pin in that if we ever get time, if I ever get to have you back, I just want to hear about the bowl system. Lovely. Yeah. You're welcome. So as you can, I'm not sure that I've seen or heard already. My passion for doing something more automated. Well, is there is given to say so. Right. And well, I think four years ago and like this, I joined five years already. I joined the email partner enablement team, which is, so we are well enabling our partners in EMEA for, for the use of our products and technologies we have. So my passion is open shift, of course. And when you as a, somebody with a developer focus has, have something like an open shift, you know, the biggest or the most important thing. When you have something with them, open shift is where you could automate everything. Right. Because suddenly instead of writing a script, which instructs, Ansible or the install or whatever to do what, what we want. We can just say, well, this is what I want. So this is my money manifest file with the deployment. Blah, blah, blah, blah, blah. And then I can tell the open shift system, please do. I just throw it, make a bowl out of it, throw it over to open shift. And then open shift takes care of the development or the deployment of everything like that. I really like that. But when you really want to do something more, how to say more special or something like this, you might want to have something like a tecton or you may want to have something like Jenkins to really have a reproducible builds at the end. Right. And then I created a new workshop for my partners, which is now, well, so my book is based on the workshop because I created some notes for the workshop because my partners after my workshop, they came back to me, asked me, well, you've told in those two days, you've told so many things. Is there anything which we can get over to download and things like this? So this was where the idea of writing a book or at least blogging was born already. And so I said, well, no, there's nothing, but I have some notes. And then I cleaned up the notes a little bit. And then I published the notes on a nice blog, which is organized by the German solution architects at Red Hat. It's called open sorcerers.org. And most of the stuff is already published. And then as mentioned, my manager came back to me saying, well, if you already have all that stuff created and published, so then please go ahead and create a book out of it. So, well, yeah. So what I wanted to do with the book or with all that writing is that you understand why it makes sense if you are on a Kubernetes or OpenShift environment that you are using the next step, which is putting all the configurations somewhere in a Git repository and then letting some tool in between, which is called ArgoCD or whatever, is reading the configuration file from the Git config and is then applying all those configurations back to the target Kubernetes cluster. And then, of course, I wanted to, well, I'm developer, I'm a technical guide, so I don't want to talk about GitOps in a theoretical perspective and things like this, but I wanted to let my people understand why it makes sense, what it is and how to use it. So the complete book is full of sources, example codes and everything. So if you, well, it's, of course, documented, but if you go to GitHub, you can find also the book examples there. Let me send you the link so that you can send it to the other audience. And by the way, with the editor together, we also did some kind of GitOps. So the authoring was GitOpsified, so to say. I wrote the book with the Markdown, and I had my Git repository on GitHub, of course, where all the sources of the book are stored in. And then I told my, the editor, if he's comfortable in using Markdown, he said, well, yes, I'm technical editor, so I am, which is absolutely great, by the way. And then I asked him, okay, if you are already comfortable by using Markdown, are you also comfortable in using Git? And he said, well, if you just help me a little bit. And then he was within a few minutes, he was up and he was absolutely excited about it, because then I didn't have to export my changes as a Word document so that he is then getting in to do all those corrections and stuff like this. And then I get back my Word documents. Everything is in red because I'm German, and if you want to understand what I was writing, you need to, well, do some corrections to say so. And then I took all those corrections and put it back into my source. And I told him this is too much work, I didn't like this. So let's do it in a way where people are doing this since ages already. So let's do it in a more developer friendly way. And he was excited about this. And so, well, we did it. This is really cool. I'm going to interrupt you really quick. Stephanie, our producer, let us know that the questions that are coming into the YouTube channel are not making it all the way through into our panel here that we can see them. So I've quickly opened YouTube, muted the site so that we're not getting an echo. And I'm just going to go ahead and let the... I'm going to go ahead and let you know what those questions are and give you a chance to answer them before we go a little bit further, just because I feel bad with these poor people. I've apparently been sitting there for a few minutes with their questions. So the first question was, you know, how would you briefly define GitOps to an absolute beginner? Okay. Think about that you are having the complete deployment configuration for your set of applications which you have saved and stored securely and auditably in a Git repository. And let's have a tool which is reading this and is applying that configuration on the life system. This is more or less GitOps. So all the changes you do, you are putting those changes back into Git so that everything what you changed is, well, auditable. Auditable. And so that you always see and know who changed what and that you can always also roll back the changes as easy as you are doing Git, whatever. Right? I hope this answers the question. Yeah. So I would say, you know, coming from, from a operations perspective, right? So you have a very strong development background. My background being quality engineering and now reliability engineering is more about scripting and operations, right? And so GitOps is basically the operations that go into, right, go into defining, deploying, and managing a software application. All of those are codified into Git and like Vanya said, like auditable, trackable, traceable, undoable. And then if you'd watched two weeks ago, our stream with Pinky, she said something, she brought up one of the points as well, also reconciles. So it has a reconciled loop, right? So which is that we were, especially if you were talking about like the system that's taking the, taking the reading, reading Git, right? If Git is the desired state, it is then applying that and making sure the actual state and the desired state meet, right? Any sort of GitOps tool is doing a reconciliation loop. So those are, especially from like my side of the house in like the operations, right? I'm going to be doing the day-to-day of like actually managing workloads in production on OpenShift. That reconcile is real important. I rely on that reconcile existing. And so GitOps, you know, GitOps tools ensure that that happens. There was an additional question, which is how much is the book and how do you get it? I believe the book is free to download from the developer's website. I think that it did not post to YouTube when Stephanie posted it. So I'm going to go ahead and, ooh, I don't know if I can post a comment to YouTube. I don't use YouTube like that. Stephanie reposted it and showed up this night. So it's there. That's good. All is well and good and true and proper. So I'm going to, now I have to double check all the windows. Okay. There's no other questions in Twitch. Sorry to interrupt. But I'm glad we had that little stop and segue because I'd like to make sure that people, you know, that we're engaging with our audience, even if it's a little extra complicated today. I'll do my best to get to everybody. And I'll let you continue. We were talking about when you left off that you were able to get, because your editor was familiar with technical markdown and so forth, you were able to actually leverage, get and make, make your examples very developer friendly. So that was roughly where we left off. And I'll let you, yeah, there's no reconcile loop happening in the comments. Thanks Christian. I'm doing my best to pay human reconcile loop right here. So I'll let you go ahead and pick back up around there in our conversation. Thank you so much. Yeah, so, well, at the end, so the first chapter was still this working with a work doc in between, which was really a bummer, so to say, or it was, it was just too much work, right? And then, but since starting from chapter two, it was quite easy to, to work with him because I did my changes. I committed everything to, to get pushed this stuff. And he then was pulling everything back to him. Then he was doing his work, which was quite a lot to be very, very, very, very honest. And then he created a pull request. I was able to accept the pull request and everything he redid, he did is already put into my source code, so to say, after I merged his pull request, of course, which was so great. So when you see this, then you know that when you're doing production work or when you're trying to deploy your application somewhere in production or so, that this is really easy. I like that so much, so that you're just automating everything by just applying a configuration change somewhere in a Git repository. But you mentioned this already. I have a demo later on. Yeah. So you use GitOps to write your book about GitOps? Yes. Like a GitOps inception. I love that. Yeah. So if you don't have anything else, like let's go into that demo. I'm excited. Okay. Cool. Great. Okay. Then let me share the screen. I tried it already. So it should work. Yes, it works. Awesome. I'm so excited about working technologies. Okay. So by the way, here you can just follow me on GitHub. It's github.com. So that's me. And here you can find the complete example directory of the book. And of course, also if you want to have a look at the working process of how we wrote the book together, the editor and myself, then have a look at ocp-book repository on GitHub. Here you can see everything. You can see the issues. You can see the pull requests and so on. Okay. Well, of course, everything is closed. But yeah, here you can see everything. And I'm very, very happy that that has worked out so well. And by the way, I do have an open shift, of course, installed somewhere here in my basement, in my attic, my hot attic. Yeah, you're in the attic. I'm in the basement. Yes. There's an entire theoretical house between us, but we're on different sides of the world. I think that's amazing. Yeah, it is. So I have around about 30, something degrees Celsius in my room. And I can hear my small server over there. It's an Intel look system. But anyway, it's still working from time to time. I think I can see some steam coming out of the server, but I hope it still lasts until I'm done today. So what I would like to show you quickly is installing and configuring a complete application, which consists of a server part. The server part is connected to a PostgreSQL database. And then we have client parts, which is a JavaScript, whatever, engine, which is connected to the server part. So when you want to have a look what I'm going to show, just go to, again, github.com, and then Quarkus grumpy cat. So this is the server part, and grumpy cat.melon.js is the client part. And what I'm going to show you now is that I'm going to install the complete application now by just issuing one call on the command line. So I'm going to the shell command. I am somewhere. I'm on my CI project. Okay, I don't need this right now. And if you have a look at the project list, you can see that there are some projects in there, but not those projects, which I'm going to create right now. So this is really clean and fresh installed now. So the first thing I'm going to do is doing an OC apply minus K for customize. And then source main github. And then, of course, I will CD. So this is a customized folder of the two applications, the development environment and the staging environment I'm going to show you. And so this installs basically the complete applications, all of them. So if I go now to the cat dev project, which you can see here, it's already done, which is cool. It is nice because, of course, I have already pulled the images locally. So everything is down here. And here you have now the server part. And you have the client part. And they are connected via some rest services between the two. And if I click on the URL here on the client part, then, of course, after a while, I get my nice game up and running, which is connected to the Grumpy Cat server, version of whatever. And this is also on my, so the URL is also on my local network. So now I can start playing with this. This is a very easy game. Well, inspired by, of course, something like Pac-Man or Grumpy Cat. In the first levels, I need to eat all those pills here. And I must not be catched by the Grumpy Cat. So the dog is my dog, and the Grumpy Cat is my Grumpy Cat. I just took some photos of them so that I have some images, some of those, for those sprites here. And then for enemies which are coming later in the second level, which are spiders or whatever. Okay, so I'm getting eaten by the cat or caught by the cat. So what I can do also is I can put a bomb in here so that the cat is not a little bit stunned. And then, of course, she is getting more angry and she is starting to follow me even more angry. So to say. So this is my project I'm currently using because when I'm doing my workshops with my customers, with my partners, then I was always, I didn't like all those Hello World applications. So which is just a basic Hello World rest service, for example, which you write in Quarkus, for example. And I wanted to provide something more special, something more meaningful. And this is really a meaningful application. As you can see, if we go back to OpenShift here, it is not a very simple application. It's still quite simple, but over the time I will also put some Kafka service in there so that... I was just going to ask you about Kafka. I'm literally wearing the ROSAC, Red Hat OpenShift Streams for Apache Kafka, shirt, long shirt, because I'm on the SRE team that supports that product. And if you remember the Summit demo for Kafka, right, they used Ship Wars, which is like a battleship game and all of the data is aggregated into some Kafka topics and then visualized. We actually are still... In the SRE team, we use this Summit demo and I would like to add your game to it if I can. We do a chaos engineering game where we spin up a standalone copy of the managed Kafka infrastructure and we install the same as production or whatever and then we have a red team going and break it and it needs to be break it in an observable way. So we use the Ship Wars to create some sort of observable environment where we can have people who are just anywhere in Red Hat coming in and playing the game to generate data, production data for us while the red team, after they have broken everything and is now reporting on the issues they're seeing from a customer perspective and so is the people playing the games. And then there's a blue and the green team who have to play like the customer success engineers is the green team and then the blue team is the SRE team to figure out what did red team break and fix it and so we've been pulling in things like these games that have been used for Summit demos to enhance the experience of the event. These are like two to three hour games and they're very fun. Anybody at Red Hat can play them. So if you want to be added to this, let me know. But yeah, and so this is how we, one of the ways that we kind of help improve the resiliency of our services by doing this for and Kafka is not the only service we will be doing it for. It's just the one that we did it for first. And yeah, and so a little games like this are really great. Your SRE teams appreciate them because if we can reuse them to make our lives more fun, we will. Okay, that's really funny. Yeah, so exactly. So I need some Kafka in here because I would like to put all the player movement data into the Kafka store and also the enemy movement data, of course. And maybe I have in a while, so in three months from now or something like this, I might have an updated version of the client where you are able to play in the team against the cats or something like this. It's just a private project I'm using for, well, also JavaScript education and things like this. And by the way, being a longtime Java developer and then getting over to do something like JavaScript, it is first of all, it's amazing what you can do with JavaScript, but then it's not type safe. I know. Issues, it's really, it's amazing. I know, I know. So TypeScript is though, so you would probably like TypeScript better. The other thing I don't love, and this is again, I'm a very operational background person. JavaScript being 100% asynchronous drives me baddie. Oh my goodness. That makes me nuts. There's a lot of things that I do in my day to day. I need things to happen in a very specific order in a very specific way. Please just do the things when I tell you, don't try to do them all at one time. If I wanted to do everything all at once, I would use Bash. Bash is great. I know what you mean. Yeah. So that was also at the beginning when I started with that, but yeah, I know what you mean. Yeah. So yeah, well, and of course, if I'm, if I would like to deploy a new version of this, then I have some I have a CI project, of course. This CI project also needs to be installed first of all, which I do by executing a simple script, right? I have prepared that with this one. So I've created that script. And this script is just generating a secret in there for accessing a Git hub and for accessing a query. Oh, and this is the reason I need to use a script here. Otherwise I could do it with an OC apply minus K as well. Right? So this one has now created the cat CI project with at least two, two, two, with at least two pipelines in there. So the pipelines are created from Tecdon, which is, if I go back to that cat CI project now here, you know, of course it also installs a nexus. I now have two pipelines, the dev pipeline, which is cloning the source code of my QuarkOS Grumpy Cat application. It's also cloning the configuration from Git Hub, then it's packaging everything, it's building the image, and then it's updating the config repo, which you can see here. And, well, let's do this now. So let's go back to this one. So now I need to execute again that script. But with a different source main GitOps, with a different parameters, of course, Tecdon and then pipeline SH. So you can, as mentioned, you can find all those scripts in the example directory of in GitHub. You can find everything. And by the way, this is also a copy of all the examples coming from my Getting GitOps book. So it's exactly the same, but it's a nice example. Right? So now I would like to start the build pipeline of this. And here, again, I need to provide the Quay token, which I change, of course, every time I do this in GitHub. So, cleared. And now if I go back to OpenShift now, we can see that the pipelines are about to run. Of course, it's the first time. Now we need to download all the dependencies from Maven, which takes a while. But after a while, it is being built. You can see the dependencies are being downloaded now. But after a while, everything is update or is built, which means the image is then stored on Quay automatically, which could be, of course, also an internal enterprise, Quay, Docker, HUB, Docker, whatever repository. It must just be compliant to the Docker specification. And well, then you can find it in here after a while. Of course, with the newer version, it still takes some time. And yeah, I need to change the base image for the GranPiket server part, which is fine, but I will do this later. And yeah, it's still downloading the... I still have to say, though, like, yeah, it's downloading that stuff, everything takes a while. The fact that we now have GUIs to have, and just like that surface level observability of our CI and CD, right? CI and CD or deploying software was probably the last thing to get some sort of observability added to it, truly, right? Christian and I talked about this when we went to KubeCon and we were talking about GitHub's Con, and I actually very specifically talked about this, that basically we started off with... We had to do things by hand, right? Like physically delivering disks. Although, usually that was done via mail or sending them to a store, not necessarily driving in your car, but that's also very valid, right? We used to do things like that to deliver our code or deliver our production systems. And like you said, now we're just promoting manifests. Exactly. And so there was like... We were able to observe how our application was doing with things like uptime checks, like Pingdom and so forth. I would say much earlier on in observability as an industry discipline compared to actually delivering production systems. I feel like that got observability almost... It almost got it last, and we were doing... Like I said, we were doing bash... Geez, I can't even speak today. Bash scripts, and then we had make files or some combination thereof. And the fact that now we can be roughly know what's going on with our promotion based off of an animated GUI is like, we have come so far. It always happens. Yes, exactly. Exactly. I really love it. So the nice thing really within where the typical compilation time now I now do have my updated... Well, my updated image here, which was just published two minutes ago, as you can see here. And then of course we now go to the Grumpy Cat config repository, which if you want to do it yourself, you have to clone everything and then you have to go to the script and to make sure that you pick up your own GitHub repository or Git repository. Now we can see... Sorry, I wanted to interrupt you real quick because we did have a question from the audience. Of course, Stephanie, just as she starts trying to adjust the visual to give everybody a better experience, I like interrupt you or whatever, right? We really have to just pause and appreciate Stephanie, our producer, who worked super hard in the background to try to give us the best possible streaming experience and is currently acting as the reconcile loop for the comments between YouTube and Twitch for us. So really just appreciate Stephanie for a second here, folks. But the question is... And thank you, Stephanie, for making me aware of it. How would you describe the distinction between GitOps and infrastructure as code? I know how I would answer this. I'll let you answer it. We'll see if we're... I think we'll probably be aligned. Okay. But distinction is the difference or the overlapping? Yeah, let's talk about the difference and then let's talk about the overlap. Because, yes, that was one of the things that I was going to bring up. Thank you. I was just wondering... I don't know the word distinction right now. So the difference is that GitOps has that one part in the middle at the end which is doing all the changes or it's applying all the changes automatically. So when you see here that my Git repository is now updated which you can see here. So the only thing which was really updated three minutes ago as you can see there is the customization file and at the end we now have a different target image which should be used. And all the rest, so the part what GitOps is doing is that the configuration apply step is coming from that external tool which is in our case AgroCity. It is not coming in a directive manner from the pipeline any longer but the pipeline ends when the configuration has been updated. And then some minutes later automatically or on request or whatever that tool in the middle is coming is picking up the changes and is of course applying those changes in the dev stage as well which means we should... It's restarted three minutes ago as you can see here in the chat. So I hope you can trust me so it really has reused that new version of the server part now. So this is the difference. The overlapping part is that you're doing of course the configuration or how to say that you have infrastructure as a code so you use all those manifest files coming from Kubernetes in this case and you're maintaining those manifest files and those manifest files which you have stored somewhere those are the master of the configuration of your application. GitOps is basically a way to manage that. So you use GitOps, Git operation Git for operations you use it to manage your code but that is not necessarily infrastructure. Your infrastructure as code can be defined completely separate from your application. Obviously in a lot of the ways that we do things now with OpenShift that's not really how that actually works in this kind of environment but it could. You could have VM configurations as code if we're not looking at Kubernetes space. So infrastructure as code is your infrastructure specifications and requirements are in like manifest files and if you need to make changes you update the changes and then that applies through GitOps as the mechanism and there's ways to do that we're talking ArgoCD which has a polling mechanism so your pipelines make the changes available to be found and then the polling mechanism within ArgoCD finds it and then applies it there are other ways of doing this which are more of a push mechanism this goes down to some fundamentals of system design the difference between when you would want to do a push when you want to do a pull the nice thing about a pull is a pull will typically happen in a state of time when the system is ready it's not going to necessarily pull when it's already in the middle of applying a configuration upgrade it's going to wait until it's done and then check back after a period of time so that's just a little bit more of a sustainable model so the GitOps is the way you promote your whatever is code your infrastructure is code your actual code, your manifests and the as code part of the actual definition piece and you may or may not actually use GitOps to apply it they're joined but separate considerations that's the best way I can describe the distinction I think I know Christians in the comments I would love to hear Christians take on that if you're still here Christian I'd love to see your input in the comments on that okay so I keep interrupting you with audience questions which is great because it means we have an engaged audience it's terrible because I don't remember where I cut you off now we are approaching we are approaching top of the hour though so just reminder we've got about nine minutes left so I'll let you continue but like because we only have like nine eight and a half minutes left get your questions and now people so both have to maintain data provenance in case of rollback okay I want you to clarify so there's a question that came in asking if both have to maintain that both GitOps and infrastructure code have to maintain a data provenance in case of rollback I think that's what you mean both GitOps and infrastructure is code and the answer is no you have that single source of truth is Git so whatever you need to change if you have to do a rollback or whatever it's all going to be initiated in your source of truth of Git except except so I'm an SRE and sometimes it's in an open shift environment in an open shift environment we are talking about we're talking about things that are very stateful and I'm an SRE for applications that are installed and somewhat maintained by what's called an operator so a GitOps well yeah they're GitOps operators but that's not what I meant to say a Kubernetes operator is automating the otherwise human tasks of running and maintaining a software application so if you want to have a highly available software application you need to have three replica sets of these UI pods spread out with node affinity rules that keep them on different nodes through a rollout as each node basically comes out of the pool and gets updated to the latest version and then goes back in your software application is still available because things get moved around and low balanced and so forth so everything being stateful operators Kubernetes operators help ensure that state they apply the stateful need of the Kubernetes infrastructure layer and they expand that paradigm into the application layer and so when you're actually managing an incident in that type of thing now keep in mind Git was the source of truth for what that all should look like and then the operator is maintaining that source of truth and you could you know update mechanisms and everything right in an actual management of an incident where there are pressure considerations on the on the system an SRE might go through and make temporary changes to the infrastructure with the idea that what is in Git goes back to being the source of truth and so there are ways that we will management I want to say like a year ago Christian had somebody on the stream that had I was showing a tool that where you could like from the operations real life like incident management side make those changes and have them be like basically stop the reconcile loop because there was a mechanism in that to stop the get off reconciliation so SREs will find find tools or ways or whatever to temporarily bypass the configuration is code or will update it and then roll back our update when the incident is done being managed but so typically speaking Git is the source of truth except when you have to have a snowflake situation in your fleet because something terrible has happened the good news is that something terrible has happened is not common so saying that Git is your source of truth yeah the big red button slash cancel the saying that Git is your source of truth is really true so you should not be maintaining separate data streams exactly and in our case here the set operator as you mentioned is the tool called AgroCity which is well has a set of applications as you can see here so it's BookDev, BookStage and those are the old book related applications then of course also CatDev and CatStage and when you click on one of those you can see exactly how or how many resources are in the target system or open truth system so that represents exactly the state inside Kubernetes for the CatDev application which consists of the server part of the database of the client part and that's it those three so this is what you can see all here and if you see that there's somewhere a lock problem or whatever then you can either refresh like this one or you can re-sync the complete stuff and when you do a re-sync you can for example say well something that has happened over the last automatic sync so let's just prune everything also delete everything what you have in the target name space and then let's please reapply everything what's called a part of the application you can even say well the head revision doesn't work any longer so maybe there is a problem in there or whatever so we can use a tag so for example the release tag 1.001 or whatever and then you can easily roll back by just using and pre-existing git tag by applying this git tag here and then you can just click on synchronize and everything will be re-synchronized based on that on the configuration part stored in git which is version control which is tagged with a given name so that you can for example that you can create a tag on the image in your target image repository you can tag the configuration repository and you can tag of course also the source code so that you can always know where everything is in which state everything is that you can download the stuff that you can apply the configuration changes and so on this is what I really like in doing this GitOps stuff yep the RBCD is a really great operator it is I always talk about this operator hub.io is a great way to see what Kubernetes operators are around if folks are interested in learning more about them like I forget the website but that's there's another place that just talks about the operator SDK it's not operator hub and I always forget I'm not even going to try to say it I need to start like a sticky note on my wall that tells me what that is because I bring it up a lot so operators are great the GitOps operator is also a particularly interesting and good one and so when we talk about the reconcile loop that's one of the things that operators have is a reconcile loop they're always double checking that the desired state matches the real time state and if it doesn't it fixes it and that reconcile loop is real important to GitOps and automation of operations in general so we are running out of time now I need to close the stream with Stephanie and I should have discussed announcements so in two weeks we will be joined when we come back here we will be joined by Nathan Yellen CEO of a tool called Robusta.dev which talks about using GitOps to manage Kubernetes learning I hope I represented that right if I'm wrong he'll correct me next show and then I will be myself in August amongst all the other August streams I will be speaking at DevConf in Boston and the actual topic is it's a talk I've given before it'll be slightly updated because technologies have changed it's called Helm and Back Again this is one that I've given in the past with Christian and also by myself so I'll be delivering it without Christian this time and it is actually a comparative analysis of Helm charts and Kubernetes operators so it's like very related to some of the stuff we started getting into at the end of the script of today's show Christian would like to remind us that the call for papers is still open for GitOpsCon North America that's a co-located event for KubeCon North America which will be in Detroit in October and the CFPs close on July 25th actually so get your CFPs in July 25th is also my birthday so double get your CFPs in just because it's my birthday and that would be awesome I will not be reviewing your CFPs but it would make me happy to know you submitted one for my birthday so and Christian will tell me because Christian Hernandez former co-host of the show he is on the review board for GitOpsCon CFPs so he will be happy and he and I have been friends for 10 years and 100% tell me so that's pretty much it for announcements the show thank you again for coming and talking to us and sharing your really cool demos and your really cool games I will email you offline about how I usurp your games for my chaos engineering exercises again anybody who is watching the stream who is part of Red Hat who would like to be involved in the chaos engineering exercises please email me my my Kerbos is HLipSig so you should be able to find me otherwise you can tweet me and then we can sync up over work email that way I'm on Slack, I'm on gchat for Red Haters for non Red Haters who would like to do this we are working towards an open source story for these games and if you follow me on Twitter you'll eventually hear about that so because for Red Hat we opens everything so I'm working on it it will happen someday I hope definitely if you don't think of anything else to tell me an ounce real quick I guess I will remind everybody to like, subscribe thank you for watching and as I always say choose your technical technical debt wisely folks bye bye thank you, bye bye