 All right, what's up friends? Thanks for coming to our session. We're here to To start talking a little bit about how get off to sort of changed everything for all five of us here And hello to all of our friends online as well. Thanks for joining us from the virtual side of kubcon The room here is packed and they were all very polite. So we almost started early, but thanks for joining us Yeah So we're gonna start off with just a little bit of intros because you know At the end of the day it kind of looks like we've got five friends up here Who are just bantering and so why should you listen to us and where do we come from right? So over here on the far side. We've got Christian Hernandez Christian I'm gonna kind of tell us a little bit about yourself. Yeah, so I am Product manager at Red Hat and I am the member of open get-ups. I'm a maintainer of open get-ups. I am also a contributor to the Argo project and the I'm part of the The Argo, I guess communications a group as well. Yeah, thanks Christian and Philip Yes, my name is Philip Johnson. I work at personal school for us as infrastructure architect So I work in our platform team with the Kubernetes and open shift Thanks Philip My name is Robert. I work as a principal cloud engineer at a Mestal 42. We do cool stuff. Check us out I am Well for this palace, I'm a CNCF ambassador. I have a lot of other things go to my LinkedIn I don't want to spend all the time talking about myself, but I'm also one of the open get-ups maintainers and heavy get-ups advocates Yeah, so just a note here. You see I also use it. I build services with it I do everything with it. We've got a couple of maintainers up here on onto the next year pinky Ravi I go by pinky I am a developer experience engineer at we've works and so Before that I was actually at a company an insurance company in the US called State Farm and There I actually helped set up get-ups on our Kubernetes platform using flux. So yeah, very cool user story That led to pinky being part of the flux team. Yeah, that's how we ended up joining. So yeah, and then hey friends also I We've worked her in spirit here. My name is Lee to healing I'll be our spicy Panel moderator today. I'm a developer advocate with the VMR tonsu team Flux project member and also fond of the folks with the Carval project and what they're building there and I will be helping these folks Showing you the guide of where get-ups has kind of come from and sort of where we're going and Thanks so much for coming and joining with us. So Yeah, Christian Philip Robert pinky. I'm gonna be opening up with a question on just what brought you to get ups and how did you decide that you needed this sort of Technical methodology in your practicum in your in your life, right? Yeah, who wants to start I I can at least start with this one the what would really like attracted me to get-ups and The you know all the other buzz around it and all the practices around it was it was really kind of an evolution of things We were we as an industry were trying to do like all along with various You know scripts and automation and processes and You know with with Kubernetes right with with the the expansion of Kubernetes and with it blowing You know blowing up the industry like we now had a platform where a lot of those things was possible And so I think the the very first thing is like get-ups to me in the beginning already seemed to mature Even if it was even if it still is kind of still a little early in the evolution of get-ups, but It's because of all the stuff that you know us as practitioners You know devops CICD folks have been trying infrastructure folks writing for structure as code We're trying to do a lot of these these things already so that what attracts to me is like it was almost like mature In the you know from the beginning so those it was kind of so you were seeing the results of maybe the decade prior of continuous delivery and Like mature operations with pole-based workflows and you saw that sort of realizing itself in a new world of tools exactly exactly It's it's kind of like and I think that's that's a great way to put it putting it in. I was seeing it realize With all these new tools and with the cloud native Where were you working when you when you first sort of made those initial realizations with kubernetes specifically actually you know I've been I was I was that red hat and I've been working, you know I've been right up for you know nine plus years, so it was you know early on that I saw the It was really in in in the evolution of kubernetes when I first saw those, you know kind of declarative Programming models. Yeah programming models and I kind of like oh, okay, like there's there's this whole new methodology of way of thinking I'd love to turn the question over to you Pinky in the same vein Because you so so christian came from a home of open-source a steward From state farm yeah, enterprise bureaucracy, right? Yeah, yeah, no, no, he's not wrong if y'all aren't familiar with state farm Here in Europe. It's actually an inch a large insurance company back in the u.s I think it's like number one in auto so it's like a very highly regulated industry like we had a lot of compliance and Auditing standards we had to meet lots of red tape so The first time I heard of get-offs and like the company heard of it was actually we sent two engineers to cube con back in 2019 And they came back and they were talking about this thing called get-offs And they were like man, we have to show you this stuff This is so cool and they were showing us flux and they were like this like tool is amazing And we were like yeah, it's great like this get-offs concept is so cool So we tried to take those principles and we were actually Moving like some stuff to AWS at the time so we were Trying to fit those principles to our AWS deployments. We didn't fully get we didn't get the pole model, right? But but we did do our best to make it get-offs we were using terraform enterprise to do our deployments for terraform and then Yeah, and then then we started doing the Kubernetes platform and When at the time we actually didn't really do a like a comparison. We just kind of were like we researched it We're like, oh, let's go with flux That's the one that had been talked about at the conference He said I think in an anecdote when we were talking that that was a Google search We typed in get-offs. Yeah, and and we've works popped up and so we were like, okay Let's let's go with this and it's been that way ever since but yeah, so um, here was that that was 29 2019 2020. Yeah. No, maybe no 2020 by that point. Yeah, so 2020 get-offs is finding its way Yeah enterprise. Yeah, and it and it was a lot of work. So like those of you that are going through I think trying to like You know start get-offs at your company I really relate with y'all especially if y'all are in regulated industries Like we had to make sure that we met compliance standards We had to have auditing requirements like and then we had a platform team that already existed for Kubernetes So we had to like convince them that this is the like a good way to go because and they weren't even doing declarative A declarative approach at all. So it was a lot of work and then the developers have to be onboarded. It was Yeah, lots of people to convince so I feel free all Philip Robert you want to kind of talk a little bit about where you discovered get-offs and and how it made its way into your workflow? Yeah, so our story is a bit different We start off with the Kubernetes about a year ago And we did it together with red hats. So we did they have a concept called container adoption journey So we have a dedicated team within the company and then they bring in Consultants to help us with the journey. So they basically say it said that yeah, this is the way you should do it. Yeah Yeah That's 2022. Yeah, that's great. Okay. So yeah, that was real nice and Along with the project, there's a lot of other dependencies around Kubernetes that you have to think about as well So our approach in the beginning was like it would be really nice to do everything Infrastructure as code like deploying stuff in public cloud and and so on on premises slow balancers everything So we didn't think that we would be able to do it because yeah We haven't really been doing this before and it seems pretty hard But yeah, we end up having like almost everything within our OpenShift deployment Defined as infrastructure as code and using github That you use those two words or those terms the infrastructure is code and github what where sort of Do you start calling a methodology infrastructure as code versus maybe when you get to kubernetes world? Yeah, so we're doing some infrastructure as code that I think is not github's so we use we're using Terraform to deploy stuff in Azure And that was basically doing Manual CLI deployments in Terraform. Yeah, and also using like you get lab pipelines So that is not really automatic reconciliation and stuff like that someone has to push the button There's interrupts. Yeah, and the people make decisions exactly you get back into the social loop of what is our problem? Where are we getting to and the infrastructure is sticky enough to where you have to do that? Yeah, and also when we were deploying kubernetes clusters, we're using ansible to do that. So there's also some manual Interactions when we do that. So it's not really github's but it's within the infrastructure as code realm at least Yeah, thanks, and then Robert you yeah, you are quite active with with open github's. Yeah. What's your story? How'd you get here? Well, I think it's kind of like the natural revolution of it all like starting out as You know what we in Norway call a potato just doing everything and and then I kind of Potato, yeah, and then and then I like I found over email. That's not great to work with because end users So I stopped doing that clients don't want to do that because end users and then I came into the infrastructure part of things and Really discovered Terraform and infrastructure as code as like the way to do stuff in in the cloud You can't really do infrastructure in any other way and through that. I was also like just like looking for ways to To to do like an even like better job at deploying stuff Efficiently and I just kind of fell over get offs at some point Don't exactly remember where but and that's just when the the get-offs working group was kicking off People are starting to define these principles So I kind of came in really early and They look easy, but there was a lot of meetings per Principle, yeah, I mean if you would so kindly could could you rattle some of these often maybe sure? Feel free to stand up. No, no, no, I'm putting Robert on the spot So the first one is declarative like you're you want to it's all about defining the you have you Desired state of your system that that is the entire thing and we want first of all it to be declarative You don't want to have imperative like steps defined. We want just this is what we want Mm-hmm, and then we'll coupe cuddle creates and coupe cuddle updates. Those are imperative. That's imperative That's that's a specific step if that fails for some reason, you know, you have to intervene That's that means you actually have to be like we wanted to just be able to define this is what we want And then but even a tool as simple as coupe cuddle apply is starting to get into declarative here Yes, one piece. Yeah, you know at some point. Yes, but and then we can kind of split hairs about that But yeah, yeah, sure And the second one is that that desired state needs to be Stored somewhere that's versioned and the mutable which basically just means that you you don't change the state As is running you're defining a complete new version, which means that if Something happens you can always revert back to a state that is actually working because you know this because it's all versioned and immutable and And and when I say these words, they are very we thought about Making sure that we don't like actually say Come on the Kubernetes controllers and gets and stuff like that because you know We feel it's it's higher than that even though it's called get-ups. So this is not a Kubernetes specific thing, right? It is not but it's so much easier if you're doing it in Kubernetes like you could do this But you would have to you have to make a lot of things happen So it's fair to say that get-ups is not Equivalence to Kubernetes and working with Kubernetes, but that no Kubernetes is a huge reason why the way of working is Changing yes amongst all of us. Yeah, right. Yeah, that that's what kind of enabled us to do this and you can do it in other things But just just show the way so to speak. This is one of the reasons I get so excited about get-ups. It's because It's not like the computer science that makes Kubernetes possible is new right, but the democratization of that Kubernetes resource model the KRM Creating a different programming model for the way that we share big distributed systems It's begging for us to change the way we work Yeah, and that's exciting right because because we need to change the way we work if we want different systems, right? Right. Well, um, then in in the spirit of changing the way we work I'd love to take it back to Philip's right because you've got an organization. That's that's changing the way it works Yeah, right. So one of our our next kind of pre-desired canned questions here in my cheat sheet is How far along is your organization in adopting get-ups then right you said you brought some smart consultants Yeah, and they said this is the way The way got some red hats right and then and then you they got in and you have this mixture of infrastructure as code as Well, but you were surprised right by by how much could be done, right? What did that look like? so We haven't come that far yet with Kubernetes said we started about a year ago But everything we do within Kubernetes that is Get ups. It feels different than the way of working before. Yeah, so we're kind of stuck in two camps We have the traditional way of working and we have the new way of working and then That's also correlates to the teams that are working with the different things like we have still our traditional infrastructure teams that are working with networking and storage and Virtualization and then we have the teams that are working with the new platform so yeah, and It's kind of a struggle for me as infrastructure architect to try to get everyone to get along and it tried to Get people to adopt the new ways of thinking and How to evolve and How you do things? It's funny because it's almost like if you were around in other communities say like DevOps days 10 years ago, you could hear somebody Almost with the same exact experience, right? Like it's not necessarily the tooling But the changing the way that we work across teams can be kind of difficult, right? Yeah, you're feeling that pain a little bit, but that it's also interesting and worth it Yeah, it is but and also I think that changing the way we think about things and Getting into this mindset of everything should be declarative It really open up opens up new ways of thinking like I think some people here have heard about the backstage So when I first heard about that was well, you can get helps this So it's basically get up seeing our old Enterprise architecture software the workflows the way that developers and everybody who wants to touch and change the system Like interacts with each other and get information and also like the the actual information the charts of the systems that you have within the company before or Now we still have those systems that keep them in big databases and it's a hassle to change. Can we just put that in get instead? That would be great Can you drill it more into that right because I think you said something really intellectually interesting there, right? We've got these old systems. Yeah, they're they're a pain to operate What what do you mean by pulling the state into get? Yes, so I was talking about the enterprise architecture systems where we basically model our entire Organization like both the systems that we sell or systems that we build and all the services that Makes up those systems. So In order to know what we have to have an inventory of all everything we have We need to document it somewhere, right? And there are systems that do this and you have to use some application put that into a database and fill in metadata and Enter like dependencies between different objects, so Why can't we just use declarative setup use get the story it we have version control Can have you seen this in practice to Christian and I see you nodding a little bit on terms In like how to deal with these stateful systems and kind of get more knowledge about them. Yeah. Yeah, and it's Some of these things so it's like a get-offs will as Philip was mentioning is Will will want you to have like some of these systems in get right so like once you start using get-offs in a Kubernetes native environment you start seeing the advantages that it gives you right and then you start looking at some of the more traditional infrastructure some of the more You know the way I hate using the word traditional because it's not like it's not like it's not valid Right, but it's like stuff we need to do those things are sitting around. Yeah, they're keeping all of our humanity Yeah, exactly. Right. Yeah, exactly. We had keeping like the lights on right like yes But like you kind of look at some of these other other You know infrastructure, you know traditional kind of workflows and like like Just trying trying to model that after you know what you're doing it here and get ups changes What you remember to document? Yes changes. Yes, how you communicate about a system? Maybe now you want to start changing the way your inputs work, too Yes, exactly. And and the how how you interface with the system, right like so like get ups in and there goes I Yeah, no, hold on. I think you might have to yeah, there we go. Okay. Cool. Awesome. Thank you. No credentials Yeah, no credit for the immediate you know, you're now certified Yeah, it's just trying to take, you know some of these the interface of working with your system now is kind of You know in get right we do it doesn't have to be in get because of the principles But you know, let's let's keep this a little simpler Yeah interface with your system with get even just making a manual change and maybe reflecting that you change something Yeah, exactly like who made that change is like well, we can do some of that stuff even now as you know They mentioned infrastructure as code were kind of differentiating between the two so you can kind of start doing some of those things as Well because now you're gonna be like, how can we can't get ups that like let's let's try to move some of that stuff over It reminds me of the crawl walk run model that was so often popularized when talking about changing our Organizations way of working in DevOps, right? So some of these lessons from the past decade. It seems like they're still haunting us Tech debt, right? Yeah. Cool. Well, that's some really interesting notes about Working with sticky systems and and also pain between teams in a real organization Pinky you said something in your intro that actually really caught my attention Which is that when you were in this high compliance high regulation environment, you know auto insurance very exciting stuff, right people Auto and home, right? And we still got branching out over here and we're gonna cut it there before it becomes a legal issue The you mentioned that in this in this desire to go to get ups that Your your platform team was using Kubernetes But they were using it in a very imperative way in a different way Yeah, and you had some convincing to do. Yes. Yeah, lots of it was that um, I'm sure that was easy No, yeah, it was so easy. No Lots of meetings where me and my teammate were basically joking about just eating popcorn and just watching the drama Because they're you know, so we had like the compliance people saying like hey Our our Kubernetes platform was not locked down. Everyone had access to admin Which is crazy because like for a highly regularly like we were constantly audited But the platform had not been audited yet, and it was going to be audited and it was going to fail So that was like the conversations we had were like, hey This is like more urgent than y'all realize because I know it's fun and games that like everybody has you know Admin access and nobody wants to lose it, but at the same time we have to lock this down and have like meet our requirements So there was a lot of conversations between like the audit like the the compliance team and the like Kubernetes platform team and we were the delivery engineering team and so it was it was a lot of Getting them on board. We had to do a lot of proof of concepts actually showing them like how it would look using multi-tenancy And this is what it would look in code and then we showed them it's like not that bad You know what I mean like we had sandbox environments that we were like this is what it would look like This is what the onboarding process would look like. It's not as painful as you think you made proof of concepts Yeah, you showed what the world could be exactly. Yeah, we were like this. It's not as scary as it sounds But that technical implementation The reason that there's convincing around it is it has social consequences, right? Yeah, like now we're talking about changing people's access Exactly the way that people describe their changes to the system completely Yeah, and that would change their jobs too because now they're gonna have to they have to own it Exactly, they're gonna have to answer to those people that are like, hey, where'd my access go? And yeah, it's what now they have to learn how to use what it's kind of you you touch a finer interject real quick it's kind of when I talk to people about like get ops and You know start talking about like what's happening that kind of like the first thing you reminded me like the first thing They say is like what do you mean? It's automatically applied to production and I always say like get ops kind of reveals some of the Pitfalls that you may have that you may not know right in in your environment. So that's that's a that's a great story That is a good point. Actually, yeah roadblocks that are actually Causing friction to shipping that then that back pressure encourages people to bundle changes into riskier blocks of code, right or changes to the system Right and so it's like if it's slow on the path to production You know things are people are going to bundle more and more and your changes themselves will become more risky Right and and that that's an anti pattern We we know from the past decade of research like we can point to data that says oh You should watch out for that. That's dangerous. So so It sounds like if if somebody's not doing continuous delivery and there's reasons for that that get ops sort of Reveals that is that what you're getting? Yeah. Yeah, it kind of just reveals that it reveals some of those things that Like you said and I think when what you said about like bundling changes That's and then you know that big change goes through all at once because as you said like it's slow You know if you if you know you're gonna make a change it's gonna take forever You're gonna bundle those changes versus, you know incrementally, you know sending those changes From the past like continuous delivery. It solves it solves that symptom Yes. Yeah, and you're saying get ops is continuous delivery or It's part it's part of continuous delivery is built into get ops. Yes. All right. You here to hear folks cool, well Friends I'm I'm delighted that we've talked so much about the lessons of the past and how they inform our new systems our new community but Get ops is kind of this this funky marketing term that we got going on I've got to ask the group Why do we need a new word for this if we've been doing it for a decade, right? Ask Alexis? No, I'm just kidding. What is the need to define get ops? So so actually you you might have a better answer, but but Alexis the CEO of we works actually coined that Sorry, I'm not talking to the mic this CEO of we works actually coined the term. I don't remember what year but 2017 Robert is saying So it was actually like so so flux was created because there was like an outage that happened at we've works And they realized like we don't want that to happen again Like it took so much time to start back up So they created this tool that would you know do that automatically and just like take the declarative nature And so then he coined this term get ops so The idea is that you know You're you're focusing on get as a single source of truth and that you're pulling versus pushing and there's so many benefits that come with this model because Like this is what I saw at State Farm even like the fact that if for some reason with their admin access Somebody goes in and deletes a deployment. It stands right back up. You know what I mean? Like there's there whatever you have in your code is what's actually realized and and that's so powerful in my opinion Yeah, I would agree the the entire idea of that the actual desired state of the system is stored in get I had discussions with people kind of don't understand that fully like what happens if someone changes Stuff, you know on the equipment is well It will be rolled back because the desired state is in get in our in this case and for me for me kind of So, you know, we have see I see the tools and we you know been using that for for this and I Like to call like I like to like not think of us to continues Delivery but it contains deployment because that is what it is when your software is deliverable. I don't want You know all these different angles of how to get it out. I wanted to just get out there And we can we can do advanced stuff like progressive delivery and have it be delivered to you know In certain number of users and all those kind of things we can do that But baseline is I just want when we have something new that's supposed to be in production I want it out there again I don't want that to be an entire pipeline to go through and that could possibly fail and then someone has a troubleshoot It's like it should just work What I which I think like the the the various get-offs tools kind of stalls in a very continuous deployment Yeah, I said delivery earlier, but yeah, I meant is even more accurate Yeah, because for you know, I people I said like I think actually Alex is Alexis also said that in the blog post that we Have been trying to do this for so many years now. We have the technology to make it so let's build tooling That's even closer to what the what the research says not that people aren't already doing it Not that existing tools can't do it already, right? But look this is an idea that we need to replicate Yeah, and that and we we've talked a lot about flux because flux was kind of first Right, but flux is definitely not the only thing around anymore, right? I mean, we've got an Argo t-shirt over here Right and and I I'm with the Carvel project. We've got a get-offs tool as well right now we've got this word and That's that same word with all of these principles can be said about all of those tool sets Right and and amongst a bunch of open interfaces a new programming model. So this is interesting, right? What's happening right Christian? Has it has it been for you to see get-offs growing and and maturing and becoming a thing with all of these New people joining the community. Yeah, and I think I think that what you mentioned is interesting because like we all kind of you know come from like different tool sets And I think it's really interesting that you know we the the Like the principles and the idea of get-offs drove the building of the tools and we all kind of ended up With the you know, you know, you know similar if not the same good kind of workflows Regardless of what tool sets what we're using Because you know, we kind of looked at like the principles We kind of looked at like the work The get-offs workflows and what it means to do get-offs and then build the tools to do that right bird And so I think It's it's very very interesting. I think I'm excited to see it evolve I've even had conversations with a few folks Around at least internally I read had folks reached out to me from the Ansible team saying hey, how can we be more get-offs like Right, so I've been kind of working with like other, you know other Other projects, you know asking me and like oh, how can we be more get-offs like what it's what you know What does actually mean can you explain the principles more in detail that sort of thing? And so I think seeing things evolve that way You know, I'm I'm excited to see how we're gonna get-offs, you know Stateful infrastructure, you know, I mean that that's I think that's like the only step highlight what what you mentioned there Folks red hat folks from the Ansible project asking you how to be more get-offs like yes, right Not that they don't know how DevOps and continuous delivery and continuous deployment work Not that they haven't been around since years before kubernetes became a thing But this word is actually important to configuration management as a whole Okay, great. Well, um We have hit a lot of points here, right where get-offs kind of came from where it wasn't our personal stories Pain and dealing with a social organization lessons learned from where we came from We though as a as a group of people who love collaborating together Recognize that all of the stories that happen from the practitioner community and all of the things that you all In the audience end up there virtually as well are fighting with are important And we want this panel to serve you well. We've got quite a few minutes still left for questions Who who wants a mic to to ask a question to our lovely panelists over here? I saw this hand. Yeah Hi, what's your name Sergio? Yeah, hello, I'm my name is Sergio and thank you everyone. It's very interesting panel So, um, my company we use in terraform for infrastructure platform as called sorry So terraform enterprise you said terraform for infrastructure as coach. Okay And the way we do it is we install kubernetes then deploy argo cd main components like um, serve manager external diner which you would expect and everything else is managed by argo cd now I often struggle with with um Knowing what's github's material what's not as in Should I install this application from argo cd or should install this application as part of my core infrastructure from a different? Installation method like terraform. So I wonder what the panel thinks what's github's material and what's not github's material Yeah, yeah, dude. Well or did because I know because you've done a lot of with with terraform Uh, so it It is kind of a hard question That's because traditionally it would be like you're setting up your infrastructure with infrastructure as code and then you would get and start to get all process, right? Um, which is perfectly fine You know that that works. There's nothing wrong with that if you're doing that and that makes sense to you and your team perfect, uh But having terraform uh in git and then running pipelines to Do your terraform stuff that doesn't that's not necessarily github's, you know Um, you can set it up in a way so that it works like that But in itself is not github's what what you can do If you want to go on the journey of like starting to do, uh actual You know, that's a trademark that how to do actual github's with terraform Yeah, uh, which I go to youtube and search for that and you find a talk from me That's called exactly that And us as well. Um, you you could use some sort of controller to kind of like getting it into that That terraform space and there's the tf controller from we works that works with flux There there's Yeah I'm gonna And there there's other operators and tools that you can also implement the problem Or not the problem with one of the things with the tf controller is it seems it's from we works It actually works with flux. So it's using some of the flux components So it doesn't have to reinvent the wheel What you could do is use this project called flamingo, which is, uh, you know, basically you you have argo running on top of the The tooling from flux Which that means you could extend it. So you could have your argo, uh, do Basically use the tf controller Otherwise, you're gonna have to find some tool that does, you know, kind of bind it all together So, um, I I have experienced tried some of them Usually it just ends up being a Something that runs a hashi corp also has one called terraform operator It works best with terraform cloud. So if you if not do using that as a tool you You know, you would be best served finding something that fits for you Uh, also, uh outside of the kubernetes world, um, there is a bot that you can install Called atlantis, uh, that will run against your source code. Yes, uh, when you make pull requests to your terraform And that can help you get to more of an active reconciliation workflow that hits some of these points if Say you're missing tooling that uh, if you're terraform kind of And you can like even with like terraform cloud you can get it to some sort of state where it's super easy to use And it's very much, you know, actually kind of get us but Uh, yeah So that's what I was doing. I was using terraform enterprise. Um, there's talk way like way back went out there If you look me up, there's a talk like way old talk about using terraform enterprise Um for for like trying to like mock this Yeah Terraform and github thank you sergio for the question. Uh, uh, we had a the mic over here for your next question Okay, it's working. I was just changing so i'm i want to give a little bit of a return of experience because we started yak infrastructure is code and github's back in my company In 2020 with our cloud transformation And we arrive at a point in which I personally have an issue with github's and I wanted to have a little bit of your opinion Which I find that it doesn't really work well today at scale We have a pletorial repository. We are also a partner of redact We have more than 300 clusters open shift clusters out there How do you manage such high volume of repositories? Which they have all dependencies on each other Yes Argo cd could be the answer when you deploy things in kubernetes completely agree with you, but when you have to Interact with resources that are completely outside of a kubernetes system and you need to manage poor request across tens of repository to make sure that your application is fully running And one pr builds fails because terraform. So sorry, but it's not bullet proof It fails. How do you reconcile with all of that today? We are at situation in which Some of us at our companies they are not really buying the github's drugs to be frank with you And I wanted to hear a little bit of your opinion on that Yeah, so um the Your question is mainly around so that I would really be asking a lot about like, you know using If you're using monorepo versus using poly repo versus using um, yeah, so versus using the you know, a lot of repos Um together right like collecting them as as as a Yeah, yeah, so there's a separation of responsibilities of like for example The a big one would be separating like the infrastructure Get ops right versus like the application specific get ops and you may have many application specific repos Having there and there are you know, you say your issues like they're all interrelated right so like one one can fail and the others You know, the it'll essentially affect everything right like you have one Not reconciling for whatever reason which then affects the other ones because they're dependent on yeah, so the um a lot of that is I think comes down to process right Trying to you know coordinate Different pull requests with you know different people different organizations right that usually like you said at scale The issue becomes um communication between Um between teams and coordination between teams trying to Yeah, yeah, no, I can hear you. Yeah, no, and I'll I'll repeat what you said Oh, there's scott. Hey scott. Yeah, I think you want to give the mike. Yeah, let me run the mike to scott over here scott rigby The main tanner open get ops Hey everybody, uh, I just wanted to reply to your comment. Um Because it is an open forum, right? We'll see. Okay, cool. Um, and I know it's like ending so people have to leave and Glad that you were able to make it. Um, but you know that I'm wondering There seems to be a jump in a way to say our company's not Not uh, not buying into get ops, but then you go on to say well, you know, the problem is really our applications are spread across like Infrastructure hundreds of repos or whatever and how do we how do we manage all that and my question to you is How would you manage that whether or not you were using get ups? Yeah, I uh, I do think yeah, thank you. Thank you for the uh detail and passionate question and um Yeah, just to be respectful of every one's time. Let's uh The answer to this question is that um To christian's point to scott's point, uh, the complexity you're running into Is not a problem with get ups tooling Uh, it is a process and complexity problem that is uh older than configuration management. Uh, and Uh, the same sorts of pains and joys and victories that are waiting for your teams Uh, have been well studied and uh that there's there's practitioners from the from the last decade But that you're feeling those things with new tools and that's that's awesome. Um, Yeah So we let's do just do you have a quick question? I can try I can try to make it short. Uh, So, uh, this is a little bit related although not as long. I can promise you that Um, the thing is at our company we have a situation where we have our components That is actually a stateful components that we need to Uh beef up its thousands of them And we need to turn down on a daily basis So it's going it's very much more dynamic in the nature So we are doing get ups in all our what what do you say kind of more static Situations that we've got that makes a whole lot of sense But when we have a very very dynamic situation that we partially do Uh, we find it difficult to have get ups actually to help us out on that one Because it takes too much time just to to save all the files and all the changes to files So what we're doing instead and you might prove me wrong because we're doing something wrong in this sense Uh, is that we in depth instead we do it directly with some coded access into get ups Sorry to uh to kubernetes in order to start up these individual Stateful sets with individual configurations So that's what we do. It's more like a comment, but I yeah, that's about it Oh great Well, you know, it's one of those things where uh, if you can keep it stateless It's always have no very much easier Yeah, so like like using external sources for those kind of things Would be like my suggestion in in most cases, but I kind of see the point and it's one of those things where Like I think all of us probably could help you look into it Based on what you're having to deal with Yeah, and also I think at some of these conversations and I don't know and you know We can do a hallway thing after you see this explain a little bit more, but I've always Get ups has made a lot of at least people at least us internally at red hat by the way we dog food So we we we run into the same problems, but with folks that I that I talked to It always ends up being that a lot of people see ci and cd as one thing Whereas get ups they all of a sudden become decoupled you have a A a a Asynchronous task and now all of a sudden the synchronous task that you're trying to coordinate with the with the asynchronous task and then some of those things I I always end up saying try to do Some of the things that you're describing maybe not all as possible in ci Right try to do some of that before it comes You know to that very last piece that little that get ups piece And and it's only like the growing pains of like trying to decouple ci and cd because for so long We we just kind of glom them together as like one process. So that's um, you know Again, like there's a lot of people having that same pain because you know like, you know We internally have that same pain trying to decouple some of these The asynchronous task and synchronous tasks. So yeah in the end of the day Stateful problem solving is a reality of what we're all trying to accomplish when in these bigger teams on these bigger systems And you'll never get away from that part, right? But in in where get ops can help us We can find the declarations we can Let the the computers these very complex computers, whether it's kubernetes or your cloud systems Follow a little bit more of a desired state So it makes it a bit easier to work with our team and focus on the the harder imperative problems So and that that gets as well to our comment or a more complex question earlier, which was thank you for that But um, yeah friends. We're a little bit over time. Thanks so much for staying with us It's a christian philip robert pinkie and lee and um, thank you for for your thank you. Thank you everyone