 Good morning and welcome to the level-up hour where we contemplate containers considered Kubernetes Opine on OpenShift and in general help you to understand a little bit more about this world of containers Kubernetes and OpenShift, especially if you are somebody coming from that Red Hat Enterprise Linux background as a systems administrator engineer and so We've got a great show today in which we're going to be talking about GitOps and developing your CI-CV pipeline and some tooling that are available to help that work a little better So I'm joined today by my colleague Scott McBrien otherwise known as Stabby and We have our guest today Vincent de Maester and Mayor Friedman who are going to share some of their knowledge about what all of this is about When you hear these these letters CI-CD. I just want to remind everybody please do like subscribe and share it at everyone let people know that we're on air and Let's jump right into this So we're talking today a little bit about about GitOps and CID CI-CD pipelines and so You know like a lot of acronyms these get bandied about and everybody assumes that everybody knows what they are but Then so maybe why don't you share with us? What a what a CI what is CI-CD? So I think the CI part is Kind of close like everybody understand the CI part. I think for everybody it means continuous integration and It means all up even continuous integration can mean different things for different people But usually it means I built on each commits or each PR or each whatever but on each change that me and my team do I run some set of tests and Sometimes I test a set of validation like I know the interest anything So the CI part I think is kind of simple to gasp grasp grasp. Yeah, grasp is better The CD one is like really really more difficult because people will put different meanings behind CD. It's gonna be continuous deployment. It's gonna be Continuous delivery and and if you dig a little bit more into those two and even more I I don't know the other meaning in my head, but it can depend different things that continuous deployment At least in my head means any time there's a change is deployed somewhere But continuous delivery doesn't necessarily mean this. I don't know if To this but yeah, it's the CD part is More influx and I think the CD foundation so the the foundation that kind of try to Work around those thing is is working on a paper around this like trying to define what? What is CD and and what behind this? well, so Let's let's dig into that a little bit and so let's first start with the CI P part because I think if you come from the Developer background Integration is something that you deal with all the time If you're somebody coming from more of an infrastructure background It might not be as as self-evident as you think and so may or maybe Before we dig into the complexities of the CD because it does potentially mean a couple of different things Explain to us our audience again thinking that about the folks were coming more from that infrastructure background What is it when you're what is it you're doing when you're doing integration? Forget it continuous forget making it continuous when you do an integration exercise. What is it that the developer is doing? so I'll try to to look at this from the infrastructure perspective and I have to start with the words continuous integration unfortunately because the whole essence of continuous integration is to supply confidence that Proposed changes will actually work as intended and and as Vincent said There are there are many perspectives to to look at CI and where it happens It can either happen on a pull request. It can happen after the merge so different schools tell different stories as to what CI is But essentially from from the infrastructure perspective I think the easiest thing will be to will be is to use an example Imagine you have a repo that controls your infrastructure, right as an infrastructure person I have multiple such repos and every time I want to make a change to any of my infrastructure I will submit a PR to that repo maybe get appeared to review and Essentially, I want to see that the outcome is what I expected to be even before I emerge and after I emerge is Even more important but harder to predict So the integration part come when when when different team members collaborate on the same code On the same source code to produce a single resort result, which is the artifact and that is also a And overloaded word, but I guess we can get up from here All right Well, so I think we we have a fairly clear picture there now on this integration piece which you know again I think is is very self-explanatory to to almost anybody who is working as a developer But the exercise might not always be quite as clear, but thank you for for that clear now Now let's get into the more complicated half of this this this little acronym. We're dealing with and that's CD Vincent you talked about how there might be a couple of different ways that D is interpreted It could be delivery or it could be deployment Maybe if you could give us a little bit of clarity on the difference between the two of those So continuous deployment for me is A way of doing continuous delivery first it's continuous delivery is I'm able To deploy continuously and by continuously means at given set of at a given period of time or at a given set of events but I want to be able to deploy continuously something and continuous deployment is just about the deployment like I do something I deploy The thing that I built I have confidence that it's it works because it was tested I can deploy it already directly for example in In production, that's usually what people think when they talk about continuous deployment itself When we talk about continuous delivery, it's It's encompass a broader scope. It can be The whole process that goes from source code commits to The actual deployment and in this process, there's there can be manual approval. There can be If the sets of events that's gonna I don't know for for example for container image that's gonna Run some security checks on an image and wait for it to get back etc. So The the continuous deployment is just a smaller view of what continuous delivery is I'm trying to think about the other thing Well Well while we contemplate that I'm actually gonna Ask Scott so so Scott coming from an infrastructure background Tell me a little bit about how how integration and you know delivery and deployment have looked to you because you know My recollection is you have you have worked in an environment that aspired to maybe some of these things But perhaps uh, you know, it's not always achieved them. Uh, and this isn't a former life, right? Yeah, and I think that even today we we meet with a lot of people that are Still very monolithic in their co-design and deployment A lot of the production operations folks like they they're used to shifting a large version forward all at once And with that comes a fair amount of risk, which is why I think it's very challenging to do those migrations and upgrades and deployments And so this is a a change in thinking and it says, hey, let's take a lot smaller bites of that elephant Um so that we can have a lot less risk at each incremental change So that overall we're getting to a better outcome With our software product and so, you know by breaking up the Changes to be much smaller and more consistent We're minimizing the amount of risk to our infrastructure for each individual change rather than being like, okay We're we're running this version on this operating system. Cool Let's completely change everything and redeploy it Right, right. Well, so, you know, there has been a big move in the industry towards more of a microservices approach as opposed to a monolithic approach for applications But it doesn't make much sense if you're going to have a microservices approach to deploy it like it was a month, right? and so CICD in my mind works very well with this larger picture of really trying to break things up Reduce that risk associated with updates and also to be able to push those updates without having every single piece Considered reflected and so on and it has a lot to do with the app design but at the end of the day the Your strategy for testing integration and pushing that thing out has to be consistent with having that actual design, right? all right, so that's that's that world of cicd and so We've talked a little bit about why will people implement cicd and And I guess I would ask again Vincent and mayor Is is there something that we've missed here that if if we're thinking about, you know, an it organization That might not be doing this yet. What is it that they're going to be think? What are going to be the big drivers for them that are going to push them to Pursuing a more cicd approach to deploying applications I think More said confidence. I think this is the main really main benefit because The more you build the more you try things the more you test your Changes the more you will be confident that it's going to work when you actually deploy it. So I would say the main benefit is going to be Confidence, so that's kind of confidence driven development and I would also Assume that the more people will drive will go towards cicd. The more they will have the wheel to automate things And simplify their day-to-day work um, and thus I think that would be kind of a side effects benefit, uh, which would be Because we testing because we do things more often we see the pain point we automate the pain points and and and we Kind of become more efficient day after day I think it will also reduce a lot of the day-to-day effort allowing People at the at the various companies to focus on what matters and what and the things that are more interesting I mean, I can do the same thing over and over again. Eventually I'll get tired. I want something to Something someone or doesn't matter something to do it for me Um, and and and automation comes comes for the rescue at that point So it allows people to focus on what's important and I think more importantly it allows Everyone to find issues Uh, more quickly Right, uh, if you if you, uh, only promote once in, uh Once in a long period of time you promote a lot of changes Be whatever promote means in in this context Um, you're also potentially Promoting a lot of bugs or maybe just a bug that was introduced in the very first commit of this batch of changes It would be very hard to to refactor back whereas where you deploy Single changes and you deploy everything you've discovered the bugs immediately if not even before you deployed anywhere Depending on what kind of continuous you do Right. Well, and you know what you were just saying suggests to me and this, you know I think has understood to be an underlying truth too is that there has to be a bit of a culture change within an organization once you start Going down this path is that it's not enough to simply say that you have erected the infrastructure and the You know the pipelines that everything flows through there has to be a different way of thinking And acting and working together That compliments that because you do have to change some things because of the very facts You know things you were just talking about and so um That kind of really moves me to another question, which is You know in your experience What are some of the recommended practices that people should be thinking about? When they are moving to this more CICD approach and they are trying to build out a CICD approach so I think One of the things that are very important um Happens during incidents when suddenly something happens in the in your production environment and you need to You need to find out what's going on immediately. So one of the more important things are traceability We need the CICD pipeline to be able to implement traceability in such a way that no matter where in the Either the supply chain or in the environments you look you can see exactly where everything came from What pipeline deployed those changes? Uh, where was that pipeline created? What triggered the pipeline? What are the the code changes that were introduced or what is the process that caused the current environment to look like it does? So traceability is super important I also I also think that to a certain degree what's important in these processes and something that's Important to design up front is reproducibility. We want the pipelines to generate the same results Every time they run regardless of when they when they run And and with that also comes Also come thoughts about When am I going to do each action in my pipeline and am I okay for choosing to do several actions more than once? one example would be If you're deploying on to open shift and you use container images And you deploy something promote something to production. It's important that the promotion notion Stays with you as in I am promoting a certain version of my Of my code base of my of my binaries artifacts Deployment manifests whatever it is. I'm promoting something that was already built and tested in stage So there are I've seen multiple examples of pipelines that build the container image More than once for example once for the deployment in state one for production So I try to avoid these things already in the design phases of the pipeline and and and and practically one of the important things to To consider is how the pipelines Match the current working practices of the organization One example will be the branching strategy the CICD pipelines need to match the branching strategy For them to actually Deliver what they need to deliver which is a seamless experience for a developer From merging their code to that code being in production. So it has to match the current working processes that are already established And and obviously more Um technical things like the structure of the different repositories and whether there are differences And what will they be and how do you how do you generalize everything into a single concept And then implement the differences Well, you know, uh, first of all, I love the idea of a promotion notion I'm gonna use that from now on. Um, so we you really covered a lot of interesting ground there in terms of How you have to think through the specifics of an organization and how they work is that there There isn't a one-size-fits-all necessarily that you just drop this infrastructure in and it's going to uh It's going to work for everybody And so, you know, one of the things that caught my ear in particular was talking about your branching strategy And how that has to actually be Congruent with the way you set up your pipelines. And so, uh, vincent, maybe uh, could you talk a little bit about the relationship of Of sort of branching strategy and the pitfalls that might arise If you don't have a good, um If you don't have a good fit between that approach to branching Versus the way you set up your pipelines I I mean, I'm not sure exactly what but more was referring to for the branching strategy Okay, well Tell me what some other pitfalls and then I'll put him back on the I'll put put put him on the So, yeah I think the for me the I'm very biased on these topics sometimes um For me one one very important pitfalls that happen a lot is time yourself whatever runs into your safety to a specific Or safety orchestra like I don't know Jenkins or our Travis or whatever and and this Used to be something you didn't really have the choice In the past But now with with the right of container images, this is right if you design your Steps and your pipeline correctly you each and every actual unit of work Doesn't even need to know that it's part of the pipeline You just need to have a set of inputs a set of outputs And and not and then you can use this Uh elsewhere I don't today you you're using tecton for your ccd tomorrow is going to be something else If you design it well You just kind of have to move to a new language of description But the actual core value of what your ccd does is there and usually you see this is one of the pitfalls You you tend to tie yourself a lot to whatever the tool you're using And then when this was not maintained anymore or when it's like I know a person performance Uh, Pete or anything then you're you're kind of stuck to reinvent a lot of the wheels um when you switch and and And now that we have container images, uh, and that's where I'm very great because I used to work for docker before etc so I'm like Container image for a long time and and I'm I'm yeah Yeah, and yeah, it's it's okay. Yeah people can work for docker and then read it Of course So on that note Um mayor, I do want to return back to that question on the on the branching again I want to sort of explore some of the pitfalls that were That an organization might bump into when they are doing cicd You call that out and I can kind of imagine that but Give us a little bit more color around what happens if for example your branching strategy Is not consistent between your pipeline and and and how you actually do branching in your code base So I think that uh, vincent actually answered that question. Maybe without even understanding it Uh, the one the pitfall that he mentioned was not tying your cicd system into the Orchestrator or build tool like Jenkins or uh, whatever else you're using now We've all done it in the past. We only you only learn what not to do by actually doing it in the past I even have blog posts saying you should tie it to the build system. It's amazing But I think the only thing that's common underneath is is really git the source code itself, which is usually git Let's stick to the git example And when you stick to git to either tag your artifacts or define what you want to deploy and promote and when You're actually loosening yourself Decoupling yourself from the build system by using the git commits. So for example A relation between a cicd Between cicd pipelines and a branching strategy, maybe Whenever someone merges something to the development branch, right? I assume some an organization is working with git flow Whenever someone merges to the develop branch, we want an image built and deployed to the dev environment And then you have the main branch and whenever someone merges or something gets merged into the main branch We want the image the artifacts built again And then we want to deploy to stage So that is already a connection to a possible very minimal merging strategy with a develop branch and a main branch And and that may also connect to the previous points about continuous deployment or delivery Which is indeed the difference in the process and if there is a button involved there in the middle or not Or at least that's what the legend says And and when you want to this so everything from the main branch At least in our branching strategy goes directly to the stage environment after it's been tested in a dev environment prior And when we want to promote something to production We again go to emerging to a branching strategy, which is don't do anything Just choose a commit shot that worked in stage and promote that So we're very github scene that way and we are decoupled from the build system Which is the pitfall and we also did the sort of Connection between a git branch and an environment So you get a direct link and you know if something is wrong in production Look at what commit shot is broken something broken in stage. Look at master at the main branch and so on All right. Yeah, so scott um anything you want to add on any of this, uh, you know because uh, I think This is a natural transition for us to maybe start talking about About what are some of these tools that people use because uh, because I think there are things that We offer and that can help with this but before I turn to that Is there anything from your experience or from what you hear from customers? That seem to be some of the pitfalls that they're bumping into I was just really excited. I mentioned this in chat to hear mayor talk about promoting Um artifacts throughout the pipeline instead of rebuilding each stage because you know, there's been more than once where uh, maybe even at the crimson haberdashery where, uh Production systems in no way match death or stage. You don't find out until you actually deploy it and then it's like, oh no So yeah, that was excellent All right. Well, so Let me come back to this question then, you know, uh on the tooling and and you know, Vincent you raised some interesting questions in points about sort of Where the build system fits in but let's look at this from a broader perspective in terms of what is the tooling That people are using To be able to build effective cicd pipelines. What is it that that we're recommending and thinking that Are tools that can lead to success in a variety of organizations? So I think the the the tool that everybody knows Is and and where everybody has a kind of a Love-hate relationship with is Jenkins. Uh, it has been around for I don't know probably 20 years or so Um, and this is like this shows this is up until even now There's new alternative and we are developing your alternative, but Up until now, I think this is still the main The main tool that everybody knows it comes with a lot of pitfalls because it It started 20 years ago. So Containers, it doesn't know cube. It doesn't know there's a lot of things it doesn't know and It has been architecture with this plugin mechanism with this which is Super nice super awesome, but when you have 1 billion plugins and How you manage updates it's like it it comes with A lot of problems and on the open shift side We do We did ship with a Jenkins instance In 3 we still do in in opportunity for But those are kind of really hard to maintain and administer mainly because depending on depending on What's your use of your cluster and what teams are there? You might need or once a different Jenkins in sleeper each team because each team has different needs each team might Want to build and need this plugin the other one the other plugin The different version xx also it becomes like a huge hard thing to maintain and This is where new alternatives come from And that that are real like that are cloud native and and one is Again, which is the one I work on Oh, what a coincidence So so Scott let me just let me let me divert a bit from that So certainly Jenkins it's been around for a long time And and I think Vincent makes a great point that that it precedes in many ways Some things that are really sort of central to the world of CICD at this point, but you know, you're you get feedback from our Rail customer base or is there any other tooling that's in this picture that they are maybe Trying to cobble together What they want to do and and and that they're maybe struggling with I mean, I think that Jenkins is kind of the Widely used tool for a lot of stuff, but because it's used for a lot of things as Vincent pointed out it it gets really gnarly Because to do everything well you can do or to do everything you can do no single thing really well um And there's a been a bunch of other kind of SaaS offerings that people can use and um But yeah, I think Jenkins is kind of the the beast everyone knows Right, right. Well, so that that brings us to what is open shift bringing to this to this endeavor, right? What is it that open shift does and allows? Um, you know some an organization to do When they have this objective of building CICD pipelines that you know Achieve those objectives we talked about earlier and that you know in some way can align to their particular approach to to building code and and you know to their application lifecycle, so mayor, um Tell me a little bit about You know what open shift brings to this picture that maybe has missing or that is especially effective right, so First of all, I'd like to point out that I Openly and publicly like Jenkins. I think it's a very good tool It does it does what it's supposed to do which is to run things. You just need to tell it how and telling it how is the difficult part because Managing Jenkins, whether it is the Jenkins instance itself, whether it is the underlying infrastructure for the nodes for the For the main for the main node master node Whether it is the jobs on top of that, which is probably the most difficult part because Even when you when you think of scale it gets even worse because then you end up with the Jenkins instance with 800 job definitions on it All composed differently. We use Jenkins job builder So it's easier for us to compose different jobs of different types and and to even allow our users to to to take advantage of that but eventually I think the the common Characteristic and and what we've been going for or what what vincent's team have been going for and and we received as customers is the concept of of building small small boxes and stacking them together in every way that we want so open shift pipelines they come They come in in a very It's a very generic tool Every pipeline that you can imagine eventually is a pipeline and a pipeline is divided into tasks And each task may have one step or more and that's and that's the the concept and when you are able to build Different pipelines, but have the steps reusable It brings a huge advantage and And it's not even something I mean with Jenkins specifically you would need to actually define each and every job and and Find your way to reuse components with yaml magic and things of that sort, which is never fun We ended up one time with a 5000 line file of Jenkins job builder definitions for For a single project for a single product. We are running and our team currently runs over 50 So imagine what would have happened if we kept going that route But I think what open shift brings to the table is is the elasticity That's the most important feature that it brings and that is the reason we migrated all our cd processes from Jenkins to Open shift pipelines because of the elasticity no matter how many we used to fight with Jenkins. We used to tell tenants not to promote to Frequently not really in those words, right? We told them. Yeah, never mind. We were encountering overload Constantly with too many deployments happening at the same time and with open shift It just doesn't happen because the cluster can be defined without the scaling and you deploy as many resources as you want Also need to clean up because these are custom resources and at cd at cd So cleanup is important But essentially you define things once you can reuse them from different places And you just build your pipelines any way you need and whenever someone wants to run them they just go into the namespace and Hit run because it's embedded in the in the console which Super nice So everything into a single place, which is also the place running your services That's double nice All right, what does sound like go ahead Yeah, I was gonna say I think just to add to that One of the goal of open shift pipeline, which is based on the upstream project data and was to Also kind of speak the same language or be more Close to what you use to deploy because In the end right at open shift pipeline or tecton is just an extension on top of q We are using crd's etc. So you're you're gonna be able to use Everything single tool that we that you use to manage your clusters, etc. You can use them also with Tecton and that's that's kind of was one of the the goal and yeah as mar said like one of the other goal was Composability and reusability be able to easily define Task with multiple steps that things that are reusable so that you define them once You can even share them somewhere and we do have community sharing place somewhere for command used task and and and from there you you just like get your pipeline and You orchestrate the unit task that you wanted I think you hit the goal on spot with Customers reusing the same systems. They already used to deploy their services. We did exactly the same thing our team Which is a customer of open shift pipelines We have this repo that we use to define and deploy all the services that we run Into different open shift clusters and and open shift pipelines just came naturally into that picture We reuse the same system of course and it was One of the one of the best migrations that we did from Jenkins to open shift within the same system It was spot on with the goal and it hit right there All right. Well, so it sounds like this is actually a path to being able to do things For lack of a better word much more easily You know with fewer pitfalls Less complexity and of course complexity brings in the possibility of things going wrong I'm particularly intrigued by the idea of being able to share out particular you know Ways of doing the you know sort of the composition and other sorts of things because that also Starts to build kind of a community of practice in which there is a way that you do things It's not that that's the only way But there is a common way that can be shared and and it becomes in a sense tribal knowledge, right? so Talking about the tribe a little bit Can we talk a little bit about the roles of the different people who are involved in building these pipelines? Okay, so first of all, let's talk about the ops role in In in in sort of building out CICD pipelines and who wants to take that one? There you go I am the No, see I was going to put scott on the That one, but you know what? No, I want to hear from you or All right, so so I think first of all the the definition of roles may be different in every organization And and eventually we need to to formulate the relationship between the different roles in an organization in order to decide who does what a sort of I'll use a word my team uses a lot a contract between each layer Of people supplying it So in our in our team, for example, we are responsible for The cd process as a whole Are the services that we run the development teams they come with their source code? And we have a sort of a contract that they put their deployables in their in their code repo And we are responsible for the pipelines themselves And and they are receiving that as a service So in this example the ops team is responsible for not only supplying the pipelines But also for building designing maintaining them Um while the developers in this equation are responsible In our case at least to define their pipelines in a declarative way We have this system. I started talking about I'll say its name up interface contract reconciled go look at our GitHub organization. It's awesome So so this system allows the themes that that we service to define in a declarative way What they want deployed where according to the branching strategies we mentioned earlier So they're going to say something like this is my repo and in this path you'll find my deployable I wanted uh, whenever there is a merge to main I want to deploy to stage and in this commit I want to deploy to production And that will be their role behind the scenes automatically Our systems or be it people depending on the level of automation each organization is that will provision these pipelines And and based on the same declarative manifest will also be Will also be automatically triggering pipelines based on different conditions For example, if the main branch is pointed at stage environment and a commit changes behind the main branch We'd want it redeployed. So the ops team in this case will also be responsible for making sure that Artifacts are being delivered into environments according to the contract once again So the collaboration in this is essentially through a contract We decide upfront who does but who's responsible for what and who stores artifacts and Based on that we define everything else I like the term contract because a contract implies a negotiation right and and there there has to be some degree of negotiation Between the outside no negotiation No, no negotiation the contract is the well There's no negotiation after the contract but there has to be some degree of agreement about what that contract looks like, right? um, so Vincent give me a little bit from the developer perspective on this and and how How the developer role participates in this and thinks about what the cicd pipeline looks like and how it works Well, uh, yeah, I as as Moritz said like In in depending on the the roles are different, but I would think like for me the The role of developer here are to tailor like if we talk about in specific technical pipelines like real machine pipelines in the notion of tasks I think developers role is to tailor the tasks that they might use the one that are important for them like the I'm building the the code testing the code etc to their needs with of course discussion negotiation contract with the operational role, but like it it becomes your pipeline becomes a set of an element that are either dev responsibility or op-response responsibility the pipeline itself might be on the ops responsibility, but you usually you have Like you will have your set of tasks to maintain because this is kind of closer to what you're doing um and yeah, but yeah, I mean it really depends on and and and it's also like We're talking about for example, open shift pipeline specifically, but you can have higher level constructs, which would abstract this contract and and and I know there's like work towards those kind of higher level construct upstream for example where we're trying to think about more like the dev wants to provide this in this set of tasks that are used for them to build like a go build and a go test task and and on the opposite side, we want to be able to easily say Ingest inject Inject these tasks into the pipeline and the pipeline will also build an image from this will also inject some other Steps that are tasks that are coming from someone else the operation for example to secure our scan the image so Yeah, I think I'm answering I mean over yeah Well, I know I followed along and and I think this is what what One of the things that comes across here is that it has to be a very collaborative process in which it's it's not something that can simply be Installed and now you have this thing that does the thing. It's there's a long long series of Again negotiations. I think discussion about well, okay So here is how we do things now and the thing about it is is that how we do things now Is not always the thing that you want to continue I mean sometimes you are undertaking this for the very reason that you do not want to do things the way you do them Now you want to move to a new and better way But at the same time it's not like you want to give up the way you do everything And so that's from a developer's perspective or from the ops perspective It's let's take the things that work for us, but then maybe move in some new directions with cicd But also anticipate how do we really actually push our apps out as opposed to what we imagine You know, we're not, you know, we're an organization that might not imagine itself as being an amazon You know, we're a netflix or whomever that you know is pushing out 10 000 code changes a day Uh Yeah, that implies a somewhat different cicd pipeline It implies a different relationship between the developers and the ops and and many other things right So let me ask um You know, if we just get a little more specific here You know, what are the administrative tasks? You know, what is what is day two? For setting up your your open shift pipelines, you know, what does that look like for an organization typically? Understanding that it's going to vary from from work to work My work? Yeah, well, sometimes I just want people to jump on in, you know, Scott you feel free to feel free to clarify My my open shift knowledge is known to be vast and deep in all things All right, so I'll try to just to tell a bit from our experience what what we had to do um So I think some of the day two operations as always Day one is usually to get started and and get things working and day two is usually the improve part or iterate part The day one operations on on an open shift, for example, they want provision a cluster. They do upgrade the cluster So I think the day two operations will be partially to to modify and improve the pipelines as as new needs arise and As bugs are discovered and there are bugs everywhere including in your pipelines and in ours and we haven't found all of them and we never will But the day two operations are more in the in the direction of of maintaining and I think for maintaining and keeping day two operations Even even different ones to keep them sane and to keep them sustainable So it will be very important to focus on the day one operations how you set up those pipelines How you really collaborate between teams just by storing the right resources in the right repository By reusing actual resources between different namespaces according to your deployment systems and and how you manage your your code Manifests our code as well. So how you deploy the same manifest into different namespaces if you're servicing More than a single team. You'd probably want to to do some compartmenting Do some compartmenting. Yeah, and and create separate namespaces for pipelines So you'd need to be able to keep maintaining those pipelines and you'll need to be able to scale everyone needs to think of scaling in in any sort of Magnitude, so how are you going to on board new teams into your process that you've created in in your in your day one operations? How are you going to handle an issue with your pipelines? by by By making a single change and having it applied to all your different environments and to all the different pipelines that are reusing this Small package So I think a lot of the day two operations actually depend on how you plan to deploy your resources And that's again, that's a that's a perspective of how you deploy your services So open chest pipelines really just doesn't bring anything new into this concept You've every organization was already thinking of these things. How do I? Deploy and how do I fix my code faster? and and those pipelines they are just code and And I think this is one of the most important things. Yeah All right. Well, yeah, so, you know, it's good Ordinarily we hear a lot of talk about day two and so on but I think the Maybe the moral of the story that I'm getting from you today is that you need to get your day one right? Because that's where you actually set up the the things that you're hoping will Will provide all these benefits down the road As far as you can predict right because it's easy to say let's plan and let's plan thoroughly and let's not be surprised But honestly what you don't know what you don't know and when you find out That it's different from what you thought you'd know It's it's already too late and I think that in that essence It's important to try to plan as much as you can but also Be aware of the fact that there may be changes along the way and iterations are good We always say that iterations are good Don't deliver everything at once deliver in pieces that already return some of your investment and Right Well, you know, so that's that's the thing that was kind of striking me as a bit of a paradox there is that on the one hand What we are trying to do with CICD and so much of the, you know, how things are being approached in this space now Is really around trying to be very agile But at the same time if you're trying to get all of this right at the same time that's starting to not sound so agile And so that in my mind Raises raises this paradoxical question is How do you evolve your CICD pipelines over time because you want to make those good decisions up front Maybe you don't make every decision at the outset But you're trying to set things up in a way that works organizationally works for the kinds of applications you have and so on But inevitably you are going to have to evolve these. What does that look like? I think Mark said something very interesting on on the on the fact that How when you have multiple teams or even multiple users on your on your cluster that are doing Uh Pipelines and Sharing things how you keep them up to date with whatever change you Provide to the pipeline or to the tasks themselves And and this is where I think when you serve this is kind of for me the day to work Of day one would be I discovered the thing Oh, this is nice. This is using cube semantics, etc I can create something and it's going to be available. So but then the data will be How I Keep this available for everybody. Where is my source of truth? And and from there what what should I use to update my my pipeline? It's it's kind of easy there It's like how do I build containers by building containers in containers? This is a bit similar like you can definitely have pipelines that manage Actually the delivery of your pipelines in your clusters and And you have the full done This is kind of what we're trying to do for your platform where we build Tecton pipeline with With tecton pipeline, but we're doing the same downstream building open chip pipeline with open chip pipeline and and trying to maintain in our top footing Cluster our own all of our Pipeline tasks reusable tasks, right? They're all defined somewhere in the source of truth, which is one of the huge reports here and we do have pipeline itself updating itself validating itself and keeping Any sort of usage of this in the cluster, but this is kind of there's there's going to be As much approaches as there is teams almost like i'm sure i'm sure what we do different from what our team does And it's going to be different from elsewhere. There's going to be emerging patterns there, but yeah Well, actually emerging patterns gets me into one of uh one of the sort of the last questions in my mind and that is is You know Where is the technology in support of cicd going? You know in in sort of that next step the next phase What are what are some of the things that you see Happening in this space and that might be might be changing. Let's say over the next year or so um well so I think every Everything is is happening on uh containers everything like every single cicd system right now As containers as its core or some sort of View of containers. The other action is like you can see them as containers because they are using the actual distribution format even though they're running vm behind the scene But so I think At least i'm going to speak as the leader of tecton is I wish Something like tecton would become kind of a standard of reusability and By standard I really mean the actual Smallest unit in tecton, which is the task Because I mean there's going to be Lots of different ways to do things but when you do want to do a go build it's going to always be the same So if you have the image And a set of parameters that you can give then your Your actual tasks never going to change If you are Right out if you're Amazon if you're a small company it's always going to be the same um So I would hope that some of This would kind of become a standard and then Any abstraction that builds on top would kind of speak the tecton language so that then you just you can switch from an absorption to another or at least you don't need to buy into some something specific to to Like if I don't know if github action and github builds I don't know the name and pipeline as code from open shift or something else is if they are all talking Tecton then it's kind of a win because you don't need to Only use github action for on github or github on github etc. You just need to you can choose whatever fits your team or what you think is the important part for your team and then you You rely on whatever is able to provide you the understanding like The same as what happens with container images with one layer on top on css tecton modules, okay um Yeah So uh mayor, uh, we're uh any last thoughts on sort of that question of where the future is Is headed Depends on who you ask if you ask the ops team. I'm asking you Yeah, yeah, of course, but this is a two perspective conversation. If you ask ops people it goes into being more uh More abstract Really and if you ask deaf people, it's going away. They don't want to be aware of of cICD They want things to work Like we don't want clusters. We want abstract concept and we want our service running It doesn't matter what's underneath. So I think I agree with Vincent. Everything is going towards abstractions And the less you know the the less you know And it still works and as long as everything works the less you know is better In being from an ops background. I also want things to just work All right, well, thank you gentlemen for uh for joining us today, uh, this has been incredibly Incredibly informative about the world of cICD those four letters get used so so so much and And yet I think that sometimes there's not a lot of clarity about what what's under the hood and what this really involves and entails Um, so anyway, thank you again It is that time for us to talk about sweet sweet internet points And again, here's where they we're at in the leaderboard nir and dev is narrowly ahead narrowly ahead and so we will uh check back in in You know, when we have our next show and we'll see how that looks but again, please Like please share Please tell friends about it when you join. Please ask us questions Let us know if there's anything that uh that you would like for us to cover in future sessions. So, uh again, thank you for attending Thank you