 Hello, my name is Chepank Witenski. I work for Accolade, which is an American company that is in the healthcare concierge space And I work for the office that that's based in Prague Today we are not going to talk to talk about what is continuous delivery and what is not continuous delivery I would rather want to share with you some of the experience that I went through while trying to Start with continuous delivery And hopefully you will be able to find out Something that will be useful for you in case that you're in in position that you won't either to start with continuous delivery or or make it Better at your company or you are just thinking about maybe that's a that that's a good way to To choose and to go Oops Yes, that's the other one. Sorry, there are two very similar ones Yeah, there we are Before I go to that To give you the context I would like to share what my journey looked like So maybe you will be able to somehow connect the dots With either your experience or why I have done some of the things some of the things The way that that I have done them. I'm in software development engineering for about a little bit longer than than 10 years I started as a as a tester and Throughout the years I ended up in in in doing a software design engineer for Tools that supports test automation and automation in general I Learned about continuous delivery I think roughly around five years ago and I have to say I immediately fell in love with that and For me that was like yes, this is the way how software should be done and no other way is right. So I'm a little bit Exaggerated in in in continuous delivery. I'm like an enthusiast you might say Though I'm not I'm not like continuous delivery consultant that would go from company to company and help them upstart it So it's really more like personal experience from what I was able to to do at the companies that I've worked that I worked with So it's gonna be very like I would say Personalized to to my experience Can I have your hands up who has ever seen this book? That's not that much. How many of you that that has seen it have actually read at least few chapters from that We're there anyone who read it cover to cover Like the whole book. Oh, I got one Cool just humble and David Farley. They have coined the the term continuous delivery Somewhere around 2010 and By the way You who who wrote the the full book? Do you think that that Jess and David they are something like gods of continuous delivery? No Do you think they are masterminds at least? Yeah, okay, cool But we're on the same page there. I think they were they are really great thinkers by being able to Basically take the concept of continuous integration and move it move it further and and evolve it as I said The book has been published in in 2010. I Have to say there is still a lot of things that are still valid today So the the basic principles they still stand even though to me personally continuous delivery changed a little bit in In things like we have now much better adoption of feature flagging so Previously to me that was it was more for the technological Organizations how to move faster nowadays it also encompasses the other side of the house that the business side of the house and gives them more power Well, it gives more power to their hands They are a little bit more self-served we can say So What I'm going to speak about When I was putting together the talk I was thinking how how to put those little shards that I got from like oh, this is how it works and Make it into something that that would make sense for you how to group those things together so I went into The internet and I searched like how transformations are done And I have found found out that when you're doing a digital transformation, which to me like adopting continuous delivery is nothing else than transformation You have three three Areas that you need to focus on a that's people process and Technology so I said well, maybe I can put those shards that I have and and put them into the same Into the same areas Together so it will make sense for you. So we will get through those three areas and then I'll open to questions But first a little disclaimer People are are very diverse and That makes continuous delivery or adoption of continuous delivery really hard because as everyone is different like Even even the teams are different and companies are different. It's When you're looking at at how to how to get continuous delivery into into a company or into your team, it's Not the same as if you were in the same situation in in different company And to me that that's also what really drives me with continuous delivery because it's so much fun trying to find out Which way would work in the different situation? So what I'm going to be talking about it's not like that you can Copy paste and apply to your team and it would work But you might find out some parallels or principles that you can adopt and And use for yourself Okay, let's get to the people section First lesson learned I Have I have Start up started with a continuous delivery with really small team. I was leading team of four We were doing some tools automation support and I was really lucky that I got product owner that was really into continuous delivery as well And he said like we are working on a new project. Let's start to make it properly Let's let's have continuous delivery in there. So with the team we started to look around What does it mean how we need to think about coding how we need to think about deployments in order to get into continuous delivery Then I moved to another job to my current one and I worked for very similar team basically also making tools and frameworks to support support automation and I have talked about continuous delivery since I was passionate about it I talked to our architects and they were like well, this sounds really great. Like maybe we can do it. Will you Lead it for us and I was like, yeah, that's awesome. I'm gonna be like the continuous delivery consultant. Great. Let's do it I have some experience well Was totally different then then with the small team. I have learned that By like hard some hard failures But the thing that I have learned is that you need to be patient With the previous team with the small team that I worked in the in the previous company It was kind of everybody was very similar minded. So everyone kind of said, yes, we want to do this Let's do this and every single person on the team started to think about This is how we are going to do it in Current position that's a little bit different because I'm part of a team that's working on something and In something that I will call like a spare time. I'm trying to help the company to to switch Switch over so it's not my like primary focus on switching accolades to continuous delivery It's really like I have my primary assignments and let's say one of them or like Site by sites to the primary assignments. I'm helping out with the continuous delivery. So What you really need is patience Because it's gonna take time the most Interesting point in continuous delivery. It's that the people need to start differently about how they think about the coding and You are going to Approach the teams especially if you're like external from the team And you're gonna approach them and say like hey, we should be doing continuous delivery. And this is what it means You're basically attacking their status quo You you're attacking how they are doing things until now and it's working for them perfectly But maybe it's not gonna be working for them in the For the speed that you want to achieve within two years So you need to start to preparing them to start thinking About what we are doing now is good, but we should get better and this takes time Some people are more eager into like yeah, we want to release more faster or we need to push more faster Some people are like we're okay with what we have at the moment. So That's That's I guess the biggest lesson learned at the people section for me is that you really need to be patient Because it really takes time Some other examples of what you will be at like how people needs to Switch the thinking is with continuous delivery It's big to me. It's something like test driven design on steroids For your pipelines to work you need to write your code, but you also at the very same time you need to write Some tests or you you well either automated or or prepare test cases that you will run as a part of the pipeline Later, but you need to think about the whole thing when you're designing Which is totally different to Somebody who's used to waterfall model and he said like okay This is like how I code it and then somebody else will do it That's not the case for continuous delivery. Usually it's the same thing that needs to do all that stuff So it's really similar to for me to a child transformation as well Like you need to turn steer the people to to start looking totally different way on things. They are doing the next biggest impact was Who's your customer So I went into into that like yes, I'm gonna do that for you I went to VP of engineering. They said that yes, we want to do it I was like, oh, yes, these are my customers. These are the ones that wants us to be delivering much much quicker and then the teams are going to be my customers in like Those will be the ones that I need To give advice and help them with what what they need to change The thing that I haven't realized is that there is the other side of the house There is the business side of the house Which might really not be happy with getting new code every single day Actually, the truth is they are really anxious about getting new code every single day Especially if they were used for like monthly releases. You're Disturbing their world which is tightly tied to Your company revenue and they won't allow you to do that so you need to start with your whoever your your Business or customers are but it's definitely not the technology organization To me the parallel will be like the technology organization is like the engine that that's producing something But the business side of the house is they are the ones that are driving the car so examples from from my Experience where like for example our business whenever they got the the new stuff New stuff delivered they had to train all the operators and those trainings took from day to one week in case that you want to make those Deployments those new features Getting new features to them faster. They were like, oh, but we need to train those people We can't let you let us Let you give us code more often than like once a week because Otherwise our operators would be just on the trainings and they wouldn't be Doing the operations as they should which is the thing that that drives the revenue and produces money for us so you need to look into what are the restrictions from business side and Look into what continuous delivery can give you an architect the final solution around that so you can use for example feature flagging to to Distinguish between deployments deployments and releases and Make sure that that you don't disturb your you don't disturb your Business day of life the other thing that I have discovered very soon is that even in the community Around the world There is this term continuous delivery. There is the term continuous integration and continuous deployment Some people think those terms are interchangeable and use them that way but that Really creates confusion between people because Continuous delivery isn't the same thing as continuous deployment and I use the terms as as Just humbly uses them So basically it means continuous delivery is something that Makes you as owner of the code sure that whenever you want to Deploy that to production this piece would work. That's what continue continuous delivery does for you same as continuous integration is like whenever you add a new piece of code it will work well with the rest of the code base and Continuous deployment is basically having everything automated So you just commit and it goes all the way into production and is available to your user So what you need to do Whenever you encounter that to two people or two groups are communicating regarding continuous delivery and they are interchanging those terms You should go there and fix fix the vocabulary So everybody understands what it exactly means to do continuous delivery example would be like I told our Technology group Let's go and do continuous delivery and they were like yeah, let's do it that that's great But then they were coming back with like continuous delivery is too hard We don't want to do that for us. It's just enough if we can deploy during daytime and They totally didn't get that continuous delivery is exactly about that about being sure that you don't disturb your your business And you can actually deploy during daytime the other thing that I have mentioned a little bit with the feature flagging is That is a difference between code deployment and release Usually from the old days whenever we said we are going to release it meant that You are going to have immediately the new features while with Continuous delivery or or adopting continuous delivery. You are going to change that a little bit. So You are going to Make make Deployments which doesn't necessarily change the behavior of your application Who is especially anxious about that is business because Whenever you released you change the behavior of the application previously. So how comes it's it's it's different now so you will have to Tell them like what's the differences in between those new in those new mechanisms that will make sure that Whenever you deploy new code Their day of life will remain the same and it's gonna be After some time when for example, they decide they want to switch the flag and change the behavior That it's gonna be different and to me it's like a building a highway bridge In Standard day, you don't even think about that like you're going on the highway and they want to build a new bridge They are just start to build a new bridge in there in piece by piece and you don't really care because you're still driving your way and and everything works for you and Once the bridge bridge is fully built then they just shift the traffic Then you just go over the new new bridge and like that this is completely normal for us No one thinks about well Can I really go on the old bridge before the new bridge is built if it's like being built in there It's not like that they would come and put the whole bridge like all advanced there, right? For business. This is like I don't get why for software software Design or software delivery. This is so so much different. It's basically very similar, right? We When we build the software in a way that that we can That we can say that The old bridge is still fully functional, but we're just adding something that you just don't see They are anxious about that for some reason But if it's like a civil engineering, they don't they are not that anxious So I guess we just need to somehow persuade them that it's it's the very same thing And they don't have to be anxious as long as you have the the right continuous delivery mechanisms in place another thing about About continuous delivery is that your teams are as I said They are going to be different and they are going to move in a different speed and that's perfectly fine Since those teams are diverse Some of them would be more eager to adopt continuous delivery and work much harder on it Some some the others would have maybe some critical features. They need to deliver Then they don't they wouldn't have time to transform just yet. That's perfectly fine It doesn't make much fuss around the technology But it it usually makes fuss around the high level management because they usually want to have everyone in the same in the same boxes Like everyone is doing the way the say the same way so Again, you will have to spend some time with the high level management to explain that that being different Throughout the teams isn't actually bad It's actually good because it's gonna work much better than if you would come and say now This is the way how you're doing it without understanding why we are doing the way Why we are doing the thing differently and Last but not the least you need the supporters Obviously when you want to start with like changing Your team or company you need to have a supporter that has enough power And that is looking at it the same way as you are that that he wants to get into into continuous delivery That happens to me so I was kind of lucky our VP of engineering. They said like yes Obviously we want to do that. Let's go and let's do it So I got the sub I got the supporters that got enough power in order to to move to steer the company's ship But more importantly you will need the other supporters Like this guy, which are actually not just like cheering for the continuous delivery But they are turning to the others and like cheering them up even more for you Those are usually Find within the teams and those will be your single point of contact for that specific teams and they will It has basically mutual benefit These are the ones that you will be Instructing or working with the most because they are going to amplify what what you're saying into their team and they are gonna follow up and On the other hand, they are going to be the ones that are going to tell you hey, this is not working for us So it's good to good to find them early One of the things that worked for us at accolade. We have something that we called That we call community of practice, which is basically gathering of people that are like thinking about doing some transformation or Want to share? Stuff around some specific technology So we have like like Java community of practice and we have CI CD community of practice and after a few of them After a few of those gatherings those people crystallized up those are the ones that are always appearing in that they are always asking questions Those are the ones that you need to target and pair up with in order to to be able to be successful Okay, let's get into the into the process part When teams that have never seen continuous delivery have never had the continuous delivery book are Going into the continuous delivery for them. It's basically like an unexplored land or unexplored Island so when I was thinking about like How can I help those people without actually having to be in there every single day? So they will be able to somehow Use their pace and get slowly towards the continuous delivery like how can I help them and One thing that that came into my mind and I was kind of lucky because at the very same time I I got a I got a mentor into a mentorship program and I got with my mentor Together and he said okay what you want to what you want to improve. I was like well those few things Let's say it's gonna be it's gonna be speaking at the conferences and he say he told me like okay But that's like I okay. I get it you want to improve that but how? So he introduced me into very simple framework Which was draw three columns in the column on the very left Put like what's your current state into the column on the very right? Put what is the state that you want to achieve and then you have this one column in the middle? That's empty originally and there you put all the steps that you will do in order to to get from the current state into the to be state and this Very simple framework actually really works very well because it's no longer I just want to improve in that or I I want to improve in that But it's it gives you like a checklist of things that you that you need to follow in order to really achieve achieve your goal As I said well, maybe I can use something similar, but for continuous delivery, so I use this framework I merged it with the maturity continuous delivery maturity model From the book which says like there are there is several levels of maturity of continuous delivery And it has this little Nice metrics that that says like from bottom bottom up. It's like the you're really not good and you're You're you're the best or you can just like only improve a little bit here and there And it like targets like the things How the things look like so it's like our As is and to be columns, but we're still missing the how to get there So I put those two together and creating something like that, which is a matrix of things That needs to be achieved the blue columns are the states The rows are different disciplines that you need to that you need to cover with continuous delivery and then the the white columns are actually the to get their steps So how the thing can use it? Basically goes to that framework look at the matrix said, okay this is the stuff that we want to improve for example, let's take logging and They they pick the line then they go and look into the blue columns and check like what is my current state So they read those little bullet points and if they are fulfilling all of them They are definitely in that state so they go left to right from the lowest level up to the highest one And whenever they find a state that there is some bullet point that is not fulfilled They identify that there to be state and then they go to this little cell in the middle That says like these are the steps that you need to do in order to get to the next level and it can be You need to add up some like shared libraries You need to set up your infrastructure in order to send the locks into some common sharing Sharing infrastructure of logs and all other kind of things those are already Tied to to how we do things in accolade so it's not like that they would open the book and find out Okay, what I need to do with the logging, but that that's just a basic principle in the book And you won't have the the things how we do it at our company So this is really tailored to how our company does stuff The other thing that you can get there because like if you would have this the book has like 300 pages So if you would really want to put there like how things needs to be done This metrics would be enormous and nobody would read it So instead of that there can be links to something that we call CI CD recipes Which is you can imagine that Similar to standard cooking recipes. So what we have done in there is that we say for example blue green deployment There is a recipe for how to do blue green deployment at accolade and it starts with like this is what blue green Deployment is so everybody understands what they are trying to achieve. There is a Little section about the theory of the of the blue green deployment then there is a section like a Playbook on what you need to do with your let's say service what you need to do with your service in order to have blue green deployment Applied and then you can have something like if you're really interested in blue green deployment. Those are the for the links or conference talks or Another resources that you should go and investigate and this really helped with starting People that weren't thinking about Anything from continuous delivery before and they were like always just saying the book is too thick. We can't like just read it This like you're serving it in a little portions and it somehow automatically gets into the teams Teams mindset. So that really helps last piece the technology Ah There are few things that the that you can do in order to help people help the teams move faster One of them is are some templates But be careful because templates can bite you back quite a lot that happens to us So what we have done and that's something that I called like a static template is that we say, okay if we want our team to get into continuous delivery, they need to have Properly defined the pipeline. So we created a template that actually defines the pipeline for them and they just fill it fill it in For a new project, they just fill in the parameters that are related to the project and it creates the pipeline the whole Pipeline definition for them accordingly to like what we believe is the latest and greatest standard up to that that's perfectly fine for green slate Project that works perfectly the problem comes with the Teams might do some things a little bit differently and they start to Adjust the pipeline and then we found out. Okay. Maybe we have done something wrong So we changed the template, but you can't really have the team just to delete whatever they have customized in there And just use the template again because they would lose their customization that might be really Specific just to that team and are perfectly valid in the pipeline so It generates something like a technical depth and I still don't have great solution For this things that I had in mind is like maybe we can version those template and then we can Also store the parameters that the team used when it when they were Creating the head start pipeline Then they run the same set of parameters on the new version and then just compare like what's different And they should know what was the what is the customization and what not, but it's a lot of work So rather than that It's better if you can use something that I call dynamic templates Which you can imagine something like a like a runtime template so instead of basically bootstrapping the project With the pipeline definition you you bootstrap it with the pipeline definition that doesn't look like it's it's all in there But parts of the pipelines that looks the same like deployment into into environment are basically just another Template placeholders and when you run your pipeline, they're going to be filled in This works a little bit better because whenever you you find out that you need to change the way how you deploy Deploy into your environments the next run of the pipeline will have it automatically for them So those are much better for the technical depth, but much harder to to maintain obviously same as the shared libraries which are like the next steps of those Dynamic templates, which is the things that that you would put into place. So everybody doesn't have to write the very same code to Do something before the deployment? Happens but they just reuse the code same like we do libraries for for the the product itself We should have the same libraries for the pipeline definition or steps within those pipeline definitions so we prevent copy paste and we keep The pipelines actually maintainable when you find out that you have been doing something wrong And I can guarantee you you will find out you have been doing something wrong And you need to change that and imagine if you have like for us We have something around 200 pipelines if you would have to go into 200 repositories and change that and You can't script it because maybe some teams have changed it a little bit to what they needed If you would rather have that step done through a library that everybody is reusing Obviously when then you change the the library everybody has the step done in a new way Without having to update all those 200 pipelines the downside of it or well, let's not say downside but The requirement to such libraries is that they need to be top-notch quality So in such case that that you are going to develop such such Libraries and we did you need to how they say it in some companies you need to drink your own champagne So those libraries should have the full Pledge continuous delivery in place before you start to use them in Pipelines of everyone else because if not and you're used Sorry for the word you screw something up You basically screw the whole company and they won't be able to deliver their software And they will they will come and literally kill you I think so you want to prevent that and in case You're not able to fix it quickly enough You should have good rollback mechanisms in place that that will be almost Instantionals so whenever there is a problem you can roll back to previous version that was proven to be working few points around the CICD engines I happened to be part of picking picking the new CICD engine for acolyte when I joined When we were like yeah, we want to go into continuous delivery But we still didn't have much experience with it So I gathered five points Which I Personally think are really important when you are going to pick up a CICD engine If you don't have one and you want to to pick a new one those five points are the ones that you should look into Number one good visibility into your pipelines. This is my biggest mistake that I have done when we were choosing that engine because our VP of engineering came and he said I want to Have overview of like every single team. I was like, yeah, that's easy We just need something that is able to show all the pipelines that because the pipelines are reflecting how the team is doing and You're there but like what you see on the picture there is just not even one-tenth of the pipelines that we have It's just like what maybe 20 and we have 200 So if you have something that that is able to display all the pipelines It's completely unusable because from that overview you can't tell what's going on You might be able to see that there is something wrong, but you're not able to to tell anything so you would have to relay Zoom in zoom in zoom in and then click on the stuff to find out what's going on So so don't focus on being able to display pipelines for the whole for the whole company Or even don't don't display all the pipelines for the whole team because there might be also if they own Like 20 to 30 microservices this like high-level overview is not giving you any value. So rather than that look for specific features around the dashboarding being able to to compose Really highly customizable dashboards that are able to show you in case there is something wrong They are able to show you that within seconds That's much more helpful and having like filtering abilities on those dashboards as well For example, like if you want to see what's the current state of your production environment You will just filter it out and it will show you like all those steps that are related to to production much better than Trying to display the whole pipeline. Obviously sometimes it's really useful to be able to see the whole pipeline Just to see like what are the steps that we are doing in order to be able to deliver stuff into production but There are much better use cases for more deep dive dashboards than that number two is Infrastructure support obviously like when we were going into that We got some cloud provider We asked whether we are going to be locked in the answer was yes, and we were okay So we are looking for this specific cloud provider support in our CI CD engine and that that's it so we found out those and by Going through through that through like a year later We found out that the support was there but we started to want To do things a little bit differently than what the CI CD engine actually supports because It really depends on who is creating though this CI CD engine and what is his philosophy around continuous delivery again It can be a little bit different towards your philosophy And they can have for example different implementation for blue green that what you would like to achieve so you need to look into not only if it supports your your either cloud provider or or like deployment into your Data center, but you need to look into how specifically it supports that like what is the what is the connections that it can do what are the mechanisms that it uses and Try to identify whether you're okay with such mechanisms or whether you want something more So you need to spend a little bit more into piloting or prototyping Your deployments in the new tool before you actually say, okay, this is the tool for us Number three pipeline definition From the small team experience like we came up almost Instantially we want to have pipeline as a code because if anything happens We'll just like spun it up again from the git repo the pipeline will be there We might might lose the history, but we'll have the the new deployment Available within few seconds in the engine So I was looking into okay our new CI CD engine needs to have pipeline as a code What I haven't realized again that the teams might be different and not everyone is from day one Comfortable with having pipeline as a code some people really like to go especially UI developers they really like to go into the tool and Put the thing together within the within the few clicks because they don't need to really understand how that runs Transforming into something like that. That would be like few clicks on the UI, but in the code It's totally different. They need to go into the documentation so make some Investigation around your teams how they would like to define the pipelines if if they have an idea and they don't if they don't then you need to Count with you will have to prepare lessons or trainings for them to be able to to Create the pipelines the other thing is Some of those engines wouldn't allow you to actually display the pipeline bit before it's committed and applied Which for people that are not used to continuous delivery It's really hard to imagine like if I do this change This is how the pipeline would look like so much better art tools that are able to first of all check the pipeline definition if you have it as a code check it on your local whether you don't have some issues in there and Able to display you or prototype the pipeline for you. So you will see how the how the end result will look like before you actually committed into into your Into your repository and it will recreate the pipeline and maybe spoil the pipeline and One more thing that we've learned was that The tool that we've picked and we haven't realized that got like CI definition was always used from the branch in which you commit it But the pipeline definition was taken only from one branch and you could have selected just one branch Which would define the pipeline which totally makes sense if you have something like a github flow And you have you are only committing to masters and you have short lived branches But if you have git flow where you have those all those different release branches That totally changes Changes the way because the people they are changing their CI Pushing it and within that that CI change that they Realize that they want to change the pipeline as well But since the pipeline is actually taken only from one branch even though they have another branch that like should have a little bit different pipeline the changes were not applied and they were really confused because the CI changes were applied and the like does the CD changes the changes for the pipeline variant and that really was a Hard thing to tackle to explain how it works and why it works that way and that we actually need to change from The github flow sorry from the git flow to github flow because that makes much more sense with continuous delivery Number four is how you can customize that engine most of the Nowadays engine and engines and there is a lot of them like there there are a lot of startups that are creating those continuous delivery engines Because it's really popular nowadays Most of them they will always like show you the console and If you want to do something special you need to Spit it up into the console. So look For those that are able to well, there is nothing wrong with the console to be honest It's just that if your job produces like 10,000 locks files Lock lines, it's really hard to find within seconds like what has gone wrong So much better is looking to something that can give you an overview of the step and it allows you to actually Say from within the step definition if something goes wrong You will display this thing or this link on on the top page for the job So when the developer comes there, he sees right away without having to scroll through those locks and try to Find the needle in the haystack, they will right away see what's wrong And so maybe even what is the remediation that they should apply? obviously Another good customers ability is having the engine should have API so you can like talk to them and Do something extra if something specific happens And The last but not the least is the extensibility most of the CI CD engines wouldn't do all the stuff that you Might want to do in the future. So look into ones that are having the API's are able to Give you the ability that from within the The current step that is running you are able to see what the step is about what are its input variables So you can hook on to that and build the additions on top that are currently missing in the engine This will really help you with moving faster in case the engine doesn't have it because otherwise if you don't have this Extensibility the only other solution for you is either just sit and wait which usually isn't good for us software engineers Where we want to be better or you would have to change the change the engine completely which Can take quite a lot of time if you have So many teams already on board it Okay, I hope You were able to find something valuable some hidden gem in this even if it would be just one That you will be able to take and use in case that you're going to get into I want to start with continuous delivery one Gem that is from myself and will always be first in my mind is Continuous delivery is not about just getting the pipeline into place and saying we're done It's about changing how people really think about how they are coding and how they are thinking about deploying their code into production Thank you questions Good question Some teams yes some teams not yet, so I have currently teams that are already Like I told them In the very beginning we're using git flow. We need to move to github flow and short-lived branches and they were like Yeah, yeah, that makes total sense But then they were just using what they were used to because they were used to it and But in changing that and I told them well It's gonna bite you and now I have teams that are coming and they are saying we think we need to change the branching model so I Think we are slowly overcoming that and it's different from team to team if that answers the question But they need to try well, I guess they need to try it out really to find out That they need to change they need to experience it that it's really slowing them down Because just telling them didn't help so you need to They didn't have pipeline so we provided the pipelines we found out the ways how to overcome the problems with the different branching model to make it easier for them and It kind of worked till we got the Release or how we cut like 14 days cycles now We changed into let's go and have those cycles long length for one day and Then they started to realize it's not working anymore. We need to change that Does it answer the question anyone else Well, if you have some question later on feel free to catch me on the on the corridor somewhere And I'm definitely eager to talk with you about continuous delivery. Thank you