 Thank you so much. We're really excited to be here and to give this presentation regarding pipelines, optimizing pipelines and continuous delivery. So, put a little gender together for this presentation. And before I move on to the agenda, if anyone has any questions throughout this presentation, please leverage or ask the questions in the Q&A portion of Zoom. I'll be monitoring it as I'm going through the presentation. I'm going to go through a couple of concepts that may, you know, be a little overwhelming for some folks. So I definitely want to answer questions as we go through this. It's not rocket science, but you know, we were talking concepts and stuff so definitely ask your questions as you, you know, think of them and I will try to monitor this as best I can and then answer it in real time. So without further ado, let's talk about the agenda. So today I'm going to be discussing software development processes and life cycles. And that as they relate to continuous delivery. Then I'm going to talk a little bit about continuous delivery at scale, which is essentially flagging a few things that you should be looking for some of things that can go wrong when you're performing continuous delivery at scale. And finally, I'll spend some time talking about how you can optimize pipelines, make them more efficient, and some, you know, tips on how you can easily implement, you know, changes that will help you manage the efficiency a little bit better. So yeah, let's jump over to a quick introduction of myself. My name is Angel Rivera, developer advocate for Circle CI. So my role as a developer advocate essentially is for me to go out to the community and discuss, you know, technology and how folks are using technology. I'm also engaging at a grassroots level with developers to get a better sense of, you know, their bottlenecks with certain technologies, some of the struggles they're having, as well as offer up some advice on how they can, you know, fix those issues or resolve them. And it also serves as a really good kind of data collection for me and my team. And once I have those learnings from the community, I can bring them back to our teams at Circle CI. And we incorporate those learnings into our design processes for new product releases and upgrades so that we can bring better value and better features, useful features to the community. And so we're helping developers and trying to make their lives easier and learning what struggles they have is really helpful when we're developing features to help, you know, resolve some of those issues within the platform. If anybody wants to reach out to me. I'm pretty much available on Twitter all the time. It's the best way to get a hold of me. So if you want to reach out just hit me up at punk data on Twitter and I will respond accordingly. That's enough about me. Let's talk about software development life cycles and continuous delivery. Pretty exciting stuff, right. But before we jump into all of that, I wanted to kind of give you a foundational overview of software development life cycles what they are with what's what's their purpose if folks don't know. But it's essentially the rules for software development that teams and organizations develop and use them to govern how they build their software right so this essentially are the rules that people build for themselves and their software development processes, which then are you know, shared amongst the teams and then use as the guide to to building that software. So, you know, software development life cycles come in many, many different flavors. Right now I'm showing you two of the more dominant software development lifecycle methodologies used. One is called waterfall development, and the other is agile development waterfall development is the process where you know it's more of a cascading model and tasks are designed and then assigned to developers more in a in a bulk capacity. What I mean by that is if you look at the diagram for waterfall. That's dotted square box around those three elements within the methodology. Developers are given, you know these these tasks to complete just like any other job right you get your tasks you have to accomplish those tasks in order to achieve your goals within software development though in the waterfall development methodology. It's it's more sequential right so there's a lot of dependencies on getting a task done prior to moving on to the next task which, as you can imagine can create a lot of bottlenecks and blockage right because you're having those dependencies. And waterfall has been around for quite some time like I mentioned earlier. The problem with waterfall is it's very inefficient, especially when you have a huge group of developers trying to develop that software. You can imagine you know that cascading effect will trickle down and create huge inefficiencies in the software development process under waterfall back in the day. So developing software you know it would take us weeks months and years, sometimes to release new software, which is pretty bad for developer because you know you could be working on something. And a year later gets released and then you have to kind of, you know, shift your mindset to that previous context of a year ago, and you know to fix things or to make enhancement so, you know, I really didn't enjoy it. And you know, as developers and engineers we figured out that hey this waterfall development process is probably not the best for you know developing software, and we can do it more efficiently and that's kind of where agile development was born. If you look at that diagram you can see a lot of movement right there's arrows circles, but one of the key elements to or differences between waterfall and agile is agile actually breaks that cascading model. And implements a more of a, you know, smaller scope and concurrent tasks. So essentially right when you have a agile development methodology, you're building your software concurrently so your developers are working on smaller tasks, right smaller in scope. In order for them to like, you know, really focus in on the thing that they need to accomplish. And then they're continuously pushing changes right so you can see all those circles where you have a sprint. And then it just, you know, gets processed passes over to like a daily scrum and some scrum master in the center there is orchestrating all of this. Right. It's an incremental way of building software which is really, really nice because again tasks are broken down smaller you can be super laser focused on things. And you're not, you know, submitting huge code changes which could take, you know, a lot of time to review. But this is the software development lifecycle which I'll also refer to as software development processes so if you hear those two terms, they are the same. So, with agile kind of, you know, bringing efficiencies and optimizations to the way we build software. And the process or a concept within a process for your, your software development processes continuous delivery is essentially, you know, born out of these findings that, you know, we can do things concurrently, and we can deliver software much much faster. So, with continuous delivery, essentially, it was the practice of building and testing software, and then also delivering those changes to code and right changes to code and user environments using automated tools. So right with continuous delivery we noticed that the agile pattern of, you know, smaller socks work doing work in smaller scopes, and then also the iterative or concurrent practice of building software can be applied to how we deliver software. So you know those two methodologies, and then be bringing continuous delivery on that to kind of facilitate releasing that code faster. So within continuous delivery, though, we have another concept called continuous integration and continuous deployment. These two concepts are actually separate, but they're often, you know, discussed at the same time, which is kind of why you see it here and see this acronym CICD stands for continuous integration continuous deployment. And again, two separate concepts which are generally addressed at the same time because they are so inter-coupled and related. So, let's talk about continuous integration. It's simply the practice of merging all developers working copies to a shared code repository so what does that mean. Within the principles of CI, that means you're writing and committing code as often as possible, right, so you're committing code to a branch or some sort of repository trunk. And then those committed changes are shared and committed to a, they're committed to a shared code repository so that the reason you're doing that is your team can actually understand and see what, you know, how you're building the features in your code. And it's also a way for you, the developer, to pull down your peers changes and the stuff they're working on and understanding, you know, kind of where they're going with their code so that you can have that nice comfortable merge of functionality, especially if you're overlapped, right, and you're touching code in an overlap kind of sense. So, the other benefit or I would say element to the principles of continuous integration is you're testing your code on every commit. So this is adding a layer of quality, you know, from the developers perspective, every time they make code changes and then are committed, you're testing that those changes and then my favorite feature or benefit from from the principles of CI is the fast feedback loops right so as we're testing our code on every commit. We're able to understand or know if we have issues with our code because the tests are either going to fail, or they're going to pass or they're going to fail and if they're failing. You definitely want to know right as quickly as possible so that you can go back and fix those issues and then resubmit them right to the whole continuous delivery process. So fast feedback loops is one of the key benefits for me at least as a developer from CI. Now with continuous deployment. It's just a practice of deploying this new software or the changes that you created right and you're releasing it to your target audience target environments. There's more on the side of basically, you know, deploying software as as the name indicates. So with the principles of continuous deployment. We are creating release artifacts for continuous delivery or continuous deployment process. We're deploying code artifacts to those target resources could be a Kubernetes cluster could be just a simple, you know, somewhere that you just hosting the website whatever the target environment is doesn't matter but you do. The fact is that you're deploying these code artifacts to those those environments. One of the other principles is sort of like that at validation and services once everything's deployed. You could have a successful deployment but then your applications could be, you know, failing miserably. It's on the server that you deployed to so you know you want to validate that those apps after deployment or functioning and the services are, you know, executing as as expected. And then finally, in the CD, or the continuous deployment principles, you want to establish some sort of monitoring or surveillance of the state of that application to ensure again that it's functioning as expected. And also, if you have any sort of resiliency or recovery type, you know, mechanisms within your, your, your application functioning. You also want to monitor that to make sure that it's recovering and being resilient as designed. So, you know, with continuous delivery, CI CD, software development processes, all those things I mentioned are basically conceptual right these are things are guidelines that people define and then operate under, but none of it means anything without automation. And the reason I say that is the automation is what brings those concepts to life, right, they're the things that execute automation executes the continuous delivery concepts and plans that we make. So wait, I have a question here before I move on. So the question is how, how it tests the commit works for new features. Okay, so I think the question is. Let me let me rephrase this. So how does it. Okay, so, so when you're testing the new feature so when I get further into the presentation I think in a couple slides they're probably a little bit more clear. But what will happen is since you're running automation in your, you know, on your code changes. Part of the execution of a CI CD pipeline is running tests right so if you define that in your CI CD pipeline, like you know run a unit test or something it's going to happen every time code is committed and pushed upstream, so that you know you're not doing it manually on your laptop you just let the system do it. And that was, that was answered. So right getting back to automation. Again, you know, this is automation is what brings all of these processes to life. And we're going to talk about CI CD pipelines right so when you have your automation. Now, the. Sorry, we have another question. Testing comes under CI or CD, both my answer to that is both because CI pertains mainly to the developer side of things where you're like writing code. And you know you're doing that iterative testing, but then on the continuous deployment side, you also want to test and validate right so when you're validating something usually testing it, meaning deployments right security scans, maybe some regulatory scans and your deployment of patterns. So, the answer is both. But primarily if you're talking about code. It's generally in the CI process. So, we're talking about CI CD pipelines right and automation the automation itself is your. The mechanism right to execute with circle CI as a platform, that's what we are we are the automation that brings your, your, your, your concepts to life your continuous delivery and software development process to life. And they're defined what in what we call a CI CD pipeline or continuous delivery pipeline. And those are built upon jobs right so if you look at this diagram it's a quick snapshot of the circle CI dashboard and if you were to use our tool. This is what kind of what you would see with your pipelines as you can see right there's every every little box there is in this area is a job right it's a block of commands that perform a specific function. So if you can look, if you see at the left bottom boxes run tests and if you work your way up. We're building a Docker image and then right I'm also creating a Kubernetes cluster in this example, but you can see it's going to you know move from the left. To the right, as our pipelines succeed. So, once you have that set up right. The CI CD pipelines are really, really important for what I'm going to talk about next which is performing continuous delivery at scale. If there's any questions that there are some questions great. So before I move on, let me answer some questions here. So what are the tools. I guess what are the tools used in CD. So, in the context of circle CI for our continuous deployment. If you're talking about that which I assume you're talking about the, the tooling is pretty much anything you use right so if you're using an Amazon AWS CLI to deploy things to Amazon to Amazon. If you're using that cloud provider that's great if you're using Google's CLI to deploy to Google that's, that's also, you know, available and you can use that. You can use infrastructures code technology such as emphasis is a sorry such as Terraform or Pulumi or Ansible right whatever whatever tooling you want at circle CI we want to meet the developer where where they're at so you can leverage any kind of tooling you want within that because it's just essentially automation right. And then the other question here is. Is it a good idea to integrate or I guess integrate CD into production environments. Yes. If you ask me yes, you should be doing that. There's a couple reasons I mean you could still have your staging you can still have you know q amp a type patterns within your software development and release cycles. I definitely recommend. Excuse me, going in and do you know leveraging continuous deployment and production the reason is your production environment. You're never going to be able to replicate it 100% right no matter how much you try. It's just not going to, going to you know pan out for you that you're never going to reach parity. You know, obviously you want to control the way you deploy that and keep really good controls around how that's deployed, but I would definitely, you know, recommend you start looking at, you know, implementing a production level deployments because that's the whole point right is to get the code that you built to production as quickly as possible and as effortlessly as possible. And there's one more question here. So, I think both CI and CD needs feedback loop for automation. May I know your view of this. Yeah, so yes, feedback loops are exactly what is required to have a successful and efficient and consistent continuous delivery operation right so like the idea is to get things done as quickly as possible but you want to have quality built in right and by testing that helps ensure it doesn't give you 100% feels right doesn't cover all these cases but it will minimize the risk that you introduce you know some weird bug, because you didn't test so. So let's go ahead and move on. I know there's some other questions here. So my voice isn't clear I'll try to speak up. Yeah, let's go ahead and jump into continuous delivery a lot of stuff to cover here. So, with, you know, talking about delivery continuous delivery at scale. What I'm showing you here is a very generic. Let me get rid of that. Okay, so it's a very generic SDLC or software development lifecycle, like, I guess pipeline. And, you know, what CI CD mainly focuses in on in this SDLC example is generally going to be in the development testing and deployment phases of your processes right. And this is where marrying CI CD and and into your software process development processes is really important. And most beneficial I should say as well, because, as we all know right code and repositories grow over time. And that's where I'm talking about, you know, the complexities growing the volume of code grows. And on top of all of that, you know, all of those things, and be able to, you know, address them as soon as possible because if you're not maintaining that growth, it's going to eventually like bite you, right it's going to impact you negatively. And establishing and maintaining mechanisms for scaling, you know, that provides solid support for pipelines is very difficult to achieve without constant surveillance and interactive reviews, meaning, you know, constantly going back and checking are our processes, you know, kind of matching our, our continuous delivery processes right. So that happens. And that's what I call like a misaligned CI CD segment. And that can get you into some trouble as well. So when your processes are not properly aligned. You know, you can start introducing huge, you know, inefficiencies slowing the pipeline and release processes down. You can also substantially hurt your quality right of the code. One of the things that happens here is if you're looking at this diagram. I'm showing you like you know the development phase things are kind of out of whack they're not really properly aligned with your software development processes or your guidelines. This happens a lot when teams are interested in kind of, you know, not working in a controlled process. It does happen occasionally amongst teams and within organizations that, you know, your software development processes don't exactly match up with your continuous delivery processes. They're like, you know, everything's kind of out of whack and in different layers. So but I'll be talking about that aligning these in a little bit here so. So, right when you have these misaligned CI CD segments that starts to introduce confusion among your teams like who's doing what when are they doing it why isn't the automation reflecting this, because again, you know, your, your CI CD or continuous delivery cycles or pipelines are a reflection of your software development practices and they, well they should be at least and when they're not aligned, it will diminish the quality of the code. You'll actually waste a ton of cycles development cycles which is very expensive because again developers are, you know, very expensive they're the most expensive piece to building software in most cases, and then it will definitely slow your relay cycles down dramatically. One of the other things that when you're operating at scale is directly attributed to the platform CI CD platform effectiveness so again the CI CD platform that I'm addressing here, or at least an example is circle CI. That's what we do, we provide the automation we are the CI CD platform for your for your pipelines. And you must ensure that you have a platform that's very robust very resilient, and that can scale CI CD platforms come in many different flavors, many different companies. They actually select, you know, circle CI we solely focus on the continuous delivery pipeline, whereas other, you know, platforms, they may be more of a all in one solution and CI CD features are kind of bolted on. And they feel kind of bulky and slow and it's actually really not their primary focus. They may provide you some, you know, effective CI CD performance on your platform but as your as again as your as your code and your project scale those solutions. The performance if they're not scaling properly or being very resilient or robust can can kind of hurt you in the end. So one of the advice I give you is to look at your CI CD platforms evaluate them make sure that they are, you know, filling the needs that you have for your CI CD pipeline and are able to scale with your workloads. So as you grow more members on your team, you know, the infrastructure needs to also be able to be resilient robust and scale with your workloads. So, one thing I will say, you know, teams really must understand that reliance on CI CD tooling and execution is essential and will increase dramatically further. You know, you're not cementing your CI CD platforms as mission critical. And if they're not working properly and you know you don't have CI CD capabilities for your team. It's going to cripple your productivity. So, you know, one emphasis here is to properly manage your, your platforms make sure that again that they're, they're resilient they're robust and can scale with your workloads, and that they're properly being maintained. And one of the like huge efficiency killers is when your CI CD platforms are not working properly, you're going to be in a world of hurt right as a software developer as a software development team. So, I'll stop here to before I jump into the next section. Let's look at some questions, which programming language would you recommend for the automation. Oh, that's a good question. I have reason why. Yeah, so so right we're all developers. The reason why this is a good question is, in general, most the ICD platforms have settled on and this isn't quite a language it's a data structure in order to define your software. Sorry, your continuous delivery pipelines, the language or the sorry, the data structure the syntax that we use is called the amel. And I'm sure if any of you have worked with any more modern cloud native type systems or solutions, you're probably quite familiar with what the amel is which is the data structure, but we use it's a it's a human readable very highly declarative data structure that we use to define things and one of the cool things we can do with it is define commands right so it's a nice. It's a good text to use human readable. It can get quite verbose and wordy, but at the end of the day, it is what we use today, and actually, most CICD modern most modern CICD platforms are defining your pipelines within the amel. The next question is, are there any approval processes in this pipelines. So, generally, you don't want to have that right manual intervention in any pipeline, but there are situations where you need it right so like if you're in a regulatory space and you need to have some eyes on a segment or a job where you can continue, and you need that manual approval. Yes. At least that circle CI we have an approval process, you know, we call it an approvals job, and you can set that in your pipeline at any point you want to see, you know, some you want to stop the pipeline until someone manually can look at it, and then let it go, you know, click the approval and it'll it'll proceed. So yes, there are approval processes. Next question, do you have any numbers regarding the performance of your machines versus the ones provided by Azure DevOps. I believe there are internally there are some some numbers I don't know them I don't have them. But yeah, we're, yeah, we probably have them. If, if you want to have a further discussion on that either hit me up on Twitter, or on our discord site and we can see what we can do to see if we have some metrics we can share, but I'm not sure where that lives. So, the next question is, except, except static code scan security testing. Are there any other security features that we can implement in CI CD pipelines. The answer is you can implement at least in the circle CI platform you can implement pretty much any kind of security. You know tooling or features or processes that that you work with. Again, you know, circle CI is designed to build to meet developers where they are so we enable you to run pretty much whatever you want with within reason on in your in your CI CD pipeline run times. So whatever tools you're using right now. And by the way that that's pretty much true across the board with with most, most modern CI CD tooling. There are limitations and restrictions but in general yeah you can pretty much run whatever you want however you want within those platforms. Is there the next question is there a best practice around making builds for different target OS environments in the pipeline. Yes, there are. So, you know, it depends on everyone's case is different but there are some common guidelines right that you can use to to build things. You know, obviously when you're building for different architectures so to speak like let's say you're building for x86 or arm which is what we introduced recently as well, the capabilities are there to now natively build, you know, software for those two architectures. It's definitely something that's like, you know, again, you can do it many, many different ways. But I would say that, you know, doing them efficiently is very important. So, you know, putting an emphasis on optimizing those build processes so that they get done as quickly as possible, but you can implement pretty much whatever kind of build process you want for those architectures. So, let's go ahead and jump into optimizing pipelines I think answered all the questions. So, what I mentioned earlier was, you know, Circle CI or not Circle CI but continuous delivery is is really important it's critical mission critical for development teams to implement this and ensure that you know their pipelines are running efficiently. So, the one of the main reasons is as I mentioned earlier right, you don't want to waste any developer cycles meaning you don't want humans sitting around waiting for, you know, feedback on the co changes that they're making, because your maybe your system is down or whatever, and the reason is, you know, we've calculated that the average cost of developers time per minute equals to about a buck or a dollar and 45 cents right every minute. So imagine if you had to. If you had, let's say, I don't know 100 developers sitting around for 60 minutes because your CI CD pipelines or your platform is not functioning right as it's down. And the developer cycles wasted is devastating. So it's in and can get very very expensive so you know with that in mind. Optimizing pipelines is super important, but you might be figure asking yourself like okay so how the heck do I, you know, optimize Angel how do I, what do I look for, and I'm going to touch on a few things. One of the ways that, or one of the like easy ways to become inefficient or or your pipelines becoming very slow and and not really in clunky right not really streamlined is again by what I talked about earlier is having misaligned CI CD segments with your software development processes. So remember this diagram. I was talking about how everything's kind of jacked up right not really aligned. But this is kind of like what you want to see moving forward you want to see that your development processes are properly aligned and executing in accordance with your guidelines right. So testing debugging all that good stuff. And that's where you can put like, in that phase security, you can do things like submitting and pushing and deploying your, your, your artifacts, like maybe QA, or staging, right you can do all those things within that phase of your software development lifecycle and then finally rate deployment, which can mean many many things. At this point, we're just going to deploy to production. Now if you notice all these things have a circle with arrows around them to indicate movement and iterative right processes so these things are happening all the time and every time code changes. So again, you want to make sure that you're aligning those bits with your software development processes so that they're just simply running efficiently and also as designed right within that higher level software development process. One other way to optimize pipelines and this is a, this is kind of a thought that I had a couple years ago, I noticed just with my engagements in the community that you know there's a lot of people talking about how there's gaps within their software development teams or release teams. You know, they're, they're just not gelling very well and you can throw DevOps into the mix as well where you know that concept there is where developers and operations teams are working kind of simultaneously and understanding each other's roles. And that is, that's a really hard adoption and implementation to make and it's never the same across the board for any organization or team. But what I'm proposing is, or what I've proposed in a blog post which I'll share the link to later is this concept of a continuous delivery engineer, and I know our industry doesn't need another role right or another, another title. I really believe that a continuous delivery engineer is kind of going to be the future. And this is why right so like in the blog post I just, I describe a lot more detail which you can read later, which I'll share that link but essentially there's two characteristics here that I define. One is the kind of the human element for this role like things you want in an individual to that's in a continuous delivery engineer position to have is like strong communication skills. I think you want that in every role but it's really important for that for this role in my opinion. You want them to have a keen analytical skills which means they're able to understand, you know, identify and understand patterns, so that they can, you know, decompose those complex patterns and processes and describe them proficiently to folks who are not in the industry right they could be business stakeholders, or people like that right outside of the software, or the technology kind of schema. You also want like proficiency in automating and optimizing these processes right so that you can also build confident teams and maintain that that rapport with your teams right so they're all kind of tied together. And it's essentially those characteristics are really important to be successful in this in this position. On the right side we have some of the duties that I would imagine, you know, this position to have. Obviously, you are essentially the CICD czar, so to speak. So, you know, you would be responsible for developing CICD principles. That involves, you know, touching the higher level software development processes and collaborating around that stuff to ensure that you know your product your software development process are direct reflection of your, of your release cycles or release processes, as well as review and modify CICD principles, iteratively, iteratively right so meaning just don't write the principles and guidelines that you're going to operate under but go back and check on them often. Also right you're going to maintain the tools and platforms. In some cases right so you will be the expert of the platform you choose whatever that platform is, and you will also be helping to maintain, you know, keeping that thing alive or the, yeah, keeping that platform alive and functional so that you don't waste developer cycles and then finally right you're going to automate all your processes and then maintain those pipeline configurations which are what, what defines your pipelines and what gets executed. So with continuous delivery engineer again it's just a role, something you may want to think about implementing inside of your organizations. So with technical optimization strategies, what I described previously was more of the human elements, and what I'm going to describe further are more, you know, the technical aspects that you can look for and optimize as well. So one of the main things I noticed that developers in general like this is a common problem in the industry is the lack of knowledge of their, the languages frameworks and resources used in your text in the tech tech sacks right so having proficiency in those languages is really important and understanding the capabilities and limitations of these languages frameworks and tooling is also critical. Because when you know how those things work, you have the ability to kind of quickly understand when when things go wrong right like why things are going wrong. Like, you know, if you're having performance issues, the code is running slower, you have an ability or at least some insight into where to look right to troubleshoot to figure out why is, why am I experiencing this performance hit, or maybe you know even bottlenecks sometimes the software just, you know your data out grows the software happens in a couple languages, your data sets you know that you're you're transacting on within your languages are just too big. And in some cases, you know, most languages frameworks have this ability to run things, again concurrently right so they have these concurrent capabilities. So you should be aware of these native concurrency capabilities within your stacks to actually implement them when you need to right so if you're if you're, if you're seeing like, you know performance hits. And, and, especially like I see this a lot with testing, the test suites, you know, as they grow they become, it takes longer and longer to, to run the test suite because you're getting into that whole situation where you know you're cascading so like you have to wait for one test to finish, and then move on to the next. In most cases, you don't have to do that you can, you know, implement the, the native concurrency capabilities in the stack, and then run all of those tests at the same time. This is similar to like the multi threading concept right. So, again, you know, understand the languages and frameworks that you're using, because you can, you can, you know, create some huge performance gains without even touching your city pipeline right these are things that can, you can do natively. Without even having to tweak a pipeline which is really important, because this is like a localized change. Now, what we're talking about next is tweaks or not tweaks but concepts within the CI CD pipeline concurrent job execution is one of them. It's huge. You know, you have to ensure that your, your jobs are running simultaneously wherever you can. It's not always possible, right to execute a job without a dependency, but if you can run and orchestrate your jobs to run as, you know, at the same time wherever possible do that, because that's where you kind of gained that velocity within your your CI CD pipelines and release cycles. Avoid building job dependencies. Again, you know, what we talked about like in the waterfall example where things are like dependent upon each other right you can't start a new top a new task. And it kind of ties back to running things concurrently. Avoid any kind of build dependencies if you can wherever you can. Again, there's going to be places where you need to, you know, have a dependency like you have to stop a job, or you can't continue until a previous job succeeded. In some cases it's a security scan right so you don't want to deploy, or you don't want to move down your process. If you have some vulnerable code that maybe has been detected right in one of your jobs. So you know, again, avoid building these dependencies wherever possible within your pipelines, but when you have to build those pipeline or those dependencies. It's okay, right, but this is the general rule that I like to go by and suggest. Let's talk about caching right so caching is is, you know, the ability to basically down a bunch of files or dependencies or code or whatever. One time and then update things only that the things that change. It's super helpful. So we know when you're when you're running pipelines because the least amount of things you have to download the faster your pipeline is. I warn you, you know, to use the cash cautiously. There's a couple cases where you know you're pulling down software, and you're pulling down all the software all the time that's not caching. I've seen that quite a bit. And then you know it's a problem, but then we're going to go ahead and talk about, you know, only update the change dependencies within your cash that's that's the proper way to implement caching is you know, the first cat the first run of your pipeline will only download all the dependencies. And then from there, you know subsequent runs of that pipeline will only update the changed dependencies or the changed files in your cash right so you know keep that in mind when you're implementing cash because it can, if you're not implementing it properly, it can cause, you know, a huge reduction in velocity in your in your pipeline. And the other, this is a this is more of a tweak right so if you notice like your pipelines are running on compute nodes everything that you do in a CI CD tool is running in a runtime on a compute node, depending you know it doesn't matter could be a server in a data center could be, you know bare metal server in a data center could be a virtual machine and Amazon whatever the resource classes in your CI CD tooling. So please make sure that, you know, your compute nodes are adequately sized they have enough memory they have CPU, they have enough this space network right all these little little details do impact your CI CD pipeline if they're not properly size. One of the biggest things I see is, you know, people add more code, things run slower, and then they just don't do anything about it, they continue to use the same compute node size. And when you do that, it's costing you money right because you're slowing things down. Now, a lot of folks don't want to upgrade or, you know, to a more capable or more capabilities on their compute node, because they fear the cost will you know be astronomical. I can tell you that that's not the case all the time. Actually, more likely than not, you will experience huge astronomical increases. So think about it this way when you implement a faster compute node things are getting done faster. So that's shortening the amount of time you're using that node. And you know, it could, I've actually seen it be cheaper to run and on a huge node right then running on a on a slower, more, you know, the cost consuming node. So you know, do your, do your due diligence due diligence there but if you want to increase some speed and certain aspects of your builds, you know, make sure that your, your, your nodes are properly. They have the proper capacity. So test execution. This is really important. You always want to be testing your code right. There's no question there. But you also don't want to, there, I believe there's a there's a thing is such as over, you know, kind of overkill for the test. And what I mean by that is, you know, if you're in the development phase, you don't want to be running end to end testing all the time right on changes that you're making every time. So what you want to do is scale that down to and scope it out to more of a unit test type execution you want to make sure that those tests are super fast right and keeping the developer going now if you promote that that that bill to a something was it. Yeah, if you're going to like a staging or QA, then at that point you want to run more extensive comprehensive tests, they're probably much longer, right to run. So you know just just keep an eye on when and how you're running your test. And actually there's one more other thing. Make sure that you're running tests that add value. So you know if you're testing something and every time you run that test. It's green all the time, you should probably look into that test and add, you know figure out like it's just adding value, because it's always green all the time, is it even functioning properly. So you know those are kind of things that you need to check out when you're running test execution. One other thing I want to talk about is conditional job execution. And this is basically how you orchestrate your jobs right. So, again, going back to that, that example of like maybe you're in a development phase of your code. Do you really need to run a certain tasks that pertain, maybe to promoting things to production or to Q&A type, you know, elements in your in your software processes. You know, you want to again align the commands being executed within jobs to to, you know, scope them out to exactly what it is that you're performing. So again, if you're just, you know, performing development of new features, scope those job executions a little bit better so you're not like, you know, doing full blown promotions across the board for no reason. You know, that that I see that all the time and it just adds more and more time to your overall pipeline run and then slows things down quite a bit. You just want to run those when certain conditions exist. And finally, I want to talk about set setting a bill duration target. So this is essentially like every, you know, every pipeline is composed of many jobs. What I mean by this statement here is you want to ensure that every job that you run is being run optimally and within a certain timeframe. So by setting a goal, I personally like to set a goal of 60 seconds. It's aggressive. It's unrealistic. But, you know, we need to we need to set these aggressive unrealistic goals so that whenever we see a job right going over like for me if I see a job going over 60 seconds, you know, then I it's a red flag, and then brings my attention to it, and I can look at it and see what is going on right analyze why is this taking longer. And you know it's like a troubleshooting kind of flag. But again right you want your pipelines to be very concise very, very fast right because you're the rest of your, your release is depending on that your release cycles are depending on that on that quick velocity right so set your build target, build duration targets, very aggressively. Now of course right if you're sending jobs off to like Amazon file build or something like that. That's out of your control. So you can't really you know if it takes three minutes, it takes three minutes but at a minimum you know that that job is taking three minutes because it's out of your control. By the way, again, if it's, you know, the threshold is exceeded that three minute average is exceeded, you need to look into why that's happening right. So again, just, you know, set some aggressive build duration targets for every job that you run in your workloads. So that's basically all of the optimization bits. I'm going to go ahead and recap and then I have some questions which I'll stick around for even if we go over a little bit. I'll definitely answer those questions. So let's go ahead and recap on the optimization of pipelines, align your CCD segments with your software development processes really important to keep those nice because again, your continuous delivery processes are reflection of your software development processes, establish, you know, if you can establish a continuous delivery engineer or at least look into that kind of role. It's definitely, you know, like I said, continuous delivery is being adopted by pretty much everybody. It just makes sense to have like a central team or central role that's kind of overlooking and overseeing and managing all of that for your teams. You know, understand the languages and frameworks and tooling that you're using have a deep knowledge of those things because proficiency in those technologies will help you learning, you know, knowing the capacities, the limitations of frameworks will definitely help you, you know, in many aspects. Execute your workloads concurrently. That's always a motto that I've adopted years and years and years ago, and I try to execute that at every turn. Avoid building job dependencies right. Like I said, there's going to be times where you just can't do it. It's just, it's impossible. You have to have a dependency, but wherever you can eliminate those. Please do so. And trust me, your life will be so much happier and faster. If you implement that, that advice, use the cash cautiously right and only update the change things. And then obviously use adequate compute nodes, adequately sized compute nodes with the proper capacities, whenever you're, you know, implementing them inside of your CICD pipelines or executing CICD pipelines, conditional job execution, don't run everything all the time, you know, try to filter out what needs to be run and in context right so again if you're developing software and your development phase, don't run all the things right in your pipeline, only run the things that you need to, to progress the next segment. And then finally set those build duration targets. You can use my number if you like 60 seconds so you can build your own, but the lower right the number, the faster things complete. So I think that's pretty much it. I just wanted to share this link, which is basically the state of the software delivery. So it's a thing with the report that we've been releasing annually for the past couple years. And it has a lot of data on how to benchmark yourselves, as far as like, you know, metrics on what a high performing team looks like, what kind of recovery developers are facing and this is all data based off of the usage on the CircleCI platform so it's pretty accurate we do millions and millions of builds per day. So like, you know, we have a lot of data and we're finally, you know, making sense of it all and releasing it to the public so that they can understand how they can improve their release cycles. The CICD engineer role that I talked about, if you want to read that blog post, just use this link. And there's actually another follow up piece to this that I wrote. So it's been about a two year project but at the end of the day I update it every year with some more thoughts and stuff so you know if you want to get some more detail, go ahead and check that out there. And finally, if you have any questions that I didn't answer here, join our Discord server. And in there, you know, CircleCI developer advocates, the rest of my team is in there, and we're generally answering questions or you know, talking with folks as well. So it's another way for you to connect. And that's about it for now. Let me answer some of these questions that are in the queue, and then we'll finish this thing. So, one of the questions is, can we implement all of the CICD tools in Docker, if we implement that, what are the pros and cons. So, like, you can use Docker within the CICD tooling I think that's the question. So can I use Docker within the CICD tooling. Yes. That's one of the, what we call executors, which is that compute note I was talking about. You can use Docker images, you know, build your code on Docker images in CircleCI that's not a problem it's actually encouraged, especially because developers right they're using Docker more and packaging their apps up. So yeah, you can definitely use that. And the pros are that right it's it's the same workflow doesn't really change for you you can use the same Docker file same code. The pros are there are limitations right so it is a Docker image or Docker container. There are performance issues as well. You know, so if you're if you're looking to do like high performance things I would recommend using a what we call a virtual machine executor, so that you know you have the full control the box, but that would be one of the cons is like you know there are limitations. The next question is, can we manage the pipelines and jobs as code without setting up everything using the UI. Do you have any recommendations for tools and practices. So, yeah, so there are many tools out there they all do it do do the CI CD pipeline definition differently at CircleCI like I said, we use YAML most tooling uses YAML. But what I tell people is when you're using CircleCI you should not be in a dashboard, unless you're, you know, checking on a status like a feedback right so if your build failed, you want to obviously debug you can do that. If you want to SSH into the box you can do that with CircleCI select the developers can, you know, debug in the environment where it failed versus oh here's a bunch of logs, and that really don't make sense, and then you try to debug on your local machine which is nearly impossible, or it takes a very very long time so yeah you can do that within CircleCI for sure. Most other tools also use, you know, depending on the tool, but you know a lot of them have that code capability to define your pipelines, and you don't have to jump into a UI. So the last question I have here is can you elaborate a little bit on avoiding job dependencies in concurrent job execution. Yeah, so avoiding job dependency so a job dependency would be let's say you have a manual approval step right. Like let's say your pipeline's coming along and it has to stop because you have a rule that says a manager or some you know person with authority needs to look at the state of the release at a given time so maybe you have a Docker image that's going to be deployed to a maybe a production environment. The dependency there would be that that approval step right. You actually absolutely need that because maybe like I said you're bound by law via compliance the regulations right. So that's a dependency now where you were I'm talking about where you can eliminate that is anywhere else you don't need that just run things, you know, concurrently, like, okay for instance, in my pipeline I usually build using infrastructure I build a Kubernetes cluster so that the release that I'm working on can be packaged up into a Docker image thrown or not thrown but deployed into a Kubernetes cluster, and then I can run some smoke testing you know to make sure that deployment is legit, and you know to verify that that Docker image works in a Kubernetes cluster. Now, most people will do things like, you know, build the cluster and then obviously you can't deploy this built image until that cluster is created right so that's a dependency right that you cannot break you need the cluster before you can deploy, but all the other things like running unit tests running security tests, building the Docker image and the Kubernetes cluster at the same time can be accomplished concurrently right. So that's what I mean by you know eliminated because a lot of people wait for the cluster to be built, then they'll build the Docker image, and then they'll deploy the Docker image. What I'm saying is now just do everything you can as you know quickly as possible because if one of those fails, then the pipeline will stop. And then you know you have to go back and figure that out. So I hope I answered that question regarding pipelines. And I think we're out of time right. I think so. Okay. So I don't see any more questions. Thank you everybody. Thank you so much to Angel for his time today and thank you to all the participants who joined us. As a reminder, this recording will be on the Linux Foundation YouTube page later today. We hope you're able to join us for future webinars. Have a wonderful day. Thank you.