 Awesome, thank you so much Christina. Hi everyone, I'm so glad to be with you all today. It's super exciting to be a part of the Linux Foundation live webinars and you know one of the nice things about live webinars is that we get to interact with each other so please feel free to ask me any questions chat and chat with me. I have it all up and I'll be able to answer questions during the presentation as well so feel free to ask anything make any comments that you'd like to make. I know that this session is a more interesting one because I think a lot of people already inherently know about how governance works right. We know about government systems we know about how rules work how standards are and and so applying it to it and to our software services is really interesting because it suddenly now we're thinking about it from the perspective of well how do I govern my software deliverables and how do I do it through this automated process. And so that's why I'm going to really enjoy the session because we're really going to be talking about how to apply governance to CI CD and a little bit more about me I am a technical evangelist at harness and we help people with simplify and scale their CI CD and their software delivery. So if you have any questions about this presentation just feel free to ask. I really do enjoy these kinds of presentations and I want to start this presentation by kind of going over some of the reasoning and history behind why we may even decide to care about governance, particularly in the of software. So in 2020 there's actually the solar winds hack and a lot of people I think were surprised but also kind of reflective, especially in the CI CD space we were thinking about well, why did something like this happen and why did it affect so many people. In 2020 December 2020 a cybersecurity firm called fire I who was a customer of solar winds, an organization that provides it monitoring solutions. Notice that some of their cybersecurity software was being used outside of their company. And when they looked into it they had noticed that there was actually a supply chain hack that had occurred where solar winds had shipped out malicious code within their software services to over 18,000 solar winds customers, and actually impacted big organizations like the Department of Homeland Security, NASA, Microsoft, and it made people even today just reconsider like what it what are they using to deliver their software services and how can they better avoid supply chain hacks and malicious targetters and malicious hackers throughout their processes. So actually, I think, in a lot of ways, we know the inherent risk of software failures right things like this that have impacts, it may have an initial small impact on a group of people, but then have this like chain and lazy chain effect that inevitably impacts more and more people right. The fact that solar winds was compromised may not have been a big deal, but now the fact that they had, they had caused a compromise to Microsoft or the Department of Homeland Security is a bigger deal. And so in 2017, they looked at over 600 software failures and noticed that it was due to a misconfiguration setup in their environment. And that actually resulted in 3.6 billion people being affected and a financial loss for the organizations of over $1.7 trillion. In newer studies today, we're seeing that the cost of correcting software failures is actually fairly expensive. So it's on average $4 million to create correct things like a data breach that's already happened to customers. And I think this is really interesting because we know this, like we know that this is what happens when we have issues that get found after we've delivered a piece of code or we delivered change, a software service change, right. We know that to be able to say like, oh crap, we made a mistake. We need to take that back. That actually costs a lot more than when we're actually coding or designing or thinking about the requirements that we need to gather and the architecture and fixing those mistakes earlier on and catching those big security, potential security vulnerabilities early on than further down in the process, right. And I think we've understood this for a few decades now, right, with the emergence of agile development processes, and now DevOps principles and practices and tooling and automation. And I think this is a good segue into why we might want to automate governance for our services for our software services, right. So that's why I want to talk about automated pipeline governance or in other words CI CD governance today and CI CD governance is how organizations attest to the integrity of assets in a delivery pipeline. So it's everything that's related to your delivery pipeline right. It's the infrastructure that you use that you provision. It's the different environments that you decide to have. It's that change control board that you may have. It's the people who have access to control your delivery pipeline. And then it's also inevitably those software services that go through your delivery pipeline right that code that gets that gets tested that gets verified that gets packaged up and put onto the server where it's running now and now it's open for all of our customers. In the session I want to go over some more definitions about software delivery in general and kind of give you a better sense of what is, what is CI CD and how can I succeed with CI CD. And then I want to go over some of the principles of governance. And then that's how we can finally get to this final part of the presentation where we can talk about practices and tooling and how to integrate that into our CI CD pipelines because it's only when we can kind of understand the foundations first that we can start building up pipelines that are that automate our governance and our standards and the things that we want to be able to accomplish or set for our for organizations and when it as it relates to delivering software. So I want to start with continuous integration and continuous delivery. If anyone has any questions I'm more than happy to answer them in chat as well so feel free to stop me if I'm, if I say something that doesn't make sense. But I do want to cover some components of continuous integration and continuous delivery now. So, CI CD, in the sense is just a way of automating the process from getting from idea to production. So it's, it's when we have this idea to serve a customer to start to have, you know, some software to do some, some type of thing that we decide well okay, we want to release it out to a customer. And so that process from idea to production can look like a couple of different things right in the most basic sense, we would, we would want to package that up right we would want to test it make sure that there's a certain level of quality right especially if we're going to put our names on it. And then that's when, after we've tested it, we can have a good idea that it works, then we can actually deploy it and release it out to our customers. And so a CI CD pipeline just automates that process it automates the process that we have for our software development lifecycle, and this process can continue again right, we can deploy something into production and say, well, wow that did a really awesome job we have new requirements we have new features we want to expand this or you know we want to have this new feature that does XYZ with for our customers. And so you can repeat that process again and deploy it again and again. And the idea is that we can do this repeatedly and much faster and even with more sustainability. And then if we were to do this manually with our people or if someone had to do these things. If like one person was in charge of doing all of these things manually without a CI CD pipeline or without any automation. And so that's what CI CD is right and a lot of times people will kind of combine CI CD into this kind of one term that talks about how we get from different parts of this cycle right maybe it's billed to deploy for some people and for some other people it's to get from you know billed to test. And so it can mean a lot of different things I think sometimes and that's where some confusion come into play. So I do want to cover about cover some of the differences between continuous integration and continuous delivery. We don't have one without the other and expect to be able to deliver software on a repeatable sustainable and quick process right. So, in continuous integration, we're really thinking about our development workflows. So it's when we have already come worked on our code, we commit it right and now we just want to package it up in some way. And so when we package it up will actually build into the package, the dependencies the different libraries that we may be using right if we're leveraging some type of framework, or some type of runtime, you know that that runtime will get compiled or interpreted. And then you'll be able to version that right say like, Okay, that worked. Everything works, you know, there's no compilation errors. Well now we can, we can say well this could be a release candidate or this could be our alpha or beta. And so you can tag it, you can store it, you can think about it, you can do other things to it but essentially you're integrating pieces of code into this artifact that's ready to be deployed and run on a computer or some type of infrastructure. And that's where continuous delivery picks up right. We have this artifact that's ready to be put on a server. So, continuous delivery, I'll actually say well, what environments are we going to deploy to, what infrastructure are we going to deploy to, is this going to go to AWS, a server EC2 instance on AWS. Is it going to go on Kubernetes? Is it going to go on, you know what kind of environment is going to go on. And one of the nice things is that we can actually automate this process, you know we can spin up new environments and just deploy on demand. And so a good part of continuous delivery is about ensuring that we have the right environments and that we can get this artifact into those environments and that we can manage this change, understand it, right. And then employ some deployment strategies so figuring out well what kind of release strategy do I want to have if it has a big impact, if this feature change has a big impact. You know, maybe I only want to release it to 10% of my customers first just in case there is some type of danger or risk associated with delivering a big change. Or maybe I want to introduce verification methods right once I've deployed an application service I want to make sure that I'm logging specific events. I can detect errors if there are errors and then roll back if there are any incidents you know just undo the deployment and go back to either go back to a new to a working state, or even just roll forward and and fix that mistake and then put it out for our customers quickly right. And so there's kind of a full proof method here that goes to say that that says like, Okay, even if we make a mistake there's there's some way that we can remedy it or there's some way that we can minimize it. That's what the stimulus delivery is about. And, and so today we're also seeing like, there's a lot more of an ecosystem around supporting this right, especially as it pertains to leveraging solutions like infrastructure is code to provision quickly vaults KMS cyber arc to protect our secrets. So like, even just general metrics like DevOps metrics and understanding for CI CD that allow us to continuously improve and track like how are we doing in our CD process and our CD process. And then there's even things like well pipeline management how who has access to particular environments who has access to particular pipelines that deploy to particular environments right. So maybe it doesn't, maybe you know the new person joining doesn't need to have access to that on their first day, or, or maybe certain teams don't need to have access to other teams resources and so it's just something all to think about here that in in the future for delivery, there's actually already a lot going on right. And so if we can build into our foundation of CI CD then we can actually achieve governance in a better way. And so now that we understand more about CI CD, I want to actually talk about governance and what it means, because I think when we talk about it governance specifically it's slightly different and it may encompass more than a typical conversation about governance or a typical conversation around standards. And I really think of it as kind of three areas that we think about when we think about it governance. We're thinking about standards, we're thinking about security, and then we're also thinking about audits. So, there's this umbrella term for government IT governance that covers three, three of these areas, and it's called GRC governance risk and compliance. And so when we talk about governance, or that's where we're talking about our standards right. So, just like we would set standards for a state of emergency in the US, right. So what happens when there is a earthquake, or what happens when there is a hurricane right certain specific specific groups of people are set up specific camps are set up, you know, some people show up. We have these things that are set in place notifications are sent out. And so that's what governance is and not shall right reset we're kind of preparing and setting up systems for people to be able to expect the unexpected. And then also setting up like the communication channels to deal with that. And so, governance is about setting these standards for what we want to do when we don't know what to do or we can't predict unpredictable. And so, the reason why we do that is so that we can oversee specific goals, achieve certain specific goals, and also maintain everything that we have in with going on in our organizations. So, you know, things like the solar winds hack happening to solar winds that's, that's a really big hit right so what, what's communication channels that they have, when they found out, you know, what were the different systems that they had to be able to look more deeply into the, these, the situation. That's what governance is trying to discover. And then we want to cover risk. The second part around risk is about security, right. So being able to identify potential risky behavior when we're delivering software, you know what is, you know, like, what happened in solar winds was that there was some people who had some risks, right and that could be a certain level that could introduce a certain level of risk to your organization and so GRC the risk part of GRC is about rating and prioritizing risks, and ensuring that people are aware of them and that we can mitigate them in the future so that we do have systems in place to say like, hey, our systems are not up to date or hey we have some configuration drift going on in our infrastructure. This is how we this is the actions that we need to take. So risk is really this kind of action plan around how we identify how we can potentially secure better secure systems. I think people understand compliance pretty well, because it's similar to how we work in the real world, right, our healthcare entities are our standards for medication or practices is what compliance means right there's a certain level of expectations around vaccines, a certain level of expectations around how drugs should work or how they should be taken right, and so the drug manufacturers of vaccine manufacturers have to comply with those standards if there are standards right. And so compliance is about meeting expectations, and being able to prove that you meet those expectations, because it's a science right. We don't want someone to just say like can we be yeah works, you know this vaccine is going to cure everybody when it doesn't, and we need to know the percentage of effectiveness. And so that means being able to document and log exactly how things were built, when with it built why would they build what are the features, what's going on to what's going on with our software services, and even being able to do it to the extent of full keeping any expectations that we may have in the future. So as more current events happen, as we change how we think about practices, our expectations are going to change as well. So we've changed how we how we perceive vaccines, and our expectations around how, how successful and how, how they perform work. And so maybe the compliance, the compliance levels and how we how we deal with even data personal information those things will happen. And so we need to be able to improve continuously improve and also have the systems to be full proof in the future. And so this is GRC and it governance. And it's really about how you monitor and control your it capabilities and decisions, when you're trying to deliver. And when we talk about DevOps when we talk about delivering software really talking talking about delivering value to key stakeholders right to our customers to the, to the target audience of our software services. So there's two parts here that I want to break down further. Right. This concept of being able to monitor, and this concept of around being able to control, because we need both of these things, right we need to be able to control a certain level of risk, we need to be able to control a certain level of risk. Same thing with compliance. Right. We need to build these into our CICD systems. And so this is the last part of the presentation that I want to cover. And it goes into details about how you can build particular. Yeah, so we had a question about do you get the slides. But I'll, I'll, I have a copy of them. So I can, I can send that out as well. Yeah. And, and again, yeah, like, you're not going to be able to introduce all these changes all at once right, and it'll go back to saying like well what can we change or what what areas do we want to change right. And so it'll be interesting to go over some of these practices and do them in a way where it makes sense for you and your part of the software development lifecycle. Again, like delivering software is not just one person's responsibility, security, code quality performance that's not just one person's responsibility right these are cross functional teams these are different groups of people who are working together to deliver something And so you may take some of these practices with a grain of salt saying like hey I have nothing to do with the system or I'm not the person who makes decisions, but I do make decisions in how we develop code, how we do it with secure practices. So this is, these are the steps I'm going to take to do that. And so that's why I broke this down in, in different parts of the software development lifecycle. So that you can kind of pick and choose where you can make that change and where you can make impact. So this is the build phase, and then the build phase, this is earlier on right this is when you have a feature developed. And now you you're just ready to go forward right you're, it's the end of the sprint. You can integrate it to the rest of the code base, potentially it could, this could be released this could be a release candidate this could be, you know, the change that's going to go out to customers. It could not be could just be one step towards that. Right, whatever it may happen to be in this build phase that we trigger. We have some code and we're ready to go. So what can you do in this process. There's a couple of things that I really recommend. The first one is just static code analysis. So this is a screenshot of something called sonar cube. They have an open source version I'm pretty sure as well. So what happens is, these tools will look at your code, your source code, and, and analyze it for things like potential vulnerabilities. So if you're doing things like hard coding, any sensitive information or infrastructure information into your application code. If you are potentially using secrets and it like doesn't look right. In terms of how you're using libraries or any of your dependencies. It'll actually tell you and flag those and actually give you a grade to so in here like you pass the quality gates, but you have 11 vulnerabilities and you got a score of a D and then you have some bugs, and they ready to see. So if you kind of look at that they also give you things like code smells so potentially like some inclinations that your code may not be as robust or as elegant as you may want it to be. And these are really great because it doesn't only look at your code that you wrote, but also like other people's code, the entire source code, the code base entire code base itself. Some other things that people will do in the build phase will also be doing things like scanning for secrets. So, you know, your secrets lie in various places of your application code so a nice thing to do is also like run, run scanners on the packaged artifact that you have and be able to detect like, do we have vulnerabilities or not. And the reason why you want to do this is not only because you may not trust your code or someone else it's not really about that right it's not really about that you don't trust the code that you write, or you don't trust your team. But it's it's a sense that we use more than just the code that we write. So there was actually a study that looked at the most popular applications in the enterprise market today, you know the code that you use every day. And they actually noticed that 96% of the most popular applications currently use open source software. There was another study that said, and this was all written in the article. And so in the same article, they said that there is more than 4000 security vulnerabilities discovered in open source projects a year. And so that dependency that you may be using, or that new runtime of Python may actually be compromised. And so when we build our systems to be able to detect these kinds of mistakes ahead of time, then we don't have to worry if then we don't have to worry as much about. Well, you know what happens if there is security vulnerability that happens like, like it did in the solar winds hack. No question here. And what phase of the development cycle. Do you monitor and control. I was thinking it would be done after releasing the software. Yeah, so you definitely monitor and control your applications after you deploy right after you've done a production release people will, you know, monitor how the application is working maybe for the next 24 hours you know if there are any major people will be on call, and that kind of thing. But I think it's really important to build some of those checks earlier on. Right. So like that solar solar cube program was the one that checks for vulnerabilities. That was another question. Like being able to have just something as simple as a scan and and like, sometimes it only takes like a, like two minutes to scan right and be able to detect. Hey, there's a, there's a vulnerability or be able to monitor potential issues, right and control them. And I kind of think of governance as a set of controls at the end of the day, at least CI CD governance, because imagine having a CI pipeline with no tests. There's no way that you know that what you integrated works, aside from the fact that it compiles. Right. And so a form a form of control can also be quality gates. And I'll actually talk about that a little bit more, because there are a couple organizations that are doing automated CI CD governance. So they kind of listed out like how they set monitoring and controlling capabilities in their CI CD pipelines. So that's a really great question. Thanks everyone. So, um, where was I. Yes, so there are issues in in codes, there's vulnerabilities in code, and they can pretty much happen everywhere. And so that was kind of the case for Microsoft, right? This was actually a blog post they had written after they noticed after the SolarWinds hack had been announced and they were investigating it. So they actually wrote a blog post trying to figure out, well, what exactly happened and why were they compromised just because they were the ones that had used SolarWinds and they had to look, you know, they had to comply and say like hey, you know, we need to figure out who else was impacted because we were impacted. And again, it was this daisy chain, right? Effect. And so actually, you know, I don't have to pay too much mind to the screen cap, but because I'll review it, but essentially what happened was there was a library DLL component within SolarWinds that was, that was hacked, because that code base was in a GitHub repository that had an exposed secret. And so what happened was a attacker injected code into that library. And that library, when it ran executed, it would actually initiate a backdoor server that would send any information that was going through SolarWinds to that backdoor service. And so I actually have a screenshot of what Microsoft investigated and noticed was that in the original function, there was actually an injected function here that started up a server, right? And this was the backdoor that any, any information that went through would send it over. So this is what happened in the SolarWinds hack. And I think a lot of different, a lot of different software services and just organizations that were a part of this process, the process for delivering software were implicated. For example, SolarWinds was using a CI tool called TeamCity. So now the government doesn't want to use TeamCity, doesn't want to use JetBrains or IntelliJs. I think it's JetBrains. Yeah, JetBrains product anymore. Because it led to a vulnerability, right? I'm not sure what SolarWinds, this one is Java. So we had a question that asked if SolarWinds was built on Python. This one is Java. Great question. So this just goes to say like, you know, you're, even though the CI, TeamCity, there is, you know, there was no compromise in TeamCity, it still led to a compromise, right? It still built that code base and shipped it out. And so at any kind of process, at any point in our delivery process, we can get, we can get our secrets or our important information compromised. So it can be as simple as a plain text password and a GitHub repository or a BitBucket repository. Or it can be a CI server that's using a secret, but it's not encrypted. It's just plain text secret. It could also be artifact repository or even an orchestrator that's using plain text. They can look into, they can hack a particular node in a particular environment, right? And gain access to your application. And so there's application secrets that live everywhere. And we need to be able to acknowledge that and be foolproof to even just simple mistakes is looking things in plain text. One thing that I recommend for people who have CI CD pipelines, who are creating resources, creating different resources for their application teams to be able to deliver is just to introduce something as simple as well base access control, right? This can be based on user groups, right? It can be based off of your organizational layout, right, with team you're under. And then you can apply it to different parts of your software delivery process. You can apply it to particular environments. You can apply it for particular types of deployments. And you can say like, hey, Team A doesn't need to have permissions to control, create update anything related to Team B's applications. They shouldn't even be able to execute a pipeline or half of a pipeline for another team's applications, right? So you need to be able to do that as well. And that's actually something that SolarWinds did not do well. And I think you can tell based off of some of the reports and I'm just trying to investigate like what happened. You know, and at one point I think they tried to blame an intern saying like, yeah, this person, it's like, and I think one of the actions that they took earlier on was just also they said they had released a report saying like, yeah, we took action and now we're limiting who has access to all of these resources. And I mean, they should have done that earlier, right? And so that's something to note is just that you have different resources, you have different things like different even just views like an admin view, a billing view, a security view. Maybe no, maybe only one or two people need access to that, not the whole organization. So this is this is about role based access control for our software delivery. All right, so I want to get to this test phase. We mentioned earlier about quality gates. And even thinking about it from the perspective of okay, well, our application code like we've run unit tests now we know that functions work as expected. You know, have some type of performance meet some type of performance criteria, etc. Well, how do they work with other services if they're in a microservices architecture or if they're in a coupled architecture. How do they work. And so there's a couple of different ways that you can test right, you can test white box from the white box or black box so black box testing. You know, you don't really you, you write the tests with the assumption that you don't know what the implementations are. Right, you don't know what language it's coded and you just, you just create those unit tests and white box testing, you want to ensure that like certain particular major parts of the code base or particular parts of the code base are tested. Right, that we, we know that this works without a beyond the doubt. And that's important right if you're going to reuse a particular part of code. Or if you're you want to make sure that one part of code never changes. And my box, my box unit testing can be really helpful and then automating this process is also a really big thing. Right, so creating test suites, ensuring that we have code coverage. I've seen so many times when people have a pipeline, a bill pipeline, and they don't keep up with the unit tests. So, after a while they just turn off unit tests, and they have the CI pipeline that doesn't have any tests and it's like, how do you, how do you know that it actually works. Do you know that actually works. You don't want to wait until you find out a customer finds out right. So this is one good example of how people will ensure that they have governance. So let's say like, okay, one of my controls is this set of unit tests. And so my as a station, or the fact that I know that this works, which is what I need to say in governance right is that I know I succeed my CI pipeline when all tests have been executed and they pass. I want you to have a CI CD pipeline and just say well, you know, half our tests pass. So we're just going to go and deploy. Right. What kinds of standards are you trying to meet. And so, if it is like, if, if that is how you want to govern your software services and you say like only 50% have to pass. Okay, that's what your association is. But again, like, we need to be able to say some of these things ahead of time right. And the other one is just clean dependencies right and as the station might be all dependencies in this build are free of known security defects. So we've run our tests, you know, if it works or not. So these are just examples of ways that you can set controls, and then monitor. Hey, do I have this working or not. You know, and so our CI pipeline can be our monitor. You know our dashboard that says pass or fail. See I pipeline pass or fail. So we know, and then it has a little report that says all tests executed and past right. Our unit test suites that we have to unit five. I'm not sure what it is in Python but right, we have, we have units, we have unit test suites that say like, yeah, I'll test pass. That that's a necessary station that's a way of monitoring do I have, did I achieve the results that I wanted to result to achieve. Did I did I achieve the results that I wanted, right. All right, so I think this is the last part. Yes, this is the last part last kind of section for practices that you can use to employ governance. And this isn't a deployment phase. So we mentioned earlier, like, you know we have this artifact that we're going to deploy into a service right, we're going to deploy into a server. Now we have to provision configure and then deliver it right. So, there was a couple practices that you want to have here. The first thing is around being able to verify and test your configurations work as they should. So you need to be able to document and figure out well. What is this particular server configured with things I need to configure it with, you know if it's running engine next is that configured. What is the password that you're using. Right. If it's admin 123, it may be a security vulnerability. And then doing things for your application itself. So there's kind of two fold, looking at your infrastructure, and then looking at your actual applications to doing things like penetration testing system tests, vulnerability scanning, while it's running. Which platform will you deploy. Right. There are different ways to test your configurations for different platforms for different cloud providers. We have a question here. Suppose we have an internal gateway which inspects inbound and outbound traffic. If we already deployed such production, do we need extra CI CD production, despite static code analysis and tests. That's a good question. Yeah, I think it depends on your actual software service. Right. Because if yours is not running. It's not like a service that other people consume as like a UI or something that's running. Right. If it's like an API. Let's get that gets compromised because you don't have static code analysis or test built in, or you know it gets malicious code injected and you wouldn't be able to find out through your CI CD pipeline. That could be a really big issue. I think it really depends on your software service. I think it's good to have. Again, it's like the sense of governance like it's good to have some means and some practices or at least some like tool kits in your toolbox right to say like well, you know now we have this very successful application. You know there's another team that we're going to spin up or there's another service that we're going to spin up that offers something else right. The use case is different and the way that people consume it is different. And so now the standards that we want to have may be different. So there's something to note there. Really awesome question. It's a top down. It's a top management responsibility of the organization for sustainable development. Oh for sure. Yeah, I definitely think that. In DevOps all the time it's just like, we need great DevOps leadership, we need people who leadership, who will say yes, we will support this will give you the tools that you need will give you the training to use those tools right it's people process technology. Doesn't work when it's just developers saying it, you know, yes, we want to have bottom up but we also want to have top down. Got an anonymous question on, will you use all kinds of orchestrators. It's a little bit of a confusing question. Yeah, you can use different kinds of CI CD tools you can use different kinds of ecosystems right different sets of tools. I'm giving really general practices and tooling that you can use. It doesn't matter what runtime you have doesn't matter what platforms you're using. So, I hope that helps get kind of like two questions around platforms and orchestrators I hope that answers both of them. Thanks for the questions. And one other thing that's not exactly built into our CI CD pipelines, but can be related to our CI CD pipelines is just this idea of being able to review, and you only review during this process or in particular parts of our process. Exactly what we're trying to deploy what where when and why and CI CD can actually help with this process right because our CI CD pipelines are taking our code from throughout all these processes right from build to test to deploy to verify and then to roll back if we need it. And so you can actually have a group of people that you really trust and they understand like all the rules and understand how application service or service works, and this can be a group of, you know, architects can be that also includes the product owner, you know, and this during that CI CD process, you know, when you're filling out a JIRA ticket or saying like hey this, this deployment to a non production environment worked. Can we review it for a production deployment. That's what a change control board can actually do. And so having a change control board probably could have helped the solar winds. It could help prevent the solar winds hack right if they had a control board, a change advisory board that could see how certain pieces of code had changed, even if they had done a code review or just like looked into the source code what what exactly was built in. They'd be able to stop that they will say hey this is this looks weird that injected code that looks weird that's not part of something that we developed. That's not a change that we had planned to make. So I kind of like any point of the software development lifecycle they could just say like nope, we're not going to, we're going to review that before we deploy, or I'm just going to, we're not going to do this deployment this deployment fails. So being able to say at any point like to control your CI CD pipeline say like no, we choose when this deploy when this goes out. Right. It's really big. Most organizations don't employ continuous deployment continuous deployment is when you can make code and it goes straight to production goes straight to your customer. Most organizations don't do that they don't need to do that. They don't even want to do that they want to pick how often do they release is it weekly is it monthly, they want to be able to do it on demand. And that's what continuous delivery helps us to do. All right. This is a screenshot of the harness platform, because I think it's really important to be able to detect things that happen to our deployment configuration changes that happen for both our deployments and for our infrastructure. I mentioned earlier, like it's super important to be able to say like, hey, what, what, you know what version of the service is this infrastructure running, you know, what version of Apache, what version of engine X, you know what version of Jenkins is running, you'll be able to track that right. We need to be able to do that with our service deployment as well. You know when we're scripting Jenkins pipelines, or we're scripting our CI CD pipelines, we need to be able to see like, what happened, we need to be able to tell like if something changes, right. And so being able to see this at a high level and being able to say like, hey, this deployment looked really weird and then be able to go to that appointment and compare it to a previous deployment and see what the change was, is really important, right. You can be able to say like, oh well crap, maybe the reason why we were not work, we're not, you know, there's a failure in our latest deployment is because we deleted our configuration variables. And so we're not actually passing in encrypted or encrypted secret for our database. All right, I'm running out of time so I am going to speed through last few slides here. Same thing with audit trails right being able to say who, when, where, why, and what's happening within our software delivery processes this is the hardest audit trail, which is this is a screenshot of like different things that had happened in the platform. So I created, if you look at the bottom, I created an artifact stream. I did someone else deleted it, I updated a serverless specification, I updated the workflow, and I know who did it. Right, I know when they did it. I know what IP address they came from. And I know what services were impacted. That's really important. So things like this will give you a better sense of like, who deployed when, where, why, when, when, where, why, losing my mind to our environments. And we need to be able to do that, especially if we're going to say like, hey, we're compliant or maybe we do audits at the end of every year, some organizations, some industries have to do that. And so being able to do that or have a record of that is really important as well. And that's what we're going to see in the future is just more deliberation and more intention towards what we deliver and why we delivered and when we deliver it. And so I want to close off by just sharing like where you can find governance in the wild because this is pretty, pretty interesting. We have two minutes left so I'll try to try to get this done. But Capital One had actually shared in a reference architecture white paper on automated pipeline governance. That's what it's called. It's by it revolution the same company that gave you the DevOps handbook and accelerate. There's a there's a white paper that John Willis helped co-author. And they, there are a bunch of other organizations that had contributed, but they shared one way that they were employing governance and they called it 16 gates. And this was the, this was the way this was how they were introducing this. And, you know, they have a bunch of practices here that we mentioned, but I really picked out the ones that made sense for anyone to be able to use right. So things like source code, version control, static code analysis, vulnerability scanning, penetration testing that I mentioned integration testing, even they were doing more interesting things around controlling who got to see specific features with feature toggling or feature flag management. That's also another great thing to employ. Right. I think a lot of people, a lot of harness customers actually doing this today as well, which is what I wanted to mention that it does exist people are doing automated pipeline governance, even though it may sometimes feel like this crazy thing when we're still trying to figure out how to really automate our governance. So I just want to close out because I know we're at the top of the 50 minute mark here and say like, automated pipeline governance is about controlling understanding and mitigating risk. Right. And even though you can't do these things all at once like snappy fingers so you have everything we can get there through increments, we can continuously improve. So I hope this session was really helpful in doing that. I'll stick around and answer any questions because I know we had a few more, but I really appreciate the time that you all spent with me. Thank you so much. If you have any questions feel free to reach out. I'm all over the social medias and you can find me wherever. But thank you all so much. Thank you so much to Tiffany for her time today and thank you to all the participants who joined us. As a reminder this recording will be on the Linux Foundation YouTube page later today, and we hope you're able to join us for future webinars. Have a wonderful day. Thank you.