 If you can measure something, you can improve it and nowhere is that more true than your DevOps journey, but where do you start? In this next session, Bobby Wensler and Nicole Schultz will tell you exactly where Northwestern Mutual started and why. Let's check it out. Hi, everyone. Welcome to GitLab Commit. My name is Nicole Schultz and I have my product manager Bobby Wensler with me today, and we are going to be presenting on how we are leading DevOps transformation with GitLab data at Northwestern Mutual. For those of you who haven't heard of Northwestern Mutual, we are a financial services and planning company. So everything you need from life insurance to investments to financial planning, NM can help you with it. We have two office locations, one in Milwaukee, Wisconsin, and another one in Greenwich Village, New York. We have a fairly large tech organization with a wide variety of technology. So if that sounds interesting to you, check us online, find us on LinkedIn. We'd love to connect with you. Let's do some quick introductions. Like I said, I'm Nicole Schultz and I'm really passionate about helping application teams move the needle in terms of CICD and DevOps. In my current role, I manage an amazing team of engineers and we create tools and capabilities to help mature and evangelize DevOps and CICD across the organization, as well as enable access and visibility to DevOps metrics. Bobby? My name is Bobby Wensler. I am the product manager for Nicole's team. I'm always up for a challenge and I really enjoy partnering with the engineering team and the clients that we have to bring solutions to life. So I'm going to talk a little bit about our agenda today. First, we're going to cover our problem. Then we're going to cover the industry research we did on that problem. Then we'll talk about the industry findings that we applied at Northwestern Mutual, talk about our solution, the CICD pipeline certification, and last talk about our outcomes and learnings. So first I wanted to talk about our problem and that is there's so much data but there's not a lot of visibility in action. So as you can see here, this is the standard pipeline that we have at Northwestern Mutual. It's got a lot of stages and jobs included, which is great. You're going to get a lot of really fast feedback from your pipeline. But the challenge becomes, as you look at this screen, so as you can see our problem is there's so much data and so much data and so little visibility in action. So from that one pipeline, you can see there's many different tools. They all output their information a little bit differently. It's really hard to tell how you're trending over time. It's hard for our organization to see, are we doing a good job? Are we getting worse? Where are we going with our CICD and DevOps rest practices? So as we looked at this, we knew we wanted to learn from what else was going on out in the industry. So we looked at this book called Accelerate and we had a lot of really great learnings I wanted to share a few with you. The first was there are a lot of gaps with maturity models. So when you look at maturity models, it's often a one size fits all, very static, never changing set of criteria that apply across the board. The other thing we learned is that quality automation gives you something great. You get speed and stability. You're not having to sacrifice one or the other. To go faster, you're not giving up stability. To get stability, you're not going slower. It's really the power of having great automation in your pipelines. Next, the ICD and SEM are supporting capabilities for the door metrics. So this means if you want to really get the sustained door metrics in your organization, you've got to have some good source code management and CI-CD best practices. If you're not sure what the door metrics are, there's another presentation at GetLevCommit actually being done by Google, would highly recommend you go listen into that. The last point was there's lightmate change management process is really important. And what this means is they found studies showed that if you had an external team approving your changes, you actually didn't get better stability at all. So we found this very intriguing and wanted to learn more about that as well. The next piece we looked at was the state of DevOps report 2019. The first finding there is the best keep getting better. If you're falling behind, you're only going to keep falling behind. You really need to start making progress now to be able to kind of catch up to what the industry is doing. The next is automate and integrate. Again, by having that robust pipeline, we saw that the automation for the high performers was way above the categories of all other performance groups. So it's really important to actually automate and build out things so you don't have to keep doing the same processes manually over and over. Community is a practice. So like many big organizations, things are going to change over time. There might be reorgs and restructures, but if you've got a solid community of practice there to bolster and kind of build up and have a grassroots foundation for your CIC and SCM, you're going to keep having that sustained community out there championing, championing, championing. Try again. So I'll go back to automate and integrate. So it's important that you continue to automate and integrate. And why this is, the book showed that basically the teams that are the highest performers have the most integration and automation built into their pipelines. The second is communities of practice. We want to make sure that we've got a good community of practice. So as things reorg and change in one company, there's still our champions out there for CICD and DevOps that doesn't, it doesn't go away if the people that were tied to it are there. Start with foundations. So this is really important. There's a lot of great practices and learnings in both of these industry researches, but you don't want to do it all at once. You want to kind of break it off just like you would in Agile and break it into small pieces and make progress over time. So then the next piece I'll talk about is how we applied the industry findings at Northwestern Mutual. We're not ready for the Dora metrics yet. So what I mean by that is we want to really start with those SEM and CICD best practices and get those off the ground. So teams have somewhere to start. If we were to tell teams, you need to have more deployments right now. How would they know how to go about that? We wanted to make sure we had the steps in place to really set them up to be successful. We wanted to address the maturity model gaps. So our criteria doesn't say static. It definitely also doesn't fit one size fits all models. We wanted to really make sure that it was flexible to support the necessary tools and technologies that were in our company. And the last piece is increased voice of customer. So it's really important that people understand why you're doing this and what they stand to benefit from it and they feel like they're part of the process. We really felt we wanted to make sure people were involved. So it didn't feel like a mandate where you must do this. It felt more of like a partnership where everyone understood what they were gonna get out of this tool. And next I'm gonna turn it over to Nicole to talk about our solution. Thanks, Bobby. So to address the gaps that Bobby just covered, we developed something we call the CICD pipeline certification, which is essentially an automated process that evaluates our repository and GitLab and gives it a maturity rating of how well it's doing in terms of predefined CICD criteria. So we actually started with a version, one of this last year called the CICD assessment and we got really positive voice of the customer feedback on it, application teams really liked it. And at the time, the assessment was running on all of the repositories we have in GitLab, which is a lot. And something that we learned is that in order to really support this and have this work, we need to create project types. So as we were looking and doing our industry research and developing kind of a new version of this assessment we had, we decided what we really need project types that we can have specific criteria for specific project types. So projects in GitLab, they have different languages which are gonna have specific needs. So for example, a Docker repository, one of the criteria we have for that is that you need to have security scanning in your pipeline. However, for a database repository, that container scanning is not gonna need to be there and there's gonna be a different tools that are used for the database repositories. So as we were thinking about how to develop our certification, one of the requirements we decided on was that we needed to be able to support project types with criteria that would apply to specific projects as well as have criteria that we knew would apply to all project types. We decided to start with just evaluating AWS Docker projects. That's really because for our cloud platform that we have, we just have more standardizations in place and set up. So it would be a lot easier for us to automatically grab all the data we needed to check each criteria using just projects that have Docker. So our maturity, our CSC pipeline certification has three maturity levels, bronze, silver, and gold. So a little bit of gamification there. Each one has a theme. So the theme for bronze is project and pipeline basics. As you can see on the screen here with all the criteria listed, we have your default branch protected. You don't wanna have people directly pushing to master. We want our source code to be tied to an asset. For us, that's really how we do ownership. So when we're tying source codes to assets, source code to assets, then we can correctly know which team owns that repository and that code and should meet the contact for any questions we have. A lot of the things on the screen here might seem really basic to you, but as Bobby pointed out, and as I said in the intro, we have a really large tech organization and everyone's on in a different place in their DevOps journey. Some teams that are more advanced, yes, these criteria are gonna be done on day one. They're gonna have them already when we roll this out. We knew that, but there are also teams that really are just beginning their CIC and DevOps journey and might not know where to start and might not have a lot of time or bandwidth to really think about what they should be doing in terms of that. So we wanted to make sure that our certification and the bronze criteria were project basics, things that could apply to everyone across the board. Our theme for silver is robust pipelines with fast feedback. So this maturity level is really focused on adding in all the different types of scanning capabilities that we have. So linting, code quality scanning, security scanning. We also have unit test coverage added in here. Unit test coverage as a whole is still something that's being adopted across our organization. So you can see for silver right now, we just have the percentage set at 20% or more. We definitely plan on looking at increasing that in the future and adding in more types of testing. In silver, we also have the fact that you're pushing your artifact to registries so that you can use it later for deployment and for production deployments, you are utilizing our change management integration. The theme for gold is addressing fast feedback and having automated builds and deployments for your default branch. So addressing fast feedback that really means that your pipeline is gonna stop when previous jobs fail. So the point is if you have security scanning in your pipeline, that should stop, right? If it fails, it should not be allowed to fail and have your pipeline keep progressing. That should stop so that you address the fast feedback, you fix any vulnerabilities that are found, you push the fix, you rerun your pipeline and then it succeeds and pushes through into production. You obviously definitely wanna have automated builds and automated deployments. And that's, you know, both signs that a team is fairly advanced in the CISD space, you know, you're continuously building your change, you're continuously deploying that most likely means you're hopefully deploying small iterative changes, you know, into production to minimize risk. Another thing we have here is the unit test coverage being at 50%. Again, we needed to just start somewhere and that's something that we'll be talking about how we're gonna be looking to increase that in the future. So future changes, I talked about a couple of them. One of them at the bottom, increasing unit test coverage, right, so as teams progress and hopefully add more and more unit tests, we wanna definitely increase those percentages to be higher in silver and gold to challenge people again to get better. We also would like to have a way to verify automatic rollbacks to see if a team has an automated rollback, that's definitely something that's a sign of maturity, that they have a way to actually rollback their application automatically. We would also like to add in a criteria saying that you should be using a Northwestern Mutual approved base Docker image. That's really just because we have base images that we do nightly patching and security on. So we really want teams to be using those for all of their Docker builds. We also wanna add additional testing types into the pipeline. So right now we just have unit tests because again, with all these criteria, we need to be able to get the data automatically and right now unit test coverage was the easiest piece to get in an automated fashion. We definitely are working with our testing organization and partnering with them to understand what other types of testing can people use in the pipeline and how can we get that data automatically and even beyond that, what are the correct threshold that should be set for silver and gold in terms of what good looks like for those types of testing. Lastly, we would like to create an exception process that allows teams to override things and when they're doing a hot fix. So we all know that hot fixes come in in production, things maybe have to go in right away. Maybe you don't have the luxury of waiting for your entire pipeline to execute or you need to allow a certain step to fail just so that you can get the fix in a timely manner. So we definitely wanna figure out what that looks like. Obviously have some sort of audit trail, et cetera, but something that customers have asked for that we'll be looking into. So I'm gonna turn it back over to Bobby now and she's gonna be walking you through the UI of our application. Yeah, so this is it. This is the pipeline certification and this is actually the report that will display for an individual project. So as you can see here, GetLab project is what the project that we're looking at. You can see bronze, we met all the criteria that Nicole talked us through. Silver is where it's gonna start us automatically expanded because we've got some criteria left to do here. One thing that we talked about was the need for upcoming feedback. So here you can see this last feedback line is gonna give you a lead time to make sure that teams have some runway to actually get out there and do these changes. So this is a feature that our customers ask for. So one day if we ask for some improvement they don't necessarily drop. So this gets to the point I made about our criteria aren't static. You can also see here clearly where things aren't met. So they still have some work to do on this particular criteria. So that's what they need to do before their pipeline will go to silver. We also have gold. So it's not expanded again because right away we want teams to focus on silver and kind of iterate through those steps. On the left hand side, we give you a lot of information about the project where we certify it as what the project is. Some of those details Nicole was telling you about how to tie it back to people. And then we also added in a few other really awesome features for our customers. The first is this project out button. So there could be teams who are sun setting something in six months. Do they wanna spend time on that project or when they're touching every day that's gonna be around for a while? They might not. So they can then opt that project out and say this project is gonna be sunset. I don't wanna focus on it right now. We really wanted to empower teams to focus on what made sense for their organization and not necessarily dictate. Everything has to be done and let the teams focus on what metrics they wanted to set. The next button is recertify. This is really important in brands. Well, you can't see all the settings. A lot of those settings are in the GetLab project themselves. Teams really loved this pipeline assessment, but like their pipeline, they wanted fast feedback. They did not wanna wait overnight for things to run. So this recertify project button will go out and get the latest and greatest GetLab settings and then update them to tell them where they are now. The next part of the pipeline certification is our dashboard. So out here we give the organization a rundown of how we're doing percentage-wise and project-count-wise for the types of projects we're certifying right now. So if you're an executive, you could come out here and say, for product A, I'm at this count gold, this count silver, bronze. My team zapped it out this many. There's also another piece of this dashboard that shows you by assets as well. So for a particular asset, where is that asset health at from a CICD pipeline perspective? And then we also show trending. So over time, do we see gold going up like we expect? Do we see silver going up? Those types of things as well. And now I'm gonna turn it over to Nicole. You're probably really excited to hear more about how we actually did this from a technical perspective. Thanks, Bobby. So this is our high-level architectural diagram on how we collect all of the necessary data for our CICD pipeline certification. As you can see on the left-hand side, it all starts with GitLab. So shout out to GitLab for having such a great API and tool that it'll always take everything we need from the application and all the repositories in it. So we utilize a system hook in GitLab. What a commit is triggered, that triggers a system hook for us, which puts the name of the project into an SQS queue that we have. That triggers or we'll get read by a Lambda and we basically have a repo metadata Lambda, we call it, which collects metadata about the repository. So lots of general information, it collects the default branch, the asset, some asset information, contributors, all kinds of different stuff, a lot of metadata information about the repository in general that we use. Once that Lambda is complete, that publishes to an SNS topic. We then have our certification Lambda, which actually reads from that SNS topic and gets the name of the project. So you can see the first Lambda that runs, the repo metadata one, that stores the data in a Dynamo database, which is accessible by a DevOps API that we have. So when our certification Lambda is running, we actually pull some of the metadata from the DevOps API to grab the asset information. You know, like I said, the default project, we pull some of the settings, like do you have your default branch protected, do you have merge request pulled on, we pull some of the criteria information from the DevOps API, and then the certification Lambda does a lot of the calculation on logic around, you know, for all the criteria we have for Brown-Silver Gold, is this met, true, false, is the criteria upcoming or not, is it active? And then it also will calculate the percentages that Bobby showed on the UI of like six of seven, what percentage is that, et cetera, for each maturity level. So after the Lambda is complete, it stores the results in another database we have just for the certification, and then that database is read by our certification API, which is our middle tier, and then that eventually is what feeds the UI, which Bobby showed, which is our front end for the certification. So let's talk about some outcomes and lessons learned. So one thing we learned, as I mentioned before, we kind of had this version one called the assessment, and that was collecting data on a nightly basis. So one thing that we learned was, people don't wanna wait overnight to see if the changes they made in their repository got them to Brown-Silver or Gold. So we, definitely with the new architecture I just showed you, we're able to collect it in near real time. For some of the data that we don't have set up real time right now, such as on this GitLab settings page. So when you go into GitLab, your project, and you go to the settings page and make some changes right now, we don't have a way to automatically trigger, have those changes trigger our data collection process. So that's where the recertify button comes in that Bobby talked about. So for Brown specifically, when people are making changes on their settings, they can go in, make those changes, and then go to our UI, click recertify, and that will basically put the name of that project into the SQS queue and run it through our entire process. As I said, if people are making changes in their GitLab CI file and their pipeline, that's gonna trigger a commit, which will automatically trigger our process. So the data collection from our side is much, much faster now, and teams can get near real time data collection and feedback when they're making changes to meet a maturity level. As Bobby talked about, people also really wanted the opt out. As teams started setting more OKRs or team goals around this, they wanted to say, hey, we wanna get 100% of our projects to bronze. Okay, well, for these 10 projects, we're sun setting them in six months, it doesn't make sense. So that's really where people wanted the opt out feature because they wanted to be able to hit team goal numbers without having to actually work on repositories where it doesn't make sense. The other thing we heard was automation. So a lot of teams have a lot of repos. In some cases, teams might have 100 or more. So they don't really necessarily want to go through all 100 repos and click four or five check boxes to meet the bronze criteria to protect their branch, turn on MRs, all that kind of stuff. So something we're actually working on right now is figuring out a way to automate that. So on our UI, adding in a button or some options to say, hey, for this group, these projects, automate these criteria, we're starting with just a simple criteria, right? We're not gonna be starting by making changes to get led CI pipeline files and stuff like that. But a lot of the bronze criteria, which are just updating settings on a GitLab project, we can automate a lot of that for the application teams. The other thing we heard was, teams wanted time for improvements. And that really gets to the point that Bobby was making, we have the upcoming criteria so teams can adjust. And in general, the fact that we can show all this data helps teams give more of a view to their leadership to say, hey, we want to make changes. We want to get to bronze or silver. These are the benefits that it gets us. And this is the steps we need to take to get there. So it actually helped them get more leadership buy in because we were showing them the data in a way that it was easy for them to show their leadership. So they could get more time to make improvements on their side to start using and adhering to some of these CISD best practices. Bobby? Yeah, thanks, Nicole. So I'll talk a little bit about some of the outcomes that we saw. So as you can see here, we had three to eight percent increases and now projects that actually have CISD pipelines. And those pipelines are actually including static application security scanning more often now. They're also leveraging artifact management tools that we have more often. They're now requiring their pipeline has to run successfully before they merge more often. They're integrating our change management integration as well. They're reporting more unit test coverage now as part of their pipeline and they're requiring their merge requests or they're requiring that merge requests are needed before you can get changes into your branch. So while three to eight percent might not sound big, we wanted to call out that we have, you know, somewhere in the neighborhood of 20,000 projects. So it's not a small drop in the bucket to get this increase across the organization. And also it's important to note that all of this was done without a lot of outreach. Nicole and I just pretty much had some dashboards out there and told people they were out there. There was no mandate. You know, teams really were doing grassroots adoption because they were so excited. They wanted to be bronze. They wanted to know when gold was coming. When is silver coming? How do I get there? So it's really important. I think the other point I stressed in an earlier slide too about voice of the customer, really getting out there and understanding what our customers wanted and what problems they were seeing also really helped. You know, I think seeing this type of increase in six months is awesome. And I can't wait to see where the next version of the pipeline certification is gonna take us now that we've gotten even more feedback from our customers. So thanks everyone for joining us today. If you have any questions, feel free to reach out to us, find us, connect with us, and we'd love to answer them. Thank you.