 Hey everyone, my name is Steven Toronto. I'm a chief engineer at Booz Allen Hamilton And today we're going to be talking about supply chain security with the Jenkins templating engine So before we start I want to take a step back and look at What is software supply chain at a glance where it covers everything from? Trusting who is contributing to your code base How is that code base getting pulled into your automated DevSecOps pipeline? What dependencies are being pulled into that pipeline? How do you trust the infrastructure of that CI pipeline? How do you make sure that the artifacts published to your artifact repository? are the ones that got built from your DevSecOps pipeline and Finally, how do you make sure that the artifacts that you're running in production are the artifacts that you expect to be running in production? Each one of these arrows Could really be its own conference talk. So I want to you know Acknowledge that and say that today we're we're gonna be focusing on your DevSecOps pipeline and more specifically What are the elements of a DevSecOps pipeline to make sure that you're doing everything you can from a security perspective? to make sure the applications that you build are secure and then You know, what are some of the challenges that organizations face as they start to scale these DevSecOps pipelines from one team To dozens or hundreds of teams throughout the organization. So let's start with a overview of DevSecOps one of the privileges of my role is I get to talk to Dozens of teams and hundreds of developers about DevOps and the first thing I learned Said no one has the same definition of what DevSecOps actually means The way I look at it DevSecOps means incorporating security into every step of the software development lifecycle In that manifests itself In a couple different ways. The first is application dependency scanning, right? So as we build applications, how do we make sure that the third-party dependencies that we're pulling into our environment? I don't have known vulnerabilities. There is lots of great tools out there in the industry to do this today I'm an open source perspective. You've got a lost dependency checker, but there's also a ton of great vendors in this space around Sonotype nexus with their firewall and lifecycle products. You've got Artifactory with x-ray Blackduck There's plenty of options to make sure that the applications or the dependencies that you're pulling into your applications don't have known issues And these dependencies are like the sort of the raw materials you're pulling into your Your assembly line that's going to produce your your artifact. You want to make sure that these To the best of your knowledge are aren't compromised Once you pulled in those dependencies, you have your own code, right? The code that you have written To comprise your application How do we make sure that that code doesn't have Known vulnerabilities So tools for that would be static code analysis. So you've got your sonar cube fortify And what these tools are looking for is really You know, how do we make sure that there aren't hard-coded password variables? You know, there's certainly more sophisticated examples like how do we make sure that there aren't memory vulnerabilities if you're writing code in and see for example once we know that the raw materials of our application are Secure and we know that the code that we've written is secure from a static perspective We can actually build an artifact here. So that would mean, you know, we're at day zero of Cubecon building a container image So with the container image comes a whole new artifact, which means a whole new set of scans that we get to do So this is where container image scanning comes in. There's tons of different tools in the space I live in the the federal space doing software delivery so, you know, our team works closely with the Department of Defense's iron bank, which is an accredited container registry That anyone across the DOD can use with hardened container images The container hardening pipeline for iron bank comprises three or four different scanning tools from anchor to twist lock to Open SCAP, you name it. They're scanning for for STIG compliance and CVE compliance with a couple different tools Once you scan these images, you know, vulnerabilities really come in two different flavors. You've got CVEs, so I pulled in, you know, Compromised code into my ecosystem But then you also have vulnerabilities that stem from misconfigurations And this is where continuous compliance comes in So continuous compliance means, you know, these same tools that do container image scanning can also do Scanning of the configuration, right? So does the container expose port 22 Within the federal space, there's all kinds of federally regulated guidelines around You know, the way you have to configure your infrastructure Most of those policies have a control profile associated with them that these continuous compliance Scanning tools can validate Making sure that you, you know, have configured your infrastructure in a way that's secure Once you have a container, you can actually go and deploy it to an environment And from there, there's still more testing to do Right, you can do penetration testing to make sure that your application is not susceptible to some of the more common Attacks things like SQL injection or, you know, not parsing user input and ending up with a remote code execution vulnerability And then the same vein of compliance. I think it's important to mention accessibility assurance So your DevSecOps pipeline once you've got a deployed application and a test environment can also scan it From a compliance standpoint to see if it's Compatible with 508 There's no tool out there that I've seen yet that can tell you with a hundred percent certainty If your application is accessible, but it can definitely tell you if you're not, right? So if you're missing some obvious Accessibility features, you know, things like I don't have an alt text on on this image These tools will catch that for you, which really gives the folks that are doing manual accessibility testing The ability to focus on the more complex areas of accessibility So there's a lot of content there, but now we need to string it all together into a DevSecOps pipeline And things start to get complicated So if we look at our path to production here all the way from plan and develop up to monitor We have all the different kinds of testing pull a word out of dictionary and add testing at the end of it And I guarantee you it's probably got a place in your DevSecOps pipeline at scale Across an organization different types of applications are going to use different tools to implement these gates, right? So take a simpler example if I've got a Java application versus a node application my Java application for unit tests and for building the application is Going to probably use Gradle or Maven If it's an older application might be using something like Ants, I've got my node application It's probably using NPM or yarn or gulp So, you know, even though on the screen right here, we've got a generic DevSecOps pipeline representing a path to production that integrates all these different kinds of security testing The implementation of this pipeline across your organization requires the ability to integrate with Different combinations of tools for each of these boxes depending upon the tech stack, right? And for some tech stacks teams might be using multiple combinations of those tools So what let's talk about what happens in in practice, right? So typically There's when you're first starting your DevSecOps journey a lot of organizations will choose You know what us consultants would call a champion. So who's your team that's going to be the first to try this out? and You'll build a pipeline for them. It will be moderately successful There'll be some bumps in the road as they experience their first Security scans there's a ton of patching that always happens the first time you start in incorporating these practices into your teams But you did it you have a DevSecOps pipeline for a specific team So now you deal with that learning curve You you successfully get the team integrated and you say awesome like let's scale this across the organization Our pilot was successful So what happens next? You wrote your pipeline presumably In one of the the more popular CICD tools these days like Jenkins or GitLab and to do that You needed to have a pipeline artifact that defined, you know, what? What is our software delivery process as code? So there's a there's a file in your source code repository called a Jenkins file Or called a .gitlabci.yaml file that's outlining What exactly is supposed to happen when developers make changes to this repository? So we want to take that artifact and we want to apply it to multiple teams So what do we do? We copy and paste it To the next repository that we want it on board and we start tweaking it right where we were doing I don't know the gradle test execution before we'll swap that out to Maven or to npm or yarn And you know that that works for a while. So we if we look at the two axes of this graph here We've got scale on the x-axis that's you know Corresponds roughly to the number of teams or applications that you're integrating and on the y-axis here. We've got complexity And the subtext being pain So that's you know as we start to onboard more than one application and we're copying and pasting and tweaking Our complexity Starts to go up at a pretty steady rate as we start to onboard things Eventually you hit a point where you start to realize like I either need every development team in my organization to understand how to create a DevSecOps pipeline and Maintain these pipeline artifacts or I need to find a better way You know, we're running into all sorts of challenges teams don't know how to edit these files These files are changing for every single team because there's no standardization So usually someone has a great idea of we should modularize these these pipelines make them a little bit more reusable You know in Jenkins that would mean let's create a Jenkins shared library where we can pull out common steps And that'll really simplify our pipeline as we scale it and you know, it does at first you do simplify Your larger scale pipeline development that way Over time as you add more and more teams you find yourself adding more and more modules to those shared libraries or modularized pipeline implementations Eventually you say this is not working either, you know, we've probably hit 50 to 75 percent of of our You know organizations application portfolio, but things are getting way too complicated this this you know shared library is gigantic So I have a better idea and I've you know, it's it's really interesting that I've probably seen this pattern Six or seven times just independently like what teams end up doing once they reach this scale is They simplify their pipeline to just say let's call an entry point method inside the shared library So you're you know, for example, your Jenkins file would just say Pipeline dot run and you'd pass it a bunch of arguments and that would try to abstract away all of the complicated bits of This DevSecOps pipeline. So, you know, you simplified the configuration for an individual team. They had to just Copy and paste this, you know file into their repository and pass it some input parameters but on the back end you have a massively complex shared library and over time as you scale more and more teams and applications the number of possible input arguments Starts to go way up to support all these different use cases So let's The gist here if I had to do a TLDR is that the more teams that you're trying to apply or have adopted DevSecOps The more pain your organization is going to feel Let's talk about why that pain happens the root of this pain From my experience is that every CI CD tool in the industry today is focused on How do I build a DevSecOps pipeline for one team for one application? and that often means that you define your Your CI CD pipeline as code inside each and every source code repository and That by itself from a software supply chain perspective poses issues that we'll get to in a second So the the first you know the first sub-bullet under I've got to duplicate my pipeline Is that most pipelines couple the business logic and the technical implementation? So and see it inside the same file you're saying, you know, this is what I want to happen When right when I open a pull request to this particular branch. These are the series of tests that I want to happen And within those conditionals, you're also putting the specific technical implementation logic of You know go run Java or go run my node app and do tests or do scans So that that has two main issues with it right the first is time I need to duplicate my pipeline across every single source code repository Which is really cumbersome. It's a you know, typically a manual process Sometimes you'll add some automation to bootstrapper repository, but at the end of the day you still have duplicated your pipeline definition The second is complexity, right all those different combinations of tools Mean different pipelines even though at the end of the day. We're really trying to automate a generic Path to production that incorporates security We have multiple definitions of those pipelines And then the the second issue here is because we're duplicating pipeline definitions where we've got separate artifacts for every single application There's no way to know that These teams are actually following the process. They're supposed to right from a site supply chain security perspective like What's to stop a developer from going and editing their Jenkins file in their feature branch and saying? Deploy to production All right, we we'd like to think that the developers that we work with wouldn't do that But you know, sometimes our accounts get compromised. There's real-world scenarios that happen where? the fact that developers have direct access to the CICD pipeline in their source code repository and many organizations poses a challenging situation And then the last major issue here is sustainment I've been on teams where We have to maintain the DevSecOps pipeline that's used by you know, a bunch of different Development teams throughout an organization and trying to keep up with that complexity It's just not sustainable. You either have to scale This DevOps team linearly to support You know the number of apps that are being onboarded or you need every dev team to become DevOps engineers Which itself is not super sustainable So let's talk about how we can make this a bit better Pipeline templates when we pulled up that you know graph earlier. I'll shoot back to it When we looked at this DevSecOps pipeline, we didn't call out a tool here We just said every app regardless of the type of app is going to do these things To simplify this picture a bit. Everyone is going to run some tests. Everyone is going to build an artifact Hopefully sign that artifact and publish it to an artifact repository. They're going to scan that artifact They're going to deploy it somewhere. They're going to run some more tests And then if everything's good, they're going to deploy to production Nothing about what I just said calls out a specific type of application a specific build tool Pacific scanning tool would have you however when we build our pipelines we couple them to specific tools So what if we didn't have to right, what if we were able to Create tool agnostic pipeline templates that call out those generic steps and then let us plug and play With what tools are being used to implement the gates? So, you know this fancy GIF here is stepping through we can apply the same template to different combinations of tools Without having to create a new pipeline from scratch and let's dive into it like a real-world example That talks through what that looks like using the Jenkins template and do The step one defining our software delivery process in the upper left in the upper right. I have two Admittedly contract pipelines to demonstrate the concept here on the left. We have an application Using Maven as their build tool and on the right. We have an application using gradle as their build tool On the left, you'll see that we're going to do a build with Maven running Maven clean package And then we're going to do static code analysis with sonar cube on the right We are going to do our build with gradle and we're also going to do static code analysis You know the big aha moment here is that regardless of the build tool being used the process is largely the same We're going to do a build now. We're going to do static code analysis So on the bottom half of the screen, you know You can see a reusable pipeline template that calls out those two steps build and static code analysis so wouldn't it be great if instead of needing to Define my pipeline for each of these teams individually I could put you know in a central source code repository This is our reusable pipeline template build and static code analysis And then we're going to plug and play with each team gets to pick are you using Maven or are you using gradle? And I can enforce that you're going to use sonar cube from a static code analysis perspective So how do we make that possible? You know the first step on the left We have a pipeline configuration repository So this is a central location where you're able to define your common pipeline template and some libraries so That pipeline configuration directory has a Jenkins file. This Jenkins file is that reusable Pipeline template we talked about on the last slide where it's going to say build and static code analysis In the center. We have what are called libraries So we have a build step in our template that has two implementations from Maven and gradle And then we have a static code analysis step from sonar cube If we look at our libraries structure You'll see that just as you would expect there's a library for Maven a library for gradle and a library for sonar cube The Maven and gradle libraries have a build.gruvi file and the sonar cube Directory has a static code analysis.gruvi file these files the base name of the file Becomes the step name when these libraries are loaded. If we look at the center We can see the source code within those The steps as we call them This pipeline code is the exact same pipeline code you would have written before Just modularized in a way and wrapped in a call method So that it can be plug-and-played Right or rather we can pull in different libraries to the same template To get different implementations of the pipeline In the bottom right there's pipeline configuration So it's specifying you know from a library's perspective I want to load maven and sonar cube from the gradle apps perspective I want to run griddle and sonar cube Let's step forward. So these libraries become the building blocks of your dev sec apps pipeline within your organization And they're only as valuable As they are reusable Right, so how do we make that possible when you load your libraries on the left? you know separate from your pipeline template you have a pipeline configuration file which makes the template mean something So this is an example for the maven application where we're saying load the maven library Which you know, there's another way to say my build step should come from the maven library The sonar cube library in this example is defining a couple parameters It's defining like the scanner version and the whether or not we should enforce the quality gate these Parameters from the pipeline configuration file get auto wired to a config variable Available within your your library. So from a developer perspective If i'm writing or contributing to a library within my organization I can define whatever input parameters are needed Uh, and then you know that makes these libraries reusable across teams If some teams are just starting out on their dev sec apps journey and aren't quite ready to enforce the quality gate As a purist I wouldn't recommend that but you could provide a pipeline configuration Uh, a variable for the sonar cube library that allows teams to tailor tailor it On the next slide, let's talk about how we can govern these pipeline configuration files a bit more so you know each repository needs to end up with a pipeline configuration That says which build tool to use and the fact that they're using sonar cube We don't want every team to have to define the fact that they're using sonar cube We want to be able to enforce that from a compliance or governance perspective We want to be able to say you have to use sonar cube, but tell me what build tool to use the way we can achieve that in The jankin sampling engine is through hierarchical pipeline configuration files So the way that works is I can define in a central location This is the pipeline configuration everyone has to inherit In our example, we're saying that Everyone has to use the sonar cube library so by default In the jankin sampling engine that means that teams aren't going to be able to override the fact that they're using sonar cube But if you'll notice in the global pipeline configuration We have an annotation there that says at merge So what that does is it says you know development teams are going to be allowed to Add additional libraries to this pipeline configuration You can't change the fact that you're using sonar cube, but you can add your build tool to the library's block At runtime when the pipeline kicks off we aggregate these pipeline configurations together And we end up with the resultant aggregated pipeline configuration that the application needs All right, so one team is using they've been in sonar cube. The other team is using cradle and sonar cube so there's There's a lot of abstractions That go on in this framework You know the heart of it is to stop creating Pipelines for individual applications and start creating reusable pipeline templates So in the upper left, we have our pipeline template right that defines in tool agnostic terms What is our teams? Our organizations software delivery process right and that can look like build scan publish deploy In practices can get more sophisticated to call up branching strategies In practice you can have multiple pipeline templates for your organization and let teams choose But the idea is that we're going to define them in a central location and teams will inherit these templates So what types of things can we templatize right? What can I inject into that pipeline template to make it mean something? And we have our steps like build and static code analysis and there's other Objects that can be created that in our framework we call the pipeline primitives So there are steps stages keywords and application environments that can be injected at runtime to the pipeline template So how do I define those things in my pipeline configuration file? And in the bottom right, we've got a couple of the different blocks that you can configure To define what is your template going to do? And then where do I define this pipeline configuration and where do I define this template? That lives in your Jenkins folder settings so You can have libraries and pipeline configurations applied globally Jenkins wide and then the manage Jenkins, you know settings page set that up And then you can create an arbitrarily complex folder structure to map whatever governance hire you need So in every folder in Jenkins, you can say here's another pipeline configuration that applies to this level of the hierarchy The teams or jobs within this folder are going to inherit And then finally you can have the application specific pipeline configuration file Which can live at the root of the team's source code repository So through this approach You can set up as complex a governance hierarchy As you have a permissions hierarchy within your organization If some applications are Super critical and they shouldn't be allowed to change anything You can create a folder with more locked down permissions On those pipeline configuration files and something that's you know, a rapid prototyping Not production-facing application yet That might need some more lenient permissions at first So what are the the key takeaways here? The first is that The Jenkins templing engine is a framework for developing tool agnostic Templated workflows that can be reused by multiple teams simultaneously Regardless of the tools being used Right and the reason that provides value is because Most ccd tools and most organizations are defining pipelines On a per application basis, which leads to A ton of complexity it allows developers in many cases to Directly edit the definition of the pipeline Bypass those checks and do a deployment to production And over time it's very difficult to sustain The second key takeaway here is that this approach separates the business logic to find in your pipeline template from the technical implementation Of your Steps of your pipeline through pipeline libraries allowing teams to configure their pipelines instead of build them from scratch Right separating business logic and technical implementation is the software best practice We've talked about for years just hasn't made its way to pipeline development yet at a large scale for some reason Right and then finally through the Jenkins templing engine You can apply organizational governance by defining these reusable templates that incorporate security Uh while making sure that developers can't bypass these checks You can optimize pipeline code reuse through these plug and play libraries in a way that Is scalable and then finally you can simplify pipeline maintainability In my experience, it's a lot easier to manage a centralized pipeline template with plug and play pipeline libraries Then it is to try to maintain copied and pasted into tweaks tweaked pipeline definitions across your entire organization So I hope this talk was helpful to help you understand How the Jenkins templing engine serves a super small role Of governing your DevSecOps pipeline as part of your supply chain security Feel free to you know check out the repository and Reach out if you have any questions. Thank you