 So, I hope everybody here is here for my presentation on securing your build pipeline from start to finish. Everybody here for that talk? Everybody here for that talk? Awesome. Great. So, my name is Dave Sarine. I am a senior solution architect with Red Hat. I've been with Red Hat for almost eight years now. I spent the first chunk of my career here at Red Hat as a technical account manager in our public sector group. I really developed my passion for security, as you can well imagine. So I've been a solution architect for about a year and a half, almost two years in the Northeast area. I cover accounts here in the Boston metro area, so I'm local as well. So that was a little bit about who I am. I also do have interests outside of technology. I am a gardener or a small-scale farmer, depending on who you ask. I live in Northeastern Connecticut, a.k.a. the quiet corner, a perfect place for that. So why are we here? We're going to be talking about securing your build pipeline, but what I like to focus on during this talk is less about the technology and more about the people and the process and the policy around securing your build pipeline. We often focus a lot on the how, the bits and bytes, the specific projects and products that you use to secure your build pipeline. We don't often enough talk about the why. And you're going to see me use jazz hands a lot or air quotes whenever I say dev ops or dev sec ops, because that means something different to everybody. There's no one real clear definition, but we'll get into that in a little bit. So how many people here are dev ops engineers? Okay, okay, okay, okay. I'd love to pick your brain on what that job title entails, because I've seen that title and about a dozen different types of people fit that job. So how many people here are systems administrators? Systems engineers, got one regular software developers, anybody in the app dev side? Okay, we've got quite a few app dev people. Great, this is good, right? Because we want a mix of different types of people in these conversations, because every single person in this process is responsible for security, is responsible for making sure their portion of the start to finish pipeline is secured. So containers very much have changed how we do our development, right? We're talking about sandboxed applications, we're talking about doing things in a much lighter, much more agile way, and I'll also use air quotes for agile as well, right? Because that's going to mean something different to everybody. So the way we're moving towards containers makes us have to change the way we've fundamentally changed the way we've deployed our finished product in the end, right? For those of you who are the DevOps engineers, right, how significantly has containerization and moving towards more cloud-native type development changed the way you do your day-to-day work? Pretty substantially, it's quiet, it's all good, it's all right, it's all right. So as well, I may pepper you guys with some questions to get some hands, but also if you have any questions for me, as we move down this conversation, please throw something large and heavy at me, right? I want to have more of a conversation. So how do we talk about it? How do we talk about securing your build pipeline? How do we talk to the key stakeholders about why you need to do what you need to do, right? So when you go to your managers or your architects and say, I'm a software engineer, right, and I need to make sure these gateways are in place in the pipeline, how do you explain that? So to understand how we explain these concepts, right, we need to focus on the three things, and I mentioned these earlier, we focus on the people, we focus on the process, and we focus on the technology. The technology, and you know, I'm going to get struck by lightning here as somebody who works for Red Hat, the technology is really kind of tertiary to the conversation, right? The technology is how we get it done, but we need to understand why we're doing it and where we need to do it through the process. So when we talk about people, we're talking about developers. We're talking about operations. We're talking about your site reliability engineers, right? How many people here bring security, you know, your security team, InfoSec, into the conversation, do you bring them in early? Do you bring them in at the very beginning of a project? Or do you bring them in at the end when you're looking for that sign-off, that rubber stamp? Bring them in early, absolutely. Yeah, yeah, so I have quite a few friends who are in the InfoSec world and in the enterprise space, and we have this conversation a lot over beverages, right, you know, when you can speak freely. And, you know, coming from a security background myself, security people are often the ones who have to say no, right? They're the ones at the end of the process where they're like, no, we can't do this because of food, right, because of ABC, XYZ. Oftentimes it's because they haven't been brought in to the conversation early enough, right? When you're in the design phase, when you're architecting out your solution, that's when security needs to be brought in, right, so they can voice their concerns early and often, right? And they can understand why you're doing the things that you're doing. All too often, security is brought in at the end of the conversation, at the end of the architecting phase, right, to say, hey, security, this is what we're going to implement, you know, can we get the sign-off? And they get this stack of, you know, 150 pages of, you know, build docs and, you know, different products and projects that you're trying to implement. And their default answer is going to be no, right? Their default answer is going to be no because it's so complex and there's so much going on that they can't, in good faith, give you a yes or no in any sort of reasonable amount of time, right? How many people have gotten that last minute no from security, right? Every hand in the room probably should go up, right? Because it happens a lot, it happens a lot, right? You're networking engineers. Another group that should be brought in early, early and often, especially when you start talking about containerization, you know, I'll use OpenShift as an example, right? When you're doing a production deployment of OpenShift, you have to carve off a slash 16, you know, subnet for OpenShift and a slash 20, right? Or slash 18, I forget, either way, it's a tremendous amount of IP space that you have to carve off for OpenShift, right? Networking often box at that because that's a lot of IPs that they need to be able to understand what's going on and they need to have insight into. So bringing them into that conversation early so they can understand why you're doing what you're doing is critically important. So your processes, right? When you're talking about, when you're talking about your build pipelines, when you're talking about going from an idea in a developer's head to running in production, what does that look like, what does that look like for you as an enterprise? What does that look like for you as a group? Because the two might not be the same, right? You can have enterprise standards, but then you can have standards for the group that you're working in, right? You know, how do you commit code, right? How do you, how does that code then get into the main branch that you're pushing out to production, right? Who does the code reviews? Is that something that's automated? Is that something that's manually done? You know, what does a code review look like? Who does the approvals for it to get out into production, for that code to get out into production? How are you doing your image scanning? How are you managing your life cycle, right? How are you managing that software life cycle? These are things that you need to think of on the front end, right? So when you're talking about technology, right? And we'll get into this a little bit more later on in the deck. But we're gonna talk about gates. We're gonna talk about gates in your pipeline and each stop along the path that involves security. Some of the technology that you're gonna be implementing into this path, you're gonna be using, you know, static and dynamic code analysis tools, things like Coverity, things like BlackDuck, you know, things like the AquaSec. You're gonna look at automation, right? Are you using Terraform, you know, the build out systems? Are you using Ansible, right, to do automation? Do you have automation built into, say, your Jenkins pipelines, right? You know, where are you automating? Can you automate in more places, right? Anytime you can take out that human element, anytime you can take out the eyes on the, the human hands that need to interact, anytime you can take those out, the more reproducible it is, the better your security posture's gonna be. As long as that automation process is reviewed over time and is implemented in a secure way, right? Compliance scanning, you know, how many people here work for a financial institution or an insurance company or a healthcare provider or anybody like that? We got one? You're gonna be my best friend today. Your hand's going up a lot, I like that. So when you're talking about financial industry, when you're talking about retail, right? You know, you have compliance, regulatory compliance standards like PCI DSS, right? Healthcare, you've got HIPAA, right? If you're doing anything with, you know, public sector, right? You know, they have their own standards, like, you know, the STIG or Secure Technology Implementation Guide, right? These are things that you need to consider while you're building your pipeline and what tools you're gonna be using to make sure as these containers are built and getting ready to be pushed out into production, do they meet these guidelines? Do they meet these standards, right? And not just before they get pushed out to production, but sometimes while they're running, you need to make sure that they maintain that level of compliance standards. And image signing, right? You know, how many people here would install an RPM that wasn't signed? Thank you for being honest. Thank you for being honest, right? We've all pulled an RPM from RPM bone, right? I mean, it's happened, but in an enterprise space, right? You wanna make sure that what you're running is verifiable, right? So are the images you're running signed? Are you signing them yourselves? Can you validate that signature? So things you need to consider, right? And there are technologies out there like Claire, for example, right? We'll do image signing and validation. So when we're talking about these, we need to be able to make the case to the business, right? As technologists, we often take the stance of, this is the right solution because I know it's the right solution, right? I've seen it in practice. I've seen people do it. It's the right solution. Is that argument really going to help make your case to your director? Is that argument gonna help make the case to the board, right? If you need to spend money and it's over a certain amount, right? So from my experience and a colleague of mine, one of our chief architects in the Northeast came up with these four key values that your CIOs, your CTOs, SVPs, even directors and regular managers, if you can position the way you're explaining the needs around one or more of these four key values, it makes it much more palatable to management to then say, okay, I get why you're trying to do what you're trying to do and it makes it easier to sell them on your solution, right? Again, I'm gonna ask a lot of questions today, so bear with me. How many people have run into that instance where you had a solution that you wanted to implement, that you had a product or technology that you wanted to deploy because it's got to solve a problem for you and you took it to your management and they said, no, we don't need, why do we need this, right? That there wasn't any sort of receptiveness to that idea. Right? We've got a couple, I mean, it happened to me, right? I mean, it happened to me tons of times, right? So we need to learn how to not just position it for the sake of the technology, not just position it because we know it's the right thing to do, which we do, that's why they pay us, right? We're the subject matter experts, right? You know, they should be able to take our word for it but they're a business. So when you're starting to talk about these things and you're starting to talk about why you need certain technologies for these gateways, these gates in your pipeline, learn how to position them in one of four kind of categories, right? You've got business capabilities. What kind of business capabilities is this going to bring to the enterprise? And when you're talking about securing your build pipeline, there's probably not many business capabilities there, right? You know, you're not providing any sort of real extrinsic, external value, right? With that cost management, right? Cost management could be a huge thing when you're building your pipeline and you're starting to add automation. You're taking man hours away. You're taking people hours away from having to do code reviews, doing code analysis, you know, things like that. You know, that's a real value. That's a real value to the business. So as you're architecting these solutions, figure out, hey, you know, we spend 30 hours per deployment on reviewing code. If we implement this technology, reviewing code is going to go down to 30 minutes, right? We'll just pull some rough numbers out of our head, right? Do you think that's going to be a value to your management? To say we're going to save this much time by implementing this technology per build, right? Yeah, that's going to provide a ton of value, right? Talk about speed of delivery, right? You're accelerating the delivery of your code that you've created to production. That's providing value to your customers, right? That's providing value to your end users. So that's a great way to position why you need certain types of technologies and why you need certain processes put in place, right? And the big one here, the big one, whenever we talk about security, we want to talk about risk, right? We want to talk about how we, and there's only a couple of ways to deal with risk. You either accept the risk, you mitigate the risk, or you offload that risk, right? Push it off to somebody else. So when you're talking about securing your build pipeline, and when you're talking about these gates that you're putting in place, and you're talking about the technologies and processes that you're going to need to make this happen, explain it in a way that says, you know, there's risk here if we keep doing it this way. Here's what that risk is. If we change our process to this new process, if we change our methodology to this new methodology, here's how it defrains that risk, right? Or even does away with that risk completely, right? Risk is a great way to have these conversations. Any questions so far? Everybody good? All right. So talking about the importance of security, I mean, it's a kind of a silly question, right? But I'm going to talk about DevSecOps, right? I'm going to do air quotes, jazz hands. One of the best definitions, and this isn't just because I work for Red Hat, but one of the best definitions I've read about DevSecOps is on Red Hat's website. And I'm going to read this directly from the slide. So DevSecOps means thinking about application and infrastructure security from the start. It also means automating some security gates to keep the DevOps workflow from slowing down, selecting the right tools to continuously integrate security can help meet your security goals, but effective DevOps security requires more than new tools. It builds on the culture changes of DevOps to integrate the work of security teams sooner rather than later. So you can see a theme here, right? Sooner, earlier, right? We need to get these things addressed earlier on in the process, because if we're going to move fast, if we're going to move fast, we can't put the brakes on something later on in the project to go back and bolt things on, right? So we need to think about these things. So here's a pretty standard build and run model, right? I don't have a whole lot of pretty pictures in my deck. I try to keep it pretty simple, pretty slimmed down, but your CI CD pipeline, you have build, test, scan, the analysis and QA, right? So there's something missing from here, right? What's missing? Right here in the front. Right here in the front, we have our people, right? This whole philosophy is really rooted in DevOps in that culture change, right? So to implement DevOps really well, right? And to get started in a healthy way with securing your build pipeline, we need to start with our people. We need to start with our people and our processes and policies. So when you're doing development, for all the developers in the room, do you have coding standards that you have to follow in your enterprise, in your group? I see a couple of chuckles. I see a couple head nods. So I'm going to go 50, 50, some don't, some do. Do you have regular guidance on recommended practices for secure coding? You know, again, a couple kind of nods, a couple of chuckles. So this is really where it starts, right? You know, if you have a set guideline, a set set of standards for your developers on how they do their development and what the expectation is of the developer for the code that is committed, you're already way ahead of the game because if you have a set set of guidelines and a set set of standards, it becomes easier to automate the checking of that code, right? Because you're going to expect things to come across in a certain way, in a certain format. If it doesn't meet that format, automatically gets kicked back, right? But once that code is committed, right? What happens from there? So this is the next game in your securing your build pipeline. So once the developer has committed that code, what happens to that code? Needs to go through probably some static code analysis, right? You know, send it through Coderity to make sure you're not hitting any known buffer overflows, SQL injection attacks, you know, anything like that. It needs to go through some code standard reviews, right? And again, that can all be automated. And then it hits another gate, you know, once it's gone through those checks, do you want just anybody to have the ability to accept pull requests, especially in production? No, we don't want just anybody to have pull, but you know, the ability to accept pull requests in production. You know, maybe if it's a web front end, sure, we can automate those pull requests. No problem if it's past these initial scans. If it's something that has to do with PII or anything that's sensitive or anything that's supposed to be secured, anything that's cryptographic, no, we're gonna want some kind of manual gate here. We can't automate everything away. As much as we'd like to try, we cannot automate everything away. So before we even hit the build process, before we even hit the point where the code can be built into a container image, we need to make sure that it's gone through the code scans, the code reviews, and that somebody has signed off on it. So those of you who are developers, you have a process like this, I'm sure, in place today, something fairly similar where you commit the code, somebody reviews that code, and then allows it to go through, correct? Yeah, so at that point, it gets into that build process. So it builds out the code, you have your base images, you have your layered images, so you have your OS image, and I'll use rel for an example, and then you throw your Tomcat on top of that, and you throw your Postgres on top of that, and then you put your code into that Tomcat image. You build that final container image and push it off, right? So then it goes into tests, right? You're gonna put it into your test environment, you're gonna make sure everything works fine, you're gonna scan it again, you're gonna make sure that there aren't any known vulnerabilities in there, so you're gonna use a tool like, Atomic Scan or, sorry. So you're gonna use certain tools to scan for those things, and to continue to make sure that this container meets your secure standards, right? Whether or not you need to meet PCI DSS compliance or HIPAA compliance or anything like that, right? Because you don't want a container to get out there that doesn't meet those minimum security standards. So at this point, we need to start thinking about where are we getting this content from, right? We've checked in that code, code's getting ready to be built, we're talking about the base images, we're talking about the layered images, so we're talking about your application stack, right? Where are you getting this content from? Is it coming from a trusted, is it coming from a trusted source? Are you pulling it from Docker Hub? Are you pulling it from Red Hat's Registry? Or are you pulling it from Bob's discount chicken and container registry, right? Which one of those are you gonna trust more, right? You need to make sure that you have a trusted source, that you're getting these images from. You know, some things to look for, right? When you're talking about looking for trusted sources, you know, what does a trusted source make, right? How often are they updating their images? Do they have a way to tell you whether or not the image, how healthy that image is, right? You know, are they performing regular maintenance on those images? Are they signing those images, right? So when you pull that image down, is there a way for you to validate that that image is the image they say it is, right? So these are things that you need to consider as part of that built pipeline. And these all need to have processes put around it. These all need to have policies put around it, right? So as you're doing this, this isn't done just ad hoc, right? Before you even get to the point where you're committing code into your pipeline and building it out, you've already thought about these things. You've already thought about, where am I getting my containers from, right? Early, often, you gotta keep reevaluating these things. So once you've created, once you've pulled in those external images, right? And you have your private registry, so you're pulling your pre-built images, right? To put on top of the base images that you're pulling, putting it through the CI CD pipeline, and you're spitting out a final container, right? What are you doing with that final container? Does that container go into your own private registry? Hopefully, right? So that's part of that, part of that where you need to think about securing your built pipeline. So when you're pumping those images into your container registry, how are those getting in there? Are you signing those images, right? Do you have a mechanism in which you sign those finished images that you've built from your code to tell you that when you go to deploy it, this is the image that you expect it to be. Sure, you've got your labeling, right? Everybody knows about labeling with containers. You've got container colon, latests, container colon, whatever your version numbering is. That's great, but is it signed? Is it cryptographically signed so you can validate that it is the container that you're expecting it to be? Does anybody hear, how many people here are actively doing container work today? Pushing out into production? Are you doing container scanning and container image signing and signature validation? Something to consider, right? So any questions? I've been going for about 20 minutes. I think I got about 10 more minutes to go, 10, 15 more minutes to go. So the rules of three continue. So we've talked about people, process and technology, right? As part of this, now what do we do with those? We need to control our environment. We need to defend our environment. And this is where we get into where DevOps really straddles the line because developers can no longer just pump that code over the wall and say, okay, operators, you do your magic and I'm not gonna care about anything else on the other side. Developers have to care about that stuff. And the opposite is true as well. Operators need to care about how developers are pushing that code through. So we're gonna, and how to extend that environment after the fact. How do we extend our security posture? How do we extend our security stance? Once the pipeline's put in. So phase one, we talk about control, right? So use public sources, right? Everybody's using public sources. Everybody's using Docker Hub, right? If you're using Docker, everybody's using Red Hat's Container Registry as well. We need to create those internal trusted images, right? Talked about this a little bit just now, right? But at the end of the day, we need to make sure that the images that we are creating are also trusted, right? Again, how do we validate that these images are trusted? They're in our private registry, probably, right? Whether it's Quay or Artifactory, any number of tools out there. Have they been scanned, right? Have they been signed? After they've been scanned, have they been signed? So we know that things haven't changed in that container image. So we need to make sure that we've created that internal trusted registry. So we need to include gates. I've talked about gates a couple of times. It's a great way to think about it, right? You know, think of it as a logical workflow or, you know, a flow chart. You know, you reach this point. Have we done this? Yes, move on to this step. Have we done it? No, okay, move on to this other step, right? So as part of these gates, you're talking about unit tests. Have we done our unit tests? Have they all succeeded? What are we using to run our unit tests? Is that still the best tool available to run our unit tests? Is it still reasonable for us to be using these tools for our unit tests? This is a living process. We need to make sure we keep reviewing these things, right? Because how we develop our code, the tools we use to develop our code, the languages we're developing in change and they're gonna start changing more rapidly. We need to keep reviewing and reevaluating the tools that we're using, right? Code quality. What tools are we using there? What are we looking for when we're talking about code quality? We're talking about code format. Are we talking about, you know, again, are we following recommended practices to avoid buffer overflows, sequel injection attacks, you know, those types of things? And vulnerability scanning, right? So, also, part of this, you know, when we're talking about these gateways, we talk about sign off, right? We need to talk about accountability. Who's accountable, ultimately, for this code being pushed out into production, right? That's part of those gates. I talked about it a little bit earlier. When I talked about who's signing off on that pull request, who's accepting that pull request to allow that code to be moved into production. Design for separation of concerns, right? Lot of tools today do a really good job of implementing things like role-based access control. Do you want your systems engineers, your systems administrators to be creating container images that have all of the libraries needed for your developers for their application stack? They don't know that information. Do you expect the developer, no offensive developers in the room, okay? Long time operations guy, but do you expect your developers to give you the full list of all the dependencies in it that they absolutely require to create? Probably not. Leave it to the subject matter experts, right? That's their area of expertise. Let them create those images, but the operators create your base images, create your gold images, right? Your operators are gonna maintain your base rel image and make sure that it's patched and up to date, they have the latest version that they need or whatever versions the developers need to deploy their images on, right? And then you pass that base image off to the architect who's working with the developers. They're gonna put, say, Tomcat or EAP or whatever framework that you need on that, on that. And then they're gonna pass that image off to the developers who are then going to deploy their applications on top of those images, right? And at that point, that all gets pushed to the pipeline, scan, check, validated, signed, and put into your registry as a discrete unit to then be deployed when needed. So you wanna be able to separate these concerns, you wanna be able to understand who's responsible for what, whose area of expertise is what and allow them to do that work, right? We're really tearing down those walls as to operations is responsible for everything, except for actually writing the code. No, the developers are now responsible for understanding what libraries they need and making sure those libraries are available in those container images. One of the stories that I love most that I heard this year at Summit, how many people here were able to go to Red Hat Summit? It's a couple of folks. I forget what insurance company it was, but they went really hard and fast towards the DevOps model, right? And the way they made sure it succeeded is they made developers carry a pager. So if something went out at two in the morning, who was getting paged? The developer. Your application's not working, why? Right? There was an interesting, what they called an unintended consequence, yes. They do, and here's the funny thing about it, right? The unintended consequence of that was the developers became much more conscious of the code that they were putting out. They made much smaller, discrete changes and deployed them more rapidly and consistently, so they're making much smaller changes every single time. So it actually improved the code quality and decreased number of outages by a large margin. I will try and dig up that information from that talk at Summit and put it as a footnote in my deck here, so when this is available to you guys, you'll be able to access that. But yes, they do still have developers there, and in fact, they are doing more work and more quality work than before. Because again, they would do every six months, again, I forget what the actual details were, but every six months they were doing a deployment, right? When you talk about a deployment every six months, how much is changing in that six months? This much, right? This much is changing in those six months. When you're talking about doing a deployment every week, how much is changing in that week? This much, right? So you have much less to go wrong, and when you're making smaller, more discrete changes, more quickly, it becomes easier to write automated tests for it, right? So when you get into the QA phase, right? A lot of that's automated. So now you're validating that your code's gonna work before it even gets out there, so you're improving quality, you're improving uptime, right? And that's all through just a process change. They didn't really change a lot in the way they did, they used technology. They implemented a lot of Ansible for automation, but developers were the same, right? Codebase largely was the same. The only thing that changed was the process. They moved more rapidly. The policy, they made developers start carrying pagers, so when there were outages, developers got called, right? So the technology was really kind of tertiary here, right? It was process and policy. That made that work. That was a good question, because I thought the same thing, because the developers aren't necessarily keen on carrying that pager, but when you're gonna be responsible for that 2AM phone call, how you go about your day job becomes very different, right? So any questions there about the separation of concerns, the separation of duties? All right. Managing your delivery, right? So once everything's done, you gotta deliver that code somewhere, right? Where's it going? Take one more look at that pipeline, right? When you're looking at that delivery cycle, when you're pushing it out, have you gone through all the gates? Have you validated signatures? Have you done your scanning? Right? So automate, automate, automate, automate. Again, we can't automate everything, but again, going back to my story about the developers having to carry the pager, right? They quickly took to Ansible to write all kinds of automated test suites. So when they made changes, it was really easy for them to spin up an environment, test it out, make sure nothing was gonna break, create discrete tests, discrete automated tests for that change, for that particular function. So now there's a lot more confidence in that code as it goes out. A lot more confidence that it's going to succeed in the build, a lot more confidence that it's not gonna make anything break, right? And here, when you can have something that's repeatable, like automation, where you don't have somebody who can accidentally overlook a step or accidentally overlook a check, right? Becomes a lot easier to find the issues before they actually become issues. Let's see here, just making sure I'm good on time here. So defend, right? This is not necessarily about your build pipeline, but your build's gonna end up somewhere. We need to be concerned about where that build's going, right? So that's gonna be your platform. So this is probably going to be a little bit more geared towards the operations folks in the room, but for the developers, this is something that you might want to be at least tacitly aware of, right? So you understand how your containers are running on that platform and what features and functionality it's taking advantage of, right? So remember, the OS matters, right? You know, take in case, in the case of rel. We use SD Linux for process segmentation. We use kernel name spaces, right? We use capabilities, kernel capabilities. You know, we enable and disable capabilities to give containers certain capabilities, you know, in the root kernel, but not others, right? That helps in that protection model of your platform. C groups, where you're starting to segment off, memory spaces and processors and things like that. So you can prevent and run away applications from consuming all of the resources on a particular machine, right? You could pin those in and set comp, right? So understanding these things is a great way to make sure that you understand what your container's running on and what features and functionality there that helps make it secure. You need to be able to secure that platform, right? So are you implementing proper role-based access control? You know, how are you doing users and groups and authentication and authorization, right? Are you tying directly into your LDAP? Are you using some kind of SSO model or using SAML, right? So is your platform multi-tenant, right? You know, are you housing multiple discreet groups on that platform? Do you need to make sure you understand how to keep those separated? How are you managing your secrets, right? Your passwords and your keys and everything like that that are stored out there, right? Are you using some kind of vault? Or is everything in plain text, right? I see a couple of chuckles out there. So I guess there's a lot of plain text passwords kicking around. How are you doing your logging and metrics, right? How are you in your auditing, right? Who's accessed the platform? Who's deployed these containers? Who's deployed these pods? Who's created these projects, right? How did they create them? When did they create them? How did they get in, right? You need to be able to understand these things as that final step in that build pipeline is that platform. You need to understand how to make sure that after everything all said and done, you have a way to understand what's happened, right? So when the auditor show up at your door knocking, those who have to be PCI DSS compliant or HIPAA compliant, you get those annual audits or sometimes quarterly audits, right? You need to be able to show them the audit logs. How things happen, why they happen, when they happen. So a little bit more on secrets management. We want to try and prevent as, we want to keep our passwords in plain text down to a minimum, right? We really do. How many people here are using some kind of vault, right? Like HashiCorp's vault or CyberArk for, perfect. Yeah, there's a couple people in here. Everybody else is doing plain text passwords, more or less. All right. And you know what, it's probably fine. In your environment, it depends, right? How secure do you need to be? This is where we go back to risk, right? So when you're talking to your management and you're talking about, hey, we really need to look at some kind of vault tool for our secrets management, why? Well, here's risk. Here's risk to the enterprise. If these passwords get compromised, what's the impact of that, right? Why should we care, right? We should care, but we need to be able to explain why and what the risk is and how we can mitigate that risk. So we want to talk about network defense, right? We need to talk about how we are separating networks between our containers. And in this case, this is more open shift centric, right? Where you have a pod that contains multiple containers, but you don't want those pods talking to each other unless you absolutely dictate that they must, right? It's almost like a default deny. So how are you segmenting that network, right? Do you have different subnets that you're using for different types of applications, right? So you want to make sure that you're separating those out. All right, and that brings us to network policy. And this is where developers and operators and networking and security all need to get together in a room with your architects to understand what applications need to talk to what applications, on what ports, on what subnets, when, in what direction, right? What protocols we're using. And then you can create network policies that dictate that this is the only way these containers can talk to one another. So for example, here, right? You've got two projects, you've got a finance project and you've got a CRM project. We want the marketing, the service, the sales, projects, or sorry, pods or instances running here in the project CRM to be able to talk to the web server, right? We want sales, service, and marketing to be able to talk directly to the bonuses, bonuses instance running over here on your finance, but we don't want sales to be able to communicate to RSU, right, but we want all the finance to be able to talk to the web server instance that's going to be able to talk to the rest of everybody else, right? So you want to be able to really start narrowing down what each container, what each pod in a project can do and how they can talk to the variety of different projects out there. Anybody have any questions here? This is a pretty standard kind of a Kubernetes way of looking at it, right? You know, you have multiple projects, they can't talk to one another unless you dictate that they can. Yes? So the question was, how do you handle ad hoc policies, right? Where you may need a project to talk to another project on a one-off basis, right? And once that is done, that policy in theory would still exist, right? You typically wouldn't have that situation, right? So if a project needs to talk to another project, there's a reason for it, right? What's the reason? What's the justification for those two projects to have to talk to one another, right? There's no specific reason for this. We can take that question offline. The gentleman over here wants to know if there's a time-based way of saying, we're going to create this policy, we want this policy to exist for a week, right? Let's use batch processing at the end of a fiscal quarter, right, as an example, or we'll take tax returns, you know, processing tax returns as an example. You know, we expect to be processing tax returns for three months, right? After the end of, let's say April 15th, whatever it is. How do you handle those policies? So yeah, in that case, you probably set up some sort of automation where you would say, and on this date, we're going to set up this policy, you'll have the config set up, so you'll kick that policy off. And then at the end of that time span, you know, have it remove that policy. So, but yeah, there's no great way today that I know of to just in a config say, keep this policy alive for a number of days. Any other questions? Attached storage, right? How are you managing access to storage? So in terms of OpenShift, you know, OpenShift setups persistent volumes and then a pod or a container would snag a persistent volume claim, you know, to claim that persistent volume and only that container will have access to that persistent volume unless otherwise dictated in a deployment config, right? You can limit who has access to that storage in a variety of different ways. How are you limiting access to that storage, right? Are you doing volume bind mounts? Please don't be doing volume bind mounts. You know, are you doing NFS? Do you have SE Linux enabled, right? For your storage? Things that you might want to consider, you know, while you're architecting this solution and especially, especially when you're a developer and you're developing something for say, let's say a financial application, right? You know, inside in-house. You don't want to use just any storage that's out there. You need to work with your operation friends, right? Cause you should be friends at this point, right? Cause we're doing DevOps, you know, everybody should be talking to one another. You need to talk to your operation friends and say, hey, I'm doing this type of application. What storage type should I be using? Cause they all have carved out in an open shift a variety of different storage classes. Fast, slow, medium, you know, large, small, secure, not secure, right? So again, this is where, you know, as you move down this path and, you know, a lot of people are already well down this path. So it might take a little bit of, you know, re-engineering, but you need to understand how you're even accessing that storage. What happens to the storage once your container spins down? Does it stay out there? Is there a process in place to remove that data on that storage? Right, is it ephemeral? Is it static? Right? Things that you need to consider. And I think I'm getting close to the end of my time. Can I get a time check? I think I've been talking for about 50 minutes now. I'm getting to the end. So API management, how are you managing your APIs? Who has access? You know, who can use what API endpoints? Are you protecting them by authentication? How are you doing the authentication? Right? You know, using OAuth 2, right? SAML, regular username and password. Are you metering the API access? Are you only allowing people to, you know, move certain amounts of data, you know, by the API, or are you just wide open? So extend, this is the last piece. You know, we want to talk about extending the ecosystem. So we talk, this is getting into the different components that fit these gateways, these gates, right? So identity access management, you know, what tools are we using? External certificate authorities, you know, are we doing all this internal self-sign? You know, are we using hardware security modules? You know, file system encryption tools, right? You know, these are a lot of the things that you want to consider to put into those gates. And here's the NASCAR slide of some third-party tools that you can use. You know, you got JFrog, HashiCorp, you know, Nginx, Tigera, Blackduck. So, you know, it's certainly worth taking a look at, you know, for some ideas on tools that are out there to facilitate these gates. And again, just kind of wanted to go over again, you know, here's the, you know, you're building, you're testing, you're scanning, you're doing your code analysis, putting it through QA, and then you're pushing it out, you know, through your deploy time policies. You know, do you need somebody to stamp it to you as approval? Is it something that you can automate and then push it down into production? And again, here's your life cycle. And I just wanted to throw that up there just as a refresher at the end. Kind of what the flow looks like. And your plan build run, you know. So, with that, that's it. Any questions? Comments? Concerns? Equipments? Silence means I was the perfect speaker. You guys fully understand how and why you want to secure your build pipeline, where you need to put your gates and how you need to talk to your management about the tools that you need to put in place to satisfy those gates. Everybody's ready to go? Awesome, perfect. Thank you very much.