 All right. So it looks like we're right at time, so let's get going. So thanks for coming. We're at a Kubernetes conference, which is very exciting. So I appreciate you taking your time to come to a topic that's not specifically about Kubernetes. I'm going to be talking about security and compliance and sort of taking our knowledge and how we do things from DevOps and applying that to security and compliance. So I wanted to talk about security in general, but it's such a broad topic. I figured I'd just zoom in on the compliance side of stuff, and then the rest of the security stuff. If you nail compliance, you can then use those same techniques for the rest of that stuff. So my name is Paul, and I'm a developer advocate at Pivotal, or as my wife would tell you, I'm a professional vacationer. So I have the privilege of being able to travel the world and coming to places like Shanghai to talk about what it is I do even when it's not specifically about things that Pivotal does, which is great. So I really appreciate you all inviting me to come here, and I really appreciate the ability to speak to you in my native language. You would not like me to try and speak to you in Chinese. I'd be able to say one word hello and that's about it. So thank you for that. So I'm going to talk a little bit about compliance, exactly what I mean when I say compliance, then DevOps go through some of the DevOps concepts and then tie the two together with just some thoughts about the two together, but also some concrete examples from what we did at a previous job. Then if we have time, we can do some questions. So we'll see how we go. So compliance is almost always about risk mitigation and risk management. Therefore, it's usually imposed on us by some government or legislative body built around consumer protection. So often as a result of some catastrophic event, some bad actor doing something causing government or similar body to say, we need to make sure this can't happen again. Then you have a bunch of different types of compliance that apply to different types of companies and different types of activities. So these are fairly American-centric stuff, but I did add a couple of non-American ones there, for examples. So you have PCI, which is really about companies are handling credit cards and people's money and how you store the information about their credit cards and their personally identifiable information. HIPAA is for say the health and medical industry. Same thing about really protecting the consumers of the health and medical industry because they're very sensitive information there. Then you have things like SOX or Sabane's Oxley, which is about protecting us from actually like bad actors like large enterprises that have like bad actions. I think a lot of that came out of like Enron and stuff like that. So really protecting us as consumers from not just like accounting errors but also from like actual fraudulent activities in those large enterprises. Then so that's your government legislative body style compliance. There's also compliance that we self-impose or impose on us by our customers. Some examples of that is maybe the CIS controls or the STIG, which is the Security Technical Implementation Guide. There are pieces of a compliance that somebody has built that really knows how to say secure a Linux system and has basically open-source that information and any company that wants to make sure they have a secure Linux system can follow these compliance guides and have a higher chance of having a secure system. There's really these three main aspects to compliance. There's the compliance docs themselves, the specifications. They're often lengthy and verbose and not very clear, which can create a lot of problems for us when we're trying to actually build controls around that compliance. That's where your checklists are, your controls, these are the policies and procedures that we've turned the specification into to make sure that we're going to follow that compliance. Then there's some verification. Usually you have like an internal team that's doing some sort of auditing of compliance and then often there's also an external body that will come in and ask to see your documentation, ask to see your controls, and validate that you are indeed following that compliance. This is an example of the STIG, the Secure Technical Implementation Guide, which I really like because for every setting that it wants you to have on your server, your Linux server to be secure, it's pretty explicit. It tells you why it exists, like SSH protocol version one is a bad thing, and it tells you exactly how to check for it, like it gives you the exact command to check for it. It also has some really good control names and rule IDs and stuff, so it's very easy to follow. I can give this to anybody who is capable of SSHing into a Linux server, and they can go through the compliance guide and make sure that a given Linux server is compliant. So this is a good example of really good compliance. This is from PCI and I would say this is an example of the type of compliance I don't like, because it's not very prescriptive as in exactly what you should do, and a lot of it you can't solve by testing settings on a server because it's more about like access control, who has access to what and how and stuff like that. So a lot of this type of compliance, like a lot of the legislative body driven compliance, has to be solved as like a policy and procedure as much as you can solve it with software. So the Stig side of type compliance is great, I can solve that with software. The PCI type compliance, there may be some pieces I can solve with the software, but often it's like people following particular procedures and policies. Then as we interpret that compliance, we may come across more and more things that we can add to software, but it's not as easy. So like you and I, things like the PCI spec looks a bit like this, and so as a large company that has to adhere to a lot of different compliance, like there's a lot of companies that have to do HIPAA and PCI and Sarbanes-Oxley, and it's complicated, and I look at that stuff and this is what I see, it's unreadable. So you have to hire experts that are able to read and decipher that compliance, but also that they're experts in your business and how you work. So they can figure out how to build the appropriate controls, the policies and procedures to make sure that you're adhering to that compliance. So you need to determine how access is granted to a particular user so you can validate that you're doing it correctly and you can order it. So the person needs to know not just the compliance, but also what systems are you using for authentication, using LDAP, Active Directory, are you doing some sort of single sign-on, that sort of stuff. So they really need to know like deep dive into what you as a business and as an IT organization is doing as well as what the compliance is saying. So to ensure accuracy, most organizations tend to opt to create these separation of duties by having a person or a team that are built to do specific activities within compliance. Compliance itself is covering yourself, you're protecting yourself, and so that tends to enter the system. So you want to make sure there's lots of auditable events. So usually you don't like talk to each other, you like send tickets to each other and have these official communication things. So people can come back and validate things later. So you end up with these very walled off silos. This is the thing that we see a lot in enterprises, and this is one of the things that DevOps really came up around to do is to break down those silos and make it easier for people to work together, but still have ways to be able to audit and see what's going on. So we can get in to talk about DevOps now. So hopefully a little bit about DevOps. This is a famous DevOps picture from nearly 10 years ago now, from Andrew Schaefer, I think was the first person to do this. It's showing that these developers and operations have different goals. The developers needs to be writing features, needs to be adding software, that's their job. The operators need to keep the servers running, so they're focused on stability. So any changes to the software adds risk to the stability. So they're at odds with each other, they have different goals, and often what happens is you have this wall that's up there which was labeled the wall of confusion where basically the development team is just throwing like software, maybe they've already built it into a package, maybe they haven't to operation just saying, I need you to run this for me. Operations like, I don't know what's in this, I'm not going to run it until I do like a lot of testing and stuff and you have this very lengthy period of trying to get stuff running. As we hit 2010 or so, we started to think about approaching infrastructure and operations in like an agile way, taking some of the things we learned from agile software development. This really started to hit up so at one of the first examples of DevOps type stuff came out here in 2009 or 2010. It was actually maybe 2008 I think actually. John Ospor and Paul Hammond at Flickr, and they're basically we're talking about how they do 10 plus deploys per day at Flickr by having like Dev and Ops working tightly together and removing that wall of confusion. So in 2009, we had our first DevOps days conference run by Patrick DuBois, and so we started to really get serious about this. So we've been doing DevOps now for about 10 years, and so we've got some pretty good ideas of how we can pull down those silos and how we can get operations and developers working together and have like software-driven operations. We built out this acronym. So John Willis and Damon Edwards kind of coined an acronym CAHNS, so Culture Automation Measurement and Sharing, and then Jess Humble added lean to it. So we have this CAHMS acronym, and basically these are the things you need to be focusing on as you're doing DevOps in whatever way to actually have good outcomes. When you follow that CAHMS methodology, you create this type feedback loop of increasing continuous improvement. So this is an example is a bunch of different types of these types of feedback loops. This is the UDA loop, which is Observe, Orient, Decide, and Act. So if you look back at what we had like automation, measurement, sharing, they kind of map into this loop. So when we use lean techniques to apply automation, measurement, et cetera, we're basically building out this feedback loop, this constant self-improving feedback loop. But that was DevOps, right? So a few years ago, people started talking about DevSecOps, and similar, and we have FinOps and all these other terms. A lot of us thought this was strange at first, because we thought that we were building out DevOps to be this very inclusive. It's not really just developer and operations, it's the whole business. But it didn't really work out that way. This is with a word cloud built out to look at I think some of the DevOps conferences and what the topics were, and there's no security and compliance listed in here. We thought we were being inclusive, but maybe we weren't being as inclusive as we thought we were. So we were just really focusing on the infrastructure side of things. We were doing config management, Chef and Puppet, doing things like Terraform. But we were just either ignoring or just paying lip service to security and compliance, and even other aspects of the business. That left the security team trying to play catch up, and generally they ended up creating checkbox security that didn't really have any teeth. Then we would make up dumb memes about them and say, ha ha, look how stupid the security team is. That didn't really help. So not only did we not help bring the security and compliance teams with us, but we also in a way actively deprived them of their ability to do anything other than very basic checkbox style security. So they tried to add themselves to the conversation by coming up with extensions to that term. So DevSecOps, Rugged DevOps, Secure DevOps. We're all terms that we're all thrown around. DevSecOps is seems to be what has stuck. So and not only that, but they've actually built this fairly strong body of work that we're in the DevOps community didn't even really know about, because we weren't paying attention to this concept. We were often actually deriding and saying, we don't need DevSecOps, it's just DevOps with another name. So companies like Fannie Mae, we're building out really strong DevSecOps pipelines, where they were bringing security and compliance into the development cycle. We realized that they needed a culture in which they incentivize developers and operations to consider security and compliance into every aspect of the life cycle. Now, the problem is as developers and operations, we're not security experts and for a long time, we were just like, hey, you need to write secure software, and you can tell me that as often as you want. I don't know how to write secure software, right? I'm not the security expert. So what companies like Fannie Mae did was they actually brought the security and compliance testing into the development life cycle. So when you compiled your code, when it went into continuous integration, it ran all of these security tests, compliance tests, and would give you fast feedback on whether or not you were making secure and compliant software, and that then helped empower developers and operations to become more secure and to be more mindful of secure, because they didn't need to know every single aspect of security and compliance. They had systems that would give them very fast feedback about what they were doing. So if we need to call that this Effort DevSecOps, I think that's cool, but we as DevOpses need to get involved and learn from them and also teach them, because there's a lot of things that we do in DevOps that we could help the DevSecOps folks with as well. So basically, John Willis was talking to some folks about it, and he was like, it's just a name, like get over it and let's just figure out how to work together. So we can just throw away the term DevOps, DevSecOps, or whatever, and just focus on like the calms side of things, because regardless what aspect of the business you're working in, if you think about these things, this is one way to get that continuous feedback, continuous improvement cycle. So we can just focus on this, whether we're security, whether we're compliance, whether we're operations, or even finance, there's plenty of ways you can apply these to finance. So let's focus on the calms side of things and not worry so much about like with DevOps or DevSecOps or rugged DevOps, or whatever we want to call it. So now we're going to tie the two together. But first of all, I want to actually talk a little bit about what we do at Pivotal, and I want to say I'm not trying to sell you anything here. I just want to talk about how a company that does some fairly mature like automated operations and automated security works, and then I'll get on to talk about zero back down on compliance and talk about how at a previous job, we really focused in on building out compliance as code and doing DevSecOps with that focus on compliance, because it's often one of the easier parts of security to start with and is also often one of the most like manual labor intensive aspects. So at Pivotal, we build and operate platforms. We work with Cloud Foundry Foundation to build open source projects like Cloud Foundry, the CFCR, which is the Kubernetes runtime, and then we used to have RIF, which was our Functions Runtime, but now we're focusing on Knative, which is the Function Runtimes that run on top of Kubernetes. And then we take those projects and we operationalize them into our product, which is called Pivotal Cloud Foundry. And as I said, I'm not trying to sell you anything here, I'm just going to set some background and how we help build like security into kind of all levels of our platform. So we have this tool called Bosch, and Bosch is a platform that we built for us to run complex distributed systems. And it kind of takes the best of like tools like Antibol and Chef, Terraform, and a bunch of other stuff, building golden images and all that, and sort of wraps it all up together. So it performs both the orchestration and the config management. I required to kind of create that immutable infrastructure in the form of golden images that it deploys and manages on like VMware and the big public clouds. So like Amazon, Google, et cetera. Unfortunately, I don't think we have support for Alibaba Cloud or any of the local clouds. And so not only does it install, but also like monitors the infrastructure and repairs the infrastructure if something changes or something goes wrong. So it's kind of not just that day one, but it's also the day two. And the day two side of things is where you can build in like automated security of things. So like if a VMware fails, if a VM fails, like Bosch sees that and will remediate it, replace the VM, et cetera, same with applications. And so like as a human, I could mess with a VM that's managed by Bosch, but Bosch would just show up and like destroy that machine and recreate it, right? It's not gonna try and fix it. It's not gonna try and undo what I did. It's gonna be pretty destructive on that machine. And then the platform on top of Bosch is pretty resilient, so it handles that okay. So I could SSH into a VM and change a setting, Bosch is gonna destroy it. Now, not just me, but also like if someone manages to find a vulnerability in that system and someone gets some malware into that system, Bosch is gonna come along pretty quickly and say that system doesn't look right. I'm gonna destroy it and create a new one. So there's less time in which a piece of malware or a piece of like a bad actor can be on a machine before it gets destroyed, right? So a lot of, you look at a lot of the big hacks that have happened. It's when a bad actor has got some malware onto a machine, I just sat there for months, sometimes even years, just looking around at the network, looking at the infrastructure and looking into getting passwords for databases and stuff like that. And eventually they find enough things that they can then grab all of your passwords, grab all of your credit card numbers, search and security numbers, whatever it is they're looking for. So by removing the amount of time that they can be on that system, it works. But Bosch is kind of, it's a big deal to us, but it should be an implementation detail to most of our customers. They shouldn't have to care about it. They just kind of use a more simplified interface to get systems running. So at Pivotal, we run a number of security compliance tests as part of our CI process and also part of our CD process. We follow the CIS compliance plus a bunch of the PCI and other stuff and we can make those available to people that wanna make sure we're compliant. But any updates to an application basically results in a complete rebuild and deployment of the immutable infrastructure in that platform. Obviously we manage the state and databases and stuff. And we provide automated deployments and updates as soon as we release them in the form of CI pipelines. So we have a lot of customers where when we release an update, they automatically consume that update and automatically upgrade their platform to the newer version that doesn't have that vulnerability. And that happens both under the platform and also above the platform with their actual applications that they're running on top of the platform. And we do it with canary deployments and stuff. So it's a pretty safe operation and we do a lot of validation to make sure it works. So basically the entire platform is actually frequently rebuilt and redeployed often several times a month. And we actually have customers, large banks, that actually forcefully repave their machines like every couple of days. So the longest a piece of like a vulnerable software or a piece of malware can be on a system is just a couple of days. And then we also do things like automatically rotating credentials and certificates on a scheduled basis. And so we call this kind of the three Rs of security, rotate, repair and repave. So if you're using Cloud Foundry today, you kind of get all of these things and it helps you have a secure, a more secure system. But most people don't have Cloud Foundry. And the people that do have Cloud Foundry also have a lot of other infrastructure that's not managed by Cloud Foundry. And I'm not gonna tell you to go and write Bosch releases and start running everything through Bosch yourselves because that would be like, that's a very large hammer. And it's not really something that you wouldn't necessarily need to do. And you already have probably some sort of automation tooling in your system, like that you use in your operation system. So like Chef or Puppet or Ansible, Terraform. So you can use those and add on to those to start bringing some of the secure and compliant stuff. So we'll talk about some of the practical steps you can take no matter where, like what sort of infrastructure you're using. And we do that by applying calms to our security or as I'm gonna be specifically talking about compliance. So culture's kind of a hard thing, right? It's at large companies, like the culture of the company is what it is. And it's very difficult as a practitioner to try and change it. But when you have a good culture, sorry, so the thing we have with large companies, any kind of companies is like behavior is what basically sets your culture and behavior is driven by incentives. And those incentives are set by like the executives at your company. So it really needs to be like to really change your culture. It needs to be a push from the executive level down. And it also needs to have like a push from the ground up as well. You kind of meet in the middle and often the people that really fight a culture change in a company is like that layer of middle management who are very comfortable where they are. And so you're gonna have to squeeze them into like doing what you're gonna do. Individual teams can have cultural elements that are much better than the larger company. But they're really driven by members of that team. And if those members of that team leave or something happens, that like better culture bubble that you've built goes away quite quickly. So it's really important to try and get your company to like focus on like a larger cultural shift if you don't have a very strong culture that's not amenable to like the calms type framework. So when you do have a DevOps culture, you create high performance teams. High performance teams have clear and effective decision-making methods, engaged leadership, high trust and clear communications. They accept failure and they embrace failure and they adopt things like blameless postmortems. They give autonomy to teams and their members. And this all helps create these high performance teams where everyone is taking ownership and responsibility for the business goals, right? And there's some really good information about adopting a DevOps culture. And there's the book Accelerate by Dr. Nicole Forsgrim and some other folks. And they also do the DevOps, the yearly DevOps report. There's a lot of amazing information there about how companies are making a culture change and how they're adopting DevOps type things and also how they're failing to adopt DevOps which is often the more important information, right? However, other people screwed up so I don't have to make those same mistakes. So there's kind of culture. As a practitioner like me, there's not a time I can do to directly affect the larger culture of a company I work for, especially if that company is like 100,000 people or even 5,000 people working at a company. It's very difficult. But there's still things I can do. You have lean. So right before you start any journey, you have to have a map. And this is the map of the flat earth because I think we're all agreeing now that we have a flat earth, right? JJ, you agree? Yes, great. So if you wanna measure success, you kind of need to know where you are and where you wanna go. And so like from a compliance perspective, how are you currently implementing compliance and where in your business are you implementing compliance? And you can use this thing called value stream mapping, which is kind of inside the lean framework to create these maps. So you're gonna take a few days to map out your entire process and your current state. And then you kind of look at that and it becomes pretty obvious where your waste is, where your bottlenecks are. And then you start to rebuild what you want your ideal state to be. And so you basically have your existing state and you have your ideal state. And then you can start mapping out what it is you need to do to get from one to the other. And you can do that in, depending on the size of the processes you're trying to map out from a couple of days to like a week or two, it's a very large IT org with very complicated like server provisioning and stuff. It might take you a couple of weeks to get a really good map up. But you can kind of start with like some like summary maps and kind of dive deeper as you're going. And then you can, then you'll wanna like perform, like create a set of actions and activities to start moving towards that future state. And you can do that by turning those into like user stories on a Kanban board. So in my experience, we have kind of a few places we can attack wasted time and security and compliance. When you're actually like provisioning infrastructure, well, it's actually like when you're writing software over here, I didn't include that. There's when you're writing software, there's when you're provisioning infrastructure, there's when you're actually releasing software, so when you're actually deploying it to production. And then also like continuous compliance audits. As someone who comes from an infrastructure background, obviously this first section is what interests me. But as you start mapping it out, you'll figure out where the most wasted time is, often it is in infrastructure provisioning and you start thinking about how you can overcome that. We also wanna think about these other aspects because if you think about them when you're building out how you're doing infrastructure provisioning, it'll actually like bleed into the others and help you solve the others later. So this is a very abbreviate value stream mapping. I'm gonna just go through these pretty quick because I should actually, so Kanban boards are super important. It's like basic like to do in progress done, anyone can do it. So here's what happened at a previous job. We had these Excel spreadsheets and we had this security and compliance team that would take this Excel spreadsheet contained like every setting on a server that needed to be set in order to be compliant, right? And there was hundreds, like five or 600 like cells of settings in this, of settings. And they would go through every single server line by line and make sure it was compliant and they'd mark if it was compliant or not and if they had to remediate it. And then when they were done, they would save it off and they would move on to the next server, do the same thing again. Now we had a couple of thousand servers and so that took a lot of time. And so by the time they got through them all, they would just have to go back to the first server and start again. And there'd probably be an extra couple of hundred servers that had been provisioned in the meantime. So we had, there was like three or four full-time people that were just doing this job. And so that was a lot of wasted toil, a lot of wasted human capital. And coming from a DevOps background where I had been doing a lot of work around automating a way, wasted labor and toil, this was like, this is cool, this is a problem I wanna help solve. And so I really dug in with them and I introduced them to a couple of things. So first of all, we were already using Ansible on the infrastructure side, the DevOps side to build out our servers, to install our software and to make sure what we wanted was what was in place on our servers. So I said, let's have a look at this Ansible hardening thing, which is a set of playbooks that were originally developed at Rackspace for deploying secure OpenStack installs. And it basically went through and implemented the Stig controls. So that control I showed you earlier, it implemented all of those into Ansible. So you can more or less run this against any modern Linux system and it will basically make sure that that system is compliant with Stig. And so I said, this is really great. And while we have our own internal compliance, we don't necessarily follow Stig, there's probably about 95% of it will match, right? And so let's implement this and then we can dig into it and see what's different. And so we did that. And so I said, I will take that on. And while I'm taking that on, I got the security compliance team to look through it and look through Stig and look through our compliance and figure out what was different and what extra controls we would need to create. And then we had InSpec, which is a tool that came out of Chef, but you don't need to be running actual Chef as the config management software to use it. And basically it basically uses RSpec to do compliance testing. And it's basically a grown-up version of service spec if you've ever used service spec. And we were already using service spec for some other things, so it made sense to use InSpec. And so it's a Ruby-based DSL specifically designed for testing compliance. And so because we were already using Ansible to Automated Linux infrastructure, we kind of looked like this though because our security team wasn't using it as well, right? So they were just trying to shovel up the rainbow poo that we were pooping out as DevOps'es. Rather than just like dumping them in it and telling them to sink or swim, I'm like, how about I take on the Antiball hardening and I get that working, sorry, how about I implement the Stig, so the testing, and make sure that our systems are compliant and testing. And while I'm doing that, you guys take the Antiball hardening and validate it on some of our test systems and make sure it's not gonna break anything, right? Because any time that you're changing settings on a system, you need to make sure you're not breaking anything important. So we did that and it only took me about a week to get the testing to all of our infrastructure. And it took a little bit longer for the security team to do it. So again, that's the Stig saying SSH needs to be version two. This is what the InSpec control looks like. So really simple, some tags and stuff, but this is the important bit. It knows what an SSH config looks like, so it knows how to check for that. And then this is some of the extra compliance we built out. And interestingly, they built out some additional tags to link to how to fix it and also why it's set that way. And that was really useful because we tied it into our monitoring and logging. And so now whenever a server was out of compliance, they would get an alert and not only would they alert say something was wrong, but it would give the exact, exactly how to fix it. And then also it would provide logs that we could use for creating compliance documents for spot audits and things like that. And so we had, we were monitoring for it, we were logging for it. And then we implemented the Ansible side of things. So now we're making sure when we're building servers, they are compliant. And so that was kind of getting the basic compliance into our Linux systems and it worked really well. And actually how we knew it worked really well was security and compliance team on their own added this to our CI infrastructure so that anytime we made a pull request to our Ansible code or whatever, it would actually run through all of the inspect checks and make sure that our changes still resulted in a set of servers that were compliant. And so that was great because we immediately got fast feedback on our changes. And so that's really hitting that like DevOps type workflow. It was great. We now had automated testing of all this stuff. So those four people that were just smashing away on keyboards were then freed up to do other stuff and to work with us and improve the rest of our stuff. We were able to measure it. And so the easiest measurement was how much wasted time were we saving? And that was like four people's worth. And so they were able to go to their management and say, we've now saved tens of thousands of dollars on labor and we're actually more effective at what we were doing. And so that was really great. Gave them a lot of trust in the org and I am out of time. So sharing stuff, be nice to share. And there's a ton of other stuff that you can automate like I did with compliance. I've got some stuff there, but basically almost any security team tool you're using you'll be able to automate and put into your continuous integration pipeline. And we are way out of time. So we probably don't have time for questions but I'll be hanging around in the hall later. So thank you.