 Hello. We are here to talk about advanced vulnerability management in Cloud Foundry. Sorry, thank you for coming. I'm Molly Crowler. I'm a technical program manager at Pivotal. I also am a member of the Cloud Foundry Foundation security team. Steven Levine. I'm a software engineer, and I do product management work also. And I'm the CF BuildPacks Project Lead for Pivotal. Cool. So what we're going to talk about a little bit today is Cloud Foundry as a secure platform, the kind of inherent nature of Cloud Foundry that makes it secure. Some vulnerability management process before and after this thing called DABOS that we're going to talk about. A sample DABOS workflow, and hopefully a call to action for all of you for how you can help make this a better tool. I just wanted to give a quick team kudos to everybody who's worked on the triage and automation security team. So far, we have a very big group of people who have kind of moved in and out of the team and have helped us along. That was us at a team outing, kayaking in New York City. So just before we get too far, this thing called DABOS that we're talking about today, it's not just the guy from Game of Thrones, and it's not just a town in Switzerland. It's a vulnerability management tool that we've been building internally that we want to start to share with the foundation. So we originally came up with the name DABOS, and then we're like, what is this thing actually? So it's also known as the Dependency and Vulnerability Overlord System, DABOS for short. It was built by the Pivotal Security Automation Team over the past 10 months or so. We got started at the beginning of this year. In a nutshell, it collects and stores data about vulnerabilities and integrates with Tracker to alert both open source and commercial teams from Pivotal about vulnerabilities and to collect information back from them about whether or not they're vulnerable to that particular issue and what versions they've fixed that issue in and released. We're talking about this today because we think this is kind of a new thing. We haven't heard a lot about security management and vulnerability management specifically integrating with Agile workflow, and that's what we think is cool, and we're going to talk about it. So why did we decide to invest in building DABOS? Just to give a little bit of background, Hartlead, which was disclosed in 2014. This was actually before I joined Pivotal. It was a vulnerability in open SSL that affected most commercial traffic on the internet. Some of our stem cells were affected, but not all of them were, and it was also kind of the first major public vulnerability that happened after Cloud Foundry came about. But before the foundation or even Pivotal really had a process in place for disclosing vulnerabilities. And then the next thing was Dirty Cow, which happened last year. It was so bad that Linus Torvalds actually patched it himself. He said that he had tried to fix it 11 years prior but just wasn't able to do it. It was one of the first CBEs to get a live patch from Canonical. All of our stem cells were affected. And we were able to release new patch stem cells a day after this went public. So it was kind of proof of concept of Cloud Foundry as a platform that's able to patch Linux very quickly. And also this one was unique in that when it was disclosed publicly, it was done with a website, with logos. They had a wiki set up. They had t-shirts printed. And we kind of figured that this sort of thing was going to happen more and more often that we get very, very public vulnerabilities that basically go viral to a wider community more quickly than you could imagine. Because previously, you think of, oh, there's a vulnerability in Linux who's actually going to care about this. But this was picked up by the news. I'm assuming people ordered t-shirts. So this was just something that we knew that in the future we were going to have to worry about. So we're quickly going to talk about Ubuntu security notices, which we also refer to as USNs. They basically are notices that are put out by Canonical about the Ubuntu distribution of Linux. And a USN usually contains one or more CVEs. So because the Bosch stem cells and the root FS rely so heavily on Ubuntu, it's easy for us to patch a lot of things quickly by what we call basically patching for a particular USN. So this is what the process looks like in theory. Somebody in the open source patches their package that gets put into Linux. Canonical updates Ubuntu with a new version of the package. We, as quickly as possible, if it's a high severity USN, we patch the root FS and we patch the stem cell. We make some sort of public notification on cloudfoundry.org slash security to let everyone know that these new root FS and stem cells are available and that they should pick them up. And then in a kind of commercial context, the pivotal runtime patches with the root FS and the stem cell, our other services patch as well. And then we make our own pivotal focus public notification that these versions are available for customers to pick up. One thing to note with this is we kind of go through this process on a relatively regular monthly cadence to pick up all kind of low and medium severity USNs that are released by Canonical. But this process has to run with relatively little notice and much more quickly when they release a high severity USN. And hopefully our customers upgrade. So what's the problem? Why couldn't we just run this and get really, really good at doing this in a repeatable way? One thing is that, as I said, high USNs need to be patched faster. And as you can see from this graph that I made, this is just showing the number of high USNs all time that have been released on a particular day of the month. They tend to be very variable. We can't ever expect any particular day a high USN could come out. We've definitely had complaints from product teams like, why can't this happen more regularly? And it's like, this is completely out of our control. It's somewhat random and it happens when it happens. So we've had several situations where we start patching for a high USN and we're in the middle of the process and another one comes out and everybody has to throw away all the work they did and start over. The other problem that we've seen is that public notification is really hard. I've obviously personally received a lot of feedback about why is this notice up? Why is that notice not up? Where is it? It's been a long time. What's going on? I see the pivotal notice is up but the open source one isn't basically runs the gamut, anything you could think of to complain about public notification. People complain about it. The other thing is that we don't have enough resources to put out a public notice for every single CVE or USN that comes out. There are a lot of things that don't affect us that we don't notify people about because if we just did that, that would literally take up an entire team's complete time. So that's also a problem that isn't completely solved just by having somebody responsible for putting up public notices. One last thing that throws a wrench in everything. I like to go on vacation sometimes. USNs come out while I'm on vacation. This has happened plenty of times. Or other reasons to be out of the office volunteering in Chrissy Field, which we've done. I think a high USN actually came out that day while we were picking weeds. I also have a really cute dog who sometimes wants me to not work on the weekends. So there are human problems in this process also. So we went through what it theoretically looks like to patch the stem cell for USNs. To give you an idea of what it really looks like, this is the Ubuntu security notices RSS feed. This is basically predavos what we were doing a few months ago. You get the feed. We did have some automation built by the BuildPacks team that would feed that RSS into GitHub issues. We actually have a private Pivotal repo and a private open source repo that both would get the notifications from Canonical. I would look at them. I would look at the Ubuntu website to figure out what the severity of the USN is. There's a little bit of calculation in that because a USN contains many CVEs, you have to look through all of the CVEs. And whatever one has the highest severity, we just assign the USN that severity. So I was in charge of doing that. I would go on Slack. I would poke all of the teams that needed to update either. Hey, Dimitri, we have a high USN. Can you please start patching? He would usually say, I already knew about it. It's already started. But you got to do it just in case. And then try to coordinate over dozens of teams who have to patch the commercial distribution. After they hopefully received the notification, maybe if they're in London, they received it the next day. That happens a lot. Then they would go to GitHub, take a look at that issue, make sure it really affected them, go into their tracker, make some stories that I didn't have a lot of visibility into, and hopefully get done with the patching work and eventually report back in GitHub what versions of the product they actually fixed. While this is going on, I would be waiting for the new stem cell and the RudaFest to come out. I'd be waiting for teams to finish their stories. And I also would spend a lot of time basically hand rolling every security notice that has to go up on cloudfoundry.org or pivotal.io. It's a lot of work in Google Docs, contacting marketing teams, putting stuff in WordPress, et cetera, et cetera. So that also takes a decent amount of time. And then putting the notices up on cloudfoundry.org for everybody to consume. So after I had been working on this for about six months or so, I started to think that maybe there could be a better way that didn't require all this manual work and all of this kind of cross-team, inter-team communication. So kind of we imagined what would this look like if we had some system that was doing all this for us, potentially even all the way to putting things up out for the public to see. So that's kind of where Davos came from and that's what we're gonna be talking about. And Steven is gonna go through what the workflow looks like for actually patching the root FS and how they do that. So I'm gonna rewind a little bit and talk about this process from the beginning. So while Molly's hard at work with the organizational response to these kinds of vulnerabilities, the engineering teams are hard at work, or hopefully not, because it's automated, responding with patched versions of the root FS and stem cells. So I'm gonna talk about the root FS first and that's the underlying user land files and things that you can think of it like a base image that Cloud Foundry uses for application containers. So the root FS patch process is basically all automated. There's a pipeline that looks at the feed of USNs and when it finds a new one, it starts building a new root FS for Cloud Foundry. It goes through a bunch of steps, validating dependencies and repackaging things. So we have a root FS for even works on Docker or the works on Cloud Foundry or all these different places. And it takes about two hours from when a USN hits to when our pipeline is finished with it and a Cloud Foundry operator can deploy into Cloud Foundry, it's very quick. It's only one manual step where I just say, okay, it looks good and it goes through, it's awesome. It's really important that this is fast because a big advantage of Cloud Foundry is that operators can take a new root FS and upgrade it on their platform and every single app, the operating system level dependencies for all of those apps are suddenly up to date and it happens live and in production and very quickly. And it's sort of a major advantage of Cloud Foundry over things that use immutable container layers where you have to rebuild each one from scratch to update the operating system level dependencies. So the fact that that's fast is really helpful. Now, not every one of these USNs is really relevant. This particular vulnerability in the eject, which is for CD drives that application containers don't really have. Maybe it could cause someone some problems, but for the most part, it's not terribly important to patch that in a Cloud Foundry application container in less than 48 hours. So the real challenge here is how do we get people to pick up the new root FS and teams and people that depend on it when it comes out and how do they know that, yes, they really need to update it immediately. At least that's the remaining challenge. And I'll talk about the stem cells a little bit too. It's a different team from what I deal with, but just quickly, the stem cells, the stem cell patch process is similar. They use a pipeline that automates parts of it, but the high kernel CVEs are really common and those don't affect the root FS because there's no kernel to deal with, but they affect the stem cells and they result in a lot of stem cell rebuilds that have to happen very quickly. Stem cells need to work on a whole bunch of different platforms. You have to build them on VMs, which is slower. So you have to build a lot of things and on slower systems. And a bunch of things the Bosch team knows about that I don't deal with too much, but I'm sure it's very exciting. So how do Cloud Foundry vendors, teams and operators know when to patch the root FS and stem cells? Is the real question for USNs? And more generally, how do they know when to patch anything if there's a security vulnerability that affected it and could affect their users? So next I'm gonna talk about Davos, which is the system Mali brought up that's responsible for sort of solving that. You could almost say it's a last mile problem. So there's a high level overview. Davos takes sources of data like USNs and new versions of really security critical dependencies. And also notices that we create where we realize, oh, this Cloud Foundry thing is uniquely vulnerable for some reason or whatever. It collates all that data into this sort of source of truth, this is black box, white box, I guess. And it delivers those notices to Cloud Foundry team Pivotal Tracker backlogs. Then it picks up data for how they responded to those notices from the stories after they deliver them. Keeps those stored in a source of truth along with the notices and also keeps track of the relationships between products so that it can create more notices of, hey, someone depended on this and it was fixed in another Cloud Foundry product and now they need to pick up that Cloud Foundry product afterwards, it's sort of smart about making sure that people get just the information they need when it's actionable. So I'm gonna turn over to Molly after this for some more information about what's coming next. Cool, so we've done a lot of work on Davos and it's mostly been in a Pivotal context but it also does affect open source teams. So open source teams are already receiving notifications for especially what Steven mentioned with new versions of Golang and Gen X, a couple other libraries that people depend on and they're able to communicate out more quickly when their team has fixed something. So what we want to do next is make Davos a little bit more generalizable to the full community so the full ecosystem can really benefit from it. We've been talking about potentially bringing it for incubation in the foundation which we need to do more follow up on basically. We also want to make it possible for other foundation members to be able to deploy it. I could imagine a world where other members of the foundation deploy Davos. They're able to run a script to basically import all old information about Ubuntu security notices and have access to all of the open source data that's currently in Davos that we'd love to share with our partners in the foundation. Other suggestions and feedback are definitely welcome. You can reach me at mcrowtherpivotal.io, actually mcrowther at cloudfoundry.org as well or same handle in the open source Slack. And that's about it. We'd love to take questions and thank you all for coming. So it sounds like the question is what about third party dependencies? Like other third party dependencies? Yeah, so one of the other things that we're doing that we didn't really get to too much in the talk is that Pivotal is actually scanning a lot of open source, basically open source Bosch releases using Blackduck which is a CVE and dependency scanning tool. And those are actually already also feeding into Davos via their API. So a lot of the open source teams are already receiving basically Blackduck notification saying like you have this old version of Ruby, it has the CVE in it, this is the severity, please patch. And then it will take information back about whether or not they, like we're actually affected by it because Blackduck can have a decent number of false positives because of this kind of fuzzy logic. The other thing that we've enabled is a workflow to allow teams to say that they are not planning on patching something in particular either because like the version of the Bosch release we're scanning is going into life or the CVE is in something that isn't really exposed in customer systems and like it's only for testing or just something that for some reason they can't patch. So we're both hitting the front of the process where Ubuntu and people are releasing new versions and we're also hitting the other end of the process where like what's in the code that we've already shipped. And also to add to that, we also have functionality in Davos that reports on new versions of really security critical dependencies. So if there are things that have vulnerabilities in them often, then we'll often just tell teams when there's a new version at all because they should always keep them up to date. So that helps with that type of open source library that security critical kind of those kind of situations. Any other questions, concerns, complaints? Okay, thank you.