 My name is Svavek Ligus. I work on the PCI security team in Cloud Foundry, San Francisco. And I'm here with Mark de Kona, an exquisite archer, and our product manager. OK, it's demo time. Here, the screen is divided into parts. So on the right side, we're seeing data as it leaves the client on its way to the server. The other side is the same data in transit. This is the on-the-wire view of how it travels. Textual user data, the chunks on the left-hand side can be clearly discerned here. That's not good. This is because the designers of the application did not inherently incorporate transport layer security into their problem. So what we would like to demonstrate today is what it takes with Pivotal Cloud Foundry, Bosch deployment with an add-on, to close that particular gap. So I'm going to upload the IPsec release, larger screen. Then I'm going to set the configuration for it. The config looks roughly like that. Yes, to kick off the deployment. So within moments, the left-hand side of the screen should become unintelligible. A little bit like that, but not quite. So if you pay attention, the chunks you see on the left-hand side are all the same. Can't possibly be the data from the other side of the screen. That's because one of the servers has already gotten up and speaks encryption, but the other one doesn't. So as soon as the other one catches up to the first one, they'll be able to communicate. And what we're expecting to see is a consistent stream of encrypted data. What we're deploying here is the IPsec add-on. This is the famous in some circles. It's been mentioned in a few talks yesterday, and infamous in the others among our early beta customers internally. Let's look at it for another few minutes. Seconds. And that's what we're looking for. So encryption of data in motion is only one of the many requirements of the PCI DSS compliance. Mark will tell us more about what PCI is and about its significance. I'm going to take you through two things, and then Slava will close. First, a little background about PCI, why it's important. The payment card industry, PCI, brought us the DSS standard, which is one of the foremost standards that we use a lot of the financial companies required to use to comply with that if they process credit card data. But a lot of customers are using it anyways as a guide to what to do. Because if you're in the Fortune 1,000, and this is a view of how the Fortune 1,000 is connected, a lot of our customers are in this. But if you look at what's happening today to them, they are under attack. If you saw the blog from Microsoft last week, they're experiencing on their cloud service 1.5 million attacks per day. We get 1 million new malware pieces out every day. And for those of you who have discovered a hacker in your enterprise, you'll know that you discovered on average that hacker after 205 days. If you didn't have logs, audit logs for a year, you didn't even know what they were doing. So all of these things are pretty important. Hackers are succeeding mostly because we haven't made it easy enough as a software community to automate and to defend against these things. So we get a lot of customers coming in asking us for help, especially now that Cloud Foundry has the platform on which to build upon. And you'll see some of the Bosch add-ons help to do this. One of our customers knows exactly which 14th floor of which building in Guangdong, China. The hackers attack them 9 to 5 every day. And we have another customer whose goal is to repave their entire IT system every day. New IP addresses, new credentials, and that'll give the hackers only one day if somebody manages to get in. So we're not there yet. So what are we to do? The good news is really the recipe to defend yourself hasn't changed. The bad news is we aren't doing it all the time. And so hopefully, I'll take you through the left-hand side of this, some of the things that we're doing before the fact. And on the right-hand side, some of the things that you need to do once you are running in PCF or in CF. One thing I want to say up front is we are all on a journey to open source. For those of you who don't know me, I grew up in Africa, hence the archery. And this is some wisdom that I always carry with me. Sometimes we build closed source because we have a customer that we've committed something to. Sometimes we build open source like Kred Hub and Kubo. And sometimes we move stuff from closed source to open source based on discussion in the community. So feel free to discuss and guide us. I want to talk a bit about the payment card industry. For those of you who remember the 70s, Visa and MasterCard didn't exist, at least not in the early 70s. In fact, this is what you would see. It started out in the 50s with Diners Club, American Express followed, and then others after them. And people liked the fact that you could use a card and not have to carry cash. It was empowering, but it really wasn't until the 1990s that you and I got credit cards, multiple credit cards, and now debit cards. And further it wasn't until 2006 that the PCI, the payment card industry, set up a security standards council with DSS to manage the DSS standard, and we're now at 3.2. So what is this standard? It is actually several things, these 12 requirements, and they go into more detail after that, but they're pretty much reasonable things that everybody should be doing. Have a firewall, right? For those of us who forgot about antivirus, until the Bansomer attack last month, we all remembered suddenly. So being able to do file monitoring, file integrity monitoring on your system, being able to do antivirus and encrypting your network traffic are three of the most important things, and those were deficient in Cloud Foundry, our customers asked us to help, and we built add-ons for that. So the things that we do outside of this, one of the things that we do is threat modeling. And again, this is not on every piece of Cloud Foundry software at the moment, but it's coming, it's getting spread across. We do threat modeling to identify the data assets, track the flow of data through its lifecycle, and then find the threats and mitigate against them. And every piece of software you build, you should be doing this too, because it's one of the most well-understood ways to protect yourself. We also do, because our customers before us have been doing a system scanning with tools like Nessus. So we now scan Nessus in a concourse pipeline to find vulnerabilities. And for all the open source software that is in Cloud Foundry, we scan that as well with Blackduck, identifying vulnerabilities, doing triage, and fixing them. The Bosch team is also doing system hardening. So as you know, Cloud Foundry uses stem cells. And stem cell hardening is part, a really important part of what we do. The reason that the Samba vulnerability that came out last week and last couple of weeks wasn't severe, it didn't affect Cloud Foundry, is that that service was not in the stem cell. And I'm gonna hand it back to Kslavik for the add-ons. I wanted to talk a little more about the work that the PCI team is doing. And what we put out there in the world is Bosch add-ons. We write three kinds of Bosch add-ons. So very quickly, short recap on what a Bosch add-on is. A Bosch add-on is a release job that gets deployed to each and every single VM managed by a given Bosch director. Example add-ons are an antivirus software, a health monitor agent, intrusion detection agent. So those are typically small programs, single release job being deployed, however, everywhere. So the point to take away for a Bosch add-on is it's super suitable if what you want to ensure is maximum coverage. And here's what our team is doing. The first add-on that we write and release is the IPsec add-on for PCF, the demo of which we've seen at the beginning. It provides network encryption and it works between two different operating systems between Linux and Windows. The other thing that we do to check off the PCI box, check box, is we deploy Clamav, antivirus. Antivirus gets deployed to a VM and then it pulls periodically for updates of the database. And finally, file integrity monitoring. The add-on inspects or monitors events to sensitive files on a file system and a suspicious activity is logged and can be alerted on. Okay, so we've worked a lot with add-ons and we've learned a bunch of lessons, but there's one particular one that I want to share. And that is to start, if you ever write an add-on, start by limiting resources. It's a little counter-intuitive because it was counter-intuitive for us. We started by implementing the prototype functionality then we developed the functionality and we added resource constraint as a feature. But what ended up happening is we undermined a little bit of trust with our early customers by well, having less than successful deployments and that also undermined the trust in our own software. If we just turned it around and if we deployed, if we started by constraining resources in a C group or in some other way, we would have avoided that. That's a practical side to it. But there's also a very good theoretical reason to do it. It allows you to reason about your software very easily. So if somebody comes to you and says, okay, deployment is failing, we think it's your add-on. You can say, sure, let's investigate that. However, I think it's highly unlikely because we could not have used that much CPU. We're constraining it. We couldn't have used this, that, the other. And when I say start by limiting resources, I mean all the resources your add-on is using, even if it's what seems like a constant resource. There is one thing that happened not so long ago to us. We constrained the CPU, the memory and the IO of one of the add-ons we were developing, but we ignored the disk. The reason being is it only used a constant amount of disk. So no big deal. Disk can never grow, it's constant. But it's not constant when there is a file descriptor leak on a temporary file that gets created and deleted over and over again. So you do not see it on the file system, but you will see it with LSOF. The references to inodes will be kept and those inodes will not be freed up and you use all the disk. So that would be our lesson learned to share. Okay, I think we're good. We can open up the floor for questions. Okay, thank you very much. Okay, so the question is, we've seen some amount of downtime during the demo. Cloud Foundry deployments take a lot longer. Does that mean that we will have that downtime, we will witness all that downtime during a deployment of PCF? Thank you for that wonderful question. Answer is no. Not so long ago, we've implemented a two-step deployment that avoids this. What we've seen here is just a simple demo that had to be self-contained, so I could not correct for that. However, in a PCF deployment, we were seeing down times of about 20% of the duration of the deployment. So if your deployment is 40 minutes, you would have seen 16 minutes of downtime. That was the original implementation. With a two-step deployment, the downtime is contained to, it still exists, but it's on the order of single seconds. Sorry. I'm not sure I understood the question. Can you say it again? Okay, I'm unaware of overhead of that magnitude. The overhead, we've had different numbers, because it really depends on the IaaS that you're running on. We've had varying numbers from no overhead to about 37% overhead. And there is little that you can do about it once you've picked a provider and you have to run on that CPU. It's the price of protection and price of doing business. If you're gonna go to higher, you know, 256 bit versus 128, then you can make some trade-offs, but you do have some overhead, but in the world of PCI compliance, that overhead is worth paying. Sorry, that was the question here. So the BlackDuck scanning is primarily for the third-party software. The stem cell hardening, I'm not sure if we're using BlackDuck or an SSR combination of both, but kind of John is here. Yeah, so not for BlackDuck, not fucking. Question over here. This is throughput. Thank you for the question. So, okay, advantages. Let's talk about differences. The IPsec operates on a transport layer. So it injects itself into the network stack. The communication that goes on becomes encrypted between machines. TLS stands for transport level security, but if I'm not mistaken, it actually is application level. It's implemented at the application level, right? Okay, so in other words, first of all, you manage this complexity somewhere else, but most importantly, your developer has to think, the developer has to think and implement inherent security into their application. If they fail to do so, one advantage is to deploy IPsec and we're done. Well, we're closer. So definitely for applications that are being developed now, you would tend towards TLS and mutual TLS. For applications that were developed and you don't want to invest effort in it, but you do want encryption, you would do it at the network layer. And trust me, we've seen customers with applications where they've lost the source code. So there is no other choice. Without downtime. They can be rotated, but with minimal downtime, not without downtime. There's downtime on the order of seconds per cluster. And it's mostly not due to IPsec, but due to something inherent in, I think, HAProxy restarting or something like that. So there's always a little bit of downtime, but. Yeah, it exists. And we should point out that all of these, all of these things we're doing with PCI compliance currently on Linux VMs are also gonna apply to Windows VMs, which have their own constraints, along with can you have two certificates when you're doing a certificate change. So not all was the case, but we are at the moment in the process of getting these already to deploy on Windows cells as well. So it's very much the same as your desktop antivirus world, except in the case of you having a data center or having a group of servers that have to be updated, you probably wanna have a local mirror to improve performance. In the case of CloudAV, they update hourly minimum twice a day. We've got customers who've developed add-ons for Trend Micro and MacAfee that fit with their enterprise antivirus. Correct, yeah. Okay, we have another question over there. Okay, for data inspection, it is possible to analyze traffic through Wireshark if you have access to the keys. So you can do it that way. Does that answer the question? Okay, no, the tunnels are created between VMs and they create a mesh. So every VM with every VM establishes a separate connection. Okay, so I do the VM. Right. Okay, thank you. But you had another question there, right? Okay, cool. You can log in to network.pivotal.io and all of the add-ons are on there. Okay, thank you. Thank you.