 All right, let's get started. My name is Sergey Madashkin. I'm Distinguished Engineer at Comcast. Next half an hour, I want to focus about removing friction between security and DevOps to essentially make DevOps to be able to deliver products faster. I'm a proud member of an application platform acceleration team at Comcast where our mission is to accelerate and our product is the developers experience. We measure the developer experience in productivity and happiness. So these are our KPIs. We do whatever it takes to reach our goals. We provide services and platforms. Cloud Foundry is just one example, but far beyond that, to the DevOps team so they can leverage those platforms. We are trying to connect different teams in large organization together so they can work together on addressing cross-cutting concerns and again do it more effectively, efficiently. Also we are studying high-performance teams in Comcast and spreading their models of brandy on the whole organization. So why security? Security is all frequently considered to be in the path of velocity. So very often we are using or between them when we describe them on the same sentence. Because very often we need to make compromise between security and velocity. Even in our day-to-day life, when you're thinking about security tools like locks in your door, speed bump on local roads, or speed limits on highways, checkpoints, gates on airports, and borders. Those are all tools that slow you down. When you want to move faster, frequently you need to compromise. Somebody might decide not to lock a door and say, if you know a few seconds of not doing that, there is a consequence, lower security. So that compromise you might decide to take or not take. I mean, if you're living in a gated community, might be it is okay but then again there is a gait. So how we can change the narrative and make security and velocity work well together? So essentially remove compromise and help security and velocity to improve each other. At homecast, we treat security as our not necessary top priority. It's really outside of priority list. It is job number zero. So it needs to be baked everywhere, and that counts with a price. Price, just one example. Yesterday, before keynote, I was chatting from Pivotal, Ruth McClain. I think she is from London office from Pivotal, and she was asking, so are developers committed to security, and how serious they are about it? My immediate answer was, yes, they are. Then I thought for a second and I say still, yes, they are, but, but they would be happy not to do that. Right? So they are committed to security, they care about security, but they would be happy to do something else. Maybe some even willing to sacrifice a small animal for not doing that. I'm not sure, but what I'm sure is that they would do extra steps not to care about security. So it does come with a price when we are saying that the bar is that high that security needs to be baked everywhere. Also, it does not come with developers' happiness, and developers' happiness is a developers' productivity. I'm not going to ask how many of you enjoy patching because I don't think it's a really interesting question, because when last time you hear from USCA, great job on all patches applied. Was it this morning or never? Yeah, nobody likes those long nights applying patches. He's last year, mid-last year, I was chatting with my boss, Greg Ather. He's, I don't think he needs at this point any introduction. Everybody knows Greg already. So he was calling me and saying, he has access to all kind of data, right? So he was calling me and saying, Sergey, how much, do you think how much time our developer and operators spent on patching? I was like, I don't know, Greg tell me and he was all right. They're spending a percent of time just on patching, and that number was a mind-blowing to me. I'm not in a position to repeat it, but that time when you multiply to all the teams, all the developers, and the time they spent on patching is times they don't spend on delivering services to our customers. Many of you are here. So that clearly was a problem that we saw, and what we wanted to do is change this problem into an opportunity. An opportunity to think how we can apply security differently so it will not, it will be less expensive, more productive, and have our developers happy. On this thing, we start to think about security by default. So how we can bake security in basic practices and tools and systems that we have. So developers do not need to care about it, and they will need to care only about their applications and customers. So that is a frictional security, remove friction between security and developers and make them to be more productive, move faster, with less efforts. So just a little bit deeper into make security by default. So Cloud Foundry is one of the tools that we provide, but the practices that we have around them are very significant. So we need to bake security in all of them. So essentially, security will be given as basic rights or as breathing care or contributing to open source, and the same security will be available to all the developers. On this journey, we've been really inspired by Justin Smith's points about Cloud Native Security. I think he delivered it first time about three years ago, and I'm not going to repeat his points and his approach. But just one example. At Comcast, we have a lot of systems that are designed for PI, for PCI, for other three-letter acronyms, and on those systems we are writing agents that are monitoring all the files, calculating checksum and compared to what it should be. Every time when there is a mismatch, a file is changed, that's an alarm. So somebody needs to see it and act upon that. So that system is all about being static and resisting to change. But in our agile world, we want to move forward, we want to change, we want to change fast. So traditional approach to security is resisting changes while what we want to do is promote changes, do it through automation, do it more proactively. In this example of resisting and promoting changes, it is really 180-degree turn. So this is a pretty drastic change from where we are, where we were and where we are moving. So this is not an evolution, this is a pretty drastic change. Just introduced three pillars, how practically can security be implemented? Next, I want to talk about how we are implementing those pillars to make security available by default to reduce friction to make developers more productive. The first is patching. So that's what we started. How we can not care about patching. When you are running systems like Cloud Foundry, you already have all the components and have automation on development process. You already have all the components for that. So for example, if you already have a pipeline that takes your Java code from Git, build, test, and deploy to Cloud Foundry, then deployment to Cloud Foundry is actually combining your code with runtime delivered by Java BuildPack. And if Java BuildPack is patched just before your new release, then you automatically have that patch for free. That means essentially the more frequent you do deployment, the quicker you get patched, right? And the patch is going for free. You don't need to think about it. Was BuildPack patched? Was Java patched? You just focus on your application, you deploy, and if you deploy it frequently, you get free patching. Even better, if you have already that pipeline and you don't deploy every day, but you want to have zero-day protection, then just in your pipeline, introduce watching for Java BuildPack change. And when there is an update to BuildPack, trigger your pipeline the same way you are triggering it when a new code is updated, right? But what we care about the whole stack not just an application, and with Bosch managed platforms, caring about the whole stack is also quite an easy task when it's automated. So if, for example, on Bosch deployments, Bosch deployments uses stem cells and releases. So if your monitor changes on a stem cell and release and you already have a continuous delivery pipeline for your target platform, Cloud Foundry, CFCR, your custom applications, you name it, then you already have all the components to trigger that pipeline and do test and deployment every time when stem cell is patched or release is patched. So if you already have all the components, like powerful platforms powered by Bosch, Cloud Foundry, Kubernetes, and you have automated deployments, then you can make patching free. And for us, that was a huge win because it release, not just release developers' time, but release less productive developers' time to make them what they really want to do. Somehow I switched, I'm not sure, somehow I missed one slide. Yep, so next was one was on repaying. Sorry, I don't have slide. Actually, I have. They moved somehow. Yeah, they moved somehow, right, sorry about that. The concept of repaying is, if malware already somehow made a pass to your system, to your application, then just redeploying application, redeploying means completely removing the container and creating a new one will remove that malware. And if you do this kind of repaying often, then for attackers, there is no time to get in. And again, we can implement it for the whole stack pretty easily if we have the right tools. You can do Bosch recreate and recreate your environment in a safe way with, you know, canary deployment and gradually replacing all the VMs and repaying all the VMs, as well as an application you can use different techniques like, you know, kill individual instances and let's cloud foundry to recreate those new instances. Using self-healing capabilities of tools like cloud foundry or Kubernetes. And Bosch has self-healing capabilities as well, right? So we can just randomly kill a VM and it will be recreated, which gives us an opportunity to combine repaying with chaos engineering. What if we will just randomly kill different VMs in infrastructure and let them recreate, be recreated and that recreation process completely repays and restarts everything. So if there is something not right on those systems, it will be removed. There are open-source solutions like the Chaos Lemur for Bosch that can be very effectively applied for this. Again, automation, self-healing, and then you can have best of two worlds. Chaos engineering will keep your environment and applications resilient, anti-fragile and ensure that it is tested. And at the same time, we will have after repaving and ensure that our environment is clean. It is the way how it is supposed to be. All right, going backwards. The last pillar of the last component is rotate, is about rotating credentials. And this is the hardest part for us because credentials we have everywhere. We have credentials during development cycle, GitHub, artifact repositories, docker registries, as well as during runtime. Applications come, talk to different services. They talk to databases and everything. Everything is based on authentication based on credentials. Those credentials very frequently shared. Shared credentials means sticking out on my laptop, Excel spreadsheets and over email. Hey, we have updated this password. You need to update it in your system. And that communication between teams are happening all the time. So shared credentials is here. We cannot rid of them in one day. But when I start to think about it and apply to some tools that we already have, like credentials, centralized credential management, Cret Hub, HashiCorp Vault, right? So at least we can somehow consolidate and keep secrets in a good secret place rather than on Excel spreadsheets. But even better, when we start to hide that credentials or a centralized delegation, we can rotate them centrally, which makes rotation much easier task that it used to be, right? Because it used to be one in the morning call, we need to have people from team A, team B, and team C, and at the same time change passwords on three different systems, verify. Five in the morning, everything looks good. Maybe, maybe not. So not very satisfied. A job that doesn't bring satisfaction, it takes time, it takes human doing things when robots can do that. So centralized credential management can facilitate rotation. And HashiCorp Vault did a great job with integrating different systems. So essentially, for example, they have Kubernetes authentication and they integrate it with the different databases, like MongoDB, Oracle, MySQL, you name it. So you can have a very easy pipeline of a flow from Kubernetes hosted application to MySQL database without actually having shared credentials anywhere. And we have implemented that and we're pretty impressed with what can be done. And right now opened a new open source platform project that we want to share with as many people as possible to get ideas on implementation and help with implementation is to have similar integration for Cloud Foundry. And the idea is, so the result, the outcome of this would be, you don't need to have shared passwords, you don't need to have passwords at all. So developer can push an application to a platform like Cloud Foundry and Kubernetes and that application will have access to resources that it needs, like databases, other services through trust relationship and policy. So essentially replace secrets, replace passwords, replace certificates with trust relationship and policies. So it is very possible we are actually working on it. It is long conversation, but if you are interested, I'm here today, tomorrow, please find and we'll be happy to talk about that. All right, been there. So to summarize, by applying cloud native security on practice, on the platforms and systems and automation that we have, it is very possible to make security into basic components. So developers, devops teams do not need to care about it and security will not be on the path of it. And very often, high velocity will improve security, right then opposite. So what we have, developers can innovate faster. Application security improves and cost is going down as well. So better quality products delivers faster to customers, happy customers. We definitely have much happier developers and we have happier finance and the investors. It's a triple win. That's what we are trying to achieve and this is the outcome that we already have and expecting to improve. That's all for the core of this and we have a few minutes for questions. But before that, I want to talk a little bit about Comcast committed to open source. We do value open source and we are trying to not just consume but work with the community to improve, improve not just what we are doing in Comcast but make our work available to everybody as well as we are big fans of and consumers of open source. You can find around 145 open source repositories at github.com slash Comcast. But outside of that, one of the big contribution last year Apache Traffic Control was graduated in Apache Foundation and that is used by many service providers now and it is used because it was open sourced outside of a small Comcast family. So please visit and check. And also, if you haven't seen our booth yet, please go there, talk to people. There are different people from development side, from operators, from operations as well. And from recruiting, yes, we are hiring. These are some principles that we are using in our day-to-day life. So bias for actions and outcomes is very important. That's what we are looking for. We are agile, we take our customers seriously and security of our customers very seriously. And also we love to work together in Comcast and outside. We firm believer in a group IQ and that together we can go further. On this note, any questions? Yeah, comcastgithub.io is one place to go as well as labs.comcast.com. This policy and trust relationship. So essentially established trust relationship between different components in our ecosystem and apply policies on how those components can communicate with each other. Yes, and just last week Adam here is a developer. So last week we are opening it to the world, not as a ready to consume piece, but ready to collaborate and ready to talk. So we have a bunch of ideas, we have started on a code. And I think we have a pretty clear vision how it can be all implemented. And that is to integrate Cloud Foundry with Vault so we can provide that password less experience based on trust, based on policies for Cloud Foundry users. Yes, it is on a, what do you call that? Help me out. Thank you. Vault Cloud Foundry plugin. It gets rid entirely. And yeah, I don't have a visual schema how it's going to work, so it's not easy to describe, but the idea is each Cloud Foundry instance has a short-lived short-term certificate and a key issued by the own Cloud Foundry authority. And that certificate contains information about that instance, which organization it is, space and application name, things like that. So we can use that information to authenticate against HasherCorp Vault and use that DM to apply policy based on specific organization and space. Right, so no passwords. And if somebody will hijack that key, then it is very short-lived. I don't remember what is the current time for that, but it is very short-lived, and it is constantly replaced by Cloud Foundry. So yeah, that is the core idea. So no secrets there are. And that can be also baked into Spring Cloud security, for example, right? So you deploy your application, you don't need to put any code, you just will have it for free by default. And that is a different approach compared to, say, Spring Cloud config when you need a key from the kingdom to unlock all your secrets from a central place. Right, so in this case, there is no shared long-term key. I forgot to mention it, but I put it just a sample pipeline, and this is a tail of the pipeline, and the tail is where deployment is happening. So if you already have that pipeline, what you can do is just to put a periodic trigger and push deployment every so often. Every day, every hour, every 15 minutes, whatever you are comfortable with, and that will essentially repay it for you, right? So that is one technique which developer can put very easily with minimal efforts, and that will take care of a periodic repaying, but does the technique is just deploy more often, iterate more frequently, deploy more often, and you will be automatically repaid, or apply some kind of chaos engineering technique and just kill randomly your instances and let Cloud Foundry to recreate them. All right, how are we doing from time? All right, thanks everybody for coming.