 Good evening, and welcome to this discussion on securing cloud native applications. I'm Kamala Desika, you're a moderator. I focus on product marketing for our Cloud Platform and ISV ecosystem at Pivotal. What that means is I get to work with a great bunch of people, some great partners that help us broaden the value that we offer to our customers and help that digital transformation go just a little bit easier. One of the things that comes up a lot in our discussions with our customers is how important security is to them, right? And this was also quite consistently brought up in the enterprise track where many of our customers spoke yesterday and also today. So we thought it would be useful to share some of our experiences and our ideas with the broader community, and so today we have some security focused colleagues of mine out here with some of our security focused partners to have what's not necessarily a comprehensive discussion given the time concerns, but at least we hope to leave you with some thought provoking as well as some actionable ideas for when we're back at work next week. So please join me in welcoming our panelists. I'll start with Joe Granja. Do you want to start out with your introduction, your name and where you work? Sure. Yeah. Hi, everyone. My name is Joe Granja, and I've been building software for over 20 years. Time flies when you're having fun. And mainly financial service sector has been my focus in the Canadian market. Toronto, actually, I'm based out of Toronto, worked with a lot of the big banks and insurance insurance industry, working with the InfoSec teams, making sure applications that was building on, you know, are compliant with the security guidelines. Got to keep that closer. In my current role, I'm actually a core committer within Spring Security focusing on the new standards today. OAuth 2 Open ID Connect. Thanks, Joe. My name is John Field, and I'm a product manager working on Pivotal Cloud Foundry and focused on security. Like to tell people I've been doing security since long before it's fashionable, actually the same as Joe here, came to Pivotal by way of EMC and RSA CTO office, basically helping product teams improve their security architecture for a number of years and been working on Cloud Foundry since about 2013. Before that, I did some time on Bank of Trust on Wall Street, working in IT security and banking environment. And I like to say that that experience taught me everything I need to know about how to successfully get through a regulatory compliance audit. So hopefully we can share some of that today. Yes, you get through those by doing a lot of drinking. I'm Tyler Shields. I run marketing strategy and partnerships for a security company called Signal Sciences based out of Los Angeles. I have spent the better part of two decades as well in security, but mostly on the breaking as opposed to the building side, breaking applications, breaking into locations, etc. And then a number of years back, I switched over to actually building companies and helping build technologies and tools to help secure, secure applications. And I'm also called Joe. I think I'll be Joe too on this panel. So I work in the CTO office in Jamalto focused on strategy. My background has been I seem to be the baby because I'm only 15 years in security, but it started with a long stint in a dark basement for government. And in the last five, six years, I've been working on their product management, product development site and a number of vendor organizations. Perfect. Now, you guys are here on the panel today because each of you is focused on a specific layer in the application security stack. So could you tell us a little bit about what that is and where your work is currently focused? Sure, yeah. So as I mentioned, I'm one of the committers on spring security, so my focus is application security. I'm just curious, just if I could get a raise of hands, who has heard of spring security or who has used it? Great. OK, I'll just do a quick high level for those of the they haven't is spring security is, you know, application security framework for the Java platform provides authentication, authorization capabilities and integration to identity systems, whether it's LDAP or SAML or OAuth, open ID, so forth. Also provides protection against common vulnerabilities like cross-layer, best forage, cross-layed scripting, session fixation, et cetera. Most of my efforts these days is focused on building out support for OAuth to an open ID connect as well as a Hosei framework. So that's where my focus is today. Great, thanks, Joe. And from my perspective, I'm focused mostly on Cloud Foundry platform, so Pivotal, Cloud Foundry in the sense of kind of two things. I'm concerned about how do we manage to deliver a platform that makes it easy for the application developers to get through a regulatory compliance audit, right? What are your security and compliance requirements? And in practice, what does that mean? It comes down to working with Cloud Foundry product managers to make sure that the platform includes the technical controls that are necessary. And it also means working with customers to help those customers understand what's necessary for them to successfully get their applications through a compliance audit when those applications are hosted on Cloud Foundry. So I also am at the application security layer and not necessarily at the OAuth layer, like the folks down on the end. My company focuses mostly on security visibility and security runtime security protection for applications in this particular context for applications built in the Cloud Foundry and Pivotal framework. So we embed right into the heart of the application either as a plug-in to the web server itself or as a library loaded into the into the source code to provide security visibility and data and anomaly and analytics, as well as runtime protection to your application. So I focus predominantly on that app sector. So I'm at the other extreme from the up guys. I'm right on the foundation layer. When you think about the security that goes into applications, encryption, TLS, all of those kind of things, they're based on encryption keys. Those encryption keys get wrapped by other encryption keys. And effectively, the bookstops with me and the products that you might have such that you're not building a stack of turtles all the way down. It comes to a foundation of what we call the hardware security module, which is effectively a vault, a brick that has been specifically security tested that it can be truly considered to be the root of the security model on which everything else in the platform is built. Great. So what would you say are some of the marketing market forces that are driving your work today? So maybe we can get that one started also with Joe Granjo. Well, yeah, as as we've been seeing in this brave new world, right, this this cloud native world where microservices live, right? I mean, that's totally changed the whole landscape of application security, you know, in the old way of doing things. The monolith, you know, let's say let's use a concrete example. We've got a banking application, right? Online banking retail commercial. It's got 50 features, right? One application. We secure that, right? Now we decompose that application into let's just say 50 microservices. Now we have all these components, way bigger attack service. Now we got to secure all these components instead of this one, right? So totally changes the landscape and super complex. How do we secure all these services, right? And it's the service to service interaction. What type of security protocols are we using? A mixture of combination security protocols. So it really, really drastically changes and really introduces complexity. Microservices are great. There's a lot of flexibility around it. But of course it introduced complexity as well on the security side of things. And how about you, John? Yeah, so I think Joe's points are basically all around application architecture. And I think that's kind of the first level. The thing that happens after that is the transformations we're seeing around DevOps and the move to cloud native from a process perspective. And I think what's important with security, the thing to understand, you know, just keep in mind, is that you can't talk about security of just the bits, right? That's important. But it's actually a combination of people and process and technology. And so that means that the technical bits are really only about a third of the problem. So I think we've been doing great in terms of adding new technical controls to the platform compared to where we were a couple of years back. The platform itself has got all of the necessary technical controls that you need. And from the people side, there's kind of no denying. Everyone is aware of security. The developers and operators I work with all want to do the right thing. We just have to kind of arm them with the education and training they need. Everybody is on the right track in that sense. From the process side, we've got a lot of work to do. And I think what's happening is DevOps running into compliance is kind of like waterfall meets agile and they annihilate each other. So I think there's work to be done there. And the key thing, kind of the one thing that I could suggest is that we have to think about really engaging early. So invite the compliance and security guys to the platform dojo. Instead of leaving them out of the loop, bring them in early. Early and often is definitely going to pay off in the long run. Tyler. What's shifting I think in the app sec world that's causing a lot of change is definitely the DevOps, the agile and the cloud movements. I call it the three tectonic plates of change underneath security. And when you look at the actual, it does a lot of things to how we develop apps. It changes a lot of the processes. It changes a lot of the way things get done. For me the number one thing it changes is speed. How quickly you can actually develop, how quickly you can put out new code, how quickly you can make changes to your code base, how many times you push into production a day. If we take the traditional static analysis code analysis tools to have a turnaround of let's say, I don't know, a day or two days to do a full code review for you before you push into production, that's fine when you're pushing into production once every six months. That fails when you're pushing into production 20 times a day. You can't do it. And so the fundamental biggest problem that security at the app layer is having is speed at the speed of business and the speed of the technology that's now supporting that business. So the app sec world really has to reinvent and put itself into a position where it can provide security, visibility, protection, knowledge with feedback loops to the devs to educate the devs on how to do it right but to do it in a way that's real time. And that I think is one of the biggest changes in market forces right now. So in my area, I would say the biggest change is how the technology is consumed ultimately because hardware security modules, the kind of technology that we're talking about has been around since the 1990s and if you look at what it looks like today and what it looked like right at the beginning in 1990, it's very similar. There's maybe a few tweaks here and there. And so that whole methodology or expectation that you need a master's degree or a PhD to configure it and it can take you a day, half a day to do something simple like add a new partition such that a new application can come on board. All of that goes out the window as you say when you're pushing code out 20 times a day. And so the fact that these things are built into compliance, built into regulations that you must use these kind of technologies hits square into the DevOps culture of, okay, I've got to use it, so how can I use it quickly? And those kind of questions are being answered. That's great. Yeah, I think, you know, kind of Ed, Joe was saying and Tyler, there's a concept, you know, what I like to think about from the point of view of the platform is the concept of inherited controls. And so everything that we're saying about the apps got to go faster and a lot of the technical controls that you need are in the platform, in the HSM that's integrated with the platform. The idea then is that the apps go faster because they can benefit from the fact that the platform itself has been certified. So the approach of security accreditation of the platform and then the apps that get deployed on top, you have a much smaller scope when you do that security and compliance assessment and therefore you can go faster. It allows the operational components of those controls to actually operate much faster too. So the compliance side is super important, but the ability, if you take the traditional web app fire while we stick a box out in front of your app, what happens when you spin up a thousand apps in the cloud and you're putting these physical boxes in front of it, right? It doesn't map and it doesn't work. So by embedding security into the heart of the application, whether it be at the HSM layer, whether it be at the app layer or at the host layer, embedding it right into the heart of the application gets it to run at the same speed, metaphorically the same speed as the rest of the system itself. Right, wow, so many questions to follow up on actually from my end. So let me do a different one for each of you, right? So I'll start with Joe Pindar actually this time. So one of the things that came up in all of your discussion is how much change there is. Now what would you say are some areas of weakness in the process and security models that Enterprise already use? So if you think about your own personal journey coming into Cloud Foundry and Cloud Native, you've had to ramp up, you've had to think about things in different ways that potentially the way that you learn to university and how to develop apps isn't the way that you do it now. Now think about coming from a security background where you're not actually necessarily paying attention to all the developments that are in this space and what you know as just it's natural to do a CF push, it's natural to have a pipeline of things. You have security professionals who are still thinking, you've only just made a transition from physically separate service that you can protect with a cage into a virtual environment. And now if you're starting to say, well actually it's gone from that virtual machine to much more granular, it's not even containers, it's apps that you're just pushing and it's all orchestrated in there. It's more and more head spinning kind of steam out of the years. So I think that's something that as was said and getting security into the dojo, security officers and people who are ultimately going to be signing off their compliance don't have to know everything about the platform and the way things work, but they have to know enough that when they apply it back to their first principles and what they know as the way things should work, they can actually answer the fundamental questions. I think that's essential. Perfect. My next one is for Tyler. So when trying to address those weaknesses that we just talked about, what would you say that enterprises would have to do to think differently in how they're investing their budget or how they're protecting their applications? I'll take the budget question first because this absolutely drives me insane. So I was an analyst prior to being an entrepreneur and as an analyst, we generated a lot of data. One piece of data was the actual number and percentage of records that got breached via the application security layer and it's by far the most records compromised come from the application security layer and things like SQL injection, cross-excripting and database breaches coming in from that direction. You had only 5% to 10% of spend happens at the app sec layer. So I totally don't get that mismatch, right? It makes no logical sense to me and so that always used to drive me insane but I think the biggest thing that has to change is actually making, well not the biggest thing because I think that the biggest thing has to change with education of people to get them to understand security inherently at the development level because that's going to solve the problem at its root. But in the short to mid term, the right way to handle that is to take care of it by putting in technologies in place that can apply security controls and feedback that data to the developer in a feedback loop. So it comes down to ownership in many ways. One of the tenants of DevOps is actually that the developer owns both DevOps, right? It's kind of one person responsible for the whole thing beginning to end soup to nuts. Well, you know it's securities in there too. So if your stuff gets popped it's not necessarily on the security team at the end of the day, right? It's the ownership of that code. And so I think those are the key fundamentals changes that are occurring there at the people section I think is going to be the that's the right way to solve it. But man, that's such an uphill battle. Right. So just in terms of a clean segue maybe the next one is actually for Joe Granja. What would you say then is if it's on the developers what would you say then in terms of metrics or what sort of outcomes technical outcomes should they be looking at from a security perspective? I've definitely been the general theme and I'm definitely riding on that theme is you got to involve the security professionals with the engineering team right from the get go. Establish that relationship, pair with them, understand work with them to establish the security requirements for your microservice. The bottom line is as a microservices team you own the security right now. Like you have to make sure you have to ensure your microservice is secured, right? Work with the security people to establish your authorization rules right? Especially and this is a key one is and I'm not seeing this too much is write security focused integration tests. Now if your service is only authorizing you know let's say you're using OAuth and you got GWT is going you know being passed into your services and you're only allowing scope A and scope B to proceed through. You got to have those negative tests. Forget about the positive tests. Obviously you got to have that but you got to have as many negative tests and the security professionals they know how vulnerabilities, how attacks you know come in. They have that knowledge so they'll be able to work with the engineers to build those test pass those negative cases right? So you have those tests and ultimately you'll have the metrics right? Through those tests they'll be logged, they'll ultimately come in handy at the end of it during the security audit and the final check before production, right? You said the magic word audit so I thought that would be like a perfect thing for... Yeah so something else to add to that too guys is it's a hell of a lot easier to teach a developer security than it is to teach a security guy to develop. Remember that okay? You guys can learn how attacks work because you understand the fundamentals of the code. You understand when a tick mark comes in on an input field you understand how that manipulates the actual SQL query right? You can get to that level. It's so much harder to take a generic network pentest guy or somebody who's run firewalls and IDS systems at a generic level from the security world and explain to them how development works. That's a whole other ballgame. So own it and drag them kicking and screaming if you have to and so I don't know how many security people are in the audience that I just pissed off but it's the truth of the matter. Actually so yeah to build on that I think we need to do more threat modeling up front right? So if you're a developer seek out someone in your organization that is a security knowledgeable person and do some serious threat modeling before you start iterating. So if you actually understand what your requirements are including the non-functional requirements. You might have reporting requirements. You might have availability requirements. Privacy confidentiality requirements that you wouldn't otherwise have thought about. The security person will bring that to your attention together as a balanced team right? We talk at Pivotal a lot about balanced teams. Security and compliance is part of that balanced team. Understand those requirements. Do the threat modeling and having done that you'll be much more successful in delivering rapidly an application that is actually secure and is capable of being compliant. So John you mentioned the magic word compliance and Joe you mentioned the magic word audit. So what do you see like what sort of advice can you give customers who want to deploy apps to PCF and are subject to a compliance audit and yeah. Yeah so you know in addition to what I said earlier about kind of people process and technology I mean I think that's a key thing. It's like the three-legged stool. It's people process and technology. The platform is just part of that problem. I think the thing that most people have trouble with is on the process side and so you know engage with that audit team early and often so that they can begin to understand how the platform operates right. The check box compliance approach is just not only not appropriate it's not going to ensure your security. It's actually better to get to the basis of why those check boxes were created in the earlier generation and then deliver on what the actual requirement is. So there's three parts to showing your compliant. It's actually be secure and demonstrate that you meet the technical control requirement. Then be able to show that you are secure by gathering the evidence that proves that and I think doing that together is key. It has to be the deployed platform right. You can't actually say that CF release is PCI compliant right. It's just bits on a disk. CF release is just software to make a statement about compliance or security. It's an actual running operating foundation and the surrounding people process and other technical controls that go with it. Perfect. So we talked quite a bit about process change and some of the cultural aspects as well. Now before we go to the audience for their questions, yeah it is that time almost already. It flies when you're having fun. So what are some key takeaways that you have for them and what are maybe some gaps that they might need to be aware of? Maybe then we'll start with Joe Pindar. So I think one of the key things that I would take away is the fact that in emerging fields, in emerging communities like DevOps like Cloud Foundry it's often easy to think we don't need no stinking compliance. We're pushing 20 times a day so that's fantastic. But the way that I suggest you think about compliance is that it is the minimum entry criteria that a certain industry has said that you have to meet this if you want to play in this environment and so what that means is if you want to play in PCI and payment cards then you have to meet PCI compliance. So then it becomes a question much like we've been talking about on the panel is so how am I going to meet that compliance? And so if you think about it as more of a hurdle rate to get into new markets to expand the opportunity of what CF can deliver and take it to new audiences I think that's a good mindset to take this forward. So from a compliance perspective I just read somewhere recently that the UK withdrew a couple of their compliance specific rules or laws or I don't remember what they were. And the reason they withdrew them is because it set a minimum bar that then gave people the permission to not exceed that minimum bar. Hey I got my XYZ compliance cert I'm good, I'm nice and safe, well guess what? A vast majority of the web breaches that occurred in the United States had PCI compliance, right? So that's not, that is totally the minimum and I definitely think that should be a takeaway. From my point of view I'm not a huge compliance guy, I don't follow the space very much, it's not an area I tend to focus on so from my point of view I think the key for me is making sure that your processes and your people take control of the change that's occurring in your security realm but really the key thing is getting tools that automate those people because you're never going to have the right amount of people with the right skill set to be properly secure, you're just not going to. So get the right tools and if you can embed them into the heart of the application itself and make them scale with the application and execute at runtime you're able to keep up with the small changes frequent deployment model that we now have to deal with. So for me it's really looking at throwing away a lot of the tools and processes that we've had, you know, for 20 years in APSAC, well APSAC's been around 15 or so, but in the last 15 years we've built up all these old static processes that just don't work anymore so don't be afraid to throw them away, reinvent just like we did with DevOps and Agile. We threw away a lot of old world development processes in favor of new techniques and new ways of doing development. Do the same thing with your security stuff. Throw away a lot of that old stuff and figure out how to do it in a new way. Yeah, so plus one to security automation for sure. The whole idea that you can do things like static code scans, CVE scans, config scans, do those things in your pipeline for the very reason that that's the cheapest place to do it, right? There's, you know, adage about finding a bug before you ship, you know, it costs 10 times more to fix it after there's been a bug shipped. In this case, when we're talking cloud native it costs way more after you've been breached and the data has actually been exfiltrated, right? So a small investment. Security is clearly paying now or pay me later. And so an investment up front to say I'm going to put those things in my pipeline and then it's just part of every release. On the runtime side I think in terms of investing in advanced detective controls, so rather than purely preventative controls think about integrating with detective controls that are going to give you that feedback loop because the platform itself actually gives you all of the preventative controls, right? We've got all of those capabilities in the platform. Your application can rely on that. What you really want is to operate with detective controls at the application tier. Yeah, key takeaways from the development side of things is do not write your own security framework or your own identity store. Don't do it. It's very difficult. It's very risky for your business. Not to mention maintaining vulnerabilities that are constantly being introduced. So don't do it. Leverage proven stable ones like spring security for example. Leverage your internal organization's identity system. That's the biggest one. Establish that relationship with the security professional. They're probably working with you guys than sitting in their cubicles doing their security usual stuff. So engage with them, learn from them because it's key in today's world, despite with distributed architecture, security is even more critical these days. So learn from them. And this other one is a big one is this term adopt the approach of lease privileged. What I mean is your service, you have authorization rules there, right? It's allowing request to proceed based on scope a scope B for example. We'll drive on an example. Microservices do one simple thing. If all of a sudden you got 10 scopes in your authorization rules, you got to step back and you got to look is that service got to get decomposed into two services? Or do I got to remodel my authorization rules? The bottom line is is keep things tight right from the beginning. You have to make sure you get go only open it up a little as you're iterating on a development process. But don't open it up too much because then that's that. I mean, it's just not safe. You just got to keep an aisle for that adopt the approach of lease privilege. So those are the main key takeaways. Again, plus one that's it's much harder to pull it back after you've actually done that. So that's one of the things that I think is really important. I think it's important to actively and start with the smallest possible scope and then actually go beyond that as you need to and in consultation with the folks who actually can help you in terms of things like what are the appropriate roles that exist in this app, right? Knowing that there are different roles. I think Joe wants to say something about how do you actually do security in these new frameworks? One of the things that you'll find is that a lot of the security professional and the security auditors have been doing this for such a long time that they know how to take the shortcuts and so they've forgotten what the principle is that they're trying to guard against and they will say something silly like you must have a static rule in a firewall to have an IP address to kind of prevent it this IP address from talking to that IP address. In a dynamic system that makes no sense. What it comes down to is can this application talk to that application and so when you're pushing back on these security auditors and the security organization it's good to try and ask them is that actually a shortcut that you've always done it that way or what is the first principle that you're actually working to? Perfect. I'd like to open it up for questions from the audience. Yes? I think you'll probably hear me. So the question was is there a role for the partners and other ISVs within the ecosystem to educate QSAs? Yes, absolutely and not just QSAs. I think people like 18F have done a fantastic job creating internally within the US government and we have equivalents in the UK but it's for everyone from a security background coming into this space to champion the cause absolutely. I'm going to push back. I have a question for you. What's the right way for us to do that? At the end of the day my company is intended to do two things. One, help the world become more secure and make money. That's a business. What's the right way that we should engage with you? Sending folks to conferences like this so we can talk about the problems doing training courses and things like that are common but it hasn't worked in 15 years. What can we do to do that better? Maybe that's a bit rhetorical you don't have to answer. Yeah, please. Yeah. I think the biggest thing is that it's going to be a community effort. Folks like you who clearly have an understanding of what some of these web attacks look like you have some experience in the space folks like you and others in the community have to help champion it for each other as well as every vendor so totally with you. I'm not sure the exact number but my understanding is that it doesn't cost that much to join the PCI standards council and so maybe what we do is to get a bunch of folks from this room to do that. I started a pivotal tracker which was mapping every single PCI requirement into a tracker story and then the idea is that you can demonstrate compliance on the platform by actually looking at the acceptance procedure that's there. So that speaks to the point of what is the essential security principle we're doing like separation of duties and here is a procedure you can follow to demonstrate that the platform actually does that and so having those little tidbits or the way we educate those auditors and those QSAs kind of in an incremental way and hopefully over the course of time we get to the point where everyone agrees it makes sense to adjust what the standard stock says right the word container doesn't appear anywhere in PCI today right. So I have a question for all of you which is it sounded from some of the comments you were making that we needed to sort of involve the developer more and the concern I have is we have this notion of a full stack developer versus somebody that's much more if you like focused narrowly and so I would prefer it if the frameworks themselves provided a lot of the support that's necessary of course you want people to have an awareness of security but security is many faceted it's everything from micro segmentation to things like the actual attack surface if you're putting out a web app etc etc there's so many different dimensions to it that it feels wrong to sort of like force to develop it down that track versus actually have the platform have built in circuit breakers I'm thinking of things like AppSera for example which can react far faster and deal with things on a sort of platform basis versus having every developer have to understand program defensively etc Well I think from the point of view of full stack I don't think there's any disagreement that it's important to understand the full technology stack including the security implications I one time gave a talk about the full stack security developer and the idea that it doesn't make sense to not be a full stack developer because you really don't fully understand what you're deploying if you don't understand the full stack but understanding it and having to actually build it are two different things and I'm going to pass over to Joe here who can explain you get the value from the framework and you don't actually have to build it you just have to know that it's being used properly Yeah like I was saying earlier it's been the common theme involving the security professional throughout the process right up front now security is very specialized full stack developer a lot of developers they don't like security and they just not really focused on that and that's okay however during the process engaging that specialized the security professional learning from them understanding at the very the first step is understanding what are the security guidelines governing the organization from a high level at the very least learn from them engage with them pair with them now the security professional they don't have to understand writing code that's why you got the full stack developer but just engaging with them understanding what capabilities there are within the organization for authorizing their micro service for authenticating with their micro service and so forth building those integration tests with security focused integration tests with the combination of those two you could achieve that full stack developer does not need to specialize in security because that takes it's complicated right but that's why you got the specialist there tag teaming with the engineers right I think Joe's next so what you're advocating Joe Granja is DevOps with security professional and your DevOps professionals so I'm going to push back on this concept of security and the full stack developer because for every full stack developer I see 10 junior developers who don't know how to do the full stack but let's say there is a junior developer doing front end web app work and they need to receive external or human input into that web app so they have some some entry fields now absolutely as part of their general education of how you do a nice web form on the front end and how you make it look pretty they should know about SQL injections and the way that people are actually attacking those kind of forms so yes there is a full stack and that can get very complicated but on a very simple level on just doing the front end interface there are security things that you've got to think about that frameworks can help you with absolutely but I think that's something that everyone from the most junior person or as soon as you put an input form on a web page you should be thinking about security security can be automated throughout that stack in many different places many different layers in many different ways and that's awesome that's the dream so that it's a self-securing system and definitely put those tools use those frameworks etc but how do you automate the security of a multi-step process that was designed incorrectly by the coder that doesn't have an input field that you're manipulating but literally if you go down a certain path of input you're able to do something that wasn't originally intended those are called multi-step process flaws or business logic flaws right those aren't anything that a framework is going to be able to solve for they're unique snowflakes to that particular application so my point of bringing that up is put in the best strong frameworks use Cloud Foundry and Pivotal to erase, recycle, reuse all the Rs and have it build up killing off the constant the persistent threats that may get in etc but at the end of the day you still have custom code that sits on top of that an application layer of text when I was breaking systems we had a running joke if you found a cross-site scripting people would laugh at you because there are a dime a dozen there throughout the system now some of the frameworks fix them whatever the real fun volns when we were breaking systems was finding those multi-step process flaws that the developer put in that was like oh man that was a good find and at the end of the day those are the hardest ones to solve so I think the answer still has to be education for that full stack developer to look at his own code he has ownership and responsibility of that code but definitely rely on underlying tools to help automate whatever can be automated perfect we are very much over time and I think no one has left the room so that's awesome so we must have had a really good discussion so I would like to thank the audience for their time as well as our guests and our panelists today so a big thank you for staying over we will still be around if you guys have individual questions to ask them but I do believe we have to leave the room soon sorry