 Hello everybody, thanks for coming out to this session here at 530 notes kind of late One of the last sessions of the day, but hang in there. I think lunch or dinner is coming after this I'm Jim Freeman. I I'm a director at Rackspace. I work in what we call security engineering This is Michael Zen. He's a senior security engineer And what we want to talk to you about is about the paradigm shift and kind of moving away from your typical information security guy and being more engineering developer type and I'm gonna talk about the differences What those are keep pushing the button we didn't have we didn't QA this Well, they do that. I'll talk a little bit about what I wanted to talk about which is kind of our agenda today So the first thing I want to talk about is background. How do we get here? We're security kind of encompasses everything. How do you go from physical security to cyber security? How do you go from cyber security into coding and testing? Those are a type of type of things. I want to talk about I want to talk about what the problem is. What do we see today from a? Security information security problem, and then what is the solution? How do we get better? Still not working. Yeah There you go. Sorry about that little technical difficulties. So back in the 1990s security kind of came around and We had physical security. We had computer security and kind of things that they were kind of designed to do is kind of keep the bad guys That kind of defensive mode built up the walls. We've got firewalls. We've got net sec Kind of monitor that. What does that look like? It wasn't at all. Think about software So software became the de facto. Let's put everything in one group your corporate security involved the information security or IT Physical security compliance your auditing and then your security awareness program these kind of things fall into your security your corporate security group Pretty common throughout a lot of companies But why does it default to corporate security teams when they're their whole entire mission is basically to defend and protect? And their people did it and assets for a company. They're not developers They're not engineers some of them are so don't want to put that in a still typical way But a lot of them don't have the skill sets to develop tools When you look at security and you look at companies, we spent a lot of money I kind of looked up and in the last couple of years We were averaging about 50 billion dollars a year on security defense security protection And when you look at the software field we about had 532 increase in our certain incidents reported since 2001 so obviously we're spending more money on security But we're getting worse at it if you look at it some software Companies are saying operation security is not working. We're spending all this money We're hiring all these great people, but they're not solving our problem That's because they don't know why we are having these issues Software is providing a tunnel apis VPNs. This is not hardware stuff. This is software stuff We have more bugs more problems the battlefield is changing since 2005 We had seen an increase not only in bugs, but vulnerabilities based on software software development and when you look at it from a cycle, it's Develop design develop release design develop a release and that's kind of quick But it goes fast when you have when you go fast you have more bugs So most security flaws are invisible to conventional testing I'm gonna talk a little bit about that later when I talk about application testing But basically we need to design and build security from the beginning when the concept is there We need to get our guys in there and help Kind of make security part of the DNA of the development So Conant talked a little bit about the problems, but what about solutions that Jim? We always have a problem. What's the solution? So? About three and a half years ago. I got a call from Rackspace and they called me up and says hey, you're doing some great things with civil engineering Nuclear engineering. What are you doing? And I said I work in security coding security testing for security on the application side and so what I mainly worked on is if you had anything to do with civil engineering and you wanted to get Build or you wanted to put PII information out there You had to build a system to talk to us so while I was not part of the corporate security I was part of the field security what I will we called in the civil engineering side field security We're make sure and sure that information that's being passed between our client and us was secure at all times So let's talk about your typical corporate security guy. I Know that's probably if you're a corporate security guy go golly, you know, you're really still typing me I used to be that guy. So although that's not me because he has more hair than I do, but he's probably better looking But the whole job is keep organizations safe I want to make sure that you're safe. I want to make sure your computer is safe You want to protect the data that's on your computer on the networks? They want to build IDS is monitoring the network have logs put ass asset controls very network-centric What these guys are focused on is the infrastructure. They want to defend and make sure it's safe So for security and operations, this is what we call information security operation centers These are the guys that monitor. They want to look if there's any suspicious activities Most companies that do this are run by operational people. They're not developers. They're not technical in the sense. They can't program Very very heavily. I'm not again. I'm not saying not all of them can't but most of them can't But again, it goes back. Why would they start it? They wanted to make sure we're protecting our band boundaries if you think about the military therefore defend and protect so Couple years ago, and I say a couple years ago early 2000 mid 2000 the term application Security kind of came around and it was built upon a need. We need to kind of test our applications before we release them The problem with that is they had a lot of canned text And if you anybody who's ever used vericode, no, they can't they can't scan Python or dynamic languages And when you look at the cup of companies that came in and did that which they did is they had a can test They ran a test they walked over to the developer and said Here's your problems. Have a nice day. Where's my money? It's kind of a short way of saying they weren't really helping them They weren't really saying this line of code right here is the problem and we need to show you how to fix it one of the things that That we we have I know for us is finding guys who can develop and find security And it's just not out there. The skills are hard to find one of the things that I Interviewed for this but not I didn't interview but I was the interviewing person for a position and I kind of one of my famous questions What do you think about third-party testing because to me? This will tell me what kind of person this is if he tells me that I think it's great We should always use them. I The interview should pretty much in there and I'll tell you why Everybody here is some technical anybody can learn how to do security testing, but can you do it? Can you develop? Security that's what I'm looking for so when you bring in third-party testers. They're going back to the application side and What we were looking for is more technical again going back to the development side the bar is too low I know fortify and denim group. We've actually used him when I first came to rack space and it was the same old story test report goodbye, I Will say this and I meant to put them on here was Montesano out of California they actually did they Were actually pretty good. They hired developers who understood security Had daily stand-ups with the developers and looked at the code and said here's where you have a cross-site scripting This is where your cross-request is coming from. This is your overbuffalo and Buffer a buffer flow. Yeah buffer overflow But they were able to sit down with them and do that and that's where the idea kind of came is We need to go to that route product security It's made a long way Product security is a lot different. It's kind of like a QA QA does does the product do what we want it to do Product security does it do what it should not do ie. Does it allow you to get PI information? Can you brute force in? Can you obtain some security numbers? It's part of the development process? You're working with developers But you're not doing it from a building standpoint Good step great place to start It also measures how secure a product is We talk about Product security. I know that a lot of people in information security get a hard time about that and I always tell them I said it if Ford builds a car and they put That's coming out there. I'm sorry. I heard When Ford builds a car or any car manufacturer they don't go up to their information security guys and say hey Can you go test our car to make sure it meets all standard? Safeties and security requirements. You just don't do that because it's fit and designed for a different skill set Product security goes beyond the simple CIA. It's excellent team to partner with Some teams actually put them in the QA some actually put them in in Development it kind of depends on where you you as a company want to put them so That sounds nice Jim great We want to have developers who have security minus stuff and then we want to put them with QA or development It's a lot harder than you think The problem is if you have somebody very strong in security and I know that David from the OSSG Sign up here, and he's really really smart love the stuff. He's doing And I'm gonna kind of kind of describe what this first bullet means CISP doesn't teach development. I know Michael has a CSSP, but he also develops so we kind of want to put the two worlds together Development when you go to school And this is getting better by the way, but when I was in school In up to a couple years. There was no security programs. They didn't teach you how to properly Program securely it was this is how the code get it out get it going because the dynamics of software security has still not come out through Then you have You know old viewpoints. Why is there a problem? I am the security guy for this corporation. You will report to me That's a problem Because that guy who says I'm cyber security doesn't really focus on physical security Sometimes looks in the corporate side. I mean the compliant side, but to that's only the risk base What that guy is really doing is saying they look security is security security. It's not true Security problems in software. So we know that we have a lack of access controls in some places We were told that developers are told to produce faster code. We talked about that earlier Faster code more bugs. We know that through studies and metrics approximately 50% of all security problems come from software And that's just a security problems, but problems in general We also know that there's a diffusion delay. Basically what that means is when something is released Takes two years approximately to find out all the bugs So a lot of that could have been found in the beginning if we had the proper setup Security is not part of the testing part of the development process. It's not part of the designing It's not part of the building. It's not part of the testing other problems Well, we know that the world opens that everybody's connecting everybody's taking the code everybody's you know deploying code How do we get better at making sure that whatever we're using or deploying is secure? We know that you don't actually need physical access anymore, which makes it very easy for a bad guy to make it automated Attack Automation to you know, get hub Someone who puts in their keys or passwords or credentials actually puts it out there. All of a sudden he has access to that Guess what? He has all the keys to your kingdom Everyone has different goals and talk about the corporate security who had set network security you have developers who have a different goal not partying together Then course security is actually off. It's an afterthought Security is kind of be all the jack of all trades I don't want to say master of none because those guys are really good at what they do But they're asked to do a lot more than what they should They're being overworked so quick Quick story if you've ever seen this the movie office space kind of put the little scenario in the dev world So the boss comes up and says you know what I need more code and I need it faster Goes up to the guys. He tells his guys we need faster code when you get out the door But where's security? Where is the testing and so you what you end up having is a bunch of employees? Developers are frustrated because security says well I'm not signing off on this this code or this product until I get to test it Which is way after the fact causes huge problems. Everybody's frustrated from the picture You can see that people are at a table to working together It's a team environment everybody's together Its problems are being addressed as they are found so that's what we're trying to push here This is what we're going to try to push here from the open-stack community How do we work together as a team from concept to end? So solving the problem so I Know Haven't talked about this with many companies and many people there needs to be a culture shift in ideology about security It's not a one-shop all it can't be a one-stop shop Software security kind of needs to borrow heavily from from software development And we must agree that software security needs to be a specialized and placed in development in QE and moved out of the Corporate world And I say that when you go to the doctor you go to somewhere You want to have something if you have a problem with your back you go to a practitioner He's kind of sends you to a back specialist. You're a problem with a knee you go to knee specialist software We need someone who specializes in software Just as you can't test quality in the software you can't build security in the process if you're not there The rise of the security engineering we need a specialist who can find design and coding flaws on the fly as it's being Addressed addressed by building better software design For design secure systems get away from the operational mind view It's cheaper to build some and fix it on the right that in there that it is to kind of deploy and fix later a lot more expensive and later Development software companies are already made shift and moving out of the out of operations and moving into Qa in development Here a couple a couple companies that I've actually talked to and it kind of got some ideas about how we should run at Ragspace CH to him Hill is the engineering company. I used to work for Cigital came in and did a what we call a B some study If you guys are familiar that they come in and kind of say how are you on a maturity model doing against other companies? They're doing the same thing of course Microsoft Google. There is an actual YouTube video out there and he talks about the difference between corporate and product security Then I mentioned Montesano earlier about what they did on a testing basis from a third party skills required if you notice I put Python up there because we want help in engineering coding and design and Michael will tell you Every time I gave him a resume he'll come back and say but yeah, they're great But they don't have no security experience or third they have security experience But they have no development experience So these guys are hard to find and if you have anybody who has this experience, please tell them to call me Not not not just because I want to hire them But I want to make a big deal out of it, you know, maybe you know for open stack Get them involved They must be able to proceed all participate in all phases of the development positive process architectural design coding testing Must be able to review code if you can't review code How do you know you're gonna find a cross-disk scripting error in code because we know Python can't be scanned by a vericode static scanner We know that for if I can't scan it We know denim group and the reason I know this because I talked to them and I says first want to market I will buy your product and and I told him write that in there I said I don't care how much it cost we need it badly And they yet to produce it because it's hard Other skiers required I put their VMware I know of a Hadoop it would be really really great if we had experts in VMware who were Security development minded it'd be great to put somebody in Hadoop who knew that inside and out If you think about it from a quality standpoint in a security standpoint, it's easier to test something You know really well So having known the background of the product a lot easier But we have a long way to go because we have we don't have that many people who with this kind of skill set Need good developer testers and I say it again. They're scarce. So My information and Michael's information will be on the end of this slide But if you know anybody if you are that person, please reach out to me because at the end of this slide The email address will be there, but please consider contacting us not not for us, but for OpenStack One thing I want to point out there Is the three parts of what we're looking for one part developer one part tester one part security You make those three parts. That's what we're looking for Suffer securities everyone's jobs and that's kind of a cliche in security role. Everybody should think about security But from a from a product standpoint is developers its security engineering its QA and I mentioned QA because they have fuss testings they can use and Automation test scripts they can kind of use to put libraries of fuzzing strings in there. They can do things like that We need security operations, you know, I kind of say they're not rights fit for development But that the right fit for operations continued to look at architecture and kind of make sure that we're being They're defending our Perimeter as far as the network We need admins to think about you know before they give rude access to everyone They should practice a priest of principal lease privilege users must understand the value of secure product Firefox was kind of came out long after next gate if it goes through the guys remember next gate Inter Explorer kind of came around But it always had security vulnerabilities and then Firefox kind of came And said you know what we're better than them So it's a story and I think from an OpenStack perspective if we can sell the story worse more secure Then it kind of goes with what Troy was talking about we're gonna be Kind of looking at a broader bigger bigger market from enterprises And then lastly our executives must understand the value in place in security engineering with the development in QA I'll talk a little bit about organizational alignment Someone asked me to put this in here and I'm gonna go through it real quick But if you if you understand the security engineering is not network security or corporate security It should be part of the QA and the development Higher software developers with security skills Rename the the department that doesn't conflict with the corporate security and of course Make sure that they're set up so they can sit day-to-day with the developers and the engineers on whatever they're working on Recommendations get the team involved from concept of development. What I mean by that is Michael as a developer and a security can sit down with the developers and say this is the concept We'll have you thought about this if you're gonna use keystone. There's some security implications that you might want to think about Actually sit there and start writing code with them Development pride provide security requirements. This is something that the open stack security group has done And I think it's a great job and something that we we should use and look at and test ourselves against have balance Don't say you cannot have a you know If you have access controls don't make it so hard that nobody can use it Don't make it so hard that users would have a hard time using it Participate in peer code reviews Don't just use scanners because scanners will only you look for what they are programmed to look for in other words There are there's only certain things they know to look for Michael has been known to break many APIs and many code in the in the open stack Community and and a lot of people have reached out to him. How did you do that because the scanner didn't pick it up and Don't toss bugs over the fence and leave them help track fix and deploy Be part of the solution Work in an open stack. So what what am I staying up here? Why am I talking all about this security developer versus corporate security because what I would like to do is kind of propose that we Kind of join together with a bqa But have a team that works together and starts working with the concepts that you know the beginning The deployments working together and not just after the fact Michael is going to show you a little bit later Shortly a demo that he's kind of worked on about building a tool in the testing automation frameworks that he can test for cross-ride scripting and buffer overflows Create templates best practices already out there guidelines already out there But what kind of metrics are as we as a community Deploying code that has one of the simple things is buffer overflows It's the most elementary error that you can find yet We still do it and then continuous improvements and then of course to get to that point where we want to be Well, there's more people out there from the security engineering mentoring and training Michael is it does a lot of a lot of time with universities and Mentors and trains them as well. So if you want to help out Be part of that group, please feel free to contact me directly or Michael and then now Michael's gonna show you This demo What Michael did is he's been working on One of the things that we struggled from from Rackspace was we had to do a lot of manual tests on API's and So Michael kind of came with the idea of says obviously we need to automate this. How do we do that? So with the help from testing automation tools And some of the other QA folks we were able to get this going Thank you, Jim Cool great thank you, Jim. So I Have been spending a couple years. I'm doing a PR testing for all the time before I start how many of you home I work Did the penetration test all API testing? Yes Let's talk about the tradition way of do API testing and pen to testing the first start point is all you ask for a pair documentation So if we want to test Open stack Keystone we just go here to check what's called how to make it call because we need to know how the product work before we do security testing and Sometimes you might find the documentation is not accurate and it's very painful Fortunately for keystone we have with the clients ready. So I just see in dog the keystone client and Here just trying to get the authentic in talking and normally You would pass the server server proxy Burp is one of the most widely used web option security audit tools So the normal process is once you've got it working burp You can check the request to see this authentication Give the username and the password the response continue the talking and everything and the good the normal good the starting point is always send this to Repeater, which is the automatic file into Sorry intruder. So if I send it to intruder and you can see here I Can choose different position different data points to test for the payload For number are gonna choose Fuzzing just automatically stuff This book building and if you click start Attacking The burp tool is gonna limit all the data points use the far defective given and send the request to choose a server in this case, I just spin up a cloud server running del stack and You can check the status here and you can check the What's the request? What's the response? normally for security testing We folks on some, you know Every message like five hundred eight hour now So whenever you say whenever I see five hundred my just like up and then from there You can send the request to repeater for special test if you want to test the sequel injection If you want to test authorization Authentication you send the each request to the repeater And we do have a checklist this specify which error wish for the test and this is just for one of the function So imagine if just for the case stone there may be 20 function calls. So we repeat the same process for How long the function calls for each function call we go through all the checklist So it takes a long time for us to finish all the security testing What happened if developer give us a new version tomorrow? So we have to repeat the whole process again it's really painful and Especially today with the why the user say I see the developer trying to release code almost every 10 minutes or something like that How could the way and security test to catch up with it? Fortunately in rock space our QE team is using their framework for their function testing And we reach out to them and they already create all the module and the client for all the function testing We will that's a perfectly To the for us to leverage and to same time So what happened if we could write our test the security test the case and given this to our QE team our QE team can run this test and the if there is some field test they can see what's the What's the security problem and that they can work with developer team or they can work with us together to get this fix Let's say the new way of do security testing here So in this gun for we're gonna show how to automatically do the test for XSE XSE is one of a security effect found by someone last year impacting lots of Open-stack products it enables people to read the file contents and scan your internet work So if we create a test case we can just give this to our QE team our QE team can just run this here and you can see the screen it tries to different file and test with the Different attacks here and this is one pass. It's okay. It means to pass the test and on the back end We can always see it in the burp see what's really going on here Let's close this if you look at this example here. It shows. What's the test is about? so basically we just define XSE and And I'll try to watch the response for this for this one because we did not inject in the payload Nothing happened. It's really successful 200. What happened if we In this this example if we include XSE here and you can see a response come back see the 400 See this is a bad X amount. So it proves that the API we test is good from this defect. So that's the reason why all the tests are passed here This is really good. Let's see another one for the same thing Let's do a buffer overflow attack here For buffer overflow We are trying to inject the long string to the data to say to check whether The application did have been doing good job of input validation For good input validation you should always check the value for its type range length and the format In this attack we generated some long string here and you can see for this case We are testing create a user function here And then you can you can see from here. There's Some test case already failed Let's stop this because take a long way to run and check what's happening in the burp If we come here Here should we want to create a user and the username and the test Followed by some random number we generate and for the password that we just Create a super strong password here. I don't know how many are here and the response is 200 that means that Our application success accept this and create a password here and for this one the same file in but way We just inject long string in the email without you know at here and they The application happily accept that email and the crisis Here so if we look at this the response if we go a little bit further and we can see here That's why the test case is a failed in this case We're trying to a longer email. Maybe with 100,000 byte of character here and give us of 500 error back calling to error message. It's It means an unexpected error printed server from filling your request and set to long for extra and the one so it means they cannot Process this long password here But if we Increase the links more and you can see here the server just give up of 500 error All those 500 error might be Abused to do denial service attack if we if we want we might send, you know 1,000 of this type of request to server to see the server can handle that or not Let's say an example of SQL injection here. So for SQL injection For SQL injection we are trying to iterate the request and Put our injection on different data entry to see what's the response back. We are looking for SQL error message and possible Time delay if we are trying to inject the sometime dealers in database So far it looks good and we can go back to to our burp to see what has been really send it to the SQL database So one of the examples are here and you can see for the email. We are trying to use the SQL injection here all one equals one and So far so good It shows that our application is good in this case I will stop the testing because it takes a long time to run Currently open stack spot two type of the data format one is x mile and the one is json So for secure testing that's really light mile because we need to test both Originally normally if we don't have enough time or we just covered x mile I'll just cover json. We just choose one side assume that everything is good But with the help of the framework, it's pretty easy to switch It's just a config file change. So for here and you can see we use double just on here if we switch that to x mile The framework are gonna change the Stabilization of the format to from x from just on to x mile And if we run the same SQL injection, you're gonna see some interesting Same happens here and you can see the first test case failed What might happen here? Let's go back to check If we go to the burp and see what the request has been sending the first thing you can see here just by Switching the config file from json to x mile on all the request automatically changes to x mile Without extra coding. So that gun already save us a half over workload. We needed to do If we check the response that you can see here, it's given us the error message But unfortunately in the error message it includes all the SQL statements If Anyone who did the penetrant test that's definitely that's not the best the secure practice You you should never retain your SQL statements back to the user So so far for those 20 functions, let's see how many test case we have Currently, we just write the SQL injection command the injection past traversing also racing authentication If we run them all let's Draw a run here. I wish I had the time to run them all but fortunately I don't have time so and you can see There are 52 so in the test case already be written. So Just by giving these tools to our QET They can just run all the tools and only focus on those failed cases and it will have both team And we can have a quick turn around The best one to you once we write this case if there's a new code push out We might adjust a while to function to the new case and we run the test case It is gonna save a lot of time. It's gonna make our life. Let's be easy make everybody life easy Thank you, Michael. I know we went over time. So we're gonna End it here. If you have any questions, we'll stick around You have any Q&A? I guess my mic was off So that concludes our presentation. I know we ran over time So instead of having somebody walk up the mic, do you have any questions? Feel free to come up here. We'll be we'll sit up here for about five minutes five ten minutes We'll have a long it takes and appreciate and have a good night