 Alright. Hello. Good to see you. Sorry that I missed you on Tuesday. I mean I did miss you on Tuesday. I'm sorry I wasn't here. I'm not sorry that I missed you. Okay, so high level logistic things off the fat. We'll be releasing homework one later tonight. I've got to get the system all set up. The first one will be mainly a coding exercise, where you'll be implementing a crazy policy that I give you. All revealed. It's kind of intentionally obtuse and not super crazy, but just in a way where you have to actually think through what are the intentions behind this security policy and it's going to be themed around kind of the house scenario that we've been talking about. So that'll be the first homework assignment. I'm sure you have a lot of questions, but let's save it so you can actually read it. Then you can hop on the Piazza and ask all the questions to your hearts content. Any other questions? Not really. Cool. Does anybody solve the assurance problem? What do you have a question? I was going to ask when it would be due. Yes. I think probably a week-ish because it'll be a pretty simple first assignment. Cool. What did you say about this assignment? It will be released tonight and all the details will be there. Cool. Okay, so assurance. So how do you quantify assurance? Testing. How does that quantify it? Because you can validate whether your system is secure by watching a series of test attacks. Okay, so you can create test attacks, but how do you, so with quantification, you want to bring it down to a number. So how do you then trust, do you trust the system 50% because of your test cases? Do you trust it up? That's up to discretion, I guess. That's up to discretion based on whatever is required. So healthcare, I guess, can be much more specific where you need 99% of your test cases to pass versus something else. Almost that 0.01% is the test case where somebody dies, right? That's true. So yeah, the context is incredibly important, right? It depends on what you're trying to build. If you're writing software that keeps airplanes in the air, we would all agree that you want a high level of assurance there that those actually work. So you may not be acceptable with having 10 test cases and having all those test cases pass and just say, yep, looks like everything's good. I 100% trust the system, right? Whereas other situations, you maybe don't have that kind of requirement. Is there a hand at the back? I want to share. Or a person attached to a hand that I wanted to share? Yeah, was it you? You want to say something? Okay. Next time. Cool. So what does it depend on? So what only thing about assurance, what does our trust in a system depend on? So we talked about test cases. We can create some kind of test cases which are really testing that the system does what we think it should. But what does that really depend on? What are all the factors that goes into developing a piece of software that just completely changes what you're saying? Well, I'll take it that way. What's the acceptable risk? Ooh, acceptable risk? Why is that important? I guess it says that how much damage you can do. So it goes back to context again. So not only what's the software supposed to do but from a business perspective, what are you most concerned with in terms of risk? Are you worried about losing your customers data? Which would suck? Are you worried about somebody arbitrarily changing your customer data? What is the risk that you absolutely cannot tolerate so you want to think about it in those terms? So yeah, so risk, if it's not a risky part of your system, you may be okay with a level of assurance that's like it's probably going to work and if it doesn't, the risk to us is very minimal. What else does assurance depend on? What do you want to be sure is correct or that you want trust in being correct? That the way we're approaching our system is the way we want to be approaching our system. Okay, but the way you, can you clarify it a little bit? So if we're talking about like a house, you know, the way I'd approach securing a house would be different from how I might secure an apartment. Yes, okay, so making sure, so this is part of assurance that you're making sure that what you're testing is appropriate for the system that you're actually testing, right? Because you have to apply housing tests to an apartment and that maybe doesn't make sense, yeah. Trusting the people that handle the data? The people that are handling the data? Yeah, what about, so what about if we think about the system itself? So we think about less of the data and the systems around that data, like the software system. So how do you write software? Does anybody write software? Everyone looked at me. So the typical way is that a professor gives you an assignment, you go, you wait till the last day and then you start working on it. You start just trying to furiously code something that works. How should the software engineering process work? A lot of planning. Planning, so what do you do in the planning phase? You design the system and then you design how the system should interact. Right, so you build a high level design of what are all the different components in the system? How do they talk to each other? In what ways do they talk to each other? Who's responsible for doing what? So you create some kind of a design which you're still, well I guess technically I guess, most homework assignments that design is kind of handed to you in the form of your specification. So what else then would you do? Really code of yours. Yeah, so you work on it, you implement, you have to actually implement that and then what do you do? Test. Test, so you need to test that. So when you think about the assurance of the system, you want to think about that at all of those different layers, right? Is the design correct? What if you're building 100% perfectly something to secure an apartment but you really care about securing a house, right? So if the design fundamentally says something like, or the specification, if the specification says that, well anybody should be able to enter this house at any time, but you as the business owner do not agree with that specification, then whatever gets built at the end, it doesn't matter that it matches the specification, because the specification doesn't match your business objective with what you need and what your security policy is. So specification really tries to answer at a high level, and this isn't a software engineering course, but it's important to think about that. And what I want you to be thinking about is it's not just the end stage, you're not just testing that software component at the very end, right? It's important to think about when you're thinking about how much do you trust the system, did you verify the design, or did you actually analyze the design, or the specification of the system? What about the design? What about the implementation? What about while you're developing the system, are you testing it for security, or do you just wait till the very end and then all the bugs get swept under the rug because we've got a ship tomorrow because the CEO said we're going to ship tomorrow, so security bugs be damned. So how do you define specifications? What are you trying to do? What does the specification actually look like? Those are all Texas specifications, right? So you think about the house, the specification could be the building plan, right? What about with software? You can use text or diagrams. Text? It's kind of what we talked about when we talked about security policies, right? So you can use text, English text to define your specification, and this has basically all the same drawbacks that we've been talking about with actual English specifications. So what are some of those drawbacks? Ambiguity? Ambiguity, right? This happens actually all the time. If you look at, there's actually a whole class of vulnerabilities that exist because the, let's say, specification for HTTP doesn't say what to do in this certain scenario, so two implementations do it differently and now you have kind of like this broken ecosystem, which can lead to security problems. What are other ways to define specifications? You can use like expressions, mathematical expressions. Yeah, so you can do mathematical formalisms, right? And then that has all the downsides we talked about already. It can be difficult to understand. It may not accurately reflect what you're trying to do, but the nice thing is maybe you can actually verify that the end system satisfies your specifications, right? There's a whole area of system verification, like automated verification that basically you write a specification and then you can verify that the program actually implement that specification correctly. Of course, then you have the problem of how do you know that specification is actually correct for what you want to do, right? So you can verify that there's no bugs in the implementation that don't jive with your formal definition of the specification, but fundamentally the specification's wrong if there's a bug there and you have the same problem. So we take the specification. This says basically what we're supposed to do. Then we sit down. We design a system. So when we design a system, what are some components that go into that? Like, sit again. What the specifications are. Yeah, so your design is based off of the specifications, right? So you take that in and the whole goal of the design is to design something so that it matches the specifications. How do you go about doing that? Try to call someone new. Sit again, louder. Implement the mechanism that are hopefully defined in this specification. Yeah, so implement. So go through a specification. You're trying to understand so the specification says what the system does is supposed to do. The design specifies how it should do that, right? And so it's actually... And how do you actually... So if you're talking about assurance, so you have the problem of is the specification itself correct with what the business wants. Then you have the problem of how does the design... How do you know that the design actually matches the specification? So how do you do that? Is it important to do that? Yeah. Why? Exactly. And maybe it does 90% of that functionality. Like, oh, you didn't say that you didn't want a backdoor in your system. And this seems like a crazy thing, but I mentioned we all have routers at home that have default username passwords, which is essentially a backdoor installed by the people who manufacture that system, right? If you're a business, you definitely don't want that, right? But as users, we actually do want that so we can go in, log, and change our SSID to something clever. So how can we actually verify that the design satisfies the specifications? Try to test cases on the design itself, right? So this can be slightly difficult because your design may not be executable, right? So what kind of formats are the designs in? Let's see if we have any future PMs in the audience. UML models? UML models? Yeah, so you can have a more formal UML model that diagrams the components in the system, how they relate to each other. Almost not aware of places that actually use that, but I would be interested in learning if you come across any. What was that? Waterfall models? Waterfall. So waterfall, I think, is kind of this whole system we're talking about, right, of like specification, design, implementation, testing. That's really, I mean, you can think of this in the agile model as well, right? This is your design scales are crunched down, so, yeah. Simulink models? Simulink models, what's simulink? I'm not familiar with that. Like MATLAB? Oh, MATLAB models. Interesting. Partitioning and passing through history. Huh, cool. Is that useful? Have you used it? Yeah, my internship. Okay. Oh, interesting. Did you find it useful? Yeah. Okay, cool. That's the important point. Okay, so yeah, you can basically, we can chalk those all up to some kind of modeling way. So you either have UML, I mean, you can model all kinds of aspects of the system. What are other ways? Use cases. So you can specify use cases of what the system should do, and this is actually very similar to the agile model, which has a lot of these, where you create use cases, you even create different users and different classes of users and say when the user X does this, then Y should happen. But they're all in English, right? Yeah. What are other ways? Yeah. User feedback. User feedback. Is your design itself user feedback? How does user feedback impact the whole process? If I deliver a product after doing my own testing, and the customer comes back and says, well, all these features were nice, but could you add this in or remove that? You know, I can't just say, well, the design said this. Yeah, tough buy it anyways, right? Because you like me. Yeah. So that, I would say, rather than the design itself, that kind of feeds back into the specifications. That's kind of the creating this loop, right? That's the whole idea with agile is you shorten that time process to on the order of weeks. That way, every two weeks, you're going to use your feedback, which is helping you refine your specification, which changes your design. But you may need to under, I mean this is kind of going slightly off topic, but you may need to understand how frequently that's going to happen because that can impact your design. If your design says we hard code everything exactly like these and we will always have five modules and you've hard coded it throughout the entire code base that there are these five modules, now if you add a six and everything breaks and blows up, you didn't think about actually changing that design in the future. Yeah. Let's say comparing to industry standards, like if you're a banker or something, make sure that you have these security implementations in place versus somebody who's like email provider or something. Your email provider is not important? Well, I'm, you know, would you like me to do wireless number provider? Wireless number provider, right? You guys are going forward. Those are all super important places. But the point is well taken. So yeah, so thinking about when we talk about thinking about assurance, we want to understand based on the context what level of analysis of the design is appropriate, right? Because we want to fix problems here. So nobody caught onto my favorite answer of how designs are done. That's with PowerPoint. This is a life. I have a friend who is a, oh man, maybe I shouldn't start calculating because I'm going to feel old, but he's been a Microsoft PM for, I want to say, seven or eight years and his entire job is basically PowerPoint. So he's the one who talks to the users, figure out the specification and kind of creates PowerPoint documents that can help the design. Or more on a technical level is a Word document, a text document describing what the system could should do. So very similar to kind of the use of how you're specifying that. So then we get to the problem of testing or thinking about the assurance, how much do we trust that the design properly implements the specification, right? So we use all the techniques we talk about. We can hire somebody to essentially, even if it's not executable code, we can still pen test this design. Is there a way from this design to do something bad? If it's bank software, can I get it to transfer money out of an account? If it's email provider, can an unauthorized person get access to the system? Can we prove it? Do you tell me your design 100% follows the specification? I say I don't believe you. How do you prove it to me? You're on tests. Can you show me your tests? I say tests are garbage. Why do I say that? Does it fail? Maybe they failed? If you try to tell me your system's secure and your tests are failing, we're already in a bad place. You could give me the program. You could basically say, we'll come up with a counter example where the system does not. I'm busy, I don't want to do that, but I don't trust it. Do you ever prove it? 100%? Yeah. This isn't like a math proof, if you're looking for that, but you, as the designer of this specification, you could make your own test suite. Okay, so separate the concerns. If you're in charge of specification, but you're not in charge of the design, then you could separate those to make sure that you write test cases. Okay. Yeah, how do you do that? Right, so then you have to, I mean, you can do that. You need to formally define the specification in a formal language. Then you need to have your design also in that formal specification, and then you can mathematically prove that the specification matches, or the design matches the specification. The problem then is, well, how do I know there's not a bug in any of your modeling components, or the design in the specification? Yeah. I mean, when I was working this summer, pretty much every job we'd get handed, the company would also have basically a whole battery of unit tests, and they would say, literally it would be like, here's the design doc, and then here's the checklist of everything we're going to go through. Make sure you get six seeds on every single one of these, and they would basically have nice, so it's almost a little bit at the second level, right? So that's making sure that the implementation matches your design, but it's a similar idea as somebody just mentioned about basically have the person at the upper layer write the test case that should pass for the next layer, right? So that was talking about have the specification person list out design test cases that should pass, and then at the next level it's to make sure that the implementation matches the design, you have the designer write out test cases that should pass. It's pretty close to impossible to truly write bug free code. Yes, I agree. I'd say with all of our knowledge now I feel very confident saying that it's impossible. Yeah? Do you think like that we can create computers that write bug free code? No. Who wrote those programs? You don't think we can write computers that can surpass human intelligence? I don't think it's a matter of intelligence more so that it's a matter of it's basically the problem is it's turtles all the way down, right? So you have a system writing code who wrote that system? A human. A human. So there's got to be bugs in that program which means some of the programs it generates are going to have bugs. I think on simpler things are much easier to convince yourself that they're actually bug free but the problem really comes in the complexity and in my mind really arises when you have systems talking to each other and you have a big complex system so you can verify each of the individual modules that this module is 100% secure. You put them all together and there's some complex interaction that you never thought of that is a security problem. We'll get better I think it'll be a lot better but it's like anything it's like thinking about Microsoft in the early 2000s almost like every month there was some new virus or some new worm that was spreading Slammer, Code Red the Melissa virus, all these things and it's because Microsoft had never really thought about security to that point and they didn't realize that as the operating system that 90% of people use one bug in their system means that basically the whole world is vulnerable to this so they actually took a really hard look they developed this secure development life cycle which you can go look up maybe I'll have you look at that at some point and they took security seriously and it took a while for that culture to be ingrained but they basically do part of the secure design lifestyle is doing what we talk about here is they do red team at the design stage of all Microsoft stuff they do red team reviews specifications, design, implementations they have whole teams dedicated to that it's all now we have other problems but we don't have this problem of you're worried about a zero day in your windows machine that somebody can remotely get group access to red team red team in like we were talking about the penetration test so a separate team that does a security analysis of the design and tries to poke flaws in the design and the important thing is doing that at every stage yeah it's actually undecidable for many non-trivial properties so for any arbitrary program you could do it on a key test the key and this part of what makes my research so fun is that the problem of finding all the security vulnerabilities in a given program and only those security vulnerabilities is equivalent to the halting problem you fundamentally can't you have to develop a system that will either miss some bugs or report some bugs as vulnerabilities that are not vulnerabilities so yeah it's tough so I can do simulations so you can do simulations yeah there's techniques it's all about improving assurance so let other people find bugs yeah so I don't know if we'll talk about specific bugs but there is a bug I think in 2014 called shell shop which was in bash this was a essentially this crazy feature of bash every no bash you're happy to be terminal or shell I guess that's the right word you know you can find variables in bash you ever do cd tilde how does that know where your home director is magic he uses an environment variable called H-O-M-E all capital that I think you do and so tilde just expands to this home variable so it turns out bash had this feature for 20 years that if you defined a variable that I think it was the value is if it was a bash function it would execute that function so this way you could pass functions to another in the form of environment variables this had been happening for 20 years and finally in 2014 somebody realized hey that's a security problem because some applications like any cgi web application some things like Ruby on Rails the user can actually create environment variables that bash would then interpret and execute so it's basically gave people remote code access on like millions of machines so it's a huge deal and that existed in an open source package bash for 20 years so open source is good but it's not a I think what's the phrase there's the open source phrase about many eyes find many bugs I think or something like that but other people have amended that to many eyes find shallow bugs because you have these really complex vulnerabilities that can that exist even in open source code so this is a good discussion so really and this is what I want you to be thinking about is if somebody is trying to tell you that something is 100% secure you should not believe them they don't have a proof of that and even if they did it's hard to trust those proofs because what's the language that the proofing system was written in how expressive is it all this fun stuff so that goes to the part that my favorite part is implementation all the code how do you actually implement the design seems like a silly question it's not a trick question either you write code so you read the design and you write the code is that it but that's how you know that your implementation maybe not implement all the features yeah but you write it write the code you terrible humans that insert security vulnerabilities by accident not you all of us including myself in that right there's probably different stages of implementation you can set up like stage gates so you have your first stage be very simple and you move on okay so you could basically take the design group the features that you definitely want to do and create kind of rather than trying to write the whole thing at once and then being so shocked when it doesn't work together write small pieces like a core component and kind of build out from there what about automatically wouldn't it be super cool if you could not code like I wanted to have computers write programs so he wants to put you out of the business so and this could if you had the design in a formal specification you actually maybe could do this right if you have some formal design you could actually have something that spits out code that matches that design you could use that as a starting point but how do you know that the implementation satisfies the design we've talked about this a lot but let's kind of talk about all the things that we've discussed so far so somebody mentioned that we have the person that created the design create test cases that you are either executable or not maybe it's just a table of test cases is that the only way? well there are different ways you can test it on individual classes or modules where you can do like a system level test yeah so you could do unit test which is touching like an individual function or module you could do more broader integration test which testing or system test like test the system or other components end to end does that still mean that your implementation 100% satisfies the design satisfy the specifications right ah yes it does need to specify the specifications so this is also part of assuming right at every layer if you're just doing what the design says but the design was wrong in terms of the specification then you're building the wrong thing but let's say that the design is a faithful representation of the specification for now so just considering so the thing about these are like different layers right so just considering the interaction between those two layers so you have some sort of like beta and alpha testing or something? similar to the staging idea of basically don't build everything at once build a small piece and then release that or test that release it to customers maybe do a pen test or have a red team review of that right then and then build out from there again we come to the problem how do we prove that the implementation actually satisfies the design stakeholders say so? stakeholders say so yeah that's a very software engineering point of view I mean at the end of the day I guess if somebody's paying you to do it then and they say it's good then it's good at the same time they may not have the expertise to identify all the security vulnerabilities that you accidentally left in so just because they say it's good I guess from an assurance perspective maybe functionality is good but then when you think about security assurance then you may not use as a trust indicator yeah exactly yeah so we need to somehow convince ourselves and it's through these types of processes that we've talked about of red team review including testing often times and this is actually something that I think came up about a year ago so I don't think I included this but one of the things so often times so people work to the company done automated testing what were your test cases testing for basically it depends on exactly what it is but essentially you're verifying the design that the system does essentially what the design does how many of the tests what the system is not supposed to do test cases that test for no pointer errors or test cases that give I don't know 60,000 bytes as input to see if any buffer overclothes are triggered do you have test cases that test if have test cases with SQL like special characters like single quotes to see if any SQL errors are generated which could be an indication of a SQL injection vulnerability yes, no you could do that do companies do that huh? some and I think I don't know if he wants me to name him so I won't say who he was or where I was from but this was actually a really interesting idea of basically automating security testing and putting it into your testing pipeline so that way rather than have it be this thing that humans do at the end actually have test cases that are dedicated to security parts right and it's again a very difficult problem because you're not it's not a positive test of saying does this function or does this module do what it's supposed to do based on the specification yes or no you're trying to prove the absence of behavior right is there a security vulnerability here and so proving that with test cases is very difficult but yeah so sort of proving what happens when something else breaks in your program exactly and so thinking through and the difficult thing is creating this in an automated way that you can do it and so it's there's something I want to expose you to to be thinking about because even the implementation is a difficult thing and the other thing to think about is I think we touched on this briefly is how do you even know so you're probably looking at the source code but how do you know that the compiler didn't accidentally introduce a vulnerability do compilers do that yes how bad compilers there's specifications can be ambiguous and like certain edge cases so like if you don't declare an initial value for a variable what does it become yes it's whatever garbage was in that memory location previously right so I've seen specifically actually the reverse of that where somebody I think it was I can't remember what application but it had in memory the user's password and so they had a line that zeroed out the user's password but the compiler being very intelligent and optimizing the code said well this password is never used later on in the program so why zero out the password that's just a waste of time so it knocked that instruction out and that didn't actually get included in the binary code and so then you had a vulnerability that existed only in the binary version but not in the source code because the compiler is being too smart so to make this problem even more complicated you have the problem of does your source code match the design and then does the actual binary that is running on people's systems match the design match your source code and match your design ultimately at the high level so really the point of all this is to get used to think about how difficult these problems are and then not even to stop there to think about well now you wrote some software now you have to deploy it you have to configure it you have to operate it are these all areas that can impact security yes yes very yes right so now you have to ask the problems well how is this being deployed how is it being configured how is it being run and operated especially when you want to think about security if you're talking about access to user data the system itself maybe is 100% secure but if everyone in your company is admin on a server where the user's data is running then that's a horrible problem right so and this really what I I mean this is why I love security and why it's so fun is all of these things have to actually be correct right you could have a security bug at the specification level design level implementation level deployment configuration and operation level and so you actually have to as an attacker understand all of that in order to see how it's supposed to work and to see if you can explain the vulnerabilities and then the cool thing from the defender's perspective is you have to defend all of that right you have to make sure you want to create a secure system work and I said you can't create a secure system if you want to have a high level of assurance that your system is secure you need to make sure that every single one of those levels is taken into consideration yeah have you ever seen bugs cancel each other out like there was a bug in the implementation but then like the way it was deployed like oh yes actually or yeah I actually have an example right from DEF CON so we so the way everything worked is we had basically a central like database that had a REST based API so that all the other components could talk to it and they would log events like this flag was captured or this thing was done so everything would feed into this REST API and then as I was doing this I realized wow I'm going to be the only person who can ever use this thing and I don't want to interact with this REST API directly I want a nice admin interface so we can do things like release a challenge or release network traffic to the teams so I coded up this really ugly admin interface that actually used the REST API and while I was doing this so we were doing this in the Kubernetes admin cluster and I changed so my original security design was the REST API should only be accessible within this admin cluster nobody else should ever be able to access it as a layer of defense so if the teams ever did something really weird they would never be able to talk to it then I realized well I want the rest of my team to be able to access this so I used the option to expose it on a node port on port 30,000 so every so we had two machines that were part of this cluster if you may request those machines on port 30,000 you would see this admin page which by itself is fine because the teams could never talk to that IP address except that they needed to be able to access a team interface and it's getting very complicated now but essentially we were so this one of the machines had basically like was impersonating an IP address so it had like three IP addresses one was this team interface I realized this stupid node port was binding to all IP addresses I thought it was just its internal IP address so the teams would access the team interface and because I didn't restrict what port they were accessing which is another problem they could scan that IP address see that port 30,000 was open go to it and actually saw the admin interface so that was a huge bug in deployment and probably configuration we figured it incorrectly and so we actually and it was like that for probably like three hours and it turned out that oh so we got some early warning signs because some of the teams were coming up to us saying hey I saw a service like up here and then go away and like that's weird what are you talking about like that didn't happen and then finally another team came to us and said yeah we found this admin interface that doesn't look like part of the game you should fix that so then we blocked it and we found out that another team had found the admin interface and was using it to toggle some services as visible or not which got up here to everyone else so talking about bugs cancelling each other out I didn't have enough time to write any documentation about how to use this rest interface which is really good because they weren't able to figure out how to like add events to give themselves more points so all they had was basically like the size of the game and a way to like change services as visible or not which is not very important and it doesn't give them much information at all so it was not as bad as it could have been because there was no documentation so that bug of me and I literally had a bug in our like github issue thing of like write documentation of how to use this API and I just never did it so yes it was a huge security through obscurity situation so long story that was too difficult for anyone to use interfaces I think the interface was so difficult to use just really bad UX design it's an API right APIs are complicated right how do you make a general purpose event API where you can log events but yeah in general documenting APIs is difficult and everyone everyone on that team had yeah so the moral is is not don't write documentation definitely write documentation to user interfaces make sure you secure things is the appropriate response cool so then again we have the problem of how do we so we implemented something we're fairly sure that it's implemented correctly which means according to the design and the specification but now how do we prove or test that the deployment configuration and operation is actually secure monitor it so put monitoring into place to it so that way you can actually respond to that and try to recover that like what Google Chrome stuff had like meet back programs yeah this is actually a really difficult problem for people shipping like desktop software like how do you know I mean they're looking for more bugs than anything else but yeah so how do you actually result back from some user's machine so that's why all these things have options to report to them and Chrome's even taken it farther and doing things like report to us that there's a weird SSL error because maybe somebody's impersonating and stolen some security certificate we can do penetration testing at any of the last three steps exactly so you can do and this would be different right than the implementation because against the implementation you maybe are just have a binary maybe you have access to the source code trying to find bugs that way but here you're actually trying to exploit the system as it is deployed configured and operated right so this is actually why most penetration tests happen at this level right because you know so you know if a penetration testing team finds a vulnerability and is able to exploit your system as it's deployed configured and operated that means that a bad person like a hacker can do that as well right like we talked about things cancelling out maybe so when you can find the implementation level maybe problems but maybe how it's deployed actually isn't a problem but you still maybe want to fix that so how expensive is it to fix problems at each of these stages what's the cheapest it's the cheapest if you fix it in the specification why because you haven't done anything I mean you've written the specification assuming you find it at that stage and not when you've got the whole process yeah so it's cheaper to fix problems like earlier in the process because each layer has to be satisfied by each previously there exactly and you may need to completely redo right you may find such a bad problem in the specification we have to completely rethink the design which means you have to more or less throw out a lot of your implementation and redo that right so it's incredibly expensive this also extends when you think about just implementing it's a lot cheaper to fix a bug on the fifth day than it is on the like hundredth day or three hundredth day why is that maybe we're going to come in with an old code base what was it different people are working there now oh yeah that's I mean a huge problem right yeah so you have the code more or less gets more complicated over time it things that were good ideas at the time right now to be not so good ideas because the specification has changed the code is brittle and you change one piece and another piece breaks and you have no idea why right all these things make it really difficult so and people often forget this like there's a lot of hate for a long time on Microsoft about like people would report vulnerabilities of Microsoft and then want to release the details about that exploit within like 30 days and Microsoft would always because and what for security researchers it's difficult to understand but anybody heard the idea of like a test matrix there's just like a table of all the combinations of operating systems and hardware or whatever that you need to test in order for something to pass so when you're talking about let's say you find an exploit that affects like all the way back what's the least supported Windows 7 still supported so so let's say really bad and say yeah all the way back to let's say Windows Server 2003's I don't remember all my windows but I think XP is finally an end of life but let's say everything after XP you think about every single operating system that whatever change and fix no matter how minor you need to test on every single version of every single operating system that has been released since then on also different types of hardware with different types of software configuration because something weird can break and so that testing takes a lot of time even if the fix itself is very small so this is something that I don't know keep in mind that like finding a bug in a deployed system can be very difficult you need to test it and reevaluate it have you heard about SimCity do you know SimCity? yeah the game yeah so this is a really interesting I think his first name is Raymond I can't remember his last name I think Chen, he's a Microsoft engineer who has a really good blog about the early days of Microsoft and Windows so when they first moved I think to Windows I want to say 95 or something they because when you make this operating system if your old software doesn't run on it anymore nobody's going to buy your new operating system and so they would test all of the popular software and they found out that SimCity was taking advantage of some undocumented features of like Windows whatever it was before 95 I don't remember if it was DOS or something I don't know it was taking advantage of some super weird feature so instead of being like SimCity you suck fix your software Microsoft actually put in a shim into the operating system that says if the program executing is SimCity then change the behavior of the operating system in this way to support it and you think that's just one example so you think about all those examples over these years it's absolutely absurd that anything still runs so stuff's complex fixing stuff is hard so okay so we've identified a problem do we fix it it depends it seems like a silly question the answer should be yes you fix it your front doors broken your windows broken you fix it am I moving houses though are you moving houses is it somebody else's problem are you moving out of your apartment or like if your screen door breaks and you never use it what are you caring your screen door is not very secure right how hard is it to break a screen door cut it or just punch it I don't think it's super I've never seen a screen door that's very secure right so why don't we fix it in this case it might also be more expensive to fix it than to just build a new one exactly so maybe the I'm trying to think of a real world example but maybe the problem that it's going back to risk maybe the risk of the company is only like $50 and you're going to spend $50,000 to fix it maybe it's not working I was going to say like more expensive to fix it than build one sometimes like cars or motorcycles or transmission problem where it's like well I'm going to spend $1,000 to fix it where I can respond so both are very good points going back to the legacy code based stuff you were talking about if we have no one at the company that really knows how to work with the legacy code base it might be and something breaks in that but we're working on a new solution to it it might be better to just kind of burn the old system down and move to a new solution yeah that's exactly it so for all kinds of reasons maybe the bug or the problem in the old system is so difficult to fix that kind of makes sense or maybe you were starting from scratch or maybe you were starting from scratch so you know we invest all this time and money into building this new system we'll just stick with the old one at least until it works this is usually the optimistic scenario from people who've never gone through that because you find out very quickly that old systems have a lot of bugs that you thought were bugs that are actually features that everyone uses and you have to replicate all of those bugs or features otherwise they get very upset and at the end of the day you have a code base that looks is just a pile of garbage to a new person because they don't understand all of those weird corner cases so this happens a lot it's called I think the second system syndrome where you build one system and you're like man it's a pile of garbage I finally know how to build it right and then you spend years building it right and then by the time it's done requirements have changed like it's not really useful anymore and this is getting across the idea that change isn't free right fixing a bug fixing a security problem is not free which makes sense we just talked about when in a life cycle of a system you find a vulnerability usually the earlier the cheaper it is to fix which clearly is better you'd rather fix a bug for ten dollars than ten thousand dollars or a hundred thousand dollars and so and this is really and this is kind of applied at all areas and this is kind of having a discussion around this and this is the other important one of the things I hope you're getting so far in this course is we don't just talk about how awesome it is for things to be secure and how security is the most important thing on earth and even though I do really like security always have to consider the business context and the business situation right do you actually want to is it worth the cost to do whatever it is you want to do it's not then why are you doing it you're wasting money so is this something we need to so what are we weighing here when we think about cost benefit in these terms what are some of the things we're trying to weigh getting whatever we need to fix the cost the cost one factor how much you value user information okay so let's say you're trying to weigh the downside in some sense with the impact right so thinking how much you weigh user information what are some of the things that you as an organization you would think about there do you really just care about user information I was going to say like your reputation if that user information was leaked so not only do you have to think about the monetary value or how many you could try to estimate well how many people would cancel accounts if we lost all their data right but you also need to think do big organizations and companies care about their brand never considered this before I'm not saying a lot of nodding the answer is yes they have marketing as the base of the entire department dedicated to brand right more or less so do they care about these things so often times they're thinking about the PR ramifications of do we want our brand associated with this latest breach like would you buy an Equifax shirt or another product Equifax lost what was it like hundreds of millions of users personal information and Equifax is one of the major credit credit bureaus yeah I think that's the term right so they lost all this huge data like you think about what your credit with these credit agencies know on you it's everything all your accounts to even check your own information you have to pass a bunch of tests so they took a huge a huge hit to their PR when that happened I actually don't know how it's recovered since then I guess I didn't trust them to start with one of the things do we consider down side visibility visibility feasibility of the fix exactly that's a great point so if your fix if your fix is well I will solve the problem of people breaking into my house by filling it with water right if I waterproof everything fill it with water that way any attacker will drown you highly unfeasible to actually do that right to have a house you want to live in unless you're some type of murder person so okay that's a great point so other things yeah like the context in terms of like who's using it like I know there are some systems of like ASU for example on like some submission sites for homework that are not very secure at all because only people who are like students are going to submit to that I guess yeah we're thinking about well for students it's a difficult case we we think about this when we're thinking about DEFGON CTF we you know you have usernames and passwords to log in but once you log in really the only functionality at least in qualification you can do is submit a flag as somebody so it's like are we really worried about people breaking into other people's accounts so they can only do a positive action on their account it's like I think this thing pops up on twitter every month in a while of people being like I can't believe I have to like verify my identity in order to pay my like student loan or something or deposit money into your account they go no I would be happy if anyone wants to do this here's how to do that right so you think about risk there the risk is insanely minimal to the person like what do they care somebody else pays off their wouldn't you if you're paying off debt on that account would it be possible that your payment information could be on that account for sure yes it all depends on what information I think the specific case I was remembering was literally just like a pay money into an account and nothing else but yes in general in general it's very easy to take a very myopic like tiny view and say well this isn't important but you forget all that other information that's there that can be used yeah yep yeah that's actually a good point right maybe the reason for this mechanism is you as a person yes you may be fine with other people depositing into your account but when you make a deposit you want to make sure that goes into your account and not somebody else's just because it was easy and you mistyped some letters I was going to say maybe the legality of it like maybe of the situation I don't know what the law says a company has to do with their security measures oh legal is a huge do you guys do you think the law is important do you think the company these things about law is important why they get fine they get shut down they get go to jail right these are all things that can happen a corporation is usually interested in making money or continuing to exist in some form you need money for that you also need to not go to jail so this is actually something that terrifies companies is legal requirements actually my job when I was at Microsoft was really born out of legal requirements so Microsoft I think it was the early 2000's was sued for antitrust because they were considered a monopoly and were abusing their position does anybody know why what's going on in your browser part of it was to the internet explorer yeah yes although those are I think well I'm not a lawyer let's first practice this the browser thing I think was more the EU the EU had a big problem with that because they forced windows to include an EU software browser selection window when they installed the operating system but it definitely had parts to do with that so what it turned out was it because they were offering so much functionality with their OS as well I think because they were adding in all these apps that people would normally have bought in other companies right so you think like the operating system provides a lot of features right to application so from Microsoft's perspective well just because we made a really good web browser and do not misunderstand internet explorer was a really good web browser way way way way back in the day the problem is they never upgraded it to an even better web browser for like five or six years until Firefox got attention but that's neither here nor there so what they found out part of the problem was is they had this big operating system which has a lot of features a lot of system calls a lot of things that applications could do they found out that Microsoft was developing their own apps that would run on their operating system that were using undocumented features of the OS that other apps weren't using and this was one of the big reasons that they got hit with this antitrust lawsuit and I went for I think it was I don't know if it was intern new employees it must have been full time employee new employee orientation one of the things that they have an automated system that every build of every Microsoft system looks for calls to private OS internals and if it does that it flags it and that's like a high priority high severity issue that you must fix immediately and so how that ties into me is the group I was working in they had this problem where now they have to either change their apps to not use that functionality which is terrible and it takes a lot of time and effort or write all this documentation of how to use these these features and so that's what this tool that I was working on was a system to help technical writers write technical content and it came out of this need this literally legal obligation for the government because you will write documentation for all of these APIs that they had to do and satisfy so a bit of a tangent but I think but it's all about you know when you think about that as Microsoft do you think they sat there being like will we comply with this request or not it's like no we want to be a company continuing to operate we also want to be one company one of the things they could have done was ripped them apart like they did to AT&T doing monopoly and then they basically ripped them apart into small companies regional companies so this is all about cool so any other comments or thoughts on the cost benefit analysis of what factors to consider like the cost of implementing the security measures versus the cost of like incurring a loss to some stores security breach how is the security best for you? exactly so cost one of the things that the really important thing here is to not just think of cost in terms of dollars right you think of dollars you think of time you think of economics opportunity cost right the time that you have a person fixing or implementing some security mechanism is time that they can't develop some new feature which your users are going to pay you for when you think about cost on the negative side right you want to think about all these type of cost of not just monetary cost but all kinds of areas any other thoughts? cool so one of the parts of basically cost benefit analysis and this is driven by reading the ISI on Tuesday was looking at how do you kind of analyze the risk in a system or a specific component and really what you're trying to answer the question is like should something actually be I mean protected or secure right and so what drives this decision? flip a coin percentages percentages what kind of percentages? like the percent of a system failing what is the percent of the system failing over this period of time and then you could pay X amount per month to keep that system updated or you could not depending on what you want to do with that part of the system yeah so you may try to quantify and try to understand what are the likelihood that something happens what type of assets for a company do they care about protecting? is it everything? trade secrets? yeah so secrets that keep the company basically in business okay so user information is that? yeah personally identifiable information right so user data that's sensitive to the users themselves not necessarily the company yeah controlled information controlled information in what sense? like the nuclear designs and well that's classified information depending on what level so it's information isn't necessarily classified by the government as hey don't release this because it's a secret for the government but it may also be information that's kind of just a category of information that's just kind of like we don't it's not classified but you really shouldn't just kind of be throwing it out in the parking lot anyway so like PII employee files or even you know just stuff like trade secrets as well so generally controlled information anything else? what about like the CEO's laptop? that's something they care about depends on what's on the laptop depends on what's on the laptop oh depends on what's on the laptop what do you think it takes to be worth protecting? what can I do with the laptop yeah what can you do with the laptop right is it can you get all of the CEO's emails do you think that's a useful can you send emails as the CEO can you access internal corporate information there might be incriminating files yeah yeah incriminating files what about like this laptop am I the CEO of a company? you're a potentially class so much fun no I'm not but should my laptop be protected and why? what do you have on it? what do you think I have on it? grades tests yes these are all things I have on this laptop should it be protected? or are you going to buy an Apple product if it's not are you connected to the network today? I am connected not right now but yes in general somebody can get it but why should it be why you would when I see that should an asset be protected I would ask who wants to take it any of you like the exams or homework assignments test cases yes please yeah perhaps how valuable an asset is yeah so probably not I mean yes so monetarily yes we have some servers that are worth like 20,000 dollars it would be very difficult to steal not super difficult but they're heavy so I think you need like a crew so legally so ASU actually has a legal obligation to keep all of your grades confidential I believe it's FERPA requirements so this is why if your parents for some reason tried to call me to see how you were doing in class I have a very easy response to that of A how did you get this number and B do that because of FERPA legal requirements also the same thing with recruiters or something unless you give me explicit permission to release that information I cannot and this is a legal requirement which is why that this asset is really important to ASU because if I lose this asset and the hard drive is not encrypted and then somebody now has access to all the grades of all the students of all my classes that's a FERPA violation so this computer is encrypted you will not guess the password so don't steal it so it's something that you really do have to understand the business right to understand what assets should be protected obviously the CEO's laptop or whatever makes perfect sense when you think about like I guess me versus Michael Crow you think like well of course Michael Crow's laptop is sensitive I am not at Michael Crow's level at all so maybe like who cares I'm just like a mid-level employee at ASU but when you realize oh actually this laptop has sensitive data that we have a legal obligation then you see why the security of this laptop is very important mixed in with everything we've talked about right is what threats does this asset face right why should we and why are the threats that it faces important your analysis of the risk sorry I need like a one or a two so again well I mean for you we don't really have to worry about like big state actors coming after your laptop right why not I mean I could be worried but is that a useful way to spend my time being worried worried about what? big state actors and I worried about a big like state actor like trying to have Russia hacking into my laptop yes I hate to break it to you but they do not care about your grades they may be interested in research although that's more like in the realm of movies especially because basically all the research all the code for research will eventually become open source anyways so it would be silly to try to break into that maybe they want to break into the laptop to try to get access to DEF CON to give their CDF team a leg up but that's a little bit far fetched so I'm not super worried about that so yeah that's what I'm thinking about with threats but what threats am I worried about am I actually worried about somebody breaking into this laptop probably not what am I worried about what was that what's stealing stealing like physical physical theft what else I trust you destroying it damage but in that case the records are released so I actually don't really care about the pile of water what if one copy was upon your laptop yeah that would just suck but legally I think I'm still good on fur but cat user what are you saying I mean if you accidentally release all of our grades then everybody else have I shared Dropbox link to you all with all of my grades what about just losing the laptop like not even theft just leaving it in a taxi cab or whatever or just accidentally leaving it here that's actually more of what threats I'm worried about than actually somebody physically stealing it and so other things we think about is so what are the consequences if it is attacked so we just talk about that so like we said I'm not really worried about the threat of dropping this in water because I don't really care I mean it would suck I guess I do have apple care on this but like from a legal standpoint those FERPA records are fine covered in water other ways to think about is what level to protect an asset why is that important yeah exactly so if you're and this goes back to the house example we talked about right it doesn't make sense if we're a normal person living in a house it doesn't make sense to like higher guards to patrol the outside to put up 12 foot electric fences because we don't face those kind of threats so it doesn't make sense the level of protection should match what we feel that the asset is what consequences we'll face if it's attacked and thinking through the threats does risk remain constant you suddenly won the lotto and had $8 million or $800 million and it was publicized your name my threat model might change if I told you all that I had which I do not remember that I have like 10,000 bitcoins on this laptop that would change things you would be very interested in my office hours laptop so the business circumstances could change with us if there's like a vulnerability that's like announced and it's like really easy to do and maybe you're more at risk of being hacked or something so when we consider we need to constantly reevaluate we're just thinking about what threats it faces and what are the consequences if we're not considering the current landscape of threats we may not be thinking about that what else keep that in mind when I'm super famous but yeah it's something that people as you have more let's say social standing people have an incentive to break into your you may think that nobody cares about your phone but if you all of a sudden become this super popular I don't know we're all set okay the other way to think about this is think about a company what do companies release every quarter earnings earnings report right which is how much money they made in the last quarter is that important information yes why yes because it determines the price of the stock right, analysts actually guess on what the earnings are going to be and then the market basically adjusts roughly to that and then if they beat the expectations of the market then the stock usually goes up so if you think about that data and the data on that financial information right up until the moment it's publicly released is super important information right, strict confidentiality requirements and I think there's probably FCC consequences if I have to say FCC consequences if you release that information early because it could be insider training or something and in that moment that it's released that data is now work less because it's public, everyone can see that data that's literally the point so there there's a temporal aspect to the risk to a system because you care about that data in that day or two leading up to the release the right one is released and you don't care anymore and then as always how to quantify quantification turning things into numbers very difficult, what's the risk and then I don't know, that's tough alright, we'll talk for a bit on laws and customs and we are almost done with this