 If you're like most companies, you're probably wondering, what is the best way to build a secure IoT solution? Of course, an important element to how you go about doing that is by performing security testing, by engaging with a company or doing it yourself in order to try to find security vulnerabilities so that you can fix them so that you can build better, more secure solutions. Now, if you're like most companies, what you've probably done is you've probably gone out to seek out the service by name of penetration testing. But what is penetration testing anyway? Why is this term so commonly misused and misunderstood about what it is and what it isn't? And that's what we're here to talk about today, is to help really demystify what this term actually means and how to think about dealing with the rampant term confusion that exists around this idea. So when we think about what the idea of security testing is anyway, the first thing we have to realize is that there's the umbrella term which is security assessment. And security assessment refers to essentially all types of security testing where the goal is to, in some way, improve the security of a system. But what's interesting is that most organizations actually ask for penetration testing. However, what they're often sold is something else, is usually vulnerability scanning. Yet what further confuses this already confusing situation is that what most organizations actually need is a third thing altogether which are called vulnerability assessments. So what are these things? How can we tell them apart? Why do we need to know what the difference is and how does this term confusion actually manifest as a problem for most organizations? So let's start with penetration testing. Penetration testing is actually a very specific thing even though the term is often misused. Penetration testing is a time constrained scope of a specific effort in order to try to arrive at a somewhat binary outcome. Did or did not an outcome was that achieved. They're really designed for very thoroughly tested hardened systems that have already gone through a tremendous amount of security testing already. And they essentially take a specific real world scenario and say, hey, what would happen in this specific scenario? The metaphor that we can use to describe this idea comes from cars. And if we think about auto makers, what do they do when they want to understand a specific real world scenario? For example, what does a car maker do when they wanna know what would happen in a head-on collision? They literally crash the car against the wall. That's kind of what a penetration test is like. You take a built system, you actually go execute a real world attack scenario and you see what would happen and you measure the result in a binary sense. Did or did not whatever you were seeking happen. Were you able to escalate privileges if that was what you were going for? Now, that's pretty different from vulnerability scanning. And unfortunately today, vulnerability scans are really presented as penetration tests when really they're not. And what vulnerability scans are is they are automated ways to look for the proverbial low-hanging fruit, the easy to find issues, the known issues, the unpatched vulnerabilities. These are the kinds of things that scanners are really good at looking for. They're really effective in identifying the first thing that the attacker's gonna see because the attacker's gonna run a scanner before the remainder of their testing effort as well. So if we go back to our car metaphor, vulnerability scanners are more like when the check engine light comes on in your car. And you take it to the mechanic and the mechanic takes that little diagnostic tool, they stick it into the dash and it essentially queries the onboard computer for what might be causing the check engine light and it spits back some codes to say, here's how you turn off the check engine light. Now, the problem with this is of course that it can only check for known issues. It's very quick, very easy, very inexpensive. But compare that to what we just described metaphorically to what penetration testing is, right? The idea of crashing the car against the wall, that's pretty different from this diagnostic scanner. And that's why it's a really big problem when someone says, hey, I need penetration testing done on my system. And yet they are given a vulnerability scan instead. And unfortunately, in the way the world works today, if you were to go do a Google search for the term penetration testing, the majority of the results that you're gonna see are actually vulnerability scans. Now, further complicating this, of course, is the idea that what most organizations actually need, in most cases, are vulnerability assessments. Now, vulnerability assessments are more comprehensive approaches to really understand how do all the different security mechanisms of the system work? So if we think holistically about the system, how well will it perform against an attack? And going back to our car metaphor here, this is sort of like the entire automotive safety engineering department. What do they do? They evaluate how do all the safety systems work together? How does the airbags, how does the site impact beams, the lane departure technology? How do all of these systems work together in order to really maximize the likelihood that in the event of an accident, the passenger will walk away safely? So those are three really, really different things. They entail different investments of time, effort, and money, and they deliver different outcomes. But this, of course, begets the question, which is, well, what should we do with this information now? Are we supposed to be memorizing these terms? And as much as I'd want people to use the right term to describe the right thing, I know that in the current marketplace, the way the world works today, this term confusion is too rampant that we probably can't solve it. So even if you, attending this talk, might now know the difference between these different terms, the people that you communicate with, they might not. The broader industry might not. So instead, what should we do? We should start with a goal. So when you're communicating about what actually needs to be achieved, what we need to do is we need to start with a goal and use the goal, the outcome that we're seeking, and use that to be the descriptor of which service we should be getting. Because then, it doesn't matter whether an organization is using the wrong term to describe the service. As long as we understand that the outcome is what we're going after, then we can make sure that the approach will be right. So if the goal is to take a mature system that's already been through a lot of security testing, it's already very well-hardened, and you want to see how it performs in a simulated attack that's very narrowly scoped and has this sort of binary outcome that you're seeking, that's when you want penetration test. If your goal is to check for common issues, to look for known vulnerabilities, to look at things in a very quick and easy and inexpensive approach, then vulnerability scanning actually makes a lot of sense. And then if your goal is to actually find vulnerabilities, including those custom exploits that would be unique to your system, you want to be able to assign severity to those issues. You want to be able to understand how to remediate the issues and prioritize those based on the prioritization derived from the severity ratings. That's really where you'd want to look at vulnerability assessments. So when we think about this idea that there is a way, there's a method that we want to actually approach how to find vulnerabilities, well, what does it mean when an ethical hacker or a security analyst or whatever term you want to use is actually looking at a system and is going through an attack progression, trying to understand how can we find these vulnerabilities? Well, there is actually a method to how this type of individual or this group might go about finding these vulnerabilities. And when we break this method down, we'll realize that this distinction between what penetration testing really is when it's actually presented as a vulnerability scan is actually presented as a pen test and vulnerability assessments. You take all these together and you look at the method, you'll realize where these deficiencies sort of appear. So let's talk about this method and all these ideas that we're talking about here, by the way, they come out of a book that I wrote called Hackable, How to Do Application Security Right. So everything that we've talked about goes into way more depth in the book, including these ideas that we're talking about today, but also many other ideas all related to how do you build better, more secure systems? So the first step to this method is to analyze the design. Now, of course, the assumption that's baked into this is that the organization that's doing this testing that's actually seeking the vulnerabilities is an outside organization. So this is why companies will work with outside security consultants, security testing firms, because they have that sort of independence, that specialization of the skills that are required in order to do this, et cetera, et cetera. And so this first step of understanding the design, that's, of course, important for someone who didn't build the system. They need to understand how did the system work? What business problem does it solve? What assets does it provide access to? What are the workflows for provisioning access or onboarding or off-boarding users? Now, here's an important thing to note as we think about this, which is that this step actually doesn't appear in a scan-oriented quote-unquote penetration test, because when you're running a scan to a scanner, it doesn't matter how the system works. The scanner is just looking for known vulnerabilities. There are fingerprints or signatures that it's looking for, and it doesn't matter that this system works in financial services or works in media entertainment or is a very specific type of internet-connected device. It doesn't matter. But very much it matters when you're thinking about how do you find those more important, most more sophisticated and more catastrophic security vulnerabilities? And we're gonna talk about this later, but in order to be able to get to the more advanced tactics, things we'll talk about in a moment, you first need to understand the design. The next step, I'm actually gonna introduce it as two stages together as one because they're highly related, but they really are two different efforts. So one is that we want to actually run scanners. Now, even though I just spent a lot of time talking about some of the shortcomings of scanners, they can only look for known vulnerabilities. They certainly can't look for custom exploits. The severity ratings of them are of sort of one size fits all. They're not customized to your particular use case. Nevertheless, we still wanna run scanners because scanners do have a lot of powerful benefits they deliver. So first is, of course, they make things that would be very onerous to do manually happen very quickly and thus reduces the cost in the effort to do that. It's also the first thing the attacker's gonna do. And what do we wanna do as defenders? Of course, we wanna think like the attacker. So the first thing we should do is the first thing they're going to do and we wanna see what will they see if they run a scanner against the system. And then the next thing is, of course, look for known vulnerabilities. So scanners do this, but of course you should do an element of this manually as well. But the idea here is that attackers, they're just like you and me. They're looking for the most efficient, most effective, best return on investment of their time, effort, and money. And so what is the logical way that they think about this? Well, they say if most people have this type of problem or if this is a common type of issue that afflicts most organizations, it might be a problem here too. So let me go and try and see if this particular system suffers that flaw as well. So that's why you wanna look for known vulnerabilities because that is a shortcut to success that many attackers will also do as well. Now, these three ideas, the combination of them are what I refer to as the basics, sort of the fundamentals of any sort of approach to try to find vulnerabilities. Because what you're doing here, these should make up the baseline of any effort to find vulnerabilities, assuming the system has something worth protecting. And that's an important assumption to draw out because maybe your system doesn't have something worth protecting. If it's not worth protecting, then it's probably not worth investing in finding vulnerabilities. But let's make the assumption that you have a system that protects something worth protecting. Now, unfortunately, this is actually where most security efforts call it quits. Now, not everyone even knows that's actually what happens because of this problem where many companies think that they are getting penetration testing and they're really getting a vulnerability scan. They don't realize that actually all that they've done is a couple of the easier elements of just the fundamentals. And they actually don't even get all of the fundamentals because as I mentioned, when scanning is the focus, there is not an effort to try to understand the design of the system. Now, the next stage is actually not a stage at all, but it's actually a gap between what we just talked about, which are the fundamentals, and what's to come next, which are the advanced tactics. And I've come to refer to this as the capability gap because really what this is is this is a barrier, a gulf almost, a gap between these fundamental steps which can be done largely with tools. They require a low to moderate level of skill and expertise. And that separates those steps from the advanced tactics which require a high degree of experience and expertise and they really rely on a manual emphasis. It doesn't work so well with tools because a lot of the things we're gonna talk about, there's really no tool to help you figure out those types of issues. So the first advanced tactic is this idea that's called abuse functionality. Now, abuse functionality is the idea that you take the existing feature set of a system and use it in the attack against the system. So there's a really interesting analog metaphor that we can take a look at of this kind of funny video that's going around the internet right now. And there's this guy, he's in, I'm not sure exactly which beach town, but some coastal beach town, I believe in Southern California, might be Santa Monica, but it's somewhere that has really difficult parking. And he has this coin and it's attached to this little plastic strip and he's standing there at a parking meter and he waits for someone to be able to get access to a parking spot where that meter is. Someone pulls out and then the next person pulls in and their day is just made because they just found this parking spot in an area that's really, really hard to park. And what does this guy do? He inserts this coin. Now he's doing this while the person's parking. They don't realize this is actually something that he's doing on their behalf. And we're all familiar with parking meters, right? They are supposed to be able to ingest coins. The functionality is insert coin, the machine registers that a certain denomination has been received and as a result, the meter says, okay, you get this many minutes of parking. So one coin is supposed to do that. The functionality is you put a coin in. But there's really nothing to prevent from pulling the coin back out. And that's what this guy figured out because he took this coin, attached it to this little plastic strip and he would insert it and then pull it right back out and then do it again and again and again and again until he maxes out the meter all with the same coin. Now, I don't know about the ethics of this, of course. He clearly was looking for some sort of, maybe donation for his services. But the point is the parking meter is not supposed to work that way. But nevertheless, it did. And that's why this idea of functionality abuse is really important because we often don't anticipate that a system can actually be abused in a certain way. Yet nevertheless, often it can. And those are the types of issues we really need to investigate for. So we can apply this to a technical exploit if we look at the work that was done by a hacker who went by the pseudonym of Manfred. Some of you have probably heard him speak at DEF CON presenting some of his research around hacking online games. And one of the elements or one of the key details that came out of some of the work that he did was he was really interested in the in-game currency element to these online games. Now, if you're not familiar with online games, you're not familiar with in-game currency, the idea is essentially that you can earn or buy currency in the game and then you can use that currency in order to enhance the gameplay in some way. You can buy a skills upgrade or a weapon upgrade or access to a secret level or change the outfit that your avatar is wearing. And there's these, for lack of a better term, these sort of banking systems in the game that facilitate these transactions. And I'm oversimplifying the way they work, but for simplicity, essentially the way that it works is you have your account balance and then as you want to get one of these upgrades, you'd withdraw a certain amount in order to get that whatever it was you're getting and then the result is a new balance that's in your account. And what Manfred did when he saw the way the system worked is he pulled out one of the greatest tools in the hacker toolkit, which is the question, what if? And he said, he noticed that the way the system worked was it used positive integers. They would say positive account balance minus positive number that is withdrawal equals your new account balance, which is also a positive number. And he said, well, what if I could get the system to react to negative integers? And that's exactly what he focused his efforts on and he in fact was able to find a way that was, in fact, in some senses, somewhat trivial to do, that he could make the system react to a negative integer. So where the system previously might have said something just using made up numbers here, it might say something like you have 500 coins and you subtract 100 coins and the resulting balance should be 400 coins. That's the way the system is supposed to work. Those are all positive integers. But because Manfred found this way to be able to use negative integers, now what would happen is it would say 500 coins minus negative 100 coins. And as we all learned in middle school, subtraction of a negative is actually addition. So as a result, he would get whatever the upgrade was that he was buying, but more importantly, he would grow his account balance. And now there are additional things he could do with that account balance. He could sell those coins and offline secondary markets for real money. I mean, if he wanted to, he could sort of crash the economy of this game. That's why looking at things like functionality abuse really matter, because the system was not supposed to work this way. Yet nevertheless, it did. There's no tool that would help you be able to investigate for something like this. No scanner is going to be able to find this type of issue. So these are the kinds of things that a problem solving human who's sort of connecting dots is really required in order to find issues. These are the more important and yet also catastrophic issues that afflict most systems. And so when people think that they're getting penetration testing, but it's really just a scanner and the scanner doesn't doing stuff like this, they're left with this sort of misplaced confidence about the security posture of the system because they haven't even attempted to investigate for issues like this. The next type of thing that will happen in the process is an investigation into things that are called to chain exploits. Now to chain exploits is to take two or more things and actually combine them in a way that can derive either more significant impact or achieve some sort of adverse outcome that otherwise wouldn't be possible. The maybe real world metaphor here is something that came from my childhood. I can remember when I was a kid, one of my neighbors growing up had a trampoline. And we were there all the time. I mean, what kid doesn't love jumping on a trampoline, right? And I remember the day we discovered this idea that's called the double jump. And the double jump is essentially where, if I time my jump so that I'm launching at the same or the correct instant when my friend is landing, his downward velocity actually amplifies my upward velocity. And we discovered this, of course, because I got shot to this crazy height I'd never ever been before. I mean, I even hit these branches that were hanging over the trampoline at the time, kind of experienced this moment of weightlessness and then came speeding, crashing back down to earth and thought I was gonna die on the way down. Obviously didn't. It was a scary moment. Now I didn't know it at the time, obviously I was a kid. But that was a really vivid, in hindsight a really vivid lesson for me about this idea that by combining multiple things you actually can exponentially increase the outcome or the impact of something that you're doing. So if we take this idea and we apply it to a technical exploit, we can talk about the outcome of a security assessment that we did at our organization, Independent Security Evaluators at ISE. We were working for a customer and we were doing this security assessment of this particular piece of software that we noticed had two issues. The first issue was what's called information leakage. Now information leakage is when a system gives up information that it probably shouldn't. Now on its own, that's not really that bad and isn't even directly exploitable. It's just not something you want to have be in existence. And in this case, essentially what it meant was that any user of the system could figure out the user identifier of any other user of the system. Again, not really something that's exploitable directly, but you don't want that to happen. The second issue was where the authorization model was broken. So if you wanted to change your credentials, if you want to change your password, you had to supply information as you do in most authorization models. But in this case, the piece of information you had to supply was your user identifier. Now in theory, each individual user only knows their own user identifier. So in order to change their credentials, someone can only change their own credentials. But when you combine these two issues, because the information leakage meant that you could identify all user IDs, and then with the user ID, you could go change the credentials for every user. That meant that an attacker could take over every single account in the entire system, complete system compromise. They could take over even the admin accounts using this method. Now that's what's crazy when you think about this idea of chaining exploits, because those two issues, one was the broken authorization was obviously worse than the information leakage, but both of those on their own had a certain degree of problematic, where they were being problematic on their own. But when you combine them, that's where it arrives at this outcome of system-wide account takeover. That's why chaining exploits is really, really important. So this is the type of thing that again, there's no tool, there's no scanner that can help you figure out these ideas automatically on its own. You really need to be thinking about the unique way that a given system works in order to say, can I take these two seemingly disconnected ideas and pair them in a way that delivers a really bad outcome for the system? And then finally, the absolute pinnacle of security testing is seeking the unknown unknowns. So these are the things that are often so unexpected that you don't even know to look for them. Now there's three types of unknown unknowns. We're gonna focus just on one for illustrative purposes today, but one type is the novel versions of common vulnerabilities. So for example, injection attack. Injection attacks of certain flavor are, you know, that's a common type of issue, but how that manifests in the unique way in a specific system, that might be new, that might have never been seen before. And so that's the kind of unknown unknowns that's the first type. The second type are when you have zero-day vulnerabilities in the supply chain. Of course, for those of you who aren't familiar with that term, zero-day vulnerabilities are issues that the defender has literally zero days to fix before it's exploitable in the wild. And the supply chain is all of the different things that a system leverages or integrates with. So for example, this might be a third-party library or a third-party software, maybe a payment platform that you integrate with or it might be the way you leverage the cloud, but this is where vulnerabilities in other elements of the supply chain, if those vulnerabilities exist and are exploitable, that could potentially harm your system. Another third type is what's known, what's referred to as previously unknown attack methods. So these are attack techniques that on the defender's side of the scenario, maybe we didn't realize were in existence until sometime after the fact. So a great example of that is this concept known as request smuggling. So to explain request smuggling, let's just briefly describe the way that a web application works. So you have a lot of traffic that any web app will ultimately handle. And there's a couple of ways that you can deal with that. So one is that you can have a very sophisticated and thus also expensive server to process all that traffic. The other way is you could have more simple, less expensive servers and have them share the traffic. And in order to do that method, which winds up being the better business decision, in most cases, you need what's called a load balancer. And you can think about a load balancer as, like imagine when you go to a sporting event or a festival or a concert or whatever, and you drive in the parking lot and there's the guy with the flags and he's sort of directing cars, this car park here and that car park there, that's sort of what the load balancer does. The load balancer directs traffic to the different web servers. Now the software is supposed to be implemented the same in both cases on the server software, the load balancer software, they should have the same essentially rules for how they're going to deal with these requests. But where there's a discrepancy, that can be exploitable in certain cases by an attacker. And essentially what happens when request smuggling is viable, it means there's a discrepancy between these two pieces of software. And ultimately what winds up happening is what an attacker can do is compare legitimate request and a malicious request. And it will get treated by the load balancer as a single legitimate request, but then the web server will actually serve them up independently as a legitimate request and a malicious request. And with the malicious request now getting all the way through, it's able to deliver all kinds of nasty things, change credentials, leak information, I mean there's just all kinds of bad stuff that can happen. So here's what's crazy about this as an attack technique. The timeline from when this first became viable, when the technology itself was invented and a attack request smuggling became a thing, the timeline from that moment until it first became documented in security literature was about 16 years. So think about that, 16 years this attack technique existed and no one really knew about it. You fast forward to today, that's another around 16 years and many organizations still don't really know about this and haven't had issues with it. So this is the kind of thing that again, there's no tool that's gonna help us scan our way to figuring these types of issues out. We really need to be looking at the unique way that different systems work in order to keep advancing and evolving and find those vulnerabilities that matter. If you have a system that protects something worth protecting, these are the kinds of things you have to do. So when we summarize this whole method, we can realize that again, this assumption being that you have something worth protecting, you really need all of these steps, not just the fundamentals, not just the scanning elements, but the advanced tactics too. But when this idea of penetration testing is misused to represent something else, represent really just running a scanner which is a vulnerability scan, you're only getting a subset of some of the easier elements or not even getting all of the basics and that is a really significant problem. And so my hope here today is that now you're able to walk away understanding the differences between all these different types and why you might need to understand the differences in order to get what you actually need to get. So I pass the baton to you. Ideas are only as good as the action that we take on them. So I want you to go back to your organization and do a few things. So first and foremost, evaluate what it is that you're doing for testing. Try to peel back the layers to the onion and look past what you're actually referring to it as and try to understand what's actually happening. Is it actually a penetration test? Is it actually vulnerability scanning? Is it actually a vulnerability assessment? Second thing I want you to do is understand and articulate your goal. What is it that you're trying to achieve with the testing? And then the third thing is determine are you actually getting the methods required in order to achieve the goal you're seeking to achieve? Because when you do these things in combination, what you're able to ultimately do is to build better, more secure systems. And that's what I hope you leave today's talk able to do. So with that, thank you so much. My name is Ted Harrington. Use me as a resource, however I can help you. Hit me up by email, follow me on socials. All these ideas are in my book hackable. But let's go out and build better, more secure software systems together. Thank you very much.