 Thank you for waiting with us. Uh, I'm Mudge. Uh, this is Sarah and we're going to give you a little uh, uh, information about the Cyber Independent Testing Lab. Uh, you may have been reading about it. Kind of pull back, uh, the curtains and um, so that's the introduction. I'm gonna let Sarah start this off. It kind of, as a heads up, uh, we go a lot from like a 30,000 foot all the way down into the weeds and then back up. So, um, it's gonna be a little bumpy ride, but hopefully it's enjoyable. We know different people are interested in different technical levels, so we wanted to make sure we hit something for everyone. Uh, but anyways, uh, this is just a preliminary data peek behind the scenes. So, um, if you visit our website, it's mostly just coming soon right now, but uh, this is what we're, but uh, uh, this is uh, what we're up to. Uh, just the quick peek. Okay, so first off, uh, what problem are we trying to address? It's the fact that, uh, we've been trying to get people to care about security for years, but then whenever somebody says, okay, I give, I get that it's important, what should I do? We don't have very much concrete data to give them. They don't have anything to act on. There's no consumer reports that, uh, for software security that tells them what's the safest browser. And if I put it out to the floor, what's the safest browser? We'd get a lot of very strong opinions, but a lot of those opinions would be based on what feels true rather than actual data. And that's the problem, that nobody actually really has data. Um, so, uh, what are the things people are trying to do right now and why don't they work? First off, there's certifications and evaluations which are frequently focused on processes and procedures and don't actually look at code or at the final product. So, they can't really speak to security. Um, there's industry and marketing labels, but these are frequently vague or misleading. Um, and they're, they're not, you can't compare between them. If two people have the same sticker on their product, you don't know which of them did it better or which one did just the bare minimum and then stopped. So, it's not useful to a consumer. Um, source code review definitely has its place, but it's not there to serve the consumer. It's the, either an internal thing by the vendor or the vendor is paying somebody to do this on their product and they're, the consumer doesn't see the results of this or whether the bugs that were found in review actually got fixed. Um, and so it's just the incentive structure there isn't right for consumer advocacy. And finally, legislation is frequently well-meaning, but a lot of the time ends up trying to fix a problem by making it illegal to look at it. And that's a terrible way to fix anything. So, is this mic on? Can folks hear me in the back? Great. Thank you. So, I wanted to dive pretty quickly into some of the data. Um, and just so you keep it in your mind, we're measuring how difficult it is for an adversary, how much work you can impose on an adversary to newly exploit a piece of software. The more work you can impose on your opponent and you know, without any work on your own, you know, the better off you are. So, to date, we've looked at about 100,000 binary applications, um, with about 100 plus features, uh, static features. We've got a whole bunch of dynamic ones. We'll go into some difference there, uh, and, uh, measure, you know, as bug hunters and attackers and exploiters, which I've been doing that for 20, almost 30 years now. Um, I'm old. Uh, you know, what do we see? So, when you get such a large amount of data and applications and binaries and libraries to look at an operating system, you can build up these continuum. So, here's the first continuum. This is Linux Ubuntu and this is the first, uh, 10,000 some odd binaries. And the way you read this is the easier to exploit is further to the left, going down to negative numbers here. And, you know, we normalized up at 100% on the far right. And then the number of binaries, which are executables and libraries, are, you know, uh, bucketed into the columns of, uh, five point bins. And with this, you can kind of, so the, let me back up. The installation is, this is all the base software that comes with it, plus most of the, most common third party applications installed. And because of this, you can kind of pull in any of the data sheets from the really detailed data, uh, extract, uh, the, the relevant metrics and kind of plot them. So, if we throw the first couple on here, uh, we're gonna do it on a relative hardening line underneath that's mapped to the top part. And we'll take two common browsers, Chrome and Firefox. And Chrome, you know, did a little bit better. It's a little bit more difficult to exploit. If you look at the underground market, the cost of a zero-day on Chrome bears that out. It's, uh, slightly more expensive than the cost of a zero-day for Firefox. Uh, and then the yellow triangles up on top, you know, show where that is. Next slide. We see a little further down, um, because we're, we're approaching the only fifth percentile mark, uh, you start to see some of the office sweeps, the client side applications. Um, for crowds like this, this isn't that much of a surprise, because we know that the people writing those, you know, it's, it's, they don't view themselves as the most common attack surface, but it's also why client side attacks and attachments that get opened up by these are, you know, the easiest path to compromise and exploit. Um, and next slide. This is OSX. So we've done this on Linux OSX and Windows, and we'll have some of the RTOS's and the other architectures coming online. And instead of just the first 10,000, this is the 38,000. So this is all of OSX, uh, El Captain, uh, and about 20,000 third party applications that we went out and measured, uh, and then plotted on here. So let's see what this looks like now that we've kind of seen Linux. We have a much wider spread for Chrome on the far right, uh, pretty decent, almost approaching the 95th percentile safari. And then Firefox is way down here. And this was surprising to us. Uh, it was also surprising to the Firefox development team. And they've now confirmed this as well. Um, so that's nice. And I'm gonna get into what makes up these numbers to show you the depth of the data that we're extracting out of the binaries to give you some confidence, uh, in how this is working and how it would be useful to you. So let's look at some other applications out of these 30-some odd thousand. Well, here are the different office suites that are available on OSX. You've got Microsoft Office coming in pretty, again pretty close to the bottom 5th percentile. Uh, Open Office and, you know, uh, uh, Apple's Office. Again, all on one particular platform. I should point out with this sort of view from the data, it is important to only do comparisons within the same platform. So it's not fair to say, oh, how did Firefox on Apple do compared to Explorer on Windows? Because they have different attributes and different traits. I mean, and the compilation settings are different, but we'll go into that and more. So let's pull out a few more examples. Because it's nice to have something on the far right and the far left. And why not let it be the very software that installs the security patches for your product anyway. Um, yeah, that's a little disappointing to see the Microsoft Office updater being a negative 7.5. Uh, I was talking to a team, uh, that I believe might be involved with, uh, some of the zero days going around that are popping a lot of the OSX boxes on El Capitan. And I said, uh, you know, any, any clues as to, you know, how you're going in. And they said, oh, you'll figure it out. And I went back to them. I said, all the boxes that you're going in on have, uh, Microsoft Office installed, don't they? And they're like, well, maybe. Um, and this is a real quick way of looking at not only as a defender being able to choose and say, hey, if all things are even and I can kind of choose which office suite, because it doesn't matter to me, I'm going to choose the one that imposes more work to my opponent. And that's kind of the goal. The flip side on the offensive side as the adversary is I want to choose the lowest hanging targets because I've got finite amount of resource and time as well. So why am I going to go after OSX's software updater if I see Microsoft's updater listening on the network and taking input? Okay. This is the base, just the base, no other apps. We haven't finished this one yet, of Windows 10. And I wanted to pull this out because this is really impressive. Because this is a level of consistency that we didn't see in the base of, um, Linux, uh, when we were looking at the base, uh, before we added in all the other ones, we did not see this in OSX. And the development life cycle and the compilation process that Microsoft has imposed, this is way different than what it looked like on Windows XP. Um, because you can see they're very consistent with what they put in. There are a few on the bottom. That's because there's this one set of applications that I did install. It's a big data analytics package. It was installed on all three. It was the bottom in all three. And we're going to point out the lessons learned on that one in a moment as well. Rest assured, as we put in more third-party applications and as we start to flesh out more of the binary and more of the dynamic feature sets, this is going to start becoming a bit more, uh, dispersed. But right now, this was, this is Kudos, Microsoft knows what the heck they're doing on Windows. Um, previous slide, we might question if they know what they're doing as well on OSX. Into the Michael Bunch. Alright, so the, we've shown you some numbers, but you probably want to know what kind of data we're looking at to produce those numbers. Uh, and we're, there's a lot of other industries that have trained consumers in how to make complex technical decisions. So in figuring out how to help consumers make security, software decisions, we are drawing from those. So first off, we've got our static analysis features, things where we're measuring aspects of the application without running it. And this is like the nutritional facts for the software, which functions were used and what are the complexity values, etc. Um, and then, uh, the runtime testing, the, uh, dynamic stuff where we're fuzzing and, you know, crash testing is like crash testing cars, you know, so the Manroni sticker that you see in any new car that you're buying, uh, you know, tells you how it didn't crash testing, what's its, uh, EPA expected miles per gallon, things like that. And then, um, the safety continuum where you see where that software falls compared to its peers or to the rest of that software environment, uh, that's like energy guide where you find out, you know, how much does this fridge to run, cost to run monthly versus another one or what have you. In the, uh, static part, which is a large amount of what went into creating those scores on the back end on the continuums that you saw, um, and is really going to be the focus of most of this talk here, although we will talk about what we're doing on the dynamic aspect, um, includes essentially the 100 plus featured, uh, extractions in the following three categories. Measurement of complexity, uh, turn out to be very, very important. Not only because, um, you know, the more complex something is, the more difficult it is for the developers or the creators to have gotten it right, and then for other people to ensure its correctness in operation and function and intent, um, but because it works across the board. I mean, we can do things like just the code size as a simple metric example, up through measures of branch prediction complexity, through, uh, stack adjusts and rack and stack those up through more complex, uh, things such as cyclomatic complexity for functions. And this works across any sort of operating system. In fact, it works on any binaries all the way down to extracting them off of firmware from televisions or cars. And this is a nice way of comparing, you know, how complex, you know, product A is versus how complex products B, product B is. The other areas are a bit more specific to the types of operating systems and development environments themselves. So application armory is a catch all that we say for all of the features that can be imposed and buttressed into and, uh, reinforced in the binary from the compilation stage. So if you do slash GS in Microsoft, it will try to go in and do stack, uh, protection by putting stack guards in. Uh, if you do dash D fortify source on OS X or Linux, uh, it'll go in and look at 72 common risky functions and see if it can't do heuristics to replace them with the safer version. Um, and, you know, they're more recent, uh, modern advances have control flow integrity, code pointer integrity to prevent, rock, et cetera, return on your programming. Then there's the linker. So this is, you know, where address space layout randomization comes in, uh, is important. And of course there's lots of measurements, not just is it turned on or not, but, you know, is it high entropy, uh, how ubiquitous is it across all of the components, et cetera, et cetera. And then the loader, which is, you know, what's ultimately run at the very end right when the application is put in and is being told how to mark memory as executable or not. These are all very important safety features. In fact, some of these are akin to automobiles with like the seat belts, the airbags, the anti-lock brakes. If I'm buying a piece of software or using a piece of software that doesn't have address space layout randomization, fortified source and stack guards, I'm buying a car that doesn't have, you know, seat belts and airbags and ABS and I need to know that because they've been around for decades and they definitively and demonstrably and quantifiably have made applications more difficult to exploit. In fact, some of the capture the flag people when they want to make an easier challenge binary, they just go get an older compiler that doesn't have those attributes and they build it because it's much easier to exploit. And then the final part is developer hygiene. So here's about 500 common function calls across POSIX and ANSI that historically have been the root urbane of memory corruption, code and data confusion, et cetera. And we break those out into the following buckets. There's IC functions. There's only a few of these. If you remember the poison sticker underneath your kitchen sink for the detergents and stuff, you know, that's IC. If you see an IC function like GitS in commercial code, run screaming. Those people should not be doing commercial code. And then there's classic bad functions which are difficult to use correctly. The unbounded stir copies, some of the mem copies, et cetera. Risky ones which, you know, the bounded versions and more recently we have some good functions that are hard to use incorrectly like the sterl cat sterl copies done very nicely. The problem is the only people who know about those good functions are folks who cared about security in the first place and we don't teach those in school or anything else. When you see those in the code in the binary, that's a really good sign. But it's not as good of a sign when you don't see a consistent use of them and you see the risky and the bad ones next to it. So anyway, these are all the sorts of things that we're pulling out of all of those, you know, tens, hundreds of thousands of binaries from the static component. Next slide please. Right before I go into the dynamic one which I'll just talk briefly about, we're going to do a bunch of deep dives just showing you deep dives looking at one or two of those static sort of features and then kind of pop the stack back up because it gets way too much in the weeds otherwise. Dynamic fuzzing because it's really nice to say like, well, this looks like it's a super soft target. But it's even better to say we know we can get a crash with a SIG bus or illegal instruction or whatever, SIG-SEG-V. Right now we're using AFL. AFL is fantastic. It gives us good enough coverage and we use it for three specific results. One of them is the exploitability because our environment, we really care about exploitability because, you know, we're bug hunters. We like to write exploit code. But that's not always the most important thing for different consumers, which is why we call out the level of disruptability as well. Think about a big business. Think about one that's doing offshore oil drilling. They care much more about the disruptability than the exploitability. And if you ask them, they'll say, if our system crashes, the drill bit stops. The molten core solidifies and that offshore oil rig goes offline for more than 12 months as we have to build a new one, and push it out there. I don't care. I don't want the system to be compromised and exploited. But if it is, I'd rather they're on IRC or doing it as a wearer's distribution site and they didn't crash the system. You go talk to a bank and it's a different story because they're saying, we want our systems to crash rather than be exploited in a way that we can't trust the integrity of the underlying data and we're propagating bad information and before too long we can't unroll and reclaim the books. And then the final one, which is a new one and I'm calling this out here because this is about two, three, four, hopefully longer off but it's coming and this is algorithmic complexity and this matters to any large distributed companies like your LinkedIn's, your Facebook's, your Google's because they're pretty impervious to distributed denial of service because they've got more bandwidth than, you know, well they are the world's bandwidth for all intents and purposes and they're distributed and decentralized but when you can find a small amount of input that causes the worst case in particular types of algorithms so a link list devolves or a hash table devolves to a link list you can start taking these guys out and they can't defend against it the traditional DDoS defenses de-aggregate decentralized increased bandwidth don't work and so based upon how we modify AFL we can get all three of these but fuzzing's expensive. Fuzzing's expensive so we'd like to do as little of it as we need to and so what but on the other hand math is cheap so what we're doing is fuzzing a statistically significant portion of the software and then using Bayesian math and linear regression so that we can model how the rest of the software would do because we don't care about finding a specific exploit we want to know what categories of function of vulnerabilities are present what sorts of problems do we expect to see and for that we can model that based off of the static features the like somebody who's actually looking for an exploit still has to do the all that heavy lifting but for our risk assessments we don't need that so some software will have a little a in the corner saying actual we really fuzz this and then some things will have little e saying it's estimated and that's the that's the icing on the cake the part that would make this scale really well for and mathematically we can show you to what level we're able to actually predict this you know 99.99 whatever we have and this isn't too unusual you're actually used to this in a different area explain you know the car so like for the EPA miles per gallon they don't run every single car until it's on fumes they do it for enough so that they can understand and model how the rest of the cars will do you know assuming no one's trying to trick them with the Volkswagen figured out a way around that right so we would continue to do spot checking and if we find an anomaly that goes back into our model but it should work really nicely long term for us and even out one of the traditional asymmetries of defender and attacker because this is something that works really well for defense not so well for offense yeah one of the things I learned when I was a deputy director of ATAP out in Google and got to see how Google did things is don't underestimate the power of Bayesian analysis and linear regression testing so go ahead okay so we're going to I'm just going to set this up for Sarah I mentioned we want to take you on a deep dive on just some small subsets to show you kind of the power of what you can do when you start to really tease out information on specific attributes from the binary extraction and then we'll pop back up to a higher level view of the world again okay so this is one of our automatically generated reports for Google Chrome on OSX and I got to spend lots of quality time with I can never remember whether people prefer LaTec LaTec yeah I read it more than I say so the first page is always first a table that gives the rubric for how scores were achieved because as we tweak things we want to be able to look at old reports and remember how we got those numbers and then after that we get a summary of any anomalies for the file so did it have any weird flags that you don't normally see or strange initial permissions or what have you uh... how did it do for function consistency which uh... we'll we'll talk about in a little bit and then uh... the file code and data size for the main application average library total libraries and then all of that together uh... because you know the I mean Google Chrome the main application is just six hundred and one bytes of code it's a stuff all the action is happening in the libraries and when you look at the libraries it links to directly and then the libraries they link to and so on down the rabbit hole eventually chrome is using a hundred and seventy six libraries so we put out that number and then the average and minimum library scores that occurred to and then the next page is that the main report obviously there's more than one page of this if there's a hundred seventy six libraries so that this is just the first page but uh... uh... the first line will be the main application and then all the libraries listed after that and uh... for it then uh... columns are whether it's thirty two sixty four bit what's going to get and then two categories features that were highlighting here are application armoring and function hygiene and so the application armoring are the you know safety features that make software better and then the uh... function hygiene is a measure of how well do the programmers know what they're doing so that's more detailed view this is a very core screen view comparing three browsers on OSX and uh... what we have here is that we're just looking at four application armoring features ASLR, non executable heap stack guards and fortified source and uh... if all the files for your browser had ASLR enabled that would be twenty five points uh... if you had all all if all files had all four features you get a hundred points which no one did uh... but google chrome comes out ahead because they had pretty consistent application of ASLR and non executable heap uh... safari did not quite so well they had all four of those things present in some cases but just not consistently and then firefox was missing ASLR entirely which was a shock to us and then if you looked at the uh... bugzilla comments it was a shock to the development team it was quite interesting because kim zedder did a very nice article uh... on us in uh... i can't remember the name of the uh... the journal uh... and some folks on firefox dev team popped up and said that doesn't make sense we've had ASLR since two thousand mumble mumble and it was enjoyable in a kind of you know uh... awkward way to watch some of the other developers point out all the situations where they intentionally disabled it and guess kind of what we might have measured and somebody said well wait a second they've got safaris and that doesn't live on other things this must be os x we have ASLR and os x don't we and then somebody goes and looks and goes no uh... not at all and they dig it out and it's like sure enough for backwards compatibility you know uh... from ten six os x you know they had to drop it out now the good news is in september they're going to rectify that they won't be able to fix i don't i don't it doesn't look like they're going to try and fix the non-excutible heap the way chrome google chrome did which is a bummer uh... so but the good news is a fixes coming based upon this data uh... we'll see how well it goes across the board the bad news is until then i don't know what the recommendation is maybe use chrome so that was a view of uh... just a few of the application armoring uh... callouts so let's dive down into just one specific one uh... since the data on the back end that we're using for all this is actually pretty rich and this is the fortify source and i give you a little for those who aren't familiar with this there are seventy two functions i think it's seventy two presently uh... that the compiler uh... has replacement safer versions for and if you say fortify source on Linux or os x it'll go through and see if you have any of the functions in your code there do a bunch of heuristics on each one to see if it can guess what you had really intended and then replace that risky function with one that's more strongly bounded so it'll kind of like do some extra safety for you and put that into the resulting binary so we looked at uh... the linux applications and across so uh... across all of those that were about two million opportunities for a risky function to be uh... enforced or improved with fortify source and the way you read this graph is that each one of these dots is a file and along along the x-axis is what percentage of the opportunities of those risky functions it found it was able to replace in the file and on the y-axis is the number of those risky functions per file so you know a little bit off uh... is there's a file with over ten thousand risky functions that was only able to replace about seven percent of those uh... for those who are linux hackers uh... system d kind of extremely important uh... uh... binary uh... for linux off the scale how many forty three thousand opportunities mostly p-reads and it was able to successfully reinforce those less than seven tenths of one percent of the time and the interesting part here is that the developer is trying to do the right thing i mean the right thing would not to be used as risky functions in the first place but sometimes you're kind of you're kind of host and have to so they told it fortify source but then they don't know the efficacy of the coverage that it got and the consumer needs to know that as well because the fact that two people have anti-lock breaks is a lot different than the fact that one of them will stop within three hundred yards and one of them will stop within ten yards and depending on your environment you need to be able to know each this gave us a nice view of source code fortification across linux how does it look across osx first a couple other things for this chart the uh... a third of the files end up being in the ninety five to a hundred percent range and then the rest were really evenly distributed from zero to ninety five percent and interesting thing to note is that some of these very well fortified close to a hundred percent files are up at like twenty five thousand functions so it it's not that they've only had five functions and that's why i got a hundred percent okay now moving on but it also means that two-thirds of the time it's not able to protect you uh... when it's putting on there so this is osx and you can see it's weighted a little bit towards the other side in fact there weren't any functions uh... any binaries that had a significant number of functions that were uh... completely fortified the ninety five to a hundred percent one of the largest one had like a hundred and twenty one uh... functions that that were replaced uh... and this led us to believe that source code fortification is lagging behind in osx as to what it is on linux so we dove into the data and decided to slice and dice this one more way and look at it per function so remember there are seventy two functions in the way you read this chart is the far left one says there are about thirty seven functions out of those seventy two that were never zero to five percent of the time where they ever able to be fortified and replaced far right you see that there were fifteen functions that almost all the time uh... were able to be replaced successfully with the safer versions far right makes up essentially your stir copies your unbounded was there that says i know what you're trying to do the far left all of your pointer arithmetic on mem copies largely there were twenty eight functions out of their that never got touched they're essentially academic uh... with our hypothesis that it's a little uh... more mature on linux than it is on osx what it is that's look like that's not that good so there were you know fifty plus uh... out of the seventy two that are you know only ever occasionally and then a whole slew of them well fifty two or zero percent out of the seventy two this is a way of using the data rather than looking at a per binary of looking at an entire environment and figuring out its maturity level as a consumer as to the different safety mechanisms that are in place where there were fifteen functions that were ninety five to a hundred percent fortified in linux here there's only one in the ninety to ninety five and then nothing it was a hundred percent fortified so you can see that it's a much less mature feature on osx okay and uh... a lot of the things we're looking at are not new things to be looking for their things that attackers always look at when they're trying to find a weak target and figure out how to target their efforts usually they look until they find something that looks juicy and then they start working on that uh... it's as far as we know nobody's ever applied these metrics across the entire ecosystem to see on a broad scale what it all looks like so and i wanted to add we've shown you at the beginning an example of looking at individual files the last couple showed you looking at an entire operating system in kind of the maturity level and now it's there is going to walk you through as if you step back and look at the institutions that made the code and what you can infer about them that maybe they don't even know about themselves so what we're going to look at is google chrome and microsoft excel on osx and what you can learn about the osx development uh... for those two organizations and what you can see uh... as i'm gonna explain is that each organization has something they do very well and each organization has a blind spot so first off uh... the application armoring features what they really tell us about is the development and build environments for the the software developers and that's an area where google chrome does very well they actually had the only sixty four bit files on osx that had the non-executable heap flag enabled uh... out of the almost forty thousand binaries that we looked at so kudos to them they did a great job there they figured out how to manually go in and hack the binaries because they knew the operating system in the a b i would allow you to explicitly call that out even though the compiler chains don't give you the option to do that and they said osx tries to make the heap non-executable by default but it's a system-wide control and we can explicitly say no it should never be executable for our app please and a lot of you might be surprised that for at least a period of time it was executable on all of your distributions some of them that might still be so uh... they went above and beyond to make sure that they had the best development and build environment on the other hand microsoft excel was still a thirty two bit binary and they didn't have some of the application armoring features that are default for thirty two bit binaries on osx uh... if you're doing a modern build chain and so what we can infer from this is that either they were using a really old build system for osx because it's not their wheelhouse their main focus is of course windows development or that they were using a modern build environment and specifically disabled it so we're giving them the benefit of the doubt and assuming that it's just a really old compiler um... but then uh... on the other hand uh... let's look at function hygiene so google chrome has more use of good functions than excel does but all those binaries that had good functions also had the risky and bad versions of the same functions which sort of defeats the purpose uh... you know that if you're gonna use the stirrel copy always use stirrel copy don't also have stir copies right next to it uh... on the other hand microsoft excel when they use the good function they just use the good function they uh... they don't have the stir copies in there anymore because they got beat up for them too many times and they don't let their developers use them now and uh... so this is an area where microsoft has learned the hard lessons and is really doing the right thing but uh... at google it's a little bit of the wild west it's some developers know about the good functions and so they'll catch it and code review or use the right function from the beginning but other ones don't and so it's sort of luck of the draw and they have i mean they have really good programmers at google this isn't a knock on them you know not all of them are security uh... as their main focus uh... and it's a little bit of luck of the draws to who you get as the reviewer as well but microsoft in their pre-compiler has a dirty words list essentially and you've seen this on their deprecated so you know these functions are too risky for security we will not let you ship you know if you're using that out of microsoft so it flags them refuses to go to a gold build and they have to go back and actually replace it so it's an institutionally enforced area which is impressive and uh... this is also just to plug a different pet peeve of mine variance in what you see from security knowledge of developers is also just the fault of computer science curriculum that it's not something that's included for general computer science uh... but anyways moving on back on talk back on topic uh... this is the the sort of curated report that we're doing automatically off of the data sets and it's only highlighting those sub-functions out of app armoring and uh... and the function hygiene but what we're doing is as the non-profit we're opening up and licensing the data sources so other folks can cut it and slice it and dice it any way they want for their own analysis but we're modeling off of consumer reports to figure out something that's in the middle that will give consumers the ability to kind of look at a high level of what's going on and if we pop back to the big picture and you think back to the histograms of linux os x in the in the the early one on windows there was something because we always we always want to know uh... you know with the attacker hat on what's in the far left what's making up all that low hanging fruit and we had installed the same package on all three of them and there was a package that had about six hundred binaries uh... if you are any startup in silicon valley working with big data you probably rely heavily on this uh... as do a lot of larger organizations and it really surprised us because uh... this would not have been picked up with source code analysis which is also another reason other than we won't don't want to be under n d a is because we want to be able to give you the output why we don't look at source the sources the developers intent but the binaries is the actual ground truth so what is that nest the story of something called anaconda we were looking at address based layout randomization uh... and measuring the efficacy uh... in some of the linux areas and this is a view of the number of dynamically of dynamic uh... dynamic symbols that are fixed on the x-axis and the number of function pointers that are fixed on the y-axis you want everything to be zero zero that means you've got all the a s l r that the colonel can do as much as it can to try and protect you against particular types of attacks anything moving up to the top right is getting worse and worse and worse and worse so we were like what the heck are all of these and we looked at it and we saw that who a bunch of moral from the same package what's going on and that's where we decided to look into this particular package uh... it's a DARPA funded package uh... which is a little embarrassing i mean i had a lot of fun at DARPA and i'm a huge fan i'm a booster you saw cyber grand challenge that was that that was history actually uh... that's going on and uh... they'll be the first ones to say look this is sometimes it's about rapid prototyping but as a consumer i need to know where i'm accepting more risk than i ever expected it's a roll-up it's a whole bunch of open source software for r and python with all of your numpy and pandas and your side plot and then it's got you know open ssl libraries and xml and libcurl all pre-compiled and packaged together it is super convenient i mean it's like backtrack or cally for linux because it's a royal pain in the butt to try and put that stuff together and get it working on your own somebody else does it yay and i rolled it out the problem is on all of our systems i mean and i'm not the only one that's rolled it out here's the customer list from their uh... it's major fortune ten companies you know everything from bank of america seamans every to do d and what have you uh... how large is the footprint when you install it and why is it the bottom score on all of these operating systems for six hundred plus binaries because we had these binaries from other kids other things used open ssl and had the binaries there and they weren't scoring as low so what was up with that and as we looked across them they had the old version of the uh... segment and section layout for linux which you know implied that was weird because you can overwrite in strange ways they were missing things like basic stack guards uh... even on osx almost everybody that's been default for the comp for the compiler settings for eons and on windows uh... no high entropy uh... uh... address based layout randomization no safe structured exception handlers yet now it refixed again microsoft thinks it's she rather than s e h that's fine so we finally were able to get the ground truth as to why this was happening because on linux uh... they put in the uh... dot info section it spits in the compiler version in the build environment and their most recent uh... batch of binaries is being recompiled from all the source code on what two thousand eight gcc running on a two thousand and five installation of linux those were different defaults back then and it missed all of those safety features all of those anti-lock brakes airbag seat belts side you know side impact et cetera i hadn't realized that they had essentially done the same trick unintentionally that the capture the flag folks do to make very easy targets for exploitation uh... they missed almost a decade worth of improvements and you saw how well that worked for google on the compiler tool chains by staying up to date on the latest and greatest uh... so that's a kind of interesting little side story we figured we'd share about why binary looking at binaries rather than source actually sometimes isn't the uh... deficiency that many people think it is just to wrap things up there's been various misconceptions about what we're doing or not so this is me making sure that we've uh... got everyone on the same page as us these are preliminary results the uh... detailed data releases are planned for end of this year early next year uh... the goal of this is just to familiarize everyone with what we're doing uh... and as we've stated we do binary on the analysis partially because we don't want to be beholden to vendors or have to have NDA signed or anything like that but also because in a lot of ways source code is the theory whereas binaries are the ground truth and we want to see what the consumer gets uh... this is not a pass fail you get a gold star sort of thing it's meant to be quantified and comparable between different products and uh... we look at overall classes of vulnerabilities and trends rather than specific instances so we're not going to be like you know giving you oh there's an exploit on this line of whatever you know but we will tell you that this will be exploitable at a ninety nine point nine percent percentage and that the adversary still has to do all the heavy work in order to do it so finally an asymmetric win for the consumer and the defender this talk focused just on the static analysis because we had to pick a silo to focus on but uh... we the dynamic analysis results are planned to be released next year uh... and we are also going to be looking at uh... internet of things devices again next year uh... we're a 501c3 which means we're a non-profit charitable organization uh... the uh... with non-exclusive rights to use the IP we are going to be offering uh... the ability to license our data or to uh... uh... other partnerships with corporations just not with the corporations that make the software this was actually a really important uh... choice that we made because we needed the incentive structures to be aligned that would enable us may force us to be able to give out the data uh... to everybody and there are a couple of other efforts here i know folks are familiar with the more traditional underwriters laboratory you might not be aware and i hope that they do well i'm not sure i'm i'm i'm uh... a booster of their approach they're now a for-profit organization and it's a for-profit based on public safety and to me those are fundamentally misaligned incentive structures uh... and then of course what they're doing is a bit more of the did you do all of the forty a l common criteria which we've already demonstrated really are just measurements of your processes and not not what the safety is in the end product we're partnering with uh... consumer reports for some things right now yeah some reports is involved with the consumer reports sort of business model is what we're think of that for what we're going to be doing yeah if you're wondering how are they going to make money or how are they gonna do anything else look at consumer reports and we're trying to do it the exact same way that they do we won't accept things from vendors we will go out and buy the software and analyze it ourselves or get it the same way you do off of the download sites and finally one thing a lot of people ask us about is we're not looking at software configuration vendor history how quickly they put out patches interpreted code corporate policies privacy any of those things it's not that we think they're unimportant it's just that that data is already out there other people are looking at it and we're trying to focus on the blind spot the thing that no one has data on yet and uh... you know we've uh... compared to what we're doing to nutritional facts so you know having the nutritional facts on the thing is great but you still sometimes need a doctor a dietitian so the security specialist that you know consultants they can tell you what diet you should be on for software and they can also bring in all these other factors and you know help uh... put it into a whole picture for you yeah we aren't telling you what to buy or what not to buy we don't want to do that we want to tell you what's inside of it so that you can make your own informed decision the same way that i enjoy my candy bar i want to eat my candy bar uh... but i do want to know you know what it's made of and i actually enjoy it more when i know how many calories and sugar and i'm cheating yeah so i'm flying through this this is what we're gonna see at the end of the year some of those curated data releases coming online we're gonna release in detail on the site our static measurement methodology so other folks can recreate and do it we are not releasing the actual source code because that's uh... a gaming issue uh... but we're releasing enough data so that you can recreate it on your own twenty seventeen the large-scale data analytics dumps and everything else uh... internet of things and then the large-scale fuzzing results of the mathematic models are going to be released more publicly and describe publicly in twenty seventeen two slides i gotta get to the these are our thanks darpey afl community capstone uh... ford foundation is funding us through consumer reports we've got some money from darpa i'm huge fans they took the risk on us and this is the mandatory slide at the end uh... because we have d o d funding inside of it but it's basic research meaning that we can publish without having to go ask permission uh... because this information just needs to be out there this does not necessarily represent their opinions or anything else from the air force darpa said me too make sure me too and then the white house says why do you keep name dropping us all the time would you please knock it off uh... but so be it thank you very much