 Yeah started everybody's like, okay, this is cool, but not that cool Apart from John who's totally cool Okay, should we start? Let's start. All right, you're up. So I'm gonna boot this off because I am I sort of started shellfish, but I sort of reaping the The benefit of it without really doing anything these guys are actually the brains behind it and The guys that stayed up all night doing all the work. I'm just looking at them thinking. Oh my god. I remember when I did that 25 years ago. Giovanni did a lot of high-level planning and sushi delivery Exactly, that's my role feed them. They will poop software. Okay Cyber Grand Challenge code that's really actually true. That's actually true. So I'm gonna be very sure in this Shellfish was born out of the sack lab, which is the security group. I you see Santa Barbara every time you say you see people say University of California, that's not right. That's Berkeley. You see Santa Monica. That does not exist It's you see Santa Barbara. So get it right sack lab is the group. That's where we come from and the group is led currently by me and my Colle Christopher Kruegel We look very professional here like professors, but we're actually Hackers behind weird handles like everybody else. I I never got the handle thing But I I needed one and so if you look about Zanardi on The on the internet is somebody with a gigantic nose and a ponytail which I once had Giovanni Would you say Chris is your life partner? I think Chris me Christopher is my academic wife. So I I have to take care of all these needs and is I wish he would be here. He would be very is super proud of everybody But this is our university not bad. And that's why Shellfish is here. Our lab is exactly there where the arrow points. We're all right on the beach We have a private beach and that's why our tagline is hex on the beach We're lucky might be back here. Is it back here? Yes, it is. It is. All right So how do you start it? It started in 2004? I know it's incredibly Such a long time ago. It's me, but then I had a bunch of grad students including Chris and we evolved into a Community and in 2005 we actually won DevCon CTF never won since then that was the good old days And it's all it's awesome because they say the older you get the more awesome you wear So I'm milking it for whatever I can But we grew up, you know, and then suddenly void Chris moved to Vienna became a professor there Recruited some more people that became more people that came back to Santa Barbara because it's awesome became more people more students more students even more students and What happened is that some people went to Boston? So we have a substantial presence in Boston and we evolved as a group more and more in the years Until at a certain point all our graduate graduate students actually became Professors as well. And so a lot of you, you know, you see people Became professor all around the world in London at Arizona State University Eurocom in France and right now Shellfish is a very big group of all academic people all around the world doing interesting stuff. So Right now our group is pretty much this we're very inclusive where you know, we foster research and that's what we care about and with this I'll give my Presentation a tawn to young Thank You Giovanni. So before we go on with the cyber grand challenge itself I'd like to give a shout out To all the other shellfish Shows in the audience. So raise your hand if you're a shellfish. Oh, yeah, right there Yeah, shellfish is Bigger than just the CGC team the CGC team is a straight subset But we have a lot of people that were cheering us from the sidelines even on the team So let's talk about the cyber grand challenge Um DARPA has a history of grand challenges, right? You guys are probably familiar with the self-driving car grand challenge and the robotics grand challenge because they got a lot of press similar to the cyber grand challenge just now and The idea behind these is DARPA finds this fledgling technology self-driving cars and they fund It with a lot of money, right? So their prizes million-dollar prizes for self-driving cars and This motivated a lot of people to put a lot of research into it at the time people were of course saying Because the time was 2006 when we didn't even have smartphones and people were saying do you really think that? Someday you'll be sitting inside a computer and it'll be driving you around. That's absurd and now we have people Driving themselves to the hospital while they're having a heart attack in their Tesla And so, you know this technology push really pays off and It's probably gonna be the same with robotics DARPA did the robotics I regret and challenge and probably in 10 years We're all gonna be dead and it's also gonna be the same with program. So the cyber grand challenge really pushed the frontier of automatic program analysis exploitation and defense Right now. It's in its infancy. I think You'll see how the CRS is did at DEF CON CTF But maybe they won't be the best humans But that's the beginning the chess systems didn't be the best humans and the self-driving cars aren't gonna be the best humans and Races right now, but eventually they will and eventually Mechanical fish will kill us all or hack us all while the actual robots kill us So as a cyber grand challenge Let's talk about shellfish is involved in the cyber grand challenge as your money said shellfish is a bunch of Academics and hackers right so we're kind of academics so at one point we decided to shift our research Interests in at UCSB closer to binary analysis, right? We started looking into doing automated binary analysis and All of the things along with that automatic vulnerability discovery and so forth Completely independent of the cyber grand challenge. We started doing this sometime in 2013 and in late 2013 DARPA announces the cyber grand challenge, right? So I have an email somewhere in my history saying hey guys check this out This is this cool thing Maybe you should participate because we're working on a lot of the same stuff and everyone said yeah, let's do it Let's go for it. I said great and then probably forgot about it for like a year right so the deadline for registration was in late 2014 I sent in the Kind of application literally 15 seconds before the deadline because that's that's how we roll and They said great. You're in congratulations. Let's you know See what you got the first court event is coming up in like four months And so we're like, okay cool. Oh, no like on the graph is like in one month, right? So we said cool. Let's let's build a CRS We're gonna we're gonna rock the scored event the first kind of practice round that that was the term DARPA used for I'm scored events. So the first practice round You're gonna we're gonna do super awesome. We were gonna kill it and we totally forgot about it the morning of the practice round I wake up and I'm like shit. There's a practice round for the Cgc stuff tonight and so we started working on our CRS, right? So the first commit to the CRS is two hours Maybe three hours. Let's say before the practice round begins, right? So we start writing our CRS practice round begins. We play the practice round with some Janky ass CRS that that kind of half works Cool, so then we like all right well now here we started we're gonna get it all Super put together before the second practice round second practice round rolls around and now we remember about it Maybe three days before right so the second commit to the CRS happens three days before the second practice round We build it up build it up build it up play in the second round So okay, cool now we have this kind of cyber reasoning system That's kind of ready to play in the CQE if we keep working on it solidly until the qualifiers and then of course we forget about it for another couple months and Then two and a half weeks before the qualifiers. We remember. Hey wait a second The qualifiers are coming up. So then we start working like crazy and not sleeping three weeks of complete insanity until the Cyber grand challenge qualifiers and we have a cyber reasoning system that we can field for the cyber grand challenge qualifiers And we qualify with three weeks of absolute insanity And so then we'll figured cool now a we're super rich because the qualifiers came with $750,000 of prize money and be we can now spend a year working solidly, right solidly with test cases test cases code freezes milestones milestones lots of milestones and absolutely, you know continuous Integration and and and you know test rounds and everything for an entire freaking year At child development. That's that's the key word here. None of that happened so for nine months we use our money to fly around the world giving conference talks and like saying how how cool we are and how you know Fish is a Chinese martial arts expert or wait that was that was Kevin Kevin's a Chinese martial arts expert And you know Antonio is mysterious and all this shit But it really we should have been doing is working on the CRS, right and three months before the finals three months ago. We realized this and we're like Crap We should really write a CRS for real actually, right like and we should take what we had in quads and actually like You know extended so it can win finals So three months ago. We started working like crazy. We stopped sleeping, right? I have a fiancee and I haven't seen her in three months. Basically that that's you know the insanity To the founding agency they're listening we're a lot more responsible than it looks yeah This is our hacker persona, right? We also have an academic persona where of course we have CI of course come on who doesn't have CI and code freezes, right? And we we finish all our papers two weeks before they're due so that our professors can go over them and Absolutely, this is the hacker shall see persona All right, anyways, so we went crazy for three months We got the final commit to the CRS 30 minutes before the air gap was established 30 minutes All right, and it was a commit in one of the core components. So shit could go wrong there's a slide for that and All right, I'm killing us so We did it we play the CAGC we got third and this is the team that we already introduced We're from all around the world Italy Germany the US India there was a guy qualifying with us who's hopefully sitting in the audience from Senegal officials from China We're from all all over the place And we are very rich because we got Two seven hundred fifty thousand dollar prizes now so That's kind of a brief intro to our involvement in the CDC I'll pass it off to Jacopo to introduce the CGC as a platform and what it means right so Very very very true and very effective introduction to the shellfish hacker very distinct from academia very distinct from the shellfish Academy All right, so just very briefly So what does it mean to actually score well in the CDC? You have to you're gonna go blind with binaries that you have never seen before You have to analyze them in whatever way you want. There's no limitation on how you do it You have to own them either by a crash or by leaking a secret and you also have to patch them So that the other guys cannot do the same to you and this is a classic Classic CTF Structure that has some modifications to the Cree in the decree operating system to make it more model more easier to model and Easier to handle for a program. Okay, so one of the simplifications is that so the architecture is Intel x86 All opcodes are legal which can lead to interesting situations that we will see in a bit um See schools are simplified much easier to model pretty much read and write select Allocate the allocate like malloc and free Random and obviously exit a lot easier to model for a program and the actual binaries are actually a lot a lot more realistic a Very real. They're not complete fake binaries So as a side note and the DEF CON CTF just finished and the DEF CON CTF was also played on the same platform So just an example of how real and complex these binaries could be one of the challenges in the DEF CON CTF was a Power PC interpreter and jitterer, which was awful. And so there's a lot of room for complexity in these programs And on the actual pony inside I Don't know if some of you guys want to barge in but basically What it means is that? There is no there is no state every program runs once there is no state It runs you either own it or it's gonna do its thing. There's no there's no state There's no file system to modify. This is a lot easier to to model for the for the qualifications And only for the qualification. It was just enough to crash the program Sec fault illegal instruction. You will get the points you have owned the binaries For the finals things a lot more nuanced and the actual exploitation as we will see is a lot is a lot more complicated And it's a very interesting application of how to use symbolic execution and static analysis But as a as a basic idea the two ways you do is either via a Control crash in which you can show that you can not only crash the program in some place But you can actually crash the program at a place that the API that that was gonna tell you please crash the program in this place And set this register to this value if you can do that you verify that you have actually control of the program or alternative that you can leak a secret flag from memory and On the patch inside just a brief note on how you analyze how this API is designed So that it does not become too easy like for instance we can submit patches to the binary Okay, so what is preventing us from just submitting a binary just exits These programs this program obviously never crashes, but also does not do anything useful So the way this is prevented is that is that there are functionality checks if you if the program does not maintain its benign function if the program is math calculator it needs to still be able to do all the math operation that it can do normally and Similarly, there is no signal handling So no way to just hide away all the sac folds if you sac fold you are crashing and finally How would prevent us from putting in an interpreter that runs everything? So checks before every possible instruction am I gonna crash am I gonna crash obviously you will never crash and The way this is prevented by that project you can actually do it. You can do it if you want But you're gonna pay a performance price You're gonna lose points for performance. This is believe me not as easy as it sound Understanding exactly how your patch is performing is definitely not an easy task many of us looking to eat I look into it a bit. I'm trying to look into it in a bit. It's definitely pretty hard And then we gave up testing performance. We just say this is our patch deal with it Yes, yes, that's very true and I know informally We know other teams also had trouble But I think none more than out of end knows very well how much of a pain How much of a big pain it can be to actually test the performance and the functionality of binary so big props to Arvin for actually pushing through this task and actually making it and This actually helped us a lot during our own internal testing even if it did not go into the live part and Are we now hand over all right, so the CQE for the qualifying event was not the full Now it was not the full cyber grand challenge It was you needed to patch binaries and needed to crash binaries. You didn't need to exploit anything You just needed to crash it the final event. You need to patch binaries crash binaries who find where vulnerabilities are and then exploit those vulnerabilities and On top of that It wasn't just a simple game or a simple program Challenge where you got a binary you crashed it. It was a game So you have to have a game theoretic aspect that Played against other actual competitors right similar to a human CTF, but all with computers So the competition was actually divided into 96 rounds and that wasn't predetermined it was you know However, many rounds they got through in a day there was a Minimum time per round and that ended up being 96 and there was a bunch of Challenged binaries as they term as DARPA terms them and which were provided to the teams to hack and for each score for each round the team would have a separate round score that When aggregated would be their total score for the game the score was calculated based on a multiplication of the team's availability, which means how much do they fuck up the binary and How fast the binary still was right how much overhead the patches had which is something Yakbo alluded to the security score Which is how exploitable were the binary still or were they still exploitable and the valuation score Which means did we find did the team find an exploit for this binary? so it was very easy to screw yourself in this context because They're all multipliers if you completely break the binary even if you have perfect Offense if you even if you find all of the exploits for this binary then you still get zero points because you broke the binary in developing for this competition we Ran into a lot of kind of Organizational things as I alluded to earlier we started super late. So for example up until Depressingly short time ago. This was our database All right After all these are research group run by an Italian Again, this is our hacker persona So we actually had to do a join on this database at one point when we got the real database up You're joining between the paper database and the actual database This is relevant because it's about our performance scores This is the best is the database of our performance scores. We're trying to analyze That was relevant to the previous slide specifically this database contains the feedback from some Practice sessions for the final event. So this is what diaper called sparring partner sessions We wrote them down and then we had to join them with the real database to get the actual information that we needed To tune our patches We also tried to go into code freeze several times So at 4 1 p.m. On some God-forsaken day We froze a component of our CRS called farnsworth and Very shortly thereafter. This is the commit log Right, so the code freeze didn't work very well There are commits such as this gem here so that that that's Francesca here that you know Beautiful beautiful good. This commit was okay. Actually, he just has very high standards. Actually, it was probably crap And then of course this is a Long time into our code freeze 1215 hours before our nose were shut down a couple days ago. We were still changing very Core components of the system. That's me upside down. I was at this point no longer sane so our CRS consisted of a lot of components, right? We had a We had a central database that we called farnsworth for some reason Which stored all of the data that we got from the Cybergan challenge API through a component That you'll talk about later. It stored network Captures it made It stored the scheduling decisions of what jobs to run and then it stored the result of those jobs So now we're gonna go One by one into all of these components probably pretty quickly We have 15 minutes left and we'll start with the Organization or the core organization components and I'll hand it over to Francesco and Kevin so obviously coordination is very important if you're running a cluster of 64 nodes and of course Since we needed to do that we since lay came up with like using one database to simply store all the ground truth that we have As a bunch of you probably know this is from future Amar So we just went with essentially funds worth because well good news everyone And it's the only component that we actually tested fairly well at about 69 percent test coverage I think the rest probably dumps around it like 1% Zero oh perfect even better And who needs testing anyways right anger has at least 15% code coverage. I think Francesco probably disagrees, but yeah, who cares Then on top of that we since they had to my star which the Germans and you know So since they just master Which looks at scheduling jobs and deciding what jobs we want to run what kind of part of our pipeline We want to run exploits patching if we want to run AFL these kind of things a schedule them based on priority End of this obviously Sorry, the last component that we actually changed with the last commit being I guess two hours and 18 minutes before The actual deadline. So yeah, this was at 1242 and the same deadline to actually the no shutdown was that 3 p.m But we made a commit. I think we rolled that commit back 30 minutes before the dead. Yeah There were a bunch of commits at like 2 p.m But we actually reverted them and cleaned up the history just to make sure that they're actually not there Because they caused a bunch of failures on our side Anyways, we would also like to give a big shout out to essentially the open source components that we should rely on One of them is Python the Microsoft research the 3 compiler all of our things runs into inside of Docker containers Which are running Ubuntu with Pi Pi? We're also using Kubernetes QAMU P we vax pos grass Obviously anger, which I'm sure a bunch of people are going to talk about now and I think that's probably young Possibly souls Andrew John I guess and pizza. Yeah, go ahead. I Agree we everything he said Angers the open source binary project binary analysis project that we have in the sec lab It's really really cool. It's been open source for like a year now. We released a defcon last year, right? Yeah It does everything. It's cool No time. It's very cool. That's our logo. It's creative commons We in order to do the actual exploitation analysis pipeline We split it up into a whole bunch of components and rearrange them into these Weirder things like we've used concoct execution in order to do some basic analysis of what can go where there's automatic Exploitation and patching which will all be talked about. I think they've all got their sections in this presentation There's crashes. I think you can slow down a little Fine Who wants to? Sorry So who's crap who wants to talk about crashing crashing? Guys, we haven't been sleeping for three days. So I'm sorry if your friends do all the funding agency. We're not doing drugs or alcohol Looks like it. I'm not even 21. All right crashes souls Nick talk about it You see how prepared we were for this huge defcon talk. Hello So crashing So our exploitation strategy is we find crashes and we turn this into exploits Pretty incredible So actually like a lot of teams the thing we do the most is fuzzing and this is what generates a lot lots of test cases Lots of crashes the majority of her crashes, but not entirely all the goodies we find so We use AFL as our core component Fuzzing we I'm more explain how AFL works like these slides do I suppose and Essentially begin by generating lots of inputs which attempt to explore different parts of the program the inputs are basically random Some of them are more or less educated guesses and how well these inputs do when exploring the program is tracked by instrumentation which is Compiled into the binary or which is provided by An emulator like QMM So See did I go over all these? So if it was a great job of doing this we've modified it slightly to work better on TGC binaries So we have a couple of hacks which I think will be open sourcing which make it perfect for CGC or at least a lot better Okay The uncrasher I don't think that's actually actually exists But I Don't think there's a non-crasher man the points of flag and all this shit. Yes. So what is like karaoke slides? right I already mentioned this right AFL. It's great. This is how fuzzing works Random stuff gets put into the binary Yep, same input all over again eventually comes up with a random thing that works This is a much harder for a fuzzer. We have to generate a very specific input fuzzing will have no luck with this of Keeps continues to lose Makes absolutely no progress if you guys can't feel like you can't keep up with my pizza. I feel like that very frequently Okay, so anger on the other hand is a symbolic execution engine It's slower and more heavy weight But it's great at finding very specific cases like the one we just described and the way this works is by Generating these states following different paths as you can see here in the control flow graph. We have different states which are being followed Eventually, there is a state which will satisfy the you win expression and we talked to z3 We ask it to generate an input which gives us the state and boom So what we tried to do is combine both AFL and anger And we this is called Driller Driller begins by fuzzing gets basic code coverage of the program the way you would expect AFL To and get maybe gets a couple test cases in this example x and y we get the cheap coverage Next slide Then it okay again, then we take those test cases and we trace all of them with anger So we make the input completely concrete almost we actually make it keep it symbolic But we constrain it to be this concrete input the AFL generated And we see at any point in the program if we could have taken a different path Which AFL failed to take if we could have taken that path we talked to z3 or anger more specifically and we say give Me an input which satisfies its new path in this case we get the CGC magic and The new test case is generated and now we continue The loop and we feed this back in the AFL which continues to mutate that further and buzz and it goes on and on until we continue to get work code coverage And then we play video games all right So this next part is the auto exploitation how we go from a crash Which is generated by AFL and Driller to actually an exploit for the CGC which scores as a flag All right, so in this example, I think there's a buffer so there's a buffer overflow inside the Heat inside this malloc object here, and when you overflow this buffer you actually control the function pointer and so We're inputting inputting inputting symbolic bytes and eventually we control the buffer the symbolic address We're gonna call into an address we control and so to exploit this we use anger We check we trace the input using anger and check that first the IP is some bog the PC here We say is the state does the state have a symbolic PC at that point? We know it's probably exploitable we can control where we're gonna jump to and so let's set the buffer to contain our shell code We ask z3 to give us input where the buffer point contains shellcode and then we jump to the buffer And that'll give us an exploit And to do this we synthesize the input in an anger. That's just called state dot posix dot dumb zero So in the CGC this is discovered by taking a crashing input and tracing that with anger So keeping all the input that AFL created symbolic and then following the path that took until we have our crashing symbolic state so Keep in mind. This is very simplified We have a bunch more techniques that handle the harder cases and that can take a Not so good crash and turn into a better crash And you can find those all when we do our open source release and when we release more details and papers later And in the open source release This component is called Rex if you're interested in auto exploitation check that out All right, so then the steps again We create a vulnerable symbolic state where we control the PC we add the constraints to set the shellcode and to set the The program counter to point to the shellcode and then we synthesize the input and that creates our exploit Okay, so this This component will be talking about augmentation of flag leaks So if you didn't know there are two types of exploits you can generate in the CGC the type one is sort of classic memory Corruption show that you can control the program counter and show that you can also control a general purpose register However, there's another type called the type to very creative which Shows that you can leak arbitrary memory from the program. So in the CGC, there's actually a sensitive sensitive data that's a map at a special address or in every single binary and if you can leak content from this page in memory you score points Like Heartbleed for example with there was a hardly challenge in this game, which Where the premise was leaking this data from this flag page the sensitive data So the way we do this in a fast way is we actually use the unicorn engine Which anger integrates to make the entire input completely? Concrete the only thing which is symbolic during the flag detection is the fly page itself so We trace the entire program and execute very fast because everything is being concretely emulated by QMM with unicorn and We can detect in transmit because we hook it with anger When the fly page is actually being emitted and then we can see exactly which transformations are done in this fly page You can tell if it's been exhored or if some complicated constraints have been applied for example This actually solved the DEF CON CTF challenge which Okay, it's enough time to talk about that but we saw the DEF CON CTF challenge this way so we'll talk about it a little more later so So of course one of the challenge was to patch this binary So we had a component called patch rex that was going from patch From a patch binary to patch binary So the general idea is we have patching techniques for this and let's add let's encrypt the return address and these patching techniques generate patches such as let's add this code here Let's add this data there and these patches were injected within the binary We had three different ways the first one was slower, but more reliable and the last one was faster, but Less a little bit less reliable and fish is probably gonna talk about the reassembly And so we had adversarial patches that were designed not to make our Binary our patch binary analysable by others and this is a one of them that is pretty cool and This is a detect QEMU detection this if you run this code in QEMU QEMU i3d6 It'll hang forever Well, not really forever as long as it takes to int to increment a 64 bit int to the 64 times That's basically forever And we actually owned the cyber grand challenge Visualization infrastructure with this they're apparently using QEMU for instruction tracing and so at one point during the CGC We noticed that their instruction tracing had just stopped and it stopped right on this code Which was designed to detect QEMU and crash will not crash, but hang for a zero day take a picture We have a lot of open source bug fixes to contribute starting now So there were other so all sorts of adversarial patches so to speak for instance our binary was starting by transmitting the flag out but They were transmitting to STDR so to STDR so that This could probably confuse an analysis system that could misidentify this as a as a type 2 Vulnerability we also have a back door that if some team was using our patch in the in their submission We could actually Exploit that and I'm not sure if the back door work during the CGC But for sure it worked during the city. Yeah, how many do you know during death corn? How many team? I know that our other teams use our back door during Can you name names? I'm sure And what three teams that fielded of our back door at the CTF? during CGC Yeah, okay, cool. So then we had also sort of genetic patches that are these are more standard academic things such as Protecting the return pointer protecting data codes and when we are going to release these Code you will see all these sort of kind of more standard techniques and then targeted patches So the general idea oh you can speak about so so targeted patches right so qualification events We just wanted to avoid crashes right because anything that crashes counts as an exploit So we had some you know, we just checked using a weird quirk of one of the syscalls Using a weird quirk of one of the syscalls we checked to see if the if memory was Readable at a certain point if it wasn't recrashed, so I would like to take Specific credit for our back one slide for our targeted patches in the final event, which were exactly nil And it worked great. So what what can I say? And one note that no functionality overhead That wasn't that and one cool thing about these that we thought we were cool Finding these weird syscall tricks to detect memory locations, but actually when we analyze Qualification binaries from other teams when they were released we found at least one other team was using exactly the same tree So you're saying they're both cool. Yeah, we're both. Yeah Okay, so we are running out of time. So Well, the only thing I want to say is anger is awesome I spent three days in writing Reassembler and another three days in ready optimizer. So it works out So what is a reassembler just real quick reassembler is a static binary? Rewriter that's basically Okay, we'll talk about it later. No Okay, I've yet we had a we had a breakdown from our I think I think Guys is Okay, it's fine the reassembler is awesome fish wrote a binary writer Where you can inject code into binaries and it'll seamlessly Reassemble the binary to include that code check it out in the open source release You go There's nothing much to say basically tried. So we that by gave us 64 Powerful servers, but how many servers? 64 64 Holy shit not 30 64. So we try to maximize this usage the usage of these nodes And yeah, we can did it with the C2 at least Not the memory, but That's it. That's it So the 64 servers we had a lot of media attention over the CGC and what we got What we got people excited about the most strangely enough is the fact that we had 64 servers all to ourselves Incredible. Anyways, so We implemented all these systems in a breakneck like three months and We pushed as hard as we could we got it all running We made commits at the last second and we played the game or rather our baby played the game She walked on her own We walked into the room and they told us hey Your guys's bot started up and it's doing a lot of this guy. Oh and we fucking lost it Because we freaking lost it because up until then We thought you know, it's gonna turn on and something will fail and and it'll all crap itself So this was incredible and then we got third place Top three is amazing for us guys I can't I can't tell you how incredible it is to have been part of this comp and we're going on It was incredible Since we played in the CTF we didn't really get much of a chance to actually look at the data However, we quickly briefly looked at it So in total there were 82 channel sense fielded at least our bot saw only 82 more have been fielded We might have actually missed them in total mechanical fish generated about 2,450 exploits We generally did a total of 1,700 exploits for 14 out of the 82 Channel sets all of them have 100% reliability and so far as a score like always leaking or essentially Crashing at a specific address. Did you check how many were like mostly reliable? I did not so this is it seems that we only got 14 out of 82 channel sets We do not know how many essentially grandma tech with tech X and Xandra got or mayhem with for all secure the rumors are that we have Top exploitation, but we didn't have the best game theory. So like always our SLA sucks our SLA is shit And Yeah, so until we'll like you back up one slit these are essentially the exploits would be actually generated some Actually, I should say the the caveats of those rumors is mayhem was only up half the game And I think they still got almost as many exploits. So yeah Yeah, and so we got two of the rematch challenges So so two of the historical challenges that DARPA introduced one of them was sequel slammer Which I think two other teams also got but don't quote me on that And then there was also crack at her which supposedly only we got right and Then in total if you look at essentially the different challenges that we had and the vulnerabilities that were in there This is the list of challenge sets that is what we got and with that from all of us. Thank you for the attention So real quick, let's talk about the next steps real quick the next steps beyond automated hacking is Machines augmenting human intelligence. So in DEF CON CTF, we hooked up our CRS mayhem as the winner They played completely autonomously We played with our CRS. So I mentioned already that the CRS actually pwned one binary Without us even realizing it it actually assisted us with five of the exploits. There were five exploits At which either after providing the crash Or after just providing interaction it created an exploit for and our CRS inserts backdoors into every binary that it patches and so you might have heard already that a lot of teams actually used our backdoor These sounds all awesome, but we didn't win even close. Yes, we almost got close to last So let's turn down the bragging that that's right. Just a tiny bit the CRS did amazing But there were some issues like for example the DEF CON organizers had to implement a separate API For the infrastructure then DARPA did right because the DARPA API had to be secret so that you know everyone was on an evil even playing field and so there were some API incompatibilities and Computers are very brittle and so these API incompatibilities Screwed us until the very last day. So the last day I feel we had a good showing up until then The CRS kept crashing the CRS kept getting invalid data. It was kind of touch-and-go So as you might have heard we're going to open source everything. We're gonna do Thank you We We're gonna do a full open source vomit because we believe in Raising the playing field for everybody so the next time a CDC runs around rolls around We expect all of you to play as well Hopefully using our stuff so We don't Have it all ready right now to push the github because we were playing the CTF We thought we'd had time, but we don't but Chris. Do you think you can do a symbolic open sourcing of engrop? All right, let's do it right on stage. I'm gonna unplug the video Kevin. So Chris isn't logging in Unless I mean just don't type your password into the wrong field I've seen that before at DEF CON. It was incredible It was someone fairly famous, too. Ah, there we go better save than sorry. I Think their password was star star star star star star star star star. I enable logging before Chow chow for is what Giovanni says. I Think that's his password though All right, so we're gonna plug it back in while we try to Desperately find the settings of the open source project. So engrop is our Rob compiler So if you are tired of writing return-oriented programming payloads by hand You can wait. Hold on. Let me explain what it is You can use engrop which uses anger to compile Rob payloads into whatever you want So you say actually just read this memory or execute the sys call and it figures out the Rob payload that it needs to generate Chris wrote it is an amazing guy and it's an amazing project and here it is being open source for the world boom the rest of the code we need to scrub free of Private keys because there are so depressingly many and other depressing Things and then we'll push it out this week Also, if you find a private key that we haven't scrub, can you please gently let us know instead of destroying our infrastructure? I will appreciate it We're hackers hackers has on the worst security in the world So and and and my password is six characters long just to give you an idea All right, Kevin. How do I get back to our? thing But I think we're done basically Thank you guys So stay in touch Hit us up on Twitter By email jump on our RC channel You can chat with us about our CRS at shellfish CRS and free node. I'm the only one there right now super exclusive or On anger at free node on anger questions. Are there any actual questions? Yeah, hi Congratulations, thank you on your work So in your driller paper you had said that the fuzzing was mostly responsible for 68 of the binaries whereas Having the symbolic execution based fuzzing only let you find Vulnerabilities in 11 more than that So why is that still the case or is the symbolic execution more effective than fuzzing now you want to talk about drill is 3.0 Sure, so one thing we've done to actually improve One thing one thing we've done in action to actually improve a driller Especially on CGC binaries is to identify functions and install sim procedures in their place So what this means is that a lot of basic block transitions Which are hard for or uninteresting for once block execution solve are more interesting We have a super seizure we can talk about it more if you want to come up here Mike Oh last question. Okay. Well, congrats guys. Thank you first second I wanted to know how compute bound you felt like where did you get enough compute power too little too much? Would you put something else in there? Backplane Ram, what do you think so at this point? We don't actually know because we haven't gotten a chance to actually look through all of the logs We had some problems in the very beginning So actually on Wednesday still to get all of our Kubernetes pot scheduled simply because Kubernetes was not catching up We kind of solved that But we at this point We don't really know what the status is in so far as the utilization of all the nodes from watching the power consumption It seemed that the way that it dropped off It seemed that it had a lot of unnecessary jobs that would be scheduled later So I think we could have used a little less even And it was still yeah, we could have probably used 32 nodes and done about the same But the more the merrier especially if you can schedule more jobs We definitely had jobs to schedule that we couldn't schedule because of delays in Kubernetes Thanks. All right. Thank you. Thank you for organizing this thing. Please give shellfish team a huge round of applause What they've accomplished is immense Thank you guys. It was a dream come true to be here Yes