 Five, four. Technology, does it work? It says recording. It says recording. Oh, I see us on Twitch and on YouTube. I can see you drinking from a blue bottle from the Twitch stream. Awesome. All right, let's tweet this bad boy out. Let's go. All right, so I have no idea what's gonna happen. Let's just jump off. So, hey everyone, welcome to CTF Radio. We have, for the first time, we're doing a lot, recording a live episode. It is Monday, October 26th, at 4 p.m. Yon's here, I'm here, and we are hopefully gonna take some live questions if we can get it. If you're watching this later, if you keep track of our Twitter feed, there'll be a link so that you can call in to the show and ask questions, which we're doing right now, and we're also streaming to Twitch and YouTube. So if anybody has any questions for us there, I suppose we'll try to answer them there, but Yon is the Twitch streaming master. He's streaming his entire course on Twitch, so he's probably much, much better at this than I am. Do you feel like you're a Twitch streaming master? I, right before this, had off-sours for Pwn College, and it took me about 15 minutes to enter my own Twitch chat, so I feel very... So the answer's no, you don't feel like an expert. Or maybe the good segue is you feel like all CTF players feel. Yes, I'm getting there. Exactly. And just to confirm, since I didn't confirm this earlier, and this is, of course, something I probably would cut, but you are using Chrome to access this website. Is this correct? Yeah, absolutely, man. Chrome all the way. Yeah, you make it seem like that's crazy, but the last episode we recorded, you did not. Exactly. Hold on. How do I... You see it? This is not as simple as it looks. So I can't see... This is something I would normally cut out, live people watching. Exactly. I can't do Twitch on my computer because it overheats. Yeah, that's a problem. So instead, I have to do Twitch on the iPad, but the iPad is very hit and miss on actually... Yes, okay, we're live. Zero viewers, perfect. Exactly how many viewers I expected us to have. Yeah. Cool, this is our first time doing this, so that's okay. So until we wait for the torrent of people that are definitely gonna be coming into our stream, I think we can start off with... So this is actually our eighth episode. So we've been doing this now for two months, which seems kind of insane. Yeah, that is crazy. Went from a tiny idea to now people actually watching this, so thanks everyone out there. I know we really appreciate it. Do you wanna say anything to the people out there, Yang? Yeah, I think it's very easy as a community to pile in weekend after weekend, not easy, but this is what the community does. Pile in weekend after weekend, and build CTFs, run them, play them, make write-ups, and then forget they ever existed. There's a lot of culture and experiences created in the CTF community that deserve to be propagated onwards into the future, to leave a legacy. So hopefully that's, you know, these last two months have started that in a nice and archivable and approachable way. Ah, I see a problem where you are streaming on my personal YouTube stream. That is nice. Okay, we'll just roll with that. That's not what I wanted, but whatever, that's cool. Do you want me to tweet out an update? No, no, no, the tweet's fine. I just also did it to YouTube and Twitch, and so the Twitch is going fine on CTF radio. But it's also on YouTube, so whatever. Cool, okay, so the first then thing we can talk about, so kind of the purpose of this live show, and maybe we'll, I don't know, do this every two months or do it on some more regular basis, depending on how it goes. One of the first things that actually came up that was incredibly amazing is we had some almost historical comments that came from the community in response to some of our first episodes. And kind of the first one I want to bring up is this comment on the CTF radio YouTube page from Bob F, and this was on our episode on what is Capture the Flag, you know, the very first episode that we came up with. And so I'll read this and then I'll kind of let you respond, Jan. So the comment is from Bob F. We, Ken Shoto, wrote that first Jeopardy board for DEF CON 14, our second year running it. Our first year we had to stick to the Ghetto Hackers sequential binaries qualifier model where there was a Linux host with each level's binary set UID in a way to give you access to the next challenge directory when you gained execution. Yeah, so this is a super fascinating part of CTF history, right? Where did Jeopardy come from? No one knew, I mean, I guess people knew obviously, but in the community, Jeopardy is just how you have most CTFs. You know, CTFD provides a Jeopardy CTF, several other frameworks as well. And you always know that there are some types of CTFs, Jeopardy versus attack defense and so forth. But the history of these has been lost in some sort of a shrouded, you know, mist of time. And this is super interesting to see a comment on our channel, right, which is super cool that already we are kind of starting to evoke these recollections, I guess. I'd always thought that Jeopardy was invented at DEF CON CTF, and it was just how the CTF was either always or just kind of emerged from there. The other thing that Bob mentioned there, the SEDUID level by level, the client. So can you explain what that is for people? Yeah, absolutely. So this is a different design of hacking challenges. A little more of a war game style, let's say, probably the true successor of this is war games like Ponyl.kr, where you just go click one by one. Nothing keeps you from skipping, but you typically rarely do. And I guess it was the original design of the DEF CON qualifiers, which is super crazy. It's this concept where you connect to a host and you're some user, user level one, for example. And user level one has access to a specific program. When you exploit that program, you exploit it to gain the privileges of user level two. And then that one has, and so on. So you go from level one to level two to level three to level four. And it has this, if I may, that has this nice gating property, right? Where it sets up kind of naturally, you can't go on to the next level until you've completed the previous level, which has been actually persisted in the original Jeopardy model of CTFs, where you originally, just like the Jeopardy board game, you have a board of Jeopardy challenges. The organizers pick one, and then the first person that solves that opens up the next one. Yes. And so on and so forth. And in the original, it was really the next one. So in every category, you could only open up the next difficulty in every category. So if you wanted, if you solved the, you know, 100 point Poneble challenge, you know, in the Jeopardy board, you could open up the 200 point Poneble challenge. Exactly. And so on. And so this set view ID race is something that you can still see on over the wire, right? So if someone says, yeah, you know, to learn Linux, go to overthewire.org and try the Bandit challenges. If you log in, you log in as level zero, and then you go and hack your way to level one and so on. I saw it the first time at UC Santa Barbara in- Same with me. Yeah, exactly. And our advisors in Giovanni's security course, where, you know, you log in, and this is a homework assignment. You log into level one, you're given a username and password. And then from there, it's just the exploitation race. I think in level 13 or 14 was the last one. And on the machine itself, there's a command, you know, scoreboard or whatever. And it'll print an ASCII scoreboard just based on the log of who logged into what. And that was a super awesome design, but I had no idea that it used to be a def, the DEF CON qualify design. That's crazy. And to connect even further, this is something I still use in my grad class. So I borrowed from Giovanni and modified levels and made it to be 15 levels, but this is still, and I think in terms of education, right? It's a nice model for approaching a security challenge because you have one thing to focus on. So on one hand, you have one thing to focus on, right? You solve that challenge and then you get to the next one. And if you design it correctly, it becomes a building up of your skill sets over time so that you learn kind of one thing in one level. And then that takes you to the next level and then the next level. So that's something that I think is a very powerful model. Now it has downsides, right? I think the key problem is when you get stuck at a level, it is insanely frustrating and you have nothing to move on to. But I've also seen the reverse where I've done it where you have all 10 levels that can be worked on at once. And yeah, exactly. So Zion is mentioning in the chat on YouTube that this is in CSE 365. So I also developed the same thing for undergrads, but this one I let them exploit any level they wanted. And while that has a lot of benefits, the downside is you're really constrained in terms of what things you actually start exploiting. So it can be a really tricky, it's a really tricky kind of trade-off because people want to spend, they may not spend enough time to exploit a certain level and they'll pop between levels trying to figure things out. So I think this is just a fascinating way and approach to think about doing a type of CTF. And it's amazing seeing that that notion that still exists today came all the way from DEF CON 14. Sorry, it came all the way from ghetto hackers with their original qualifying event. So they must have just let the teams SSH into a system and they must have been a shared machine that they all had gated levels. And maybe, I don't know, this too early for SSH? Were they telemetting in? Like, I don't even know. So that's an interesting question. I hadn't actually considered that. No, I would have met, well, so SSH was just starting to get adoption in 2001, right? So it could have been potentially, it could have been pre-SSH, but yeah, it depends on how far back in ghetto hackers history. Yeah, so it's, and I think that we actually did bring this up to Giovanni Vigna at UCSB and he said he actually got the idea of his levels from over the wire. So it actually didn't come directly from DEF CON that he got that idea. So it's super interesting seeing the seeds of an idea kind of go throughout history and that are still alive and well today. So it's fascinating to see that. Yeah, yeah, yeah. I mean, that's, I think it's very easy to lose track of that. Ken Shoto has done some talks, some members of Ken Shoto, but there isn't a, let's say a book on, you know, or even a series of blog posts necessarily on how, you know, they went through and designed the qualifiers and what they took as inspiration and et cetera. And there have been talks and private conversations with legit BS. And I'm sure there have been talks and private conversations with ghetto hackers and DD Tech. And their presentation actually, DD Tech has quite a few that hopefully are still somewhere on YouTube or something, but it's hard to get all of this information in a digestible way. I don't know, maybe we'll just be yet another hard to digest, you know, hours and hours of me babbling on and on about legacy and stuff, but, you know. I think it's useful, right? Hannah Milley in the Twitch stream suggested CTF. CTF work. GTFVO.OO slash K. Yeah, amazing. So yeah, that's, and I think it's something that one of our goals as we go forward with this podcast, right? We want to talk about not just the current state of CTF and security, but also looking back to the past and understanding how we got to where we are today by hopefully talking with some of these people. So if we can get, you know, somebody from Ken Shoto on the record to talk about how they created that or what they remember about that time because it's been such a long time since then. So yeah, I think that's awesome. Cool, anything else on this topic? You know, it's, what is really interesting is the history of attack defense. That's also a little bit, you know, oh, it probably came out of DEF CON, right? But initially the early days of DEF CON CTF, again, from word of mouth and rumors and, you know, mythology is that the organizers would set up boxes for attendees to attack, right? So at what point did it become attack defense? At what point did the attendees themselves have to defend their boxes? And it's one of these things where this seems like I'm sure that there's someone that we can sit down with and maybe we should track them down and have them as a guest that could tell us all of this. But the fact that the organizers of DEF CON don't know is interesting. Yeah, that's kind of like- It does seem like a minor thing when you first kind of think about it, right? Like, oh, of course, well, you're just attacking things. Then why not do the defensive component? But it actually fundamentally changes the architecture of the game, how you have to create this thing. You can't just have one box. You have to have end boxes for all the teams. It brings up patching strategies. It brings up a whole different aspect. It brings up network analysis, which you don't have necessarily in an attack game. So what may seem like a minor conceptual, like, well, of course that's the next step, brings with it so much complexity that it would be fascinating to understand who made that leap and maybe how did they expect it to go versus how did it actually go? Yeah, absolutely. Another example that is much more recent is King of the Hill, right? So I was under the impression that SECON created King of the Hill, but I vaguely remember an offhand comment recently, so I should remember it better, but I haven't been sleeping a lot this semester. A vague offhand comment recently, that they might have gotten inspiration from another CTF. Oh, so yeah, I have, this is actually our next comment. So this is a comment that was on our YouTube page. I'm gonna butcher, you wanna take a shot at that name? Let me pull up the duck. Okay, you can't do it. I got it, I got it, let's see. These are the quiet moments that I cut, too. Potetti sensei. Potetti sensei, I don't know what that means, or if it's offensive, so I think we'll hopefully do our best. So they said basically, as far as I know, before SECON CTF started, there had already been a few CTFs played in the King of the Hill style. For example, at least I know, no con named Facebook CTF finals 2013 was one of them, but I'm sure that SECON named it though. So there's something very interesting there about being the one to actually name a thing. If you have the power over naming a thing, then you have power over that thing, right? So people don't associate that with you. So yeah, this is a classic example, like you said, of an idea that existed beforehand, but you kind of associated with SECON as far as other people do, but at least there were other CTFs in that timeframe that were doing similar King of the Hill style CTF challenges. Yeah, absolutely, and oftentimes, one thing is naming and the other thing is having the kind of position in the community to popularize it. For example, I had thought that OCTF invented dynamic scoring, but according to people on OOPS that I've discussed this with, that's not actually the case. Dynamic scoring was around before OCTF is just the first very prominent CTF that used it. That's another classic, one of these examples that's a kind of a, well, of course, dynamic scoring, but until somebody actually does it, shows that it works, it's kind of difficult to make that jump from static scoring to dynamic scoring. Yeah, yeah, yeah, for sure. All right, awesome, yeah, that's great. So those are some of the terms and forms of feedback, right? This is some of the interesting bits of CTF history, CTF trivia that have come from doing this show that we're like super interested to get. So if you have more of these things, please send it in, comment on our pages, do whatever, and we'll, as long as you get those comments to us, we will definitely be able to turn them around and include them in a future episode. You know what might be a cool thing, Adam, to actually have a section on CTF3.ol slash trivia or slash, you know, something like that. That would be cool, that's, yeah, I like that a lot. Awesome, so cool. Now that Yann has assigned me more work, exactly. Yeah. Adam, just put all these links onto your page, it'll be great. What can go wrong? Okay, so we have a couple of questions here. I'm gonna start off with this first one because it's the very first question that we got on our very first episode. And this is, I like that we're gonna answer this question because it's in terms of being honest about who we are and how we came to kind of where we be, where we came from and where we are and not trying to hide or, I don't know, trying to make us seem more fancier than we are. So the question was, did you guys ever get a black badge? And so this comes from in full disclosure, my cousin Trevor, who actually knows nothing, he's not a security person, he's actually a, he works as an assistant director in LA. So if you have commercials or other stuff you wanna hire him for, he's online on Twitter, at TrevMonster, T-R-E-V-M-U-N-S-T-E-R. He's been listening to our podcast even though he knows nothing about security. So I don't know if that says about our podcast or how much of a good cousin he is to me, so. So, yeah, did we, did us guys ever get win a black badge? The two of us, Adam, are absolute news. Black badge lists. Black badge lists. That's why we wear so much black. Exactly, compensate. No, so interestingly, obviously, in case you don't know, Adam, we played for Shellfish before we started. We did. And Shellfish did win Def Con CTF, but they won Def Con CTF in much earlier days than when we started. So it was Shellfish won in 2005, right? I believe so. Do you remember what Def Con that was? I look it up. I wanna say it was 13, but. Yeah, I think it was 13. I think it was, so I was attending Def Con but not as a participant in Def Con CTF since Def Con nine. Def Con 13, I actually missed for personal reasons that in retrospect, we're unfortunate, but I missed Def Con 13. I think that's the Def Con when Shellfish won. It is Def Con 13. I did look it up. So, wouldn't you miss it? On order of overflow, we have people that, you know. Exactly, so we compensate to get that legitimacy for OO by having some of the black badge holders of that Def Con victory with us. Exactly, yeah. In wheelchair and on oxygen and so forth to keep them around. Yeah, they still using, they still use Object Dump, but we love them. Exactly, but we love them. Anyways, the key thing is that it's obviously hard to win a black badge. You need to be very, very good, but you shouldn't let that gait you, right? Exactly. From participating and contributing to the community, right? You don't, I mean, obviously the more accolades you get, the better it is for you, but there are extremely impactful people all over the community that aren't allowed to get recognition that work behind the scenes and so forth. Very best. And then there's the- Object Dump is only valid if you use Intel syntax, so. Exactly, yes, exactly. That's very, very, very real. And so yeah. Oh, the other thing I wanna hype real quick. We are trying to set up a, hopefully sometime in the near-ish term future, we hope to have a podcast with some of those original OGs from Shellfish who won that DEF CON 13 CTF. So I think that'll be super interesting hearing their side of the story and like Jan says, whatever their old age adult memories can dredge up. Hopefully there's some good stuff in there. Yeah, so the important thing I think that you're getting at, Adam, is people have a tendency to look up to all the DEF CON organizers and think of these are hacking gods. I did when I was wandering around DEF CON 9 or DEF CON 10 or DEF CON 11 or every DEF CON and then I would look, okay, these players are gods, the organizers are gods, and then I started playing. I realized, wait, shit, all these players are massive noobs just like me, right? We're noobs in a very targeted way, right? And, but then I figured, okay, the organizers, they're still gods. And then of course, legit BS retired and maybe the organizers were gods, but now I can tell you the organizers are also massive noobs. I think it's just noobs all the way up, basically probably if you get to the point where you're a dark tangent and you take over his life somehow, you'll realize, oh crap, I'm still a noob, you know? Though I'm sure that dark tangent is a hacking god right now. So yeah, it's very easy to look up and be frozen by these people must have learned so much, must have put so much time in. Yon has been playing CTFs for over a decade, whereas I just started three years ago, something along these lines, right? Right. But it's very important to correct and realize everybody is faking it until they make it. And just some people have been doing it for much longer. And you don't have to keep putting the time and the work in. I think that's the key takeaway, right? If you put in the time, the work, the effort, you too can be wherever it is you wanna be. Yeah, absolutely. We have a discussion going on in Twitch right now about Objadump and multicolored pens as the optimal reverse engineering tool. You laugh, but I have been involved in government meetings on government research programs with quite a lot of resources where because of the kind of alignment, what's the term when the sun's aligned, or the stars align? It's like an astrological term. Yeah, exactly, some crazy astrological thing that I don't know. Venus and Mars are... They're in like the same house. Conjunction. Conjunction, there we go. Due to a conjunction of insane events, Objadump and multicolored pens were the optimal reversing technique where a team of very elite people, actually printed out disassembly and stapled it onto a wall and wrote on it. One of my first times teaching a grad class, I had a student who I won't out right now, but they came into my office and said, hey, I don't think level, whatever, 15 is exploitable. I'm like, what do you mean? And then they bring a stack of, they had like 15 pages of paper printed out of the assembly and like laid it all out on my desk and we went over it and like I'm like, all right, yep, you are correct. That does not get you what I wanted. And so then I had to go back and fix it. And yeah, sometimes that's the best way to prove that you're right is a stack of papers. And this person did not have any colored pens or pencils as far as I know, but... That's the problem. That's how you get, how you pull up. Ah, that makes sense. Yeah, yeah, cool. Awesome. All right, so this actually then feeds into our next question. So I prioritized this person's question because they were the first one in our chat. So shout out to at Han Emel, I'm gonna say E-M-I-L-E. And this person asked us, so what kind of modification slash changes to the world of CTF are you expecting in the future? And I'll extend this a little bit and you can either talk about what you expect to see or what maybe you want to see. Of course, without, again, the caveat we're not giving away anything that the order of the overflow is or is not planning to do for any future DEF CON events. So don't try to get that in there. Exactly. I mean, I think if I could pose an area of maybe a genre of answer that we should discuss is how does CTF remain relevant? Right, and there are two parts to this. One is CTF and CTF participants love ponables and very subtle, crazy ponables, insane heap things, crazy kernel exploits, et cetera. We're slowly moving toward a future where that doesn't exist. How do you CTF in a world of rust if we get there? Or a world of... I have a clear answer for that. It's a web, man, like look at the web. There's entire languages that are dedicated to preventing the old classic ponestile vulnerabilities. Yeah, I definitely agree that obviously there will still be CTF opportunities, but try creating a CTF without a ponable. There was a recent one. It was in Newt Duhak. I don't remember what it was, but they said, we're just not gonna have any ponables. And they got a huge amount of blowback, right? So it's something that maybe it'll naturally happen as the community shift technology. For example, last month, Intel released chips with control flow enforcement technology, right? Which is going to wreck ROP pretty heavily. For one, I'm gonna have to rewrite my entire ROP module next year or two years from now or whenever they become actually adopted. But if you take a binary on Ubuntu 20.04 and you disassemble it, you'll see this NBR64 instruction everywhere, right? When those are actually enforced, ROP is not quite gone, but it's going to get much, much, much less relevant, right? But could you, to play devil's advocate a little bit, could somebody have said that about buffer overflows before ROP was really kind of conceptualized and like something like, let's say ASLR. When ASLR took off, right? I think you could reasonably have suspected like, oh yeah, okay, this kind of type of exploitation is going away, but it took new techniques like ROP to make it so that that's not the case. So do you foresee that as a fundamental shift or is it a shift until attackers until we get better about exploiting this weird machine? I mean, you already have answers to this in some sense, right? Block-oriented programming in academia, right? So you say, okay, well, given enough control over data, we can do the equivalent of ROP, but on entire basic blocks or entire paths, control paths, right? And you've had that for a while in 2015 was the height of this ROP, anti-ROP insanity, right? That was in academia was a huge cat and mouse game. So I think CET isn't going to kill ROP, I agree, but we're gradually moving further and further into kind of the sunset of memory corruption. I mean, maybe, hopefully from the perspective of security engineers, but hopefully not from the perspective of hackers. Yeah, for the benefit of computer users everywhere, I think we all want a future where these types of vulnerabilities go away or these types of exploits go away, right? I think that'll make definitely everything better. So hopefully that does make the difference, but my gut reaction from seeing a lot of this stuff is that it will cause people to look elsewhere for other types of bugs, right? There'll be data only attacks or there'll be different types of ways of that maybe aren't as beautiful and fun as we like ROP to be, but that still allow different types of, let's say, I mean, that's different types of exploits, right? It's all about what is a vulnerability at the end of the day or an exploit? It's something that allows you to compromise the security of an application. So if that causes us to get more creative and more clever with what types of applications we go after. I think the thing to think about is Ethereum and smart contracts, right? They are written in a language that doesn't have buffer overflows, doesn't have this kind of thing. And yet people make silly programming mistakes that allow you to steal money and have real financial consequences. No, and you can see a web assembly and all of this where, and you could see actually talk about this exact microcosm of this in heap, right? So back in the day, and again, 2015, 2016, it was like the first heap renaissance and you had a house of this, house of that and everything worked, everything was the wild west and then the heap slowly started hardening, right? And things were harder and harder and harder and then teacash came out, right? And so suddenly it was Christmas time, open season on the heap once again. And now things are starting to harden again, right? So maybe it's cyclical, maybe, like you said, RAP will go in and there'll be some other crazy thing and it'll be web assembly or YAN 85 or something crazy, some new style of computing where all of this is relevant or Ethereum. But I don't know, I do think that it, I mean, eventually, right, looking ahead decades, we have to think about is CTF going to be as fun or how can we keep CTF as fun without Ponebles? I think this non-ponible CTF was an interesting experiment. I'm not sure what the motivations behind it was. Yeah, well, you definitely heard it here first, no Ponebles in future DEF CONs, CTFs. That's what YAN sang. Exactly, exactly. Yeah, and then the other question is how do we see automation in CTFs? Wait, let's put a pause in that question for one second. I wanna see if something happens but because I'd like to weigh in first on what I see in CTFs. Oh, okay, yeah, because I, go ahead. Yeah, so I think actually the, and that's why I kinda wanted to interpret this question more so is what changes what I'd like to see in the future while YAN messes with his audio driver, which is super fun in the middle of recording a podcast. What, I think the question we really have to ask ourselves is, and it's something you touched on a little bit, YAN, it's like, why do people leave CTF and stop CTFing after four or five years? I think everyone kind of sees that it's a, in some sense, one way to phrase it is it's a young person's game, right? People, you got to spend weekends, you got to give up your weekends to CTF, you have to give up your time during the week. And I think time spent preparing for CTF is always time spent worthwhile, but is it really sustainable to be competing in 48 hour CTFs weekend after weekend after weekend? And I think we really need to start thinking about burnout, like, people stop CTFing, right? And some of them go and be security professionals at companies and maybe CTF occasionally, but there's a very clear thing that about every five or six years kind of people rotate out of CTFs. So I'd like people to start thinking about how can we try to change CTFs so that they are more sustainable so that we can actually maybe play CTFs? And I know there's a lot of insane things. I feel I'm confident that I can say this because I was shot down many, many, many times, but I've was pitched, like, why don't we have, can we have a month of CTF? Could you have a CTF spread out over a month so that you have challenges leaked out but given maybe a few at a time so people could work on it over the course of a week? I know there's a lot of kind of problems and flip sides to that, but I'd really like to see something that is sustainable that people can play CTFs for 15 or 20 years so that I don't wanna get burnt out as I get older, more responsibilities and more of my free time goes away. I still wanna be able to play and contribute to CTFs. So that's kind of the thing I'd like to see people think about and discuss. Yeah, yeah. It's, I think it's something that everyone sees in the community, right? There are some, I think of it as a half-life, right? So it's not that you're guaranteed to cycle out in five years, but half of your team is going to cycle out in five years or at the very least in five years after they stop being students. Yeah, it's kind of tricky. And of course you get the hangers on this, right? Right. Like Chris Eagle or Giovanni, the sort of, you know, pillars of the community, but yeah, I wonder what that would look like. You have stuff like CTF 365, right? Where just theoretically it's a constantly running CTF, but it's tricky, right? There's something different about being forced to learn something in a week or a weekend. Exactly, those time constraints, I mean, are definitely a huge part of it. And maybe, I don't know, maybe one thing it goes from is from a novice style thing to, you know, can we think about, I don't know, it's esports like the good definition, right? You can drop in and play. I'm gonna sound dumb because I don't know any latest. I'm gonna say it's StarCraft too, maybe. I think some old people still play that, yeah. Yeah, I don't know. I was gonna say Counter Strike, but that would really, really date me, so. Counter Strike 1-6. There you go. So yeah, so you can pop in, play a game and pop out, right? But there's a whole competitive aspect of people who are doing it literally professionally, right? So I think that that's kind of like an interesting maybe way to take that. So yeah, okay, cool. So yeah, going back to then the question that came up in the chat. So thanks, Ghost Raptor. So in the Twitch chat, so where do you see the future of automation in CTF? And do you see automated analysis and exploitation taking a bigger part? So actually, you already started to see that. And it was- Exactly, we saw action and reaction, right? Yes, so shortly after, I don't remember when this started up, but with the rise, it was also around the Cyber Grand Challenge, right? With the rise of these complex program analysis techniques that can then rewrite binaries automatically, et cetera, et cetera, there were teams that have shown up with these automated Superman defenses, right? And whether that is, you know, you go and do something simple, like have an LD preload, module that or just change up the PLT for free to just remove it. So, you know, break a lot of heap vulnerabilities or complex stuff. Like rewrite a binary to extend every stack frame or re-randomize locations or, you know, all sorts of stuff, right? Or varying complexity. So automatic patching has been a thing for a very, very long time. And it's something that has basically been disallowed, whether by just statements in the rules saying Superman defenses are not allowed and we will go to you and force you to roll something back or like what do we do at DEF CON or what we have done at least in our previous three years where we have a limit on the number of bytes you can patch in a binary, right? Just to make sure that people are really patching. Actually fixing the bug and not just fixing, trying to not even fix but obfuscate an entire class of bugs, let's say. Exactly. Yeah, because vulnerabilities exploitation is extremely precise. And that's part of the point. It wouldn't be very fun if it wasn't very precise. It wasn't a careful skill. And of course you can screw up a subtle exploit very easily. It's much easier to break an exploit than it is to land an exploit. So there's this multi bug exploits, right? So if you have a bug that has multiple stages, you may, your Superman defense may have disrupted the second phase without really understanding what the underlying problem is, which is not cool. Yeah, exactly, exactly. So it's, there are problems. And so people, the organizers fight against these. Then the question on the automated side, I think it's going to be a very, very long time before we see automation being like true automation, not just helpful tooling, but true automation being competitive on the high levels. Because by definition, the high levels are the very, very edge of the state of the art. So for example, if someone discovers some novel, novel, crazy heap technique and has a CTF challenge with that crazy heap technique, I've created automatic exploitation or I've been involved in the creation of automatic exploitation. Are you creating all of mechanical fish? That's what you're stating right here. All me, no. But I was involved in research on the creation of these systems. And they are very fragile, right? They are, and the way that you create them is you have an idea of what the vulnerabilities you need to target and exploit are and you create an automated system that kind of thinks like you do. When you create something that thinks like you do and then you receive a challenge that for you would require you to learn something extra, the system that you created does not have that capability. That's the core problem with automatic attack. Now that's for fully automatic attack. I think we are seeing there are a lot of research, large overarching research programs including ones going on that our lab is involved with for aiding a human in the ability to approach these complex things. That I think we'll see quite a lot of new tools or new extensions to existing tools that'll basically bring automation to it or human guided automation. Full automation, I don't see being very effective. Sorry, I think that's, and the interesting thing if you look at it, you could say, well, there's already a lot of automation in CTFing, right? I mean, part of, at least, part of it is being familiar with things. I mean, Pone Tools is probably one of the best examples of, I don't know if you'd call that necessarily automation but it does automate a lot of the tedious and difficult parts of poning things, right? Or like you said, a Rop compiler, right? Would be as a clear example of automating something that you could definitely do by hand, but it's such a tedious part that you want a machine to do it. But like you said, it really gets to the, what is the core of a CTF, right? The core is not to run another cybergrime challenge to see who has the best automated systems. And people I think could have different views on that. You may want to, Rodeo Day is I think a good example of something where the whole point is to create automated systems. But in our view, and I think we have a shared view of this, Jan, is that the point of a CTF is to figure out who has the best human hackers. And yeah, sure they may be augmented with scripts and tools and other kinds of stuff. But at the end of the day, if it's a question of two different teams, you want the one that's the better hackers to win, not the one that has better automation, right? Yeah, yeah, exactly. I mean, there's a place for automation and that is also part of your preparation. But the automation that you want to see is extremely general, right? Because you want to see, okay, not who is the best x8664 heap challenges on libc 2.31 hacker, right? You want to see who's the best hacker. And that's the same for automation too, right? So if they really create an automation that is really useful for example, when we released anger, right? And before we released, especially before we released anger, every, not every, but many, many reversing challenges up to like three, 400 point challenges would be angerable, right? We would load them in, we would say, okay, so I'll ask you from here to here, we deal with a couple of explosions along the way and then we'd have a flag. Then we released it and for the next year to 200 and 300 challenges were still angerable, then only 200 ones, right? And what we want to see is, if someone creates the next system that can start tackling exploitation challenges or reversing challenges or something along those lines on its own, great. That is very cool to see, but we want to see a very significant step, not just like, you know. So a follow-up question. So in Twitch, QuazTL, QAZT, IELS are abstraction and automation the same? Which I think a great question because I think yes and no, right? So I think in this context, maybe we're thinking abstraction in terms of, you can think of it in maybe decompilers, right? Something that helps you abstract the complexity of a binary by essentially automatically translating it into a higher level language. Yeah, but I don't know that every form of abstraction would be. I feel like this is a crazy question for like a PL theory person that could respond in monads and stuff that I don't think I could definitely do. So I think, I'm going to say for our purposes, yes. I think we're very broadly saying if there's any kind of machine or script or computing thing that you create and write that allows you to do better at CTF, then yeah, I think that would be within kind of the broad automation umbrella that we're talking about. Yeah, it's, I think abstraction is almost more the automation that we would almost like to, you know, that I think is interesting. I think it can help a human more, exactly, right? Because rather than an automation that's just fit to a specific task, like, oh, I can, like a Rop compiler, right? Can generate a Rop payload under very specific circumstances, right? It has to look for exact gadgets. It can't really reason about new gadgets or new scenarios or new ways around things. But whereas if you had something that maybe abstracted the Rop gadgets into a way that you as a human could easily understand to be able to piece them together easier. Yeah, I think that's something that would be definitely useful. That's really interesting. I think a Rop compiler or a good example of automation as a Rop gadget founder, right? Back in the early days of Rop, I mean, some of the first Rop challenges, trying to remember, like, maybe based off of my best effort memory, Harry Potter at, I think, Vlad. What's it at, Vlad? That was an awesome Rop challenge, where it was a super constrained Rop that at the time, if I remember correctly, the tooling available for it, we had to basically obj-dump, we would cat out the binary, randomly offset it, and obj-dump, right? And that was our Rop gadget finder, right? And then things like, you know, X-Rop, RP++, Rop gadget, all of these tools. I think, I don't know, it probably you can't draw direct correlation between CTFs and those tools, but I think it's very clear to somebody like yourself, right, once you spend time doing this manually, say, you know what, it'd be really nice to have something to automate this, right? And then that automation then helps maybe you, but other people are gonna have those same thoughts, and so those tools will be used by everyone very shortly. For sure, for sure. I mean, I think Pone Tools is a good example, right? What tools was created for CTFs by GallupSled, and is now being used in cybersecurity education, I use it extensively in my class, it's being used in real-world exploit writing. It's because the tool you learn through CTF and then start applying widely. But I think Rop gadget finders, in terms of abstraction, is the perfect example, right? It is automation. It automates this, you know, ridiculous, objedump shot script that we had, but then it abstracts over the binary into the Rop gadget, and then there's a further level of abstraction into, you know, whether it's Rop gadgets, automatic chain builder, or something like Aang Rop that, you know, produces the full gadget. Does that automate away hacking and take away human's know? And it doesn't necessarily even, you know, I mean, it doesn't close out specific types of challenges or anything, but it's a very, very useful abstraction tool. Awesome. Cool, so then, switching gears a bit. So we have a question from Twitter that is, and this is from X3ERO0. So I'll link that in the shout-outs. I think that's probably pronounced zero. Oh, yeah it is, okay, that makes sense. Actually, zero, let me double check to make sure this is still the case, is one of the worldwide students participating in the Pwn College, my type of career education, yeah, he's currently in seventh place. Oh, there we go, awesome, it's from zero to hero. So the question is, what's the most crazy as CTF challenge you guys made slash solved in reverse engineering or binary exploitation? I have mine answered, but I'll go next. You wanna go first, Jan? You've actually solved more of these styles of challenge. I think that from you might be more interesting, but. Sure, I mean, I don't know, I think it's one of these laws of, I mean, kind of almost the opposite of laws of big numbers. You could solve a lot of different challenges, but which ones are these, all of them have interesting technical nuggets, right? They're like, for example, in binary reverse engineering, there was one challenge that was at, wanna say code gate, I think it was either code gate or secu inside, one of those where I solved a VHDL reversing, if I remember correctly, and it was like, I've never even seen VHDL before in my life, I've been vaguely aware of it. And so within eight hours, I had to learn VHDL to a good enough point to reverse-compile the VHDL and extract the flag. And I understand that, and then we created this dynamic technique. I don't remember, I think we injected delays or something. Anyways, so we were able to induce a timing attack on the VHDL output, and. That's cool. So then wait, what did you guys use then for? Did you take a hardware class in undergrad or no? I took a, not quite, the close I took to a hardware class was like, we used like the precursor to Arduino, essentially. Right, so we did it in programming like FPGA or anything like that? I don't know. I'm not in school, I never programed. I'm missing out, man. I got little lights to blink and stuff, and it was like a thing. Yeah, and this is very cool stuff. I had the summer after my freshman year of college when I learned, I did computer organization. Computer organization was very cool because it was kind of a hacking course, but at the end I came home that summer, like I'm gonna build an architecture from scratch out of logic gates and stuff, and I never did. Until now when I created Yon85. But yeah, so from that perspective, that's a cool one, but it's cool to me. I think probably looking back, probably as a, to a VHDL Pro, they'd be like, yeah, I mean, so it's like, but I have a couple, like my favorite solution, did we talk about it on the podcast, the rooting of the reason I'm in grad school, kind of my definite origin stories. Yeah, with the pod permissions and stuff. Yeah, and so that's probably my favorite solution to this day because the solution was so, went beyond what was intended. It's not that the solution was unintended, the actual exploit was intended, but then the stage three or whatever, the stage N plus one of that exploit resulted in root compromise of a lot of machines of competitors and landed being grad school. Okay, wait, no, the answer is no. I thought you were talking about when you and your friend who we won't name were fighting over their machines, over your IRC machines. No, we haven't, you haven't told us so, so go ahead. So we have talked about how me and you, Adam, reversed the, reversing 500, virtual machine at DEF CON 2008, right, for 2009. Yeah, we talked about that. And so then we did a great job, shellfish qualified and we went to finals. At finals I show up and me and actually the unnamed friend that you mentioned earlier sit down to do this one challenge. I forget what it was called, but it was like a crazy Lisp interpreter with some memory corruption inside and blah, blah, blah, blah, blah. And so we exploit this thing, read an exploit and we are out to dinner as a team Saturday night. And I'm thinking, man, something is just not adding up with the way that the file system is set up in the vulnerable box and it's giving me vibes similar to that crazy rooting of my friend's machine that we had talked about in our origin stories, right? And so I realized, wait a second, it's the exact same problem. If you move, if you get command execution, the back then the challenges were written to run as root, open up the low ports they needed and then drop privileges. And I realized if you don't have access to the binary itself, it's owned by root. But if you move it out of the way because you have read access to the directory and then you copy it back in, now you have read access to the file and then you can patch out the privilege drop and then you can kill the parent because the parent was now running it with an effective user idea of yourself. And then the next time when the team would restart, it wouldn't drop privileges and it's running as root. The next time you attack, it's your machine, right? And so we wrote this exploit and we launched it on Sunday and on Sunday morning, we took over, I don't know how many machines, many and vaulted ourselves. And this is actually a monument to how disorganized we love being, right? We are just showing up and hacking. No organization, barely, there are people in charge but fuck it, we're all just like piling in and hacking. So probably if you had been more ready, that could have been a very easy victory. We could have locked people out of their machines and then collected all the flags. And as it was, we just collected a bunch of flags and then fell back to fourth place by that of the game. But that was probably my favorite solution. The challenge itself I barely remember but the solution of the challenge was awesome. And that actually, that solution is what got me when I reached out to Giovanni afterwards about doing graduate work, you know, that's what he remembered when I reached out to him. That makes sense now, okay, yeah. Nice. So, yeah, I mean, I have a couple others I can talk about but I think we should bounce forward. Yeah, sure, I think for me, my one's going to be the, I think the pinnacle of my craziness was this last one with the parallel AF challenge for DEF CON 28 CTF. So I haven't mentioned it, I briefly mentioned it on the podcast but not, I haven't mentioned it in terms of, we did an in-depth discussion during DEF CON about it during some audience interaction. So anyways, I will briefly, so basically, I don't know, maybe it should be probably clear by now that I like thinking about computation in different terms or thinking, kind of trying to boil hacking down to its raw essence and say, okay, what is it that we're really doing here? And it actually goes back to the things we just talked about, about automation, right? Like, if you're a hacker on a desert island or an alien comes down and gives you some new system, like, can you hack that system? Are you so reliant on the tools that you have that you can't adapt to this new situation? And so, those are some of the things I love thinking about and especially if it's historical in nature because then it forces you to think about something else that like, and it's also a glimpse into what could have been, right? So like, thinking through different historical architectures, it's like, yeah, but we could be living, who knows what kind of historical accidents occurred in order to shift us into the current architecture we have now. So I actually, I don't know if you do this, Jan, but I throughout the year keep a list of notes of different types of ideas for CTF challenges. And then when it gets time to prepare for something like quals or finals, I kind of go through there, see if there's anything interesting that sparks my interest. And I had written down and I still don't know where this idea came from. I've asked several people that I think were the culprits, but I don't know. I had down there a data flow machine. And I start doing like all these things. You start Googling and researching and being like, okay, what the heck is a data flow machine? So if you look up in Wikipedia, you can, there's a Wikipedia article about data flow machine architectures. And so I started there, I read this, I read, there's a whole bunch of papers from the 70s and 80s where basically they were trying at the time. Like it's crazy to think about that like the current computer architecture we use, right? x86 von Neumann style architecture. That wasn't set in stone. Like it was not a thing that just like it was assumed that we were going to use that architecture. And so there was, and essentially what really got me going and what really intrigued me about this was that you read about this and then you understand there's no program counter. So it's a computer architecture which is designed to compute something without a program counter. And that runs contrary to just about, and you know, you did how many architectures for that DEF CON 26 finals challenge? Was it 15, 16? 11, 15, 100. Yeah, so different architectures. Did any of them, did any of them not have a program counter? No. Yeah, so program counter is basically, so the way most CPU architectures work is you have your program instruction as bytes somewhere in memory and the program counter is a register inside the CPU that says you are currently at this instruction. This is the next instruction that's going to execute. So it fetches the memory. The CPU has to fetch the instruction from memory, decode it, figure out what it is. Oh, it's an ad. I need to add these two registers together and put the data over here. And then it updates the program counter after this instruction is done executing to what the next instruction is. And that's how you get things like jumps. So it'll say execute this, this, this. Oh, and then jump down here to this other thing. And, but this is a data flow machines do not have a program counter, which seems bonkers. Like how could you compute something if you don't know what instruction you're on and what things you're supposed to do next? And so the idea is rather than thinking about your computation in terms of the, like what should execute next in a sequential order, what you encode in the architecture is how should data flow throughout the system? So if you wanna calculate what's one plus two, you would have an instruction that says, take one and take two and add them together. And then you'd send the results to the next instruction that needed it. So in this way, you have this graph that defines how to do the computation. So it actually gives you even a different way of representing your instructions at the architecture level. And actually all of the papers on this topic actually show you this. So this is what I kind of started running with and I was like, oh, this would be a super cool reversing slash poning problem if like I could create an emulator for this data flow machine. And the beautiful thing that I found was these machines were actually created. Like they created physical machines about, they created these physical machines. And so then it becomes something that's not just a theoretical exercise. Like people actually built these things. So I implemented an emulator. So I implemented like a CPU and basically created a C program that had, and the way all these things work is there's different modules and they have message passing between them. So I had cues between them. So basically like everything did, you basically like any instruction that can execute gets passed to an instruction unit that can execute it. And then it's data, it encodes where it's data is supposed to go next. And then that goes into something that then figures out, okay, the next instruction, here's the input, does it have its second input yet? If it does, then it can execute. If it doesn't, then it can't execute. And it just keeps going like this. And so that was super fun. And I mean, it's, it is bonkers. Like it's crazy. So I wrote this essentially like the machine. So I wrote the machine and said, okay, I have the machine. I have all the instructions for the machine. Crap, I've gotta now figure out how do I program this effing machine? So I programmed it by hand with simple test cases. And then I had to extend that. And now you have to start thinking like, okay, I'm building up this base, but what do I want it to be in terms of a challenge? So what I realized is I had to write an operating system basically, and then user space programs. And to do that, I created my own simple programming language that would be a language that I could write a program in that would compile down to this Dataflow architecture. Anyways, it was all kinds of insanity. It took me months. And the, yeah, go ahead, Jan. Do you know the concept of like a conservatorship when you can basically say, okay, this adult person is incapable of existing in societies? I do, but I don't know how you're gonna use this right now. So I think at the court hearing where we'll establish legal control where you're gonna play this, they're like, oh yeah, that was a crazy person. So yeah, but then you need, and then the key thing that I always think of with these type of challenges is just like what, and then the other thing is like, think of the vulnerability, right? And what type of exploits do you want people to write from this, right? So it's an attack defense challenge. I didn't want it to be just some just quote, quote, a reversing challenge, right? So I like the vulnerability should then tie into, because if it's just a trivial, but for overflow in the emulator, you could completely exploit it without understanding any of this insanity that I just probably spent way too long explaining. And so then what I realized is by the nature of how this architecture works, it's fundamentally impossible without additional features to support a separation between the operating system and user space because instructions can send their result wherever they want it to go. That reminds me, I did have to write like a, basically like a loader and an elf and a relocatable. Yeah, man, that was crazy. Now I think about it, but anyways, so any instruction can essentially send data anywhere, including to the operating system. So you can load a program. So if you had, let's say like a, like permission checks in an operating system, right? Which is essentially what I emulated. So I had a function in the operating system that was open file and you could open any file except the flag file, right? So there's in my code, there's a check for if flag, but the crazy thing that I really wanted the teams to understand is that actually you could just call the open function directly from the, from user space. You could bypass the checks completely by setting up data correctly and sending it essentially into the operating system at a very targeted location. And so yeah, that was definitely, I think this is crazier than the list machine. I don't know if anybody out there on Twitch has done both then could comment. I think it's crazier than the list machine. No, you look. You sound crazier when talking about it. Yeah, I think it's because the list machine built on stuff that was already there. So I think that was less crazy. But yeah, the, you know, and I made some mistakes with that challenge. I think with where we released it, the idea was to have three different stages. The first stage being that there was a back door in a user space program that you could use to call into the kernel to get the flag. So, and then there was a second that removed that where you actually had to do what I wanted and had people upload code essentially their own program that they would then execute that would get the flag. And PBP was the only team that made it there. I even, Jan convinced me, this is part of what we do as we're developing challenges. I tell Jan this insanity that I'm thinking of and he, yes, part of this is spurred on by you, man. You're the one who keeps like pushing and being like, you know, there's a single step, there's single step mode in, what is it? 8086 or something, but yeah. And so I created a whole single step mode. So I had a switch in the binary that teams could flip and the first 100 bytes then would be a program that would get called on every instruction so that the teams could actually patch this. So they could patch it in phase two by like moving things around, but they really could like actually patch it by doing like introspection to understand in their own language what things could or could not be called. So yeah, that was crazy. Whew. So I think they answer. Oh, that's super, that's what you see what Perry Bus just wrote in our Twitch stream. Doing the Lismissing Challenge was how Lucas and Perry met. So I fully take responsibility for that. So feel free to name any future children after me. I would highly appreciate that. Or more cats. I'm allergic to cats. I'd rather take a kid. I'll take a middle name, how about that? Middle name is pretty good. There's a very important question on Twitch about my lion painting. So for those who are listening. Yeah, I think it might be a bit of a underwhelming story, but for those that are listening, I'm holding up a very tasteful lion painting into the camera. And he's gonna mess it up when he hangs it, I bet. Oh my God. It may have put his fist through that painting, that'd be amazing. It's only incredibly crooked right now, people that only listen to the podcast. All right, we're back. Anyway, so yeah, the lion painting is there because my partner Mrs. Brooke, Mrs. Yon, but her, she's Brooke, hates it. So I have it in my office, it's awesome. Wait, so the reason it's here is because it's essentially super ugly is what? No, it's awesome, come on. It's philosophically- Do you consider yourself a lion or is that your favorite version of OSX? Let's go with the second one. I think considering yourself a lion, that's just like a level of douchiness that is a little lower. But you get married, you kill all of your partners' babies as a form of making sure that your genes went out, right? Just like a lion. Just like a lion. There's actually, so in Russia, there was a surprisingly common occurrence of like families having a pet lion or a pet tiger. And then, you know, if the dad dies, then the lion kills the rest of the family. Because the lion might recognize the dad as the head of the pride, and then it's over. Nice, wow. That's like a terrible story with nothing to do with where you got that lion painting. It's just why it's in there, so. It's just why it's in there. Where I got it is a mystery. It's not impossible that lion paintings like that instantiate spontaneously. Maybe. Okay, you want to do one more craziest CDF challenge you've made or solved in reverse engineering or by an expectation? Yeah, absolutely. I mean, there are a lot of, obviously, a lot of. So many challenges that you've solved. It's so difficult. There's so many that I solved. It's so hard to choose. Yeah, exactly. I might not have a black badge, but I've solved a lot of 100 point challenges. But what challenge I didn't solve until five minutes after the CTF ended? Actually, there's many of these. Three specific ones that I'm very proud of solving right after the CTF ended. I'm sure. Hey, there we go. I think there's just a timer until Chrome crashes. Oh. Yeah, probably right around this time. So that's awesome. I love all of your uploads. It's great. I don't know why. All right. So there's two exploitation ones and one crypto. The crypto, I guess, is out of the scope of the question. But in solving the crypto, me and Riamar, and Anthony, we found an error in the only crypto book to discuss that problem. We found a typo where instead of minus, they had to plus. It's insane. We found it. We got the wrong solution. We tracked backwards. We tracked down error, and we solved the freaking problem literally five minutes after the CTF ended. Wow. Did you follow up with the authors? Is that why they created that challenge? Maybe they found the same bug and were like, screw these guys. I'm going to make a challenge about this. That's an interesting question. We did not. We should have. Then, for a phoneable, there was a year of, I think it was Secure Inside that had 1,000 point challenges. So they had the standard 100, 200, whatever. Then they had three or four 1,000 point challenges. The phoneable one was an insane thing. This concept of in-video games, sometimes you bust out of the level and you start running around memory. It was that in SoCoBan. When you got out of the level, you dug into the global offset table. Oh, cool. Back then, the global offset table could be writable sometimes. In SoCoBan, you rearrange parts of the level. By doing that, you have to basically hijack control flow and win. The reason that's very memorable to me and the reason I didn't solve it until after the CTF is there's a lot of randomization in that level. So I patched out the PLT stub of random to just red. Because of that, the GOT entry of random never got initialized. On the version of libc that was running remotely that I retrieved from fingerprinting the other values, the version of random had the values or the address of random had the values that I needed. Because I patched it out, it was never initialized and never noticed those values while exploring. I wasn't able to solve the problem. This is actually the inspiration for me writing pre-need. I wrote pre-need to avoid having to patch stuff out on these challenges. At one point, I had some crazy compiler pass. I never actually committed that, I think, that would lay out pre-need so that you would have your function addresses would line up with libc that you were preloading, or at least out the last three bytes. It was insane. Nice. Actually, both those stories are very interesting because they actually weren't about challenges that you solved during the competition. So I think it's interesting to point out and make clear that you cannot solve something during the competition, and that's fine. You actually learn a lot. Maybe in the moment, I'm sure it sucked not to get those points for the crypto challenge, but you came away with it finding bugs. You realize there's bugs in books and papers just like everywhere else. This other thing actually helped you to come up with a new tool and a new, would you call it an abstraction or an automation? Abstraction. So the pre-need abstraction layer. Abstraction sounds fancier. I feel like that's why you gravitate towards using that. Exactly. Yeah, pre-need was my first open-source success. Pre-need is a set of LD preload libraries where instead of calling, I don't know, open, it'll call something, whatever. Yon underscore open. Yeah, exactly. So it's stuff for like disabling randomization, turning a network-alarm, exactly, disabling alarm, turning a network binary into a standard in binary, not crazy stuff at all, and I managed to write up a pitch of 3D more or less as a meme and presented it at Black Hat Tools Arsenal, which, fine, it's not so insanely selective, but it's still funny that I presented LD preload stubs there. Nice. That's awesome, man. Okay, so I think we have one final question. Before we do that, Perry Bus in the Twitch chat has a great quote. It's not about the flags that you capture. It's about the friends you make along the way, so we should all take that to heart. Awesome. So, okay, then the last question let's go with is from Engen-Hell, like Engen values hell, on Twitter who said, if I don't have any local CTF teams in my city or country, what's the best way to start a team or join a team somewhere else? This is a tricky challenge. Right now, in the age of everyone being remote anyways, it's probably the best time to not have a team in your country or city. But Adam, you often have good advice on open to all, right? Yeah, that's what I would recommend. I mean, I'm not affiliated with them. I've never played with open for all, but there's literally a CTF team out there that is dedicated to being open to all people that want to participate, so I would highly recommend checking that out, using that as a way to get your foot in the door. You know, see if you like CTFs, and then I'd recommend playing with them for a bit, get your feet wet, and then once you feel like you kind of the lay of the land, maybe after six months, maybe after a year, start your own damn team. Like, you know, I think this is one of the things that it's totally, it's not only reasonable, but it's important to start, you know, when you see something like this that doesn't exist and that you think should go out there and do it. That's why we created this silly podcast because we've been talking about this for years, about how, you know, so much of CTF information is lost, there's really no place for people to do it, and so we said, hey, let's do it. And it took us, what, a month to release the first episode or something insane? Like, you know, we just made it happen, and, you know, I know there's always, it's always easier to convince yourself that it's impossible or hard to do or that you don't have the time or the bandwidth right now, and some of that stuff may be true, but at the end of the day, you have to decide. You know, I like to think like you are, you are what you spend your time doing, not what you say you are, right? So like, if you want to be the person to run a CTF podcast, make a CTF podcast. If you want to be the person to start a local CTF team, be that person and, you know, be nice, be welcoming and try to get people in, and it's going to take a lot of effort and a lot of work, right? But we've seen from talking with, you know, organizers and captains and all of that that as you bring people in, you start to create a self-perpetuating system. Like, I think one of the really cool examples that I can point to for me personally was the Pone Devils. So the Pone Devils was when I started here at ASU in 2014, they had a CTF team that had basically been dormant for years, and I think they basically occasionally played an ICTF. And so I joined and I said, okay, I'm going to start up this group again. And I'll tell you what, I failed a lot starting that group up. And I keep trying to tell Jan and the current folks about all the things that I tried because they often don't believe me that they don't work and people don't need to find out for themselves why things do or do not work. Like I did, you know, two hours a week where we'd have meetings, two hours a week. We played in CTFs on weekends and I would, you know, you have to be the cheerleader. It's the thing that you don't realize about being a captain and a leader is oftentimes what your job and role is is to be a cheerleader and to get people going. So, you know, and then the Pone Devils went from something that I put in a ton of time and effort into it, to now being a thing where they've essentially merged with shellfish and they're now part of shellfish and it's a self-perpetuating thing. So currently, Zion is the ASU captain of shellfish Pone Devils. I actually don't know your exact role. Sorry, Zion, I'm sure it's somewhere in my brain, but, you know, and he runs the meetings. They're actually concurrently having a meeting right now going over reversing and stuff, which is hopefully why we lost them on the stream. And so that's, you know, I think that's great. And then I think, Zion, you have an example. I can say definitely of, you know, being at UCSB doing your PhD, seeing that shellfish was a kind of maybe losing its way CTF-wise, right? We compete in like the major CTFs. And then what did you decide to do? Did you just like complain about it or? So my big thing was how to keep shellfish able to qualify for DEF CON, right? And so we would do the hack meetings. I would needle people mercilessly about playing DEF CON of quals going to DEF CON. And actually we have to give credit where credit is due because fish showed up, right? And fish is the one that demanded we play every CTF. Oh, awesome. Cool. Credit to you. So I'm glad you put it where it did. Yeah, yeah, yeah. And so that was a game changer. You know, fish said, we should play this CTF. We should play this CTF. We should play this CTF. And we started. And you know, it's tough when you're rusty. It's one thing to get your skills back out for DEF CON quals. And I think actually no longer viable as an approach with how difficult a CTF has gotten in general. But back then, you know, it was difficult. You know, see ourselves, okay, we're number, I don't remember 40th and like Mozilla CTF, right? Or, you know, these, and then we just eventually we're, you know, top five, top five, top two, top one and so on. I think a good example of the modern generation of shellfish that has players not, I mean, from all over the place. ASU, UCSB, the world, a lot of amazing people. Right. And the undergrad is just one seesaw, quals, right? You know, and that's incredible. That's something that we had never managed to do back in my day, even when grad students could play with. Right. So it takes a while. Over years, you'll start seeing payoffs. And it's very tricky to see those payoffs because you still have tough times. Let me look up how shellfish didn't hack the vote, for example, last weekend. I don't think. 20 second, I believe, or something. Exactly. So you'll, you'll, you know, it's hard to maintain that momentum. You'll still have tough, hard times, but it's, yeah, 21st. It's important not to say, oh shit, you know, we got 20, screw this, we're leaving. Yeah, exactly. You know, that's important. And it's important to keep playing CTFs, you know, start, you know, start small with your team. Don't, you know, I would also encourage you, you know, you want to build up that core. So if you can try, you know, even if all of you specialize on one problem area, I think that's totally fine for the start to just like try all the ponying challenge, like, you know, decide what challenges you want to get good at and just work on those challenges and work on them together as a group, meet during the week. You know, you could definitely do that. And then you'll be the person that built up that, that CTF scene in your town. And then maybe, you know, I don't know where you are or what, you know, but there's always the chances in the medium to longer term future of joining a big CTF team or whatever or joining one of these, like, you know, the sour cloud or the M Hecarones or the, you know, entire country or region based CTF super teams, like it definitely happens. So yeah, I think that's really good. And you had something else you wanted to say, John. Yeah, I was going to say the other thing is an alternative route. Get good. And this is tricky, right? You can get good by following along with online war games or educational platforms, right? So if you complete all the phoneable that KR, you'll and then start applying not necessarily to teams that's kind of rare for teams to take these sort of applications, but you could apply to, and this requires a lot more commitment to graduate programs around the world. And if on your CV, on your application package, you say, hey, I actually solved, you know, half a phoneable KR, that means something, right? And there are certain faculty members, us included, that will notice, right? And it's an important part. You can, and all of the education platforms are like that. Right now I am running Pwn College, which is, you know, binary exploitation, educational platform that takes you from kind of, I phrase it like, white belt to yellow belt, right? And people are now starting to email me and say, hey, I've been following along with Pwn College. It's awesome. I want to do this. Can I, you know, come do this at Arizona State University? And that's awesome because I see already someone already, you know, is fighting so hard to understand all this stuff. And, you know, maybe they'll be great to keep working with. And then of course, through that, that's how you pull you into CTF and then you start CTFing with our students and so forth. Yeah. Cool. So, okay, final question in the chat, because I think this one will be very quick. QuazTL asks, do we have any business ventures? I think there are no, I think our job is essentially a business venture. So I think it's an important thing to point out, like we're both professors, right? And at some point, as we were finishing up our graduate studies, we had to make the decision, do we follow along the professor route and, you know, give up quite a lot of potential business venturing and so forth for the sake of education and research, or do we, you know, dive into business ventures? And we chose academia, right? That's an interesting sort of choice. It's not for everybody. We are here to teach you to research and create the next generation of, you know, cybersecurity techniques and stuff and to spread knowledge about them. Great. We're not here to make tons of money, but if you want to give us money, that's fine. Yeah, if somebody wants to back up, you know, a bunch of money for this podcast or really kind of anything, but if you want to back up a dump truck full of money, we're definitely not going to say no. But, you know, we're in a nice spot now where we have the freedom to do something like a silly CTF podcast that, you know, I think some people out there do listen to, which is super nice. And so, yeah, we, you know, we're about kind of giving back, right, with in terms of being DEFCON organizers is our way of giving back to the DEFCON community. We do research, publish papers, publish source code for our papers, our research projects to help kind of push the community forward. So that's what we're going for there. So, yeah, thanks for everyone for joining us today. This was a new experiment in the CTF radio experience. I'm Adam Dee. You can find me on Twitter at Adam DuPay. He's Zardis. You can find him on Twitter at Zardis. Together we're CTF Radio. You can find us on YouTube or Twitter at CTF Radio with three O's. You can send us questions through email at CTF Radio at gmail.com. And who knows, maybe we'll use your questions on another live episode of CTF Radio. So take care everyone and happy hacking. Happy hacking.