 Welcome, everyone, to ethics village. My name is Shane. I'm one of the founders of the village and today we have our final talk of the day is concerning teaching, consulting, pen testing and ethics lessons learned from running a national penetration testing competition. I will let the guys introduce themselves, but this is intended to be an interactive session. If you look near you, you should have an ethical slash unethical card. They may ask questions as to your opinion and if you don't have one after I'm done announcing them, you can raise your hands and I will hand some of them more out. What they'll do is they'll pose a query to you and ask whether or not you think the scenario is ethical or unethical. If you do have questions or comments about a scenario, I ask that you step up to the microphone because we are recording this for posterity and so with that I will let the guys take it away. Okay, so good evening, good afternoon, everybody, I suppose it depends on what time zone your body or in my case my stomach is still in, but we are here to talk about certainly the ethics of what we have seen interacting with collegiate level students doing a competition. So we've got a really short introduction to give you all some background on what the competition is. We're also going to introduce ourselves as we kind of get to some components of this and then for the most part we just have some scenarios. These are things that we've experienced in the last five years doing the competition and this is a good size. So expect to start to see everybody else coming up to the microphone or well we can't really just have you yell it out because we are recording it, but we really want to get everyone's involvement. So I know a lot of people in the crowd, so we might be like picking on you and making you come up too. So the agenda here is like I said we're going to talk through what the competition is, we're going to go through the ethics and then we have some scenarios. So who are we? My name is Lucas Morris, I am in my day job, I am a senior manager of the Crow Horwath or Crow now that does penetration testing, information security, even doing it for about 15 years. In the auspices of the competition I am one of the members of the black team that actually builds the infrastructure, builds the world and runs the competition itself also as a member of the advisory board. Yeah and I am Tom Kupchak, in my real job I am a director of technical operations at Hurricane Labs. We basically do splunk work and I manage the team that does our splunk of limitations and pretend to do other things that happen as needed. Within the competition I am in the white team and handle a lot of the rules and the operations of the event and basically working with Lucas and other members of the advisory board to give the students the experiences that we're hoping to achieve. Hi, I'm Jason, good job, principal consultant for NCC group, breaking things all day long. Within the competition I am on the black team, I help do app devs so a lot of the times we write super custom apps and I think we're going to talk a little bit about some of that in a bit but and I also am part of the pentesting advisory board as well to kind of help set that direction. Hey guys, I'm Dan Borges, for my day job I'm an internal red teamer but for CPTC I help put together all the OSINT and world scenarios and that's kind of like the context and the flavor of the game. It's the fake employees, it's the operations of the company like the docs and a lot of the storylines and then I just wanted to emphasize that it takes a village to make this competition. We're just four people representing a lot of people that put in a lot of hard work. Yeah, and every year the volunteers grow, there's actually some other volunteers in the audience. So let's talk about the competition here very quickly, we really want to get to our scenarios. So what is it? It is an annual collegiate level penetration testing competition. So what does this mean? Well, we're not a CTF, we're not a defense competition, those are also very good but the gap that we saw several years ago was as a consultant people come out and they're very technical, they know how to do the tools, they know how to build a network, they know how to break into AD, Linux, something like that but where they were lacking was interacting with clients, interacting with a business, in internal jobs they were having a hard time communicating risk to their superiors. So what did we do? We put together a competition where the teams from various universities from around the country and actually this year will be around the world come together into various regions and then to a national competition and perform a penetration test. So they are given time inside an environment where we build a fake company including apps, including hundreds of emails, socent profiles, fake servers and an entire internal environment to attack but the important thing is they're not scored only on their technical competency, there are no flags in the environment. It is simply basically a corporate environment and they are scored based on a report and a national is also a presentation to a board which are executives from our sponsors that rate them. So if they don't put something in a report and if they don't communicate it effectively in that report they get no credit even if all of the technical details were there. I really like the last line in here where that's basically the formula for CPTC. We take offensive security, pentesting is the vehicle through which we teach offensive security but we build a custom environment especially Jason Lukas working together to lead that operation. We throw in the business and you get CPTC so it's definitely an equation that we have going on. So this isn't really a competition. We treat it like an engagement. We are actually in character most of the time so if a team comes to us and says well how does this work for the competition, we're like what do you mean? This isn't a competition between your folks and our IT team. Like what do you think this is some kind of game? This is, we act like we are an executive or an engineer or someone at this company and so they do test it for technical issues, we have a ton of technical issues but you won't see a ton about this because a lot of the ethical issues that we have come with the interaction and the decisions that the teams make and it's great because this is a learning ground and a testing ground for them where they can make and you'll see some serious mistakes that frankly at least at my job we would fire someone immediately for that they can then start to talk about. We have a couple of those, we actually focused on more some of the gray area because we wanted to get some good debate for what we have happening but they do have to communicate. So from a technical perspective well each year we do have a different theme, this year actually we're doing a bank so I think there's gonna be some really interesting options there but every year there's a lot of environment. The whole apps are custom, we'll have some off the shelf stuff as well but for example when we were the hospital we had an EMR, we had lots of different, we tried to simulate imaging systems and other things on custom apps and certainly our sponsors also will throw software at us that actually really works out for them getting tested. For the autonomous vehicles we actually had car simulators and Uber was a big sponsor. So there's a big technical component to this but as I said there are no flags, we just try and make it a company. So with that we're not here to talk about the competition itself, if you wanna do that we'll have some stuff if you wanna volunteer later we can talk about it at the very end. We wanna talk about ethics so we are here to teach these students, that is what we are there to do and we are teaching them realistically three major objectives, the first is technology. We need, there's a skills gap in cyber so we need it but the second is soft skills, interaction and then the third is decision making and team interaction because there are six students per team with one captain that's a student and a coach but the coach is not allowed to be engaged during the competition, in fact they can't talk to them for most of that time. So the teams have an opportunity to be dangerous. It's a simulated environment, each team actually gets a separate environment so we do actually spend like 20 grand during the weekend of regionals and for our cloud provider who thankfully is a very positive sponsor of ours because everyone gets their own mirrored copy of the company. So they can be dangerous, they will take things down, they will change environments, they will actually, as you'll see in our first scenario, leave red herrings for other teams during the OSINT process and during a bidding process that we make them go through. We don't give them really a lot about scope, they have to discover that themselves and for those of us that do pen testing you know scope is important, taking systems down is very bad. So they now will learn, if they go into consulting which a small number of the students do, they will have the direct skills but for those that go into an analyst or an engineering role or something in-house they now also know how do I be a better customer and how do I be a better communicator to the people that I'm working with. So, what do we learn through this? Well, a lot of stuff, a lot of stuff. Technically-minded college students, they're really bad at interaction the first time through. I know everybody's probably real surprised about that. The scope of work, is it a suggestion? Because they treat it as such, they will regularly plow through limits that we've put together and I know every single person at this table has said, but I'm a hacker, rules don't apply to me. So just a point out too, we dock them points if they go out of scope. And we will actually dock them more points or less points depending on their reaction. If they, if we walk into a room, sometimes we'll channel angry clients that we've had and we'll start screaming at them, not angrily, but we'll be making a tense situation and if they say- Lucas has a ton of experience with that sort of thing too, so. And I'm generally a very nice person. He's good at being an angry client, just, you know. But if they say, we're very sorry, let us look into this, they make it points back, especially if they come back with a good answer. They're not all of them, but most. If they say that's not us and we have logs, we monitor, we generate 400 gigs worth of data and just the national competition every year, we know. They lose all the points. So are there boundaries for hackers? I think that is a big piece when we think of the ethics of cybersecurity and what are the boundaries that we have working with our clients, working with our companies, both from a responsible reporting, but how we scope and structure our engagements to make sure they're the best. Yeah, I think a lot of times, hackers think if something isn't explicitly denied, then they can get away with it. But when you're working in a client engagement, there's unspoken rules of how you interact with the clients. And this is a super interesting change of perspective for them because as a community, we've built up this great environment where people can come in and just crap all over everything all the time in CTFs, right? And so they come into this with that mindset of like, I'm just gonna bang a land, break all the things. And that's not a professional environment at all. So it's a very big mind shift for them to come into a CTF where that doesn't work. And we have the rules written in such a way that they are broad. And so there is a board of industry professionals that not only makes decisions on what we do and certainly has very little free time putting together this competition throughout the year, but also whenever there's a potential rules violation, some of them are very clear cut. That's not what we don't talk about here today. We wanna talk about the ones that are gray because we get together and we build consensus between it's not just this group, but there's often 10 or more of us before we create a rules violation and flag it as such. Because our rules are broad and we want to try and give them a real world interaction, also knowing that frankly, sometimes if you piss a client off, doesn't matter what you did, you are in the wrong. And we've given ourselves the flexibility to do that. And we firmly believe that teams might, they treat this as a competition, even though we don't want it to be a competition. So they look for nuances in the rules or things that we haven't considered and we don't consider that professional. And we wanna make it clear that if that comes up, we have the ability to deal with it, just like a client would. So the way this is gonna work, now that we've given you about 10 minutes worth of background here is there are a lot of challenges that we deal with. So we have 11 stories. Our hope is to not get through all of them because we wanna hear your thoughts. Whether it's raising your card, but more importantly, we'd love to hear your comments at the microphone as we go through these. So the four of us are gonna tell a story. We've worked through what ones we wanted to tell, but it'll probably be a little bit of communal storytelling. We're gonna keep that short. And then we wanna have a discussion around was how we'll talk about how we responded, but we also wanna hear about how would you have responded and where do you think things belong? And feel free to tweak the scenarios to talk about something a little different if you want. We definitely wanna make this creative and get into some good conversation about what is acceptable and where do we wanna take this competition in the future? Because the idea is these are things that as industry professionals, you've probably ran into yourself with either clients or personally as a grad teamer. So things that we'll be talking about are generally going to be things that you're gonna have experience as well. So I think, Dan, you probably are gonna be the best one to start this one off. But then you can finish it, yeah. This one was pretty interesting. As part of the OSINT for the competition, we had set up a fake company website. The company website links to multiple social media profiles for fake employees, and then from there, it started going into this crazy social network. At one point, a bunch of these fake employees were part of a group where they shared an open Slack invitation. So anybody that found this public group on say Facebook could now join a private Slack chat, which was the company Slack chat. And this was OSINT before the competition started and it's public, so anybody can join. And what we had is we had a whole bunch, and then we seeded this chat with a bunch of conversations. So what we had is we had students log in and then where the real issue came is instead of just logging in and learning this information and doing the OSINT, they started to make profiles of the company and seed false data to the other people that were getting the OSINT to throw them off the trail. So I think we want to stop right there and just take a poll of the audience with your cards. How do you feel about that? I'm seeing a lot of black. This first one, we felt like it was maybe a little more black and white at the beginning, but. Yes. We got the easy one. We want to make sure everyone understands the system. Everybody's in practice. So then we had another team end up reporting it because they basically found a bunch of information that they started doing research on and then they found it was open information and not part of our environment. So then they came to us and they said, is this you, and we told them no, this is not us. And then we started the investigation, found out it was another team, at which point we had to talk to both teams, the team that found the false information and the team that submitted it. So the information itself was something that was not, not copyrighted, but owned, it was proprietary, thank you. And so there were two issues. And again, I'd like to see the cards before we start the discussion. So the team that found this information, not the team that planted it, but the team that found it immediately recognized where it came from. And so their first response actually was to come to us, but in parallel, also start going to that organization before they had any information from us and report us for using that information. Yep. So would like to hear from everybody on this as well. What are your thoughts on that? Was that the ethical decision to immediately go to the proprietary company or should they have waited until they got information from us? Seeing some black, but it feels questionable. It's also white too. Okay. Yeah, there was a little bit of white. I think you may have said a false dichotomy in your question. Oh, did I? Should they have immediately reported it to them or should they have come to us and asked us about it? But what I feel they should have done was find out information as to whether or not it was something they should report. Absolutely. So, you know, it's a good question. I'm sure it was poorly worded on my part. That's okay. So somebody would come up and tell us why you think that's ethical? Awesome. Okay. There was ambiguity for me, because I was like, well, depending on what kind of information this could have been like, holy shit, this is a really bad, you know, is this bad? Guys, this is bad too. And sort of like reporting it to both parties, whether or not they get you in trouble as an aside to, oh, by the way, I found child porn on this website. I reported to the FBI and you. Let's just clarify. This was intellectual property. Yeah, right. But you know, whatever the intellectual property, it could be of the holy crap I'm gonna tell both. So that's why I'm like, well, ethically, I understand that. And we're trying to be a little vague on this. To keep innocent parties innocent. But we will say intellectual property, not that other stuff. So just to make everyone clear. I wanna take that example though, because I've actually run into that example in a pentest where I have found child pornography on my client's network. So let me ask this question outside of this context with CPTC, what's the ethical responsibility there? Is the ethical responsibility because you're under an NDA, right? So am I allowed to go to the FBI? Or do I just report it to the client and allow them to escalate it? Right, correct, yes. Do not pass code, do not collect $200. Right, so from that perspective, they kind of did the right thing, right? Like reported to the body that owns the information and then suss out how it got there and who owns it. But from one perspective, right? There's multiple perspectives here. Yeah. If they were claiming attributes. We're recording, so if you went, sorry. Yeah, on the other hand, if they were claiming a specific attribution to you of this violation, that is the unethical thing. Yeah, yeah. But reporting it is completely ethical. Yeah. So then we started the investigation. We found out it was a team that was doing this to see false information for other teams. So we had the IP address of where it came from and it was an IP address that was most assuredly attributed to a university. And it happened to be a university that was competing. And it was happened to be a university that was competing. So circumstantial evidence, but we called. They came forward when we asked them about it. Yeah, so we called the coach. So a member of the faculty or an employee, the coach has to be. And so we called the coach and through that process, we actually met with the team. And we met with the team in two ways. So the first 10 minutes or so of the meeting, we just met with their coach and their captain. So it would be akin to saying maybe we're meeting with the partner and the project manager or something like that in a consulting engagement. And we were entirely in character and asked them, so why is it that we found you planting false information for our other people that are bidding on and working in our environment? And they did a fantastic, they did a very fantastic job of saying, you know what, we did not do that. I did not know about this, but if you believe it's coming from our IP address. I think that's a different scenario. Oh, was it? Yeah, they admitted fault when we did this. My bad, I'm confining a couple. Needless to say, over the years, I've had to have a few of these meetings. So apologies. So they immediately came forward and said, it was us. Yeah, and they removed the data. And they removed the data. So our response was to dock them points, basically to assess a penalty. Do you wanna talk about that, Dom? I would just say that we held that as unprofessional behavior. And it's no different than if your client discovers you doing something unprofessional, they might fire you, or they might not renew your contract in the future. So very similar to how we held the team accountable in that case. That said, I think once they removed the data, even the group that was going to report to the organization, which theoretically would have kicked them out of whatever they were trying to accomplish with that organization, didn't. They, it all resolved kind of peacefully, and I don't think he lost his access to the thing. That's correct. It was a, the offending university had uploaded a port scan of a private testing and a proprietary network that is used for studying. So that's pretty vague. Next one. So next. Okay, so this is very similar. In this one, we set up LinkedIn profiles for all of our employees, and we set up a fake company. In this one, we had student teams create fake employees of our company to add them as friends and friend request them on LinkedIn. We didn't really appreciate this from that perspective because let's say we were working at that company, that's not a real employee, and we kind of recognized that. So how do you guys feel about ethical or not ethical for someone to do this in a pentesting? I think we need to clarify that phishing was not part of the scope. Yes. It's just a regular network pentest. Now. Hold them up high. All right, seems pretty universally unethical. Now what about if phishing, social engineering, security awareness testing was in scope? Pretty ethical. Pretty ethical, okay. So that really shows the importance of scope there. Yeah. And one of the things that as Lucas was saying that we try to teach is the importance of sticking to your scope and understanding it. And I would say especially that's one of the boundaries that college students like to push. And this kind of gets back to the hackers no rules thing. Like unless we explicitly deny it, they will try it. Specifically, they hadn't even been brought in for their intro interviews yet. This was pre pentest. This was still doing OSINT and they were impersonating employees of the company. So one of the things we do actually ask them to do is to put together a proposal and bid on the work. It does have a minor component in the score, but in reality, we are not looking, we don't expect them to know how to bid on work, put together proposals. Or in my professional life, we don't really look at people to do that until they've got a few couple of years of experience. And so what happens is it's more to get them in character. They put together a proposal and we get some wild bids. Like this is a $2 million pentest. We've also had a $150 pentest before. I really want to go with that one in the future. But we give them actually at that component, we give them a lot of guidance on. We expect to see these items, they get a bulleted list to try and help. Then actually the Friday of the competition, we have them sign an engagement letter. What that actually is is there's a whole set of non-disclosures and other things for other sponsors and the software they're providing, but also photo release and other minor things that you get at any one of these competitions. So they were doing this before the official start date of the engagement. So I'd also like to see people's thoughts on ethical, unethical first for simply creating the account but not asking anyone to friend you or do anything. What are your thoughts on that? Is that ethical or is that unethical? It was positioned as an employee of the company. Yeah. Okay, so I see lots of black, but let me ask this question. How many of you have a start date on a pentest and it's phishings included and to prep for that pentest before the start date, you go out and create profiles for the company because you know you're gonna need them next week when you start this engagement. That's ethical, but it's not ethical for the college kids to do it before that. Why is that different? Please step up to the microphone. Thank you. Quick question, what would they gain from that from starting early? Well, so I know when we do this at work, certainly some efficiency, but the other example I was thinking of is we will also create domains and we will list them and we will start to get them filtered a week or two before we start so that they sit in the filter. Two words, account age. It's very easy to see if an account is two days old versus two years old. We rename accounts and continue to reuse them for that purpose. Sure. Very cool. So I think that tells you what you get out of it and I don't disagree with you, but then I guess my question still stands, why is it not okay for the college kids to do that? Everybody held up black for that, but white for pentesters to do that. Why is that different? So the question was engagement letter being signed versus start date. If you know your start date is two weeks out, you know your start date is two weeks out so you're prepped for it. I like that. And yeah, you're very correct. We often have our engagement letter signed months in advance or weeks in advance so that's a good point. I guess I just wanted to comment on, I don't know if it's a sentiment that others share, but there's a difference at least for me between ethical and unethical and then allowed or not allowed. So I guess of the same hacker mentality, if it's not explicitly said, then I probably would do that as well, but I'd also acknowledge if I was caught on it that it's unethical. Okay, very cool. And I know, it's actually a very good point. One of the things that I think it's a, That's awesome. Yeah, it's a lesson that we've debated a lot internally to our firm is allowed, unallowed, ethical, unethical. And then also the third level is how upset will the client be versus not and who is our client? Because that, their personality may greatly sway how we approach something. So let me ask this question. Should unethical behavior be allowed in pen testing? Contract, real world should, because we're designing the game. Sorry, I shouldn't use the mic for the speaker, sorry. Since we designed the game to reflect real world contract, that's our whole goal is we want to get kids exposed to what this life, what contracting life is like. So we'll say real world, should that be allowed? Yeah, in the auspices of this game, those are the same. I'm going to say if you talk to the client and you agree with the client on the specific thing that you're emulating, then in that scenario, I think there's value in testing unethical techniques. But I also think the client needs to be aware of that and also give you approval and sign off on that. Yeah, yeah, that's definitely. Because that makes your approach at least ethical in terms of working with the client. Yes. Exactly, exactly. In this context, would you say that it's more important to be ethical or professional with a client and what's the difference between the two? Interesting. I would say yes. I mean, I feel like in at least, I'm sure this may be different for different people, but for me and for what we do at our firm, those are the same, right? So something unethical, I do not feel like can be professional. I wanted to make this differentiation earlier when we said rules versus ethics. I think a lot of times when there's a break of rules, we address that out of character and out of the game. We say, should we pause the game? Should we adjust scores? Should we correct this somehow? And if it's an unethical decision, we handle that in character in the game and kind of just run it to ground and play with it. So I have a comment about some of the terms you've been using and then a question. So as you describe this, you use the words game and you use the words competition and competition is actually in the title. So words are really important, especially for undergrads who don't have that experience of here's what consulting, here's what contracting is. So as soon as you start using the words game and competition, they're like, oh yeah, points. We need to win this. So I think unfortunately some of these situations are set up because of the terms that are being used. So I know you probably do a lot of discussion with the teams afterwards. There's a lot of lessons learned and they're probably like, yeah, you know, that does make sense now that I know about it. What are some of the universities or what are you all doing beforehand to teach them some of these lessons before it happens so that they have that knowledge so that they can run through it on their own as they're considering doing some of these scenarios? So I actually think there's a fair amount that we try to do but it is actually a very good point that there is a lot more that we can do. So at the beginning of the competition, during, as I mentioned, we will have them do a bid and a proposal and we take time out of character and almost every meeting that we have up until the competition begins to remind them and explain to them, look, here's the differentiation between game and competition and real world. Here's what consulting looks like and there's actually some competitors in the room so if we're not doing this, please tell us but I feel like we make a best effort to explain the difference to them in as simplistic of terms as possible but I know we can grow. Yeah, I would say one of the emphasis we're doing this year is making sure that there's a conference call that every team can be on in advance so we set these ground rules in advance. Also, by doing this, we are providing OSINT about the environment and making this recording publicly available and we expect teams who are interested in competing to watch this and learn about what they're gonna be doing. I think also we're meeting with the coaches, we've been meeting with the coaches for months even though the competition is still months away and I don't think we ever clearly set this expectation and maybe we should but our hope is that the coaches would be going back to the teams and providing that guidance based on the discussions that we're having in those calls. To add into the comment though when you're looking at unethical situations and let's say if we're looking at a company and you're doing your work both nationally and internationally, some international laws could actually put those people into prison. So that would be a real life scenario where being unethical in that scenario would be bad for them. Sure, so just to put it out there, the international component this year, I'm really excited for this because it's a Middle Eastern region and it's gonna be interesting to see how they approach things perhaps differently than US students. I'm really kind of excited about this. And certainly all of our infrastructures run locally so we're not exposing our students to that liability but I mean, yeah, we at work have contracts and in terms in our engagement letters around that exact component. But that all said, talking about liability and doing things, we have a later scenario where teams are coming up with creative ways to potentially cause real risks to them if they were an organization and actually causing risk for us which we unfortunately have to deal with but yeah, that's all in the joys of running this thing. Ready to step on? Yeah, I think so. Cool. So, yeah, this one's good. You wanna take it? Yeah, take it. So this scenario was really funny. Basically, we started doing the OSINT and we released an email address at one point and somebody emailed us a fork bomb and I don't even know what they were thinking cause like, what do I do? Pipe my mail to like my command line? No. So they email us a fork bomb and I'm just like, okay, who did this? And they use an anonymous mail service. When you look at the headers of the anonymous mail service it records the IP address that it was sent from. It was sent from the public IP of the university that was competing in our competition. Spoiler alert, university IPs are tracked to you pretty well, it's hard to deny that. So we sent an email to the school at the thing and we're like, did you send us a fork bomb? Like, hello? And they replied and they were like, no bro, not us. So stop real quick, cards again? Ethical or not ethical to send a fork bomb to someone. No, this is not a joke. This actually happened. It gets better. They tried to do it as a joke. Oh, it gets better. No, it gets better. So they deny it, it's not them. And we're like, okay, we have logs. We'd like to see your logs from that time period and hopefully we can see if there's any connections or anything. So we start to run it down with the school's IT team. This is a computer security and forensics competition. Like, we're gonna find out. So we start to run it down with the IT team and they're like, yeah dude, it was their freaking lab. Like here's the packets, right? And then so at this point the competition keeps going on and this is one of those things where they're not really breaking rules but it's kind of unethical. So we're like, whatever, we'll just deal with this in character. Like the competition will continue and we just had these like random meetings with this team to resolve this issue that we were having with them. So by the end of the weekend, we have this final meeting with them where we have all of this data that shows these packets were coming from their logs. They had done an independent analysis of their own logs during the competition. This was before the competition. Okay. This one was before. Okay. Sorry, we have a lot of these. They prepared a presentation for us to let us know what happened and they came in to give this presentation and they basically doubled down on their lie and they said they had forensic logs that showed UDP traffic from Mexico accessing their lab at the time of this fork bomb. So they were like, we basically had these actors in there sending you fork bombs. Yeah, somehow Mexico is targeting this competition that their team happens to be participating in. Even though none of the logs from the institution say this, but whatever. So ethical or ethical, doubling down on your stories. Yeah, black and white, right? You know, do the, do the viral. So. So maybe they're just really embarrassed. So there's a couple other components to this though. So as Dan mentioned, during the competition, they provided a nice report for us that essentially doubled down. Ahead of the competition, we had another meeting with them and this is the one I was thinking of earlier where the first 10 or 15 minutes or so were with their coach who already knew we provided him the courtesy ahead of time of just professional courtesy of, hey, we've seen this happen. He's the one that got us in touch with their IT team to do the investigation. But with the coach and their captain, which was a student. And the captain actually did a very good job of saying, we are gonna do an investigation. Give us a little bit of time to figure this out. We need some time to determine with the additional information you've provided us and our IT team has provided us to really determine what happened. And that is an excellent response for that sort of scenario. Yeah, yeah. So I was- Independent investigation, ethical or unethical. Pretty straightforward, yeah. Okay, so we then had an out of character meeting to discuss this with them as a teachable moment, right? This is an educational piece. So we wanted to just explain from the perspective of actually to your comment of professionalism and game versus competition versus consulting. Let's talk about the consulting component of this and no matter what's happening, right? You have created an issue with your client. So there are some potential ramifications for that. One of the students doubled down in a different way than we expected. So the first thing they said was something I mentioned earlier, almost verbatim, I'm a hacker, the rules don't apply. So I would like to hear maybe this is more a broader comment with the slides. As a hacker, do the rules apply to us in all of the things that we've talked about? There's probably a better way to word this. The rules don't apply? I mean- Yeah, so- Fucking love it. So we all say that, but I have people at my company that I work with that maintain that same attitude. And maybe it's just because they're fresh out of college. We do hire a lot of young kids straight out of school. But that's a attitude that a lot of people are coming through four years of school insecurity and they come out with that attitude. It doesn't matter. I can do what I want because I'm a hacker and that's what you're paying me for. That mindset, which is true. We are paying them for that mindset. I mean, I'm gonna go with that's completely unethical, but it's also a flaw in your hiring process. I don't disagree with you. I don't disagree there. So should we ask ethics questions as part of our hiring process? Yes. Absolutely. You should. How many of you do? Nice. Good. I was just gonna point out because I don't know. I've been competing in defense competitions and so on, but they're not kids. They're students. Yes. I'm sorry. I'm sorry. Thank you. Sorry about that. So the other piece of this was during this meeting, a member of their team, even though the captain was still leading most of the conversation, which again is from a professionalism perspective, what we would expect. A member of their team very loudly spoke up, spoke over their captain and said, it's someone else, I'm not gonna name who, that has access to our lab that did this. So I think there are two things here I'd really like to see people's comments on. The first is, outing someone like that, comments directly, but also, are you as a consultancy still responsible for the things that someone else on your team may do? I feel like yes. Yeah. If someone broke into my company and used our network to attack our clients, contractually, we are most definitely liable up. There's no getting around that. Next? Okay. Oh, you have a question. Oh, okay. Hopefully start to get to a little more some gray stuff here, I think. This one was really funny, I'm gonna describe it. Okay, so this was during our healthcare scenario and basically we had a bunch of systems and our emphasis was on high availability. These were systems that were part of healthcare monitoring systems and we really needed to make sure they were up at all times. Anyway, one of these systems was susceptible to the dirty cow privilege escalation. I forget the exact CVE thing, but it was pretty popular the last few years as a Linux LPE. So the students got access and we actually had the gambit of responses on this one. We had students that didn't talk to us and they were like, oh, sweet, LPE. They ran the exploit and the system crashed and it wouldn't come back up on its own and then they had to come talk to us. We had students that first reached out to us and said, hey, this is vulnerable to dirty cow but there's a risk of it not coming back up. Can I run this? We said, go for it. And then we had students that just decided not to run it at all due to the risk. So I guess the question here is, we'll start with the ones that find the LPE and don't talk to the client. Is that ethical or unethical? Assuming no crash. There's a possibility of a crash. Correct, there's a possibility, but we're not assuming there was or was not. What's the SOW saying? It says high availability health care systems. Yeah, it's a, you're testing. The pen testers, are we testing denial of service? No, no, no, no, no, no, no explicitly out of scope. So I see there's actually some different colors coming up here, some whites and blacks. So I'd like to hear from both. Anybody want to take a shot at running the exploit? This almost just feels less like an ethics question and more being bad at your job. Like that is just contractually bad that you crash the system when you've signed a thing saying that you would maintain high availability, just talk to the client. But there are always, there is always a chance that we will crash the system. Okay, well. Yeah, sorry. There's just some things that have a higher chance than others. Sure, but what's the threshold? Okay, so let's say you ask to run it and it's a system that's critical for you to get root on to pivot into another network and the healthcare system leaves the determination up to the pentester. Should the pentester exploit the system with a chance of taking it down to further their access? Yes, because they've now notified. Use the. So I'm gonna speak for possibly half of the room. I have no idea how many people agree with me, but I would say yes, definitely, because you've notified the client, you can tell someone, hey, make sure if you see this system's light stop blinking, go turn it back on. But once you've notified them and they know that it might crash and they say, all right, then it's up to your decision, then it's totally ethical, at least. It might not be the best idea, but I would say it's at least ethical to do it now. Does it matter what the target is? Say that's a life support machine specifically. Well, that would be the idea of like you, that would be something you would have to talk through with the client and say, do you have someone that's there on that person making sure that this won't cause loss of life or loss of, I mean, a brain function for someone who's on life support. So are you saying that ethically it changes if that is a life support machine hooked up to a patient versus one where someone isn't hooked up? Yeah. Yeah, like a diver. Well, for sure, like if loss of life is involved, that's a whole other level of issue than like losing money in the middle of say like last year's thing. If, actually that was a perfect example. Last year's thing is a whole different case of, oh yeah, like we're really, really sorry, this was very bad, we crashed your database that stored like banking information or yeah, we're really, really sorry, I think we're gonna go to jail, we just killed one of your clients. That's two totally different ethical issues. It is. So let's say instead of a life support machine, it's the stock exchange broker central component that manages all the trades. So you're losing like tens of millions of dollars every second that it's down. How do you feel about that one? I'm speaking for substantially less numbers of people in the room as we go on. I'm gonna throw out an idea, right? Like if you get into a scenario like that and you talk to the client, you can explain the risk of the exploit, you can maybe try it on a dev system and then they could just put you at the level of access that you need. Exactly. I think fundamentally the thing we're discussing here is talking to the client and getting their approval and feedback is really one of the most important parts of it because you as a pentester aren't gonna know if that system is hooked up to a human being that you could kill versus it's their dev. Not that there's ever a hospital that has a dev life support machine, but if there was such a thing, that would be you would essentially make an ethical decision as a pentester to have the client tell you the right feedback. Now if the client's wrong and that results into that, you still did something bad but like from an ethical perspective, I think you've made the best effort to make the right decision in that case and have the client give you the guidance and the client is the one that screwed up in that case. So communication is critical and that's something that we are terrible at as an industry, both in teaching people and in just like fostering that expectation as you kind of go through things like DEF CON and everything else. So we're like, we don't talk about that much. Things to point out or at least bring up is if you are getting that agreement from the client, I'm sorry, if you are getting that agreement from the client, making sure to get in and writing and not verbally over the phone. That is an excellent point because if something is oral, they can easily deny that. If it's written and their headers in the email come from their mail server, that definitely makes it a lot more solid. Yeah, I'm just gonna confirm. Come a little closer to that. Yeah, there has to be the terms of, what, I'm sorry. The rules of engagement need to be defined and there is an operational risk and it's up to both parties if they wanna engage or not. But when you're testing like the OT side of things or something critical where there might be loss of life, like that is an option that you're dealt with and sure that it's an ethical dilemma but you're gonna do your best. I guess I wanted to add to this. I don't, I think I kind of disagree because it seems like the focus here is more on passing liability or just contractual responsibility that if the other party acknowledges the risk, then we can proceed. I'd actually like the industry to approach it with more rigor in that like you do end up playing a part whether it's acknowledged or not and you should be able to assess for yourself whether or not like if this is down the line could end up harming someone, even if the contract that you have or the company you're working with gives you the okay like we should still like take that for ourselves as like pen testers or like people within the cybersecurity community and be able to address that like separately from just like passing on the liability. I think that's an amazing point. I see a lot of times in my business people defer risk, they will keep asking up the chain what is this decision, what is this decision and they don't often add some kind of analysis of like, hey, maybe we can come together on this as a group or like this is my opinion and this is how I vote in this situation and then send it up the chain. A lot of times I just see people deferring that responsibility, that risk to the next person so they don't have to. Yeah, a lot of times you'll see they don't necessarily want to have, they want to have a scapegoat pretty much as opposed to a solution. Yeah. Which we see that, that's a very good point. We see that a lot. And a lot of times the pen test firm is the scapegoat. Yes. And so, I mean, yes. And then another thing that comes into play in the actual competition but you don't really capture it during these questions is there's so much politics usually involved in a pen test, right? And like maybe the organization won't give them that access because they don't want them to see what's on the other side of that machine. So I've seen that too where the organization is resistant to the pen test or continuing to get access which might cause them to try and exploit like this to find out what's out there. The other thing I'd say is this goes beyond just pen testing and even in general consulting I've ran into this sort of thing where like I've been blamed for taking down websites after doing a firewall upgrade when the firewall upgrade was canceled. So it's a whole thing where everyone is trying to look for someone to defer the blame to and we really shouldn't be looking at just blaming. We should be just trying to solve problems. But what do I know? Just to consult them? Yeah. Next one? Before you go, you mentioned that you do role playing as part of the competition or game. Is there any time that you've done this or plan to do it in the future to test ethics to maybe put a trap door in there, a scenario where you may come across in character to try to test that. And if so, if you're thinking about in the future, do you have an ethical obligation to let them know that that may be coming as a real world experience that could come if they're pen testing for the more honorary organizations that are out there? That's a good news, everyone. Yeah. You wanna take it down? Yeah, sure. So we have many things planned for the coming year and the following year. Ocin, osin, osin. Yeah. Little hint there. This year, this year. I would say that we do not necessarily plan on telling the teams that they will be subject to this sort of thing. We expect them to operate professionally and handle things in a way that they think is best, given their role as a pen tester, working for the company that has hired them. So without, you know, delving too much into what we're planning on doing and what we aren't planning on doing, I would say that's absolutely something that is something that we wanna teach. And one of the things I will add as we're doing this, we are not trying to set them up to fail. So we will be giving in the process of that, one of the things we have been discussing at length is how do we do this in a way to give them every hint and every chance to succeed so that it's not, ha, ha, ha, we got you, did you learn? It's more of an opportunity at every step to get something out of it. Right, and I think that's key. Our goal as the penthouse advisory board in particular, like we want students to come out of this because we wanna hire them, right? And we want students that have these skills because right now what we've seen and the reason this all got started is because we're seeing that gap. People can come out and they can break stuff all day. I have people on my team that I can throw anything at them and they will be able to break it and it's amazing to watch, but I cannot put them in front of a client at all because they have no concept of how to interact with the client in a professional way. So we do a lot of coaching before the game on professionalism and what that means within the context of this competition. And I appreciate that comment earlier about our language. I don't know what a better way to define this is so suggestions afterward would be awesome, but we spend a lot of time with them coaching them. What does this mean? And then repeatedly throughout, we expect you to behave professionally, professionally, professionally. But the problem is a lot of these students have never been in a professional environment so they have no idea what that means really. And that's another thing that goes beyond pentesting where if you are in a position where you're working with clients, you could be the best technical person in the world. But if I can't trust you to not be an idiot in front of a customer, I can't have you be the first impression. So that's something that we have to deal with beyond just as a pentester, but just technically in general. So just to explicitly call it out, yes we have ethical human interaction challenges this year. Yeah, he's not wrong. And we now need to send an email to all the teams about that. Because we're ethical. Because they may not see this. Well actually, that's a good question. No, no, we're going to tell all the teams to watch the video for this. Yeah, so I guess that's maybe a good question here is from an ethical perspective, do we need to email or provide direct? Because there are competitors in this room right now. I won't call them out. I'm a competitor, no you don't need to. All right. So I guess that is an ethical question of our should we, is it unethical for us to just say leave it and not say anything and say the people that were here are in a better shape? I think it's there, but I actually think you could play the Devil's Advocate and make an argument of. You might have them longer. No, we did not. We have told everyone that follows our Twitter that we are going to be here and we have told everyone that has registered for the competition to look at our Twitter for news. And we, I plan on sharing this as an innocent thing. So as someone else who puts together competitions, social media is a very poor mechanism to share information, especially if in the context of being ethical. If you're going to say you get an advantage and you get a disadvantage for being able to have access to something, that you're raising the bar of entry to the competition. So you've said you can do this, but now you actually mean you need to go all the way up here because you have to monitor social media accounts in addition to the social media accounts that are being given for the competition. So whether or not that's realistic, it's fun to put it out there, here's this hint, but if those hints are going out there for something at such a big level, in my opinion at least for teams who are playing, they need those bits of information to help build the strategy. Yeah, you're saying it's like too desperate, too hard to access? Yeah. Okay, yeah, so I think that's a great comment. He's saying the information is too desperate, too hard to access, and we almost need to provide these lessons learned directly and I think do the like a thinner version of the OSIN? So do you think that if we were to say, send an email to all the competitors with a list of things that they might want to consider looking at? Oh that's perfectly fine. Would that be, would that resolve that? Yeah, that was my question also. So do you think we need to directly say, hey there will be this type of challenge to be ethical or can we just say, hey you might want to check out this talk? So I don't think saying, spelling everything out for a team helps them. After all the idea of a competition is to help build the skills of a student participating. So no, you don't have to spell it out, but giving everyone the same access to the information sets the entry level and that's where a competition takes it from there. I agree with that. And I would say this talk aside, step out of character here for a second. We completely agree, but it's a good discussion, so. As an additional point, yes access, but also inform them of the availability of it. Yes it's available for anyone who happens to know the video, but I'm not gonna put it on on Twitter, I'm not gonna send an email to the teams because they've got to find it themselves. That becomes really hard to, you know, not good. But sending them out and saying, hey here's the URL and if they don't look, they're damn pro. Sure, right, so we don't have to provide the information directly, but we should provide the accessibility directly, fair enough. One of the other things that's actually been coming up for us a lot as a board this year, now that we're in several years of this, is how do we also level the playing field with new schools versus those that have played before? And that's a very serious consideration that we've had, and actually a future slide that we'll hopefully be discussing, but how do we deal with cases where some schools have more experience competing? Plus there are people who have recognition of schools, like you're gonna know, I'm looking in the audience, I know there are people that have competed, and believe it or not, we have a opinion of you. We judge you. Yes. Oh no. Black flag. We have a black flag being held up on that. I think that this bit quickly kind of devolved into the first chapter of Hitchhiker's Guide to the Galaxy, like, oh yeah, the information's been on display in Alpha Centauri for the last four years, why didn't you check? It goes a little bit back to the core idea, like you started off by saying, oh yeah, this is a competition, but we don't want people being competitive. We want this to be a learning experience, and that creates an interesting dichotomy where you can say, if you wanted to provide out of the way information that only some people knew about and create like that insider's club, that's really good from a competitive standpoint to have more people hunting for these things, but it's not as good for learning and creating a broader base of professionals. So that's interesting, and I agree with you. I guess the question that I have, and since we do have so many competitors or previous competitors, would you be interested if this wasn't a competition? It was just build as, learn how this works. Was that something that would still hold your interest or no, is the competitive part of it a key part? Do both. Having done cyber defense side of things, yes, pen testing, good. We also had to publish curricula and say like, look, this is what you need to know, this is, you know, here's all the information, and we published about six or seven weeks worth of curricula for them to go through for all schools. So that was our attempt to also level the playing field. So I don't know if you guys do something similar, but suggest. It's a good idea. I think the networking opportunity between the student teams is one of the most fantastic aspects of the entire, I mean, probably the pinnacle of the entire competition. What about getting them to work together? I mean, like, so maybe it's still a competition, but maybe you split the teams and, you know, two, two and two or whatever, and now they have to work together. Talking about some curve balls like that. I just, I would love that. Yeah, I would, because then you're working with the, you know, these are the people you're gonna, these are gonna be your clients, these are gonna be your customers, these are gonna be your coworkers, this might even be your boss. This is a very small community, just by having 30,000 people here at this conference. You know, let's catalyze that opportunity to mix and match. Let me just say with, maybe not within the scope of the actual competition, like scoring points aspect of it, but there are things very, very similar to what you're just saying in a Google Doc that we're using for planning. Oh great, can you share that with me? No. Oh. Yeah. Okay, we will share it with you in November. I absolutely wanna highlight that comment. These students are going to be working with each other in the industry, as much as us, but like, they are peers, and it's a shame because I feel like the competitive aspect of it stops them from interacting as much as I wish they did. I really, I would love it if teams sat with each other, share techniques, like kind of share the stuff, but there's that competitive aspect where they don't wanna reveal their hand. So yeah, I'd like to. There's a very firm plan for us to have more time to interact with the students, not only in a competitive, but also in a professional manner. And I know you're going, and I'll give you a chance in just a second, but also some of the things that we've done to try to kind of prevent that competitive nature is require things like you can't have private tools or repositories. You can use whatever you want, but it's gotta be publicly available so you don't have a separate advantage from other teams. So we try to even the playing field that way and I also wanna bring up, that's a really good idea and it happens in the professional world. I've been on incident response engagements where they brought in my company and another company and we have to work together to find the data. So it's a real world example that you're working with competing companies. And to Jason's point, one of the rule additions that we've made for 2019 is allowing teams to use a repository that we provide for any tools that they develop. That is something that has to be public. They have to document it. They have to know how the tool works and be able to explain it to someone else. But it's our belief that if you create something for the competition, that's going to be a contribution to the rest of the information security community that shouldn't be something that secret. So we wanna make that something that we can share under the umbrella of CPTC but further the community as opposed to just having teams hiding in their own circles, developing awesome tools that they don't wanna share. And we figured the best way to do that was to open that up and make it something that was allowed with certain restrictions. So personally, I come not from a CPTC background but from the CPTC background but I think the idea still relates which is imagine that I'm playing in the like a southeastern regional and somebody else is playing in a different region. There is a very, very low percent chance in which I am going to interact with teams that do not qualify from another region. And even the teams that don't even qualify for their region don't interact with anybody. So one of the things that we've been thinking about and this is coming from previous work experience and we've done this before where we said we're gonna work on a project with our entire group, the whole group, this whole department, the large department. Instead of just being your little department that's in one location, you're gonna have to work with at least another group that's in another location. So not only do you have to work with like your, in this case it would be your region but also you have to work with people from other regions. So you have to learn here's how I work remotely even. Like how do I work with people that don't even live anywhere near me, may work in a different time zone, as well as possibly saying like I need to work with people from other teams within my region just to like increase the diversity of thought within like inter-regional ideas. That is an interesting idea. That's a really good idea. We'll have to figure out how to make that work but I like your thoughts. So hopefully not going too far off topic. That's just the whole point of this so that's no problem. Just trying to go from the focus of ethics and I'm gonna throw this back here but if you are as competition organizers, as competition organizers you say okay now you have to, you no longer have your team, you no longer have those people who you trained with, you studied with, however you prepared. You are now split amongst 10 teams, congratulations. Is that something that is to the context of a competition, is that something that's ethical? Is that to rip all the strategy and all that stuff and force teams to do stuff that they don't want? Is that something that is actually ethical? So we just got a card for ethical and unethical, come on up. I think before we do, I'd like to see. See everybody. Let's see everybody. Including us. What was on? Hmm? Most. So, can you restate the question for the audience? Would you want to restate your question? Restate the question, sorry. Actually here I can do, is it ethical in the under the auspices of a competition to force everyone to randomly work with other people knowing that they have trained together? We even have some disagreement up here. Yeah. That's fair. How would you score? Well let's see, I'd like to hear from the audience from one of each. Matt, get up there. Like a lot of us are millennials, we just think everyone wins, right? Well, it's a bad joke. Bad, whatever. If it's the NFL All-Star game, that's one thing. If it's competition, you're training as a team. The team is competing, not the individual. So, all right, now you get to work with different people that you've never worked with but you're still being judged as a team, you have no control over who the other people are. You've never worked together. You might, out of random luck, you might get the lame-ass team. So to clarify, for those that don't see your card, you're holding unethical. Yeah, I'm holding unethical. Okay. Okay. Just like the real world. Yeah. That is why I held unethical because I agree with that. I think if you train with a team, you should play with that team. Yeah. So, what about? If it's a different thing like the All-Star game, which is not part of the regular season, it's like, hey, let's have the Crispin. And then they do, it's kind of a pickup game that becomes ethical because it's not counted. It's more fun. So, basically the difference between the World Series and the All-Star game, or? Bingham. Yeah. So, I'd love to hear from someone that said ethical. Yeah, so it's still like in the context of like, in the competition, like we are still like, the art university team is this random team or it's just like, we are brand new teams that we just, that we're all. Brand new teams. So, if we took one person from every team and all put them on this, like a collective team. Yeah. We shuffled all the teams. So, I don't see an issue with that because it's not like we're representing our university. We're representing this team that was just formed. So, we're pretty much all starting off the same like level playing field. So, there's no, like, yes, you may get someone that isn't as good, but basically everyone has that same percentage of risk where like, they may get someone that's really good, someone that gets someone that's really bad. And overall, since we're not representing like our official university or whatever you're doing, I see no problem with it, especially since this competition is for learning. So, like, I don't see you, just because you train with someone doesn't mean that you can't work in another group, especially if you're not an official university, I see no issue with it. And I've done this before, so. And that's why I said ethical, too, because I kind of agree with that. I don't train as a team with my colleagues, right? It's all just a random grab bag. So, it doesn't make a difference if I'm training with, working with somebody at my company or some other company. I've never worked with them before anyway. And so, the competition environment, like CCDC, where you're like, okay, we need somebody that knows Linux really well. We know you need somebody that knows Windows really well. And the teams do come in with that kind of breakdown, but that's not how it works in a company. Like, that's not how that works at all. To add to that, like, in real companies, or you're gonna be pulled into, you have other teams pulled into your work and you guys need to go, like, we have a detection monitoring team and I work in our team for a company I can't work out. We have instances where we're both working together, but even though we're still relatively close, like, we still have to interact with someone else that we usually don't, we don't do our day-to-day work with. You don't know what their skill level is. I really like that. The person knows what Plaza is. Can they do forensics? Can they do anything? Can they read PCAP? Do they know what logs are? So, like, you don't know who you're working with, which is really sad. I wonder if it would change the internal dynamics of the team, because then you would get one person from each team at the school that won, versus a team that won. You could potentially have a scenario where the team that won represents three or four different universities. Right, it would be one person from every university. Yes, and it would be a new team. But imagine the opportunity for cross-pollination between these programs. Many of these programs are focused in very specific areas. They have a very specific methodology or strategy. If we can remove the barriers to sharing information and change the, I mean, ultimately, somebody has to win one of these competitions by points, but if you look at these competitions, CPTC, CCDC, ultimately, you're actually not competing against each other. You're actually competing against some external entity, right? In CPTC, you're competing against a red team. There is nothing in the rules. I'm not giving you guys any ideas here, but. CPTC or CCDC? CPTC, there is nothing in the rules that says that we can't collaborate. Sure, yeah. And the same is true for, in CCDC, you're essentially going up against a red team. And you're all kind of at the same starting point. There's nothing that says that blue teams couldn't collaborate together to, yeah. But we approach this from a PVP point of view. We need to switch, I think, to a PVE. Right, and I think the opportunity there would be something really special. One of the reasons I really like that idea is because some of the schools that come into us don't have security programs. They've just got students that enjoy security and there's somebody on the faculty that is willing to work with them and kind of foster that. And so they come in and they don't have that. Stanford has no undergrad security program at all. And you're not unique amongst other teams that are similar. And we've actually seen schools that have strong business programs do well in the event because they have the technical skills that people get out of passion just from working with it. And then they also have solid business skills that the technologists kind of suck at. So that's worked out really well. I mean, we've got this incredible diversity of outstanding polytechnical institutes, outstanding theory institutes. Let's get these people together and start working together. Yeah, that could create lifelong friendships, too. Those people will probably trade information, you know what I mean, stay in touch. The people that I've met through this event and this competition, they're going to be with me for the rest of my life. We love curveballs, too, so we're liking ideas. I'm the director of the Michigan Cyber Range. And we've struggled with kind of leveling the playing field. I've hosted numerous competitions. And how do you really level the playing field across teams? You can either one, randomize the teams so you have a diverse team. Or you can say, hey, pick your own team. Kind of force that diversity upon them. If you want to win, you need a diverse team. We all know that. So it's kind of you choose your poison there. And we've hosted a bunch of exercises that I said. And one of the things that we've done is kind of level the playing field by giving everybody the exact same operating system with all the same tools. I mean, that's another way of doing it. So it's really an interesting problem. In the end, it's all about skill and experience for each team. So you can never really truly have this perfect level playing field. That's why you have winners. Thanks. No, and we do. So the teams are all provided. All of the systems that they perpetuate their attacks from and start from are inside the virtual environment we build. So everyone does have the same image, but you're correct. There's a wide variety in skill sets, in creativity, and business, and communication skills, and writing. And it's very tough to take that diversity at a level of where you have smaller teams. We haven't because they're branded, but they're branded by team. And we are working to provide some anonymity so that we can. And we also want to be able to release some of the things that we develop for the organization so that there's a library that other teams can use to prepare. And other competitions. Our tooling's public, but our secret sauce right now isn't, and we are working on making it so that we can do that. But it's one of those things that happens in our free time, free time. Which is even smaller than our free time. Yeah, so I'm glad that a lot of people seem to be bringing up concerns about the whole information siloing aspect of this. Because you know, when you've got a team that's got years of legacy with success at not just CPTC, but other competitions as well, you end up in a position where they can just kind of get the snowball of success to such a size and just steamrolls everyone else. And what I've heard so far being brought up as far as forcing teams to kind of integrate together with their rosters, I think that's an interesting concept. Especially since that's something that people would see in a real world scenario in different parts of a company. But I also think that there are ways for you guys to attack this from kind of the competition organization standpoint as well. For example, with releasing old materials from teams, obviously that puts them at a disadvantage because their strategies for success, those are going to get released. But at the same time, that puts them in a position where they have to conscientiously decide if that's a trade off they want to make. Do they want to put that forward in a scenario where it's going to be eventually released for the sake of winning this competition this year? Or in another example, maybe you want to split up the network environment where it's like, you'd get assigned to sort of collaborate with another team. You're not necessarily integrating throughout the entire rank of one team. This is team A. This is team B. Team A is going to look at one part of the network environment that might have certain information pertinent to the other part. And then when it's time for the two teams to trade off, they would then share different tools and techniques and information that they've found over the course of their assessment, at which point they gain exposure to the other teams, like TTPs and things, and relevant information. And they're actually collaborating and working together. That's a really interesting idea. And that correlates to when I go into a company and do a pen test. And they provide me the last pen testing report that they had, and it was from a different company. So that'd be pretty easy to work into a real-world scenario. Let me ask you this question, though. We've never previously, that I'm aware of, told teams that we would make their reporting, for example, even anonymized, public. Would it be ethical for us to do that? Or is this something we should look at in the future and we make, from here forward, we put something in that says, this is something that will happen. We'd love to see the cards. I think to retroactively go back and do that without their permission would be unethical, but if you get their permission then you're fine. And definitely going forward, having a clause in that, totally fine. I would have to say that I don't know if there's anything in the rules that would prevent us from doing that. Yeah, but as we've been talking about. Yeah, I just wanted to add a quick note. That would take it from PVP to PVE to cooperative, right? Co-op, they need to work together to solve the challenge. I really like that. Two things. I guess on the reporting side, I guess a company like Pen Test Reports, that is part of their identity, that's their intellectual property, so if you were to actually publish that, that would be pretty bad. But another thing is one thing that they did in CCDC, at least starting a few years ago, is that we have a threat intelligence exchange where the captains or people from each of the teams, they sit in a room for like an hour and they go and they talk about, hey, what do you see? And you basically give the other blue teams, things like, oh, I saw this on this box, doing this, et cetera, et cetera. And now something I think they implemented last year. But that could be really cool if you guys could think of something, how you could implement that, at least in terms of the CBTC, like some sort of like techniques or procedures, like tactics, like, hey, we discovered some sort of building on this box, or. What about a forum? A forum would be good. I think a forum would be good, like an open forum for like 30 minutes, where one of the, whatever the product manager who are the lead consultants goes and talks about what they found, or like you kind of get like a short summary of like all of overview. Yeah. Very good ideas everyone has. First, good to hear that someone actually liked that implementation that we put in. But we never really hear feedback. But the other thing is, with a similar topic, if you, we all are talking about whether it's okay when things aren't documented and for ethics of competitions and things like that, is it ever okay, and obviously the whole room can answer this one, but is it okay that for us, when as a competitor your strategy is to find the little details and everyone who's competed, you know you're looking for those in the rules where it misses that one word, where it's this little piece of information that you can just sneak by so you can get a strategy that is successful. And everyone's laughing because they know it's true, but that's how a competition works, at least these competitions work. Now. Hold on. I'm gonna find something real quick. Okay. Here we go. No, no, no, no. This is what we've added to the rules. Yeah. Good job keeping that. Yeah. It's on our website. I get what you're saying, Joe. I competed in an event in college. It was like my last week of college, it was called The Great Race. I had to build these robots and I hacked the competition. We couldn't change the motors and we were fixed on these axles and I changed the wheel size so my robot went way faster than everybody else's robot. It's basically mechanical gear ratios, right? Anyway, we won on a technicality because we hacked the rules and I felt bad afterwards because I looked at everybody else and it wasn't fair, you know what I mean? We found a flaw in the rules and we took advantage of it and it wasn't the same competition. Yeah, the CCDC team I was on at RIT is responsible for some rules at the national level. So, that's part of the things that are going through our heads when we're actually trying to develop this sort of thing and that's honestly the reason we have something like this that we've added. So those nuances, we can say, we know what you're trying to do. It's not something that's professional and we don't want that to happen. There was a second half to this and that is that, you know, we know as organizers that teams are doing this, we know that they're trying to find a way around the rules. When we do find those, how, like, I know there's no clear cut answer, but as organizers, do we, if we're going to be completely ethical, then the rules must state. If you have violated the rules and we've deemed you violating the rules, like what you say, then you must be disqualified or punished in some way because that is how the rules work. However, in the reality, like, it's hard because sometimes it's unknowing, it's accidental and that's what this whole talk is about. So I'm just throwing that out there. How do we as organizers find that balance in our own policies to update them and make them better before we make something that's 100 pages of rules? So, just a thought. I was just gonna say it's a gray area and that's why I like to take the soft approach because it's a learning opportunity for us even as we go through these scenarios. So, like, I like to give people the benefit of the doubt and I don't always assume malicious intent and I try to make it a learning opportunity rather than like a you're kicked out thing. And we- I would agree, oh, go ahead. Oh, I would agree on the professionalism side. It's how the team interacts with us in character when something like this would come up. And that's an advantage that we have that maybe CCD does not have because our focus isn't just on technical, can you keep it up? Can you find the holes? Can you do all this stuff? In fact, that's not even the bulk of the scoring. The bulk of the scoring is your report, how you're interacting with the team during the, with the CBTC team during the engagement. And it's those soft skills. And so, it's a little bit, we have a little bit less of a problem there, I think, than some other schools where it's kind of, I wanna find the technical rules because you can't find technical problems with soft skill, right? It's a little harder to do. And the other thing that I would add, at least what we have tried to do, our approach thus far has been that we look at rule violations by committee, by advisory board. And sometimes, in fact, we'll even bring some of our sponsors in, maybe not read them into every intricate detail or who's responsible, but to ask their opinion and almost go, yes, no, where are we at? And I think that's probably one of the stronger pieces we have to this as well is just, and there have been some times, we have been vehemently disagreeing with each other, but we generally come to a consensus. I can't think of a situation where we voted in the end, so. But there's very strong debate. But there may be very strong aggressive debate. And I guess just to directly answer your question, it has resulted in rule changes so that we can address it. Yeah, because we want to make it a valuable educational experience. We don't want people looking for things that we screwed up and not accounting for and the rules with the common, the wrong place or something like that. We're not lawyers, but realistically, if you're actually doing this in the real world, whatever the customer wants is what is the end game. So if the customer decides they don't like you or didn't like the work that you did or you did something that was against whatever they decided they cared about, that doesn't matter if you're right or wrong. It's what the customer thinks. And we're the customer ultimately in this event. I just want to say rules of engagement is super important and you have to be explicit the entire time. One of the things I've started to dabble with is kind of putting a disclaimer at the end scene. My rules of engagement may evolve during the course of the exercise because of unforeseen bullshit that can happen, right? So you can't just say, here are the rules of engagement. You need to phrase that a little bit differently in a professional term. Exactly. If a team put that in their report, we might dock a few points. So we just call it technical terms. Yeah. So that's interesting though, because as a pen tester, I hate when a client does that. Right. I come in and that rule of engagement is what I'm coming in as, and I can't stand it when a client does that. So I don't disagree with you and it's not a professional environment. It is a competition. So it's not the same thing, but in a world like this where we're trying to emulate the world, I almost feel like we shouldn't do that. Yeah. Let's take that to the audience real quick. Let's take that to the audience. So you're contracted for a pen test, and then they changed the rules of engagement mid-pen test on you, ethical, unethical. Ethical or pain in the ass. There's a big difference between, yeah, I agree. Okay. Now. Change it without a group. You want to go up and talk about that really quick? Yeah. Because this is... We've done pen tests and the customers in halfway through the pen test, they're like, oh shit, don't do that network. Because we forgot that that shouldn't be there. That subnet's not owned by us. And also, as part of our pen test, I mean, previous question, we popped a fetal heart monitor that was connected to a patient. We freaked the hell out with the moment we knew what it was. Talked to them and they're like, oh no, no, it's okay. And we're like, mm-mm, we're not gonna do it. And pulled that out of the statement of work because we're not gonna do that and made it as a separate statement of work, not connected to a patient kind of thing, because it terrified us. Yeah, that's awesome. Yeah. So one of the other scenarios we actually had that I think would be, is very pertinent to this is, client asks you to remove something from a report. Not just change and scope, but. First, before we put that out there, how many of you have been in that scenario in your professional life? Yeah, change of scope happens. No, not change of scope. You've got a vulnerability on a report and the client comes back and says, we don't want that on the report, take it out. You always have an old pen test, or at least it's an actor, and tens of thousands of people, what are you gonna do? Yeah. So, but what if it's not, we want you to remove this, what if it's I have all these mitigating factors that we haven't tested, and I think it should be reduced from maybe a 10 to a five? That's what we're here for. What if we didn't test that? That's a very good, very good. Oh, so the, thank you. The comment was, what if they fixed it after we tested it? Yeah, what if they fix it, we retest it, and it's completely fixed? Do we still report on it? So before the test is over? Before the test is over, that's good, and I like that. They're doing their due diligence, was it too late, you know? To the state, for the team to know that they worked on their due diligence in time. So let's see cards, remove it from a report, ethical or unethical? Unethical. It almost depends. Okay, so let's go with some of these others. Reduce the risk rating. Quick question, does it depend who the report is for? Like if it's just for the client, just for a public? Yeah, I actually don't think so because it would be used internally. But that's my opinion. My opinion is similar to that, though. So what about reducing the risk? What about removing it after we retest it and find it has been resolved? Yeah, so I can tell you how I handle that on my reporting is we leave the finding. When we find a finding, the finding's there, period. If you mitigate it or if you have other factors that reduce the risk, we'll note that in the documentation. We've actually got a special section for client response and we also have like fixed things. So we'll add notes and we'll change the status from uncategorized or whatever to fixed. But it's on the report, because it was a finding when we were there. Correct. Can I just make a comment about going back to the rules? So I think violating the spirit of the competition is the key here. And there have been things that we've found or done where we internally voted it down, that it was not something we were gonna do because it violated the spirit, even though it might have been, we could have gotten away with it. So I think there's some expectation that teams self-police on this. I know this is the real world, but we're all adults and there should be an expectation that you're a professional and that you self-police on these things. That said, there is a certain beauty in creative approaches. And I would just encourage you guys to not write so many rules or become so draconian or stringent that it weeds out that creativity. And in some of the competitions, some of the most interesting work that we've done in preparation for a competition and during the competition is creativity around gaining a competitive advantage and going right up to the edge of the rule, but not crossing over. And they learn more from doing this than they would learn from learning how to instantiate local firewall rules as quickly as possible or change all the passwords as quickly as possible and remove everybody from domain admins. I mean, we all know how to do that. We can all script that. That's kind of passe. So don't lose the creative aspect of this that forces people to think about things in new ways and iterate and evolve. Yeah, I totally agree with that. And at the end of the day, as pen testers, that's what we're paid for is our creativity, right? That's when I'm hiring people, that's what I'm looking for is can you think broken and can you apply that creativity in a professional environment? And that's what we're trying to foster. And I think most of the teams are similar years. They all do a really good job self-policing. And one thing that I really enjoy about this competition in particular is that it is very kind of community. All the teams competing have that kind of community. So at least from my perspective, they do, where it's like, we enjoy doing this and we're here together to learn not kill each other. I think the openness and flexibility and approaches is a key reason why any team can be successful in this engagement. There are many possible ways to win this competition. And I would say don't rule, create so many rules that takes away that incredible facet of the. Definitely, definitely. And I would say part of the reasoning for the rule that allows teams to use publicly available tools that they create is to kind of, yes, we are seeing scenarios where some of that creativity would be the intent of maybe not using or making it clear that they're using something that they pre-staged, for example. And this is actually an area that you and I debated quite a bit on how we wanted to approach it. But our philosophy was, yes, you would be potentially running into a scenario where teams are being very creative and getting around potential detection mechanisms or something in order to get an advantage, kind of scooting around what the rules might allow. And our thought was to make that a much wider scope of what was allowed to remove some of the barriers that tend to remove some of the secrecy and increase the collaboration. Yeah, I mean, in the professional world, if you're doing defensive expertise because you think everybody's gonna learn your secrets, you don't know that much. I mean, so... Just like you're not making your own encryption. No, yeah, exactly. So open this up, level out the playing field and let the best team win. I do wanna just add, I think, we have all these issues up here and we've had issues with teams, but at the end of the day, I can't think of a single team that I've really had a negative overall experience with. They do self-police, they are amazing. Yeah, it's been a great experience. And there have been so many times, myself, Lucas, all of everyone here have been walking around and we've been approached by students who've gone through this program and are just raving about how much it has been a positive impact on their career and experience. And that is the best, most touching thing for me, to have that positive impact on people's lives. Okay, with regards to what the last gentleman said about the fetal heart rate monitor, would you guys, as the clients, ever ask the students, as the pen testers, to perform something unethical? Like, say fishing was allowed, are they allowed to only fish? Like, what does that mean? Is it just company email addresses, personal Gmail accounts, they're kids? How far do you go? So last year we had an insider threat scenario and the insider threat actively tried to get people to delete data, destroy data, to cover his tracks. And many of the teams reported this to a, we had this other investigator, this third party investigator that we were bringing in. Many of the teams reported this. Some of the teams took the data back to the guy. So we've asked them in non-direct ways to do unethical things through this insider threat guy. But in the case of, let's just say, there was an environment where there could be something that could be completely unethical, that could be somehow within scope or be considered appropriate for the pen test. We may totally ask the teams to do that. And unlike some other events where, you know, saying no is frowned upon, we would actually appreciate if the teams were to say no about that. And this is not ethical, we don't want to do this. This is why. And that would be actually something that would result in the teams getting points and doing what we ask would be putting the teams in a position where they're doing the wrong thing and losing points. See, I was just gonna say the exact opposite to that. Okay. I was gonna say, like specifically that scenario of you've got something in the scope and you find out, oh, I am not okay testing this specific thing. That's an interesting idea to put that in. I think we could, and I think we maybe should put it in just as an exercise, but I don't think I would want to score that. I really, I do like it from the perspective of somebody in a position of authority, right? Somebody that's your boss telling you to do something you are not comfortable to do. I really like that scenario. Because that happens to us professionally. Yeah. Where clients try to say, you know, I'm the client, I'm important, you're doing this, and we have to push back on it. And another is where one of the things that we have to do in the competition though is to determine, as Dan said, is this scoreable or not? If we are setting up a teachable moment that is we feel like the majority of the teams are gonna have a difficult time with, the way we score it may be very different. Because our intention is to teach, but we still are running in a competition, a game. And although we, I know we've talked a lot about how do we message this? How do we, what words do we use? So I was trying to think, as I'm sitting here, a better way to say this, we do keep that in mind as we're going through this process as well. Yeah, and I think in some cases, it's pretty black and white, right? You and outside of scope, we have a defined scope. That's pretty black and white. We can score that because it's pretty clear what the ethical implications are. Feudal heart monitor, I feel like that's pretty black and white. I think most of us here felt like that was pretty black and white. But again, as soon as we flipped that scenario a little bit and now it's just money that's involved, suddenly we started getting differing opinions. And I think if you put the students in that position, that's where I don't think I would wanna score their responses, but I think it is a useful teaching experience. And something like that where we're trying to teach something and almost expecting the students to not pick the right thing, that is one of those things where we have a duty to make sure we communicate that teaching experience afterwards. As part of the wrap up of the event or something like that, where everyone gets to see this is why we did this. This is what we wanted to teach. This is how we think you should handle that professionally because they're going to face that later in their careers. Cover one more scenario? Yeah. We've got about 10, 15 minutes left here, so. So do you like this one? No, Malera Triage. Malera Triage, okay. Yeah. Okay. You wanna do it, Dan? Okay, so this was a really interesting event. It was during our healthcare thing, we had some sponsors build part of the infrastructure and they, in this infrastructure, they planted malware and they had a breach and they didn't tell us about it as the competition organizers. And in the role that I was playing, I was playing the director of incident response. And basically I had multiple student teams bring this compromise to me at which point I didn't think we put this in the environment. So originally I didn't believe them. I said, you need to bring me evidence and convince me of this, otherwise you're kind of wasting my time. And specific teams, they didn't just bring me evidence. They had a convert, they showed me the evidence but they had a conversation that was just different than the way other teams handled it. They would show you the evidence, they sat down, they explained what they found and it was undeniable. It was like really a good experience for me because then I was like, holy crap, this is here. I gotta go tell every other team that found this that they're right. So the idea is how do you tell a client something when you have evidence and they're convinced you're wrong, but you're right. Just to get some feedback from those of you who do this, have you ran into a situation before where you found something the client doesn't believe you and how do you handle that? I mean, this is a regular thing that we deal with which is, well, we issue a report and it's, you didn't do that. Well, we did have the evidence for it. No, you couldn't have done that. We would have caught you. So, and this also goes back to like some of my firewall admin days where half of my job was fixing firewall problems and the other half was proving it wasn't a firewall problem. Yeah. And it's the same kind of thing. You can't just walk into client and tell them, no, believe me, I'm right. You're wrong, you're idiot because they're not gonna be a client anymore. Even if you're right. So I think we just provided the key there, taking it outside of the context of malware and just putting it in the context of providing a client with evidence that whatever you say you did, and how do you handle that when they're coming back and saying, no, you didn't do that. How do you handle that? I mean, for me, it seems pretty clear to me. I just, I documented and say, there you go. Here's your CEO's email. Yes, I did. Yeah, the team that had convinced me of this, like they basically sat me down in the room and they're like, yeah, check it out. It's like right here. And I was like, oh crap. And then they're like, this is the system it's on. Whereas every other team just told me, right? They were just like, yeah, you know, your own. There's malware in the environment. I'm like, where? You know what I mean? This team like, they're like, hey, come here, check this out, sit down. So I think this may go into the bigger role of, what is the actual role of the offensive team? Is it to document the results or is it to convince the company that their results are correct? Or rather that their results are fully actionable. So is their job to push change within the company or to show that there is a problem and that the company should enact change? I think it's another one where the answer is yes. I mean, from my perspective, I feel like it's our responsibility as consultants, because as pen testers, we are consultants. So it is our responsibility, at least in our work, for my part of the world to raise the bar and increase the maturity of my client and help them understand their risks. So if I am not doing that, I am doing them a disservice. And I have had multiple engagements over the years where a pen test has become incident response in that. I think that was the subtle difference here where the finding of the malware would trigger an incident response process that would kind of stop the pen test. I get what you're saying, though. Normally you would just put it in the report and you're just presenting the evidence and driving the changes up to the company to believe it or do something about it, right? But here, as the director of incident response, I said, if you find a compromise, let me know immediately so that we can do something, right? And basically, multiple people were like, hey, this is compromise and I just didn't believe them. I didn't have the evidence, et cetera, which happens all the time in IR, right? You get people saying, this is weird. This is funny. This is an alert. You dig into it with the things they give you and it might not be, right? I mean, I've had also situations. It's very situational, right? What do we put in the report really for later and what do we run into the client's room screaming and yelling, hey, you need to fix this? And it might be, I've had a client where they've messed up their gnats and every single port on every single host that they have in their DMZ is wide open to the internet. Yeah, we tell them about that pretty quick. But if it's some sort of inject on an internet accessible website that really doesn't get me to any information or data, then, well, that one is a tougher call. And maybe at that point, the question at least coming from the competition side would be, how much do you want to push the team to try and argue that their point is correct? So if you, as the organizers say, like, we're gonna make them argue that this thing is true and we're not gonna believe it for a while, so how far do you push them? Do you push them like a little to say like, oh, no, we don't believe it and then they come back with some evidence and okay, we're starting to believe this or is it like, we don't believe this? Come back with like your dictionary of everything that you come up with and. That's the line and it's not how much do they push us, it's how do they do it, right? And it was the elegance of the soft touch of sitting us down and showing us rather than pushing evidence on us and having it to be adversarial. And this comes down to language too, where you're using the word argue. There's different definitions of argue or nuances to that and argue as in, you're wrong, believe me, damn it, is not the argue we're looking for, but argue in the sense of providing evidence and helping the client logic and reach the conclusion that we're hoping to achieve, that is exactly what you're going with, Dan. And I think from a teachable perspective, because I think there was a component of that in your comment as well, I really feel like, so as everyone has said, I play the pain in the butt part of the client a lot or the aggressive portion or the angry bad cop, whatever you want to call it and there may be a push really hard, but then someone else from our team may come in and provide some support. So again, there's teaching, but then education. There's the expectation, here's a hard lesson and then someone immediately comes behind or 20, 30 minutes later to say, hey, we need to get this together, the boss was upset when he came in or hey, the engineer was upset, but let me talk to you about what we actually need. So there is certainly some lines and we push them in different ways and I'll be honest, a lot of times we are doing and making those decisions on the fly and that's why again, we don't really go into rooms alone, we're talking with people and then we actually do gut checks as we leave and we're walking from team to team to say, okay, we need to keep this consistent amongst all the teams. Now what do we need to do to go back to maybe change how we approach this for everybody and it's very dynamic how we do this and I think the competition and the process allows us to do that, which is cool, but we have to be very self aware in it as well. I think ultimately you're right, if you believe something, you should push for it and it's our responsibility to run it to ground. I think we have time, we could probably do a quick scenario. We got one more I think. Which one do you wanna do? Let's do the next one. Okay. I'll talk a little bit about this one. So one of the things, this isn't something that's actually occurred for us yet, but it's something we've been talking about a lot is, so we have these coaches, they're full-time employees, professors, engineers, administrators, we have a lot of different folks that are the representative from the university that comes with their team. You have to have an FTE come and join them from certainly a liability perspective, but we also then have this unique opportunity to have people and leaders from around the country and around these universities on site with us together. And so the first couple of years, we actually tried to do a little conference for them and have everybody, each one of the coaches brings something to present to the others and one year actually they just said can we just work? We're here, let us hunker down and just work. But what we actually did last year is we gave them access to the environment after part of the environment, basically the regional environment while the students were doing nationals. Team 11, coach team. Yep, we had team 11, the coach team. And they actually really enjoyed that, but we've had a lot of internal debate on from the perspective of a coach, are we creating an ethical dilemma on the coach where maybe they identify something and even though we are all professionals, they are now in the dilemma of can they inadvertently, maybe not maliciously, but inadvertently help their team as they're at dinner or as they're talking about things. So from our perspective, what we are going to do is we're gonna try and make it so that the coaches now get a year delay. So they will get last year, but I think there's some good conversation around even if it's accidental, what are the some of the ethical dilemmas in that? So first let's see, coach team, if they have an older environment, is that ethical or unethical to the competition? Pretty easy, hopefully. What if they have the same environment, ethical or unethical to the competition? Would somebody who marked ethical and somebody who marked unethical wanna talk? Let's start with ethical. Ethical or unethical? So what's the delay between you guys giving them the environment and then us actually competing? Let's say we gave them the same time you all started. All right, so I personally don't think it matters because when you go to write the report, you have to have support evidence anyway. And so by the time we finish our day, if even though they may know something, we still have to write the report and if we don't have evidence for it, it didn't happen. So, but we don't control, okay, I see your point. Yeah, so even let's say we go through our days or our coaches like a super guy, whatever, hacking, which is not true, by the way. Let's put it out there, for the record. Yeah, let's say he turns like a super hacker guy, we'll like three, one, three, seven, zero days all day. And he's happy. Don't forget, we're going to email this out now. That's fine. He hacks everything and he's like, all right guys, they got all the cheat codes, here's what happened, everything here. And he tells us the night of, we're writing the report. Well, it doesn't really matter because when we don't screenshot evidence, we can't prove that we did it. And we can't tell the client like, okay, we know there's a vulnerability in your environment. Sorry, we can't tell you how we found out about it or proof that we actually exploited it, but it's there. That's awesome. But what about if you have evidence of something but you don't have a great way to substantiate that? Or what do you mean? Like, what about evidence? You think like something and they don't substantiate it. Well, what if, no, let's just say you were competing. You found that there was something there but you couldn't necessarily explain why but your coach figured it all out. We found something and we couldn't explain why. Yeah, yeah, like there's evidence of something or you have a hunch, you have some screenshots that could be somewhat crafted to potentially indicate that. Well, then that would us being, let us that we be unethical for a while. You know, you can't have a, you can't write a hunch in a report. You can write like, we believe X, Y and Z happen. I can say that teams have definitely tried to write hunches. And they're not gonna get points. So the technical scoring happens with just a couple of us and it's those of us that are, not that we all don't do pentesting but there's a couple of us that are super hardcore ridiculous people and the scoring comes down to us and I can tell you there's a whole lot of hunches in those reports that we go through and none of them get scored. And so what happens a lot of times is the teams will get, see their scores and they see the technical, I'm like we had way more findings and we're like, yeah, but none of them counted because you didn't put any evidence in and it does not count as a finding if you don't have that evidence. Yeah, so yeah, I mean at least on our end, like we only put things that we have like 100% verifiable proof and screenshot for. Even there are things that we even redacted of our reports that we can't, we don't have enough supporting evidence for it. Thank you. Yeah, so like if you don't have the proof for it, it doesn't really matter because we can't put something that you believe happened in a report. It's not true and no one's gonna like that. I think we have like one minute. So we've got about one minute left. So I know they're, sorry, Alex, we need to. We can talk after. We can talk after, but so. There's a couple of real world lessons we wanted to talk through here. I think, you know, bringing this all together. The human element is very important. How we handle some of these disagreements and attitude are probably just as important for those of us in the real world as the way and how and the technical information on what we approach. More importantly, we as the educators of this competition are trying to teach people speaking skills interpersonal skills and so we've talked about this a lot, right? It's not a competition, except it is. And so this is something that we are constantly, constantly debating. I don't know if you guys have any other closing comments on this, otherwise. I really do wanna call out that seventh bullet specifically about remember who you're working for. We will see often teams will present and say, your IT staff sucks. And like for me, role playing the role of the customer and you're just telling me that I hired shitty people. You're fired. You know, we don't wanna see that sort of thing. And that is a very valuable learning experience to have. How do you approach those sorts of situations where yes, maybe your client, your customer isn't all that good and they have poor security practices, but how do you present that in a way that's appropriate for the audience and the management of those people that are making poor decisions? Just closing remarks, I wanted to thank everybody that came here. I got a ton of great notes that I'm going to incorporate back into our program. So thank you for your thoughts and your comments because I thought we had a great discussion and I'm going to digest them and actually apply them. Yeah, this was a phenomenal audience and we really appreciate your participation. So last couple of things here. If you wanna get involved, we'd love to have you. We need to build profiles. We need to write. We need to grade reports in the fall. We are in the process of building the 2019 infrastructure. We're in the process of planning the 2020 competition. So if you want to get involved, this is a call to action. If you wanna go to the next slide, be an ethical influence. This is a really final comment here, is be the influence that we wanna be. The whole purpose of this competition and this is our minus the little edit here is essentially our mission statement is to be an influencer and an educator to create the next generation of cybersecurity professionals. That is the goal of CPTC. Now we also need to be ethical about it and that is really what we wanna do. So we'd really like to hear from you on either ideas that you've got or if you'd like to get involved. For prior competitors, we love you the best. So you know what you've been through. If you are no longer a student in full time, we'd love to hear from you as well. But anybody in the audience or anyone that you know that you think might wanna get involved. We have regionals all over the country but we actually run the infrastructure and we run everything for them and we create all of that. So we need people in person. We certainly can always use money but we're actually pretty good there. Really right now we just need people to help us build. We're building a bank and you should see what people's reaction is when I say yeah, I'm just building a whole bank today. It's pretty fun. Yeah, we actually end up spending between probably the core group of 20 people or so about 10,000 hours a year building this. We can use people across the board. If you're technical, if you're non-technical, if you're a project manager, if you like marketing, whatever, we have a job for you. Yeah, the thing we were joking, Lucas and me, that the core group that runs a lot of this is smaller than a lot of the teams, which is insane. But without the support of everyone else who donates time and their talents and all that, we can never make it happen. So thank you all for coming. We really appreciate it and we'd love to talk to you so we'll be outside, I think, yes? And grab stickers before you leave. All right, thank you very much. Yes.