 Hey, thank you Omar. I'm super excited to post the luminaries of the AI village and the Red Team village for I think what may be the first joint panel in this space. So to set a little bit of context, AI red teams are kind of mushrooming everywhere from Facebook, Microsoft, Nvidia, and even in the government. But there's a lot of like misconceptions about what it is and we really want to take this time to kind of like tease out how different it is from regular red team. And most importantly, how can the security community and the machine learning community come together to kind of like a deal with this team. So I want to quickly introduce the panel. We have with us the celebrated security guru, he's been thinking a lot about attacks on AI systems, you know, in the recently, so welcome Bruce. We also have Omar Santos, who leads the Red Team village. We really need to be thankful to him for all the wonderful CCFs that are happening, as well as his support from Nvidia. He's kind of the Nvidia's AI red team head honcho. So thank you Omar and Chris for representing the Red Team village. From the AI village, we have Dr. Anita Nicolich, who's the director of research and technology innovation at the University of Illinois Urbana Champaign. And Dr. Anita is really going to ground us today on what is actually possible and what is actually happening in the cutting edge research from academia. And finally, we have with us a security data scientist, Rich Perang from DuoSec, who's also now put out very exciting Twitter's bias bug bounty, a bug bounty, which we will be talking about today. So, you know, but this, I really want to get quickly started and want all of you to hear from the experts. Bruce, I want to start with you. You know, you, you had a very thoughtful road to the keynote at the AI village recently. You know, we spoke about how AI systems will eventually find loopholes at ladies' feet. And I really love your example with tax evasion. So how do you think about, can you just elaborate about that for just a quick minute for people from the Red Team village? And I really want you to kind of like touch on how you think about humans attacking AI systems. So I've been watching AIs become hackers. It happened at DEFCON I think 2016 when AIs had their own capture the flag contest. And more interestingly, there's a lot of research in AIs finding vulnerabilities in code. It's kind of the thing you'd expect AIs to be good at. It's a, it's pattern matching. There's a lot of data. It's a lot of repetitive work. And they're not very good at it yet. They're going to get better at it. So when we think about hacking and vulnerability finding, it's no longer going to be a human only creative endeavor. There are two parts to hacking. There's the creative part, figuring out what the hack is, and there's the execution. And yes, we can automate the execution. Now that's pretty easy. But automating the finding the clever vulnerability, finding the exploit, making it work, that's going to become increasingly automated. So when you look at hacking AI systems now, and you know, Ron, you all know that it's primarily human, it's done by humans. It's a human creative process that's going to change probably the next few years slowly and then quickly, like all of these things change. Oh, I did a great time to kind of break Omar in. Omar, you've been like, you know, you have a wealth of experience in kind of like red teaming traditional systems. So right now hacking involves like human, you know, like how Bruce pointed out, how do you see the space kind of like evolving? Yeah, I think that actually Bruce mentioned something extremely relevant, right? But before, before I go deeper into the red teaming of AI systems, let me actually define the two aspects of that word, or the phrase AI red teaming that you mentioned, right? So one is about attacks against the environment. So we're going to talk about that a little bit later. And the other one is what Bruce was actually mentioning is actually using AI or machine learning to, to attack, you know, different platforms or to perform data manipulation. Because at the end of the day, a lot of the attacks that you're going to be seeing is data manipulation and poisoning of data, of training data, so that you will, you know, cause some damage into into the AI system. But you know, going, going back to what Bruce mentioned, if you remember four to five years ago, DARPA had a competition. That's what Bruce mentioned here in DEF CON, right? It was called the DARPA cyber challenge. They had different teams that they created machine learning environments to do both things. To attack and emulate different type of attack methods, find vulnerabilities, and then the other side of the coin, of course, protect, protect against those vulnerabilities, right? So there were, we're trying to actually patch at the same time and defend based on that behavior, the adversary behavior, right? Now AI, you know, of course, and machine learning will definitely be a big, big role of the attacks in the future from manipulating people, right? The masses in social engineering, a type of tactics and so on to learning about weaknesses and the underlying systems and learning how humans may also defend against those attacks and respond to those attacks. So remember that in a, in a traditional incident response, you know, you, you have the ability to, you know, detect whatever the threat actor is actually doing, what we call the tactics techniques and procedures. What if, you know, the attacker is actually able to learn what are the mitigations and the responses from the, from the, you know, security team, and then, you know, of course, evolve into that, right? And then I can think of, you know, things like anti-forensics capabilities, you know, being inserted into these environments and so on. Now shifting back to the other concept, attacking the AI and ML systems, the first thing that the traditional, you know, is the traditional vulnerabilities against the underlying system itself. You're going to continue to see capabilities. At the end of the day, you're actually using, you know, proven technology nowadays for machine learning. However, one thing that you've seen a lot nowadays are companies using black box machine learning solutions that you have in the cloud and so on, right? And even though they actually try to sell you some machine learning, you know, thing, they're actually using somebody else's technology that is probably cloud driven. They're sending a whole bunch of data to that system. What if you actually can manipulate that system and as a whole, now not only attack one, but many, you know, different solutions, right? At the same time, data manipulation, if you can actually poison data, you can actually manipulate the results. So I work at Cisco, as you mentioned, we use machine learning solutions, of course, you know, and the rich here, you know, is from Duo as well. And one of the things that we actually do is that what if you can manipulate and poison network telemetry that could lead into different attack vectors or attack evasions, post negatives or unnecessary actions taken by an administrator or the security team, right? So all those aspects actually come into play from an adversarial perspective. I mean, I think that's a, I kind of like, I'm very excited to hear about, like, Rich's point of view on this as well, because Rich, you've been like building like machine learning systems, like forever for security now. So what is, in the terms of AI reckoning, how are you even propelling in this case? So I think Omar was right on point when he pointed out that there's an entire system involved in doing these, in doing these attacks, right? You're not just attacking only the amount, you're attacking the system that it's embedded in. And you still have these same traditional vulnerabilities that you can go after. And I think what's maybe a little bit different about the AI space when you get into stuff that's specific to AI, it's sort of just, it's the same, really, it's there's new capabilities that exist, right? So you have, like, you can attack it, you can do like training data extraction, you can attack it, you can do model stealing or something like that. But you have to do it when this model is in sort of an embedded system. And so we have academic research which says, oh yeah, these things are possible, right? We can do a black box attack and we can extract some of the training data or we can, you know, find if there was, you know, PPI that was used to train the model or something like that. But most of those techniques don't seem to have really made the leap into practice. And I think part of that is because we have this disconnect between treating the model as sort of part of a, you know, treating the model on its own, which is kind of how it typically is handled in an academic setting versus as part of a complete system. So when you're talking to people that are building and deploying models in the industry right now, we are thinking about those things, right? We fuzz our feature extraction to make sure that there's no crashes there. We sanitize our inputs, we double check our telemetry and we keep an eye on telemetry to see if weird things are going. So all of sort of like the good software hygiene that comes along with deploying any application, ML or not, it's very similar. You have very strong parallels between this. Where we sort of fall over is we know that there are these other kinds of attacks that can be launched against AI driven systems. But right now it's all sort of in the academic space. And so I think this is where we're kind of hoping that red teams can lead the way, right? As they find which of these theoretical attacks are actually sort of practically achievable and practically deployable. That gives people who are invested in defending these models specifics, right? As opposed to just trying to prove a negative, right? Tell me that, you know, to me that this model cannot is not subject to a data extraction attack versus we've done this one now defend against it. Yeah, I really like your point about like how a lot of these attacks are in this academic space. I want to kind of like get to Anita just real fast. But we have somebody with us who's actually reading leading an AI red team. Chris, you kind of like, you know, are leading this like NVIDIA's AI red team. You know, we just heard from like Rich that a lot of these attacks are still like in the academic phase. And kind of like Bruce also kind of said, Hey, you've got like humans going after like machine learning systems, there is no like AI behind it at this point. So what exactly do you do, Chris at NVIDIA? What exactly is like, you know, the NVIDIA AI red teams charter? Why did you create this team? And what exactly are you doing in terms of attacking ML systems? Sure. I'm super excited to be here, by the way. This is a new field for a lot of us. And we started it because it sounded cool, first and foremost. And second of all, because like specifically out in video, we have a lot of AI things. We have a lot of machine learning things. And like Omar mentioned, you get a lot of these companies that have like black boxes, they take models, they ingest models from other places. We are actually creating some models on our own. So it behooves us to figure out how to do this stuff. Because not only are we consumers, we're also suppliers in some regards. So a lot of people came to me and my team, and they were like, when are you going to start doing AI red team? Or how do we hack an AI? Or how do we hack ML? And, you know, a year ago, we really can answer that question. And, you know, red teamers, we like challenges. And we just tried to do what we could. And we started researching it. And a year later, you know, we've we've gotten into partnerships like these. And it's still very tabletop-ish right now. But we are getting to the point where we're going to start doing some operations probably soon. Chris, can I follow up? You know, I think you mentioned the core of today's talk, which is like hack AI systems. What does that mean in practice? Like, you know, even if you're doing tabletop access, can you just, like, tell the red team members what an exercise would look like? Yeah, sure. So some of the things we have scoped out so far, there's, to me, there's three main attack areas that we can go into is like when you're actually attacking an AI system. There's, as the models being trained, as like after the models trained and after the models deployed. So each of those three phases, you have different attack scenarios. And depending on what our customer was, or how we wanted to do things, we would have to figure out what type of operation we wanted to do. Do we want to poison the model? Do we want to replicate the model? Do we want to, you know, after the model's been deployed, like, are we going to try to test systems to see if we can confuse it? So there's, there's, there's a lot to it. It's like a mile wide and an inch deep. So at least that's what I'm thinking so far. Like I said, we're pretty new to this too. Oh, thank you. I'm going to pull that for a little bit more. But I first want to also thank Dr. Nicolich into their conversation. So you're hearing like, you know, Rich's comment about how a lot of these are like academic. Can you paint us a picture on what exactly are those types of attacks that academicians are kind of like thinking about on machine learning systems? And do they have actual real impact? Or is this more theoretical kind of exercise? So since I'm not at my day job, I can be a little down on AI and AI security and academia. So AI security is a new concept for academicians. AI gets a lot of federal funding, by the way. NSF just put another round of these AI institutes. None of them are focused on AI security, but the focus being on agriculture, chemicals, water, physics, science, if you can imagine, nothing in there talks about the security of any of this AI. I think much like, you know, a few years ago, disinformation, that whole concept was like a big no-no for researching. But now we figured out it's important. I think the same thing is going to happen with AI security. But to get back to your question, many of these attacks on AI are very esoteric and they are focused on different portions in the pipeline. My frustration is that AI is a pipeline right through data collection and cleaning and ethics and system and software. And most of the academic attacks focus on one esoteric niche of that, not the entire pipeline. And I think that's that's a big problem. And can you also help us explain like when when people talk about AI security, is it like attacking AI systems? You mentioned misinformation like GPT-3 to kind of like generate like disinformation. Then there's like what Bruce kind of alluded to in this awesome keynote about the coming up AI attackers. So what exactly is AI security and how are people perceiving that Anita? So I mean, AI security, we know that, you know, as more of these AI systems proliferate, we know that there's unanticipated behaviors. One of the places academia can focus is on how do we quantify? How do we identify these unanticipated behaviors? How do we deal with the unknowns? And right now that's becoming really impossible. One thing I'll throw out that I think I've seen a little bit in academia and I wish I would see more, but many people are not keen on this is to think about design thinking and futures thinking. And that really takes, you know, the whole pipeline to say, okay, there's this type of system. What's the best use of it? What's the worst use of it? What's the optimal use of it? How could an attacker do it? So really almost like almost a table type exercise but think it in the future. And this is facilitated not by technical people like many of us, but by design thinkers to think how society can use this. And then thinking about the pipeline, then you could have the, you know, the more academic people think all along that way, where can I then attack it? Just one suggestion. Absolutely. What do you mean by design thinking, Anita? I'm sorry. Can you just like explain that concept a little bit? Yeah. So designers, I mean, there's whole schools of design. And one of the things they do is called futures thinking. This is something that came out in the 70s. So you might imagine like this. And I took a futures design class for fun and somebody did this, which is what they said, what if funding for public art was no longer? What would happen? Well, maybe a company like Amazon would make public art. But maybe, and this is part of the design thinking, you sit and think about these futures. If Amazon funded public art, maybe it has to be a prime member to see the public art. Well, how would that work? And so the kind of storyboard I think all through, and Ashken Sultani, who is an FTC CTO, had this great coin, this great phrase about abusability thinking. So how can we use, how can we abuse AI? And I think if we think in the broader context, that's when you can really think about red teaming. I mean, much of red teaming is the physical aspect, the social engineering, the technical trashing of models, easy. It's everything around it. How do you get to it? How are you going to poison this model? I think design thinking can help us. That's awesome. Because like, I have a question for Bruce, which I think just tailors on to the brass tacks aspect. Are we even prepared? If a tax on agencies are coming, are we prepared for this? You know, I think we're never prepared for abusability thinking is a security mindset, right? How does it fail? How can it be made to fail? And if you think back to the things Omar was saying, all of that, those adversarial ways of thinking about AI machine learning. And we're never prepared because people don't think about security. I mean, you just point that the funding is for topics, for things the system does, not the things the system prevents. And we saw this, we saw this two days ago, Apple announced this system and they're going to scan your iPhone looking for abuse images. Yeah. And you read there, they have a lot of security. They do not talk about the adversarial ML aspects at all. They do not assume an adversarial model. They do not assume data poisoning. They do not, they do not assume any of those things. And this is Apple. This isn't some bunch of idiots. Right. So here again, and I see it again and again, people design systems for functionality for how they work. They don't design it for how they fail. And I like the phrase of usability thinking, I think it is a security mindset. It is something we at DEF CON have been doing since forever. And it is the way we think of systems. And no, of course, we're not ready because nobody's calling us when they're designing the systems. They're just designing it. And Bruce, for people who are just like, you know, who are not exposed to adversarial ML, can you give us an example of what adversarial ML thinking in the Apple case would look like? Is there an example that comes to your mind? Can I use an example? So image classifiers classify images and they assume the images are like regular images. There's an entire class of research of making changes to images that the human eye don'ts and detect that the image classifier deliberately fails. There is a, you can put stickers on stop signs, turn them into speed limit signs that the ML systems and cars will ignore. There are ways you can change. The famous one is changing a turtle into a rifle. Just look up image classifier hacks and you'll see lots of them. And that's an easy example. That's not even a hard one. That is, I mean, I feel like especially attacks on images are captured people's imagination. And I kind of want to bring Rich in here. Rich, before we get into like, you know, the challenge, I want to kind of like, can you talk through the chaos that happened with Twitter's image cropping algorithm? Like, so I should clarify that I don't work for Twitter. I was I was an interested bystander in this. And yeah, essentially what happened was people began to notice. So Twitter has a cropping algorithm by which if you have a very large photo that you post into which week, it will actually try to reduce down to a narrow portion of that that can be shown in the complete stream of tweets. So that it doesn't, you know, one image doesn't blow up, you know, and dominate an entire in some of these entire people. What people noticed was that it was beginning, it was cropping in strange ways. So if you put a person of African descent next to a person next to a white person, it might preferentially crop the white person. If you put a woman up, it very often tended to focus on your chest. And a lot of this was driven by the fact that what they did was they used a gaze tracking algorithm to train a saliency model to say what parts of the image would be most interesting to look at. Unfortunately, the population, I believe, skewed white and male. And so that drove the saliency algorithm, which then drove the cropping. So again, it's back to what Anita was talking about. It's this entire pipeline of decisions that you have to consider. So to their credit, Twitter did immediately address that. They released a blog post where they analyzed the results and they found that really there was there was some small bias there. But what was happening was people were finding sort of the remarkable events and highlighting those because those had the most impact. And they've also been very, very open and transparent with this ethics bug bounty that they've launched in collaboration with the AI village. And again, that's that's something that all credit goes to the mom and Utah and their teams for pushing that through Twitter. Really, we hosted it and we sort of helped them kick the tires and think through some of the issues with the bounty a little bit. But yeah, it's it's another illustration of how what seems like a series of pretty good ideas can actually lead to a machine learning classifier that can have an impact but actually upsets or even harms some people by cropping them out of photos where they really should have been the central focus or highlighting bits of their anatomy when really what should be highlighted is what they did or things like that. So yeah, I guess does answer the question. I really enjoyed the Twitter bug bounty because I think it's a concrete example of how an organization is trying to bring Anita is like Anita screaming of abusability thinking into this like space. I'd love for you to kind of like tell like people who want to get into this AI red teaming. Do you need to know math to actually contest AI systems? Do you actually have already machine learning knowledge to work in this space? So I think it depends on how deep into the space you want to get like with any other pen testing there is sort of different levels you can do it at if you want to just do sort of high level throw stuff at the wall use known attacks redeploy them. We're starting to see tools and frameworks come out. So you've got things like two can strike counterfeit. Clever Hans is another example. So frameworks that will actually let you execute attacks against NO systems yourself and really you don't need much technical depth to be able to do that. Basically, you need to be able to figure out how to run a model. Metasploit for machine learning with sorry. Is that how it works? Counterfeit is yeah, broadly I think you can you can think of it as Metasploit for machine learning. The other frameworks are a little bit two can strike tries to sort of go in the same direction. Clever Hans, you need to get your hands a little bit dirtier. None of them require, you know, PhD in mathematics or statistics to really get into. As you start to push the boundaries a little more and you want to think about developing your own attacks or you want to do stranger things with the models, then you begin. I think that's when you begin to need a little bit more of the specialized knowledge about how machine learning works, what sort of inputs and outputs you expect from them. But if you take a look, you know, living it back to the Twitter ethics bug bounty, a lot of that was entirely data driven. All people needed to do was collect data that showed some sort of disparate impact or some sort of harm, run it through the model to demonstrate that harm or that that impact. And you could produce, you know, essentially about right, it's undesired behavior, even undesired behavior in the sense that the entire pipeline is working kind of as it's programmed to, it just gave you what you asked for, not what you really wanted. So again, any, you know, there's entry points at all levels. I really do encourage people to get into this because this is a really fascinating space. But to really sort of push the boundaries of it, I don't think there's much of a way around getting your hands into at least a little bit of calculus. Yeah, I like your perspective of like, how should like red team members know machine learning? I think Omar, this is more like a question for you. You've seen like red team members kind of grow. What advice do you have for machine learning people who want to learn red teaming skills? Like, is that required for, in your perspective, for building like AI pentestine? Yeah, that's an interesting question, because let me define the red team part of it. So most people, whenever they mention about what they call red teaming is, you know, infiltrating a building, trying to impersonate somebody and then launching some exploits to attack a system. So it's a bigger scope pentesting, right? Now, what Bruce mentioned about people not even paying attention to security from the beginning, when designing things, that's the number one thing that I'm going to start highlighting, right? How you fix this is actually not by doing a pentestor or a red team engagement after the fact. You fix it, you know, what the boss worries, you know, moving security to the left, right? At that moment that you're trying to design these systems, try to think about the adversarial methodologies, right? That will apply, right? Whether it's to data, as we were talking about before. And actually, Rich brought an amazing point related to ethics. And I think Anita touched on that. What you're also going to see is that a lot of people will concentrate, okay, I know security, because I know how to do this fuzzing technique. So I know how to do this type of traditional, you know, penetration testing, you know, typing activities. But in order to solve this problem, you also have to think about, if I'm an attacker, and I'm able to manipulate this type of behavior, that then has collateral damage to ethics, like what we're talking about, pixelation, and manipulating of images, what if I actually can do that, and then affect the background of the foreground of an image and then have face recognition network for certain people, right? Things like that, just thinking outside of the traditional confinements of a pentest, and then truly understanding one, how the technology is going to be used. Second, how can it be abused? And third one, what are going to be the evasion techniques from an attacker perspective, that then, you know, is trying to cover the tracks, how can I detect it? What happens if this is actually compromised? Because it's not so much about protecting, you're never going to be able to protect everything 100%. So it's also how to react. What is going to be the way? One thing that I'm actually publicly discussing with many other entities is the disclosure of vulnerabilities, right? How are you also going to be disclosing vulnerabilities in an AI system? It has to be manipulated, especially something that can be potentially affecting many vendors, many implementations, many software out there. So that's something to also keep in mind, it's not only about the cool pentesting or red teaming methodologies, it's also the whole ecosystem, the design, the adversarial techniques, and the response. I really like that point because, you know, I want to get to the vulnerability disclosure in just a moment. Anita, like some of the tools that Rich mentioned, like Two-Con Strike and Flaver Hands, actually has its roots in academia. Like, and there seems to be considerable work happening in academia in this space, especially when it comes to tooling. Why is that? I think that it's easy to come up with these cute tools. You get a grant, you have a study, and you come up with these cute tools, and then somebody comes up with a better tool, and it's like this cat and mouse game, it escalates, and then the next conference they're saying, I got a better tool. I don't think that's the best way to do it. I truly, and this was like, just like with regular security, I think truly like it's not just a buzzword, we need to think about multidisciplinary teams. I mean, you know, I'm not a physicist, but I work with physicists on the whole end-to-end pipeline when they're trying to do AI for physics now with the Large Hadron Collider. I don't really think about physics, but I do know what could happen if someone walks in their unprotected data center with the USB drive. So I think like, you know, having more humanities technical and people who are not just machine learning math experts is really important. But to get back to your question, you get tenure based on your cute paper and your cute tool, not based on not harming people. Yeah. Well, Bruce, should people like think about releasing these cute tools, like, you know, when people aren't even thinking about securing machine learning systems, or should they just fold back a little bit, you know, and also sometimes even protect it? Like, you know, releasing these cute tools can actually cause more harm? You know, so like, we've been doing this at DEF CON for a bunch of decades, and we know by now that releasing the tools, doing the research, making it public, improve systems. So the ML people might not like it, but the car people didn't like it, and Microsoft didn't like it in the 90s, and no one likes it. But this is how we improve security. If we don't do the work, we don't release the tools, then the lousy stuff just stays in production. And this is the lesson that we can teach everybody else, instead of we in the DEF CON community, because we know this, we've been known it for decades. Kind of like, can you pull that turn a little bit, Bruce, like, you could have like, you know, mentioned even like a little, you know, for a couple of minutes per go, like how these systems are kind of left unguarded. So for somebody in the machine learning space who's just like, you know, wrapping their head around security, can you give an example of how releasing a security tool actually went to improving the net security posture? Like ML researchers might think, oh, if I release this tool, I'm buying attackers. Why is that a bad, why is that a bad thing? So it's a common belief that you're giving attackers ideas, you're giving attackers tools, attackers don't need ideas, attackers have. Right. Who doesn't have the tools are the defenders, are the non security people doing the designing. Those are the most ignorant, you know, in this entire system. So that is, and also it spurs companies to action. Microsoft took security seriously because the community kept pushing them. Automobiles are like, what, 10, 15 years later doing the same thing. And here is ML systems again behind. So we in security see this cycle again and again and again. And it is only by doing the research in public with disclosure that you actually get improvement. And we'll see that here in ML systems. This is going to be the way it's going to work. And we know it. It's almost like Bruce, especially for, you know, for you and the rest of the veterans in this like chat, this is not new. Perhaps you all have seen this like multiple times. I feel like Omar's taking me aside and Bruce taking us by the rich mileage. Listen to what Chris said, right? She might not know the domain, but she knows security. Turns out that security knowledge is important and it transfers. And it doesn't matter if the computer is attached to a car or refrigerator or a phone. It's a computer. How to attack and secure computers. It's software all the way down. Chris, I want to kind of like, you know, bring this question to you. What kind of tools like we've been speaking a little bit about tooling because we want to talk about brass tax or red teaming. What kind of tools do you use as part of your AI red teaming effort at Nvidia? Like, can you just talk a little bit about that? Sure. That's a great question. And there was a bunch of really awesome topics that were presented like literally right before you asked me this question. So it's an awesome segue. So there's been talks about what does it mean to be a red team for AI? Omar brought up and Bruce was talking about how releasing tools and things like that actually emboldens the security community and makes things better. So there's three main tools that we have started to use. One is counterfeit that Microsoft has released. That is, and I co-heartedly agree that that basically is metaspoly for attacking AI systems and that would be attacking an AI system after the model has been deployed. So that's one of the phases of attacking it. There's the second tool that I think is great that we as Nvidia released to bring the security community into the fold. People don't need math. You don't need math to attack certain aspects. And that tool is called Mint NV. And that is a Docker container that you pull down. It's a boot to root. You attack it. The initial access is you have to circumvent a deployed AI model. You can use counterfeit with it. But that way you can, security people can go. They can pull this down. They can use the two tools and they can see what it looks like to actually attack like one of the phases of an AI system. And the third tool is not really a tool. Unfortunately, it's a NIST publication. It's NIST-IR-269, adversarial machine learning. Get a big ol' pot of coffee and just read it. And you only really have to read it once to kind of digest it. But as you're going through the taxonomy and terminology of adversarial machine learning, you'll start to get an idea of where you can attack these things and what tools would be applicable for each of these different phases. And the last thing that I would like to mention about tooling is to think about a concept of what tools you would need to red team an AI system like Omar was saying, what does that mean? I started to come to the term that when you attack an AI, this is the first case where you're actually socially engineering a technology. So build your tools around how do I trick, how do I socially engineer this technology? Hey, I know you mentioned, especially Rich and Chris, some tools. So can you treat it out from your handle at our Karang and at Ice Bear friend after the end of this live panel? But people can have pointers for these tools. I'll be very helpful. Of course. Yeah, thank you. You know, I kind of like now want to switch gears a little bit and talk about the future of like this field. And Bruce, I'm going to come back to you. And I want you to kind of like paint this picture for us. Based on your experience in this field, when do you think you'll have the next Stuxnet for AI systems? Like when is whenever we're going to have like, we'll be using like half machine learning system. And RSA vendors are going to like sell us on the solution. When is that going to happen? So interesting. So you pick Stuxnet Stuxnet is it was a very, very targeted hack. That was not a general, you know, general hack that that affected hundreds or thousands of system that was against one particular Iranian nuclear plant. And we'll have that when a government decides that that is an efficacious way of advancing their foreign policy. And it could be tomorrow and it could be years from now. And we're going to see criminal hacks when they become profitable. And I mean, this follows the trajectory of these systems being deployed. As they are deployed in more places, you will see them use. So again, back to the apple and image classifying and looking for child abuse material on your phone. So, you know, we could expect to see hacks that will attempt to frame somebody. We will see attempt we were going to see hacks that will bypass the system. And we'll see hacks that just, you know, cause general mayhem. Then I don't know, I mean, I think we're going to see the first papers on this in a year or two. And this would go down to the criminals and governments, you know, when it when it does and all this stuff flows downhill. The top of the NSA program is tomorrow's PhD thesis, the next day's hacker tool. It's hard to know when, but we know it's coming. And it's driven by how they're used. And what do you mean by how they're used? Like we always see like machine learning systems powering like healthcare, finance, and all these like important like fields. And we all of you, they're unguarded. Why are you seeing attacks on machine learning systems more than? I'm not convinced we're not. I mean, the question is whether they make the news. So, you know, a proxy for whether things occur or whether we know about them. And that's an imperfect proxy. I think already doctors know how to code patient information in order to get the AIs at the insurance companies to produce the outcomes they want to approve the procedure. I mean, so I think there is that kind of adversarial ML going on right now. I think image classifiers and the stuff that kills people, you tend not to see that because a lot of people don't want to kill people. You know, is annoying, right? The way Microsoft's Tay was turned into a misogynistic Nazi in 48 hours by 4chan. That was an adversarial ML attack. So we do see them. I don't think they're picking the news because they're still under the radar and the systems aren't as widely deployed and understood. I feel that's an interesting point. I think like Chris, I'm bringing it back again because I think like for clearly like all these like flagship companies that, you know, Microsoft, NVIDIA, IBM, Twitter, Google, they're all putting like machine learning systems front and center of like, you know, their competitive like advantage. So all of these teams tend to have red teams, right? Like NVIDIA I'm sure had like a vanilla red team before this. So based on what you're seeing, like how can they invest in the space? Like what are some of the organizational challenges and opportunities at hand? No, it's a great point. And I've heard a lot of really good feedback from the panelists so far and I think Anita nailed it. And she said just getting in the room, that is going to be like the best entry point for any red team. You have a red team, you have, if you're going to be using AI systems or models or anything like that, just get them in the room, get them at the table. It's not a technical upfront, but they know how to attack things. They know types of attacks that have adversarial mindsets. Just having them in the room and being able to ask questions to scientists or policy makers will change the trajectory of whatever AI or ML system is being created. And if it's being, if the model is being created, you've now shifted how that model is going to be secured. People are going to start asking questions. So for entry points for places that are using AI technology or anything like that or making their own models, I would say do tabletops first and then the next phase after that, after you can start affecting policy would be try to start doing some of those more later stage AI attacks because that's more aligned with what I would consider traditional pen testing. You're going to be bleeding APIs, you're going to be pulling data, stuff like that. And as more familiarity gets accomplished with the red team, you can start moving into attacking the model, poisoning it as it's being created, replicating the model, how to attack those things, offline attacks, and you can just get more deeper with it. Sorry, Chris. I didn't mean to interrupt you, but when you meant policy, were you meaning governmental policy or like Nvidia's policy? Can you just clarify that? Sure, yeah. I meant policy like that stakeholders would implement. If we're going to use an ML system, we're going to have to do this. I think Rich talked about the bias. That stuff is, I mean, that's incredible once you start reading about those things. And there has to be policy around, you can't just take a AI system and just use it. There's going to be inherent things you need to account for. And being in the room and saying, hey, I think we need to account for this type of attack. Just being there does wonders. You went from 0% to 100%. So when I met policy, I meant stakeholders internally. How are you going to use the models? How are they going to be implemented and things like that? That is awesome, Chris. I want to pull that just a little bit more. For somebody who's listening to this panel and they're super pumped, they want to go think about, say, even talking to stakeholders, what are two or three questions that they can ask in the ML developer as a security person to just get the ball rolling? Anything on top of your mind? Yeah, where'd you get the model from? Like, is that our data or is it somebody else's? So it would be like, where'd you get the model? Do we trust them? Is the data that goes that the model is used against? Is it ours or somebody else's? You start asking those questions and then you can start figuring out where it comes from. I need to talk about pipelines attacks. Well, sometimes the pipeline is like the model doesn't come from you. It comes from somebody else. So if the model was compromised way upstream, by the time it needs to you, you may not even know it's compromised. There may be a backdoor sitting in it. I don't know this, but there may be a backdoor sitting in it that has made it through like five different channels that nobody knows about. So those would be my two biggest questions. It's like, where'd the model come from? Where'd the data come from? I want to jump in on that just to emphasize data, data and data. And where are you hosting something like a simple S3 bucket that's probably a more likely vector. Where is the data kept? How long are you keeping it? All these boring security compliance things, which we go through and people don't like to go through it. Where's it backed up? Who has it? When are you going to deaccession? Who has access to it? Those fundamentals that we know in security? First thing I'd ask. That's awesome. As I listened to what you're saying, Anita, and as I'm processing like Chris's information, I really want to get your thought here. The questions that Chris and Anita are asking, in some levels, you must have been asking for basic software for a long time. So using the questions that Chris and Anita disposed, how do you think the role of a traditional vet team member is going to change in five years with ML systems? Can you just talk a little bit about that? Yeah, if you go back in time, at least I guess I'm getting old, but whenever we started with router switches, embedded devices, a lot of people didn't know about that. It was this taboo thing. They concentrated in n-host machines and probably Windows. We saw a big gap in the talent on how to look and manipulate those type of systems. In some cases, because it was cost prohibited by some people, putting a $2 million core router of the internet, that was pretty challenging. But we evolved. We looked into not only the red team part, but also the forensics. There's a huge lack to this day of really good people that knows how to do forensics in these type of systems. Now, fast forward now with ML and AI, that's something that we have to think through. What are the skills that are necessary, not only from the red team perspective? I'm touching that in a second, but also in other areas of security to protect these systems. Again, going back, what if I'm already compromised? What are the things I'm going to be doing from this ML system to see if, again, if an attacker is actually compromising this a year ago? Same thing that you're going to see. You're not going to see the perfect candidate or the perfect person that will know all AI and all red teams. What Chris mentioned about being on the table, that's number one. Having those conversations because it's an evolving thing. Second, you're not seeing specializations within these red teaming, pentesting, offensive security. I'm going to generalize it off of offensive security. Just like I have probably more knowledge on web applications and somebody else in embedded devices and somebody else in all the type of technologies, traditionally, you will see that. Same thing goes with quantum computing. We haven't even got the kind of worms. Another thing that you're going to be seeing is AI or NML is going to augment the task of a red teamer slash offensive security person. In that case, it's not so much of attacking those things. How can you use those systems to then do all the types of attacks? Do all the types of obfuscation, other types of evasion techniques, other types of manipulation that does not assist today or probably not a scale. There's the augmenting of the red teamer or the offensive security type person using these type of systems for some other manipulation. I mean, Omar, that was such a comprehensive answer. Especially if you're interested, if you're listening to this panel, if you're interested in the last part of how a red team can be automated, if you really listen to Bruce's keynote that's hosted in the AI village right now, it really does open your mind to AI hackers. Thank you for bringing that point, Omar. It feels like we're tying down a lot of different points. Rich, I also want to ask you this question. You've been an ML engineer since the title didn't exist. How do you see the role of an ML engineer changing in the next five years when you have to think about actually securing these machine learning systems? I think we have sort of the good path and the bad path. Tell us both the paths. Okay. The bad path is we do nothing. What we end up with is the systems which are deployed everywhere. We don't have a standard way of understanding what the vulnerabilities are in them or how to address them or what the best practices are. We haven't invested sufficiently in secure tooling for them. And as a result, it's the wild west like the internet was 20 years ago. And it's just a complete disaster. I think the good path is we form these sort of multi-skilled, multi-disciplinary teams who can think about these things holistically. And there's just back to the data thing. There's a whole issue there with forensics and data security and data privacy. If you have PII that's going for personally identifiable information that's going into those models that in some jurisdictions is controlled, that's actually telemetry that you would need to do forensics on the model. So how can you handle that securely? You've got all of these questions that we've barely even started to tackle that we need to think about to be able to essentially secure these models and defend them properly. So I think the good path is essentially the inverse of the bad path. We spent time thinking about these things, you know, with the help perhaps of red teamers. We've identified these are the classes of attacks that are feasible. These are how they usually happen in production systems and not just in sort of academic settings. And this is how we can then go about defending against them. And we're definitely seeing moves I think towards the good path. We've got NIST is seeking input on how to do trustworthy, reliable AI computing. We have attack frameworks and the Atlas framework from Microsoft and MITRE I believe. So we're starting to sort of systematize and categorize these. What we don't have at the moment is again this transition from the academic space where we have all of these theoretical attacks, one of these theoretical vulnerabilities that are very specific to machine learning models that haven't made the jump into oh yeah that's actually happening all the time. I mean we have, as far as I know, we still today since 2019 have exactly one CVE that has been filed in relationship to a machine learning model and that was Will and Nick with the CVE against Group 9. And again, that's a good example because that required both a weird configuration in how the mail bounce was handled and the leaking of the scores for that attack to be effective. So again, it's the entire system that you have to consider and so those kinds of things right in five years maybe will be like oh yeah obviously we should write on it obviously we shouldn't cross scores obviously we should do differential privacy on the inputs stuff like this hopefully it'll just be like routine part of ML engineering the same way that software engineers these days think about very routine security tasks that for a long time nobody bothered with because we didn't have sort of the framework to think about them in a systematic manner. Yeah I really like that point about not having frameworks and especially like how contrasting it with like how Chris mentioned the miss framework you know for like adversarial ML so it feels like there is some piece of the puzzle but there's still like integration is still missing is that a fair statement Rich? Yeah I think so I think I think there's two pieces that are missing so the first piece is we sort of need to pull these different threads together sort of within the security community in so far as such a thing exists have an agreement that yeah this is how we should think about vulnerabilities to machine learning research into machine in machine learning models I think the second thing we need to do though is actually focus on how we transition these theoretical attacks we have an unbelievable number of favors of all kinds of different attacks against machine learning models in the academic space so in some sense it's kind of scary because we know that there are these potential vulnerabilities against machine learning models we could be leaking data we could be you know having models stolen all the time and it's sort of the question that Bruce brought up is it that it's happening that we don't know about it or is it just those they haven't made the jump yeah and so I think what we really need to see is more collaboration sort of industry academic to make these transitions happen so that it can be like okay yeah that's that's realistic we really do need to worry about that we need to think about how to protect against that this other thing maybe not so much it's a cute trick but it doesn't actually fly in reality yeah I feel like I one of Anita's point is really sticking to me it's like how people get funding and they write cute papers and cute tools so it's going to like you know be a big takeaway for me Anita I want to like I think it's a good closing question from at Mammon on Twitter this is for I think it was a great question for everybody in the panel just like a quick if you can just take 30 seconds and point to one piece of information that people you think should know about AI red teaming I know it's pretty like you know pretty generic but there's one key takeaway that you want people to take away from you know AI red teaming what would it be Anita do you want to start us do you want to start start us off I guess the key takeaway for me would be intended use and that AI red teaming is not in my opinion it's not a technical exercise that's a great like you know way to throw the ball at Chris because you are doing technical exercises well and that is a great ball because my takeaway would be you don't have to be a data scientist to attack it and we should like pass the ball to the data scientist now rich well Chris stole what I was going to say which was that the field is the field is wide open I think maybe I would just expand about on that a little bit and I think like I've maybe been harbing on a bit too much we don't know what we don't know yet the field really is wide open I think tools are making it more accessible there's so much free content out there to just get it enough of an idea about how machine learning kind of works to get a sense for it what the inputs and outputs look like I really would encourage people to you know grab the docker image that Chris was talking about or take a look at what was posted for the twitter ethics bug bounty or take a look at some of these other ml problems and see where you can dive in and see what you can find because there's there's a lot that's still out there to discover so please get involved and rich can I ask you to like tweet this out from a twitter handle at her rank sure thank you Omar how do how what is your key takeaway I like the controversy between the non-technical versus technical so so I'm gonna be between the two I think that is a it's a absolutely it's a combination of what I what I think that is paramount instead of giving you just one resource is a call to action right is because a lot of the things that we talked about before even Bruce mentioned some of these conversations are not taking place and whenever you're creating technology right and so on and it's as Rich mentioned it's an evolving thing right so it's up to us in this panel and in this community right to to come up with resources that the you know the newcomers actually are will will take advantage of and I think I'm really curious you know to actually collect all the different feedback all the tools that Chris mentioned all the research that Anita mentioned and everything and probably putting together really a github repository you know tweet it out and you know whatever we want to do but at least starting cooperating that you know I guess it's a call to action versus giving you the perfect solution thank you Omar and Bruce bring us home like tell us tell us from what we need to know about this field oh Bruce uh this is this I love it it's 2021 yes it is 2021 I know it is and it feels like March 2020 all over again what Anita said that right think of a big picture this is do your threat modeling okay this is not just technical this this is at least is the entire system it's a socio-technical system and the better you can threat model the better you are on both sides of this fantastic thank you Bruce I you know first of all a big thank you to all of you on the panel today I know how busy the most important account is I really appreciate all of you coming here if you are listening to the panel I strongly encourage you to go check out AI villages discord channel they have an entire channel on attacking AI systems and you know you can you can poke Anita and Rich there and if you hang out at the red team village now we've got Omar and Chris also kind of guiding you there so please make use of these resources and a rich and Chris will be tweeting out some of these resources uh so you should also go a look into that so with that thank you very much I really appreciate all of your time today thank you raw