 All right, hello. I'm very excited and honored to introduce a panel that is in collaboration with the AI Village here today. I will host and moderator today will be Ram Shankar from Microsoft. Without any further ado, I'll pass it back to you, Ram. Hey, thank you, Omar. I'm super excited, you know, both the luminaries of the AI Village and the Red Team Village, where I think what may be the first joint panel in the space. So to set a little bit of context, AI Red teams are kind of mushrooming everywhere from Facebook, from Microsoft, to Nvidia, and even in the government. But there's a lot of, like, misconception about what it is. And we really don't, we really want to take this time to kind of like tease out how different it is from the regular Red Team. And most importantly, how can the security community and the machine learning community come together to kind of, like, deal with the screen? So I want to quickly introduce the panel. We have with us, we celebrated security guru, he's Naira, he's been thinking a lot about attacks on AI systems, you know, in the recently. So welcome, Bruce. We also have Omar Estanjos, who leads the Red Team Village. We really need to be thankful to him for all the wonderful CCFs that are happening, as well as Chris Sopral from Nvidia. He's kind of the Nvidia's AI Red Team head honcho. Thank you, Omar and Chris, from representing the Red Team Village. From the AI Village, we have Dr. Anita Nikolich, who's the director of research and technology innovation at the University of Illinois, Urbana-Champaign. And Dr. Anita is really going to ground us today on what is actually possible and what is actually happening in the cutting edge research from academia. And finally, we have with us our security data scientist, Rich Horan from Duo Sack, who's also now put out very exciting Twitter's bias bug bounty, which we will be talking about today. So, you know, but this, I really want to get quickly started and want all of you to hear from the experts. Bruce, I want to start with you. You know, you, you had a great talk for real to the keynote at the AI Village recently. You know, we spoke about how AI systems will eventually find loopholes at blazing speeds. And I really love your example with tax evasion. So, how do you think about, can you just elaborate about that for just a quick minute for people from the Red Team Village? And I really want you to kind of like touch on how you think about humans attacking AI systems. So, I've been watching AIs become hackers. It happened at DEF CON, I think 2016, when AIs had their own capture the flag contest. And more interestingly, there's a lot of research in AIs finding vulnerabilities in code. It's kind of the thing you'd expect AIs to be good at. It's a, it's pattern matching. There's a lot of data. It's a lot of, it's a lot of repetitive work. And they're not very good at it yet. They're going to get better at it. So, when we think about hacking and vulnerability finding, it's no longer going to be a human only creative endeavor, right? There are two parts to hacking. There's the creative part. Figuring out what the hack is, and there's the execution. And yes, we can automate the execution. Now that's pretty easy. But automating the finding the clever vulnerability, finding the exploit, making it work, that's going to become increasingly automated. So, when you look at hacking AI systems now, and you know, Ron, you all know that it's primarily human, it's done by humans. It's a human creative process. That's going to change probably the next few years, slowly and then quickly, like all of these things change. Oh, I think it's a great time to kind of bring Omar in. Omar, you've been like, you know, you have a wealth of experience in kind of like red teaming traditional systems. So, right now, hacking involves like humans. You know, like how Bruce pointed out, how do you see the space kind of like evolving? Yeah, I think that actually Bruce mentioned something extremely relevant, right? But before I go deeper into the red teaming of AI systems, let me actually define the two aspects of that word, or the phrase AI red teaming that you mentioned, right? So, one is about attacks against the environment. So, we're going to talk about that a little bit later. And the other one is what Bruce was actually mentioning, is actually using AI or machine learning to attack, you know, different platforms or to perform data manipulation. Because at the end of the day, a lot of the attacks that you're going to be seeing is data manipulation and poisoning of data, of training data, so that you will, you know, cause some damage into the NAI system. But, you know, going back to what Bruce mentioned, if you remember four to five years ago, DARPA had a competition. That's what Bruce mentioned here in DEF CON, right? It was called the DARPA Cyber Challenge. They had different teams that they created machine learning environments to do both things, to attack and emulate different type of attack methods, find vulnerabilities and then the other side of the coin, of course, protect against those vulnerabilities, right? So, they were trying to actually patch at the same time and defend based on the adversary behavior, right? Now, AI, you know, of course, machine learning will definitely pay a big role of the attacks in the future from manipulating people, right? The masses in social engineering, a type of tactics and so on, to learning about weaknesses and the underlying systems and learning how humans may also defend against those attacks and respond to those attacks. So remember that in a traditional incident response, you have the ability to detect whatever the threat actor is actually doing what we call the tactics, techniques and procedures. What if, you know, the attacker is actually able to learn what are the mitigations and the responses from the security team and then, you know, of course, evolving to that, right? And then I can think of, you know, things like anti-forensics capabilities, you know, being inserted into these environments and so on. Now, shifting back to the other concept, attacking the AI and ML systems, the first thing that the traditional, you know, is the traditional vulnerabilities against the underlying system itself. You're gonna continue to see additional capabilities. At the end of the day, you're actually using, you know, a proven technology nowadays for machine learning. However, one thing that you've seen a lot nowadays are companies using black box machine learning solutions that you have in the cloud and so on, right? And even though they actually try to sell you some machine learning, you know, thing, they're actually using somebody else's technology that is probably cloud-driven. They're sending a whole bunch of data to that system. What if you actually can manipulate that system and assign the whole, now not only attack one, but many, you know, different solutions, right? At the same time, data manipulation, if you can actually poison data, you can actually manipulate the results. So I work at Cisco, as you mentioned, we use machine learning solutions, of course, you know, and Rich here, you know, is from Duo as well. And one of the things that we actually do is that what if you can manipulate and poison network telemetry that could lead into different attack vectors or attack evasions, post negatives or unnecessary actions taken by an administrator or the security team, right? So all those aspects actually come into play from an adversarial perspective. So. I mean, I think that's a, I kind of like, I'm very excited to hear about like, which is point of view on this as well. Cause which you've been like building like machine learning systems, like forever for security now. So what is, it was done as an AI red teaming. How are we even propelling in this case? So I think Omar was right on point when he pointed out that there's an entire system involved in doing these, in doing these attacks, right? You're not just attacking only the ML, you're attacking the system that it's embedded in and you still have these same traditional vulnerabilities that you can go after. And I think what's maybe a little bit different about the AI space when you get into stuff that's specific to AI, it's sort of just, it's the same, really it's, there's new capabilities that exist, right? So you have like, you can attack it, you can do like training data extraction, you can attack it, you can do model stealing or something like that. But you have to do it when this model is in sort of an embedded system. And so we have academic research which says, oh yeah, these things are possible, right? We can do a black box attack and we can extract some of the training data or we can find if there was PPI that was used to train the model or something like that. But most of those techniques don't seem to have really made the leap into practice. And I think part of that is because we have this disconnect between treating the model as sort of part of a, treating the model on its own, which is kind of how it typically is handled in an academic setting versus as part of a complete system. So when you're talking to people that are building and deploying models in the industry right now, we are thinking about those things, right? We fuzz our feature extraction to make sure that there's no crashes there. We sanitize our inputs, we double check our telemetry and we keep an eye on telemetry to see if weird things are going. So all of sort of like the good software hygiene that comes along with deploying any application, ML or not, it's very similar. You have very strong parallels between this. Where we sort of fall over is we know that there are these other kinds of attacks that can be launched against AI driven systems. But right now it's all sort of in the academic space. And so I think this is where we're kind of hoping that red teams can lead the way, right? As they find which of these theoretical attacks are actually sort of practically achievable and practically deployable, that gives people who are invested in defending these models specifics, right? As opposed to just trying to prove a negative, right? Tell me that, you know, to me that this model cannot is not subject to a data extraction attack versus we've done this one now defending instead. Yeah, I really liked your point about like how a lot of these attacks are in this academic space. I want to kind of like get to Anita just real fast. But we have somebody with us who's actually leading an AI red team. Chris, you kind of like, you know, are leading this like NVIDIA's AI red team. You know, we just heard from like Rich that a lot of these attacks are still like in the academic phase and kind of like Bruce also kind of said, hey, you've got like humans going after like machine learning systems. There is no like AI behind it at this point. So what exactly do you do, Chris at NVIDIA? What exactly is like, you know, the NVIDIA AI red teams charter? Why did you create this team? And what exactly are you doing in terms of attacking ML systems? Sure. I'm super excited to be here, by the way. This is a new field for a lot of us. And we started it because it sounded cool. First and foremost, and second of all, because like specifically at NVIDIA, we have a lot of AI things. We have a lot of machine learning things. And like Omar and Mitch and you get a lot of these companies that have like black boxes, they take models, they ingest models from other places. We are actually creating some models on our own. So it behooves us to figure out how to do this stuff. Because not only are we consumers, we're also suppliers in some regards. So a lot of people came to me and my team and they were like, when are you gonna start doing AI red team? Or how do we hack an AI or how do we hack ML? And, you know, a year ago, we really can answer that question. And, you know, red teamers, we like challenges. And we just tried to do what we could and we started researching it. And a year later, you know, we've got leading into partnerships like these. And it's still very tabletop-ish right now, but we are getting to the point where we're gonna start doing some operations probably soon. Chris, can I follow up? You know, I think you mentioned the core of today's talk, which is like hack AI systems. What does that mean in practice? Like, you know, even if you're doing tabletop access, can you just like tell the red team members what an exercise would look like? Yeah, sure. So some of the things that we have scoped out so far, there's, to me, there's three main attack areas that we can go into is like when you're actually attacking an AI system, there's as the model's being trained, as like after the model's trained and after the model's deployed. So each of those three phases, you have different attack scenarios. And depending on what our customer was or how we wanted to do things, we would have to figure out what type of operation we wanted to do. Do we wanna poison the model? Do we wanna replicate the model? Do we want to, you know, after the model's been deployed, like are we gonna try to test systems to see if we can confuse it? So there's a lot to it. It's like a mile wide and an inch deep. So at least that's what I'm thinking so far. Like I said, we're pretty new to this too. Oh, thank you. I wanna pull that's for a little bit more, but at first one, Dr. Nicolich into their conversation. So you're hearing like, you know, Rich's comment about how a lot of these are like academic. Can you paint as a picture on what exactly are those types of attacks that academicians are kind of like thinking about on machine learning systems? And do they have actual real impact? Or is this more theoretical kind of exercise? So since I'm not at my day job, I can be a little down on AI and AI security and academia. So AI security is a new concept for academicians. AI gets a lot of federal funding, by the way. NSF just put another round of these AI institutes. None of them are focused on AI security, but the focus being on agriculture, chemicals, water, physics, science, if you can imagine nothing in there talks about the security of any of this AI. I think much like, you know, a few years ago, disinformation, that whole concept was like a big no-no for researching. But now we figured out it's important. I think the same thing is gonna happen with AI security. But to get back to your question, many of these attacks on AI are very esoteric and they are focused on different portions in the pipeline. My frustration is that AI is a pipeline, right? There's data collection and cleaning and ethics and system and software. And most of the academic attacks focus on one esoteric niche, not the entire pipeline. And I think that's a big problem. And can you also help us explain, like when people talk about AI security, is it like attacking AI systems? You mentioned misinformation, like GPT-3 to kind of like generate like disinformation. Then there's like what Bruce kind of alluded to in this awesome keynote about the coming of AI attackers. So what exactly is AI security and how are people perceiving that Anita? So I mean, AI security, we know that, you know, as more of these AI systems proliferate, we know that there's unanticipated behaviors. One of the places academia can focus is on how do we quantify, how do we identify these unanticipated behavior? How do we deal with the unknowns? And right now that's becoming really impossible. One thing I'll throw out that I think I've seen a little bit in academia and I wish I would see more, but many people are not keen on this, is to think about design thinking and futures thinking. And that really takes, you know, the whole pipeline to say, okay, there's this type of system. What's the best use of it? What's the worst use of it? What's the optimal use of it? How could it attack or do it? So really almost like, almost a table type exercise, but think it into future. And this is facilitated not by technical people like many of us, but by design thinkers to think how society can use this and then thinking about the pipeline so you could have the, you know, the more academic people think all along that way, where can I then attack it? Just one one suggestion. Absolutely, what do you mean by design thinking Anita? I'm sorry, can you just like explain that concept a little bit? Yeah, so designers, I mean, there's whole schools of design and one of the things they do is called futures thinking. And this is something that came out in the 70s. So you might imagine like this and I took a futures design class for fun and somebody did this, which is what they said, what if funding for public art was no longer? What would happen? Well, maybe a company like Amazon would make public art, but maybe, and this is part of the design thinking, you sit and think about these futures. Amazon funded public art, maybe it has to be a prime member to see the public art. Well, how would that work? And so the kind of storyboard to think all through and Ashken Sultani, who was an FTC CTO, had this great coin, this great phrase about abusability thinking. So how can we use, how can we abuse the AI? And I think if we think in the broader context, that's when you can really think about red teaming. I mean, much of red teaming is the physical aspect, the social engineering, the technical trashing of models, easy. It's everything around it. How do you get to it? How are you gonna poison this model? I think design thinking can help us. That's awesome because like, I have a question for Bruce, which I think just tailors on to the brass tacks aspect. Bruce, are we even prepared? If attacks on agencies are coming, are we prepared for this? You know, I think we're never prepared. Abusability thinking is a security mindset, right? How does it fail? How can it be made to fail? And if you think back to the things Omar was saying, all of that, those adversarial ways of thinking about AI machine learning, and we're never prepared because people don't think about security. I mean, you just point that the funding is for topics, for things the system does, not the things the system prevents. And we saw this. We saw this two days ago. Apple announced this system. They're gonna scan your iPhone looking for abuse images. You read there, they have a lot of security. They do not talk about the adversarial ML aspects at all. They do not assume an adversarial model. They do not assume data poisoning. They do not assume any of those things. And this is Apple. This isn't some bunch of idiots, right? So here again, and I've seen again and again, people design systems for functionality, for how they work. They don't design them for how they fail. And I like the phrase of usability thinking. I think of it as a security mindset. It is something we at DEF CON have been doing since forever. And it is the way we think of systems. And no, of course we're not ready because nobody's calling us when they're designing the systems. They're just designing them. And Bruce, for people who are just like, who are not exposed to adversarial ML, can you give us an example of what adversarial ML thinking in the Apple case would look like? Is there an example that comes to your mind? So there is an example. So image classifiers classify images. And they assume the images are like regular images. There's an entire class of research of making changes to images that the human eye don't detect that the image classifier deliberately fails. There is a, you can put stickers on stop signs, turn them into speed limit signs that the ML systems and cars will ignore. There are ways you can change, though the famous one is changing a turtle into a rifle. Just look up image classifier hacks and you'll see lots of them. And that's an easy example. That's not even a hard one. That is, I mean, I feel like, especially attacks on images are so captured people's imagination. And I kind of want to bring Rich in here. Rich, before we get into like, you know, the challenge, I want to kind of like, can you talk through the chaos that happened with Twitter's image cropping algorithm? So I should clarify that I don't work for Twitter. I have, so I was interested bystander in this. And yeah, essentially what happened was people began to notice. So Twitter has a cropping algorithm by which if you have a very large photo that you post into a tweet, it will actually try to reduce down to a narrow portion of that that can be shown in the complete stream of tweets. So that it doesn't, you know, one image doesn't blow up and dominate an entire, some of these entire people. What people noticed was that it was beginning, it was cropping in strange ways. So if you put a person of African descent next to a person, next to a white person, it might preferentially crop the white person. If you put a woman up, it very often tended to focus on your chest. And a lot of this was driven by the fact that what they did was they used a gaze tracking algorithm to train a saliency model to say what parts of the image would be most interesting to look at. Unfortunately, the population, I believe, skewed white and male. And so that drove the saliency algorithm, which then drove the cropping. So again, it's back to what Anita was talking about. It's this entire pipeline of decisions that you have to consider. So to their credit, Twitter did immediately address that. They released a blog post where they analyzed the results and they found that really, there was some small bias there, but what was happening was people were finding sort of the remarkable events and highlighting those because those had the most impact. And they've also been very, very open and transparent with this ethics bug bounty that they've launched in collaboration with the AI Village. And again, that's something that all credit goes to Ramon and Yuta and their teams for pushing that through Twitter. Really, we hosted it and we sort of helped them kick the tires and think through some of the issues with the bounty a little bit. But yeah, it's another illustration of how what seems like a series of pretty good ideas can actually lead to a machine learning classifier that can have an impact that actually upsets or even harms some people by cropping them out of photos where they really should have been the center of focus or highlighting what's of their anatomy when really what should be highlighted is what they did or things like that. So yeah, I guess does that answer the question? Yeah, I was, I really enjoyed the Twitter bug bounty because I think it's a concrete example of how an organization is trying to bring Anita's like Anita's framing of abusability thinking into this like space. I'd love for you to kind of like tell like people who want to get into this AI red teaming, do you need to know math to actually contest AI systems? Could you actually have already machine learning knowledge to work in this space? So I think it depends on how deep into the space you want to get. Like with any other pen testing, there is sort of different levels you can do it at. If you want to just do sort of high level, throw stuff at the wall, use known attacks, redeploy them. We're starting to see tools and frameworks come out. So you've got things like two-hand strike, counterfeit, clever Hans is another example. So frameworks that will actually let you execute attacks against NL systems yourself. And really you don't need much technical depth to be able to do that. Basically you need to be able to figure out how to run a model. Just to clarify for machine learning with, sorry, is that how it works? Counterfeit is, yeah, broadly, I think you can think of it as meta-sploit for machine learning. The other frameworks are a little bit, two-hand strike tries to sort of go in the same direction. Clever Hans, you need to get your hands a little bit dirtier. None of them require a PhD in mathematics or statistics to really get into. As you start to push the boundaries a little more and you want to think about developing your own attacks or you want to do stranger things with the models, then you begin, I think that's when you begin to need a little bit more of the specialized knowledge about how machine learning works, what sort of inputs and outputs you expect from them. But if you take a look, living it back to the Twitter ethics bug bounty, a lot of that was entirely data driven. All people needed to do was collect data that showed some sort of disparate impact or some sort of harm. Run it through the model to demonstrate that harm or that impact and you could produce essentially about its undesired behavior, even undesired behavior in the sense that the entire pipeline is working kind of as it's programmed to. It just gave you what you asked for and not what you really wanted. So again, there's entry points at all levels. I really do encourage people to get into this because this is a really fascinating space. But to really sort of push the boundaries of it, I don't think there's much of a way around getting your hands into at least a little bit of calculus. Yeah, I like your perspective of like, how should like red team members know machine learning? I think Omar, this is more like a question for you. You've seen like red team members kind of grow. What advice do you have for machine learning people who want to learn retaining skills? Like, is that required for in your perspective for building like AI pentestine? Yeah, that's an interesting question because let me define the red team part of it. So most people, whenever they mention about what they call red teaming is infiltrating a building, trying to impersonate somebody and then launching some exploits to attack a system. So it's a bigger scope pentesting. Now, what Bruce mentioned about people not even paying attention to security from the beginning when designing things, that's the number one thing that I'm gonna start highlighting. How you fix this is actually not by doing a pentest or a red team engagement after the fact, you fix it, what the boss worries, moving security to the left. At that moment that you're trying to design these systems, try to think about the adversarial methodologies that will apply, whether it's to data, as we were talking about before. And actually, Rich brought an amazing point related to ethics. And I think Anita touched on that. What you also gonna see is that a lot of people will concentrate, okay, I know security because I know how to do this fuzzing techniques or I know how to do this type of traditional penetration testing type activities. But in order to solve this problem, you also have to think about if I'm an attacker and I'm able to manipulate this type of behavior that then has collateral damage to ethics. Like we were talking about pixelation and manipulating of images. What if I actually can do that and then affect the background and the foreground of an image and then have face recognition network for certain people, right? It's things like that, just thinking outside of the traditional confinements of a pentest and then truly understanding one, how the technology is gonna be used. Second, how can it be abused? Then third one, what are gonna be the evasion techniques from an attacker perspective that then is trying to cover the tracks? How can I detect it? What happens if this is actually compromised? Because it's not so much about protecting. You're never gonna be able to protect everything 100%. So it's also how to react. What is it gonna be the way? One thing that I'm actually publicly discussing with many other entities is the disclosure of vulnerabilities, right? How are you also gonna be disclosing vulnerabilities in an AI system that has been manipulated? Especially something that can be potentially affecting many vendors, many implementations, many software out there. So that's something to also keep in mind. It's not only about the cool pentesting or red teaming methodologies. It's also the whole ecosystem, the design, the adversarial techniques, and the response. I really like that point because I wanna get to the vulnerability disclosure in just a moment. Anita, some of the tools that Rich mentioned, like Two Count Strike and Flabberhands actually has its roots in academia. And there seems to be considerable work happening in academia in this space, especially when it comes to tooling. Why is that? I think that it's easy to come up with these cute tools. Even when you get a grant or you have a study and you come up with these cute tools and then somebody comes up with a better tool and it's like this cat and mouse game, it escalates and then the next conference, they're saying I got a better tool. I don't think that's the best way to do it. I truly, and this is like, it's just like with regular security. I think truly like it's not just a buzzword. We need to think about multidisciplinary teams. I mean, I'm not a physicist, but I work with physicists on the whole end to end pipeline when they're trying to do AI for physics now with the Large Hadron Collider. I don't really think about physics, but I do know what could happen if someone walks in their unprotected data center with a USB drive. So I think like having more humanities technical and people who are not just machine learning math experts is really important. But to get back to your question, you get tenure based on your cute paper and your cute tool, not based on not harming people. Well, Bruce, should people like think about releasing these cute tools, like when people aren't even thinking about securing machine learning systems, or should they just fold back a little bit? And how systems don't even protect it? Like, releasing these cute tools can actually cause more harm. You know, so like we've been doing this at DEF CON for a bunch of decades. And we know by now that releasing the tools, doing the research, making it public, improve systems. So the ML people might not like it, but the car people didn't like it, and Microsoft didn't like it in the 90s, and no one likes it. But this is how we improve security. If we don't do the work, we don't release the tools, then the lousy stuff just stays in production. And this is a lesson that we can teach everybody else, instead of we in the DEF CON community, because we know this. We've been known it for decades. Kind of like, can you pull that turn a little bit first? Like, you kind of like, you know, mentioned even like a little, you know, for a couple of minutes to go, like how these systems are kind of left unguarded. So for somebody in the machine learning space, who's just like, you know, wrapping their head around security, can you give an example of how releasing a security tool actually led to improving the net security posture? Like, ML research was my thing, I thought I'd release this tool and buy them attackers. Why is that a bad thing? So it's a common belief that you're giving attackers ideas, you're giving attackers tools, attackers don't need ideas, attackers have to. Who doesn't have the tools are the defenders, are the non-security people doing the designing. Those are the most ignorant, you know, in this entire system. So that is, and also it spurs companies to action. Microsoft took security seriously because the community kept pushing them. Automobiles are like what, 10, 15 years later doing the same thing. And here's ML systems again behind. So we in security see this cycle again and again and again. And it is only by doing the research in public with disclosure that you actually get improvement. And you'll see that here in ML systems. This is gonna be the way it's gonna work. And we know it. It's almost like Bruce, especially for you and the rest of the veterans in this like chat. This is not new. Perhaps you all have seen this like multiple times. I see that Omar shaking his head and Bruce shaking his head and Rich smiling. Listen to what Chris said, right? She might not know the domain, but she knows security. Turns out that security knowledge is important and it transfers. And it doesn't matter if the computer is attached to a car or a refrigerator or a phone. It's a computer. We know how to attack and secure computers. It's software all the way down. Chris, I wanna kind of like, bring this question to you. What kind of tools? Like we've been speaking a little bit about tooling because we wanna talk about brass tacks of preparing for red teaming. What kind of tools do you use as part of your AI red teaming effort at NVIDIA? Like, can you just talk a little bit about that? Sure. That's a great question and there was a bunch of really awesome topics that represented like, literally right before you asked me this question. So it's an awesome segue. So there's been talks about like, what does it mean to be a red team for AI? And I think that's what Neil Maher brought up and Bruce was talking about, how releasing tools and things like that actually emboldens the security community and makes things better. So there's three main tools that we have started to use. One is counterfeit that Microsoft has released. That is, and I wholeheartedly agree that that basically is metaspolate for attacking AI systems. And that would be attacking an AI system after the model has been deployed. So that's one of the phases of attacking it. There's the second tool that I think what is great that we as NVIDIA released to bring the security community into the fold. You know, if people don't need math, like you need to talk about, like, you don't need math to attack certain aspects. And that tool is called Mint NV and that is a Docker container that you pull down. It's a boot to root. You attack it, the initial access is you have to circumvent a deployed AI model. You can use counterfeit with it, but that way you can, security people can go, they can pull this down, they can use the two tools and they can see what it looks like to actually attack like one of the phases of an AI system. And the other, the third tool is not really a tool. Unfortunately, it's a, it's a NIST publication. It's NIST IR 269, adversarial machine learning. Get a big old pot of coffee and just read it. And you only really have to read it once to kind of digest it. But as you're going through the taxonomy and terminology of like adversarial machine learning, you'll start to get an idea of like where you can attack these things and what tools would be applicable for each of these different phases. And the last thing that I would like to mention about tooling is to think about a concept of what tools you would need to read team an AI system like Omar was saying, what does that mean? I've started to come to the term that when you attack an AI, this is the first case where you're actually socially engineering a technology. So build your tools around, how do I trick? How do I socially engineer this technology? Hey, I know you mentioned, especially Rich and Chris, some tools. So can you treat it out from your handle at our Karang and at Ice Bear Friends after the end of this live panel, but people can have pointers for these tools. That'll be very helpful. Of course. Yeah, thank you. You know, I kind of like now want to switch gears a little bit and talk about the future of like this field. And Bruce, I'm going to come back to you and I want you to kind of like paint this picture for us. Based on your experience in this field, when do you think we'll have the next Stuxnet for AI systems? Like when are we going to have like APT using like half machine learning system? And honestly, vendors are going to like sell us on the solution. When is that going to happen, Bruce? So interesting. So you picked Stuxnet. Stuxnet is a very, very targeted hack. That was not a general hack that affected hundreds or thousands of systems. That was against one particular Iranian nuclear plant. And we'll have that when a government decides that that is an efficacious way of advancing their foreign policy. And it could be tomorrow and it could be years from now. And we're going to see criminal hacks when they become profitable. I mean, this follows the trajectory of these systems being deployed. As they are deployed in more places, you will see them used. So again, back to the Apple and image classifying and looking for child abuse material on your phone. So, we could expect to see hacks that will attempt to frame somebody. We will see hacks that will bypass the system. And we'll see hacks that just cause general mayhem. Then, I don't know. I mean, I think we're going to see the first papers on this in a year or two and this would go down to the criminals and governments when it does. And all this stuff flows downhill. The top of the NSA program is tomorrow's PhD thesis and the next day's hacker tool. It's hard to know when, but we know it's coming. And it's driven by how they're used. And when you mean by how they're used, like we already see like machine learning systems powering like healthcare, finance, and all these like important like fields. And we all of you, they're unguarded, why aren't you seeing a tax on machine learning systems more then? I'm not convinced we're not. I mean, the question is whether they make the news. I mean, so, you know, a proxy for whether things occur or whether we know about them. And that's an imperfect proxy. I think already doctors know how to code patient information in order to get the AIs at the insurance companies to produce the outcomes they want to make to improve the procedure. I mean, so I think there is that kind of adversarial ML going on right now. I think image classifiers and the stuff that kills people, you tend not to see that because a lot of people don't want to kill people. But just, you know, is annoying, right? The way Microsoft's Tay was turned into a misogynistic Nazi in 48 hours by 4chan, right? That was an adversarial ML attack. So we do see that. I don't think they're making the news because they're still under the radar and the systems aren't as widely deployed and understood. I feel that's an interesting point. I think, like, Chris, I'm bringing it back again because I think, like, for clearly, like, all these, like, flagship companies that, you know, Microsoft, NVIDIA, IBM, Twitter, Google, they're all putting, like, machine learning systems front and center of, like, you know, they're competitive, like, advantage. So all of these teams tend to have red teams, right? Like, NVIDIA, I'm sure, had, like, a vanilla red team before this. So based on what you're seeing, like, how can they invest in this space? Like, what are some of the organizational, like, challenges and opportunities at hand? No, it's a great point. And I've heard a lot of really good feedback from the panelists so far, and I think Anita nailed it. She said just getting in the room. That is gonna be, like, the best entry point for any red team. You have a red team, you have. If you're gonna be using AI systems or models or anything like that, just get them in the room. Get them at the table. It's not a technical upfront, but they know how to attack things. They know types of attacks. I have adversarial mindsets. Just having them in the room and being able to ask questions to scientists or policy makers will change the trajectory of whatever AI or ML system is being created. And if it's being, if a model is being created, you've now shifted how that model is gonna be secured. People are gonna start asking questions. So for entry points for places that are using AI technology or anything like that or making their own models, I would say do table tops first. And then the next phase after that, after you can start affecting policy would be try to start doing some of those more later stage AI attacks because that's more aligned with what I would consider traditional pen testing. You're gonna be bleeding APIs, you're gonna be pulling data, stuff like that. And as more familiarity gets accomplished with the red team, you can start moving into attacking the model, poisoning it as it's being created, replicating the model, how to attack those things, offline attacks. And you can just get more deeper with it. Oh, sorry, Chris, I didn't mean to interrupt you, but when you meant policy, what were you meaning governmental policy or like NVIDIA's like policy? Like can you just like clarify that? Sure, yeah, I meant policy like that, that stakeholders would implement. Like if we're gonna use an ML system, we're gonna have to do this. I think Rich talked about bias. That stuff is, I mean, that's incredible once you start reading about those things and there has to be policy around, you can't just take a AI system and just use it. Like there's gonna be inherent things that you need to account for. And being in the room and saying, hey, I think we need to account for this type of attack. Just being there does wonders. Like you went from 0% to 100%. So like when I meant policy, I meant stakeholders internally, how are you gonna use the models? How are they gonna be implemented and things like that? That is awesome, Chris. I wanna just pull that sort of just a little bit more. For somebody who's listening to this panel and they're super pumped, they wanna go think about like, think about like say even talking to stakeholders, what are like two or three questions that they can ask an ML developer as a security person to just get the balls rolling. Anything on top of your mind? Yeah, where'd you get the model from? Like is that our data or is it somebody else's? So it'd be like, where'd you get the model? Do we trust them? Is the data that goes that the model was used against? Is it ours or somebody else's? You start asking those questions and then you can start figuring out like where it come from and need to talk about pipelines attacks or like sometimes the pipeline is, like the model doesn't come from you. It comes from somebody else. So if the model was compromised way upstream, by the time you see you, you may not even know it's compromised. There may be a backdoor sitting in it. I don't know this, but like there may be a backdoor sitting in it that has made it through like five different channels that nobody knows about. So those would be my two biggest questions. It's like where the model come from? Where did the data come from? I wanna jump in on that just to emphasize data, data and data and where are you hosting, something like a simple S3 bucket that's probably a likely, more likely vector. Where is the data cabinet? How long are you keeping it? All these boring security compliance things which we go through and people don't like to go through it. Where's it backed up? Who has it? When are you gonna deaccession? Who has access to it? Those fundamentals that we know in security, first thing I'd ask. That's awesome. As I listened to what you're saying, Anita, and as I'm processing like Chris's information, and this Omar, I really wanna get your thought here. The questions that Chris and Anita are asking for in some levels, you must have been asking from like basic software for a long time. So using the questions that Chris and Anita kind of like just posed, how do you think the role of a traditional black team member is gonna change in five years with ML systems? Can you just talk a little bit about that? Yeah, if you go back in time, at least I guess I'm getting old, but whenever we started with router switches, embedded devices, a lot of people didn't know about that. It was this taboo thing. They concentrated in end host machines and probably windows. We saw a big gap in the talent, right? So on how to look and manipulate those type of systems. In some cases, it was because it was cost prohibited by some people, right? Putting a $2 million core router on the internet, that was pretty challenging. But we evolved, right? We looked into not only the red team part, but also the forensics. There's a huge lack this day of really good people that knows how to do forensics in these type of systems. Now, fast forward now with ML and AI, that's something that we have to think through, right? What are the skills that are necessary, not only from the red team perspective, I'm touching that in a second, but also in other areas of security to protect these systems, right? Again, going back, what if I'm already compromised? What are the things I'm gonna be doing from this ML system to see if, again, if an attacker is actually compromising this a year from ago? And same thing that you're gonna see, you're not gonna see the perfect candidate or the perfect person that will know all AI and all red teams, right? What Chris mentioned about being on the table, that's number one, right? At least having those conversations because it's an evolving thing, right? Second is you're gonna see specializations within these red teaming, pen testing, offensive security. I'm gonna generalize it about offensive security. Just like I have probably more knowledge on web applications and somebody else in embedded devices and somebody else in all the type of technologies, you will, traditionally, you will see that, right? Same thing goes with quantum computing, right? We haven't even got the kind of worms, right? Now, another thing that you're gonna be seeing is AI or NML is gonna augment the task of a red teamer slash offensive security person, right? So in that case, it's not so much of attacking those things. It's how can you use those systems to then do all the types of attacks, right? All the types of obfuscation, all the types of evasion techniques, all the types of manipulation, right? That does not assist today or probably not a scale, right? So it's the augmenting of the red teamer or the offensive security type person using these type of systems for some other manipulation. I mean, Omar, that was such a comprehensive answer. And especially if you're interested, you know, if you're listening to this panel or if you're interested in the last part of how like a red team can be like automated, you should really listen to Bruce's keynote that's gonna be, that's hosted in the AI village right now. It really does open your mind to AI hackers. So thank you for bringing that point, Omar. Feels like we're tying down a lot of like, tying down a lot of different points. But Rich, I also want to kind of like ask you this question. You're an ML, you've been an ML engineer since the titles didn't exist. So how do you see like, you know, the role of an ML engineer changing in the next five years? Then you have to think about actually securing these machine learning systems. I think we have sort of the good path and the bad path. So. Tell us both the paths. Okay, so the bad path is we do a thing and what we end up with is the systems which are deployed every month. We don't have a standard way of understanding what the vulnerabilities are in them or how to address them or what the best practices are. We haven't invested sufficiently in secure tooling for them. And as a result, it's the wild west like, you know, the internet was about 20 years ago and it's just, it's a complete disaster. I think the good path is we form these sort of multi-skilled, multi-disciplinary teams who can think about these things holistically, right? And there's, you know, just back to the data thing, right? There's a whole issue there with like forensics and data security and data privacy. If you have PII that's going personally identifiable information that's going into those models that in some jurisdictions is controlled, that's actually telemetry that you would need to do forensics on the model. So how can you handle that securely, right? You've got all of these questions that we've barely even started to tackle that we need to think about it to be able to essentially secure these models and defend them properly. So I think the good path is essentially the inverse of the bad path. We spent time thinking about these things, you know, with the help perhaps of red teamers. We've identified these are the classes of attacks that are feasible. These are how they usually happen in production systems and not just in sort of academic settings. And this is how we can then go about defending against them. And we're definitely seeing moves, I think, towards the good path. We've got NIST is seeking input on how to do like trustworthy and reliable AI computing. We have attack frameworks and the Atlas framework. From Microsoft and MITRE, I believe. So we're starting to sort of systematize and categorize these. What we don't have at the moment is, again, this transition from the academic space where we have all of these theoretical attacks and all of these theoretical vulnerabilities that are very specific to machine learning models that haven't made the jump into, oh yeah, that's actually happening all the time now. We have, as far as I know, we still today, since 2019, have exactly one CDE that has been filed in relationship to a machine learning model. And that was Will and Nick with the CDE against Group 9. And again, that's a good example because that require both a weird configuration in how the mail bounce was handled and the leaking of the scores for that attack to be effective. So again, it's the entire system that you have to consider. And so those kinds of things in five years, maybe we'll be like, oh yeah, obviously, we should write them. Obviously, we shouldn't leak raw scores. Obviously, we should do differential privacy on the inputs, stuff like this. Hopefully, it'll just be routine. Part of ML engineering, the same way that software engineers these days think about very routine security tasks that for a long time, nobody bothered with because we didn't have sort of the framework to think about them in a systematic manner. Yeah, I really like that point about not having frameworks and especially like how you're contrasting it with like how Chris mentioned the missed framework for like adversarial ML. So it feels like there is some piece of the puzzle but there's still like integration is still missing. Is that a fair statement, Rich? Yeah, I think so. I think there's two pieces that are missing. So the first piece is we sort of need to pull these different threads together sort of within the security community insofar as such a thing exists. Have an agreement that yeah, this is how we should think about vulnerabilities to machine learning research in machine learning models. I think the second thing we need to do though is actually focus on how do we transition these theoretical attacks. We have an unbelievable number of favors of all kinds of different attacks against machine learning models in the academic space. So in some sense, it's kind of scary because we know that there are these potential vulnerabilities against machine learning models. We could be leaking data. We could be having models stolen all the time. And it's sort of the question that Bruce brought up. Is it that it's happening that we don't know about it or is it just those, they haven't made the jump yet? And so I think what we really need to see is more collaboration sort of industry academic to make these transitions happen. So they could be like, okay, yeah, that's realistic. We really do need to worry about that. We need to think about how to protect against that. This other thing, maybe not so much. It's a cute trick, but it doesn't actually fly in reality. Yeah, I feel like one of Anita's point is really sticking to me is like how people get funding and they'd write cute papers and cute tools. So that's gonna be a big takeaway for me, Anita. I wanna like, I think this is a good closing question from at Mammon on Twitter. This is for, I think it was a great question for everybody in the panel. Just like a quick, if you can just take 30 seconds and point to one piece of information that people you think should know about AI redeeming. I know it's pretty generic, but if there's one key takeaway that you want people to take away from AI red teaming, what would it be? Anita, do you wanna start us off? I guess the key takeaway for me would be intended use and that AI red teaming is not, in my opinion, it's not a technical exercise. That's a great way to throw the ball at Chris because you are doing technical exercises. Well, and that is a great ball because my takeaway would be you don't have to be a data scientist to attack it. Mm, and we should pass the ball to the data scientist now. Rich? Well, Chris stole what I was gonna say, which was that the field is wide open. I think maybe I would just expand about on that a little bit. And I think I've maybe been arguing on it too much. We don't know what we don't know yet. The field really is wide open. I think tools are making it more accessible. There's so much free content out there to just get enough of an idea about how machine learning kinda works, get a sense for it, what the inputs and outputs look like. I really would encourage people to, you know, grab the docker niche that Chris was talking about or take a look at what was posted for the Twitter ethics bug bounty or take a look at some of these other ML problems and see where you can dive in and see what you can find, because there's a lot that's still out there to discover. So please get involved. And Rich, can I ask you to like tweet this out for my Twitter handle at harangue? Sure. Awesome, thank you. Omar, what is your key takeaway? I like the controversy between the non-technical versus technical, so I'm gonna be between the two. I think that is absolutely is a combination of what I think that is paramount instead of giving you just one resource is the call to action, right? It's because a lot of the things that we talked about before, even Bruce mentioned, some of these conversations are not taking place. And whenever you're creating technology, right? And so on. And as Rich mentioned, it's an evolving thing, right? So it's up to us in this panel and in this community, right, to come up with resources that, you know, the newcomers actually will take advantage of. And I think I'm really curious, you know, to actually collect all the different feedback, all the tools that Chris mentioned, all the research that Anita mentioned and everything, and probably putting together, probably the GitHub repository, you know, tweet it out and you know, whatever we want to do, but at least starting cooperating that, you know, I guess it's a call to action versus giving you the perfect solution. Thank you, Omar. And Bruce, bring us home, like, tell us from what we need to know about this field. Oh, Bruce, this is, I love it. It's 2021, yes. It is 2021, I know, I just feel it. And it feels like March 2020, you're all over again. I don't care what Anita said, that, right, think of the big picture. This is, do your threat modeling. This is not just technical, this is the entire system, this is the socio-technical system and the better you can threat model, the better you are on both sides of this. Fantastic. Thank you, Bruce. I, you know, first of all, a big thank you to all of you on the panel today. I know how busy we are supporting the panelists. I really appreciate all of you coming here. If you are listening to the panel, I strongly encourage you to go check out AI Village's Discord channel. They have an entire channel on attacking AI systems. And, you know, you can poke Anita and Rich there. And if you hang out at the Red Team Village, now we've got Omar and Chris also kind of guiding you there. So please make use of these resources. And Rich and Chris will be tweeting out some of these resources. So you should also go and look into that. So with that, thank you very much. I really appreciate all of your time today. Thank you, Ron. Thank you, this is great.