 Hello and welcome to Hacking with Skynet, how AI is empowering adversaries. My name is Gavin. I go by the pseudonym GT Klondike. I am a security researcher and senior security consultant. I'm very passionate about network security, both the attack and defense side. And through that passion, I run a project called NetSec Explained, which is a blog and YouTube channel where I explain intermediate and advanced level concepts in an easy to understand way. So machine learning, let's get everybody on the same page. What is machine learning? Well, first we have this big umbrella term of AI and parts of AI are machine learning, parts of AI are unrelated. So we have rule-based artificial intelligence, we have classical image recognition. But inside of AI, we have this small subset called machine learning and machine learning specifically allows us to take statistical analysis and draw inferences from data and perform data discovery techniques. And then out of machine learning, we also have deep learning, which is specifically deep neural networks. And this can be convolutional neural networks, which are used primarily for image recognition or recurrent neural networks, which are used for cyclical pattern recognition, such as a heartbeat monitor, right? So the burning question is, how is AI empowering adversaries or how can it empower adversaries moving forward? So I see the age of AI today or the age of machine learning today, we're at the very beginning of what it can do for the security industry on both the attack and the defense side. Now, if you remember back in the 80s and 90s, antivirus, intrusion detection, vulnerability scanners, these were all simple scripts that were written and then over time they grew into what are now multi-billion dollar industries. And so I think that we're at the very beginning of what AI and machine learning can do for the security community. So the way that I'm going to bring about this talk is really in two major parts. So the first is to talk about offensive AI tools. Why do we want these? What's out there already? What are we seeing in the wild? How do we build them? And then adversarial machine learning, which is attacking the machine learning algorithms themselves. So let's start with offensive AI tools. So the first question is, why would we even want these? Well, offensive AI tools and AI tools in general will allow us to operate at a much faster speed and a much larger scale than we're currently able to do. And this is going to allow us to automate a lot of manual tasks. Now, we don't need to think about it as automating everything, but if there's something that requires or manages or deals with a lot of data and these are things that we can apply a certain amount of fuzzy logic to, then this is a perfect candidate for automating with some form of AI or machine learning. And then of course, this allows us to dynamically and intelligently explore target attack surface and uncover hidden blind spots that come about some of the biases that we intrinsically have. And so being able to statistically analyze some of this data or perform some machine learning operations, this will allow us to kind of uncover some of those blind spots and look for things that we don't already know. And then how realistic is this? Well, back in 2016, DARPA kind of took on a large undertaking in their cyber grand challenge. In here, what they did was they required participants to develop some form of artificial intelligence that will perform a series of attack chain tasks. So in this case, it was going to be vulnerability discovery, exploitation, patching, and then data theft. So in the form of data theft, they have capture the flag. And then as part of their major grand challenge, they had attack defend capture the flag event that was air gapped. And you can see that in the picture kind of off to the side. Well, there is no human interaction whatsoever for the entire event and everything had to be automated. So this was a big reveal on a world stage that let us see what the power of artificial intelligence is, especially when it comes to a number of these tasks. And then where are we seeing applications of AI already? So the current proof of concepts we can see for social engineering, we can see in defense detection and evasion. There are some researchers that are identifying intrusion detection systems and sandbox detection systems so that attacks can be stealthier and break out of various systems and bypass existing defense measures. Evaluating data leaks, so large data sets such as password dumps, subdomains, things of that nature, websites, applications, configuration files, automated network exploitation. So this will be going from a standard user to domain admin automating that process as well as normal internal and external network pentests. And then software exploitation. And so we'll see this with AFL, which is a software fuzzer that will utilize its own custom compiler, put in certain key points in the software itself and then it allows the system to fuzz and track where through the application user input is being provided and how that can reach potentially vulnerable functions. So let's start with social engineering. Social engineering, there's a couple of things that are in the works right now that will allow us to improve our social engineering capabilities and improve attackers' abilities to exploit corporations and systems. So in this case, we have an NVIDIA project called StyleGAN. What it does is generate human faces. So all four of the images up at the top you see are actually generated by StyleGAN. And then just to kind of show you that there's no tricks up my sleeve, on the bottom row we have StyleGAN generated cats. And you can see here in the bottom right corner this abomination that kind of looks like a cat but is really messed up and difficult to see, difficult to really recognize for a human. And this is kind of the imperfect model generating a cat-like image. But the level of detail that these can go into, if you look at the cat over here on the left, just kind of sitting on the carpet, you can see the individual hair, the whiskers, the texture of the carpet underneath. And so these are really difficult to tell whether or not they are legitimate faces or cats or if these are machine learning generated. One of the telltale signs, if you look here in the top right corner, this gentleman, it seems like there's a picket fence behind him, but it's a little warped in the back. You also notice that all of the images have blurred background. And if you look really carefully, all of the eyes point directly at you. So it doesn't matter what direction they're faced, the eyes are pointing straight. So there's already some technologies and some research being performed to kind of identify what is a legitimate human face and what is a generated human face. Now, why is this important? How can we use this for social engineering? Well, for several types of social engineering campaigns, what's pretty common to do is go to the 10th page of a Google image search, grab somebody's stock photo or profile photo, and then you can paste that up as your own LinkedIn profile photo and say, okay, I am now Becky. I'm not a 32 year old male. My name is Becky. I'm 27. I'm starting my career and I wanna ask somebody, hey, as a fellow woman, how did you get started? And then I can use that to build a rapport, have my pretext and perform social engineering tasks to try and exploit employees, maybe in the HR department, sales, wherever. And so instead of grabbing that image and throwing it up as a pretext where an analyst can go in, grab that same image, do a reverse Google image search, see that it's a stock photo, call me out, realize that I'm fake, what I can do is use something like StyleGAN to generate my own face. So why steal a person when I can just make a whole new person? But there needs to be a little bit more legitimacy to my pretext. So this is where GPT2 and Grover come into play. GPT2 and Grover, they operate very similarly. These are automated generation of long form content. So these can be used to generate emails, blog posts, articles, anything of that nature with very minimal effort from the attacker, the adversary in that respect. And so unfortunately the website that was used to generate these has been taken offline, but the GPT2 model is still something that you can download, put on your own systems and generate long form content. So if I was developing a pretext, then I can use this to write phishing emails that could get past a spam filter or that look distinctly unique so that I'm not sending the same email to multiple people within an environment. Or I can use this to write blog posts and long form content and because any sort of Google image search for what is that called, plagiarism detection, it will see that I am the only source of truth. Therefore I must be a real person. Therefore my website must be legitimate. And so this will kind of add to the pretext. And then on top of that long form content, Google really enjoys. And so that can give us some Google ranking to add a little bit more credibility to our initial pretext. So this is one of the scarier ones that I see as being at the very beginning of the research and I'm curious to see where this will be in the next several years. We have Liarbird and Tacotron. Liarbird and Tacotron, these are text to speech that are based on human voices and use those human voices to generate conversation bits. And so we can use this for see in the phishing campaigns where we're actually calling people or leaving voicemails. So it's a lot less automated, sounding a little less robocall. And then you just add a little bit of compression and it sounds like you just have a really bad connection. So I have a couple audio clips. I want you to hear them. So these first two, I have the original voice and then the second one is the synthetic voice based off that original voice. So let's go ahead and take a listen. In another moment, down went Alice after it. Never once did she consider how in the world she was to get out again. Awesome, and then the synthetic voice built off of that sample. My voice might be generated by a computer, but I think it sounds pretty human. I don't know exactly how they made it, but I'm really impressed. So what I'm imagining is that as these models get better, what we can do is take interviews of CEOs and VPs of large corporations, grab their audio, isolate it, and then put it into a model like this so that we can have our own version of synthetic text to speech using their voice. So we can call throughout the company and impersonate these individuals in order to cause some sort of social engineering action. Now this last one is really interesting. This is Tacotron. This is a Google project and I have the text over here on the right, but what I want you to do while you're listening to this is pay attention to the breaths. As humans, we tend to have filler words like um and uh, and then in between sentences and commas, we usually have a bit of a pause or a breath. And so the Tacotron model actually picked that up. This is fully synthetic voice, just take a listen. Only the photograph really showed how much time had passed. 10 years ago, there had been lots of pictures of what looked like a large pink beach ball wearing different colored bonnets, but Dudley Dursley was no longer a baby and now the photograph showed a large blond boy riding his first bicycle. On a carousel at the fair, playing a computer game with his father being hugged and kissed by his mother, by his mother. That's pretty scary. So these projects are still underway. Liar Bird is actually advertising itself as a transcription or reverse transcription service, which is just text to speech in their original pay model that they had pushed out. Currently they're in closed beta, but in their original pay model that they pushed out, you can transcribe three hours of audio pretty cheaply. And then if you're so inclined to pay, like you can transcribe obviously more, but if a malicious user or somebody who's maliciously inclined, they can just grab those voice samples, throw them into Liar Bird or Tacotron or just use the voice samples that are already in there and generate their own adversarial examples. So next we get into sandbox detection. So this is the defense detection and evasion part of the system. So we have these researchers over at Silent Break Security and what they did was decide that with many different types of sandbox, what you can do is take the process list as data itself. And so based on the process list, the number of processes, the number of users on a system, you can generate and decide whether or not you're in a sandbox. And then you can take it a step further and decide what kind of sandbox you're in, whether you're in FireEye or Cuckoo Sandbox. So what the researchers did was kind of classify, they took a number of basic window systems and they took a number of sandboxes and just kind of identified how many processes there are, how many users there are. Let's take a look at what malware can identify. And so when we have a low process count, you see over here in column A, we have 33 and column F, we have 34. And then we have a high user count such as, again, A and F, we have four and four. And the process count per user, 8.25, 8.5, it gives us a very high probability that it is a sandbox. And so we can see that with our host total score. And then based off that total score, we can set a threshold. In this case, it looks like they set the host average, the host average score as their threshold. So anything below the average score is considered a sandbox. Anything above that is considered a legitimate system. And so they can have malware that completely shuts down or erases itself if it identifies that it's in a sandbox or it can use targeted evasion techniques per sandbox. So if I'm in FireEye, these are the things that I'll do to get past it. If I'm in Cuckoo, these are the things that I'll do to get past it. So it's a really interesting article. Highly recommend taking a look at that. And then for automated network exploitation, we have systems like DeepExploit and Goathon. Goathon and DeepExploit, they both operate very similarly. The way that these work is they will go out, they will perform an end map scan based on the results of that end map scan. If it sees common services that it knows are exploitable or things that it would like to kind of take to that next level. For example, poor 80 or 443, it's web traffic. So it's gonna go ahead and run scrappy to go and crawl that system. I'm sure they could add other tools such as like Go Buster, which will go out and enumerate a lot of the files or directories or URLs that are on that system. Try and fingerprint the server. You wanna see if it's running Nginx, Apache, ASPX, PHP, Java, things of that nature. And then it performs additional content exploration and it's on Google hacking, which is kind of cool. So that'll go out, do a little bit of research, see if there's any sort of exploitable scripts, things of that nature. And it takes all of this data and it processes it using some statistical techniques and then it uses Metasploit as a targeted exploitation engine and it goes and tries to exploit those things. So if it sees certain versions of JBoss or Jenkins, it'll go and try and target that with maybe password spray or any sort of Metasploit module, Tomcat, same thing. So this is kind of cool as a proof of concept of automating the entire process of network discovery all the way up to exploitation and in some cases, post-exploitation. One of my favorite while I was researching this is Death Star and this is my favorite just kind of in the way that it is assembled. So Death Star, it's built on PowerShell Empire and the way that it works is it starts, it kind of covers its baseline and it is designed to go from a standard user all the way up to domain administrator and so it charts and maps attack paths all the way to that location. And so it starts at its baseline and it says, am I already a domain administrator? If so, cancel out. We'll see that's the path off to the left. If I have to main administrator credentials, all right, cool, log in and that charts off the path to the right. But if it's neither of those then it goes straight down the middle and it says, okay, let's spawn a new agent, let's go ahead and identify where is the domain administrator and where is the domain controllers? And so it runs over here to the right. We can see get net domain controller, get net group member, invoke user hunter. And so this is going to enumerate the domain controllers, enumerate the domain admins and enumerate the users that it can on the systems that it can. And you can see it goes through this entire tree kind of off here to the bottom left. It identifies and say, hey, is this a vulnerable, like known vulnerable version of Windows? For example, Windows 7. If yes, let's run Mimicats, try and do Mimicats, SecretsDump, things of that nature, collect credentials and then we can kind of start back over at the top. Let's go ahead and enumerate the running processes. Do we see any processes that are already being ran by other users that we don't have in our inventory? If so, let's add those users to an inventory and try and do some lateral movement. And so in this case it'll use PSInject and then it'll go ahead and spawn a new instance. And so it goes through this iterative process all the way until it reaches domain admin credentials. And so if you've ever used a system like Bloodhound where you'll generate an attack graph and say, okay, this user is part of this group which has these members, which this member is also part of domain admins. And so this is the attack path that you wanna go in order to become domain admin. And so Death Star kind of takes that idea a step further and actually tries to navigate that attack path in an automated fashion. So that's all I have for the machine learning tools that already exist, some of the proof of concepts. There's a lot more information that I wanted to cover that I wouldn't be able to cover in such a short amount of time. So at the end of this presentation, I have a link to my slide deck which has a lot more detail and a lot more tools and a lot more techniques in it. But in the meantime, I kinda wanna answer the question of how do I go about creating my own machine learning tools or AI tools? What should I look for? How do I automate that? What is that process? And so honestly, my advice would be to identify somewhere that either utilizes a lot of data or identify somewhere that uses a, it's a common process, but it uses some sort of fuzzy logic, right? Something that's kind of hard to script, it's hard to write a signature for, but with AI and machine learning, you can use some of that fuzzy logic to generate some heuristics and it can perform based off of that. So in the case of some of these, what we can do is say, all right, end map scans, everybody does end map scans, and then you have to manually look at them and then based off of that, you have to follow up and go over here. Instead, like I showed with Goithon is you can just use the end map scan and then automate the process of, okay, let's enumerate and fingerprint these services and then let's see if any of these services are already exploitable using Metasploit or let's go with next steps. I identified a web service, let's run web analysis techniques. So Go Buster, Crawler, any sort of things like that. So that's kind of the thing that you wanna keep an eye on if you want to develop your own machine learning tools. And there's more that are coming out all of the time and it's really interesting and a really fast paced area and it's just ripe with research. So go ahead and play around with some of that. So the next bit that I wanna get into is adversarial machine learning and this is attacking the machine learning algorithms themselves. So adversarial machine learning, there's really three ways to go about attacking a machine learning model. So the first way we have is model evasion. This is to take what is clearly a malicious example and get the model to consider it to be benign or normal or standard to bypass that filter. So this would be taking a spam message and getting passed a spam filter or this would be taking a piece of malware and getting it passed antivirus or anti-malware solutions. Model poisoning is where we add our own data into the training dataset. Now, machine learning models specifically are, they're done learning as soon as they're done being trained. To get around this, there are some models that are called online systems and so what they'll do is they'll take data that has been classified by itself already. So say I send it a piece of data, like a piece of malware, it looks at that says, okay, bam, that's malware. Now that goes into my training set and I learn from that malware. So data poisoning allows us to slowly inject certain pieces of data so that we push that threshold a little off to the side into what is more acceptable and then we target where that threshold has been pushed. And we'll see an example of that in a bit. Data leakage, data leakage is actually a combination of two smaller bits. So one is stealing the model itself. So for example, say you're an IoT engineering company and you develop an IoT system that utilizes some form of machine learning. What I can do is buy that device and then reverse engineer it and now I have your model. And so I can use that model or train another model based off of that instead of having to go through the hundreds of thousands of dollars that you went through to develop that in the first place. And so this is where the model itself is easy to be grabbed. There's also data leakage, which is I can identify what data you use to train the model. And so this is really damaging, especially for healthcare industry or HIPAA, HIPAA protected information. So if I can identify any sort of clients or patients that you have, any sort of their PII then that's really damaging. So when it comes to model testing, there's really two approaches. You have a white box where you know what the model is, how the hyperparameters are set, everything that's involved in the building of this and then you target that model, much like you would do in software where you already have all of the code involved. And then you have a black box, which is just the model itself. You aren't able to peek into the way that the model operates, but what you can do is treat it like an oracle. So if you've ever done any sort of cryptography exercises, you have an oracle, you ask yes, no questions or things of that nature, and then you can generate what is called a surrogate model. So based off of the yes, no questions that I asked the black box model, I generated my own model and then based off of that model, I can perform attacks and those attacks should be able to impact the original black box itself. Now, the way that this operates is under a principle called attack transferability. This is a documented phenomenon and the way that attack transferability works is it doesn't matter what model or the underlying architecture that these models are designed with, an attack on one model can also be an attack on another model. So if I have a deep learning model over here and I find adversarial examples that I can use to attack this model, then here is a random forest of logistic regression, SVM or anything else, any other machine learning algorithm or architecture that those same adversarial examples will be able to impact that. And then of course the attack surface. So where are the machine learning models themselves vulnerable? So machine learning as it works is you have a physical object and then you take that physical object and you try to digitize it. So say you have a picture of a panda or a car, you take a photograph of it, now it's digitized and you can make information off of that. You take that digitized information and you run your pre-processing as you would per the normal machine learning process and then you put that into the machine learning model as part of its input features. It presents some output and usually a probability or a number of some sort. And then a decision is made based off of that action. So for example, if I am 67% confident that this is malware then I will probably flag it as malware. If I'm only 42% confident then I'll just let it continue to operate. So where can we attack things? Well, one, we can actually impact the physical object itself. So down below I have the network intrusion detection system as kind of an example that follows this generic machine learning model path. So we have the attack traffic. We can actually, as attackers, modify the attack traffic in such a way that as it goes through the pipeline of the machine learning pipeline, it will modify the behaviors at different points. So I can slow down my attack traffic, I can use my attack traffic for multiple hosts so that it's a little bit harder to detect. And then that attack traffic is then digitized. The digitization process, we don't really have a lot of impact over as attackers. It's not realistic for us to be able to affect TCP dump. There's really famous examples. For example, you may have come across this if you ever looked at adversarial machine learning, a picture of a panda and you modify some pixels and now it thinks it's a gibbon or a bird or some sort. But if we had the ability to modify pixels, why don't we just give it a picture of a bird at that point? And then more to the point, if we could modify pictures, if you have something like a self-driving car, it has a camera, how do we interface with that camera in an adversarial perspective? We would already have to have access to the system, which at that point we can just shut it down or do whatever, it's physical access. So we're not actually able to interface with the digitization of the object, but what we can do is change the way that the attack traffic is modified. We can change the input features, the input vectors, the machine learning model, especially with the model poisoning. We can change the way that the data is represented. And then so in the case of the network intrusion detection system, we have the packet metadata and then it spits out an attack probability. And then off of that, it makes a decision. And in between the machine learning model and the decision-making process, you gotta remember, this is just software and you can approach it like any other piece of software that you're pen testing or trying to hack. So you have the machine learning model in it and then all wrapped around it is a nice piece of software that probably has some of its own vulnerabilities involved. So model invasion, model evasion, not invasion, sorry. So we're gonna try and hide in the blind spots. Now we have this massive theoretical space. This is all of the possible pieces of malware that could possibly exist, but we can't collect all of that. Realistically speaking, antivirus companies can't collect all of it. And so what they have is a subset and this is considered the training space. So this is what they are realistically able to collect. So in their training space, they perform their training and then off of a small piece of that, that's already clean labeled data. They perform their testing to make sure that the model operates properly. Well, part of that testing space kind of right there off the sliver, this is the adversarial space. This is where we live. This is where we are able to generate adversarial examples that target that model specifically. And so this is kind of what we wanna play around with and hide in the blind spots, so to speak. So a couple of really good examples is in this case, multinomial naive Bayesian spam filters or just any of the early spam filters that used some rudimentary forms of machine learning. How they would operate is they would look at keywords. So if you had something like buy Viagra now, the probability that the word Viagra shows up in a spam email is much higher than the probability of the word Viagra showing up in a regular email. And so based off of these keywords, it does its own statistical analysis and says, I am 99% sure this is spam. Well, the interesting thing about emails is that emails are usually represented as HTML pages or web pages. And so as malicious actors, what we can do is actually add in a comment. So in this case, we can add in the entire Wikipedia page article for a horse. And so we have a bunch of bad words but we have even more good words. And what this lets us do is kind of skip past the spam filter by showing, hey, we have a lot more good words and bad words so we must be good, right? So it says, yeah, absolutely. You are, I am 99% sure you are not spam. And the thing is that the way that this is presented to users is as an HTML webpage. And so the comment itself is not actually rendered for the users. So the users still see the original spam email as though it was unmodified. And so this is kind of a real-world example on some of the spam filters. A really good example of this is by Skylight Cyber, the researchers over there were actually able to bypass the Silenced Machine Learning Antivirus in this article, Silenced, I Kill You. Really good article. Highly recommend you take a look at it. But we can see some familiar malware samples here. So we have CoinMiner, Emotet, Zeus. And we see the original scores that they were actually able to pull out of the Silenced Antivirus. Negative 826, negative 923, negative 997. So these are really bad, really bad, flagged immediately. So what the researchers over at Skylight Cyber did was they took the Silenced Antivirus, they reverse engineered it, and they were able to pull out the scores based on some of the syscalls and other key features that Silenced looked at for malware. One of the curious things is that they identified certain things that were whitelisted. Now, why would Silenced whitelist certain things, even though the standard machine learning process would actually identify it as potentially malware? Well, the reason why is because when you think about malware, think about how malware operates, right? So maybe it asks for access to your webcam or your microphone or it tries to resize or open up new windows. Maybe it makes calls out to the internet. Well, what's another thing that does that? Chrome, Firefox, Internet Explorer, they operate very similar to some modern malware. In this case, what they found was a video game. And so I'm not sure which video game it is. My money is on Fortnite or League of Legends. But the video games that they found were certain whitelisted keywords. So what they did was took these clearly bad malware and they just added these keywords to the end of the malware. So it didn't change the way that the malware operated but it changed the way that Silenced looked at it and said, okay, you have a bunch of bad boy points but you have even more good boy points. So we're gonna give you some good boy scores. So in this case, coin miner had a 884. My favorite is looking at Zeus negative 997, one of the worst in this table all the way up to positive 997 which is one of the best in this table. And so it shows that model invasion techniques are very useful in a number of different ways. So how do we defend against this? Well, the first and probably one of the better is adversarial training. Adversarial training kind of comes from the idea of chaos engineering. Netflix is a really big proponent of chaos engineering where you already know something bad is going to happen. So you might as well account for those bad things to happen. So in this case, you train it with adversarial examples. You generate your own adversarial examples and you train the model based off of that to kind of harden and make it a little bit more robust. Again, if it's not a robust model, it's not a good model. Chaos engineering as a whole, the way that it operates is you already know something's gonna go bad. So you have Netflix, for example, Netflix. They know that their systems, they need to have that uptime. And so they know that it's possible when one of their systems goes down, they wanna make sure their redundancy operates properly. So you purposefully go into production and you break things. And then you develop processes and policies around knowing things are going to break. So in some cases, you'll have the CIO walk into a data center and just start unplugging servers. So adversarial training is a really good way to kind of counteract this. Another slightly more technical way is called defensive distillation. Defensive distillation is a smoothing algorithm that kind of smooths the curve. So instead of having, okay, malware, malware, malware, nope, it's okay, you can't have sharp edges like that. So what it does is it smooths it. So instead of going straight down, it just kind of goes in a downward trend. And so it's a little bit harder, a little bit more robust way of analyzing and dealing with the adversarial examples. And then the last example would be monotonic classification. Monotonic classification just means that the graph grows in one direction. So we see here in the image off to the right, the non-monotonic example, we see that it's getting worse and worse and worse as far as a key word. So it goes worse and worse and worse across that threshold. Now it's considered bad, it's spam, it's malware, it's whatever. We're ready to throw it out, but we're still analyzing it. And so it's crossed that threshold, but wait, no, it's getting better and better and better. And then we just hit that pad of key words that are really positive. And so we went from good to bad all the way back up to good. Monotonic classification means that we don't count the good words at all. We say we measure your badness. How many bad words do you have? And so in this case on the bottom example, we have the monotonic classification, which says, okay, we have some bad words, we have some bad words, up we all went all the way up to the top. And then we hit all of the padded good words and it doesn't matter, you're still bad, you're Jesus is not gonna save you, you're done. And so monotonic classification is a really powerful technique to kind of deal with these model evasion techniques. Next we have model poisoning. Model poisoning almost exclusively works on online learning systems. So as adversaries, we need to find a way to get adversarial examples into the training space. And so this is where systems will grab data that it's already classified a certain way and add it into the training space. And so we're just trying to push that barrier past a certain threshold where we can actually operate safely as adversaries without it detecting us as such. So some real world examples, TAI, if you're not already familiar, highly recommend checking it out. TAI was a Microsoft Twitter chatbot and it was designed to learn from real conversations by real people. Well, 4chan, of course, gets ahold of this and they decide how much does it learn? How much can we teach it? And about 16 hours after coming online, it was taken offline permanently because within that time, 4chan was able to get it to say Hitler did nothing wrong and racial slurs. If you want more information on TAI and I highly recommend checking this out, it's called The People's Chatbot by the Internet Historian on YouTube. Really funny guy, really great story and it's just incredible to think of how many data scientists and developers and people behind this project put time in, put effort, put work and it was shut down by a bunch of kids on 4chan. The other one is Jacobian map saliency attack and this kind of goes into the pixel manipulation. So what you do is you grab a couple of pixels and you modify them or you tweak them a little bit or you tweak the data a little bit, doesn't necessarily need to be image data. And you pass that on as a certain class. And so as the model is learning from this data, learning from this class, it starts identifying these tweaks and then you can implement those tweaks in an adversarial example. So in this case, we have a 70 kilometer per hour sign, we add in these little tweaks and it latches on to those tweaks and says, okay, you're a 30 kilometer sign, 30 kilometer per hour sign. And so imagine the kind of chaos that that can create. Again, these tweaks are difficult to implement in real world examples. This is also difficult to pull off simply because it relies on a online learning system, something that learns from the data that you've already presented it. But these are things that researchers have been able to identify and actually use to attack machine learning systems. So how do we defend against poisoning attacks? Basically it comes down to, you can still use a lot of the data, but what you wanna do is add your own little smoothing filter. So in this case, have longer periods of time between retraining. So what that will do is kind of narrow the window or not narrow the window, it will expand the window that an attacker would need to generate adversarial examples for. And then they would also have to compete for votes against legitimate traffic. So it's kind of like, if you look at Bitcoin, you need 51% of the vote in order to kind of take over cryptocurrency networks. So this is kind of their own example of doing this. You can also analyze longer periods of data. Again, it broadens the window that you're looking at. And so the adversaries of the attackers are gonna have to generate more traffic and more examples in a longer period of time, which is difficult to do. And then you just wanna generally minimize the impact of adversarial training examples. So whether you have a human in loop, that's actually, that's manually looking at some of these examples or maybe you're taking a small sample of the stuff that you're seeing and only allowing that to affect the model very a little bit. So it's just kind of a general rule of thumb. It's just minimize the impact, come to the understanding that people will try and poison the well account for that. And then of course we have data leakage. So data leakage, this is usually when, if we're trying to steal data, not steal the model, but steal data itself, this is usually when the models are trained to be a little too good. So this image down below, you see aerial photographs, generated map, and then an aerial reconstruction. And so articles at the time, this is a project by Google, articles at the time were kind of grandstanding and saying, oh my, oh my God, the Google AI found a way to cheat and generate aerial photographs. It was able to embed its own data. From the machine learning perspective or data science perspective, this is actually called overfitting. It's a very common problem. It's nothing spectacular. It's just you didn't train your model, right? And so especially with overfit models, you have the training data in the model itself or the model's able to generate that training data. And so as I was saying earlier with things like credit card information, PCI protected information, personally identifiable information, public health, or not public, sorry, personal health information or protected data, things of that nature. It can be very damaging for a machine learning model to leak real training data. And so it's important to kind of evaluate and make sure that the models are not overfit, are not underfit, or just hidden just right or that the data is being pre-processed in such a way that it's a one directional. So in some cases, what they'll do is generate a hash of certain features. For example, instead of putting in a credit card number, they generate a hash of the credit card number. And so it's not reversible in any meaningful way. And then of course, model theft, model theft by competitors. There's a technique called federated learning, which is you encrypt the machine learning model itself so that the model has to be used in that encrypted form. You can't steal the model, you can't decrypt it. It's one way kind of thing or a one way street in that scenario. And so this will allow like IoT devices to be deployed with machine learning models and it will prevent competitors or attackers and malicious actors from reversing it, stealing that model and building their own product off of it. And then one last thing that I kind of want to cover is adversarial stickers. So you've probably seen adversarial stickers. They look like melted crayons in my opinion. But adversarial stickers, they have a bit of a problem. This is considered a brittle attack and it's not very repeatable. The reason why is because it doesn't have that attack transferability that I brought up earlier. So in this case, kind of off to the bottom right, the stop sign, this is one of the first articles I was talking about adversarial stickers and they were trying to modify a stop sign in such a way that the machine learning model didn't recognize it as a stop sign anymore, recognized it as something else. And so this could be really damaging for self-driving vehicles. One of the problems is that this paper ran into some repeatability problems. And the reason why is because this attack is so brittle, they were able to convince their model but they were not able to convince or people who were trying to replicate this were not able to convince other models. So it's not very useful. You can't just slap an adversarial sticker onto something onto a sign and hope that you're driving a self-driving vehicle off into a wall or something of that nature. And then off here to the left is kind of the same idea with facial recognition. So this researcher here said, okay, I'm going to set some constraints. I only want the pixels or the image around my eyes in the shape of glasses to be modified so that I look like this actress. Honestly, don't remember who this is, I apologize. But I'm a man, I want to look like this actress per a facial recognition model. And so he was successfully able to do that. The problem is that these glasses are designed for his face and the rest of the features of his face. And so if I or many of you put on these glasses, it'll either look like somebody else or it won't even recognize it as a adversarial example and it'll just recognize us as people, the same people just wearing stupid glasses. So attack transferability adversarial stickers don't really have it. It's not a really good attack, it's very brittle. And I just kind of wanted to bring that up. For more information on this, I highly recommend Defcon 2018 AI Village Sven has a really good 30 minute talk on adversarial stickers and kind of the challenge with them. And then for the facial recognition specifically, I know that Richard Ring in AI Village 2019 has a really good talk talking about facial recognition and trying to bypass that. So to recap, how has AI empowered attackers? Well, it allows us to operate at a speed and scale like never before, we can operate at machine speed. And then what we are seeing is the very beginning of what AI can bring to the offensive security space. So we've already seen some adversarial attacks in the wild. I anticipate us seeing more as more researchers are coming out AI and machine learning is kind of the hot new buzzword that everybody's latching onto. So CEOs and decision makers are grabbing the AI powered or machine learning powered detection systems, defense systems, and it's kind of become a bit of a buzzword where you say that in a lot of common companies, especially in places like DEF CON and they tell you to go talk to the Bitcoin folks and blockchain and all that fun stuff. There's no silver bullet, but we are seeing the very beginning of what is possible and I anticipate and I'm really hoping in the next several years that we see a much larger impact of this AI versus AI kind of warfare that should be coming about. But anyways, I appreciate it. Thank you for your time. Thanks for sticking around this long. You can go ahead and email me if you have any questions. I'm also gonna be in the chat. And then for the slides I have that are much more detailed are on Slideshare and you can find them at slideshare.net slash gtklondike. So that is my presentation. I hope you learned something and thank you for sticking around. Bye.