 Alright, so our next talk is Ram here, and he's going to speak about a framework to quantitatively assess AI safety, which includes challenges, open questions, and opportunities. So, give it up for him, and I'm going to let him take it away. Thank you. So, a quick show of hands. How many of you are currently using machine learning as part of your product, or your company wants you to? Okay, perfect. How many of you want to? Okay, sweet. Okay, so that's great. You know, let's go back to the little bit of early 90s. The internet is just starting out, you know, things are fine. And all of a sudden, you know, we see a spate of cyber attacks, like, you know, and I don't have to explain this to you guys. And then what happened was the regulatory landscape kind of quickly changed. You know, federal laws started coming into place, like, you know, the Computer Fraud and Abuse Act. Apparently Reagan saw the war games movies, got really inspired, and you know, within like a couple of years, this act was formed. Hopefully, people are not watching Mr. Robot, and we get like stronger acts like that. State laws started cropping up to fill in the gaps. Anybody wants to take a wild guess, which was the first state to actually make ransomware illegal? You're in the right track. Closer. It starts with a W. Oh my gosh, you're right. Wyoming. Yeah, Wyoming, interesting enough, is the first state. I think in 2015, legislated ransomware is illegal. And we have like a spate of like standardized cybersecurity frameworks. If you are a company and you want to convince your customers that your system is secure, you would adopt one of these frameworks. And that's how you would convince your customers that, hey, I pass all these controls, and this is pretty much I'm a secure system. Let's kind of shift to like machine learning today. This is like the machine learning landscape by Siobhan Zillis, a venture capitalist at Bloomberg Beta. If you can't see anything, which I'm sure you can, a lot of companies out there who are actually investing in machine learning as the core value proposition. And the interesting thing is the attacks on machine learning is now grown at an exponential rate. Archive, which kind of documents how many, you know, the, it's a public repository of documents has close like 100 papers on adversarial machine learning just in the past two years. And this is kind of like a timeline. Sorry, you can't see it, but I know I'll treat the slides out by Batista Bishio kind of like mapping out extensively like what are the different attacks that are possible. You know, starting all the way back from 2004. But today, our regulatory rent landscape with respect to machine learning looks pretty empty. You know, there's no way there's no guidance from the government. There's no guidance from standards organizations to tell you which ML algorithm is safe. And this kind of is a problem because we've now all bought into this Kool-Aid of machine learning and we have no way to tell our customers. Hey, what I'm doing is safe from my machine learning perspective. So that's what we're going to talk about today. You know, my name is Ram. I work in the Azure Security Data Science team. Our promise to customers is, you know, we'll keep Azure and you secure. And this work is done as I'm also a research fellow at Berkman Client Center. And this is what I'm going to be working on there. So a lot of this is ongoing work. So it's rough around the edges. And if you're working on this area, I'd love to learn, talk to you more about. And this talk basically has two parts. So the first half is going to be about how do I assess the safety rating of a machine learning system? So, you know, your development team, say, you know, let's say that they built a machine learning system, you know, can we pass it through some sort of framework and get a safety rating? Just like, you know, how you would expect of like the health of a food or, you know, the safety of your car seats, which you can very much look at, what kind of guarantees can you get about the safety of your machine learning system? That's going to be the first half of the talk. The second part is going to be about the legal underpinning of a machine learning system. You know, we all have, we all heard about the doomsday scenario, right? Like there's a drone, you know, using an open source system. And instead of landing in your backyard, what if it kind of like, you know, chops your hand off? What do you do then? Like, who do you, who do you sue? Do you sue the drone maker? Do you sue the open source software? What kind of like legal underpinning is there for like machine learning safety? So those are the two things. So first of all, like the call for like machine learning safety is not new, newish. So, Urs Gasser, you know, wrote a fantastic paper on AI governance and said, hey, the first thing you want to be tackling is some sort of standard, some sort of data governance with respect to machine learning safety. Ryan Kahlo, the robotics, I was a law professor at UW, has pretty much laid a lot of open questions on this field and proposing a lot of fantastic ideas. And things are getting real. You know, Europe's AI strategy specifically says by 2019, they're going to come up with a standard framework and a liability framework for ML safety. So things are picking up at a great space. So it's not like I am, all of these are new ideas. I'm building on top of them. So first off, why do you, why should it care? Like, you know, what is, why is this an important topic for us to discuss at DEF CON? I think the most important thing is we all, we all want to hold our service providers to some accountability. Like you are trust, just like how you trust your data to say the cloud and you assume that the cloud is secure. If you're going to use a machine learning system, like, you know, you want to know that the machine learning system is secure as well. Can we give a fantastic talk just like an hour before about the software vulnerabilities that come in, that you inherit when you build machine learning systems. And that's kind of like unacceptable because if you are running a production system, you want your customers to feel safe from that perspective. And from a regulatory standpoint, just like, you know, if you want to take the cloud as an example, you have a bunch of laws that, you know, Congress, you know, regulatory agencies, and even councils, independent councils have created. So, you know, if you want to host, say, the credit card data, you would have to be PCI compliant. And if you fail, you pretty much get fines, you lose the ability to operate, and you can be criminally prosecuted for this. So, hopefully, we don't want any of you guys to go to jail. So the first half is going to be about, like, you know, how to assess safety ratings. So before, like, you know, we go to the most important part about how to assess, a bulk of this talk is going to talk about why this is a very difficult problem to solve at. So end goal, I'm not going to have, like, one framework, one definitive answer. It's just going to be a proposal. So what are the challenges? First off, you cannot escape adversarial machine learning. There's no vendor who can claim that, you know, my machine learning system is adversarial ML-free, because it's ubiquitous among all ML paradigms, whether you take supervised learning, whether you take, like, reinforcement learning, adversarial examples are just a force of nature. And paper not kind of figured out that adversarial examples are also transferable. So if you have a system, say, a logistic regression system, and you train on that, it's pretty much those attacks are transferable to, say, decision trees as well. So not only are you kind of, like, guaranteed, but it's also transferable. Skegady in, like, 2016 said that, hey, it's not only transferable, but even if you have two completely different data sets, you will still have, like, adversarial machine learning. You're pretty much caught in, like, a triple whammy at this point. And the second challenge is verifying if an algorithm is safe is practically difficult. In fact, like, it's an NP, it's computationally intractable. That was, like, one of the stellar results that we got this year, that even if you can, it might be theoretically possible, but computationally it's impossible because of limited training data. And the biggest kicker is it's an NP complete problem, especially in the context of deep neural nets, which is, like, really big, popular right now. So if somebody, you know, if somebody gives you a deep neural net, and if they ask you to verify a property about it, hey, is it safe? It's an NP complete problem. There's no, like, exact, you can come up with a solution, but in polynomial time it can't really verify it. There's been some good news. I don't want to, like, damper the whole system. You can get some sort of upper bounds, some confidence bounds, but this work is super nascent at this point. The third challenge is, you know, completely safe ML systems can only occur, you know, if you have a non-zero tester. So just to, like, you know, what this statement means is if you, when you train your machine learning systems, you get, like, two, you know, two things that you want to really care about. Like, how does it perform in the training set and how does it perform in the test set? And even if you get, like, a machine learning system that performs really well on a training test, on a training data, there's no guarantee it's going to perform well on a test data. So imagine that you built, like, a... Oh, it's not me. Thank you. Yeah, there's no music as part of this presentation. So the only way, one of the insights that two recent papers had is the only way that you can kind of assume, like, a system is safe is if it has non-zero tester. And non-zero tester is really not possible. We'll see why. This is kind of like a relation. It's called a VC bound. It bounds your tester as, like, you know, your training error and as a function of something called a VC bound. We won't go into that. But just, like, you know, keep in mind that tester is kind of, like, proportional to some, you know, some additive nature of VC. The problem is these bounds are really loose. Like, if you have a linear classifier, it is, like, you know... And if you have, like, a million features, you know, your VC bound is kind of, like, a million plus one. So if you think, if you go back here and you put, like, a million, you know, the bound is really loose. And for some learners, it's kind of infinity. So imagine, like, you know, you've always wanted as a kid to put an infinity in an equation. Now is your time. So it's really, these bounds are very loose and they're getting, like, a zero tester is impossible, which means it has big safety implications. And this being DEF CON, you know, I found this, like, cool... Oh, now you can see the picture. But this is Vapnik, who kind of, like, you know, found the VC bound. And if you could read it, it says, all your bays are belong to us, but bays spelled with a Y, like, Bayesian learning, like, nerds, right? So... Of course, there's a lot of, like, defenses that have been published and archived, you know, scores and scores of those. But there is no single defense, unfortunately, from a machine learning perspective that you can use to protect yourself against attackers. There's a great paper, which actually I think won the best paper at ICLR, a top-notch machine learning academic conference. What the paper showed was, like, out of the... They took, like, the nine defenses that were published as part of the conference and they broke seven of them. So it's very easy, you know, to kind of, like, get over these defenses. Goodfellow is kind of like the father of this field and kind of, like, ranked some of these defenses along certain axes and we'll see that a little bit. The next thing is, like, the next big challenge of constructing a framework and, you know, guaranteeing safety is there's a big tension between adversarial examples and interpretability. How many of you know what GDPR is? Okay, perfect. You have all gotten those annoying emails, right? Because if a company didn't, then awesome, you know, they're all gonna get slapped big fines. So one of the things that GDPR articles say is that they want to support explanation. So, you know, which means, like, some sort of, like, explainability should be in their model. So this way, if your mortgage gets denied, you know exactly why it's been denied if it's there's a machine learning system behind it. So a lot of people are now using, like, explainable models as, you know, part of, you know, to get, like, explainability. And the crux of most explainable models is that they're linear learners. And linear learners are extremely vulnerable to tampering. So if you have this tension, wherein, like, you want to bring explainability, so you use linear models, and if you use linear ML models, you're, you know, you kind of have this adversarial tampering. So this is kind of, like, a big tension just like how we face in security field where we have a tension between usability and security as well. And finally, like, ML safety is a lot more about adversarial tampering. Justin Gilmer, who wrote a paper this time about, like, what it means to do adversarial example in real world, gave this anecdote. If you think of, like, you know, the big poster child for adversarial examples is autonomous cars. You know, hey, what if, you know, somebody puts a sticker on my stop sign and now a stop sign is now, like, recognizing a speed limit? Well, they said, hey, let's be more realistic here. What if there's no stop sign in the beginning in the first place? What if a stop sign gets, like, knocked over by Augusta Wind? What if, like, a guy's wearing a stop sign and, like, stop sign t-shirt and walking on the road? Does that mean the car is going to just come to a halt? So a lot of these things, in a practical sense, kind of dictate, would still, like, bigger than the scope of, you know, training or testing within, like, a dev machine under your box. So these are kind of, like, the challenges at this point. You know, you cannot escape attacks. There is no single defense. And it's very, it's extremely difficult to verify the security properties and really, you know, the safety is, like, more than one predominant technique. Existential dread. You fall under a bed. And now, like, you know, wait for the doom to get over it. But you guys are the perfect audience. Like, in a security field, we face this every single time. Like, everything that makes ML systems kind of insecure is what we face in a security setting on a day-to-day basis. And we tackle this risk, we tackle this by minimizing risk, right? You know, this is just one framework, like, you know, the dread framework of, like, you know, you try to see how much an attacker can cause damage. Now, how hard is it for an attacker to kind of reproduce other attackers to kind of reproduce it? Are you always worrying about, like, APT, or is it going to take a strip kitty and bring your system down? How easy is it for somebody to discover this particular vulnerability? So with that in mind, like, you know, here's, like, one possible framework that, you know, that just came to my mind is the goal is to kind of score the kind of attackers score different parts of, like, your ML pipeline. So the first thing is, like, you want to score the attacker's capability. Hey, does the attacker have access to my architecture and training data is very different from, like, if the attacker can just, like, query my API. So you want to have two different, like, you know, just, like, how do you calculate a dread score? You want to have a score for that. Hey, what are, like, you know, some of the possible attacks? You know, is it, like, a white box attack? Or is it, like, you know, a black box attack with some sort of limited querying? And have some, you know, a score for defense. Kind of rank them in some aspect. If your AI, if your ML engineer is just using, like, rate decay as a defense technique, it's really not in a good position. Whereas, you know, if they're using some sort of, if they're using some sort of strong, adversarial training with strong attacks or they're doing something that's logit-pairing, you're probably in a better place. So the same framework that you use in the context of assessing security risk, making an argument that you want to do the same thing for ML risk. And once you get that, you can kind of, like, you know, from a regulatory perspective, it becomes a very easy conversation. Let's assume that you get a score and say, hey, then you can make a claim that all ML, you know, if there's a medical device that uses machine learning, that must have a safety rating of nine or about. Or, you know, if you're using, like, military-grade drones versus civilian-grade drones should have, like, different levels of safety ratings. But of course, there's, like, a lot of open questions, right? We saw, like, the challenges. We saw, like, one possible framework. Like, who's going to certify, you know, that ML system is safe? If you go to, like, a CDC or, like, is there going to be, like, an FAA that takes care of the safety of airplanes, who's going to take care of the safety of, like, ML systems, especially that they're so pervasive? Ryan Kahlo kind of argues for, I mean, mentions, like, a federal robotics commission. And that idea is not too far fetch. Like, you know, given the fact that ML is so pervasive, we still don't know what the process of certifying it looks like. If anybody has been through, like, the GDPR process, you kind of know how an immense amount of burden is put on folks to kind of, like, get that GDPR-safe systems. And how do you ensure verifiability? If somebody, you know, if you go to a customer, if you buy a service from a service provider and they say that their machine learning is safe and we have a safety rating of X, how do you verify that the safety rating is still the same? And there are also, like, very practical questions, right? When do you calculate the safety ratings? Do you calculate it every day? Every time you push a feature? Well, tough luck because we're all now in the DevOps world and we're all, like, you know, shipping features every single day. So does that mean every time somebody touches code do you calculate this ML safety rating? A lot of the mechanics of ML safety has not been nailed yet. So lots of open questions do have ideas. I'd love to hear more about it. So that kind of, like, brings our first half of the talk, you know, to a kind of a close. We'll quickly kind of, like, look at, you know, what are the legal underpinnings of machine learning safety? And we're going to look at it from the lens of internet safety. Just because, like, you know, internet and machine learning seem to have a lot in common. Like, you know, both are transformative technologies. Nobody could imagine that people can send cat pictures when they had DARPA put out DARPANet. And just like how, like, machine learning today is used for very similar purposes, you also have the idea of, like, transforming it from so many different angles. And Jonathan's a train called Internet Genitive Technology which means that it has the capacity to produce new content for broad and varied audience. And the same thing is with machine learning. Yesterday there was this amazing demo of creating music with code from this DJ, which I thought was very interesting. Really shows the power of machine learning and the internet. And both have, like, great degrees of instability. In the loft talk, they said that how you could take the internet in 30 minutes or less. The same thing was with the machine learning. Hang Lee said how he was very easily able to open a shell in, like, a top, like, image recognition system in very short period of time. And from a legal aspect, you know, it makes sense to look at ML in the ends of internet just because of precedent. And precedent is really, really core to common law legal system, right? My favorite quote is you know, in one of the verdicts from, like, Vasquez and Hilary, they said that the court really relies on precedent because it establishes a bedrock of principles as opposed to some individuals, like, you know, making up law as they go. And the cases that God decided in, like, 1997 is still having an effect today in the internet. For instance, like, in Reno versus ACLU is essentially why you can have profane content on the internet because on the legal basis of, like, freedom of the First Amendment. And in 2018 there was a case Packingham versus North Carolina where in North Carolina said, you know, sex offenders do not have the ability, should not be on social media anymore. But the court said no on a unanimous decision. And they cited Reno as one of their legal precedents. And really precedents do not get overturned by the court so easily. They wait for societal changes. So really looking at it from the lens of internet safety makes a lot of sense. And why are we doing this from a legal perspective? So imagine, like, poisoning attacks in machine learning. Poisoning attacks are essentially when an attacker can send chaff data to your machine learning system with the goal of subverting it. So to either misclassify it or, you know, or even completely change the goal. A relevant cyber law case could be, like, CompuServe via cyber promotions. So this was the case in 1997. So the basic just was CompuServe was an ISP and cyber promotions with a direct marketing email. And they were basically sending spam via CompuServe. And CompuServe customers were obviously, like, super pissed off. They were like, hey, man, like, you guys need to stop doing this because my customers are going to bail out. And cyber promotion says, hey, no, that's kind of like, you know, you're a common carrier. This is our First Amendment. And the court basically said, no, it's not the case. And they established presidents that even on a non-electronic, even for an electronic property, you can pretty much have, like, trespassing. And this might be possible. This might be relevant for poisoning attacks when an attacker is sending like spam images to kind of derail your machine learning system. Or let's think of liability, right? Imagine that, you know, you go to a scenario where you, you know, you get some sort of, like, a robotic system to detect art forgery. Like, you know, you're expecting this top-notch, like, ML system to detect forgery, but you basically get a lemon. And now you lost a bunch of money because instead of Picasso, you've gotten some other, like, weird art student painting. Now, can you take the person who sold you this machine learning system who said that it can, like, identify fake art to court? Because you suffered, like, millions of dollars in damage. Is it an interesting system? Because liability in the context of machine learning safety is still a nascent system. And irrelevant cyber-law cases trans-corp the IBM. So essentially, Trans-Corp America bought, like, a lot of equipment back in 1994 from IBM and those discs failed and they suffered economic loss. So obviously, TCA sued IBM for economic loss and the initial, and this is, like, back when the internet was very nascent and the court basically tried to limit the liability because the courts basically said you can sue them for contractual, you know, for them reneging on the contract but you cannot sue them on the basis of economic loss. You cannot sue them for all the money that you've lost. And it's kind of crazy because today when we buy computers and say you're your OS failed and you lost all your documents we do not think of going and suing our computer you know, the people who wrote the OS for economic loss because we kind of implicitly assume computer systems are faulty and today, you know, we kind of learned that machine learning systems can be attacked and machine learning systems can be implicitly faulty. So because of the changing landscape, it is very much possible that the courts will also try to limit the liability because in order to foster innovation I mean, there's a big exception, like, if you got you know, if there was a drone that came and chopped your hand off there might be, you know, this probably doesn't apply. But here's a very interesting thing that Ryan Kalo pointed out. If you have the scenario where there was a bodily harm caused by, you know you know, like a machine learning system that used open source software, what do you do then? Who do you assign proximate cost to? The people who built the drones or the thousands of developers who contributed code to the open source systems, and these are again, like, very not easy questions to answer. Oh, this is by the way cafes, like, if you're wondering why this was an open source symbol, cafes are very popular image recognition, like, system right there. So, yeah, to kind of like, you know, wrap up with the open questions here, like, when an ML system breaks down how do you get relief? Like, where the damage is foreseeable? Do you go after the company that made the product, the open source tool kit, the company use, or if the company kind of hosted it on a different service, a service provider or the researchers who kind of like whose algorithm the computer use. And all of these are important questions for answers we do not really know because there is no case that's been tested so far. So, to wrap up, if you were to think of, like, all the countries, you know, there's been a spate of them releasing AI strategy so far, like, you know, and these are some of the handful of companies, countries who have, you know, pretty much put AI safety and privacy as a big strategy, you know, in their vision. Of course you might see our red, white and blue is not here, and that's because we do not have one yet. The US's approach to machine learning safety has been very different. Us being like us, our focus is a lot on autonomous weapons. And the Obama administration back in the 2016 created, like, you know, the US artificial intelligence R&D task force. They produced two reports and recommendations. Obviously with, take it back, with this new administration they disbanded the task force and kind of, like, you know, pushed those reports aside. And if you're wondering where we are, they created, like, a select committee of which it's co-chaired by, wait for it, our president and a bunch of other people. So we don't, very recently there was a bill called a future of cyber artificial intelligence act that's been proposed and nowhere are these as Congress even mentioned about AI safety. So we're not really in a great place in terms of, like, even thinking about this problem yet. And it's a shame because a lot of the companies have taken this on themselves to kind of define standards and try to get ahead of the curve. So to kind of conclude, you know, safety frameworks are super critical for machine learning systems. And that will help us, like, create robust standards and certifications that we can use to our advantage, whether to convince customers or to add value. And this instead will help us get meaningful relief should there be or when there is a case that involves ML safety. And with that, like, you know, I want to end with these things. The court is still grappling with the effects of internet. This was just as Anthony Kennedy in Packing MV North Carolina in 2018. And he said that the forces and directions of the internet are so new, so protein and so far reaching that courts must be conscious of what they say today might be obsolete tomorrow. And they're talking about internet which was started in the 1980s. And think about, like, a feel like internet is something and what effects and how the legal system is going to handle it. And this is a big open question. And if I were to wear my engineer hat on, you know, Ken Thompson, you know, a Turing Prize Award winner, you know, asks this question, you know, to what extent, like, you know, should you trust a statement that a program is free from Trojan Horses? And the answer is, like, that you did not write yourself. And that's shocking. But think of, like, you know, how ML is being viewed. Karpati, you know, one of open as rector for machine learning and probably a big titan this field compared machine learning as software 2.0. You know, that machines will now be able to write better programs than humans. And I'm going to end with this unsettling note, which is how much are you going to be trusting ML models that we did not build ourselves? And that's pretty much, I feel, is going to define the field in the next coming years. So that's about it.