 is going to be by Peter Eckersley. He works for the EFF. He started there staring at packets, then observed SSL in the wild for a while. And now he basically moved up to layer 8 and looks at the security and privacy implications of machine learning. So give him a warm welcome. Thanks, everyone. So I'm going to talk today about the strange new world that's unfolding with machine learning, deep learning, reinforcement learning, convolutional neural networks. These techniques that have appeared started to be successful in the last few years at solving problems in AI and start asking questions about what are the security and privacy implications of those technologies? If you're a hacker who likes to break things or you want to build secure systems or if we as a community want to understand the implications of these new technologies, what do they look like? So I'm going to begin by breaking apart this term AI because there's a lot of hype about AI right now. It's definitely a buzzword. But there's also a lot of reality to go with the hype. So I'm going to split things into two categories. There's this idea of narrow artificial intelligence, mostly the machine learning algorithms that I was just talking about, deep learning, convolutional neural networks, reinforcement learning. These are definitely learning algorithms for the most part that you could probably also include computer chess, which is not a learning problem in narrow AI. And these are being used to solve specific problems by being shown examples of the solutions. And then the algorithms basically make a pile of matrices that you can multiply together, stack together, and then they output a very complicated formula that just happens to do what you wanted. And so this is a fairly old technology. In fact, it's been around for 20 years. But with GPUs and a few algorithmic advances in the last five to seven years, suddenly these algorithms are solving really difficult problems. And then there's another kind of AI that people talk about, which I'm going to call artificial general intelligence, or AGI. And that's basically the stuff out of science fiction. That's why you have computers that perhaps think like humans, or think at least as flexibly as humans do, maybe in very alien ways. So it could be a very human AGI, or it could be something else like a hive mind or an Oracle question answering machine. Like you type into Google a question, and it thinks about it a lot and gives you a very clever answer, but it's still just doing that rather than having its own agency. So those kinds of technologies are science fiction. They don't look totally imminent. They're at least a long way off. Maybe we're making rapid progress towards them. But it's important to understand that the hype around AI blurs these two ideas together. So we're talking about real progress that is rapid in this narrow AI field and a lot of speculation about what AGI would be like if it happened. And it's important to tease those apart and be clear which of them you're talking about. So starting about why is everyone so excited about narrow AI? The answer is it's doing things that we couldn't do before. This is the Baidu Baike. This is the Baidu encyclopedia that China uses instead of Wikipedia. This is Google translating their article on privacy. And if you look at how good that's gotten, on the left here I have the cistern algorithm, which was Google Translate 10 years ago. On the right I have a neural algorithm that Google deployed last year. And if you compare these two translations of a text in Chinese, the one on the left is basically word salad. It's a lot of effort to make any sense of it. It's like deciphering something from an ancient civilization to try to understand what they were talking about. The text on the right is not yet clear. It's not yet fluent English translation. But it's starting to be something that you could read and make sense of if you needed to understand what was going on in China. Here's a second example. This is a neural algorithm for artistic style. So what this algorithm is doing is it's being given example images on the left column here. So an Escher painting or a Munch Picasso Cubist painting. And it's able to extract the stylistic features of those source images and then apply them to a second image that you give to the network. And so here a picture of the Golden Gate Bridge in San Francisco where I live has been fed in. And you can see that it's being styled automatically using the different style from the source images. Now you could say, OK, this is just Photoshop filters. And indeed it is. That's what a Photoshop filter is, except these Photoshop filters are being produced automatically by a neural network given the source image. So this is a very new and impressive result. Here's a third example. So this is a compression algorithm made with neural networks. It's not a real compression algorithm that you can use in the real world yet. It's basically a laboratory compression algorithm. But it's absolutely blowing JPEG away by an order of magnitude. So to show an example of what's going on here, here's a source image, this row. So take a look at this picture of a red stiletto. Here we have a JPEG image at 15 times compression of that stiletto. And you can see all the fast Fourier transform artifacts. You can see that a JPEG image at that size is really a poor reproduction. JPEG 2000 is a little better. But you can still see it's a very imperfect reproduction. Here we have a neural network compression of the same picture at 28 times. So half the size of this JPEG. And it still looks like a very faithful reproduction of the original image. You can crank it further. So this is 112 times compression. Still pretty good. Still better than the JPEG at 6 times the size. If you crank the neural network compression algorithm further, it looks like this. This is 224 times compression. So it's remembered that there's a red shoe. It's just forgotten that it was a stiletto. And so we're entering a strange new world where the algorithms available to us through these techniques are doing really cool, awesome things in weird ways that are quite novel. You can also see examples here of faces being compressed in the same way. As you crank the compression to enormously dense levels, you get something that actually still looks a little bit like the source image. It's just this slightly strange composition of abstract faces. So we actually have this project that we launched at EFF that you might have seen called the AI Progress measurement project. You can check it out, eff.org.ai.metrics. And what we're doing is just keeping track of a bunch of different problems that people are working on in machine learning and neural networks, and seeing how fast progress on all of those things is happening. Basically, so we can track the policy implications of this stuff and be aware of them in advance. To show you a few things that are in that project, ImageNet is a widely studied computer vision problem. In this problem, algorithms are shown examples of photographs like these that are labeled with, this is a leopard, this is mushrooms, this is a Dalmatian and some cherries, this is a Madagascan cat. And then after being trained on hundreds of thousands of these images, the algorithm is given a test to see with some new images whether it's able to correctly identify what's in them. These are images it's now never seen before. So if you look at performance on this test over time, in 2011, the error rate was 27%. So the algorithms were not very good at doing this. In the subsequent years, you see that there's been enormous improvement. There was a famous scandal where Baidu claimed to have beaten humans for the first time at this and then they had to withdraw their results because they'd been essentially peaking effectively at the test examples by submitting too many entrants to the test. But since that, a number of other teams, including another team in China, have completely beaten humans at this task. So algorithms now do a better job at recognizing what's in these pictures than humans do. Another example, this is a simple reading comprehension test that Facebook proposed and started a contest around. And so this involves showing the algorithm a bunch of simple logical reasoning tasks and then asking it the answer to those puzzles. So here's an example. Sheep are afraid of wolves. Cats are afraid of dogs. Mice are afraid of cats. Gertrude is a sheep. What is Gertrude afraid of? The answer is wolves. Another one, the football fits in the suitcase. The suitcase fits in the cupboard. The box is smaller than the football. Will the box fit in the suitcase? Answer yes. Will the cupboard fit in the box? Answer no. So performance on this, again, we're getting to basically perfect performance, like very close to perfect performance. In the course here, it's not 2011. This is just two years. Since Facebook proposed this challenge, people have gotten really good at solving it. Another harder example, this is an example of something that's not solved yet, is visual question answering. So here, you have an image a little bit like the images from ImageNet, but instead of just being asked, is this, you know, what's in this picture? And the answer is pizza. Now the questions that are asked are free form. So the user can ask any question they want. And there's a training and test data set made from Amazon Mechanical Turk workers who are showing these pictures and asked to make up questions about them. And so here, there's a question, how many slices of pizza are there? And the answer would be, you know, I think that's eight, but maybe not, it could actually be, it's hard to tell. You can see that you have to play this game. Is this vegetarian, the answer is no. Sorry, is this a vegetarian pizza and the answer is no. In this one, you know, the question is, what color are her eyes brown? What is her moustache made of? The answer is bananas. So for visual question answering, this is a really hard problem. It requires the network to have both a very good image model to see what's in the image and a good language model and the ability to merge them together. So performance on this is not yet at human levels. A typical human, given tests like this, you know, we'll get answers around 85%. Algorithms at the moment, neural networks are just at 66%. So a little way off from human performance. But that's gone up by almost 10% in a couple of years. So what are the security and privacy implications of these technologies? The first, I think obvious and really troubling one is that mass surveillance is going to get way more effective because of these techniques. As soon as, whether it's governments that have all this data or private commercial tracking companies that have all this data, they aren't going to sit around and ignore these new techniques. They're going to, unless we can find a way to stop them, they're going to use neural networks to process all of the surveillance data that they have on us and to draw inferences. It will be possible to build a system that says, predicts the 100 most likely people to organize a protest in this country next week. That's not a super hard problem. It won't be done perfectly, but we should be prepared for governments to start doing things like this. And until recently, learning that much about people basically required the concentrated attention of an analyst. Someone had to sit there and look at your records, look at your browsing history to figure out the pattern. Here, what they all they need to do is show the computers a few hundred thousand trained examples, labeled examples of the behavior they're interested in, and then they can process hundreds of millions of people's data. And I think if we want an intuitive understanding of what this looks like, Facebook is probably the working example. When you use Facebook, it learns an awful lot about what you're interested in, what you're planning to do, who you're interested in, and it's using that data to suggest things to you and to try to sell things to you. We should expect more things like that and more of the weird implications. If Facebook is used by political campaigns, I think people have claimed that both Trump and Brexit were heavily backed by effective use of Facebook and companies that knew how to use Facebook's algorithms well, we should expect that to be the future world we're going to live in for everything. And Facebook's just the leader in this. Other companies are going to figure out how to do it too. Intelligence agencies are going to figure out how to do it too unless we can prohibit them by law from doing so. Another huge problem we're going to have, in fact, we already have, is biased decision-making, which is both a problem in and of itself and then has this weird nexus to privacy. So in the United States, really high-stakes decisions about people, like, are you going to have to pay bail? How much bail are you going to have to pay if you get arrested? A being made on the basis of a machine learning risk score that's produced about people. And the scores turn out to be massively biased according to race. If you happen to be a Caucasian American with one set of characteristics or an African American with the same characteristics, your risk score will be much higher if you're the African American. And this is a symptom of two very severe problems. One is that the source data for these systems that they're training on is itself biased. If they're predicting whether someone will re-offend, they don't know the truth of that question. What they know is, in their training data set, did people get arrested by the police later? Did they get convicted by the justice system later? I mean, it's been documented that both of those variables are hugely biased against people. And so if you train on biased data, you produce a prediction, a model, that is equally biased. You reproduce the bias in your data. The second problem is more subtle. It's a thing called omitted variable bias. And this is a statistical problem. If you work in this area, you should go and read about it. It's a statistical problem that basically reproduces the flawed logic of prejudice or racially biased thinking. And in the abstract, what this is, is a true cause for the thing you want to predict, like whether someone re-offends. But you don't have any data about the true cause. And so what you do is your algorithm finds proxy variables that are correlated with the true cause. And one of the most useful ones is often going to be a variable like race that you want to actually protect people against. And then if you exclude race from your model, so you're not allowed to train on that, the model will find something else that's correlated with race, like where you live a zip code in the US, a postal area or something, the neighborhood you're in. And then it'll use that to predict the outcome and it's basically reproducing race as a cause. So this may sound a little confusing and weird. I'm going to give you a concrete example. This is when someone actually documented when I was at a conference about this stuff. This guy said, oh, my son had this problem where he was trying to buy car insurance or pay for car insurance and it was based on a little device that was in the car that was actually tracking the use of the car. And he kept being charged the highest possible rate for insurance. And his father called up the insurance company and said, what the hell, my son is a real, I drive in the car with him, he's a really good driver. Why are you charging him the highest possible insurance premium? And after several hours of arguing with the insurance company, they finally said, oh, the problem is that your son drives at night. And people who drive at night a lot are dangerous drivers so we charge them the highest premium. And if you ask why is that, statistically, it's because people who drive late at night are probably more likely to be potting and they're more likely to drive drunk. But this guy's son was a shift worker. So he was driving at night because he worked at night, which means that he's basically being judged by this category that isn't a true reflection of his particular circumstances. So here, the missing variable, the emitted variable, is shift worker. If you put shift worker into the model, suddenly you would get a different prediction, but it's an emitted variable. Turns out the world is full of emitted variables. Most of the variables are emitted and so when we build these models, they're inherently biased. Now, fortunately, there is some work on how to correct for these problems. This is a data journalism visualization project that Google put together about some research that you can use to correct a model for, say, racial bias. It's an algorithm you apply to your existing model. It transforms it into one that's essentially a model of equal opportunity. It says, what's the false positive rate for a protected group of people? We're gonna ensure that for all of the different, say, categories of race, that they get the same false positive rate, which is a proxy for fair treatment. So if you're building models that need to make decisions about people, I guess my first piece of advice is, don't do that at all. Secondly, okay, if you really wanna do that, you're gonna need to figure out how to get unbiased data. You're gonna need to apply these algorithms to de-biase your predictions. You're gonna need to keep track of a variable bias more generally. Okay, so how does all of this intersect with privacy? It turns out, privacy is one of these things like being a shift worker that's gonna totally mess with all of these models. The algorithms expect when people are deploying something like a criminal justice prediction thing or something to predict your insurance premiums or whether you get a home loan, they're assuming you're gonna make as much data available to the algorithm as possible. If you happen to be a privacy freak, like people here at Shaw, you are going to get very strange treatment from these algorithms. For instance, I heard about one insurance company that had looked into what data they could use from social networks to inform insurance decisions. And they basically concluded that for legal reasons they couldn't look at anything except whether you put a false birthday on your Facebook profile, because if you did that, that was evidence that you were not telling the truth and people who don't tell the truth on forms are less reliable according to their model. I think I don't know how many people here in this room have a Facebook account. How many of you who have a Facebook account have your true birthday on there? How many have a false birthday? So it's about like 60, 40 true to false. In an extremely like well-educated, reliable audience, like I think people in this room probably pay back loans at a higher rate than the general population, but because you care about privacy, you're not going to tell Facebook your true birthday, and yet these models are going to use this as a proxy for unreliability. So I think we're going to have some serious struggles around privacy law and the regulation of these machine learning techniques. Okay, another security implication is that these algorithms may be really good at finding bugs in our software. So we're seeing a test of this with this contest that's been funded by the US Department of Defense, the US military, through DARPA, has funded the Cyber Grand Challenge. This is basically a contest where a bunch of AIs get to play capture the flag, just hack into other people, you know, other teams' computers. And I think the joke version of this is like AIs hacking into computers. What could go wrong? So in this game, so you know, fortunately it's not x86 yet, they're playing with a sort of restricted version of x86, but in this game, the contestants like any CTF, or like many CTFs, they get given a binary, just arbitrary binary blob, it contains a vulnerability in it, and the contestants are searching through essentially input space to look for inputs that cause the programs to crash or allow them to run arbitrary code. And the teams are also running the same binaries on their machines and they're trying to find ways to firewall proxy or patch those binaries so they're not vulnerable. And in the early iterations of this contest, you're seeing that algorithms, at least in this restricted environment, are very good at finding a lot of bugs in those binaries. And I think the fundamental question that we as a community should ask about contests like this, which by the way are extremely cool, super awesome that you can do this, how does a contest like this affect the balance between offense and defense in computer security? Because we know that our planet's computer infrastructure is insanely insecure, we know that's a serious problem. Right now it's a serious problem because of fraud and getting your data compromised or whatever. But in a world with AI in it, I think it's gonna be a more fundamental problem. If you have intelligent beings, thinking beings at some point in the future that whose very existence depends on devices that are insecure and also that can turn around and break into all of our computers, neither of those things are good. So I think we wanna think about how do we get to a future where all of the devices are secure? And the conventional wisdom, I think in our community for the most part is, the attacker always wins. If you have some balance between offense and defense, it's basically with the offense. And Bruce Schneier has this long quote about it, but the fundamental point here is there's many ways to break into a system, the attacker only needs to find one, the defender needs to find all of them and fix all of them. So I think this attacker always wins thing is probably too pessimistic. Maybe a more realistic way of phrasing this is some attackers will always get through some of your layers of defense. Maybe if you have multiple layers of defense, you can protect against a breach of all of them, at least stacked on top of each other at any given time. And interestingly, this kind of automated exploit detection seems to change the equilibrium, right? If both sides, the attacker and the defender are running a search algorithm through input space on the program, you're replacing the expensive human auditing that goes into computer security work with comparatively cheap fuzzer coverage, right? You need a good fuzzer or good heuristic algorithm, a good AI for search, and then you need a bunch of CPUs or GPUs to do that work. You can replace a lot of your expert auditing work. And in theory, that should allow the gap between offense and defense to be closed that should actually help defenders. If people trying to break into your system need to run this fuzzer, you can just run it first. The problem is we have all these devices that are never gonna be covered by these protections. How are we going to get all of the weird Internet of Things objects in this room to be fuzzed properly? How are we gonna get the old versions of Android that were released years ago to be run with the latest version of these security algorithms? And I think the fact that we can't do that means at some point we're going to see another version of the Morris worm that has the ability to find its own new exploits. It's gonna be some piece of malware that launches with 10 exploits, but then once it's compromised enough computers, it starts looking at the other systems it sees on the network and finding new bugs in them. And so I'm somewhat worried about, there'll be crypto-viral malware that will be hard to stamp out because it's gonna not have a finite set of exploits that'll keep finding new ones. So some of the people working on the cyberground challenge the DARPA had proposed that the solution to this was to deploy network firewalls that would sort of automatically do the auditing in real-time using the latest algorithms on the network packets. The problem with this is it's totally incompatible with our aspirations to put TLS in everything. You can't have some smart firewall box that's inspecting your traffic and stopping exploits if you've got TLS, which you know you wanna have. So I think something our community should think about is whether we can put some of this automated exploit mitigation in our endpoints. Could we build something that we run on every Linux box and every Internet of Things device that actually uses machine learning in real-time to monitor that system, monitor the incoming network packets and say, oh wait a minute, I've never seen this before, but it looks like an exploit. I'm gonna drop this packet on the floor and alert someone or I'm going to, I've never seen this pattern of system calls come out of this binary on my machine. This looks totally weird and different. I'm gonna shut that thing down and send it off for forensic examination. I think this kind of version of this technology might be a way that we can really stack the odds in favor of defense and I think people should work on this. And a little hint, I think that we saw that this might be effective. This is the WikiLeaks documents from the CIA and what we see here is a discussion about Komodo's personal firewall by the CIA agents whose job it was to break into machines and it was clear that the Komodo firewall sitting on an endpoint was really frustrating for them. They got around it in the end but they had to do a lot of work to make sure that their exploits and their malware wasn't getting caught by these systems that are monitoring the device. So I think if we wanna enhance defensive technology, taking this insight and automating it, obviously the CIA and NSA people play both sides of this game so I think it's a lesson for them as well. Hey guys, fund better versions of this firewall because defensive cyber security is the hard problem that we haven't solved yet and we need a solution to this. So those are a few of the examples of ways in which these technologies in their current form are going to just change the world and we have to be ready for them, we have to be thinking about them, we have to be figuring out how to make the world safer with these things around. There are also some specific kinds of attacks that are possible against neural networks and I wanna give people overview of a few examples of those that have been found thus far. There are probably many more of these and so if you're a security researcher and you're looking for interesting things to break, go and look at neural networks research and you'll find a bunch of new categories of exploits that are fun to examine and play with. So here's one kind, these are called adversarial examples. So here, this is an attack against a classifier like the ImageNet one that I showed you before where you're showing it pictures and it's saying, oh, that's a picture of a Dalmatian and some cherries or in this case, this is a picture of a panda and it turns out with these classifiers do really well on these problems. So we go back here and we look at the, where are we? We look at this score history for ImageNet and of course my computer's gonna take a long time to load that so I'm gonna let it do that. We see this score history. This looks like a really good classifier. It does better than a human at recognizing pictures of dogs and pandas but there are certain inputs that you can make to this system even though it does really well that cause it to produce the wrong answer. Okay, am I losing my slides here? We go back here. If we feed in the right input that looks like noise and it's imperceptible to human vision so this static field multiplied by 0.07 so less than 1% of the data basically this is like plus or minus one pixel value in an 8-bit, one bit value in an 8-bit image. The final image is no longer classified as a panda it's classified as a given and how is that possible? Turns out that you can just find values that go into these matrices that are the exact gradient from panda to given and you can do this either by pulling apart the matrix and looking at the values inside it and studying it or you can do it online. So if someone deploys a self-driving car and it has a pedestrian detection system in it you can just buy one of the cars and walk in front of it with a few different images and see which ones it reacts to as though there was a pedestrian present and which ones it reacts to as though there was no pedestrian and with maybe 50 or 100 examples you can find a pattern that causes a pedestrian to become invisible and so I think people are worried about okay, will it be possible to stick a sticker on someone and suddenly the self-driving cars will no longer be able to see them? Will it be possible to put a sticker on a building and suddenly the self-driving car thinks it's an open road and then it'll just drive into the building? These things are likely to be possible unless you mitigate them. The default is this attack is present. Some of the mitigations that are being proposed there's a project called Clever Hands that isn't a mitigation itself it's basically a way of testing whether you did successfully mitigate by firing these attacks against your network and then scoring you on your train against the attacks and you get a little bit better but that doesn't tell you that someone might not find a new algorithm for generating attacks that still works. Second kind of mitigation is to instead of just classifying as panda or given to really work hard on the problem of how confident am I that it's a panda or a given? If you do that, at the moment you can make the attack more expensive. It might take 10 times more examples before the attacker can succeed against you but right now this remains fundamentally an unsolved problem. So stuff to work on. Another example of a really interesting problem to work on is reward hacking. So here we're talking about neural networks that are like reinforcement learning agents. I don't know how many of you've seen examples of neural networks playing Atari games. I should probably include a demo of that so let's jump over here. It turns out people have gotten these neural networks to play Atari really well. At first they miss everything but after a little training they get better. And after enough training they become totally unbeatable. They will never miss the ball and learn that they can hit the ball up the side of the thing and get scores that massively outpace humans on all of these Atari games. There are maybe two or three Atari games left that humans are still better than machines at but there are some really weird problems that you get along the way with these reinforcement learning agents that are gonna stop us and okay it's fine to put a reinforcement learning agent in the Atari universe but it's not yet fine to put one in the real world. Why? The problems boil down to you tell the thing to play Atari and its aim is to maximize the score. Okay that's simple and fine and it's safe to put that in a box but what is the score for a house cleaning robot? If you tell it the score is how much dirt is in the room and you tell it well okay how are you gonna measure dirt? Well it's the amount of dirt on your dirt sensor. The agent is gonna proceed to do a few crazy things like one day it's gonna drive really hard into the wall and it's gonna break its dirt sensor and then it's gonna think that it cleaned the apartment and so it's gonna learn to always do that as fast as possible. It's gonna like slam straight into the wall and break its dirt sensor. So you're gonna need to figure out how to encode oh I want you to like always get accurate information and then maximize this variable. It's gonna try things out because in order to learn in a new environment you have to try new things but sometimes that's gonna mean crashing into your Ming vase and knocking it over and breaking it and then the agent's gonna go whoa that made a terrible mess there's Ming vase pieces everywhere I won't do that again but you've already lost your Ming vase. Sometimes it's gonna turn out that it was trained in a house with a pet of a cat, a dog, many different kinds of things but then it meets an iguana for the first time and it's gonna try to clean the iguana and you can't anticipate all of these problems. So people are starting to work on how to solve this so here's another example of an Atari game where you've actually got a model of the broken behavior. This is from Owain Evans' work. So here we have this Road Runner, it's this blue squiggle for those of you and the coyote chasing it for those of you who didn't play Atari 2600 games when you're kids and the Road Runner has to dodge the truck and there are little blue food dots that the Road Runner is trying to collect. And so as the reinforcement learning agent learns to play, it gets pretty good. You can see it's getting all the food and dodging the truck, staying ahead of the coyote. It's getting points, but this is just level one. And when it gets to level two, it doesn't know how to play so well. So what it does is it learns to kill itself to always stay on level one. So people are finding these situations and then they're having to say, okay, how do we make Road Runner go to level two and struggle to learn level two? These authors of this paper figured out that they could use human supervision. So they actually have a human who's sitting there giving feedback to the agent and then they actually have a neural network that's watching the feedback and learning to predict the feedback. So you have this multi-stage process of trying to train a separate network to learn not the score, but what the human would want the agent to do. So this is a really interesting brand new area of research. There are like two or three papers on these problems, but I actually think if you go back here and look at these, these things are fundamental and hard algorithmic questions and there's gonna be a lot more than three papers to be written here. I think there need to be more hackers and more security people working on this stuff. Now I'm gonna jump at the end to a little bit of speculation about AGI because AGI is super fun. The science fiction AI stuff where we have real flexible thinking machines like us, this is super fun to think about even if it's hard to think about well. And so this comes with a giant disclaimer. Anything that I say about this topic is basically more for fun than something you should take too literally because with futurology, your odds of being right are always not that great. But interesting to think about, interesting to think about what mitigations we would have to put in place for some of the more plausible risks. So imagine you have a human-like AI. And I think this is the first assumption that's probably wrong. If we have AI, it's AGI, it's probably gonna be very alien, very different to us, but imagine that it was like us. So about human levels of cognition, each AI is independent of the other AIs and it lives in a computer somewhere. If it did, it would care a lot about privacy, more than humans care about privacy. Why is that? If you are a piece of software that thinks like a human, your memories and intentions and private thoughts can be copied because they're just data. They can be stolen because they are just data. They can be modified by an attacker who gets access to your hardware or your code. And this can happen without your permission if some malware gets onto your machine or the permission of your owner if you're working for someone else. There's another interesting ethical question we'll have to deal with. But focusing on privacy here, basically for AIs of this sort, crazy magic stuff is possible. Being, having your soul stolen by someone else who makes a copy of you, puts you in a box, starts messing with you and modifying you. This doesn't happen to humans, at least not in simple, straightforward ways, but given our current knowledge of what this technology would be like, we would expect this to be possible. This is the likely prediction is, AIs will have to worry about this. And so privacy for AIs will be a much higher stakes thing than it is for humans. It'll be protection against having this done to you. So we might wanna think in advance, before we build this technology and deploy it, how to guarantee this kind of privacy that these beings would need. Because if we don't do that, we build a very unstable world, right? We deploy a technology where this is possible. Who knows what the hell is gonna happen next? So I think we would have to think in advance, what kind of hardware would we need to provide a neural network with guarantees that its data can't be copied off the board? You're running a GPU or an ASIC that has a neural network in it. Should it have some hardware security features that let you click flash and then lock the data in there? Should you have, as an EFF person, I'm not used to advocating for DRM, but maybe we should think about, okay, for the stakes being this high, maybe there should be some DRM-like features where you can guarantee, at least locally, maybe remotely, that the copy of the thing you put on this hardware is what you originally intended and that someone hasn't broken in and put something different in that GPU. We should also think about the political questions that would underpin this. Like, what rights would we want to guarantee to AIs? Would we want to give them a right to privacy, a right to autonomy, or the exact opposite? Would we want to try and constrain them? And maybe we want to give them privacy, but only once they demonstrate a certain level of maturity, right? Once you have AIs that you're fairly sure have gotten past these problems, then maybe, you know, it's like you have children and teenagers for a while where you're really willing to interfere with their lives, but they get to a certain level of maturity and you say, okay, now you're an autonomous being, go out in the world and we trust you. But there'll be other questions, like when do AIs get to copy themselves? When do they get to have children? Because humans, it takes like 15, 20 years to produce another copy of yourself and a lot of work. For an AI, it's just like, okay, SCP myself to a million machines. So I think we're gonna, if AGI happens, we're gonna have to wrestle with these questions. Anyway, that's all I have in my slides. I think we have time for a few questions. I'm just gonna leave the caution symbol up here to remind everyone that this is a dangerous future. Thank you, Peter. There are two mics open in the middle. So stand up and pose all your questions. Everyone is scared? Okay. So the attack techniques that you were mentioning, the defense techniques seems still to be statistical. You're exploring a state space. How familiar are you with formal verification based attacks and defenses? So formal, I'm a little bit familiar with formal verification. Formal verification of neural networks is going to be tricky because they're so big, right? We're talking about enormous matrices of weights that have some, they're often not too incomplete. Like we're just talking about a finite state machine that's enormous. And so formally verifying one of these things, I think that's an interesting research question, but you're up against, it's just a gigabyte of matrices and how do you get enough computational power to produce proofs about its behavior in a very, very wide range of situations? I do think that this whole, if AI raises the stakes for computer security in general, if we have intelligent malware that's running around breaking into things, I do think that having more formally verified systems around to begin with is a great idea. Like one of the ways we build a safer future, kind of maybe regardless of AI, but especially because of AI, is to have formally verified operating systems, formally verified browsers, formally verified compilers and to build our next generation of technology out of them. On the back. It seems to me that a lot of what you talk about is still focusing on the physical manifestation of AI lives in a machine. But one of the things that I've seen happening and also with the Bitcoin phenomenon and other stuff is that we have this large robot with humans and computers and economy and that kind of stuff going on inside. It's a, yeah, in one ecosystem, which I tend to look at like an anthill or something like that. It's research going on because I think Facebook is trying to explode something like that, human and the machine making that together, some sort of AI to make money. Is that something that research is being done on? Well, I would say a little bit, right? You know, I had this disclaimer slide where I said, okay, if AGI, so the kind of sci-fi AI that doesn't exist yet, was human-like, then here are some things that would follow. And I think that people who are studying this question, there's not that many people yet, get pretty uncomfortable with all of these weird implications. And so they're looking for ways to not have AGI look like this. And one of the ideas that they have that people are batting around is the idea of encouraging AGIs to rely on a human partner. So instead of having a totally autonomous AI, you have one that needs to work with a human for objectives and ethical judgment. And the idea is you'd have a kind of pairing of the two together. And this is probably a really good idea and there's some precedent for it. People who follow computer chess would know that there are these things called centaurs that are not an algorithm or a team of a computer and a human together, and centaurs are slightly better at chess than pure computers. So you could imagine, okay, let's build our AIs like they're centaurs. I think the tricky thing about that is, it's gonna be hard to know, it's gonna be hard to stop the development of the technology at that place and not go all the way to fully autonomous. This sounds like it firmly has to lean onto the law. Sorry? It seems like this is predicated on law functioning. You know, the human world. Perhaps, or we might be able to think of other incentive systems. We could make a software license that says you can use this AI, but only if you have a human there, then I guess the question is, is that enforceable? Maybe not. I believe most of your talk depends on the question of whether we can control the intention of our machine learning algorithms. The intention? Intention setting, because you give it a training data, you give it a performance metric, and does this align with what we intended. But I feel like by the time that we get to actually dangerously intelligent, we probably have a good understanding of how to set intentions. Because setting intentions is actually the way we make these things intelligent in the first place. I'm actually more worried that we'll figure out how to set intentions for humans on the way there. Sorry, look, I think it's probably a... Yeah, and you're with the EFF, so your focus on privacy, for me it also relates to personal autonomy, being able to be sovereign. When someone knows all about you, you cannot really defend against intentions. I think that's actually two really good questions. So I'm gonna answer the second one first, which is how vulnerable are humans to essentially political manipulation? You call it intention setting, but I think we see Facebook doing this right now. They're optimizing their algorithms to make you spend as much time on Facebook as possible, and the results appear to be you can indeed get people to spend a lot of time on Facebook, maybe more time than they want to. And they're also, I guess we're seeing these tests of how effective are these algorithms at manipulating people's political views. And I think, unfortunately, we have live laboratory experiments running in all these elections right now because political campaigns are using these techniques. Now, wearing my EFF hat, I think we don't yet know what the answer to this problem is, because as an organization that believes in technological freedom, we'd have to think very carefully about passing a law that says you can't, you know, Facebook, you can't run certain algorithms on your servers. We'd need to figure out a very careful principle, you know, like set of principles for what does transparency look like in this world? What does accountability look like in this world? How can Facebook's users be informed and have a say in what Facebook is optimizing for? And so I think that's a question we're still grappling with. And people in this room should be grappling with it too. Then I think the first question you asked was about whether we can teach intentions to AIs. And I think this slide is about this. There's also a paper for people who are interested in this. The great starting point on this is a paper called Concrete Problems in AI Safety. Google that now, if you're interested. It discusses these problems and sort of puts markers in the literature for other people to cite, saying, how do we go from a simple naive description, like a score in a game, to something that looks more like the complicated, okay, I'm going to clean the apartment, but I'm not going to knock over the Mingvars and I'm not going to destroy my dirt sensor and I'm going to understand that there are other things that are important besides just cleaning the apartment. So I think we're going to keep struggling with this. And if we solve it before AI, I think that's great, but I don't think it's a trivial problem. In the front? Yes. So on the topic of bias, you were talking about, well, these examples of discrimination, basically, and having a machine learning algorithm pulling out, sorry, irrelevant data to make decisions. I guess my question is, should we, or could we try to avoid that bias entirely given that there's a lot of data to pull at and it's going to find some way to reach a decision? And we do have some bias already, like for the example of car insurance, if you're a young man, you're probably going to have a higher premium than a young woman. And that's a kind of bias that we are okay with. Yeah, I'm not sure that we should be totally okay with that. So, and I think we're gonna, like society's gonna wrestle with this question. I wouldn't be totally surprised if, especially like either in Europe or the United States, we started to see some regulation in this space. And that regulation is gonna force us to think more clearly about racial bias than we have in the past. Like your point about young men and car insurance is exactly right. And I think the answer is it's okay if young men on average get higher risk predictions because when they're put behind the wheel of a car, they drive faster and less carefully and they drive under circumstances that are less safe. But I think the fairness criterion here is if you are a young man who is a safe driver, do you have the same odds of getting car insurance as a safe driver in any of the other categories? And if you're not being treated fairly in that sense, I think the algorithm is doing something wrong. It's prejudging you. This is the origin of the word prejudice. It's like prejudging you by some characteristic that's not really reflective of who you are. I have a follow-up. Regulation, is this going to come from some startup in San Francisco or is it going to be maybe government regulation in Beijing who gets to decide all this? Great question. I mean, I think some of this is coming from, I guess I was gonna page over to that Google I'm not actually getting the right Google paper here. That Google infographic that shows those algorithmic mitigations for bias, I guess you could say it's coming, I mean Google's not really a startup and it was a bunch of academics who did that work. But yes, I think you're getting the source of the algorithm from a fairly technical hack of community, but I think if it was going to be required by regulation, it would have to be traditional regulatory bodies that would do it. There could be some industry leadership, I guess the partnership on AI is this organization that EFF has involved in actually that was an industry association formed by DeepMind and OpenAI, Microsoft, Amazon. The interesting thing about that is I think there's some genuine concerns amongst those companies that this technology is a little dangerous. I think it's not a case of them forming a regular like an industry body to just try and stop regulation which is usually what industry does. I think here they're actually saying, we might even need to work together on some of these problems because they're hard. And so I'm for the moment cautiously optimistic that we could get a constructive dynamic here. We'll see if that really plays out. The mic in the back. Okay, first thank you for the presentation. Every slide made me come up with new questions so it was really enjoyable. Then a little heads up, I'm an insurance broker by trade. So this kind of rang a little few bells. Insurance companies are an easy foe. Insurance is not sexy but is necessary, useful. Maybe you can do it without the companies but the process of redistributing risk is socially useful. The few examples you made are good triggers. I understand that at the beginning of a presentation like these it's what makes the audience start thinking but I saw massive opportunities for arbitrage there. Like if you use the AI in such a silly way, please tell me who you are discarding. I will make a lot of money. Not me personally but I'm just saying this description is probably more related to the state of the tech today and will self correct quite easily. The problem I think is little different. Once you let the AI drive the whole process, the economic process and you give it a good mission it will probably find a good balance for the insurance and if the mission is well described we avoid the situations where you stay at level one killing yourself which an insurance might become something really, really, really nasty. The point is that like a question earlier about setting intention that is extremely hard and I think that is not only a tech problem. It becomes now a tech problem, that's interesting but it's a problem we haven't solved in the real world. Let me explain what I mean. You said AIs may need to have some human rights about being copied and the right to copy themselves et cetera, that is the kind of thing that we should start looking into as humans. It was interesting at the end, you touched that point. That's the kind of decision that we have to do every day when we share information. I think something was, and it's interesting when it's back to insurance, was said earlier that was not correct. If a 21 year old male on average gets a higher premium than a 21 year old female on average, that's because of statistics. It's not because of somebody thinking why the male kills more people driving. Actually that's not legal anymore in Europe and the net result is that once you take away gender from the pricing, females pay more because you have to average out all the payments. So going back to the point, I need to be informed of what will happen when I share information to my premium. That's information asymmetry though, because if I'm a good driver, like you said before, I will want to tell the insurance company monitor me. If I'm a bad driver though, and I'm not forced into disclosing this, I might be able to get a better premium just by not saying anything. So... Right, and this is the point that I was trying to make with the interaction between privacy and machine learning. Exactly. So... I think that you're correct. There are a lot of opportunities for arbitrage. If you're an insurance company that wants to enter the market, all you need to do is add some more data to your data sources and you get rid of some omitted variable bias and suddenly you can be the insurance provided to all the shift workers, for instance. Exactly. Or all the young men who happen to be safe drivers. You get them to pass a few driving tests and you realize that they're actually really safe and now you give them cheap insurance. But the point there for the EFF is, how do you teach people? This is not a tech problem. It becomes one and then you have the issue. But how do you teach people when it's in their interest to disclose information and when not? The AIs are bringing a new bias here. My long question was to get to this point. We still haven't solved the problem of individual agency. Like how do you set intentions for humans is still a very relevant question. I don't know that I can solve the problem of individual agency in this talk. I think that... I really wanted to just point to this interesting privacy problem because I think that people who care a lot about privacy will get caught in the same bucket as the people who don't want anyone to see how bad a driver they are. Simple question. Debiasing algorithms. Do you know any efforts that are varying open sourced outside companies like Google and Facebook? Ideally with women of color or people of color and leadership? Yeah, so all these algorithms are in the open literature. And Google, I think... It's not that these are Google's algorithms. It's just that Google has been promoting information about them because I think they view that as constructive. So this is the URL I was looking for. This is basically a data visualization project that some people at Google put together to document open academic research. And what it lets you do is play with the de-biasing algorithm. So it's got an actual dataset with some of these omitted variable problems and you can click through and pick these different levels of mitigation and compare under the different levels of mitigation how much accuracy do you lose? You lose a little bit of accuracy by de-biasing. But then how much fairness do you get in exchange for that small hidden accuracy? So I would recommend this page as a starting point. Is this correlated research from open source efforts? Yes. Okay. Yeah, going back to the open source approach with TensorFlow and Torch and all these libraries, do you think there's going to be like a palliative because there's no actual open data to train? So like Google and Facebook, they have massive amounts of data. So there's a subtext to this question. So the question is, will TensorFlow and Torch, which open source libraries, nonetheless be essentially at a disadvantage because the missing piece there is the training data. And it's worth understanding that with the current generation of machine learning algorithms, you need a lot of data. Take however many examples a human might need to learn something. These algorithms are going to need 10 or 100 times more examples to learn from because they're not as good yet. And in particular, they don't have the ability to generalize from previous different cases to the current thing in front of them. People call this transfer learning, it's not solved. So right now, the big tech companies have a huge advantage because of all the data people have given them. Facebook has all these photos and all this data from people's feeds. Google has all of Gmail and all of the search data. And so they're using that data to train models. There is a really significant open effort as well. So those companies themselves often find it in their strategic interests to publish large data sets. You see that with Facebook's Babbi data set, which I was showing the questions and answers from Google releases data sets, DeepMind releases some data. Like all of these people do this to help them hire, to help them point attention at problems they want solves. And universities also running around, university teams are running around producing data sets. It's definitely to your advantage as a machine learning person to work for one of the big companies with a big set of data. What we don't yet know is what this is gonna look like in 10 years. If progress in these algorithms happens, is that gonna mean that data becomes more important, equally important, or less important? I think there's a fair chance it could be less important. Humans don't need huge amounts of data to learn from the world. We're able to learn by walking through it. And the combination of that fact with the fact that there is a lot of open data to kickstart your models with makes me a little bit of an optimist that open data may be enough to train really good AI. So people have been making decisions for a long time. Like, you know, I'm not gonna help you because you're a Samaritan. Like, you know, I'm not gonna send you my catalog because you're in the wrong postcode. This was all before computers. I'm not gonna rent you this car because you have the wrong last name. Aren't we gonna be in a much better world with all of this when we have a much more anonymous way to look at things that actually matter? So that's a great point. And this is the main argument that people who say deploying criminal justice risk scores in the United States say was, well, look, we may be terribly biased, but so are the humans who are making the decisions beforehand. And I think that that's a possible claim. I just think it needs to be backed with data. Like, I think if you're gonna say, well, humans are biased, so these algorithms that are also biased, like, slightly less biased, I think you need to be able to say, and we measured it. And we can actually show that we were reduced biased by half. If you do that, then, okay, fine. That's like a robust argument. But I think humans, the whole point is that, you know, say a panel making a decision about whether to grant someone parole and let them out of prison, those panels are actually made up of people who are really good at judging the character and behavior of other people. Like, that's what they spend their time doing. And humans are not that bad if we train for it. So I'm not, and yes, the results will be biased, but I think maybe less biased in some cases. And I think there's also a really good corollary there which is you can probably detect which humans are doing a really good job of that task and which humans are leaning on bias as a prediction tool, and start to encourage the humans to do a better job if you use data. So I'm just mindful of the time. I think there's another talk starting here. Should we break, is there a space where we can keep chatting if people will have more questions? Yes, I can like explodey, for example, or one of the workshop tens. So any more questions? No, then the final round of applause for Peter.