 Hey, everyone, I'm being socially distant. I'm David Roncek. I'm head of open-source machine learning strategy here at Azure, and I'm here to talk to you about statistics and what they can do to attack you. Specifically, machine learning and the way that folks need to start thinking about machine learning from a security prospect. With that, let's begin. Okay. Microsoft has been in the security business, excuse me, in the ML business for a long time, and this is something we're really quite proud of. We've done enormous work in vision, in speech, in language. You can see some of the benchmarks that we helped launch recently, and of course we do give them all back to the open-source. But most of all, we use them in our products as well. Virtually every product at Microsoft has some form of machine learning included, from consumer to enterprise, open-source applications, operating systems, gaming, Office, and of course Microsoft Research. ML interacts with nearly every one of the businesses that make up Microsoft and Azure. The reason we have to do this is because we need things at such enormous scale, there's really no alternative but to use ML to accomplish it. In this case, we have over 180 million monthly active users that engage with Office ML tooling. 18 billion questions asked of Cortana, and 6.5 trillion security events are analyzed every single day on behalf of our users. The numbers here, the amount of data is in fact so large, that there's simply no way that we could accomplish any of this without using tools like machine learning. That's great, and that's something certainly that I love to talk about, talk about all the ways that we're doing it. But when you look at the average user, and what they're interacting when they see ML, they see all these great headlines about how machine learning is changing the world and bringing things forward. Yet on the other side of it, they realize that ML is really hard. It's a lot of really challenging math. It's very complicated models, it's hard to get right. One of the biggest reasons that it's so hard to get right is because so much of the time, the news, the articles and things like that, cover this, the building of a model. Don't get me wrong, building out great model is very hard, and it is core to the overall machine learning experience. But it's not the only part of the overall machine learning experience. The reality is that most machine learning exercises look much more like this, where there's long chains of various microservices built together that are designed to solve domain-specific problems, whether or not they impact data, validation splitting, how you train, how you roll out, how you measure the rollout when it's reached production, and then how you take all that information and fold it back into the original. That is what makes up machine learning, not just building a model. And so let's say you're a data scientist, you're like, well, no, really, I only care about building a model. But the reality is you care about those things too, because if you don't focus on how to bring that model forward into production, you have tweets like this come about, where it takes three weeks to develop the model, which is great, but then it takes another 11 months to get the thing out the door. And that's obviously not ideal. Data scientists, just like anyone else, wanna see their tools being used in production and not just being used locally on their own machines. So the answer to this, that the community has begun to snap to is something called ML Ops, Machine Learning Operations. And the idea here is to merge these two disciplines together. On the left-hand side, you might have a machine learning iterative loop, where someone is going through all of those steps, acquiring the data, building it into featureization, tokenizing it, then building the model and going through and making sure that the accuracy and the other statistics are there as you would want. But then you fold them into your standard GitOps or DevOps pipelines, where it becomes part of your overall source tree, you build it into applications, and then you roll it out using the same tools you would use to roll anything else out. And this kind of integration is incredibly important. To date, most often, machine learning and data scientists have really been left out on an island. They stood in that first circle and when it's time to move it forward to development of inclusion in the main application or rolling out to operations, it's throwing it over the wall and hoping that it works. If you do use something like ML Ops, you actually get a whole bunch of benefits. You get automation and observability, being able to know exactly what is ending up in production, including things like reproducibility and verifiability. You get validation. If that code actually becomes part of your overall validation framework, you're able to do things like AB test, rollout canaries, you're able to do explanations on that code and make sure that it's avoiding bias and so on. And then finally, you get reproducibility and audibility because the only thing that made it out to code was the things that were checked in to your overall source tree and pushed through a pipeline, you're able to make sure that you know exactly what your customers saw. Instead of having someone SSH into production and drop a random binary on there, you know that the only thing that had permissions to roll out was this centralized pipeline. And all of this together combines to give you velocity and security, but for machine learning. So with all of that as the groundwork, you may be asking yourself, wasn't this supposed to be a talk about security? And in fact, it is. Machine learning operations, ML ops is the baseline for security. That you're saying to yourself, but it's just math, machine learning is just math, it's fancy statistics, how bad could it be? We're gonna talk about three attacks that have come forward in the past few years and how you might be able to use something like machine learning operations to defend yourself against them. So the first is your attacker gets your machine learning to lie to you. How do you avoid this? Well, we're gonna go through and build ourselves a very simple system here. This is gonna be a circle detector, a very important machine learning experiment that is designed to solve a critical business case. The first thing you do is you start with something like an ingestion step and then you take it to an engineer step. So in that ingestion step, you're gonna be pulling in a large amount of data and you're gonna wanna split it. You're gonna wanna have a training set, a testing set and then a hold back set. And you do that split very intentionally because what you don't wanna do is you don't want to train on your test or your verification or hold back set. You always wanna have those completely separate from your overall process. You take that, you move the training data into your trained pipeline and then once it's done training, you verify it against your test or your hold back sets and then you roll it out to a serving endpoint. So this is very common prototypical kind of machine learning pipeline here and this can all be driven by something like ML Ops. So we're gonna build a circle detector. Well, like I said, the first step is you're gonna go out and you're gonna collect a whole bunch of examples of circles. Here's a whole bunch of different sizes, different colors, different positions and you throw it in there and you build your great model and you say you train and then you present that circle and you say, hey, is this a circle? And the machine learning model says, yes, fantastic. We're done, right? Unfortunately, of course not. So now I'm gonna come along and I'm gonna present it with a new example. In this case, I'm presenting it with this thing and it looks like a square to me, but I don't know, let's see what the model says. Model says it's a circle. Well, that's crazy town. What did I do wrong? These look very, very different. One is round and blue, the other one's green and looks like a square. How could this possibly be mistaken? Well, it turns out that if you look at a square, all the pixels that make up a circle are inside the square. All right, well, that seems pretty straightforward to me. That's why the machine learning model ended up figuring out it's a circle. What I didn't do as part of my training was identify all these bits around the edges here that are not a circle and inform the machine learning model that that was the case. And so what I needed to do was not just have a bunch of positive cases that showed that this was the case, but I also needed to show negative cases where this wasn't the case and hand those both to the machine learning model in order to get an accurate model. Okay, so obviously that's just a toy example. Real advanced models are much better than this, surely. Here is a wonderful paper. And by the way, during this process, I've been incredibly lucky to read so many interesting papers. Don't feel like you have to all write them all down. At the end, I list every paper that I mentioned here in a big long list for your screenshotting pleasure. All right, so here is a paper came out of UCI, I believe, where they did a Wolf versus Husky detector, which is obviously quite critical if you want to keep your sheep safe or reindeer, as the case may be. So in this case, they presented a series of wolves and Huskies to the detector, and they only got one wrong. That's pretty good, right? This is a pretty complicated thing. How are you going to figure this out? Well, it turns out that they came along and created an entire framework for identifying what's going on here and it's called explaining what's going on and as you start to look at it, you're like, hey, what is happening here? When I go and look at the Husky picture, it's actually got pieces, pixels of the Husky that make the difference here. Here's another one. But when I look at the wolf, the funny part is the wolf makes up almost none of it. It's all these little bits around the edge. What happened? Well, it turns out that we didn't detect whether or not there was a wolf in the picture. He detected whether or not there was snow in the picture and because the correlation between snow and wolves was so high in these pictures, that's what the machine learning model determined was the right thing. So, and we're just getting started here, right? Again, that's where you have explanations. What happens in something like this? Tumblr recently came along and wanted to start excluding not-safer work content and their models had all kinds of things very publicly, unfortunately. They had a whole bunch of false positives. This is fried chicken, or excuse me, raw chicken here. This is a cartoon of a unicorn and this is boot cleaners, which they determined were false positives. They also found a bunch of false negatives. The green here is a substitute for something that is actually not safe for work. But when you took that and you layered on top and something that was totally orthogonal, in this case, just an owl, the machine learning model trained on the wrong thing. And so, adding this without this, it said no, but with this, it said, yes, that's clear to go. And that's another example where simply adding something on top now makes it much harder. And now I think you can see where the attack vectors start to come in. Because what you're able to do is, if you are able to figure out the underlying nature of that model and figure out what really matters to the model, if you can figure out like, oh, really what it cares about is snow, not the wolf, then you can start layering in additional information that starts to fool the model and starts to do bad things. So in this case, for example, you have a stop sign and some researchers have layered some real world stickers across it. And in this case, they have a turtle where they 3D printed some additional coloring on top of the turtle's shell. And in this case, because they layered these stickers on, what was being detected as a stop sign is now being detected as a speed limit sign. And this turtle is now being detected as a rifle. And again, no model is perfect. You're always gonna have mistakes, but this kind of stuff starts to get into really serious issues. Here's another wonderful paper by some researchers at Google where they took standard pictures of researchers. In this case, this is Jeff Dean and these are the two researchers who actually worked on it. And simply by layering over the top these glasses, this is all done in computers, not in real life. Excuse me, it's all done digitally. It's not, these aren't physical artifacts. They were able to convince the machine learning model that they were in fact completely different people. Jeff Dean became Mila Josevic. This researcher became a different researcher and this became Carson Daly. Now, while we would all like to be Carson Daly and Mila Josevic, I think we all can agree that this is a pretty bad security implications. And it's obviously not just for, kind of neutral situations like this. You may have seen a lot of stuff in the news recently about face detection algorithms and identification. And it falls into giant classes. In this case, Amazon's face recognition system falsely matched 28 members of Congress with mug shots. And that kind of stuff happens all over the place. And the worst is it's not just something that you opt into, in this case, you're at an airport where you actually can't opt out at all, which is obviously terrible. And again, just to take it even further, there's a technique out there called federated learning where the idea is you're starting to be able to provide privacy and not have to pass back an enormous amount of data to that central server to do your training on. But in fact, security researchers have even begun to attack that as well and show where their flaws are. In this case, the way it works is pretty straightforward. You'll have your training step, but instead of doing all your training here, where you had to input all your data, what you'll do is you'll break the model into a bunch of different pieces and hand one of those pieces to the individual. They will train on their local device without passing any information back and then they'll just pass the weights back. That's my visual pun there. They'll pass the weights back to the central server to aggregate and use going forward. And then that central server aggregates all those weights together, creates a model and is able to push down that model. Now, in a lot of ways, this is fantastic, right? None of these three individuals passed back any private information that did all their training locally and the central machine learning model didn't know about that data. So it's able to pass back an improved model without knowing anything about the individuals. Unfortunately, what happens when you get a bad person here? So here on the left-hand side, we have that evil doer who has swooped in. They get passed a model and instead of passing back accurate weights, they're gonna pass back something that they've figured out how to poison that central model. And then it doesn't just poison them, but it in fact poisons the entire model. This is not great because now these people who did nothing wrong and are off using the model on their own don't realize they're using something poison. Worse, the central authority here, the training system, doesn't know anything bad either because it had already broken all the model up and handed it out. This is why we can't have nice things. So there are additional techniques that are coming along to help detecting these malicious clients. Here's a fantastic paper by some of the folks at WeBank. But the reality is that you're never gonna be able to stop everything. There are lots of tools here. You can build in more edge cases. You can detect bad data. You can use better evaluation metrics. Don't just look for snow, look actually for wolves. You can use different models, things like one-hot models are much more defended against this because they're not as granular. Additionally, you should really be attacking your own models first. You should be doing a lot of alerting and monitoring. And most importantly, you wanna do continuous training where you pull back all the findings that you're having in production back to your original ingestion step and augment that as an additional set of features. Now, with all that said, there are some pretty big rules here. For as much as you're gonna augment it, you really, this second point is really what I wanna stress. The more impact your model is gonna have in the world, the more scrutiny you need to deploy against it. Meaning, if you're just saying, whether or not something is a circle or a square, not the end of the world if you get it wrong and someone can ask the question 10 times and so be it. If you're detecting whether or not someone should go to jail, that needs to be looked at with a whole ton of scrutiny. But what I really wanna get at here is the only way to build this in a repeatable way, excuse me, in a reliable way is to build it using a fast and modular pipeline. So let me show you what that might look like and why this gives you benefits. In this case, building a pipeline goes to three steps, basically. First, you want CI CD. Most folks have this already. You can use something like GitHub Actions or Jenkins. Second, you add in modular components. You want them to be very loosely coupled so you can add and remove them as it makes sense to you. Monoliths here are your enemy. You really don't wanna do that. And you can mix and match, and I'll show you how to do that in a little bit. Don't think about this as all just one thing in one cloud. Mixing and matching is perfectly okay. We also have an entire array of actions for doing this on mlops-github.com, and you should go check that out. And then third, and this is really important, measure, measure, measure. I cannot stress that enough. The more that you measure, the better off you'll be. And it's up to you to catch these things before you're on the front page of The New York Times because your models will go stale quickly. So the more that you continuously train, the better off. So to lay this out in a kind of rough architecture format, let's say you start up here with something like Visual Studio Code, and you're just gonna do that iterative loop that I was talking about earlier. You check that into a code source repository in this case, I'm gonna highlight GitHub, excuse me, but it could be any codes or surprises here you like. And then you plug that into your overall CICD system. In this case, we're gonna talk about GitHub actions, but anything that allows you to have that microservice-oriented action and basically have triggers from code will put you in a great spot. From there, you're gonna kick off your pipeline. So we talked about already processing data. That might occur on your CICD runner. Maybe you do something big, like feature engineering, and you're really gonna wanna do on something like Spark, which has its own cluster. It's completely okay to have that outside of your overall pipeline. In this case, this feature engineering step reaches out to Spark and executes that pipeline on Spark. And then when Spark is done, it hands back to that course thing saying, hey, you know what, I'm done, and maybe it hands back the data. Maybe it just hands back a pointer to the data. Oh, I'm being hosted on this Azure Blob or this GitHub, or excuse me, this S3 bucket or on this NFS file share. We might do the same thing for the actual training. You certainly don't wanna do that in your runner itself. You'll probably wanna do that on an external cluster or a hosted service like Kubeflow. And then maybe you go through packaging. Packaging is small. It's throwing it into a container. You can do that in your runner. And then it comes to serving. And in this case, we're gonna use a hosted service because it's globally distributed and five, nine sub time and so on and so forth. We'll use something like Azure Machine Learning, but again, could be any cloud that provides it. You'll also wanna take all of the inputs and outputs here and store them in an immutable metadata store so you can always go back and figure out what it is. But this is it. This is really the standards of a pipeline. And this is something that folks are already doing today. Very, very powerful stuff. And it just matters that you wanna bring that same sort of learning that you have from all of your other application and development rollout to machine learning. Now, what is one of the benefits here? I mentioned is the loose coupling, meaning you can add and subtract things wherever you like. Again, here's our standard pipeline. Let's say I wanna go out and implement an explainer, like that thing that it showed. It was the snow that was being detected, not the wolf. Well, if this was a monolithic thing, I would have to go and tear apart one of these components and figure out what was going on before I could move forward. In this case, because I built it, thinking about these as microservices using external things to layer on top, I'm able to just add that explainer. In this case, I'm running it on an external machine. And I add it, add the start and finish into the overall pipeline. And so without doing anything significant, I'm able to just pass it from one step to another using the standard flow that I've already done. Cool. So that's the, you know, you get your ML models to lie to you. Let's talk about some even more terrifying things. In this case, an attacker takes your model. Now, what a malicious user is trying to do here is they're trying to reproduce the original model. And basically, you know, the first goal is really just private access, meaning if they can make a copy of her model with some degree of accuracy, 70, 80%, that's more than enough to begin, you know, further activities, right? Maybe they want to do a more complete extraction later after they have a lot more information about the model. Maybe they want to extract private information that you've built into the model. And I'll show you how some of that stuff leaks through in a minute. Or maybe they want to use this model to construct those adversarial examples that I showed you earlier, that, you know, being able to build out what a turtle should look like or how to, you know, make Jeff Dean look like Mila Josephic is a really hard problem. You need to, you're going to have to execute hundreds and thousands and hundreds of thousands of queries against your model to do that. If you're doing that against the public endpoint, that's not great, that's much more easy to detect. Now the reality is extraction is really hard to defend against. But let me show you how this works and then we'll talk about what you can do around it. So I'm going to talk about two main attacks right now. The first is called a distillation attack. And the second is called a model extraction attack. So a distillation attack is really straightforward. The first thing you're going to do is, let's say you have a model over here on the left-hand side. It's a black box model. We don't know what's inside it. In this case, for the sake of argument, the attacker doesn't even really know what it's supposed to detect. So in this case, we're going to start handing examples. In this case, first I'm going to hand it a heart and it says, oh, that's no. Now I'm handing it this Pentagon and it says yes. All right, well, let's beginning to detect something. So as an attacker, the first thing I'm going to do is start handing it a lot more. And I'm starting to give the shape of roughly what it's trying to detect. But really what you need to do is hand it a whole lot of examples. And so now we have a pretty good picture of what it's designed to detect. Anyone want to take a guess? The answer is it's designed to detect Nina Simone, singer and civil rights advocate. No, in fact, it's not. It's, of course, designed to detect a triangle. But now that I've collected this as the attacker, I've collected a whole bunch of examples of where it works and where it doesn't, I'm able to reproduce the model in a pretty accurate way. And now that means that this thing that costs this original organization millions of dollars, hundreds of hours to go and reproduce, I am able to go and use it for my nefarious purposes. Maybe I create my own endpoint around it. Maybe I use it to do some of the things I talked about earlier, extract data or attack or various things like that. But regardless, I don't need to touch that original model again. Now the question is, how close do you get to accuracy here when I go and do this and how much is there? Here is an additional search from those authors who put this original paper together. And it turns out that against Amazon's logistic regression model or Big ML's decision tree model, they were able to get to 99% accuracy even in the most complicated cases of under 5,000 queries. That ain't good because 5,000 queries is once or twice every minute for two days. That's basically noise for anyone who's looking at it from an alerting or monitoring perspective. Now that's a very specific, very generic type of attack. Let's talk about a more specific attack against some of the language models that are out there. In this case, it's called a model extraction attack and it's really, really cool. You might have heard of BERT. BERT is a very popular language model that's come out in the past few years. It started by some folks at Google. Most language models that you see out there today are derivatives of this work. It's really transformative stuff. And in fact, at Google, or excuse me, at Azure, we use this too. Many of our cognitive services use derivations of these language models behind these very rich APIs to do things like language understanding, Q&A, and so on. And we've done tweets on them, but under the hood, the same core is there. And the same attack will work unless you defend it. So the way it works, the way we're going to go and attack this model is we're going to use some of the techniques created by Squad. So Squad is the Stanford question answering data set. And the idea is you take a set of Wikipedia articles and you train your model on it. And then you ask questions of the overall model. So in this case, I'm going to say, here, this is an article about prints. And I say, question, how many instruments did prints play? And the answer is 27. And as you can see, this is actually a pretty hard question to get because there were instruments all over here. Is he talking about guitars and drums and percussion? He played is separate from including 27 instruments. And there are other instruments down here, bass and keyboard. So again, this model is quite sophisticated and it does really cool things like this. Okay. So how are you going to attack it? Here are two of those articles done by UMass, Google and Northeastern University. Really cool stuff. But the way you're going to attack it is you're going to start with the same thing. You're going to have your model trained on Wikipedia or some corpus of information. And you can attack it either by using random strings. In this case, a string that says, how workforce? Stop who knew of Jordan at Wood displayed the, obviously a nonsensical sentence. But Burt and other language models like this are going to do their best to respond from the corpus that they know. Or they can say unknown, but even that gives leaks information as well. So they're saying, for some reason, the model decided that this was a good fit to this. So all right. Or you can use Wikipedia, the same Wikipedia common set that was trained on in order to develop the back end. And in this case, we're going to take a bunch of these words and words from all over the Wikipedia data set, mix them up because we don't know what a real sentence looks like. But it's also going to give you an answer. And in this case, it goes even faster. So once you've done this and you've done this a lot, you're able to reproduce the model. And how close you're able to reproduce it? Well, if you give it just 0.1% of the 10% of the underlying queries, you're able to get at 72% accuracy. If you do the same amount of queries used to train the model, you're able to get up to 86%. And again, this is without knowing anything about the underlying corpus. You're just attacking the model. And this is extremely cheap against some of those data sets that I talked about earlier. Here for 62 bucks, I'm able to get the sentiments against 67,000 queries, which is the amount necessary in order to train the model up here. Here's the speech recognition cost. And even to do something incredibly complicated like this, it costs just a couple of grand. So now I've reproduced this incredibly complicated model that I didn't do any work around. And I've been able to make a reproduction on my own. Now, there are ways to defend against this. You can try and detect those queries. You can watermark predictions. So it's going to come back and give you answers to models, regardless of whether or not it's an appropriate fit for you. And you could query that model and say, hey, does this respond to, does the canary fly in the nighttime? If it does, that implies that they have scraped your model. And things of that sort. But what I want to repeat here, the most important bit is that the pipeline is the most important and not the model. Building a rich pipeline will let you solve your own business problem, solve domain specificity, do retraining, continuous retraining for more accuracy. It'll give you a better SLA, rather than just focusing on the model. Because the reality is, if you allow arbitrary access to your model endpoint, it will be stolen. Because for better or worse, someone who is stealing your model looks exactly like someone who is just using your model. They just happen to be asking it in a very specific way. To give you the high level, you want to spend the majority of your engineering time here on this left-hand side, making this incredibly smooth and powerful and reliable and so on. I'm not saying just open it up to the entire world. Throttling and other techniques can be quite valuable here. But the reality is, no matter how hard you lock that down, you're really not going to avoid being stolen. Moving on to the final attack. We're going to get to a hidden data attack. And this is really scary stuff. So let's say a malicious user comes along and wants to probe into my system to get information about what I trained on. Now, they don't have to be logged in at all. They could just query public endpoints. And I'll show you what that looks like. But these look very similar to user behavior. And the worst part about it is, the better your model is, the more it's going to leak. There's kind of no way around that. Now, I want to say something else. And this is a really common thing that happens in machine learning. Machine learning is not at fault here. You were already at fault. You're probably having this problem today, whether or not you explicitly know about it or not. But you are having this issue. And we'll get to it. And the machine learning just kind of exacerbated it. So some example ways that the leakage is coming. You talk about things like recommendations here. This is ways. And maybe it's recommending something that I've done previously at this time or when I was in this location. In fact, I wanted to go unionize a bunch of folks. Or maybe it's leaking my network graph. In this case, here you can see this is Twitter. And it's suggesting people for me to follow based on who my friends follow. Which is not great. This may be public information, but maybe it's not explicit. It may be private if you don't do the right job of preventing it from leaking through. Or we talk about maps. Here's another really bad example. My mapping, my exercise application is recommending a whole bunch of runs for me to go on. Which is great until you realize that it's building those maps off the private running data of people in my community. Now, I don't even have to be connected to you. I'm just geographically nearby. And it's telling me where to go. So not great. But there's nothing so bad that it can't be worse, old Irish proverb, to layer on, especially with ML. So this is a technique called secret memorization. Inside your model, especially if you do something like text suggestion, you're going to include a whole bunch of private corpus of information. And here you can have an amazing XKCD. But let me dive into that just for a second. What you do is here, you're going to type a word in here, let the ruling classes. And then, in this case, Google suggests, but any suggestion, any text generation, is going to make the suggestion at the remainder of it. Now, I didn't leak that stuff, right? That was just me typing in emails. But it built a model that took this and said what the rest of it is. And you can imagine, if I had access to this information, it might be very easy for me to pick apart and say, oh, this person is trying to lead a communist overthrow. And what happens is, you might have to break into someone's email in order to accomplish this, but a lot of times you don't because in the background, a lot of these models are beginning to be aggregated together. Because again, the more data you have, the smarter it's going to be. And that's really troubling because while I kind of joke about the communist revolution, these are all pretty real things here, right? Maybe I just type what the start of my address is and it recommends the rest. Maybe I type the beginning of my phone number or my relationship status. Here I'm putting my credit card. And this is really troubling because the credit card information, the beginning of a credit card is the same across an entire brand of credit cards. Meaning all I have to know is that you use a visa from whatever Wells Fargo and it might fill in the rest. Or social security numbers, another example. The beginning of a social security number is based on where you were born. This is Florida, for example. So if I go out and say, hey, I know that that person is from Florida and I get into their email or make arbitrary suggestions against its text generation, it will provide the remainder for me. So again, really scary stuff. There has been some work recently about how to begin to detect whether or not this happens. And this is, again, I highly recommend you read all of these papers. They're just so fantastic. The way that they do the detection, not prevention, but just detection of if you have leaking, they go and they inject canaries earlier on in your process and say, you know, the random number is 012345 and so on. And then they test for leakage here because they were the ones that controlled this string. They were able to fill in the beginning part of this string here and see whether or not the remainder matches. So really cool stuff, great thinking around it. Now, I want to stress this doesn't prevent leakage. All it does is tell you how bad it is leaking. And again, like I said earlier, the real problem here is the better your model performs, the more accurately it's able to guess at what you want to say, the more that it is now a threat to leakage. And there's really no answer there because I want it to be accurate. I want it to sound exactly like me. I would love it to fill in my credit card information if I'm doing it. I just don't want anyone else to. There's some other work going on right now around differential privacy and I link to the paper later. That's where you encrypt the data on the way in and then do the training on it. And that should be a good start. But again, I can't stress enough, at the end here, your data is still coming out and will have to be decoded and translated for end users. And at that point, that's where things begin to leak. So again, no fantastic happy answer here. Ultimately, some of this will leak. And like I said, text that feels right will be based on the user's corpus because it's supposed to. That's the entire point. But it's up to you to figure out how to balance it, how to make sure that you are avoiding the worst cases of leakage. And look, I cannot stress this enough. If you can't defend it, don't launch it. If you can't correctly identify your threat model, it is better not existing in the world than having something out there that is hurting people. Let me stress again that building a pipeline, and again, I know I sound like a broker record, is the key here. You want to be able to understand that exposure. You want to inject new tools as they become available as fast as you possibly can. If you are detecting leakage, you want to be able to rebuild that model very, very quickly. And unless you have that built, repeatable model or repeatable pipeline, you're going to run into challenges. Okay. So summary, MLOps gives you a whole bunch of good stuff, even outside of security, repeatable workflows, best practices, immutable records. I can't stress that enough. You're always going to want to, at some point in the future, go back and figure out what's going on. It does require a little bit of human, and sometimes not a little, human and software work. But ultimately, it's really the only way to solve a whole bunch of these problems. And I like to end my every talk with something like this. It really is a whole new world out there. The things that you've seen here, machine learning will touch every discipline. There's really no way around that. And it's up to us, the people who are watching this, the software engineers, the data scientists, the machine learning SREs, all of us to build solutions that empower the next set of people. We can't ask everyone to get a PhD in statistics. Let's figure out what we can build to empower the frontline people, the home healthcare workers, the real estate individuals, the people looking for bias and journalists and things like that. Those are the things that we need to really empower rather than trying to solve these problems ourselves. Again, just wrapping up, you will be attacked. Your pipeline will have issues. The game is all about mitigation of harms and quick recovery. And so I can't stress that enough. Your number one metric should be how fast it takes you to make a change either in your data or in your code and have that out in production. Everything else will be kind of secondary if you can move quickly. And like I said, all of the papers and information are here. You can take your screenshots. You can visit our app. You can ping me on Twitter or at my work email there. But with that, let's take some questions. Okay, am I doing this wrong? It doesn't look like we have any questions. Am I misusing this? Anyone? Oh, maybe it's over in chat. Oh, it's not my screen share. Got it. All right. Hi, everyone. Oh, it's not my slide. Yes, happy to. How do I... Sorry, speakers. Can you help me? I don't know how to not share my screen but show the slides. I'll tell you what. I am going to, in the text of the chat, I'm just going to paste the entire slide in. That's okay. See if this works. Actually, you know what? I'm going to do even better. I'm going to, so that you don't have to take screenshots. I'm going to create a gist right now. I should have done this earlier. We'll paste the gist in. How about... And while I'm doing this over on the side, let's start answering questions. We'll answer... Okay. What do I think of AutoML? AutoML is fantastic. There's no question. I think that the vast majority of work in machine learning right now is pretty grunt work. And so things like Auto... For those that don't know, AutoML is a tool that lets you walk down an entire set of hyperparameters, explore an entire space simultaneously. And also, they're getting to some really advanced stuff where they're doing things like auto pruning, dropouts, and so on. Really powerful stuff. I think that is critical to this process, but I really think that it is only one part of the process. It will represent one of those boxes, and there's the gist. It will represent only one of the boxes there. It won't solve all of your other problems, nor should you ever expect it to solve all of those problems. So I would certainly add it to your overall flow, but I wouldn't say like, this is the end, I'll be all. Moving down, am I using AI models to detect leakage? Yeah, that's the dream, put more ML on it. The problem is that you need a large enough data set in order to detect the leakages. And in order to do that, that just requires this iterative loop, like you need more and more leakage and so on. The reality is that the best papers today, using things like that canary stuff, is your best bet. Get yourself a great red team, pay them well, and have them attack the hell out of your model, and really think about threat models as things, where things are going to leak through. Once you do that, once you identify the places things can leak through, implementing things like canaries is pretty straightforward, and it's really, really powerful technique. Go read that paper, it's fantastic. Model drift. Yeah, so that's a really, really interesting problem. For those that don't know, model drift is where you have tools, excuse me, your model was trained on a certain set of information, and then by the time you roll it out to production or years later, you're using the same model that you had originally, but it's been trained on out-of-date data. A classic example I like to give is, it's Saturday before a football game, and it trains on one thing, and if you're a ticket, seller, your recommendation might be for that Sunday's football tickets, but on Monday, that might be a terrible match for it. The underlying data change, the underlying reality of the world change, but the recommendations didn't, unless you fold in that new stuff. My recommendation here is, there's lots of tools, services out there to help you do this. Azure offers a data drift as a service, there are lots of other ones out there as well. This is something you really need to build in, and the best thing you can do around data drift is build automated solutions to probe your own model, continually feedback that production data, what people are querying on, and then fill it in. If you don't have that, absolutely. Human in the Loop is a fantastic bridge, but as your data gets bigger and as you do more stuff, there's really no way around starting to build automated systems around that. Interesting examples of attack systems. Interestingly, everything you saw here was real and live. What you'll commonly see attacked more often than not right now is where the majority of folks are. Images, objects, things like that are very frequently attacked systems, but this is a really important bit that I want to stress to everyone. Machine learning is just a new iteration of existing applications. So anything that I was talking about earlier during this talk, probably was happening before. It's just obfuscated because it sits behind a black box. Our differential privacy measures being adopted in Microsoft products. Yes, we do offer a number of differential privacy things and we're going to come out with a whole bunch of architecture around this. But the interesting bit is a lot of the differential privacy stuff happens before any of these products even get into the pipeline. The ideal is that your model is being trained on encrypted data. That's the ideal. And if you're able to accomplish that, then you're off to a good start. As far as the differential privacy for Microsoft, Microsoft doesn't ever crack your data. We don't look at your data. We have no idea what you're training on whatsoever. So you don't have to worry about us not seeing it. The question is, are you in an industry like HIPAA or other regulated industries where even within your organization two groups can't see each other's data? That's something we called eyes off training. There are other techniques for that. And we'll be coming out with a number of architectures around that. Our systems are absolutely built to do that. Would I recommend adversarial hardening? Absolutely. I mean, like I would throw everything you have here and more. That said, your hardening, your threat modeling shouldn't be a black box. The idea that you just say, hey, you know what, here's this model. I went out and spent $100,000 on this company to attack my endpoint and it came back with no results as a terrible solution. You shouldn't feel secure just because it came back with terrible results. They don't know your system as well as you do. The best thing you can do is get real trained red team humans in there, thinking about threat models, thinking about how to attack. And then and only then, once you see where your biggest threats are, that's when you start building automated tooling and more sophisticated things like adversarial hardening. Do I recommend ML Ops for ML models that are used internally for a community? Absolutely. If you're not using ML Ops today, but your model is in production, we should have a talk because that is really bad. ML Ops gives you a repeatable pipeline. If someone is doing a training in Jupiter, on your laptop, throwing the weights over the wall and just saying go to work, you are going to have a bad time. Maybe not today, maybe not tomorrow, but soon and for the rest of your life. It's brutal. Building in standard tooling around building and reproducing and rolling out models is critical. And that's what ML Ops is. What you saw there was not any brand new software other than your standard CICD tooling. But it was doing CICD with an awareness that this is machine learning specific, that we are going to run this based on data updates or based on code updates and so on and so forth. So that's what's really incredible, powerful. Are there any explainability platforms? There aren't platforms right now, but and again, forgive my bias, some folks at Microsoft in the Azure Group have come out with Interpret ML, a wonderful, completely open source SDK that is available anywhere that is under MIT license, so you can use it in any product you like. And it's something you're able to download, throw in a container and just drop it in a pipeline. And we'll be coming out with some really cool actions around that as well that you saw here. Can recommendation systems get hacked? Absolutely. Again, anything where you are leaking through the private data you have been trained on, absolutely. It is incredibly up to you, but again, let me stress, if you haven't already done this threat model analysis and the red team analysis, you're already being attacked today, even if you're nowhere near ML models. You're being attacked. Build your threat models, understand what's going on, and you'll be in the best position for defense and for building into the future. And I guess that's it. So I'll just throw this up one more time. With that, thank you very much for everything. All the slides will be available and like I said, if you have any questions whatsoever, please email me at any of these places, email Twitter, or just visit our website and you can check it out for yourself. Thank you so much.