 Okay, say something no the audio is definitely broken again Now say something Time with the audio issues. There we go. The audio seems to be working now. All right, awesome Yeah, I almost feel like we got to have whoever Is the poor soul stuck with jockey and this in the future almost needs to have two laptops like one with like Twitch and everything going and one with just like OBS and zoom. Yeah Yeah, that's nothing can confuse OBS's tiny little brain But then the problem is when you forget to mute that thing and then this thing and then there's one of your tabs Is doing the wrong noise and you're like, oh, oh, this is wrong Yep, Sven. You're being heroic. It's just Yep, we are live by the way. The internet is listening to us talk. So enjoying the banter. You're still being heroic. Yes Yeah, so for those of you watching Sven has been the the main guy troubleshooting all of the audio issues and fighting with the stream for basically all of DEF CON and it's a Testament to his hard work that things have gone as smoothly as they have You've made it a lot easier that things were pre-recorded because you can you imagine trying to hunt people down? Hunt speakers down and then their sharing is not quite right and all that Yeah, no, I mean, I mean this year at least they're not passed out in the bar How many speakers have been passed out in the bar? None of ours that I know of but we've definitely hooked a few of them out from the nearest bar Yeah, well, you know So Expecting anyone else or we want to get this show on the road. Yeah, let me well Let me pull up this thing for myself I'm gonna try a new layout as soon as I can find things so while I'm looking for it Rich, do you want to introduce the paper to everyone? Oh? Sorry, so we're gonna do a couple things. Hi, Barton so we're gonna start with just the four of us talking about it for a bit and then I'm gonna open up the and join the voice chat in the DEF CON discord and so you guys will be able to talk on stream and talk with us and everything So for the first like 20 minutes is gonna be just us And then we'll bring everyone we can in and have a bit more of a Wild West thing We're gonna try out how that works and the other thing is I'm just gonna play with the layouts a little bit so That's the structure of it. We've also got a journal club coming up on this Wednesday We will put the link to that paper in the DEF CON discord and then you if you want to participate You should join our discord Link in the Twitter bio Yeah, it's in the Twitter bio And you can just DM one of us and I'll we'll give you and send you an invite to the a village discord Please don't use the average discord while DEF CON is going on. Please be respectful of the hard work They have put in into building their discord and getting it running And only use it like tomorrow or Tuesday So take it away rich Okay, cool. So this is actually a slightly older paper. It's from 2017 It is called summoning demons the pursuit of exploitable bugs in machine learning, which is probably one of the coolest paper titles I've come across in a while and it's by rock Stevens and Tutor Demetrius's group. So the basic idea of the paper It's kind of an interesting spin on the adversarial ML papers That I think started coming out about the same time and in this rather than Actually like using the ML model itself to sort of guide inputs They're looking for more They're more looking for flaws in how the different components of the system hook up and to do that they actually Crack out a note old classic American fuzzy well the fuzzer and they re-instrument a lot of it so that weird Actions from the ML components of the system register as crashes to AFL and so that lets them Essentially do sort of like a black box adversarial attack if you want to think of it that way against the ML system And what they find is that in addition to sort of the usual stuff You can get where you have like inputs that look like faces that don't register as faces because they've had some pixels perturbed They also run into like image format parsing errors where a component of the ML system Won't parse for example a face correctly completely fails to find the face But if you open up open it up in other programs, it actually looks good. So I I think this is kind of In the vein of the same sort of things that aerial Herbert boss talks about a lot where It's not just attacking the ML system itself. That's important You've got all of these other components in the pipeline from Injusting the actual image all the way through feature extraction Pre-processing and so on and each of those is actually a point that you can attack And so in this paper, they're they're basically using fuzzing as a way to Find some of those, you know broken assumptions or breakdowns in the link from from one stage of processing to the next And I thought it was a super clever paper with a really good name and they dig a couple of CVs out of it, too so super cool application of sort of more classic security methodology to What from my perspective is kind of an adversarial and I'll problem So that's my that's my I guess like two-minute summary of the paper So I guess maybe to kick off conversation Is so does this do you feel like does anyone have an opinion strong opinion one way or the other? if this falls into the same sort of general field of adversarial and L is sort of like gradient-based attacks like I'm trying to think of the name escapes me off the top of my head like Fast gradient sign method or like these Jacobian based methods or saliency based methods Or maybe even like you could almost argue. It's like a genetic algorithm based attack against a particular ML pipeline well to be honest, I thought that adversarial ML covered all this already. I was a bit surprised that it was new Yeah, it's it's genetic algorithm based because they're using fuzzy lock which is that it go with them based which is also interesting because they were network Extensions genetic algorithms might be more appropriate if you're searching spaces So there's some interesting stuff on the front that could be done. I can't want to say what happened next Yeah context healthcare, you know practical realities of securing systems like it seems like a lot more Emphasis right now It's placed on more theoretical attacks where you know, it's more so once you have like Unhibited access to being able to query this API. It's being explored. So it's nice to see a paper like this That's a bit more than an old system set of things because at least from Healthcare IT it seems like is a more more likely a counter that you'll find Yeah, I did one thing that I did really like was their focus on implementations rather than just theoretical properties, right because like in some sense a A fast like something like the fast gradient sign method would Apply to almost any of these models But here they're sort of like okay This model was instantiated in this way and is doing these things and now how can we attack sort of that specific Version of the model which I thought was is cool because it's Yeah, it's it's a lot less academic in some ways and a lot of the adversarial ML research seems to be very academic still Yeah, this was like down and dirty and how do we attack this thing? It's kind of in my vein of stuff, which is look at the system work out the how to break the system And take it pieces. I loved the way that they they hack that little hack on fuzzy lock where they went Okay, we're gonna take out the things that actually broke and then we're gonna induce breaks for the things we care about Yeah, changing the target. Yeah, changing the target It was just beautifully done. I Thought it would be useful for defense as well Because we're also looking now at things like No, it's trace work So Brittany Foster cost work on attacking with robots What could we do to? Screw up their vision systems So how how could we we mess up? There's some beautiful adversarial there, too Mm-hmm. I can they can't see they can't move Yeah Yeah, so actually I thought table one in this was I don't know. I'm surprised It's the table one It's I think it's on page four. Yeah. Yeah, so yeah, they actually like I feel like again This is very much like Ariel's jam right they go over sort of the attack surface and the kinds of things you can do to it and it reminds me a little bit of The contest there was so there I think it was late a late entry to the contest last year that was announced at the AI village where they were trying to do ML evasions and one guy actually found Essentially a bunch of ways to just completely trash the future extraction so that the ML prediction act just wouldn't run on the samples He uploaded And so yeah, he was he was a little upset that that that didn't count as a win Which I think is justifiable right because if you can just like stop the analysis cold, right? That's almost as good as evading it, but yeah It depends what you're trying to do if you kind of sneak into a system and there's a bunch of systems But we just want to know the usual thing get in. Do your thing get out without no noticing Yeah, and we we see enough of that in real life of the stuff. It's that oh crap We didn't realize that was a problem Before someone else adds them Like there's also another thing for that like if you wanted to tack the ember The ember model with the ember dataset it only looks at your headers and you could just change other parts like Don't even think about it in terms of adversarial ML just copy like a notepad.exe's header and just put your own Entry point function in and start launching weird stuff from there And that's a different way to bypass the thing that is also not academics also not aware of Yeah, so that's sort of like I guess that's Maybe Like you can classify that if you're gonna I'm gonna screw this up So if I'm getting the terminology wrong security people, please like yell at me but it's almost sort of like a logic attack on The feature extraction rather than what they're doing here, which seems to be more going for for crashes, right? If you know how the feature extraction process works Then you can sort of right your way around it deliberately rather than just throwing shit at it throwing shit at it until something blows up Some of it's blowing it up some of it's sneaky in the back Some of it's like sneaking in the back conventionally in depending on your system. I The other thing interests me is that only some of these became CVE's That the response to the people who are responsible for the algorithms for some of this was like won't fix and who cares and that as part of our Our journey, I guess It is just like every other security thing is trying to get people to accept that yes you may have to drop a little bit of Responsiveness in your system, but you'll get it more secure It matters Yeah I'm kind of reminded a little bit of some of the argument that went around for the the proof point the CVE where Where people were arguing whether or not that actually counted as like an information leak CVE because So if people aren't familiar the short version is The people who did it will curse and someone whose name I'm forgetting. I'm sorry They found a way to essentially like Send an email through proof points spam detection service that would then bounce back to them and in the header of the email it had a score for for how spammy it looked and So that essentially was the information leak the fact that they could get that score back And that was enough for them to rebuild essentially a proxy model for it and yeah over on in various like AI village discussion channels we had very strong debates over whether or not that actually should have counted as like an information leak CVE because you know, it's In a lot of cases it's useful contextual information But when you look at it in an ML context, you're actually enabling like a model stealing attack Which again, it's one of these we're kind of in new territory as far as the security of these ML based systems goes and and a lot of people that are more familiar with conventional security maybe don't quite get what these various information leaks or Attack surfaces could do once you wrap that in like how how an also based systems work And I think another thing we get out of this is by doing this sort of attack We start showing where they at the things that break out And one of things I'd love about the fact that ML sec exists is it makes machine learning better Just by dint of the the spotlight we throw on it you chuck hackers at it They take it to pieces to show you where it's broken And hell we spend most of our life trying to get our systems to work But I'm going to give you a little bit of history. I'm old Yep jump in for just a second. I just saw in the discord chat So it was will Pierce and Nick Landers who got the proof point CV So I'm I'm very sorry that I forgot your name Nick. Okay. Sorry carry on I was just going to say about sort of some things that finally that it's double cloak precision thing So way back before we had I think we had Python, but we sure as hell didn't have psychic learner friends We had that plot lip. I'm not map not map We use matlab for a hell of a lot of machine learning stuff and there was a neural network kit in the matlab And that neural network kit suffered horrendously from our underflow errors That you could break it really really really easily so Aston University built its own They built a separate kit because they couldn't get the first one fixed And this this sort of stuff like the float to double errors. I Mean let I mean crossing my fingers that actually all of those hidden faults They're probably still going on with your network stuff and deep learning. We're not even noticing because we just trust the systems Yeah, I mean even in even in Python, right? Like under the hood a ton of these neural network libraries and PyTorch at least SK learn as well Use pickle which is just broken kind of by default, right? It's a stack machine that you can write arbitrary code for and Yeah People still share weight files that are essentially pickles and so you know you're downloading this off the internet and Running code and I keep threatening one of these days I'm actually gonna write a pickle poisoner that is just like Replaces calls to like the relu function with like some sort of stochastic Function so that half the time it works and half the time it gives you garbage comment from Comment from discord. Yeah, numpy as well uses pebbles. So it's it's everywhere Keras. Yeah I'm not sure. I know I'm pretty sure it has one One or two it's got one or two formats that That are not apparently there's a couple that the psychic learn von that these guys found Was numpy was part of that. I'm just flicking to the page page seven at the bottom if you're reading along with us One interesting thing is that they weren't able to induce like misclassification or from the precision loss of problems So they they try that different values of epsilon and basically so that it's possible theory But it's interesting to consider how would you model that epsilon to based on this attack to Induce the specification well How much of this is induced by Basically all the machine learning code is written by academics and over enthusiastic like over enthusiastic academics LT Who aren't experienced in Like it's incredibly hard to write Numerically efficient code. You don't have time to like really learn how to like multiply matrices of incredibly efficiently and Make sure you open files correctly so if we've chosen a lot of lazy paths to Formatting and opening things versus secure paths and And also you've got tiny numbers and people don't I mean they you go waiting on in build your stuff and don't think about How those numbers interact you just assume the system is going to magically deal with it Yeah, and I mean there's also the fact that Writing secure code is very much its own skill set, right? You've got you know Some people are writing research code because they don't expect it to be used in anger They don't expect it to be used in a production context Some people are writing numerically efficient code because that's their jam and God bless them and then you've got sort of security People who are probably gazing at a lot of this with sort of a feeling of vague horror that they're actually going into production Yeah, I know even Yeah, in my in my day job, right a lot of what we've been doing recently is sort of going back and and like Rethinking how all of this stuff is put together so that we can like feel confident and actually like pushing out products contain in it There's a middle ground where you can enable basically more flexible code, but also executed in a way that's Perhaps minimizes the blast radius and I mean, I don't mean put it in the container obviously, but it's something more to find of like running code and secure enclaves for instance the work with silo type of workflows or do you see anything where You try to mitigate the impacts of this code And could we have code checkers? I mean literally just some way some sort of test data sets But we know this stuff is bad run it through through your system Yeah, I mean we I had a great amount of fun Again at my day job actually like taking our feature abstraction code and fuzzing it just to see what would happen with it And you know fortunately it turned out that that it you know, I Spent more time than I care to think of beating on it and it turned out fine, but yeah Like if it hadn't what would we have done then right? It would have been and I think the idea of like building test vectors of just like, you know, really bizarre But technically conformant files for the sort of thing is is an awesome idea but extending that to the actual data science part like how often have you Issued a PR about some data science thing and sent it to someone and they were gone. It looks good to me But like a lot of the stuff like there's some stuff that you can check easily but like a lot of the Like a Burt like the code that goes into Burt is incredibly complicated very hard to check When you make a small change you could have broken things fundamentally because it's You know software How do you like write a unit test for I broke Burt when you input this one slight thing and now it Causes a whole class of things to be false positives false negatives Yeah, I mean there's a learning There's a I mean you can do I mean essentially you you do test vectors, right? You say, you know If I load because the weights you can consider the weights are obviously like part of the model You say I've got this set of weights if I load in, you know, this set of features I should get this exact set of outputs up to like numerical precision, which again, it's it's numerical precision is kind of its own nightmare, but Yeah, but then if you change the weights, right then then kind of all bets are off and that's that's like the one of the painful things about Debugging ML even when you're not under adversarial conditions is just I retrained the model How do I still trust that I mean there's a difference tree in I messed up my models Which is like we do this all the time and somebody is actively trying to attack my models Or may actively try to attack my models using known known vulnerabilities Because sure as hell half the system is out there haven't patched any of this How many machine learning developers even know unit testing or even know good software practices like they can't a lot of Thus came from academia. I have a PhD in math not software engineering So I have a philosophical question. I want to pose to the panel from discord and yaga asks what is an a terror and oh, sorry, what's a terror? This is terror right here. What is an error and what's an attack and like is there Any meaningful distinction between the two I guess I would follow up I think if you're looking at adversarial images I mean the fact you have like the little image patches to screw up your system is an attack It's not an error. You just don't randomly get sort of extra patches on a stop sign Yeah, no matter how much undergrowth you got on it I think the difference is intention Mm-hmm. Yeah, I mean that's the same thing in security right bugs can be I mean exploits are essentially software bugs right or I don't maybe that's being too general but if you You know a subset of software bugs or exploits and you use them to make the system Do things you don't want it. It shouldn't be doing under normal circumstances I wasn't intended to be doing Yeah, right and sometimes they're useful like Would you consider a well-informed person building an attack against a machine learning system where they understand the machine learning system They understand the feature extraction and all that stuff and they have they've built this And you know a false positive Is that an adversarial attack against the machine learning or is that like an exploit? Like where would you classify the mistake? Is it like a software mistake or the Or a exploit or what? I think that's the tricky bit about ml models, right? I mean they're inherently statistical so It kind of gets back to like testing after you retrain it, right? So I might if If you if you have a single example and it turns out to be a false positive And that's maybe that's one of the the bridges that I feel like as a data scientist I have to cross a lot when i'm talking to security specialists If I have a model and fps on a single file That doesn't call into question the efficacy of the model, right? You have to look at the statistics of what the model does Right, and if you say well, yeah, it got this fp and that was a pretty boneheaded fp but For 99% detection you get one fp and in 10,000, you know negative samples That's a pretty good model. And so being able to sort of like Drive that switch from. Hey, here's the you know, here's the good bad To hey, you know, you have to think of this sort of heuristically and statistically Is is a big part of just like communicating as a data scientist, I think But I mean, I'm sorry. Go. Sorry go ahead, Sarah. Yeah, if you've got somebody actively attacking There may be some pre-processing is enough to The cut down the attacks. Yeah. Yeah, for sure Although that goes out the window when you've got someone maliciously trying to Yeah, but oh so my question is so if you have like a fancy ml from an academic paper bypass of your model where Versus someone trying things What would be when is it like a machine learning mistake or versus a like a mistake in your featureizer and stuff If you look at the Decision boundary basically for something in like seeing how the perturbations relate to that Yeah, I kind of feel like Maybe to to pull it back to this paper a little bit that If it gets through your feature extraction process in one shape Or if it like in one piece then maybe like It's a modeling issue and if it like crashes your feature extraction process or it gives you something that Like you wouldn't expect from your feature extraction process then then it's then it's a featureization bug, right? Like if you if you feed garbage into a model, you can't blame the model for producing garbage outputs Which how would you basically what are some methods for quantifying like unexpectedness in this scenario? Is it more like error counts or like? Latencies in response. Like what would you pay attention to if you were trying to understand? If something was misleading You know what when you see it. Yeah, like if you take a look in the in the paper, right? They've got an example I want to find a Figure two, right? So they show an example of two different images. One is an image of Sort of that the very top of a person's head and then the rest of it is gray And one is a picture of a person's Person's head. Yes. There you go. Thank you um H6 if you're following your law, right? So open cv has a bug that these guys found by their guided fuzzing technique which Means that open cv can't properly load the file and then you run it Obviously you run it through feature extraction and it doesn't get anywhere. So It's like all these different individual components that you got to worry about, right? But I think on that topic there was a From hacker factor in discord. He's got a question. Are you making a distinction between the ml model versus software that drives the ml system? He says fundamental error versus algorithmic and I I think here, right? This is what this paper is going on going after right? It's saying the software that drives the ml system the implementation of the system Also has bugs and they find them here and you know if you go to table table two Right, they've got in open cv. They've got like two heat corruptions. They've got a weird rendering bug that they found Page six. Thank you. Yeah Right. So all these different like they literally have like heat corruption Bugs that they're discovering with these funky inputs And that has nothing to do with the model. That's you're you're attacking the plumbing around the model, which is But you can use that plumbing to go and tweak the model Right, exactly. Yeah, so that's that's what the the third line down does. That's the the figure Figure one figure two. I forget figure two Yeah, yeah, so you break the rendering and then you're the rendering goes to the feature extraction and feature extraction You can't get anything because it's just got a bunch of gray pixels to look at And then the model right garbage and garbage out the model can't do anything because it doesn't have good features So i'm going to try to open this up to discord So If you are in the different discord go to air village dash general dash voice We'll try and mute you and get you so you can talk Um, if you're human plus that you may that's the correct way to do it, but we have a workaround So head over to a i village dash general dash voice to ask any questions If you can't we'll still be paying attention to the journal the text strat in a i village dash journal dash text The moment of truth I'm watching jono club text Yeah, you too Audio managing audio is i'm so glad I don't have to deal with it But they um I wish I had seen this paper When before I started like really trying to solve adversarial machine learning as a security problem because there's a lot This seems a lot easier like to attack for a lot of attack I think a lot of attackers have more knowledge on how to break a Parcer than how to break a neural network So this to me feels like more of a realistic threat model of like your Pipeline was they they broke your pipeline. You did something wrong in aws. You did something wrong with your, uh Container setup and it's parsing your images incorrectly now Yeah, I mean, that's tons of the bugs that you find in sort of real systems, right as this parsing bugs and Yeah, a lot of a lot of the research right now focuses less on that and more on sort of these theoretical properties of You know machine learning models, which I mean, I don't want to dig on those. They're fascinating, right? I love these papers, but they're also Again keep coming back to this paper This paper makes the really good point that these sort of the oracle models that have these nice properties are embedded in real world Systems and we know that real world systems are always kind of full of bugs that can be attacked and exploited And so it gives you sort of multiple entries Into the problem right on the one hand you can exploit the preprocessor And maybe that gives you some sort of heat corruption or or some sort of like direct You know denial of service or or even code execution And on the other hand you can break the feature extraction enough so that everything downstream is getting garbage data Or mildly corrupted. I mean I I say again with like the sneaking in You want to do low and slows on some of this? I Again, I am really like the idea of using this as defense. I mean there's a whole bunch of applications where the bad guys are using machine learning based systems I'm like the idea of breaking the systems Yeah, as I would great hat song Yeah Yeah, just like yesterday's paper with the forks masks Yeah, so um Will has a comment he says i'm surprised to paper is the vehicle for this knowledge there's a whole security industry that has advice for this stuff and I think yeah, this is part of what We're trying to do right with a i-village is draw a link between security and machine learning right so We have a lot of like wicked smart security people Who know about all about these implementation bugs and then we've got a bunch of like really smart uh academics and machine learning people who know about like sort of these You know feature based attacks or or weird properties of ML systems and getting sort of that knowledge transfer going on so that we can Find out where those intersect and how those two, you know sort of sets of attacks inform each other I think that's like a really rich area for future development and it Kind of makes me sad a little bit that this paper came out in 2017 and and got comparatively little attention When it really is going right to the heart of a lot of really big questions in this, you know in ml slash security, so In practice, you see a lot of data science teams disconnected from both like infrastructure application network security sort of teams and so like the whole mlops Sort of formation of teams that can interface between data scientists and implementation has has been emerging, but i'm curious what the panel thinks as far as What's the appropriate like Role a person across functional team do we want to have security people sitting with data scientists as these things are being developed And Yeah, I mean with enough I used to be one of people going into large companies working out where to put the data scientists And there was always this argument between you want to embed the the data scientists out into the rest of the team And have them work with because you're informing the rest of the team everywhere um But then you've got data scientists out on their own and we're kind of you know You see in this paper that we get ignored quite a bit when we're on our own Or having these unical and pens full of data scientists who talk to each other but not really to anybody else And there's a so yeah sense of you you put people out But you make sure they're still part of tripe So you literally build Tribe or whatever the the good word now is for that It used to be called tribes a cross village cross. Yeah, village. Yeah village So you build villages for the data scientists to keep connected to each other, but they need to be out in the rest of the teams Because because otherwise, you know, we're we our work is everywhere. We should be everywhere too That's from the belief though Yeah, I mean, I think it's but it's it's got to be sort of a two-way street, right on the one hand Yeah, there is kind of for data scientists some sort of ml practitioners, I think There yeah, we do sort of tend to stay in our little bubble and don't think about these sort of Externalities, I don't know what the right word for it is but like these other ways, you know It's not just going to be like fast gradient sign based attacks. It's going to be like no someone someone broke the parsing but then at the same time And this is kind of going back to maybe a little bit of the height panel yesterday like having People that aren't data scientists understand kind of like what the models can and can't do and what kind of behavior To expect from them so that they don't freak out when they see something weird happen and be like, oh, you know Whatever we flagged this DLL as being malware when it's not we're clearly under attack Right, and you're just like it's statistics. Sometimes it makes mistakes Well, just explaining it's not magic It's hard, but it's not magic. Yeah, it's hard. It's useful. It does cool stuff, but it's a tool Are you guys talking about implementing the security development lifecycle? And this pure software development likes lifecycle into model creation and upkeep It'll make a data scientist cry Yeah, this I had Issues with just being like cool. We need to like just write a few unit tests to make sure that Functions all kind of do what they're supposed to do Um, there's there's a bit of pushback in our for some of our more academically minded Folk of like just a little bit of basics like that um Which may have solved some of these like open cv issues because I interesting that um, if you basically treat, um, the life cycle of like a data project as More internal products for like a company and treat it as a product that which has which serves a purpose provides value and then Is is integrated with other ways of developing software products then Data science machine learning they just become tools and engineering toolbox But then everything else falls within like methods of like continuous integration and testing and delivery So then perhaps the risk of rather than the unicorn depends Yeah, I mean, that's what we're doing. We're doing things like, uh, hypothesis-based, um Development, which is like the the next thing up from behavior-based development Uh, which is next thing up for test-based develop test test room development So you very much like work out what the universe you'd like to see is go do the go to the maths on it Build the systems in it So there there are ways of working Yeah Yeah, so maybe looping. Sorry go ahead Yeah, so I was gonna gonna loop back to the paper one more time Um, and talk about the disclosure experience that they they mentioned Towards the end of the article So basically they they got three of the vulnerabilities got new cvs Because they enabled arbitrary code execution or denial of service attacks Right, so those are both very firmly within the wheelhouse of what people page seven page seven. Thank you Very much within the wheelhouse of what security people think of is oh, yeah, this is that's a bug That's an issue. That's a security gap that we need to fix But um, there were a couple of other ones that they found Where they could impact or manipulate the prediction and then they they call out in particular the mouth Here memory corruption they found where basically they can rewrite the feature vector which Um Allows you to basically like make the model give you any output you want essentially Uh, it doesn't have to have anything to do with what the actual file that you were analyzing was And those specifically are the ones that didn't get the Uh, didn't get a cv. You didn't get called out and mostly got labeled as won't fixed or is won't fix So that's sort of like, you know, we were kind of bashing the data scientists earlier saying Oh, you know, we got to like think of this as part of a security system But then it goes back the other way, right? We need to be able to communicate to people that as data scientists This machine learning experts look, you know, this is this is bad, right? And how do we convince people that? You know something that allows arbitrary misclassification or arbitrary predictions to be to be produced is as serious as or maybe not as serious but But has some degree of severity just like a remote code execution or a denial of service I like the example that eric brought up in his talk of a turtle rifle Like that is a very effective way of communicating potential problems whereas for those who haven't seen it, uh, like a turtle, uh pre-printed object could have Be applied to a turtle to make it misclassified being as classified as a rifle you can imagine like systems that uh, like security of Like it's a studio and stuff that you can do certain events I mean, it's like any other ethics discussion. We have to talk about consequences And then track it back and just keep pushing and pushing the examples real examples if we've got them It's it's hard Yeah, I mean, I guess like that's You you have a joke But we're not really a joke that like airline regulations are written in blood Right, and so when you've got all of these safety requirements It's because a lot of people got hurt before someone was like, oh, we ought to do something about that and it would It would be nice to think that we could we could find a way to not have like ml disclosure requirements written in blood, right like maybe, you know, let's let's We're we're out of place now where we could maybe like jump ahead of the game a little bit and and Get an understanding of this out there so that we don't have to see people hurt before people are like Oh, yeah, we should maybe take that seriously. I actually have a historical example for that too for being old So I ran one of the unmanned air vehicle safety teams back in the day before UAVs was sexy and we didn't have the Privilege, I guess that the the rest of the space industry had So aerospace regulations are literally written in blood So an aircraft would crash into something else or Hit the ground and you would change the regulations and that's literally how the regulations were written same with fire safety fire happens change the regulations and we were told explicitly and understood explicitly that We couldn't just put UAVs in demand airspace have the crash into stuff and then rewrite the world rules We literally had to think all the scenarios and write the rules for how to do it safely And it's whole space safety not just individual aircraft safety before we were allowed to fly in the same spaces So there's some precedent. I mean there's maybe some looking in those spaces to see how it was done well as a good way to look But a lot of it's just getting the will Now nothing in focus is retention like a whole bunch of airliners about to crash, but yeah Yeah, you know, we could maybe get there before that point that would be So, um One more question from discord, which is good because that's where I wanted to go next So y'all guys the question. How do we force open source software to fix itself? We don't fix it ourselves That's why it's open source and the authors actually make a comment in the future work Where where they they say that it's unclear who should be responsible for fixing them as well. So When they found a bug in in the malware feature processing that was because malware relied on lib archive And the bug was actually in lib archive. And again, this gets back to this notion of like you have all these different components that feed into the system So even though the bug was in lib archive it affected malware So, where do you where do you put the responsibility for fixing it? And how do you convince? You know, like whoever's maintaining lib archive. Look, this is serious enough that You know, yes, I don't think it doesn't affect you directly but It does affect this other system and it could be a critical impact like how do we How can we navigate that sort of? uncertainty How does linux and like a one two and others Obviously will software ecosystems do it Do they have liability? I mean, I know for a lot of systems the end users, especially if they're big end users just get involved with the open source communities and work in there, but if you're using Even like python and your physical system crashes Who's responsible? Yeah Yeah, I mean a lot of licensing argument a lot of licensing agreements basically say like yeah You agree that you're using this at your own risk too. So You know, basically everyone's going around like not me Which you know, it's fair, right? And people have like they have they can't assume liability for for everything that they do, but You know, and they can't agree to fix every single thing But at some point, right? There's there's a balance, right? If something's widely adopted It's widely used and there's like a bug three steps up the chain. That's that's causing it to behave weirdly Feels like somebody ought to There ought to be like some impetus to fix it somehow The long tail of companies like fortune 500 and what bother to use these open source libraries I mean in theory, you know, this is a I so you have a lot of Boost and productivity like and there's a lot of value that's being derived this and especially if they've started to use it for critical applications Decisions so I think the honest responsibility also needs to fall in the end users of like widely adopted packages and then somehow making making it very clear to decision makers in those companies that hey if you're using this to predict insurance rates or something and it fails That's what you could potentially face and then somehow channeling that into the projects themselves Reproposing that large companies that make a lot of money using open source tools should actually give back to the open source community It's kind of terrifying that we all just laughed when you said that Sorry, I gotta get my gotta get my commentary in once in a while These are complex systems and like just communicating the impact of some small like overflow bug to somebody who only cares about like the bottom line It's a problem that maybe there needs to be like a awesome ml failures could have repository or something like that where We haven't had that Sounds like actually pretty good weekend project for someone. Yeah I mean like you know when we start up the ethics stuff years ago It just just takes some people determined to make this thing a think a visible thing Yeah, hey community Anyone want to take this one on But so how many like solid examples of ml failure do we have? If we want a lot of it is compile those. Yeah, how much do we actually know? Yeah, that's um Let's say Andrew Davis over in different discord had a Pretty good comment on it, which is that a lot of time when ml fails It kind of fails silently, right? It just you you don't flag something and it comes back to A lot of the earlier discussion about is it is it a bug or is it like normal just sort of like statistical Failure right a 1% A 1% failure rate means that if it's not failing 1% of the time you should be vaguely surprised, right? So Yeah, like I think it's possible that there's a lot of ml failures that we just kind of assume are in the statistical noise A lot of them are probably kept quiet because you know, there's Potentially like pr risks or pr damage, right? Or you know, like like actual like was You know liability attached to it Yeah, I mean, it's it's an open question how much is is actually sort of flying beneath the radar out there Imagine a rental company say or a background check company who uses ml going Uh, so we didn't give a lot of people their uh Approve, you know their credit check approval Because of a mistake and we want to publicly apologize for that When it's in the ml mistake and they can just be like well the model just was yeah, sweep it under the rug fix it quietly never mention it Like that would be an ethical nightmare Yeah, I mean there's lots of startups um That have proposed to do things. There was one I was about two three years ago Which was like oh, we'll use ml to like pre-screen your babysitters by like digging through their social media profiles and stuff right and I have my own opinions on how likely that was to succeed versus how long it how much how likely it was to simply just like recommend white people but um You know, like how do you how do you even like measure or quantify failure? Failures in that case, right because it's all going to be completely internal and all you're going to like all you can do is you know Essentially like try and like get a proxy model to be like, aha. Here are the features, right? I've submitted a thousand different profiles and here are the features that seem to be Like triggering right and that's you have to like Affirmatively dig for it right those failures aren't going to be obvious unless you're actually doing something like that recent streak of sort of facial recognition failures that lead to Incorrect arrests or several articles that the list is like a visceral got reactions like yes, this is wrong So I think it's in some cases just like seeing it like the outcomes. Well, like there's a This gets yeah, I mean The bias question like how do you tell if something's biased or Just like for this one instance, you were wrong. It was wrong Yeah, we're gonna have to clean that one out somebody. Yeah racial bias in images is oh god It's yeah, how much longer do we have for this conversation? We can solve it in eight minutes. We've got eight. Yeah. Yeah, no problem. Yeah, I mean the problem The problem in a lot of the bias stuff, right is like the terms aren't even defined sometimes so like The the the example I keep coming back to is like recidivism and like predicting re-arrest rights Right and someone might point out. Hey, look we we you know, this is biased, right? It's clearly predicting re-arrest rates that are much higher for this group Right and one argument we would be well, that's clearly wrong, right? They're you know, it's biased because it's trained on Biased data and the counter argument would be like look it's reproducing the data, right? We don't live in a perfect society. This the data is what it is and within that box The ml has done as much as you can ask it to do Right and so it always come it always boils down to these normative questions and in this is my own totally personal opinion that You you can't like talking about unbiased systems unbiased ml is is almost inherently impossible right, it's it's a question of how Is it producing the kinds of outcomes that you want and how is how is the ml system moving power around, right? Who is it empowering? Who is it disenfranchising? And you know, who is it helping and who is it who is it hurting and Chasing after some like vague notion of fairness, you know a fair system or an unbiased system is Is kind of like a distraction from like more fundamental questions of like who's being helped who's being hurt What what's being reinforced or whose concerns are being downplayed? Who's accountable? I guess also because of like reference recent foot or storm between tinnitus every now and again Basically, like is it the ML engineer or is it actually the data scientists throughout the entire chain? yep But yeah, and I mean Sorry, go ahead. I'm talking too much. I'm seeing over in the chat that we already have awesome ml failures in the AI village as a repo And stellar Athena seems to be getting in there already That's the suggestions for implementation failures ethical failures hci failures and security failures We got that site stellar Yeah, yeah, I think uh, do we do we have a category for because we were playing around with this cat does not exist Um a couple days ago. I think we maybe need a category this ml should not exist Yeah, nightmare fuel Yeah, well, I mean I'm thinking of things like you're right There are a whole bunch of machine learning systems. So just should not be on this planet Yeah, yeah, and anyone asked to contribute to it should have been like Perhaps not Yeah, as as far as responsibility, we would never have facebook if anyone had had that call. Um, anyway, I mean hindsight but you can look at like really uh short-term stuff like Even just like gender if I right it was should have been pretty obvious everyone involved that this was just Like a losing proposition from the start Yeah, and the positivism through face images stuff from china. Oh god. Yeah, okay, so Way to replicate somebody's biasing We do have we do have I just so we do have a a workshop coming up in a couple minutes. So I think I want to kind of like Put a pin in this I think if everyone's enjoying this conversation Stella's having an awesome ethics in AI panel, which is coming up in a couple of hours I'm not sure someone someone Two o'clock two o'clock. Thank you. So everyone should absolutely turn tune in for that if they're enjoying this discussion um But yeah, I guess I want to just like go around just to wrap this up And see if we have any like last minute like big idea takeaways from this paper that people thought were really really cool Uh, I love the way they did it The idea of attacking the system and looking at taking it apart is just kind of my thing And I like the idea of using to attack systems that I That shouldn't exist Yep It's fun Uh, I really like the idea of attacking the feature extractors And I think if we should also add like bypassing the feature extractors by just doing something You know the MLM system is going to miss Uh, because You understand it well To this whole thing Cool Barton Yeah, that also really enjoyed the paper in discussion. Um, I I think um, to me most clear that it can occur on many levels of the Basically stack so just Instrumentation and stuff become very very important to monitoring what happens and then I would really like to see Follow on work that applies this to tensorflow by torch basically the learning libraries and seeing how you can play with Uh intermediate representations and things like that to reduce Yeah, for sure. Yeah, the reminder that like these ML things ML Projects don't exist in some abstract theoretical space and are implemented in real code in real systems that come with their own flaws was Yeah, like you know it but being reminded of it sort of viscerally by look we have CVEs is always is always a great thing So cool. Um, so I think this is we probably want to leave a few minutes for people to fight with the audio as we transition over to the workshop So, um, I think we want to call that a panel so or a discussion. So awesome. Thanks everyone. Thanks for one second Oh before we jump off, uh, we have one on wednesday at five p.m. Pacific time Uh, and we have the I'm guessing I think we should go with your paper rich The slide deck on Yeah, the anomaly detection Outside the closed world on using machine learning for network intrusion detection by robin summan and uh, verand paxson So we'll post that in the Defconn discord and also be posted to our discord and twitter So if you want to participate you should find your way over to our discord We can't post a link to that in The defconn discord, but the links are places and you can just dms for an invite Uh, and we'll be discussing that for probably an hour and a half on, uh Wednesday and we might be able to get one of the authors in even though that this is a 10 year old paper Far we have come All right, thanks everyone catch you later