 Hi, everyone. How's it going? Are you enjoying big data? No, big things. Formally known as Big Data Spain. Before we even get started, how many people here have built a machine learning model, deep learning model before? Something like that? Okay, very good. You're probably in the right place at the right conference then. What kind of accuracy have you gotten before? Like a 97, 98%? Anybody? No, okay. We'll get a couple people. Very good. All right, so then you probably wouldn't trust your model with your life, so that was going to be the next thing. So if you haven't even gotten a 98, maybe you wouldn't consider putting it into some of the systems that I'll be talking about. But that's basically the idea here is to talk about, again, you probably wouldn't trust your model if you're only a 98% accuracy, but this is the kind of thing. Would you actually trust it to have your family in front of that truck or yourself? You know, these are the kinds of things that we need to think about as AI models are being incorporated into these kinds of systems. And so that's what we're going to talk about in this session. You know, what are the things that you need to think about? What are the challenges in these systems? And what are some of the techniques we can use to actually be more confident? Hopefully, by the end, you again may not trust it with your life, but you'll have some techniques to feel more confident in what your model is doing. So I'm Heather. Hello. And I work at the MathWorks. So you may have heard of MATLAB, Simulink. We have a lot of scientists, engineers. You know, this is kind of the target audience. Well, not really target, but this is the user base. And so we hear this a lot, actually. You know, people are really starting to get interested, you know, AI is all over the place, and they really want to incorporate these things into real systems, like the plane landing on its own and the car, you know, stopping. So this is something that comes up a lot. Again, AI is all over the place. You guys are probably working on these things as we speak. Hopefully not right this second, but feel free to code. That's fine. So again, you know, what are these kinds of things? These are really important systems. You know, so, you know, those of you that raised your hand, you know, might have built recommender systems or the taxi data analysis or the airline data kind of thing. But what's the, you know, the impact of an error? Isn't too bad. You know, maybe I got the wrong movie. You just turn it off. That's okay. But what happens again whenever that fails and the brakes don't work and somebody gets hurt? You know, what there's so many things to think about in these kinds of systems. So, you know, this is sort of your traditional, you know, AI kind of workflow. You have your data coming from a device. You have, you know, pre-processing and data cleaning. You know, we know all of this kind of stuff, right? Finally, we deploy because again, these are real systems. We want to make sure that our model is useful. That's important to engineers, scientists. You want to do something with it. But again, when you start thinking about incorporating it into these kind of systems, you also generally include simulations. You know, again, depending on what you're trying to model, in this case of a vehicle, most days we are probably a good driver, probably not out hitting things. So, we could simulate what it seems like or what it is like to hit something so that we don't actually have to break anything or hurt anyone. And then testing. I'm going to say this like 100 million times probably in this talk. So, you probably get tired of it. But it's really, really important. We'll talk about some of the strategies. But at every single phase of this flow, you want to test. You want to think about the requirements from each step of the way. And this is also just one part of another really big system. You know, again, the deployment part is really important. And that's also completely non-trivial. You know, you have to step back and think about what you're doing in the end before you even get started. So, you know, you have, especially in a vehicle or airplane, something like that, you have not only your AI models, but you have physical models, you have your control algorithms, logic for all this stuff, right? And all this has to come into play together and, you know, really work together. And then, you know, again, depending on your target, you know, device, or if you're going to the cloud, you're going to need to take all those things in that left box and wrap it up and deploy it somewhere else, right? So, that it can just run on a device, be very fast and seamless. And of course, there's an entire system. So, that also involves a big view of what's going on and test every step of the way. There's even certification, kind of, I don't know if you can read my handwriting, hopefully it's big enough and not too messy. But certification is also important. You know, there have been many, many years of research and experience, and before you put a model into an airplane or a vehicle, there's a whole certification body that has to validate and verify every single thing that is going on. So, this is very challenging. And there are lots and lots of people involved, especially with something like a vehicle, right? I think this is a fine representation of a great team. But, you know, you have end users. In this case, it's not a person that's savvy with computer science or data, it's just a driver of a car, right? That's all of us. Well, maybe some of us. We have the managers, you know, again, these are things I guess we're sort of used to, or many of us are used to the data scientists, the system architect, the engineers kind of working together. But again, we've got lawyers and plant managers and use all sorts of different skill sets, all sorts of different cares, you know, you need to communicate these things. We'll also talk a lot about these strategies for communicating across the board. But the way that you talk about your AI model to the system architect is very different from what you say to the lawyer, right? And, you know, you don't think about these things all the time when you're building a model. But these are the kinds of things you need to think about before you start even building it whenever you're in this kind of system. All right, so that's sort of the landscape kind of give you an intro of what craziness we're working with. So we'll talk now about some of the techniques that we can use to make, you know, these decisions and bring in models like this into these kinds of systems. So model explainability, very important, we'll spend a good bit of time on that. The deployment process, again, you know, one of the things that you need to think about to take that next phase, you know, generally we want to work in something, you know, nice to use, but then, you know, send it out to FPGA or, you know, machine code testing. So I won't even say anything more about that right now, because you're going to hear a lot about that. And finally, you know, again, thinking about everybody involved in this world, you know, how can we bring it all together? How can we get on the same page? Again, thinking about all the different skill sets. So we'll start with explainability. I think of this in a really basic way. So you probably have heard, I mean, I think there are a number of talks on this kind of thing this week. And I just came from another conference last week. These are, you know, things that are really important to people right now, especially when it comes to the lawyers, the plant managers. So, you know, for example, it's basically asking questions. How did you come to this decision? Why are you giving me errors? I ask all the time, I ask my code, why are you giving me errors? But that's really important. Especially again, if you're, if you have hit a pedestrian, you have to actually go to court. So you better be ready to explain why that failed. Also, even currently, at least in the United States, you can actually get some information about a loan application that was denied. For example, I recently applied for a credit card that I did not get. I feel fine talking about this because I'm, you know, U.S. citizen, I had to pay a bunch of money for school. I'm still paying off my student loans. So when I was denied, I was like, well, let's, let's see. I've heard about this world where you can ask, there's a, you know, they have to document everything. And it turns out that was the number one influencing factor by a lot. So, you know, even now, you can actually, you know, many people are starting to really incorporate this stuff. And unfortunately, I didn't get a really cool classification tree. I was really hoping to get some kind of like cool path that I could follow or whatever. But, you know, it was a little bit more verbose than that. But so, you know, those are the questions basically from the human. But you're also, you know, communicating with a computer. I know that sounds really basic. But, you know, what do you want from me? It's kind of the biggest question to answer in that realm. And I think that, I think of that as just requirements, right? And just making sure you're very, very clear about what's going to happen, you know, all the logic, all of the models, you know, that you're thinking about. Make sure it's clear to everyone, including the machine. So, again, you know, we're hearing a lot about this in the community. We've probably heard a lot about this in talks this week. And, you know, there are model or frameworks and things out there that can really help right now. They tend to be in visual ways, very helpful. So, you know, sort of look into your model, you know, TF interpretability is really great for that. And it's also, again, really super important. So, it's not only just the community trying to, you know, come up with these things and really analyze this, but also DARPA is an association in the U.S. for, you know, defense and things like that. Again, that you want to make sure that you're prepared to talk about what these models are doing. And so, I really like this visual, you know, it has some of those questions that we're talking about, you know, the things that you actually want to think about for, you know, explainability purposes. And basically they've just sort of inserted an explainable interface in the middle of this workflow. But again, it kind of, I think it just, you know, it reinforces the fact that you just want to make sure before you even just start diving in you want to just think about it first, you know, make sure that this is something that people are going to be able to understand aside from yourself. Usually I like to understand those too. But so, again, in this, in these cases, you know, we love TF explain all these things are really, really great. But it needs to be explainable to everyone, even the lawyer, right? And the end users or the community, the world, if people have questions of somebody was hit by a car, we all want to know. We all want to make sure that this is a safe world for us before we just start saying, yeah, yeah, go ahead, automated, drive yourself around. You know, we're all part of this, right? So communication and, you know, again, kind of using simpler models will really help. And so these are the, again, kind of the techniques that, you know, in our experience are really helpful for this, you know, visualizations, documentation, sort of what you might expect. But then also simulations, like we've mentioned and testing, you know, again, there's just a lot going on in this system. So in terms of simpler models, I tend to use classification trees because they're really not that simple. I don't think so anyway, especially if you're like, oh, yeah, sure, why not? These are just, you know, if the sixth variable of the acceleration is less than 54 point blah, blah, blah. So it's not necessarily explainable or super, super clear, but it is something you can at least track, you can at least understand what decision was made and why. I enjoy pruning trees because I love puns in machine learning. I think it's really fabulous. But, you know, that's another technique is just, you know, there's, you could be fitting noise. That could be very confusing. Just make it as simple as possible and keep simplifying. This is definitely too simple, but just for the slide, it doesn't even have some of the classifications on it. But again, just to kind of show, you know, most humans are able to track this and understand that you start at the top, go down, if it's greater than this or less than that, go down, continue on the branch, right? So unfortunately, again, I didn't get that. I was really hoping for that in the credit card thing. But even simpler, you know, consider your old friend F equals MA whenever we're talking to scientists and engineers, especially with these kind of systems, it's a physical model doing physical things. Like, I don't probably need machine learning to tell me if I'm speeding. It's like, pretty easy thing, you know, acceleration, physics 101. But again, it's, that's really important, you know, don't just jump in and be like, let me just deep learn it and get all the kinds of crazy layers here. Just step back, remember what you're doing. Sometimes that's appropriate and that's great. Other times, simpler the better, right? So in terms of the hierarchy of like interpretability, I guess, you know, deep learning, deep networks, probably are the least explainable, but most accurate, you know, so again, they're always trade offs. It's life. That's just how it is. So, you know, you kind of have to think about what's most important here. And you know, they're still able, you're still able to communicate these things. So it's not like you shouldn't use deep learning. Otherwise, a lot of people would be very mad at me in this room, especially. But you actually, you know, you can actually get more information about it. But if you're really looking for the most simple models, just to rule, you know, some kind of your own kind of decision trees and F equals MA, those are just great. So again, just encourage you to really think about who you're working with. What are your needs? Maybe you can sacrifice a little bit of accuracy to achieve more explainability. And so visualizations are really important. We'll talk about that next. Of course, I think of this as more natural whenever you're talking about, you know, visualizing images or classifying images, those kinds of things. But it's really important to even start with visualizations, right? So again, you know, MATLAB scientists, engineers, they're very hesitant to just, okay, let me just take this model and go to town. I'll just grab this TensorFlow and whatever TensorFlow model, you know, they want to know what's going on from the get go. So just having, you know, we have a nice app that just shows very clearly what the layers mean, what is going on with those. So people understand convolutions and, you know, aggregations. Those are pretty simple techniques. And when you can see it kind of laid out in the architecture, it's much easier for people to understand what's going on from the beginning. And then, of course, also the last part, I mean, I think we think of it as trivial, but, you know, it's really important to visualize the results, slice and dice them all different ways and really understand why classifications were made the way they were. It would not be a talk about AI without some nice cats and dogs, right? So this is an example of using Gradcam to understand what parts of an image would make the prediction that prediction. So this golden retriever sort of is your eye head area is the thing that's really making that a golden retriever. Another technique that I think, well, it's easier to explain is maybe occlusion sensitivity. So for example, this little guy, it's kind of a toss up to me, whether it's a toy or an actual poodle, based on the leash is probably a real thing, I'm guessing. But also the model is not super clear on that. But if we use occlusion sensitivity, it's really well designed for images, you end up getting back like a matrix basically that you can kind of overlay. And it just again shows you the most influential parts of that image. So again, I think of it's still a bit of a toss up, could be a toy, could be a real thing. But the thing that makes it more terrier like is the beard, right? So that's really clear, you know, from this. And so again, maybe this will help you, you know, go back and make sure your images are processed the way they should be or, you know, again, understand why these decisions are being made. So again, I think easier whenever you're working with images, but that's not, you know, all of us. But in this case, this is a just predictor importance from a tree. You can, you know, do this a lot and just take a look at what variables are influencing, you know, pair them down and, you know, use visuals in other ways too. For example, you know, if you're doing speech recognition, you could use the spectrograms to see, you know, what what's making this a yes or a no or, you know, the classifications the way they were. So again, you know, it's not like you don't have image or visual visualization options available to you if you're not working on images. You know, so I encourage you to take a look at that. And again, many of the community tools are really great with that. That's actually kind of the first step of making things explainable. Visualizations are the way that humans understand things, right? So, you know, take a look for those. Documentation. So I probably sound like your freshman computer science teacher or something, but it's so important here and even just documenting in a way that people can understand, again, not just yourself when you're coming back to write your code or not just your data science team, but just communicate and remember your audience basically, right? So you want to have a little bit different conversation with the lawyers, but again, so important to document everything from the data prep, where the data came from, you know, how you split everything up, set your seed, random numbers, you know, like just be really responsible and remember that you might have to defend this in court. Hopefully you don't, but you know, that's kind of the worst case scenario, basically. So simulations are probably, I mean, I think at least in data science, we tend to sort of, here's my data, I've got plenty of it, let's carry on, right? But again, you know, this is anomaly detection and even without the AI part of it, it's actually extremely important to simulate the stuff, right? So this is some stats from, right, I guess, R-A-N-D, I don't know if that's the way you say it, but basically you need 11 billion miles of testing to even come close, right? And that's just to make sure that your components are working, that's without AI. So that's a lot of driving and again, most of that time, you're going to be driving pretty well. You're not going to be changing lanes or, you know, doing things that are outside the norm that, you know, we would actually want to build a proper model, right? And so again, just simulating the data from the beginning and then simulating all different parts of the system, all the components, you know, all this is really important before you even try to put it onto a piece of hardware or, you know, a system like that. So again, I think it's more natural, at least for me, whenever it comes to, like, the data science part because you may be familiar with anomaly detection, right? So you have, again, many driving experiences are very good. Maybe you're going a little fast, a little slow, but you're not crashing, right? So you can actually just simulate that in a bunch. You can create your, you know, big data lake or whatever with just simulation. So that's a really, really good way. You could also use, you know, standard kind of techniques for imbalanced data sets. That's also great. But again, you want to make sure that you're documenting all of that and being careful to consider that in your testing as well. This is an example of a simulation for lane detection. The braking one actually looked a little bit boring, so I didn't make the slide. But it's just an idea. You have a bunch of different components. You have, you know, all these things together and you actually can get a sense of, you know, what's going on. You know, this is simulating a lane detection, you know, just driving situation. So, you know, again, it's really, really important. You can run these over and over and over again, collect the data, visualize, be pretty confident about that. All right. So now let's say you're pretty confident. You've got a bunch of data. You're good to go. You're ready to hit the button, so to speak, and deploy. What do you need to think about? Right? So, you know, again, there are many, many components. Often these are, you know, FPGA, GPU units. Could be just in a cloud. You might just want some, you know, dashboard to show your equipment or your maintenance. And that is, that can be extremely difficult. So again, I, before you do anything, you really, really need to think ahead. So, you know, how much data is going to be coming into the system? Is it one data point? Is it a whole hour of data? You know, what are these assumptions? Get everybody in the room, draw out the requirements. I have way, way crazier looking drawings for this that are actually reality that I would never make it to the slide because it just looks crazy. But again, everybody sit down. What's going to happen? Because some models are not going to be appropriate for that amount of data. You want to make sure you're pre-processing and you're doing the data properly based on those assumptions later on. Again, you know, we're for MathWorks. A lot of people are really in this boat. A lot of people using these tools, like they really, really need to do something with it. I mean, you can do research plenty, but you know, you really want to act on these models, right? So we've tried to like put little apps together that just basically package everything up, including all the dependencies. And it's really kind of enables the handoff, right? Remember that picture that was the lovely team? You know, you're going to hand something to the system architect. And the data scientist, the, you know, process engineer may or may not be familiar with that next step. You know, do you know about CUDA code or whatever? So, you know, trying to just make that handoff easier is really key. And then again, just that person, the next person just go ahead and implement it, right? Sounds really simple. It's not, you know. All right, finally. I have to take a drink and a big breath for testing because it's so, so, so important. So I said rigorous testing in my little bulleted point list, but that's a complete understatement. So, I mean, again, we've saw the stats. It was 11 billion miles or some crazy thing. Just so important to think through all the edge cases. You know, you might have that 1% or 2% of the model, you know, or the accuracy of the one that we're working. What was going on? Really, really understand and think through all the edge cases. So, you know, at least at MathWorks we have a bunch of test engineers. Like, that's a whole thing. It's a whole world. So it's their job to think through all those edge cases and, you know, write all these things down and communicate. So again, very, very important for real systems. Also, you want to actually test this in reality. So you also want to think about that handoff, right? So before the data science test hands over the explainable model, make sure you've tested that, right? And not just in your data science testing way. Some of us are just, yeah, it works. Here's my validation code, it's great. But again, make sure you have all those edge cases. Try out all different combinations of your data and in production. This is an example of using a server, you know, just spinning up a one node server to make that data scientist, again, really confident that in the whole system architecture without the rest of their world that this is really gonna work. So, very key. Excuse me. So this was a way messier version of the early slide that I was gonna try to do, but I thought it was, well, I was told that it was a bit much, saying test, test, test, test, test. I can just say that, but I think it's really important because when you're drawing out the system, okay, yes, I want to make sure that the data prep is tested. You know, oftentimes you're doing smoothing or you're doing some, you know, interpolation, things like that. You want to test that also. You know, it's not just the model, it's not just, you know, the rest of the system. And again, requirements, gathering those requirements at every stage is just really, really important and before you even start, right? And so, I should also emphasize, test driven development is a really, really great way is just to get everybody thinking about testing before you even start building the model, right? I do work with a lot of scientists, a lot of engineers and they're like, I hate testing, it's the worst, I never write tests. And I'm like, what do you mean you don't test? It's like, you didn't use your model? Oh yeah, I used my model. That's at least a baby step in testing, right? You know that it worked. So all you have to do is really take that, package it up and do it so that you can run it over and over again. So it really doesn't have to be a boring, scary, weird thing. I think I would love to see a million talks about unit testing at these conferences, but maybe it's just not as appealing sometimes. But you know, I think it's a really, really important skill and it's not even a big skill. If you can write a model, you can test your model, you're good to go. So again, that's really important. We, you know, I mentioned at Mather's, we have a bunch and bunches of QEs and quality engineers, this is their job, their entire world, I mean, during work. It's just about testing, right? So we've actually, many years ago, we actually exposed the unit testing framework that we've developed in-house and added script-based unit testing. And so again, the folks that are just like, yeah, yeah, I wrote a bit of code, okay, copy and paste in your script, cool. Then you can integrate it into the test suite, no problem. Again, it's just really mindful of the people that whose life is, you know, who thinks about testing like this versus people that it's kind of an afterthought, we can still all test together. And it'll be great. So the next step, again, just kind of the final step would be to test in the actual real system. This is not quite the real system, it's the very next step would be the real system. But there are techniques, again, you just try different driving scenarios in this case over and over and over again. If they fail, it's okay, maybe you don't have to pull the plug, necessarily. But you know, document all of that. And again, you know, you can create, in this case, just a report that you can share with other people, you know, how did you come to these results? You know, what are the cases when they failed? And you know, what can we do about them? Can we do better? Again, very, very important to pretty much everybody in this process to just be on the same page. So, you know, now we suppose that it's tested, we need to kind of bring it all together. Remember, lots and lots of people are involved, many systems, it's just crazy. But really, it's been together the whole time, at least it would be good if it was together the whole time, that's the ideal. Right, so before you even get started, you could use Git, you can use tools like this to just kind of keep everybody on the same page, so you're always being very transparent about what you're working on. And you know, if you don't have to have, you don't have to call a meeting necessarily to get on the same page, you can just take a look at the code. Another thing, you know, again, Mather's working on kind of bringing everybody into this world of testing, of using Git. So even if you're not familiar, you know, you could use apps, you can just, just get people to really buy in, again, enable everybody to be part of this world. You know, I don't know about the lawyer, but at least everybody else in the system. So I've made it sound quite difficult and scary and challenging, but there are cases where this is honestly being incorporated into these kinds of systems. There are a couple of links, these should be available, I guess, afterwards if you wanna check out some of the ways that it's being used. And even some of the videos, these are, this is, you know, from a real system. So hopefully you've gotten some ideas or some techniques about how to get more confident, how to be more confident in your models and some of the things you need to think about if you are working on these kinds of systems or you're hoping to work on these kinds of systems, don't forget, testing is the most important thing to me. So thank you, we have several minutes for questions and I very much thank you for your attention. Thanks. A question, yay. I haven't heard a single question in any of the talks. This is great. Yeah. Hi. Hi. Thank you, it was interesting. So I have such question. You said that for model to be explainable, sometimes it's better to lower complexity of your decision. And for example, if we use deep neural networks, it's always like a black box for us. And we can lower our accuracy, but get model easier, easy to understand why we choose, we get such results. But I think sometimes it can be difficult because for example, when we diagnose this cancer, we need higher accuracy, it's really important. And how we can make right balance between accuracy and explainable? What do you think about it? That's a very important question, thank you for that. Because that's exactly what's happening. So I think those of us in the field just having, doing AI models, deep learning, it's awesome. And sometimes it's a semi-transparent box or sort of an opaque box. If you are doing say a CNN, you know what a convolution is, like it makes sense, especially if you're working on images already. So I think sometimes these things can still be explainable, even all the visualizations that you saw with TF Explained, some of the MATLAB things you saw. So it's not like you can't use one of the deep learning models that it's just off the table. But I will say that involving lawyers, involving managers, it's very challenging. For example, we had a panel, I've done a couple of panels on this, and we've had researchers, professors, and people from the certification bodies that I was talking about. And the questions and the hesitation is just profound coming from that area. It's just like, no, do not talk to me about this. I can barely handle F equals MA, just because of the seriousness of how it is. So I think that's the important trade-off, is maybe to bring people along a bit. We could start a little simpler, but again, it's obviously, you don't want that to be the only thing detecting a pedestrian, right? You would also want to use other systems for a while. So I do think there's, we have a little bit of a communication challenge, but I do agree that accuracy is very, very important. Definitely don't be afraid to use deep learning models, just try to document them as best you can, and you don't have to use 8,000 layers necessarily, maybe even trim stuff down just as simple as you can. Great, thank you. Anything else? We'll be, we have a booth somewhere out there, I don't remember the number, but I'm happy to show some of these models, talk more about this kind of stuff. So again, thank you very much.