 All right. Good afternoon, everyone. Welcome to the Purdue Engineering Distinguished Lecture Series. My name is Dimitri Peroulis. I'm the head of the Elmer Family School of Electrical and Computer Engineering. And I'm excited to open this event today. If you haven't been to one of those lecture series before, let me just say a couple of things. The Purdue Engineering Distinguished Lecture Series was introduced in 2018. And the goal is to invite world-renowned faculty and professionals to Purdue University to encourage thought-provoking conversations and ideas with faculty, students, regarding the grand challenges that we're facing in our fields. Besides presenting a lecture, which we're about to enjoy in a few minutes, to a broad audience of graduate, undergraduate students and faculty, the speaker engages with an interactive panel of Purdue faculty and students, and that panel will follow right after the lecture series. And now to introduce our speaker, I would like to invite our Dean of Engineering, Dr. Mark Langstrom. Well, hello, everyone. And welcome to today's Purdue Engineering Distinguished Lecture. So as Dmitri said, for the past four years, this series has brought leaders in academia and industry here to Purdue. And today we're very happy to welcome Professor Nancy Levison, whose lecture, A Paradigm Change for Safety Engineering and Security, will set the stage for our panel discussion. Now, this word paradigm is interesting. You know, I think we hear it not infrequently these days, but I can remember the first time I encountered the word. I was an undergrad at a friend's house, and I spotted a book titled The Structure of Scientific Revolutions by a fellow named Thomas Kuhn, who I didn't know who he was. I don't know if he was famous at that time, but he certainly became famous later on. Was he famous at that? I asked my friend if I could borrow the book. I took it home and I read it straight through. It was fascinating. Kuhn defined a paradigm as an overarching conceptualization of science within which scientists operate. It provides the guiding principles that enable science. The paradigm serves as a taken for granted view of the basic accepted scientific propositions. And he discussed how scientific revolutions are a paradigm shift. So problems in the current paradigm crop up. They don't appear to have solutions. No one worries. They just assume that we're just not clever enough to solve the problem. We'll figure out how to do it. It's only after years and years of the best minds in the field being unable to explain the new data or to solve the problem within the current paradigm that you begin to think that maybe a new paradigm is needed. And even then, you're very careful not to switch to a new paradigm because the new paradigm must explain everything the old paradigm explained, plus the new data or problems that can't be explained within the old paradigm. So when we talk about changing a paradigm, we're talking about something very significant. And that's what Professor Levinson is discussing today. So I'm looking forward to her talk. Professor Nancy Levinson is the Jerome C. Hunsaker Professor of Aeronautics and Astronautics at MIT. Professor Levinson conducts research on all aspects of system safety, including modeling and analysis, design, operations, management, and human factors, and in the larger arena of systems engineering as well. Her techniques are used in a wide variety of safety critical industries including aerospace, transportation, chemical plants, nuclear power, health care, and many others. A common element throughout all of her work is an emphasis on applying systems theory to complex problems. Professor Levinson received all of her degrees in math, management, and computer science from UCLA. She spent her formative years as a computer science professor at the University of California, Irvine. She then moved to Seattle in 1993, she says in search of rain, and more recently to MIT, she says in search of even worse weather and new fields to conquer. So in the process, she morphed herself into an aerospace engineer. So this venue in the Neil Armstrong Hall of Engineering is particularly appropriate for this talk today. In keeping with the interdisciplinary theme of her work, her host for this visit is the School of Electrical and Computer Engineering. And I also hope that she notices what wonderful weather we have here in West Lafayette. Professor Levinson has received many honors, most recently the 2020 IEEE Medal for Environmental and Safety Technologies, and she was elected to the National Academy of Engineering in 2000. Please join me in welcoming Nancy Levinson. Thank you. Alright, so this is interesting because they haven't given me a thing. This is the only way I'm going to see what's going on, and I haven't memorized all this. So I have to look down occasionally just to remember what I put in this. And I'm going to talk about, this is a true paradigm change. And Cun, when he talked about paradigm change, said it takes a generation for a paradigm change to occur. And oh, this might, oh good. And so, and I've been surprised because this isn't taking, this is a true paradigm change. I think I'll be able to hopefully convince you of that, but it's not taking a generation. And I think the thing is that people are so desperate for a solution to this problem that they're willing to try anything. And when I came up with it about 10 years ago, I had a small meeting and people were screaming and yelling at me about what an idiot I was. I mean, it was amazing. In the halls, they were still screaming and yelling at me. And I thought, well, this is going to be difficult. And especially some people from General Motors, GM, said, oh, you're just an idiot. This can't possibly work. And so they said, well, we're going to go back and we're going to try your technique and ours. We're going to show you it doesn't work. I thought, oh, great. And the next year I had another meeting and they wrote an abstract and they said, we're going to tell the results of this comparison. I said, well, we have to sort of invite them. And we did. And amazingly, they stood up and said we tried it and it worked. And they've been the biggest users since. So what I'm going to tell you is a little bit about what this is. And first of all, how did I get into this crazy area of safety engineering? I bet there's no course here at Purdue called safety engineering. And there isn't, most anywhere. And it turned out I had just gotten my PhD at the UCLA. And I went to my first faculty position at UC Irvine. And it was a week. My dissertation had been plagiarized. I couldn't publish it. I was bored with the topic anyway. And I got a call from someone in a local aerospace company. And they said, well, we have this problem. We're building this torpedo. And they said, we're not so concerned it misses the other guy. But we're concerned it turns around 180 degrees and hits us. And we got 15 computers on this thing, 1980. And we never had any built anything with that many computers on it. At that time, this was pretty new. And we're really nervous about this. So we're looking for someone who knows something about software safety. And I said, well, it sounds like software reliability to me, but I don't do that. I do formal methods and I do beautiful math. And I don't know anything about a torpedo. Never seen a torpedo. And they said, well, we got some money. So I said, well, let me put Professor Levison on the phone. She's the software safety expert. I didn't really fool them. They knew I knew nothing about software safety. But they told me they called everyone else in the area and couldn't find anybody else who would even talk to them. So they said, well, we'll try her. And I didn't help them very much. But they taught me a lot about system safety engineering. And these were people who had been on the early ballistic missile systems and early warning systems in the 1950s. And they were really good engineers. And so I learned a lot, but I don't think I helped them much. And this is actually a picture that torpedo. And they called me after a year and they said, well, are you interested in what happened? I said, well, sure. They said, well, we went out and tested this torpedo. And it comes out of a, oops, that's not what I wanted to do. Where's the, never mind. Too many buttons. Too many things I can't figure out what they are. I mean, nothing ever has letters anymore on it. Everything's a picture. Why don't they just take pointer? Anyway, it comes out of the torpedo tube up on that right there. And they said, came out of the torpedo tube every time they fired it, came out of the torpedo tube, turned itself off, went down onto the bottom of the ocean and just sort of lay there. And I said, well, it's safe. And they said, well, the Navy doesn't want to pay for this safe torpedo system. So I said, well, what did you do about it? And I was out of there by then. And they said, well, one by one we took off safety devices until we got it reliable enough and safe enough and the Navy was willing to pay for it. And I thought, well, this long term doesn't seem like a great solution. Build something and then take things off. There's got to be a better solution. And I'll spend six months on this problem and then I'll go back to my beautiful math and I'll solve it for them. It's now been 42 years now, but I have the solution. So what I'm going to do is tell you about what I think the solution is. But it took me a while because the first thing I did was try and do all the things that people were doing then and just extend them to computers and software. And it didn't work. So I tried that for 20 years and gave up and said, we have to do something new. So I'm going to, unlike most academic speakers, I'm going to end with the conclusions. In industry they call this bottom line upfront or bluff. And so here's the conclusions and what I'm going to talk about and then I'm going to fill in the details. The first is that complexity is reaching a new level, a tipping point in engineering. As you all know, we're making things more and more complex. The old safety approaches are becoming less effective and new causes of accidents. We aren't even having the same cause of the accidents, especially those related to the use of software and the use of autonomy. The traditional analysis methods, which everybody's been taught and used, don't provide the information necessary to prevent these accidents and these losses. So I believe, and came to the conclusion finally, we needed a paradigm change from the old approach because it just didn't, I just didn't believe it was going to extend. And what is the change? Well the change in focus from increasing component reliability, which we're all taught, to enforcing safe behavior or enforcing constraints on system behavior. What is that bias? It allows us to create totally new kinds of analysis methods. And I'll talk about some of them that allows us to do things we just couldn't do. They're more powerful, they're more inclusive and effective, they're orders of magnitude less expensive, which surprised me because it's more powerful, it has to be more expensive I assumed. Turns out it is, doesn't, works on extremely complex systems. These techniques now that I'm going to, in this approach I'm going to talk to you about is now being used on the most complex system being built today, mostly military systems. And ballistic missile defense systems, early warning systems, defense systems of various kinds, and it's working like a charm. And it helps you design safety, security, and other properties into the system from the beginning. The problem in industry is that people build, start building systems and they get them to the test stage and they find out they've got a problem. And then they have to change something. Downingly expensive. It can be a thousand times more expensive to fix something when you find it at test than at the very beginning, which costs nothing to do it right. So we just can't afford this. This is why all these systems you hear about are, you know, hundred times, a thousand times over budget, 20 years late, the web telescope, which is doing wonderful things with astoundingly late, all because of software problems. And I really want to hear about the budget. That was all our money. And so we can design these things from the very beginning, which cost them nothing. And does it work? Well, we've now had probably at least a thousand empirical and controlled studies, controlled or careful experiments. Empirical what I mean is industry has a system. They'll make a shadow project. They'll do it their normal way. Then they'll do it using our tools and then compare the results. It's not controlled in any way. But every single one of them has shown that this is better. There's a couple university experiments that didn't, but mainly because they didn't know how to do our analysis techniques. They did them all wrong. What was amazing to me is they actually came out pretty good given that they didn't do it right. But otherwise, this stuff works. So let's start with the definition first. What do I mean by safety? Because it's not something that's normally that you hear about in your classes. I define an accident. Mishap is the word that the Defense Department uses. Either anything is just a loss. It's any undesired and unplanned event that results in some kind of loss that someone cares about. It could be loss of human life or injury as you expect. Property damage, you expect that. It could be environmental pollution. It could be mission loss. It could be lots of protected information. Security. It could be a negative business impact even. Any kind of thing that you want to avoid, you can use this on. And I didn't say anything about whether it was intentional or unintentional. So it includes security. Although security hasn't used as much. I'll talk to you a little bit about that later. So let's look at the history here. There's a timeline there. It goes back to 1850. I can find journal papers and mechanical engineering journals on safety and designed for safety that long ago. It really has actually a long history. But in the early, in the 1800s, mostly they were concerned with designing guards on equipment, protecting against certain kinds of behaviors. Unfortunately, after around 1922, to World War II, they went astray and decided everything was a human error. So everything was a human's fault. And then World War II came along. And after World War II, they realized that we were building systems like ballistic missile systems that were blowing up regularly and doing all kinds of things. And there were no humans on these things. So they couldn't blame the pilots anymore on the plane. So they had to figure out what the world was going on. And they assumed that accidents were caused by component failures. All right? And so it's a reliability problem then. And they came up with all kinds of techniques. For me, it was failure modes and effects analysis. That was after World War II. That was after the electromechanical systems in the World War II. 1960s, lots of new techniques. FTIS fault tree analysis used widely. HAZOP is something used in the chemical industry. Event tree ETA is used in nuclear. And after about 1980, when I got involved with my torpedo, we started to introduce computer controls in everything. And now nothing's built without a computer. I used to say a bicycle, and people now tell me they have computers in the bicycle gears now. So I give up. Toasters. There's nothing without toothbrushes. Nothing without a computer in it anymore. The computers themselves weren't the problem, but they allowed exponential increases in the complexity of the systems we were building. We had all kinds of new technology coming in. And changes of the human role in systems. There were no longer just pushing buttons and reading dials. They were making cognitively complex decisions in monitoring all of these computers. The pilots, by the way, and an airplane and commercial aircraft, pilots fly the plane by hand about seven minutes on average during a flight. During takeoff and landing, they may have some. Otherwise, it's all flown by computers. And the pilots are just monitoring this stuff, but they're there, and I want them there because if something goes wrong, they're the ones that have to go save the day. But I believe that this changes that underlying assumption about the cause of accidents, and it's no longer component reliability that's the problem. And I'll show you examples of real accidents that will do that. The only new thing is our new tools, which came in about 2005, although what we were doing was so weird. I didn't want to talk about it. I thought, you know, I have a good reputation. People think I'm absolutely nuts. You know, she did good work when she was young, but now she's lost it. So I wouldn't even let my students do theses on it because I didn't want to ruin their careers before they started. And so we played around with it for about seven years, and we used it on some of the most complex systems of the world, like missile defense systems, as I mentioned, and it worked. And so then I started talking about it and published a book on it. And we have new techniques and tools. So what is the traditional approach? Well, it assumes that accidents are caused by chains of failures, and then we have all this analysis techniques to deal with that. We evaluate the reliability of the component separately and then put it all together and figure that's the reliability of the system, and therefore that's the safety of the system. And in design, we deal with component failure here. Auburn taught about redundancy, of course, barriers to prevent failure propagation, high component integrity and over design. You make it stronger than it needs to be. Fail, safe, divine. For humans we give them operational procedures, checklist training and so on. And I contend that this doesn't work anymore. And so let's look at some examples. If the problem is, remember the assumption is that accidents are caused by component failure. This is a real accident, and this is all I can tell you about it. It was some Navy aircraft, and they were just moving some missiles from one location, Air Force Base to another, actually Naval Base, I guess, to another Naval Base. And what they do is to move them is they just put them up on the wings of the plane like they would normally do if they were in combat and then move them that way. Well, someone decided the pilot shouldn't just be sitting there all the time. And so they said, well, why don't we have them test the aim and fire software? And so he aimed, he was told to aim at the aircraft in front of him and fire a dummy missile at it, which he did. Did not make a mistake. Nobody knew that this was so-called smart software. And it was built to, in case there wasn't a good shot at the target, it was built to move to another missile on the wing and shoot that missile. And it turned out there was an antenna underneath the aircraft which was between the dummy missile and the target. So it switched to the next, and fired a live missile at the guy in front. Now, the first thing I want you to think about, what failed? Yeah? The computer did what it told. I guess the failure is that there was a live missile loaded in the first place. Yeah, there was a system design problem, a system engineering problem. Absolutely. But the things are getting so complex and they're so big and the groups that are creating these things are so large. On the F-35, the software had 8,000 engineers, and not only were there 4,000 software engineers, but they're located in every state in the country. Why? Because that's where you get congressional funding. You make sure your senator knows that money's coming into the state and so they keep funding the project. So now we've got 4,000 people and 8,000 all together with the system engineers all spread out across the country, all with different companies. You can see how this could happen. But you're right, it's a system engineering problem. Good answer. All right. But just making that software better, you know, getting software errors out wouldn't have helped. Making sure the requirements were right wouldn't have helped. The requirements were right. They just didn't fit in everyone's conception of what the requirements, software requirements should do and it was a good idea to do it. Of course, you know, the guy in front, now, when the missile's coming at him, all the alarms in his cockpit are going off because we can detect all that. Someone's electronically shooting at us and so he took a base of action. But you can imagine what he's thinking. You know, I just went out with the guy for beers last night, you know. What? Why is he shooting at me? And you also know what happened. As soon as they got down on the ground, the management said, well, wait a minute, how did he get away from that? We got to fix that missile. Another accident. This is non-military. This is commercial aircraft, Warsaw. Landing in Warsaw is a Lufthansa A320 and when you stop, one of the ways you stop an aircraft, there's multiple things you use, but one of the ways is something called reverse thrusters. You probably feel it when you land in the airplane, you know, you feel like it's stopping quickly. All it does is change the direction of the power of the engines in the opposite direction so it stops you. Now, the problem is that this has come on the reverse thrusters when the aircraft was in the air and this is real bad news because it stalls you and you go plump down into the ground. So they put in protection and they told the software, well, don't allow the pilot. They blame the accident on the pilot, although no one knows why this happened beforehand, why it happened in the past. But anyway, they said, don't let that stupid pilot turn on the software when it's in the air. Now, how does the software know that the aircraft has landed? Well, there's certain clues, things like weight on wheels, the wheel spinning rate, and other things, but it turned out it had been raining really hard. There was a lot of water on the runway, the wheels hydroplained, but basically the software didn't think the plane had landed so it wouldn't allow the pilot to activate the reverse thrusters and you can see the result. It ran over a little hill at the end of the runway and caught on fire. Again, this was certified software. That's why all our current certification requirements, the software didn't fail. Pilots didn't fail. It's system engineering. It's the design. It's not understanding all of the factors that they had to consider. Here's another accident. There's been lots of them with the reverse thrusters and the reason is that the aircraft standards methods for certified aircraft don't include these. I've been yelling at the FAA and others for a while. I have a paper I call my shit storm report because when I wrote this, someone told me it caused the shit storm at the FAA, so I'm very proud about my shit storm report. But I've been telling them this for a while. They're finally getting, finally winning 10 years later. This was a Tupelo different plane landing in Moscow. It had a soft touchdown, but it hit the runway later than normal. And there was a crosswind. So the aircraft, the software didn't think it had landed. Wouldn't allow the pilots to activate the reverse thrusters. Now that looks like the same accident, except there was something a little different here because the pilots believed that the reverse thrusters were going to come on. And so because they landed, remember down the runway, they had less space. They engaged high engine power to stop faster, but the reverse thrusters didn't come on so it just went off the end even faster. And in fact, in complex systems, human and technical considerations can't be isolated. But that's what we do. Human factors engineers tend to look at the screen out. Hardware and software engineers look at the screen in. The interface is the screen designer, the controls design of various kinds. But nobody is looking at the integrated system as a whole. And our problems now aren't in the screens. Our problems, for example, we have things like modes confusion. The software so confuses the pilot that thinks it's in a different mode than it really is. And so the pilot does what seems to be right to him, but turns out to be wrong. Situation awareness, all of these things can't be found by separating the engineering activities. So what we do now, we have software analysis. It says, how can the software contribute to a loss? We have the hardware engineers looking at how can the hardware fail? We have the operators and the human factors people analyzing how can the operators deviate from the intended or specificated procedures, but there are these new interactions between them. And our tools don't handle that. We need to do something about this. And I have a feeling that if Purdue, there may be a few enlightened people, but a lot of you are taught that these are really separate engineering activities. So we need to look at the integrated system as a whole. We're overlooking problems by breaking the analysis into pieces. So the problem, today the traditional techniques don't work on today's systems. They don't handle complex systems. They're too late. After you've written a car today has 100 million lines of software, you can't after the fact then decided, well, we wrote the wrong software. You can't change it without introducing lots of new problems. The hardware, software and humans are treated separately. There's no way to extend, and I tried for 20 years, to extend the old techniques to include these things, and they don't extend. And there's good reasons why, because the underlying assumptions are wrong. So I concluded we needed a paradigm change. This is my favorite cartoon. It's still hungry and I've been stuffing words into it all day. It's not that people don't mean well. They're really trying to solve the problem. But the problem changed on them. And they're still using the old solution on the new problems, and they're not working. So the problem is complexity. And how do we cope with complexity? Well, about 200 years ago, more, Descartes and others created the scientific revolution by figuring out we could decompose systems into components, analyze the components individually, and put them back together again. Sometimes we use statistics on complex systems, and the third possibility is systems theory. So let's just real quickly, and link to composition. As I said, you separate in separate parts. If you're looking at behavior, you're looking at events over time, each event leading to the next one, or we have a system with all these interactions between the components, and then we analyze each one separately and combine the results. But there's assumptions under there. One of the assumptions is that looking at them separately doesn't distort the phenomenon, and that they operate independently, and so on. There's a lot of various assumptions under this. So this is a way, by the way, this is right from the certification standard for commercial aircraft. So what happens is we have a... Sorry, no pointer. The loss of the aircraft at the top of the tree, and we decompose it into functions. So we have... Oh, I went too far. We have the control thrusts, control the flight path, determine the orientation, and so on. Then when I said control aircraft on the ground, and then we have a second level, and we break those into pieces until we get down to the bottom. And then we start analyzing bottom up. So what happens is this is just one piece of it. We look at the probability of failure of each of those things. We combine them together, and we get a probability at the top, and it has to be 10 to the minus 9, so it will be, because that's what you need for certification. There's some problems, though. The failures of the components have to be independent. How do you show they're independent? Very hard. Complex system. They don't work for non-failure accidents where nothing failed. It doesn't work for software, new technology. They don't put software in those trees. But if they put a box that says software error, then they put a probability 10 to the minus 4. Why 10 to the minus 4? Well, because some very bright computer scientists wrote a paper a long time ago that said if you tested software under ideal conditions for a thousand years, you might be able to get 10 to the minus 4. They said, oh good, we'll use that. That's where they came from. There's nothing wrong with the paper. It's just stupid use of it. It doesn't work for human errors. Humans don't behave randomly anymore. It used to be human error was you could put people in a laboratory situation and see how many times they misread a dial or they hit the wrong button. That's not what they're doing anymore. They're doing these complex, very sophisticated complex activities in their minds. And there's no way to get a probability that they did that wrong. So they just ignore it. There have been some... I've looked through the last 80 years and I found two accidents that this actually... Two examples of someone trying to see whether this works. One was in the 1980s and it was total failure and I can't get the original paper. Someone just wrote about it. But there's a newer one in 2002 where they carefully evaluated probabilistic risk assessment and it was a total failure. They had different teams calculating experts calculating the probability of failure of the same system and there were three to four orders of magnitude different. That's way too much. Not useful. The empirical results are terrible. All the accident I've ever seen had a probabilistic risk assessment that said it shouldn't happen. I was telling some people earlier in the day I was called by some people in the UK who had a drone. They bought it from the U.S. It was a surveillance drone, a military drone. And it had a calculated meantime between failure of 10 to the minus 12, which if you look at the number of hours and stuff it was about 500,000 years before an accident. And I said, well, yeah, that sounds great. And they said, well, we've been flying this for a year and it's crashed five times. I said, that doesn't sound so great. So they said, do you think what you're doing will help? And I said, yeah, I think we can help you and they're using it now. If we knew enough to measure it we'd fix it. If it's a design error, you don't leave the design error and hope it won't happen. If you know enough about the design error to know what it is, fix it. So what we have is a situation where the things at the top where there's a high level of randomness we can use probabilities and statistics for. If things are relatively simple in terms of coupling and complexity we can use analytic decomposition. But that blue stuff is where almost all of our systems are now. And it just doesn't work. People like to think it does, but it doesn't. Our systems are just too tightly coupled software intensive, highly automated and connected. They're just not going to work. So I believe we need a paradigm change. What's the paradigm, promising this, what's the paradigm change? Well, the old paradigm says prevent failures or accidents and treat safety as a reliability problem. The new paradigm says enforce constraints on behavior. Now, that includes constraints on the component behavior. So it includes preventing failures of components when that's necessary. It includes all the old things. As I think you said in your introduction paradigm changes usually include the old things but it goes beyond. And it includes things like the interactions controlling the interactions among the components. And that's treating safety as a control problem. And it's based on something called system theory. I didn't create system theory. System theory comes in the 1940s, 1950s. It's been adopted in a lot widely in biology and in other fields. Not so much in engineering, though I'm not sure why it should have been. And basically it says that interactions of components when the components are operating create these emergent properties that emerge from the interactions among the components. That you've probably all heard of the hole is great in the sum of the part. That's all it's saying. That you can't just sum the parts. And safety and security are emergent properties. So what do you do about them? Well, we control them, I said. And let's just use a simple controller. It's not going to be quite so simple always. But say we put a controller and most of you are probably engineering students. You see this. This is just a simple feedback control loop. The controller enforces certain properties by issuing control actions on the process and gets feedback to see what the results were, uses that feedback then to adjust and figure out what other control actions are needed. It's a broad view of control, though. I'm not assuming that there's actually this PID controller in charge or this controller in charge of everything. We can control interactions with design and do all the things we're doing now. Redundancy, interlocks, fail-safe design. That's a way of controlling the interactions among components or through processes, manufacturing, maintenance, operational processes or through social controls of various kinds. So this also includes social, the socio-technical systems, the social aspects of system. The controllers enforce constraints. There's just some example of constraints. Aircraft must maintain sufficient lift to remain airborne. Vehicles must maintain minimum separation. Public health system must prevent exposure to the public to contaminate water, food products and viruses. We've all lived through that in the last few years. So how do we treat safety as a control problem? Here's a really simple example. We have a controller. It can be human or automation, so it doesn't matter. It has some kind of algorithm or some kind of decision-making process. Every controller also has to include a model of the process that's controlling. The thermostat in this room or the building has to know what state of the system is at any one time. Incidents often occur that involve software humans when the process model is incorrect. It is inconsistent with the real state. So the software on the A320 and Warsaw thought the plane had landed when it hadn't. That explains most software errors and software unsafe behavior. Not their errors, but unsafe behavior. Catch your software errors, human errors, flawed requirements. Here's the Warsaw accident. I'll show you how you can do this. And you can put this all together. And this is all intuitive, but there are real algorithms beneath this. So the hazard is inadequate aircraft deceleration after landing. So we have a pilot now controlling the software which controls the actual physical aircraft components. So what happens? Well, feedback indicated the plane hadn't landed. The software controller's process model says the plane hasn't landed. The pilot says the plane has landed. His pilot crossed this model. He blinded it. You can see it's on the ground. So he turns on the reverse thrusters. The software decision-making says ignore that command because I was told to ignore it because we're not on the ground and we have a crash. Now what about the one in Moscow? Which was a little different. We had some human behavior was more involved here. But the same things. But in addition, the pilot said the reverse thrusters are going to come on because they always come on. It's a short runway. I need more power to stop. So I'm also going to issue an engage high power on my engine. The software says well I'm going to ignore the reverse thruster command but nobody told me to ignore the engage power command. Now I realize this is just an intuitive thing. We have tools, careful tools, step-by-step tools that will work through all these scenarios so you can identify them. And lots of math underneath because it's not considered academically respectable unless there's a lot of math underneath. But just going teaching you the math isn't going to really help you understand this first. You do that later. But there is math underneath. It's a whole formal foundation. What about cybersecurity? I haven't mentioned that. Lots of interest in cybersecurity in universities. We can apply this. It's a paradigm change from the way security is done today. It can be used for also the social organizational aspects of systems so we can look at supply chains, we can look at management decision making, all these other things, factors in security. It's being used now on Air Force Project. It is taking longer to get integrated in the industry. It's much harder to do, change the world with this. But they will. Because what we're doing now doesn't work. People break into everything. I mean, the Chinese know more about me than my mother because they've stolen all my security information off the DoD computers which are supposedly have the highest security in the world. I don't have time to tell you how this works, but we applied SCPA, that's our analysis method, the one DoD program before the SolarWinds attack. Most of you who are in security know about SolarWinds. One of the only programs that wasn't affected by SolarWinds was the one we applied the SCPA to. And if you look at there, the top is program office and program buyer, vendor, all the way down to operating system and the software and all the stuff. You can look at every level of this. The blue there and the orange are the vulnerabilities that SCPA found. The blue ones are the ones that were later exploited by SolarWinds. The orange ones are additional things we found which weren't used by SolarWinds attack. And as I said, it was unable to get into this program. So, what can you do with this? We have an underlying formal model of accident causality which I call stamp. It's not a tool. It's a formal model of why accidents are caused. On that, we built all kinds of tools, accident analysis, hazard analysis, security analysis, all kinds of things. Identifying the indicators of increasing risk. You can use those and put them together. What are we doing now? Because that's all being used. We're doing a lot more on top-down concept development, creating requirements, architecture and design using this technique. So, you build safety in from the beginning. We're trying to convince people they should do this, build security in from the very beginning. You can come up with new approaches to risk assessment, extensions to new types of technology such as machine learning, sophisticated teaming. Now, people want to build systems where we have robots or drones and we have humans and it's not like the humans are controlling the software. Now, the humans and the robots are working as teams of things. Maybe operating on the same thing or perhaps having to cooperate but in different activities. And that's just a problem we can't solve right now and we will. It also is applicable to all kinds of emerging properties. Evaluations, as I said, we've had hundreds of evaluations. All of them show that it works better and is cheaper. Is it practical? It's been used a lot in just about every industry. We used to get about 100, less than 100 people to my research group meetings every year. We now get between 2,000 and 3,000 from last year, 87 countries. All people are using this. New standards are being created. If you want more information, as I said, I couldn't teach you this and I thought you would be better to stay at the higher level. But there's lots of information, all free. We have a website, lots of papers. We have a book. You can download the book free from MIT Press. There's a handbook on SCPA, which has been downloaded now over 200,000 times. It's been translated into Japanese, Chinese, and Korean. If you want it in a different language, CAST is the accident analysis method. That's also in Korean and Japanese. And the one advertisement I want in my last 60 seconds is I have a new book coming out. This one won't be for free. Sorry. I had to get the publisher angry enough that I'm giving out free copies of his old book. It's an introduction to system safety engineering because we aren't teaching this in universities. And I don't know why. It's been around 150 years. It's time. But there aren't classes, almost never classes. What happens is people get hired out of engineering schools and they get assigned urine safety engineering. Go learn about it. And they have no training, no nothing. So I've got a new book coming out. It's historical perspectives, looks at risk, fundamental concepts, why accidents occur, the role of software in accidents, the role of humans in accidents, accident causality models, accident analysis, hazard analysis, all the different, all the old stuff too. It's not just the new stuff. I've included all the old things that people are still using. Design, how to design for safety, human factors, assurance assessment, safety management system. And then I have a bunch of appendices, real examples. Many of these, I was on the formal accident analysis investigation board and helped write the reports. So I have all kinds of information not in the reports. Each of these, I have 20 to 30 pages of description of these in Space Challenger, in Columbia spacecraft, in petrochemicals, things like, well, the newer ones, Texas City, Macado's Deepwater Horizon, and in Nuclear, Three Mile Island, Chernobyl, and Fukushima. So that's it. I hope that I haven't blown your mind too much or turn you off and say, oh, this woman, what's she saying about these crazy things? We've never been taught any of this. And unfortunately, you haven't. But you will be in the future, or students will be in the future and should be. And I've actually talked to some people today who are using this here at Purdue in their research. Not too many. Most of it's in industry now. But that's because academia is very conservative about changing things. Industry says, if it works, I'm won't. But that's not necessarily the same view in the academic world. So thank you very much. Thank you so much, Professor Libeson. That's an amazingly inspiring talk. Thank you. We will take about five minutes for questions now. And then we're going to have a short break and we'll go into our panel. So let's go for questions. I want to hear more of my cynical view of what people are doing now in system and software engineering. Well, thank you for a very interesting presentation. And I'm looking forward to us getting together tomorrow morning to talk when we'll change your paradigm on Purdue, that two buildings down, we do teach safety engineering in the chemical engineering department. Oh, they do. And we have a research group. And not only do we teach it, it's required of every senior. So I have 180 kids in my class. I'm amazed. That's wonderful. But your paradigm is right. We are one of the only schools in America to teach it. Yeah, that's what I thought. I look forward to our discussion. Oh, that's wonderful. You know, you are the only one. I just haven't heard about this in the world. They don't teach it in the rest of the world either. But you're right. The companies expect to teach folks when they get there and not know it. And this is very highly thought of by the recruiters that come to campus. One major company hired 16 kids out of the class one semester. I had an undergraduate. It took my graduate class one semester. And I sort of wasn't so sure about it. He sort of had a C minus average. And I wasn't too sure about letting him take it. But I said, well, he'll try and then he'll kick and drop out of me. And he got the highest offer that year of any of the graduating seniors. And I was amazed because his grades were not good. But that's wonderful. I am so excited. I can't wait to talk to you about this. Go ahead, sir. Hello. Thank you for the lecture. My name is Sod. I'm a sophomore in computer science and math. So going back to what you mentioned about the demerits of probabilistic risk assessment, it is used in financial markets and economics too. And in my opinion, it has mixed results. Well, it's been successful for quantitative research at hedge funds. We can't really say the same for global economics because we all here, we can remember at least one time where the economists at the Fed have said, I don't see a financial crisis occurring in our lifetimes. And yet millennials today, they're going through the third once in a lifetime financial crisis. So what are your thoughts on using this new approach for this use case? It's not exactly a system we can build, but I guess we could control it to an extent. So I sort of got most of that banana. I'm getting old. I'm losing my hearing. Let me paraphrase and correct me if I'm wrong. I think it's related to applying systems and this type of paradigm in controlling the financial system. Yes. Actually, there's a big group doing system dynamics in the slow management school at MIT. And this is used in finance. It's been used in management. It's been used in Margaret Mead, who's really a systems theorist in anthropology. It's engineering that isn't. Biology is all now changed because it was actually started by a biologist, Ben Bertolinfey, in the 1940s, 1950s. He gave it the name system theory. There were some people in computer science that came up with it. Norbert Wiener is one. They called it cybernetics, but it's no longer, that's sort of an old outdated name. But it's used a lot and it's time for us, for engineers to do it because we are building fantastically complex systems and we ignore the humans in them for the most part and we ignore the social parts. If you look at the Boeing 737 MAX accident, this was not a technical asset. There were such stupid things that were done that it wasn't. It was all because of management decision making. They forced them to do these, make these stupid decisions. And so we've got to look beyond just engineering, the standard engineering. And I understand a lot of people aren't comfortable with that. We also need people looking in depth in the technical parts of systems too. But there's not enough looking a little broader at the system engineering aspects. Let me ask you a question unless there is another one we'll break here. Let's say I'm a student. I'm really inspired, which I am, by what you're presenting. And I want to work in this field. What technical or other background do I need to study before I can understand your books? Well, you need to get my new book and I'll send you a free copy because it's still until it's published. Until it's published, I can do that. Otherwise, then the publisher is going to scream at me. It's not going to be that expensive. I've been holding out for making it less expensive. But you can start with that. You can start looking at our paper. We have papers on the use of these tools on every kind of system you can imagine. Students are all in different fields, so you can look at it in your field. Do I need to study control theory, computer science, aerospace engineering? All the control theory you need, you saw the day. You need to know what a feedback control loop. That's what we do. The underlying math is kind of, we do it because we want to make sure that the stuff's right and has a theoretical foundation. But it's not really necessary. Engineers don't know it and don't care about it and don't need to. People are just picking up and using these tools. Mostly students. I get a lot of mail, a lot of mail. But from students all over the world who say, I want to do a dissertation on this, but nobody at my university, no faculty know anything about it. Would you help me? And nobody in my university is willing to learn about it. So you do have to find somebody who's willing to be open-minded, but it's not hard to teach yourself. There's nothing hard about this stuff. It's just different. It's a different way of thinking about the problem. Instead of thinking about all the ways that will go wrong, like Femia, forward techniques, you start from the thing you don't want to happen, and then you say, how could it get there in the worst case, and I'm just going to prevent those scenarios. And it works. Wonderful. Last question. Hello. I'm a freshman going to mechanical engineering, and I was just curious about the role of human overlids in the system in the sense that if there's a conflict between the human decision and the software's decision, the human can override and have manual control of it. Yeah, that's a difficult decision, of course. As engineers, we have to make a decision. Who's going to have the final say? Somebody has to have the final say, right? We are so used to blaming humans, the accidents on humans, that we then trust software more, and we shouldn't, especially on these new systems. But a lot of it is just not understanding how humans think, and we've got to get better human factors classes, just introduction to human factors. So the autonomous vehicles, look, except in very limited circumstances within a plant, the AGVs, the automated ground vehicles, going around plants, we can do that. That's easy. Roads and autonomous cars, we are decades away from that. We spend $100 million or more on that, and it's not going anywhere, and it's not going to. And humans are going to have to be involved. But then, humans make mistakes. I'm not saying humans are perfect. We make mistakes. But humans have a lot of advantages. We're adaptable. We can solve problems that we haven't seen before. We can come up with unique solutions, and we do all the time. The solve things, and you say, where did they think of that? But save the day. But that means also, sometimes they'll do the wrong thing. Nothing can be perfect. So it's a real difficult problem. Who do we trust? And at this point, you know, we could have, I told you, pilots fly seven minutes out of a flight, and that's not critical. We could easily fly all our commercial aircraft without pilots, and the airlines really love it because pilots are expensive, and they've been trying to do this. There's no way I'm getting on a plane unless there's a pilot, human pilot in that cockpit. Not that I think that that human pilot is perfect, but two reasons. One, if something goes wrong, they may be able to figure out how to fix it where the software isn't going to. This thing about intelligence software. Yeah, it's intelligent, like a five-year-old, but I don't want a five-year-old flying my plane. And I want someone up there who's going to go down with me. And those are going to go down with me, and they're not just sitting in some bungalows somewhere and some trailer, like they do now with the drones, military drones, somewhere in Arizona, sitting there and watching this as if it's a video game. This is real, and I'm up there. Well, let's thank our distinguished lecturer once more.