 Welcome to my lecture on Validating the Simulation Model. gear gives you an idea of the place of validation in a simulation study. We've seen this before and here we're highlighting the validation stage. So naturally the simulation study is a large scale study. Simulation is just one part of it. It starts with the problem definition. Probably even before that recognizing that there is a problem. The system analysis and design phase including data collection, doing your analysis. And then finally model design meaning the conceptual model, the logic flow of the model. And then finally the implementation of the model in a particular software system of your choice. You could be using a general purpose programming language. You could be using a simulation tool like Simeo or Arena or SimScript. And from there you look at what your input to the model will be, the input data. We've done that. The programming part, we're actually doing that converting the model from a conceptual model to a runnable model. The verification phase, we're going to talk about the difference between verification and validation. Basically verification is similar in concept to debugging a program. And then here we have validating. And you can see how we could validate back to the model stage, the model building stage. Or we could validate back to the system, looking at the system in the real world. Verification pretty much does one and validation does the other. Once we finish the validation stage, once we come out of that, we have a valid simulation model to work with. And we have to finish up constructing our experimental design. What is it that we will want to do with this model? And which we, by the way, should have thought a little bit about before building it. And then finally simulation where we're actually running the model and conducting the experiment, the statistical experiment with the simulation model. So once we've done that, we've generated all the data of the experiment using simulation, then goes statistical analysis, decision making and implementing any decisions that might come out of this project experience. Why validate? It's an integral part of a simulation study, of course. What's the purpose of validation? Why does it have such a solid central place in the study? Well, of course, we want to produce a better model. And validating the model will help us to do that. It will also help us to determine if the model that we do have is a good one. So it helps us to prove model credibility. And we're never ever doing this kind of work on our own just for ourselves. We always have other people to bring in to the decision making process. And we have to justify what we are doing here and show that, yes, this really is a good model, and here's what the model does. Let's talk more about the difference between verification and validation. Verifying the model, validating the model. Verification is simply, here is my simulation model. Is it doing what it's supposed to be doing? Did I implement the conceptual model correctly? So it's really all about the implementation, not necessarily validating the conceptual model itself. So we want to know if the model works. Does it work the way it was intended to work? That's internal validation. Another name for it is internal validation. On the other hand, external validation is asking the question, is this simulation model an accurate true representation of the real system that we're studying? So it's more than just debugging. You're questioning the validity of the model itself, no matter what language or what software system it was implemented in. This is generalizing to the external world. This is external validation. We want to make sure our model was correct, validation. We want to make sure our model was implemented the way it says that it's supposed to be, that the actual running of the simulation model, the working simulation model, does what the conceptual model says that it would do. And finally, validation itself is goal-oriented. So you might validate your model for one purpose, but then the same exact model might not be valid if you're validating it for some other purpose. So validation depends not only on the model itself, but also on the outcome. What are you doing this simulation for? It's a mistake to think of validation, model validation, as an either-or idea. It's not. There's a degree to this notion of validation. Can you absolutely say for certain that a model is valid? No. Same way you can't absolutely for certain say that a program is 100% bug-free because that will never happen. And similarly, a model, and in particular a simulation model, you can never absolutely say for sure, with 100% certainty, that your model is a 100% valid representation of the real world. If you think of validity as going from 0 to 100, then as the validity of your model goes up, well, of course, the development cost of the model goes up. Your model becomes increasingly complex in order to mirror the real world. And therefore, the value to the decision maker of anything that comes out of this model may increase slightly, but at a decreasing rate. You reach the point of diminishing returns. In fact, the benefit to cost ratio, or the BCR curve, would likely peak at something less than the most valid model money can buy. And again, not 100% valid model, but even the most valid model money can buy. So as in a lot of areas of decision making, we're going to end up learning how to satisfy. There are a number of different ways that we can validate our simulation models. We're going to look at a small number of these, but they're the goodies. Number one, we're going to look at face validity, which is appropriate for many, many models. Testing the assumptions of the model, which is always a good way of validating. Comparing input-output transformations, which if it's possible for you to do it with your simulation model, that's kind of like the gold standard. And then, of course, using field tests. This validity means looking at our simulation model, does it seem valid on the face of it, on its face? In order to do this, we line up subject experts, some experts of this particular system that you're simulating. And maybe some of them will be simulation experts, but maybe some of them will not be. They'll only be experts of this system, the way it exists in the real world. Your model, if it has documentation, it will be a big help. Otherwise, you will have to be there step-by-step explaining. And sometimes your explanations may be like leading questions. So it's better as far as possible for your presence not to be required. Have the experts look carefully at your flow diagram, at your conceptual model. Walk through it. Have them walk through the system and ask, is this the way things happen in the real world? Does this seem right to you? Are these the kinds of processes and in this order and to this degree that entities flow through in the system you're familiar with? How about the input data? Could you take a look at my input data and see if that seems reasonable to you or if anything raises any flags? How about the outputs? Let's look at an output at one run, let's say, or several runs and see what the system produces. And does this look like something that would come out of the true life system that you're familiar with? You can also, in this sense, test the robustness of your model. Try running your model with different outputs, low, medium, high. Stress the model and see what happens and see what it produces and ask the experts, does this look like what would happen in the real world? Or is this something that we should try to make sure the model never reaches this state because it never happens in the real world? Of course, you not only want to check with experts, you want to check with other people who've been modeling in this domain because they have already done the work ahead of you and why not learn from their mistakes? Well, or from their successes. Close observation of the real system is very good and compare it with close observation of your simulation model, your running simulation model. And of course, when it comes to face validity, even looking at a close theoretical model, something that's close to what you're simulating, will help to validate and to justify the logic flow of your model. It's a good idea to do that together with subject experts as well. Part of building your simulation model and validating it includes testing the assumptions under which you built that model in the first place. You can do them one at a time or several at once, but this is where it happens. First, you've got structural assumptions, which basically include how does the system operate? Is your model operating in a similar manner? How do customers decide to enter the queue, leave the queue, choose a server? If there are multiple queues, do customers make their choice right to left or do they look to see if they kind of like the way the server looks? What influences their decision and does it make a difference? Is that influence important enough for you to build it into your model? Then we look at the data assumptions, which are also very, very critical. Look at the parameters that you used in your input distributions and fiddle with them. Try changing them, see what happens to the output and if it still looks reasonable. In that sense, I'm going to skip over data fitting for a minute and go right to sensitivity analysis because that's part of what we're doing here. Let's try crazy values and see how robust this data is. This model is to crazy data. Change the values of the parameter, make them very, very tiny. Try a negative value, make them very large. See how far out you have to get before the model really deteriorates. This is all under the heading of a stress test. This is really basically kind of like working with machinery. You're stressing it and you're looking to see what happens and how badly you can stress it before it actually goes kablooey. Getting back to data fitting, which probably is misplaced here, but I'm not redoing the slide. This has to do with looking at your input data and seeing what am I trying to do here? Do I have input data that I collected from the real world? Am I using the data as is? Am I using it in an empirical model? Am I trying to fit this data to a theoretical model? Or how about my output data, the output from the simulation? If theoretically I have information that tells me the output from the simulation should be log normal, then maybe I should test the output from the simulation model as part of my validation process. Any computer program, any algorithm, and most particularly a simulation model, is an input-output transformation device. It takes input, runs it through some kind of processing and produces output. And indeed the real world system that we're simulating is the same. We look at the inputs and then we look at the outputs. So the real system is also an input-output transformation device. We could have a stream of inputs and we could have a stream of outputs from several systems or from the simulation model and the system that it's trying to model. And we can compare them in some way. And if they're close enough, we can declare the simulation model to have been validated. Well, declare is a little bit strong, but you know what I mean. That's a matter of degree. So how do we do that? Well, one way is to look at this like a Turing test. Remember that a Turing test is looking at a computer program and trying to determine if the output that's produced by the computer is indistinguishable from behavior that the human would be capable of. And usually you may have seen the movie The Imitation Game. So it might have to do with taking some sort of input, simple input, producing an output like this person who just spoke to me is female or this person who just spoke to me is male, and see if the human user and the computer produce relatively similar outputs over the course of several trials. And so we can look at the output from the simulation model and look at the output from the real world and say, well, you know, are they indistinguishable? Do they look about the same? That's the Turing test in simulation validation. And then we have something that you might call predicting the past. We have a model of a real system, but we also have the data from this real system, the input data and the output data. We can use the input data as real input data from the real world into the computer simulation model. So in other words, in this case, we won't be sampling from a distribution that we have fit to the data. We'll be using the actual input data itself. And so the two input streams are the same. And now we look at the output. And how do you decide if two groups of data are the same or different? Well, we've all learned about the T test and it might be indeed, it might be a paired T test because for the same input, we have two different outputs, one from one system and one from the model. And similarly, if we don't collect enough of an N of a sample size, we can use a nonparametric two group test like the Kolmogorov Smirnov in order to compare output probability distributions. Finally, we know that when we create a simulation model, there may not be a real model to look at to either do a Turing test or a T test and look at the input output transformations because the real system didn't exist. I might be creating my simulation model because we need a lot of data before we build a very, very expensive system. It could be we already have the real world system but we're planning to make very costly changes to it and so we want to run the simulation first. So there are a few things we can do. Obviously, we can't compare our simulation model to the real system in order to validate it. But we can possibly simplify the simulation a little bit so that it's close to an analytic version of our model, let's say an analytic approximation and compare that with the real system and then when that's with the simulation model and the analytic approximation. And if we can get that validated, then we can bring up the simulation model again to where we want it. And in the case where we're looking at an alternative system, we could model the current system and do the comparisons to the current system and then make the changes. So we won't know that the simulation model really is valid but we will know it's valid to the system as it is now and that may be good enough once we make our changes to look back and say, well, here's how the system seems to change from what we have today to what we will have based on the simulated model. Predicting the past is all well and good, but perhaps we can validate our simulation model in the future with all of the other ways of validating our simulation model that we've looked at at some point you just have to accept that you've done the best you can and you build the simulation model. But that doesn't necessarily mean that your validation is over. Especially in a very, very large scale project, you may be going back to make adjustments. So how about build the system after you have what you think is a valid simulation model and then go ahead and do the runs and compare the real system to the simulated model. If you find that the model does not reflect reality, you for sure will want to make some adjustments and see what that might mean for the real-world system. Of course, one of the things you want to decide is this real system that I've just created, am I happy with it? If I am, maybe it's best to leave it alone. Still, knowledge is power and knowledge is never wasted.