 This is a continuation of the introduction to modeling and simulation lecture, and it continues with the simulation introduction to simulation portion. What kind of systems do we simulate? Well, just about anything. Here you see some systems we've already looked at in the queuing lecture, depending on the order in which you're doing the lectures. These are examples of queuing systems. We could add to this inventory systems, channels of distribution, network analysis. Here you have examples of small and large, barbershop, gas station, a loading dock, bicycles, the city bike in New York City, the city bike system, clerical pools, and even, as you could see, some social service systems like foster care and family court. Let's look at simulation compared to other kinds of experiments we might run and other techniques we might use. Here you have presented some very good reasons for using simulation, as opposed to, on the left side, experimenting with the real system that you're studying, as opposed to, on the right side, using analytical or mathematical formulas and solving those instead of simulating. As a matter of fact, there are more reasons. I'm looking at the left side now, and there are many more reasons you can come up with for using simulation over experimenting with the real system. Not listed there is the fact that, in simulation, you control complexity, and in the real world, you don't. How do you control complexity in simulation with random numbers, with randomness and sampling from probability distributions, that instead of just looking at the system in all its randomness and complexity, without understanding where it came from. Now granted, what you want to study may not be tractable even for a simulation, in which case all that goes out the window. But for the most part, almost every system can be reduced to a simulation. But let's look at what we do have there. Suppose you need a longitudinal study. You want to study a system over time, over a month, over a quarter, over a year, over a decade. Well, you certainly don't want to wait that long to collect your data from the real system. So simulation can compress time. And that's one of the things, as you know, as you remember from our lecture on models, that's one of the things you want to do with models. Sometimes we can't even experiment with the real system if we wanted to. It's impossible. If I'm planning a hospital emergency room for a new hospital building that doesn't exist yet, well, clearly I can't experiment with the system. Even if I'm renovating, even if I'm redoing a hospital emergency room, I can't play with the real system. That's like playing with people's lives. We don't want certain systems you really don't want to experiment with unless you know what you're doing, unless you have first tried it in a pretend format, which is what simulation really is. So it could disrupt too much. It might be non-existent. And even more, a third reason for not being able to experiment with the real system is this experimentation might itself destroy the system or destroy pieces of the system and be very costly. That's the case often in military simulations. Now, on the right side, we're looking at simulation compared to an analytic solution, an analytic formula, which is what you already have looked at in previous lectures, especially with regard to queuing formulas, queuing theory. So why simulate? Well, maybe we're simulating a system that isn't a simple queuing model. And maybe the analytic solution just doesn't exist. Maybe we're trying to figure out what an analytic solution might be, what form it might take, what variables are important. That's one very good reason to do a numerical analysis first. Sometimes there is a mathematical solution, but the mathematics involved are just so complicated and intricate that it doesn't make any difference whether you do that or simulation. And then the simulation itself provides some benefits. And of course, sometimes you want to simulate because you need to train. You want to show output from a system, trying different strategies. You want to show what happens if there are a lot of things you could do in simulation that a mathematical model doesn't really lend itself well to. When you're looking at simulation as opposed to a mathematical model or an analytic solution for the system under study, it's nice to know that we don't have to choose. Often we need all the techniques that we have at our disposal in order to understand the system. If you look at some previous research in the area, whatever area you're interested in, you may find that simulation is sometimes used in order to verify an analytical model. How do we know that the analytical model under study, like for instance the queuing models, how can we be sure that they're correct? Well, you can simulate a queuing system, output data, and test the model by checking the inputs versus the outputs of your simulation data, or put in the inputs and see what your output is. It should match the predictive model. So the second bullet item is sometimes simulation is used to test the accuracy of analytic models. We all know analytic models are simplified versions of reality. Simulation can be a lot more complex. We can throw in lots of levels of complexity. So if you first develop a simulation that matches your analytic model and validate it, at least you know it's similar to the world so far. And then you can add several layers of complexity to it and hope the connection with the real world is still maintained. A third bullet item there is, and we're going to see more of this, sometimes we simulate first, generate data as if we're generating data from the real system, and then take our data in order to come up with an analytic model, an analytic solution. And obviously this is not going to happen too often in the kinds of systems we simulate, but just imagine if you find that the data is related in a linear fashion and well, all you have to do is do regression analysis and come up with your regression formulas, and you've got an analytic solution to a system that you have just simulated. This is exactly what we're doing here. A meta model is a mathematical model that's developed by using the data that was generated from the simulation. So you've got the input data, the X's, you've got the output data, the Y's, and you generate a model for this data the same way you would with data that you captured from the real world. It's just two steps removed from the real world because remember the simulation is a model constructed to simulate the real world. It's a model of a real system. And then the meta model is a mathematical model that's developed using data from that simulation. So as you can imagine, validation is very important when you're using a meta model. What do we do with a meta model? Well, anything that you can do with a mathematical formula, that's the beauty of it. It's easier with a meta model than with simulation to do sensitivity analysis answer what if questions. It's not impossible with the simulation. It's easier if you have a mathematical model. It's much easier to answer inverse questions. In other words, if I want a particular value of the variable I'm studying, the Y, what must my X be? You have to do it iteratively in a simulation, but you can solve that question mathematically with a formula, if you have a mathematical formula. And finally, of course, if we're looking at a formula, we're looking at relationships that might not be immediately obvious when we just look at the data generated by the simulation. And that might be the biggest benefit of all to combining the techniques. This next section is going to be about the simulation study. The simulation, remember, is part of a larger study and is only one piece of it. And here, we're going to look at that. A simulation study has simulation at its core, but that's not all it has. The simulation itself is part of the larger study that has all the characteristics of any project that you've seen and worked on already. And I'm going to remind you here that simulation means experimentation. Simulation is the single run, a single run of a simulation. You may have more than that. You may have complicated input variables. And so you may be running the simulation several times for different levels of the variables. But simulation means playing with the system. Before we do that, we have all that model building. After we do that, we have the data analysis. That's what you're looking at here. You're looking at the place of simulation in the total context of the simulation study, the progress of the simulation study. It starts out like a lot of them, like a lot of other studies you've seen with problem definition. You know what? Even before that, you have to recognize that there is a problem. And sometimes there's a lot of data analysis that goes along with that. But you have to start somewhere. So we define the problem. We define the scope of the problem. We define the objectives of the simulation study. Why are we simulating? What do we expect to get out of it? What are the limits of the simulated system beyond that are exogenous variables and issues that we may not be looking at in this study? Then we've got the typical systems analysis and design. We have a system we're investigating. We're investigating it with connection to looking forward to simulating it, of course. And in this case, what do we mean by data collection? Well, we may have to collect data so we can figure out what random inputs to use into the simulation. What probability distribution do we use for the various stochastic elements of the simulation? Then you have the pieces of the simulation study that have to do with modeling. First, you want to have the model design. We've been calling that the conceptual model. And often it takes the form of a flow chart or flow through a system with pretty pictures like you've been using and you will use more when you work with clouds or cloudists. Basically, even this conceptual model, you want to test for validity with whatever way you can. The best way is probably to just show it to someone who works in the area who has expertise in the discipline in the area of the simulation for the model that you're studying. And say, does this conceptual model look like what the system should look like? It's a very, very introductory look at the model. It's going to be more complex in the next stage. And you'll also want to go back to those experts in the next stage. When you build your model in simulation, since simulation is a digital technique, model building means programming. But even before you write the simulation program or the simulation process, you have a few things to do. You have to know what you're doing it in. Are we using Microsoft Excel? If you're doing your homework now, probably not. Are we using arena clouds? Are we using a general programming language like C or C++ or Java or Python? Are we using a large simulation programming language like SimScript? We're going to study more about different languages and what they're good for and where they came from in another lecture. Before you write your simulation model, you really need to get your input data prepared. You looked at a little bit of that in the systems analysis and design. Now, if you find it, see any errors or something that doesn't make sense in the context of the simulation program, you might have to go back to the systems analysis and design phase and look at it again and prepare that again. Alter it, change it, enhance it, do it all over again. Then finally, you can put together the simulation model. But of course, now you have to test it. If you've done any programming at all, you always have to test the program. We usually call that debugging and there are various ways of doing that. You do the same thing, the same kinds of techniques with simulation. For example, try it with crazy data. Use extreme values. See how robust the program is. What does it output if you put in values that were unexpected? That's one technique. In simulation lingo, we use the word verification. With the verification, we're looking at the simulation program to see if it does what it says it's doing. It's very similar to what we might call internal validation. Then you've got validation, which is more like in statistical experiments, we might call that generalization. We also do call it validation where you're looking to see if your model is generalizable to the real world system. How close is this model that I just built to the real world system? You want to validate the model by comparing it to the real world system. We're going to have an entire lecture on simulation model validation. It will be very important, especially for those of you that are doing simulation models for your term projects. Once we've gotten to this point, we have done the simulation modeling portion of the simulation study. We validated it. We know it's a good model. It's a model we're going to work with. The rest of the study involves working with it. We're not doing model building anymore. What do we do with it? Well, what we would do with any statistical study where we're collecting data in the field, let's say. Here, we're using the simulation in order to collect data. You need to design the experiment to make sure that you're going to have the most relevant information as possible. Of course, there are going to be feasibility constraints always, less so than in a field experiment that the variable cost, the feasibility constraint cost, you don't really worry about nowadays with simulation because time and space is actually relatively cheap when it comes to computers. There are presumably other constraints and you're going to look at the size of the experiment and design the various factors. What you're going to do is look at the goal of the study in order to design your experiment. Once you've done that and you do your simulation runs, then you've got all the data that you asked for and you want to analyze it in order to extract as much information from the experiment as possible. The part of the project that's called simulation, the simulation runs, is bookended between experimental design and statistical analysis. The design and analysis of a study are part of it is done before, part of it is done after, and in the middle you have where in the real world system in a field study you might be collecting data there. Well, you're doing the same thing here. You're just collecting the data by using your simulation model. Some of the things that go into designing the experiment are different with a simulation study than in the real world because we want to know things like run length. How long am I going to let the simulation run? There are going to be several factors going into that decision. How many replications? Well, that's the same question as we have with a real world study. Certain variance reduction techniques, certain controlling randomness, some things we can do with a simulation happily and we can't do them in the real world. In the real world if we want to reduce the variance, we're going to run up against a limit depending on the variance of the population perhaps. And we're going to have to do sampling techniques that might be complicated and maybe even difficult to implement. All of that is much easier in a simulation. And of course the statistical analysis on the other side of the simulation is very much like statistical analysis in a field study and what you might call your regular normal statistical analysis. But in addition here you may do something where you're analyzing your data and then creating a mathematical model from the data to enable you to just throw the simulation away and just work with the analytic formula which we discussed already and is called the meta model. It's a good technique, it's not the only technique, but from a statistical point of view it's very interesting. Finally we have the final pieces of a large project any large project and simulation projects are no different. Once you've done all the work and you've collected the data and you've analyzed the data and you've put it into a form to make recommendations to the decision maker, well then comes the decision making. We started with a problem, now what are we going to do? We might come up with several solutions or several possible solutions. What are we going to do about it? We're going to use the results of the simulation study to make these decisions and then finally the implementation which is presumably not going to be in the hands of the simulation researcher, but it's going to be very satisfying to see the results of the simulation actually put into practice. You'll notice that in this lecture and throughout the course I emphasize that you can't really do a simulation without knowing why you're doing it. What are the objectives? What are you trying to accomplish? What are you trying to learn? For the most part those objectives are going to be the same as the objectives of any statistical study and you can see many of them listed here that you've studied in your other courses, except maybe for the first one, a better understanding of the system itself. That's something that simulation has an advantage over almost any other technique. Parameter estimation, we may want to use the simulation, we're going to get simulation generated data, that's empirical data, only it was generated by a simulation. We may want to use it to estimate a parameter that we don't know and we're just like any other statistical experiments, we're using our sampled data, our simulation generated data, to make inferences about a population parameter. We may want to look at alternative systems or alternative strategies and generate data from each of them and then compare. We may want to do that and then say okay, which is best? I need to optimize, I need to have a metric that I can depend on to help me select the best system. Similarly, if what I want to do is rank the systems, I need some sort of ranking, I need some sort of basis, a metric to base the ranking on. Why might I want to rank instead of just selecting the best system? Well, sometimes we're willing to pick second best because there's always going to be a downside, maybe the best system is too costly, not cost effective. We may want to predict a parameter based on a relationship in the data, of course, optimization, we already talked about that. Factor screening, when you may have had this in a multivariate course, you've got large numbers of variables, especially on the right side of your equation that goes into the variable or variables that you're studying and you want to see which of them is more important to the output and you may even want to generate coefficients for them. And of course, you may want to come up with a functional relationship like a meta model, which actually goes back to the first bullet point here, it'll help with a better understanding of the system itself. Of course, it's not always easy to determine what makes a good simulation and there might be divisiveness among researchers to what makes a good simulation. However, there are some things that we can use as guidelines and that's what you see here. It should be something that's simple to understand, not only by the researcher, but by the user. You want to be able to explain what you did. No one wants to hire a modeler to do some magic and then only look at the output. Because the decision maker is responsible for the decision and the decision maker should really understand what went into it. A simulation, as stated many times, it has to have a specified goal. It has to have an objective. A simulation should be robust. It doesn't give observed answers. You don't want to have to explain, oh, don't worry about that answer. It just came out wrong. And that's one thing we do with a verification, validation, and sensitivity analysis. It should be easy to control and manipulate if that's what you wish. So if I run the simulation and then I say, oh wait, I want to include another variable, it shouldn't be that difficult to do. And a lot of that is really related to good programming techniques in general, no matter what language you're using or what software tool you're using. Adaptive, easy to modify, easy to update, all related to that. And then also a good simulation should really be evolutionary. You start out with a very simple model, validate it, and then add complexity to it. Look at it again, do whatever levels of validation you can do, or at least just look and see if it makes sense. And then add some more complexity. And so the fact that it's evolutionary, it's not just a good in itself. It helps you to explain the model. It helps to understand the model. And it also helps you to validate the model. Thank you for joining me in this lecture and for sticking all the way to the end. To summarize, we have looked here in this very brief lecture, we have looked at when to use simulation, simulation versus the real system, simulation versus analytical solutions, where the simulation itself fits in in the context of the greater simulation study, which is a statistical experiment. What are the steps in the progress of the simulation study? What kind of objectives might we see in a typical simulation study? And finally, what do we consider a good simulation? What do we look for in order to decide if this simulation can be called a good simulation or not? Again, thank you very much for your time.