 Welcome to this lecture on documentation in a simulation study. We know from other courses that we want to document a program, a bit of statistical analysis, a major study that encompasses a large areas of the organization. Why? The reason we document a simulation study is the same reasons that we want to document all of those others. In addition to the reasons here, just think about this. Some of the work that you have been doing is so complicated that if you come back to it a year from now, you yourself may not remember what you were doing or why you did it or how you did it. And we're not even talking about something that's really, really large necessarily. So you want to document the simulation so that other people in the field can understand what you did. And also, perhaps you may have a need to evaluate it. There may be a need for others to evaluate it. There may be a need for others to extend the work. People who work on it after you have finished, there may be a desire for replication of the study, whether it's replication in its exact form or with minor changes. But we always come back to, even for the simplest project, you yourself may not completely remember what you did when you come back to it later. And so even at that simple level, documentation is one of the most important things you could be doing and you should be doing throughout the course of the project. You don't wait until it's finished and then say, well, let me document. For one thing, you'll have already forgotten what you did at the beginning. And for another thing, we all know because we're all human that chances are very good that we'll just let it go and procrastinate and forget to take care of that little thing called documentation. Here's an outline that you can use to consider documenting your simulation study. You all want project information, the information about how the model was developed, the description of the model itself, a conceptual model and how it was implemented, the actual simulation, the results that you got, which it would include the statistical analysis. And of course, any and all additions and appendices that you want to include together with the documentation so that you or someone else would have the full information when you look at this later on. Included in project information are some of the obvious elements like what's the title of the project, who did it, who's the organization and the individuals that are responsible for doing the study. Contact information for more than one individual because who knows how long this will sit around before someone takes an interest in it. The objective of the study, as you know, from all of these lectures, that's the one of the first things you want to come up with the title and the objective. When this was done from what date to what date, possible funding sources, anything that would be considered overall project information would go into this category. Model development is a large process that includes a lot of different things. What do the list you see in front of you is just one example of things that you might include when you're documenting the model development process. Information about model development, the name, who the modelers were, perhaps contact information, perhaps some literature that you used if you based your model development on an article that was published in the literature. Obviously the objective was this model used for something else and it was redeveloped for the project at hand. Is this something that you would consider maybe a transportation model? Would you consider it a population growth model? What kind of data do you require for this model? Could you find it? How was the model developed? What did you base your model development on? Was it only theory? Did you look at empirical data? Did you watch the way the model behaved? Some assumptions that went into the model, things you might do differently if you could in the future and that may also be related to the next item, the cost of development. Who is entitled to access the model and the compatibility, the computer language, computer system, computer network. Does it have to be completely implemented again in order to be used elsewhere? In addition, what is not here, but really should be here, is how the model is validated. Don't forget about that. This is more or less a continuation of the previous slide or maybe an enhancement of the previous slide where the model is more fully described as opposed to discussions about how it came to be, let's say, or who uses it. So you would have the actual model itself in whatever format, maybe in a flow chart, maybe in natural language. You want to be able to see the conceptual design, you want to see the program flow, what notation was used, and I see here we have validation. I'm not redoing the previous slide. I could see that validation could be documented either on the macro level or the micro level. And the reference information in this case means the actual source code. If this is a program, let's say that was written in C++, you might actually have the C++ code. Python R, you'd have the actual code here as part of your documentation. So the code could be found elsewhere, but you want to make sure that in the documentation package that's available to people interested in the simulation that the code is there and available to be worked on. I want to distinguish the features including, but not the least important of which is limitations. If I could do this again, what would I do differently? If I had more money, what would I do differently? What assumptions did I make that I might want to actually go out and collect more data about so that I could have a more realistic model? If there's nothing in that category, then that's what you say. This was the most complete model that I could possibly have, although we know that doesn't usually happen. Recall that a simulation is a statistical experiment. The word simulation implies a statistical experiment. So you want to have to document this. You want to have everything you would need to have in any other statistical experiment. Of course, you'd have the title, the objective, the assumptions, and even most important, don't forget the statistical aspects, the experimental design, what you are planning to do with the data, the requirements for the data. What data was used for input? How did you design that? How did you develop the input distribution about the simulation itself? The run length. How long did it run? How did you pick up the replications? What was the cost, cost per run cost for the whole study, the whole simulation study? The actual data results could be the actual raw data and or summarized data, data summarized in some way. Maybe there were, you had graphs output to show the, over the course of the length of the run to show the value of the measure of effectiveness that you were looking at. And of course, something about the assumptions that were used and then finally the data analysis. This looks very much like a documentation for any statistical study because indeed it is. We do want a complete analysis, not just the raw data, not just the summary statistics or pretty graphs and charts. We want the discussion of the results. We want the actual narrative of where you're looking at the results and explaining, understanding, reviewing and coming to some conclusions based on those results. What are some of the things that we might include in an appendix to this documentation of the simulation study? Well, you might have various project reports. You might have various pieces of documentation that were used through the ongoing study along the way. You might have quarterly reports. You might have a report at the end of each piece of the study. You might have published articles that were used as references that you based some of your conclusions or maybe that you based your model on. And some other publications that might be, might have been used like in house company publications that you anything. Basically, it's like it's just like a scholarly article where anything that you refer to that's from an external source should be either in a bibliography in a listing at the end, or if possible should be included. If it was something created for the purpose of the study, it can be included wholesale in the appendix. Thank you for attending this very brief lecture on documenting the simulation study. It contains one view, one way proposed of conducting this documentation of what to include in the study. But of course, the ultimately documentation is up to you in whatever capacity you have as either the simulationist, the manager, the supervisor, the liaison with the simulation consultant. And I hope this gives you some idea of not only what needs to be included in the simulation documentation, but of the scope of the simulation study itself. Thank you again for attending this lecture.