 Let us look at more on model testing today, where we will be looking at topics on model validation, sensitivity analysis and policy analysis. I will give a brief on all these topics and like yesterday we will try to look at a couple of examples to reinforce our ideas on what is sensitivity analysis and what is policy analysis. Some of these things we have already been doing in the past, we are just giving it a formal name and structure right now. So, model validation talks about whether we build the right model. In verification, we do build the model right, that was the question, that means given the specification we want to check whether we captured all those requirements within our model. In model validation, we want to ensure that we model the correct problem. This largely corresponds testing of the model can replicate the past behavior as well as historical pattern. Three broad categories of tests are typically done when we talk about model validation. Some of these things may have overlap with model verification, so that is why when we actually talk about we say model verification and validation as a combined activity with some subtle differences are there. So, model validation, now the first test that we can try is something called a direct structure test. In this we are just going to observe the model structure that we have built without simulating the behavior and see whether the structure makes sense. Second one is to look at structure oriented behavior test with our understanding of structure if we can make some basic simulations does the behavior mimics the structure that we expect. Suppose we have a positive feedback loop only then we can expect exponential growth is that really happening or if you have both positive negative feedback loops in the system then what kind of behaviors can occur and testing that out. Structure oriented behavior tests, then behavior reproduction test that is to statistically compare the model output with past behavior and see whether we can actually represent the past as accurately as possible based on which we can then be more confident that it can affect the that it can be used to predict or analyze what can happen in the future. Let us take a little more detail look at this each of these broad test categories. The direct structure test is used to check if the relation assumptions are based on accepted theories and that all important variables are included in the model. Like for example, if you are setting up initial values as initial values follow with say for example, what is actually been represented as initially available materials and stocks. If you are setting up some desired variables and in reality people are following C. Little's law to use compute those variables there is a model also include that. So these can be tested without even simulating the model. For the direct boundary adequacy test, test of boundaries are adequate that is all the important variables are endogenous. So, if your system has no controlling parameters nothing for you to control, then you are attributed all the problems to something external on which you have no control of that means you cannot improve the system unless externally somehow that system changes for the better to affect. So, that cannot be the reason right even in case of demand in a common inventory system or supply chain demand is considered external, but still people try to influence demand by offering reduction in their sale prices by timely locating the material at the right locations, opening up multiple sales channels etc. So, they do try to influence the demand somehow. So, if that is essential part of doing the business then it has to be part of the entire model itself that is what the first one says. So, as soon as you look at the variables we can check whether should they be exogenous endogenous. If so, then how do we include it? Direct structure assessment test, test if the structure conforms with the real system and laws of nature if any like for example, when death sucker population has to reduce, birth sucker population has to increase. So, those kind of laws of nature has to be abided by if births fall to 0 then population should not increase. It is good to check all those relations within the model that is what we may direct structure assessment test. So, it is theoretical or empirical structure parameter confirmation test, a test if the structures and parameters have some real world counterparts that we had build a now classical model on stock management structure if you recall we had two stocks in that one is a working process a supply line then inventory and then we took a decision saying that inventory is going to be adjusted based on desired inventory and actual inventory and working process will be adjusted by desired inventory or desired supply line and actual supply line then we also had a forecasting component. So, it is good to check whether the company is actually doing all that are the company having a forecasting module in it or not are they actually considering the inventory information supply line information to actually make the decision. So, this is what we mean by theoretical parameter confirmations like parameter we had used in that model a parameter called as inventory coverage. By that we meant how much invent my weeks of inventory we need to have. So, the values you are using here is it realistic to what people are using maybe the for simulation purpose we might I use some value, but for when you go to analysis and interpreting the results is a reasonable amount of inventory to hold. So, that is what we mean here. Direct extreme condition test test without simulation of structure and equation makes sense under assumed extreme conditions or what the limits are for the model to be possible or useful. If demand is too high or demand falls to 0 a birth rate falls to 0 to those kind of things what can happen or when backlog becomes too high then what kind of things could this model result in can be actually traced back. Face validation test whether domain experts find the model structure and equation appropriate for the intended purposes. So, now we are now more in the model verification when you look at verification it is in our hands we look at the model we play with the computers and figure out once it comes to validation then we are looking at whether the model is going to be used by someone else out there who is going to use this model to make the key decisions are we able to convince them. So, if you are a consultant we need to ensure that the client actually agrees to whatever the model. They can say no no this does not work this does not represent it so that has to be minimized that will only come through dialogue. So, that is that part of test is called as face validation when you look sit across table and try to convince them that this model structure actually captures what we intend to. There are some broad tests for direct structure tests. One class is called as structure oriented behavior test. Testive modes of behavior or frequencies of mechanisms causing the behavior correspond to what would one expect right if you have infectious diseases kind of models then we do export to S shape growth there for new product introductions we ended up using S shape growth because there is a positive feedback, but a later time a negative feedback starts to dominate which results in a S shape pattern. So, the model should actually result in those kind of expected patterns of behavior like if there is large delays in the system then we can expect some oscillation as it indeed occur. If there is no delays and still some oscillations are occurring then we have to figure out what is happening there is it because of some non-linearity introduced which inadvertently is causing some unexplained behaviors. One is extreme condition test similar to what we observed the structure previous step we can actually plug in run it and see whether the model is actually robust in those extreme conditions or does it throw errors, qualitative features analysis, test in a specific condition model generates a particular behavior see model we try to make it as generic as possible but in reality we will find that only one scenario has occurred right. So, under that scenario we know what kind of behavior can occur. So, if you are able to plug that scenario into the model can we get a similar behavior under those conditions. Third is the behavior anomaly test, test if changing and deleting any assumptions lead to anomalous behaviors is that assumption so critical that it is just made a positive feedback system negative feedback or things like that and does it make sense. Surprise behavior test test if model is generating surprising behaviors this is little more difficult to generate unless you generate large amount of scenarios very difficult to figure out whether any surprises occur. But one is to see whether once you plug in numbers once you look at the results we need to spend time analyzing it that is what today's lecture is about building the model is easy but that alone is not the end to it we need to spend some time outside the model to see whether the results actually make sense there is a small blip in the model you know then we have to zoom in and understand why that blip is even occurring overall the behavior will be ok we are getting positive feedback but maybe it goes up and then there is a small blip and then again it increases then we may want to zoom in to figure out why or what caused that is it because of model structure integration issues etc. Family member test can model generate the behavior of other instances same class of systems for example we are looking at say infectious diseases model there are so many infectious diseases out there can the model be used for different kinds of infectious diseases can the model be used for in fact is a different locations by changing the parameters it still makes sense. So, more test we are able to do and show that it covers a wider range then more generate the model becomes and more confident that it can be about the model that it can replicate so many scenarios and more if you have more confidence then we can be more sure that what the model is saying we can actually use it for further analysis and things like that. Behavior reproduction test basically this one has the straight forward ones where if you guys are more quantitatively inclined this is what may appeal to you to test statistically whether the model generates a behavior of interest we had used it in reverse when we try to do the modeling for I think it was part of our new product introduction when you try to fit the curve with existing data to see whether we are getting the to compute our values of I think coefficient of imitation and innovation p and q in ket of bus model and then we used it to calculate p and q but other way suppose we get a some value of p and q we can use it to figure out whether it is able to correctly capture the behavior. So, this is pretty much done to see that you know once the model is built we want to have all the feedbacks so that the same can be used to explain what happened in the last 50 years or last 30 years. So, if we are able to reproduce that behavior as accurately as possible then we can have confident that if I simulate the further 30 years and I am going to use a similar structure then what our model predicts could be reasonable and if I am going to introduce some changes in the model then I can expect that all the changes are caused because of the intervention we are doing nothing else. So, this is another popular mode of behavior reproduction test is used. One of the dangers I need to say here about this behavior reproduction is quite a few it becomes very tempting as well as difficult in the sense we ended up overfitting that with the past data. Many times happens that you get the model and tune it so much that it very very accurately predicts the past data but in the process the meaning of all those parameters just went for a toss like maybe the best fit gave a value of p and q which further or let us forget p and q let us involve say infectious diseases model we are looking at contact rate as well as the infectivity parameters can be computed using the past data because of us trying to fit the data so hard to that contact rates and infectivity parameters and we got some values we may later find that those values cannot exist in nature like the biologist doesn't agree that this is the true infectivity that can actually occur there is no other evidence to it problem is because you overfitted to the given data maybe there are other feedbacks that is occurring which we forgot to capture. So, those are small dangers in trying to fit the model so tightly with the existing behavior. So, that also has to be kept in some practical things about checking this models is unfortunately or fortunately only the users or the modeler must critically assess the model boundary you have to decide what the model boundary has to be how long time horizon needs to be what is the simulation time step or is level of aggregation we need to use you want to build a factory model or each assembly line in the factory has to be modeled or each machine level we have to model you only need to pick up the level of aggregation various type of data is used in the model you must have seen by now we have numerical data which is what we are using but there is written and mental data that we use and try to capture this comes in the form of relations or multiplications that we do as a fudge factors and do some values between 0 to 1 these catch the table functions and things like that and these are all useful because most of or rather everything that we have in the simulation model we would like to have it a real world counterpart so that we can go ahead and measure it whether it is actually indeed happening or not and use it for further improving our model. So, that means whenever you use something then we need to have a name for that quantity we need to select select the scale of measure that is units in the state the reason for actually even having those values and that gets closely tied up with model documentation it is not a print out of equation and graphs we need to list the assumptions that is happening like how often decision is being made we are indeed following exponential smoothing based or can a forecasting method. In class we covered exponential smoothing but the actual world company may use some other forecasting method just because we know only exponential smoothing does not mean we only use exponential smoothing we then try to use a forecasting model which mimics what they are having. So, these things come out when we actually write the assumption we assume that the forecasting exponential smoothing method with so much parameters and stuff like that. These are all very easy to say quite difficult to practice sometimes it becomes innate meaning you keep practicing and as soon as you look at a model you can see there is some structural anomalies but to get to there you have to practice so that is why we are playing with simple structures look at the structure we just start with first principles is it a first order system is there positive feedback is a negative feedback flows in and flows out then only certain behaviors can be done model is showing any other behavior then we can be sure that or we can at least have doubted saying that no this doesn't look field right and then once you get that feeling you have to act on it or either you have to get more confidence in the model that what it showing is right or you have to update the model to confirm with your beliefs. This is broadly what we can talk about as model validation once you have verified and valid model one of the most common and popular thing that we end up doing is called a sensitivity analysis. Sensitivity analysis computation of the effect of changes the input value or assumptions on the outputs typically we start with very small changes or typically change only one parameter value or one link or one assumption and see what is the impact of it within the entire dynamic. So, as to isolate the reasons the reasons or the effect of a single parameter changes the change in assumptions whatever it assumption can be any parameter value or anything if it results in only change the numerical value of the result that is called as numerical sensitivity. So, the pattern of behavior generated itself changes then it is called as behavior mode sensitivity yesterday in the masquerade example when we changed the what is it proportionality variable we got exponential growth or we got exponential decay or we got a stable parameter. So, that is a behavior mode sensitivity and that behavior mode is sensitive to that parameter value. So, the entire behavior changes, but for certain parameters maybe it is exponential growth, but only the numerical value changed after a few years maybe. So, then it is called numerical sensitivity if it reverses the impact of proposed policy it is called as policy sensitivity we look at policy analysis a minute. So, this is broadly what sensitivity analysis is going to do and we will learn how to change one parameter at a time and see how we can document and compare the results to see whether what is more sensitive or less sensitive to changes that we can see. This slide actually summarizes what is actual use of us doing all this modeling and modern building exercises that we are doing throughout the semester and beyond is as follows these are broad uses why we are even doing all the model is to make explicit assumptions and mental models that we may have to communicate the mental or formal models to analyze and understand the link between structures and behaviors to test theories to generate possible futures and explore uncertainties risks and opportunities to design policies that improve system behavior to test robustness of the policies and their effectiveness and a deep uncertainty to experiment in virtual lab to train, teach, learn, experience etc are all the broad purposes of model building. The first few ones may seem quite abstract this is when you self learn, but then you can for example go to the last one right. So, using this models as a virtual lab you yourself can train games to understand what kind of dynamics is happening within the reality or a supply chain or a small factory or how infection disease spread right. So, that will give an idea because we cannot gain real world experience in everything, but it is important that you look at the model play with it and observe the reality in a better light. Like in campus I am not sure what happened last few years, but a decade ago conjunctivitis outbreak is common come monsoon season around that time there will be few case of conjunctivitis and suddenly it is going to spike up and then it is going to come down. In fact, so much that the conjunctivitis even the drug the medicine there will be small shortage because so many people get infected which you can now understand much better because even a few people get it given the contact rate with the amount the population density inside campus and hostels more people are eventually bound to get it before they actually seek treatment and then take precautions and without that it is not going to fall right. So, these are some simple models you can use to relate and observe what is happening in reality better. So, that is what the last point is trying to say. So, the two points highlighted in bold is after building all these models what we actually want to do is come up with better policies to improve our actual system that is in place. Test the robustness of it see how effective it is when things are becoming so uncertain. For example, considering infectious diseases if suppose our hospital wants to know how much to stock and because all these unfortunate medicines come with expiry date you cannot stock infinite amount one you do not have space it is very expensive. So, if you want to come up with some policies or to control those infected diseases what will be the most effective policies it is better to do kind of simulations and test these policies. So, before we actually test the policy we need to design the policies then we test it or simulate it and then we analyze the policies. So, policy design the first stage typically in policy design we need to incorporate suppose we know the policy like for the infectious diseases the policies if anybody gets infected there will be quarantine or kept in a separate room so that they can avoid contact with people. Yes, so is it a good measure or not because quarantine whatever we do is going to cost money. If it is a quarantine they need to ensure it is a whatever contact free room enough beds has to be provided enough facilities then people has to wear all mask this that so that the contact is all minimized then when the patients identified as put in quarantine. Would that be better or should people do more vaccinations and awareness drives. So, what are the different policies that can adopt could be very large which is the most effective one we can help uncover and when we incorporate any of these policy designs within our simulation model we end up changing the things in simulation model. Over years people have done various kind of policy analysis and in policy what you want to do you want to introduce some sort of structural change in the model right there is some decision making happening we want to introduce some structural change. So, decreasing order of effectiveness these are structural changes that typically helps in making high impact in your outcome. The first one is adding breaking or changing information based feedback loops, decision routines and boundaries of systems and responsibilities. So, once you have model we know what is the physical flow and we want to know what is the information flows. Here it says that the by changing it by modifying that information feedback part will have the maximum impact in your systems output. The second one which can have the maximum impact could be adding breaking or changing the physical stock flow structures themselves. We really need to have so many levels of stock or reserve way in which I can cut down one level within the system itself. Strengthening on weakening existing feedback loops under flow variables, adding, eliminating delays or smoothings, change in high leverage policy parameters that is parameters that come in control by those involved and they have large effects for relatively small changes. So, you have to identify them with sensitivity analysis. So, typically whatever policy analysis we do these are typical things will end up affecting in our model. Since we have built a model we will be able to figure out that okay these are feedback loops let us see where I can reduce this delay somewhere. Let us see where I can use this additional thing to make as a information and make a better decision within the model. Our control theory is also very useful, control engineering or control theory methods are also useful. For example, the systems are all second order what he is. So, we can actually do compute his eigenvalues and perform stability analysis. You can do what is it? You can do Z transforms for discrete even systems and calculate its stability boundaries, bounded input, bounded output stability analysis can be done, there is a Bebo stability. So, all those things will be useful even here because underlying system is differential equation or difference equation however you see it and whatever control systems ideas you have can figure out what are the boundary conditions for the parameters which will result in stable behavior in system, unstable behaviors, oscillatory patterns can also be computed using the control theory ideas. For example, if all the eigenvalue roots are positive then the system is unstable. So, like that you can compute it, if it is negative then the system is stable and so on. So, this is also form of policy design and testing where it is looking at extreme conditions. So, once you have these policies we need to test it with a simulation model. A good practice is to build the policy in your model such that it can be switched on and off. So, it is a single model policy on policy off we can have a variable. Please use different run names or different policy runs or dynamics generated by different problems can be actually studied and compared. Check robustness of the policy across many runs or parameter settings because it is not just one comparison we need to compare across various settings similar to sensitivity analysis for each policy. Adaptive close loop policies are more powerful than open loop ones. As soon as you start closing the loop system will be more powerful than if you have open loop. So, whenever there is a policy I should request that think of close loop policies. And remember SD modeling requires lot of reflection beyond the model and the behavior patterns because here it assumes that your real system is also quite structured and it is going to behave the way we expect it to behave. But reality may not work like that people may not even agree to these results and there will be strong resistance to changes. So, how to even incorporate the policy changes within reality will require more reflection beyond what SD can actually show. So, this is a broad summary or outline for how to do model from a studies class model debugging then verification validation and then moving on to sensitivity analysis or policy analysis. It necessarily goes in that order because first we want to ensure that we have a verified debug model without any errors and then it is valid enough based on accepted theories or based on real problem that we are making. And once you are sure of that then we can do sensitivity analysis and policy analysis to uncover or understand the system better.