 This is a very brief PowerPoint because most of today will be in NetLogo. I'll just be demonstrating things there, but I wanted to highlight a couple of concepts. So the concepts and planning of experiments, a little bit broadly about the behavior space, which is how you automatically run experiments in NetLogo, a little bit about output and then basically that's the end of the PowerPoint presentation. There will be time for Q&A at the end of the session as well. So the first thing I want to say is that working with models and experiments is iterative. So you write code in the code tab in NetLogo. You flip it over to the interface tab and you hit run and you see what happens. This is kind of running experiments. It's more like bug testing and things like that. But broadly, you're getting a sense of how the experiments will work when you switch back and forth between writing code and running the model to test it. But then when you get the model pretty much the way you want it, you proceed to a wide sweep and then onto maybe more writing code if your wide sweep shows you that things aren't the way you want back to a wide sweep or onto a narrow sweep and then probably back to a wide sweep, narrow sweep and eventually when you've gone through the loop several times onto experiments at which point probably you'll go back to writing code. And I'll clarify this a little bit more as we go through. So just to talk about today in the experiment section, I wanted to find a basic model with very few parameters to show you how experiments go with parameters. Although the tram commute model that we looked at last time and some of the other models that we've looked at, they're very interesting, but they have so many parameters that it can be hard to get your head around what experiments look like. So I thought just pick this one. It's in the models library. It's super easy. Only three parameters as the number of cars, acceleration rate, deceleration rate. And this model essentially it highlights one agent, which is in a red car. All the other cars are blue and it tracks the red cars speed, but also tracks, you know, the slowest speed, the max speed, all the, you know, the various speeds of the cars. And what we're trying to find in a model like this or in our experiments, other experiments might try to find other things, but what we want to find is the point, the break point in behavior. So the set of parameters or the combinations of parameters where the red car behaviors change. So if it's on its own, it can go max speed continuously. There's nothing in its way. There's no reason to slow down. But when there's other traffic, it has to accelerate and break. And so it's not consistently going the same speed. So we're trying to find the combination of parameters where that behavior changes. How much other traffic is too much other traffic? How much acceleration and deceleration are too much acceleration to deceleration, that kind of thing. So why just sweeps? Let me just briefly highlight here. You get a brief example of how to set up the sweeps. So what you do is you choose the variables. These are all the variables in your model that can vary. And you set them up depending on how you want them to vary. So you can either set them at specific points. In this case, 0.001, 0.003, et cetera, through 0.009. That's basically its full range. Let me just say, all parameters, you start wide with all the parameters varying. You want a full or wide range of parameters to vary. You want big increments. You want few repetitions. And the goal is to find errors that are not found in basic testing, broad areas of interesting behavior and conceptual problems. Generally, wide parameter sweeps on models that you've written yourself are likely to motivate some major code restrictions. I'm going to have to turn off this rest break in a second because it's come at exactly the wrong time for me. So hold on just a moment. And I'll skip that. OK. So these are all the parameters. We have them all varying across all of their ranges. You can either set them to vary at a fixed list of points or within a range according to a certain increment. And this gives you instructions about how to do that. So you can have by adding the extra brackets, you can say, beginning of the range, increment, end of the range. You can do this as much as you want. Also important to notice is the repetitions. This is how many times the model will run an experiment automatically at each combination of these parameter settings. And also important to note is what you are measuring. So this is the output of the model. And you can choose to measure the runs at every time step. So it puts this thing that you're measuring into the output at every time step. But if it's unchecked, you only get the final time step. So that's very different depending on what kind of model you're producing. So you might only want the end. You set up this model. You let it run. Where does it end? That's all you care about. Other models, you set it up. You care about how it gets there. And generally you set a time limit. If there's a natural way that the model ends, like maybe when all the agents die or all the agents get infected and recover from a virus or all the agents reach a certain level of health points or something like that, the model might end naturally according to your code. If not, it's very important you set a time limit. OK. So that's a wide sweep. Now a narrow sweep has maybe not all of the parameters. Maybe you might decide, right, I don't care about acceleration. I'm only interested in the number of cards in deceleration. It's narrower ranges and smaller increments. So instead of the entire range of number of cars, I'm only looking at about half the range. And I'm moving in smaller increments. And there's more repetitions. So instead of 10 repetitions, I now have 50. And the goal of this is not necessarily to find errors. You probably will have found all the errors in the code run sort of switching or in the broad sweep. But you are really looking to clearly find the break points where the behavior changes or to find the ranges at which different runs break the break point and behavior. There is a chance that you will need some code rewrites. Probably they will not be major rewrites. They will be detailed rewrites. Finally, if you've gone through broad sweeps, if you've gone through narrow sweeps, you've done all the code rewriting that you're going to do. Essentially, experiments are really very narrow sweeps, you might say. There are many, many repetitions on a single or very few settings. So this one, the increments is even narrower. The list of different settings is much smaller. And the repetitions is much bigger. And the goal here really is to find statistically useful volumes of data on areas that the sweeps suggested were important. So the sweeps probably tell you that the very end ranges of your experiment are always, you know, the red car always goes max speed, no problems. Or the red car never goes max speed, and he's always stuck in traffic. So really you're trying to find the area in the middle where things aren't always consistently max speed or never max speed. And that's why you set the experiments to narrow in on the bits that are interesting. Now let me just switch over to the model real quick. Here we go. I'm going to set it up. Well, I'll tell you briefly. In the tools menu at the top is behavior space. You can also hit control shift B to open behavior space. This is where you set your experiments to run automatically. So I have these three experiments that I set, wide sweep, narrow sweep and experiments. And you'll notice they have the number of individual model runs that this set of, you know, that the sweep or the experiments contains. So the wide sweep, I'll just open this so you can see, has this wide range, big increments, 10 repetitions. And that comes out to 1,250 runs. Now the narrow sweep has fewer settings, you know, smaller increments, but many more repetitions. So it comes out to 3,200 runs. The experiments, even narrower ranges, even smaller increments, but many, many more repetitions. So it comes out to 4,500 runs. Now this is not impossible to do with Excel, but I really think I really encourage you to consider using R or Python or Julia or some kind of, or even Stato or SPSS to analyze your results because they are potentially much, much larger than this and they will break Excel and you will go crazy trying to get it to work. Now, if I wanted to set up even more narrow experiments, I could hit duplicate and let's say I just want to look at when there are 20 cars and 22 cars. So I've changed from a range with the increments to a fixed list. And let's say I just want one acceleration and two decelerations. Now I'll keep the repetitions at 100, you can move it to 200 or 500 or whatever and I'll call this experiments focus. Why am I typing focus? Today is not my day, folks. And hit OK, keeping everything else the same. So now it's got 400 runs because I didn't increase the increments and I narrowed down the number of experiments I was running. Do we have any questions so far? I don't see any in the Q&A. So that's fine. I'll just pause for a minute here and let you all, if you have your NetLogo open, you can open this model. I loaded this model with these experiments, the wide sweep, narrow sweep and experiments, the first one. I loaded to the GitHub repo. So if you want to see exactly what I've created, you can pull these changes from the GitHub repo. Alternatively, you can just go to file models library. Well, I'll cancel this. But within models library, it's called traffic basic. And you can recreate these experiments within behavior space, which remember is at the bottom, near the bottom of the tools menu. So what happens? I'll move back to the PowerPoint, I think. That's not what I wanted. I wanted the full. There we go. So those, that's what I have in my behavior space. You'll notice run that actually sets the experiments to run. Let me, there we go. But you will need to choose an output. Now it will give you, when you hit run, it will give you the option of a spreadsheet output, a table output. You can update the view. This is the view. And you can update plots of monitors, which is in the actual interface. It will run just as if you run it directly from the interface. Simultaneous runs in parallel. This is good if you don't have a creaky old laptop and you can run lots of things at the same time. If you're concerned about computer power, just run one or two at a time. The difference between spreadsheet and table output is the orientation. So spreadsheet has the variable names going down in the first column and the output of the different runs going across in the y direction. Whereas table output has the variable names going across and the results going down. I find table output more natural to work with. But maybe you find spreadsheet and that's fine. And this is a little bit of when you hit OK, it'll ask you where you want to save it and what you want to save the output as. But when you pick a place to save it and a name to save it under, it starts running. This got to run 188 of 3200. And I think I ran all of the 3200 experiments in about 60 or 70 seconds because this model is very, very straightforward, very small and trivial. It runs fast. You can of course, where's my cursor gone? Here it is. If you have to, if things are going terribly, terribly wrong, you can just chuck the whole thing and start over. You can also, I mean, it's not very important for this model because this model is so small, you can change the speed. This kind of only matters if you're updating the view. It doesn't really matter for most things. That's pretty much the end of the PowerPoint. So let me get back to traffic. And do we have any questions? Let me just take a moment if anyone has questions. Okay. So I'll go ahead and hit run just to show you what this looks like. Gives you this little window. I've chosen table output because I think it makes more sense than spreadsheet. Hit OK. Gives me where do I want to store this? This is the folder that I have. I'll copy this up to the GitHub repo, but this is the folder where I'm storing my local copy of that repo. I'll hit save. And you see these 400 runs are done already. That's very quick. So what does it look like in that? Let me just see if I can open. Now, I don't want to open that. Let's just go with PowerPoint. I want Excel. I'll go with one of these other runs that I had already exported. It's taken a while to open. That's all right. Let me move it over. Excel. My nemesis, I feel. It's refusing to move the window. There we go. Okay. So let's make this big. All right. Let's decide to open a bunch. Okay. So you'll notice in the automatic export from behavior space, it gives you six lines of stuff about the model. So it was a metadata, basically, sort of like, you know, what was the version of NetLogo that ran this model? What's the model called? What's the experiment called? When was it run? What was the world? What are the dimensions of the world that were used in this model? Now, if you're analyzing this with R or SPSS or Python or something like that, you probably don't need these rows. You can delete them and re-save the file as something else. So I will delete these. I want to actually delete them, not just delete the contents and save as traffic, where is it? Traffic-wide tidy. That's what I'll call it. And it's going to tell me that it already exists. Why aren't you? Okay. Let's try that. I want to replace it. I basically replaced it with itself. That's how I created traffic-wide tidy. And let me just open our studio, because this will show you a little bit about how we can work with that. Now, you remember if you saw my previous workshop, the one about adding in real-world data, we talked about using the CSV extension. And we looked a little bit at the code for how to get the model code to export things, maybe not just at the end or on every turn, but when a certain thing has been accomplished within the model, you can ask agents to open a file, write something and then close the file. And that's very useful if you want more complex, more interesting sort of model-specific data, rather than just what is happening in basic parameters, like global parameters at this point in the model or at the end. So here's my art studio. It's taken its time to open, because that's basically how it goes. Let's see. Let's close PowerPoint, because I think I'm done with it. Close all windows. Wants to know if I want to keep my ink annotations. No, don't be silly. Okay. Let's close that. Why is this taking so long? Clearly I should have practiced a bit. So I've created a little bit of code, and this art code is in the GitHub repo if you want to use it. Basically, I install a few packages and make them available. I actually don't need the read Excel. That's not helpful. So you want to get your working directory and set it, if necessary, to the directory in which your code, your output is working. Read in this case, wide. Let's clear out the old variables. Yes. Why didn't you clear them? Well, it'll get around to it. I'm guessing it's just slow, because everything is slow right now. So you want to read in the variables. Is it not? Yeah. It seems to be working. Yeah. Slowly. Slowly. Feel free to ask questions at any point. You don't have to wait until the end. I am ready and able to clarify anything if I'm going to quickly. If you want another example of something that I've shown here, could not find function as tipple. You know why? Because I didn't install the packages. So install those first. Always good. This may take a little while. Sometimes package installation is a bit slow. Again, I could have done this before the workshop started. But then you wouldn't see the genuine silliness that is like working it with R. So we've got these libraries. That's good. Let's just get my working directory to make sure. I know where I am. Yeah, it's fine. Cannot open file. No such file or directory. Okay. Let's see what's going on with that. It's always a problem, isn't it? That's modeling. 22 training. Come on. Oh, it's not showing my... Let's try this again. Get working base modeling. 22. There we go. Traffic model. Okay. So traffic underscore wide underscore tidy should be there. Maybe it was just because I had it open in Excel Oops. Control enter runs the highlighted code. Nope. Cannot open the connection. I have no idea. Today is really not my day. Oh, right. I see. Because I want... I will change my working directory to source file location. Let's try that to see if the working directory is correct now. Yeah. Now it's in traffic model experiments. So the problem was despite having hit get working directory, I did not pay proper attention to what it said. Note to self, pay attention to self. It's time. Try it again. Run the code to read in the CSV as a table called wild. Wide. Sorry. We see over here it is quite a lot of observations of six variables. And it will view coming up here in a second. So the, you know, it's sort of automatically coming in from CSV. It adds this X dot thing. It puts dots in the middle of the names, but they are run number, acceleration, number of cars, deceleration, time step, and the speed of the sample car, the red car. So that is all the things I need, but you know what? Those names are a bit fussy. So I'm going to rename the variables to simply run acceleration, number cars, deceleration, step, and red speed. So now they are renamed. Great. Now briefly, I want to just do a little bit of summarizing. And this is going to find the average of the 10 runs in which all the other variables were held the same. So if you remember from our wide sweep, we had 10 runs and there's 10 repetitions at each combination of these settings. And what this code does is it finds the average of those 10 and stores that in a new table called wide summary. I think it's a table. So if I run this briefly, it finds there are now 10%, it's only 10% of the size, so it's dropped a digit essentially because it's fine for every combination of 10 runs where all the variables are similar. It's only found the average. And briefly, I'll just plot this. And this is a big stupid meaningless plot. I wouldn't recommend that you try and get a paper published with a plot like this. But you can see sort of, so this is when there is one car. And here is the acceleration. So when the car can accelerate very quickly and there is only one car, the red car very quickly gets up to max speed and then just stays there. But the lowest acceleration, the car never makes it to max speed. So we can eliminate one car as like an interesting combination of parameters. Now with 11 cars, again, low acceleration never gets up to max speed. In fact, if we look, even the low acceleration never helps anything. So this is why we drop low acceleration in our narrow sweeps and experiments. High acceleration, still pretty good. I mean, right up through sort of medium acceleration. That's not bad. But what's really interesting is when the traffic is big, when there's 41 cars in the model, the actual speed is hugely variable. So while acceleration was the only thing that matters when there's a small number of cars, even, you know, 11 broadly up to 21, like as soon as you get a lot of cars, there's other things are driving behavior differences. So some runs, the red car goes very fast. Some runs, the red car goes very slow. So we're kind of trying to zone in on that. And what we can do is replicate these same basic steps but with traffic narrow tidy. So let's call it narrow and then traffic narrow tidy. If we run this, we'll see another variable here. Eventually it's taken its time. It does not seem to have worked. Oh well. And then when it arrives, we will rename its variables. Right. It seems I heard not. Oh, right. The problem is I did not save a version without all of the headers and stuff. Now I could say do this in R. I could remove those rows, but that would make importing it more difficult. So this is another reason to do the CSV extension, the kind of detailed exports that I did in, yes, the previous workshop where I showed how to build reporting into the model operation is that the exports, the export file in a CSV extension like when the code properly, I've just renamed wide. Oh well. We'll go back, rename this. Sorry, trying to do too many things at once. When you properly export, say PaaS, when you try and export things properly through code within the model, you don't get these headers with the version of the model software with the dimensions of the world. You can if you want, of course, but you have to set that up. Whereas the export that comes automatically within the behavior space really wants the, it always puts that stuff on and I never want it. Like I find you can always just make that part of the file name or something like that if you want. I don't know. Or it would be more useful as a column of in the spreadsheet rather than like a heading that I don't know. I don't particularly like it, but what can you do? So let's change these things. And now we see here, we did finally get it imported. It is a huge number of observations because there are now 50 times 50 repetitions at each combination. So when we find the narrow summary, it's going to take the volume right down. And it does narrow summary is now the same number of observations we have here. So it's much, much smaller. Oh, that's because it's, let's try this again. That's still going to be small, but probably not the exact same size. Yeah, it is now even smaller. Okay, so that's very good. So now let's plot this and see what this smaller set of experiments looks like. Here we go. All right, we see much more interesting behavior here because we have a smaller number of, you know, car numbers to compare. And we see that the behavior is much more streamlined. It's especially streamlined when there are fewer cars, but 25 and 30 are very interesting, aren't they? Because we get good ranges of behavior. So there's still chaos. There's still individual circumstances within each of these experiments. But there are, I'll just post upon this, but there are interesting behaviors. So 25 is still, okay, you know, we'll see how it goes. Certainly the highest acceleration 0.008 and 0.07 reach max speed, but, you know, lower accelerations 0.006 and 5 don't. But when there's 30 cars, nobody reaches max speed really. Maybe some higher accelerations. And the ranges are very big. So let me save this code. Do we have any questions? I feel like I'm moving quite quickly. And I am surprised that no one wants clarification on any of the things that I've covered. Within NetLogo, within are about the logic of experimentation. I mean, maybe you all are just whiz, you know, really whizzy for how you approach, you know, thinking and experimenting. None of this is shocking to you and you just need an example. But I'm surprised. So please do feel free to ask questions. Okay. And again, we can repeat these same steps for experiment. But let's have a look at that experiment and think about is actually this where we want it to do. Now, our model suggests between 20 and 25. There's maybe not so much interesting. We think might be interesting is between 25 and 30. There's a big change here. But maybe we could go slightly beyond that. So let's go with instead of 18 to 30, let's go with maybe 26, 28, 30 and 32. And what about the acceleration? 0.05 is certainly, you know, it's not that interesting. 0.06 is just about interesting. So let's say, yeah, 6, 6, 5, 7. This is pretty good. 100 repetitions. That's it. That's because I forgot to take away these brackets. If you want a range, you need an extra set of square brackets. If you want a list of individual points to consider, you just put in the list. So it's now 3,600 runs. Let's run that table output. Okay. Yep. Let's just call it experiments. Save that. It will go pretty quickly, I think. Yep. It's moving through this very well. I don't think I'll have time to finish this micro break. Maybe I will. Let's see. Total elapsed time so far, 22 seconds. Let's see how far it gets. I think it'll finish. Yeah. Ooh, it's a bit of a race. Will the runs finish before my micro break? They did, yes. But I'll wait three seconds more. Thank you very much. I will go back to my folder. Let's see. 1602. That's today. No. Yeah. Is this the one I just ran? I don't know. We'll see. 13. This is not the same one. Let's try again. Excel my nemesis. Okay. So let's delete these. Delete. Save as traffic underscore experiments. We need to stop this so that our can open it. And let's experiment. Underscore one. All of these things in. Place. And we can rerun all of this code. To see what it looks like. Now again, no questions. Please do feel free to ask, you know, why am I doing this? Why have I chosen the parameters that I mean, what do I mean by break points and critical area? These are pretty important topics. And if you're not clear on them, please feel free to ask. You can of course ask anonymously. So nobody needs to know that you're the one who's confused. If that's what you're worried about. We name that. Find the summary. Now. You'll see the original that I imported had. 643,200 observations. But after summarizing it. After averaging all of the means. We get one, two, eight, six, four. Now you might think, why is that bridging the means important? Because that's, you know, that's. Removing interesting variation possibly. This is just a demonstration. We don't, you can of course. Focus in on these differently. So again, 15, 20, this is definitely not the setup that I had just exported. No idea where that went. So let's try it again. Hermann's tidy. Save. Yes. So it's going to run again. It's time. Probably again take around 30 seconds, which is very fast. Well, what one thing that's useful about all these mistakes that I've made and problems that I've had with files and exports and all this is that it's, it's really demonstrating how hugely fast it is to run thousands of Asian based modeling experiments. How easy it is to get the data that you want. How easy it is to change the experiments and rerun them. So, you know, we can really very, very quickly do a lot of work. The slow part is absolutely. Yeah. Setting up the model functioning. If you're writing a new model, that is a slow process. I mean, it doesn't have to be very slow if you're good at writing models, but getting it correct is typically very, very slow. Whereas the experimentation is super fast. Delete these. Save that. Close this again. Go back to our rerun these things. And we'll see. See what we get. Okay. Again, just, just not seem, I have no idea where. I don't know why it's not changing. It's tidy. It's just now. Oh, I see. This was my problem. What did I call this experiments? So, there you go. Everybody makes mistakes. Sometimes quite goofy, silly, obvious mistakes. The problem is now that I have misspelled experiment. Now, if you're not interested in using our, of course, you're not obligated. I, I like it because it makes these nice graphs quite straightforwardly. You can use anything else you want instead. All right. So here we see an interesting down slope at the beginning. It's not something we've seen previously. Go back to previous graphs. Well, there's a little bit of it down here. But if we go back a bit further, come on. Valid plot index here. The very tiny dip, but mostly, you know, with, with very small numbers, it's almost instant acceleration. Whereas in the later graphs, we do get these dips. What's going on? Why are there, why are people slowing down instantly at first? And they're spending quite a lot of time accelerating after that because they, they dropped so slow. So it's interesting. It's, you know, they're reaching pretty much zero speed in all of these quite quickly. It's an interesting one. And again, much more variation when there are higher numbers of cars. So it doesn't necessarily answer all of the questions. You know, like if our question was how many cars can we get on here that flow smoothly, we would have to revise because all of this suggests that even as many as 26, you know, that's too many that we're, they're not flowing smoothly because they're, they're dipping pretty much instantly. But maybe we assume there's, there's, you know, set it at 26, for example, cancel. Why is this so difficult sometimes? Okay. Let's set acceleration at 60. Deceleration, let's increase that just appropriately. And run this and see, see what's going on. Why, why is it? Okay. So let's look and we see, yeah, there's lots of start and stop. You know, the max speed is, they're never reaching the max speed. There's lots of starts and stops. Maybe this is just too many cars. So if we drop it down to 15, same, same settings otherwise. It is still starting and stopping, but much less so. Hmm. So if we, you know, we can just keep trying. You can try exploring. You can run experiments to see here what it looks like at the settings that your sort of analysis and averages suggest are interesting. There's a lot of different things you can do to find out the logic of what's going on. Now, if anyone wants to suggest another model that I can, you know, look at and maybe run some experiments on. That's fine. We can, we can look at other things. I can show you again how to set up more experiments. Maybe you're interested in knowing more detail about how to set up ranges. Maybe I can show you, you know, we can try importing the CSV export function. I can, for example, by open, go ahead and save those changes. Oh, no, that's not what I wanted. I can show you the, let's see. If I go up, I can look at the GIS commute model, you know, and we can, we can look at this, how the CSV export works in that and why it's different to analyze that. Okay. Let's look at the file exports. Okay. So in this model, whenever one of the little agents arrives at a destination, it, and the model is set to export data. It opens this CSV file, which is created based on an input in the interface. So this is the CSV file. So you could name it chicken sandwich, if you wanted, and it would create an output file called chicken sandwich, plus some numbers, plus, you know, CSV. And then it would print its own name, a comma, it's starting place, a comma, the local authority. Of that starting place, a comma, its destination, a comma, the local authority of that destination, a comma, and how long it took to reach that destination from its starting place. And then it would close that file. So in this case, the output will not have those headers about, you know, what, what version of net logo this was, what the dimensions of the world were, the date and time that it was created. I mean, you could keep those if you want, you would edit this, the file print to include those features if you wanted it. So it would need more commas and more, you know, variables. And it would include those in the file export, which might be useful to you. It might not. You could also put them in the name of the output file if you wanted to rather than just the input code from the interface, you could put code, the name input into the interface, plus the date, plus the version of net logo, plus, I don't know, something else, whatever you wanted. And that's why these files were created. And that's why these files are different in structure. Yeah. So these, these are different kinds of output you can get from your model, depending on what you think is important. Now the basic output that comes with net logo, you know, it gives you the option to go to interface tools, behavior space. Notice here, there's no experiments. You'll have to hit new. It gives you all of the variables, all the things that can vary. You do not have to set them all. So you could set export data to true. Oh, it would help if I could spell true. If I could get a list of false number places could be 50 and 100. And that's it. Look, label trim stops false. That's number generated trim stops only. You know, if this only matters if the projection is set to random. So I could set the projection to both random. And I would have to set it to GM. I think it's GM Alice. So I would need quotes around both of them, but it would just list them here. And if it's using random, I could set it to 20 and 25. For example, chicken sandwich. I could also set this to, you know, you know, so it would run. This is an interesting one. In this case, you probably, it's not that helpful to have variables here, because it will run all of the other things with all of the repetitions for each output file. So you will get two completely different sets of output files. Some of them called chicken sandwich. Some called verbal, but they will have all the same combinations of statistics. And, you know, all these other things you can set as you like. Repetitions, you might want to increase. Now, when you create a new experiment, it sets it automatically to be count turtles. This is basically to show you what reporters, how you enter reporters in here. But you could set that to other things. You could set it to, you know, average speed of turtles, or like we saw in the red car. It was the current speed of red car, because it was measuring runs at every step. You could set, yeah, it depends on how your model is built. And then obviously you want to set a time limit. Maybe you want more time in this because it's a bigger model. And ultimately, if you know that you have export data set to true, and that's really the interesting sort of data output that you want, you don't have to use the behavior space automatic commands at any, at all. You can remove measure runs at every step. You can hit OK. And when you hit run, you just uncheck both of these. So you won't get any automatic output. You will only get the export data output that you create in the model code. I'm going to cancel that because I don't actually want to run an experiment with those things. All right. So we are into the Q&A session portion of today's workshop. Please do ask me any questions about running experiments, parameter sweeps, how you choose parameters, all of that. You can, in fact, also ask me anything else about agent-based modeling because so far we don't have any questions on experiments. And so you are welcome to ask me anything else about the upcoming workshop maybe, or how I got started in agent-based modeling, or how useful it is to create different kinds of export data commands, what kind of useful things happen on there. You can ask me anything you like. I am here to help. We have someone who's raised their hand. If you're OK with that, then with Sorsha making you audible, you can just ask your question. Sorsha, if you want to make Peter unmuted. Yeah, we'll do. OK. I'm going to share my screen again because I assume that Peter will ask a question that it's useful to have an example for. We've also got a question come in on the chat, so it's fine. Peter, ask yours first. OK, can you hear me? I can, yes. OK, thanks. Well, thanks very much for that presentation, which I'm really, really valuable. I didn't ask questions before because I've been using that logo for some time. I got kind of stuck with the time it was taking to run the models and got something to run it in Python and then on the supercomputers at Durham University. And so my question is this, that my main focus has really been parameterization, which seems to have taken an earthly amount of time. And we're essentially using experiments to improve the parameterization. And I wonder, I've got my terminology mixed up here. Sure. Running what you're doing, but not to see what the outcomes are in order to improve the model. Is that the right way of thinking? I mean, early on in the PowerPoint, I had that sort of flow chart with lots of iterative loops because you absolutely do have to run a thing and then see what it is and then run it again maybe with a narrower focus or a different focus if your first look turned up nothing interesting. And yeah, if you're going to, if you're getting absolutely massive quantities of data, it does make sense to book time on big supercomputers and get them to run it because as I said, when I hit run from experiments, it gave me the option to run experiments in parallel and that works fine to a certain extent on most people's computers, but it will work a whole lot better on a supercomputer. But yeah, you, I mean, we all go into experiments thinking we know broadly what we're going to get. It was certainly we have, if we're trying to recreate a real world situation, for example, traffic jams in this place or, you know, in our rate of some of this that or the other, if we're trying to model pandemics, we'll try and play around with the settings until we can replicate the things we see in the real world. So in that case, you kind of have something to aim for and you can set up broadly like the full range and say, let's look at the output, which is close, which settings get us closest to the real world thing that we think and then we'll narrow in. And you can obviously do things with like, you know, Monte Carlo sampling and different kinds of things like that. And if you're trying to get big statistical pictures of lots of variables when they're all changing, it can be quite difficult. I mean, that's not a feature that's true only of agent-based modeling, like sampling complex sets of data with lots of, you know, variables is a problem for any statistical analysis. The benefit of agent-based modeling is that you can increase the number of repetitions so that every combination of variables is statistically relevant. So that helps a little bit. In fact, that touches on the next question, that question that someone has asked in the Q&A, which is, how do you change the number of runs? And so I'll just show that one more time. Did I answer your question, Peter, or do you want to, like, repeat it again? No, that's fine. That's good. Very reassuring. Yeah, absolutely. You kind of just kind of, like, the wide parameter sweeps are just thrown things at the wall and see if they stick. And then, like, you get to, you know, whatever is closest. Okay, so if you want to change the number of runs, you edit either when in the, you can edit an existing or you can create a new or duplicate set of experiments. Give it a name. So you might call it experiments. Underscore two. Well, that's bar two. That's not what I want. Okay, there we go. Repetition this here below the variables. That's what you change. If you want to change how many times the behavior space automatically runs an experiment with this combination of features. So it could be 10,000, 10,000. I mean, it kind of depends on how many combinations settings you have and how good your computational resources are. In theory, more experiment, more repetitions is better statistically anyway. But, you know, certainly early on in, in your narrowing in on the interesting critical spaces, 30 at least, like 30 is statistically relevant. Yeah, and that's a good number to do statistics. So start there. I believe I hope that's answered your question. If anyone else has questions coming in. Yeah, I'm happy to demonstrate other things because I, yeah, I really did whiz through a lot of that. So I don't blame you at all. If you missed an explanation that I gave or, you know, an example that I could have written another way. I mean, maybe everyone just wants to leave early, which you are also allowed to do. You're not obligated to stay to the end. I am, but you're not. But you can ask questions about models, about exports, about experiments, about code, you know, about other kinds of modeling platforms. I can't guarantee that I have very useful answers for that because mostly I use NetLogo. But yeah, you can, we can talk about it if you like. The, yeah, the, to say a little bit more about sort of experimenting, it's not, it's not easy to know exactly where to narrow in, especially if you're, if your model is quite like fact-free. If it's not trying to replicate a real-world scenario very well or if it's not trying to explore a really specific kind of case, if you're just sort of trying to figure out like, what would happen if I did this? Then it's, it's not easy to know what interesting behavior to look for in your statistical analysis. So it's not easy to identify interesting features of the results that suggest you should look closer. But yeah, if, if you are trying to replicate a real-world situation, it's much easier to, to sort of fine-tune your parameters because you can first start out with finding, fine-tuning the parameters to represent genuinely what, what you think the real world is like. So, you know, if you think, you know, if you know for sure that there are, you know, 10,000 people using the tram network every day, start out with, you know, 9,000, 10,000 and 11,000 people is the variable range for your people using the tram network every day. On the other hand, if you're trying to see what would happen if 15,000 people were to suddenly start using the tram every day, yeah, you can, you can put that in as a parameter setting. It kind of depends on your research questions. I mean, I can, I can open other models if we want to look at those. I can certainly look at code examples like CSB extension. I mean, this is the one that we, that we looked at with the sort of writing to file, the little agents that were placed randomly. GIS extension, if you want to look at mapping. Lots of other things though, if you want, I really recommend code examples if you want to learn how to, for example, get your agents to move in a circular path. Or, you know, like create fully connected networks. You know, there's a file output example. So that's, that's another example showing file print. Okay. That's probably useful to look at if you want to give the standard basic behavior space tables and spreadsheets and go with something that's more specifically useful to your model. Yeah. If you want hexagonal cells instead of square patches. That's good stuff in there. I really, I think it's very useful the way that they've organized the code, the models library with the code examples. I think there's a lot of good, good examples of how you can get, you know, turtles to occupy a space and not let another turtle occupy the same space, for example, if you're trying to show how people move through a room to take their seats in a, in a classroom, for example, you want turtles to not walk over the top of each other because that's not what people do. You want turtles to find a place and sit down and occupy it. Arguably, if you were being realistic, you'd want turtles to find a place, sit down, occupy it, put their bag on one other place and their jacket on a third place and then occupy all of those until someone comes along and gives them the eye. So, you know, even this is not super example, super realistic example, but it's better than, you know, turtles who walk over the top of each other. Yeah. There's a lot of good network examples as well. There's not those random walks. Yeah. There are no further questions. I'm astonished. I'm astonished there are no questions. I mean, maybe I shouldn't be. But this is how it goes, I guess. Maybe I've completely overwhelmed you. But there's still 15 minutes left. So I'll give it a few minutes. If no questions come in in the next, like, 15 minutes, then I'll just say goodbye and we'll end this slightly early. But please do feel free to ask questions about this model, about other models, about exports, about experiments, about writing code in R to model to, you know, to have a look at your results. Let me know. I'm here for you. So let me know if there's other things about agent-based modeling that you'd like workshops on, because we can, of course, have more workshops in the future about, you know, if you're interested in building, I don't know, economic models or supply chain models or, I don't know, travel models. Like there's all kinds of interesting behaviors that people might want to look at. And we can, you know, maybe take a broad kind of question and try and build a model together about how to approach it. Julia? Yep. We've got a question that's come in from YouTube. All right. So I have heard that Julia is very good at agent modeling. Have you tried it? Thanks. Is that a program or? Agent model. I mean, there's agent spring. Yeah. Because this next question is, how is it better or worse than net logo? Okay. I would need more clarity about exactly which model that is. But I would direct people to my, the first sort of webinar in the series. We'll just stop sharing. So my face is big. In the first one I did, which I believe was January 19th, the video is available on YouTube. The slides are available in either our GitHub repo or on our UK DS past events pages. In that slide deck, I cover several different kinds of modeling software and some of the pros and cons of each of them. So I would say that the best thing that I could do for a repast, I think agent spring, is another one. Doing agent based modeling directly in Python. I mean, because it's an object oriented programming language. So it is logically compatible with agent based modeling, but it is obviously a very broad tool. And so it doesn't have things like an automatic visualizer. It doesn't have a behavior space to automatically run your code multiple times at different combinations. It's powerful, but it's not user friendly. I mean, the reason I choose NetLogo is because it is extremely fast to get going with prototype ideas because it is absolutely designed for agent based modeling. It has a lot of built in language that makes it very easy to develop a concept from nothing to functional and experiment ready agent based model quickly. So the downsides to NetLogo and the reason why another language might be better is that NetLogo sort of maxes out if you have a complex model. So when we try and run this tram commute model with the real world GIS and real world census data, that's tens of thousands of people with, I think, 98 tram stops or something like that. And it's absolutely at the limits of what NetLogo on my own laptop can do. NetLogo is obviously much stronger if you want to run it on a high powered computer through your university or some kind of remote server allocation. But ultimately even then NetLogo will max out a bit faster than some other languages. So if nothing else, I would say start in NetLogo, prototype your concept, make sure it works logically on the cheap, easy, fast NetLogo, and then rewrite it properly in another more advanced language after you've already worked out the basic logical bugs in NetLogo. Because more advanced agent based modeling languages like RePast or Mason or Agent Spring are very, very strong and operate under the same basic logical principles as NetLogo, but they are not easy to learn and they are not user friendly interfaces, at least not in the same way, not as user friendly. So that would be my advice there is that start with NetLogo, get a working prototype, move it somewhere else if NetLogo maxes out on you. I promise I'm not being paid by NetLogo. The person who asked the question has added, I mean agents.jl which is a Julia framework for agent based modeling. I haven't used it. Actually some members of my team are looking into learning Julia as a programming language. I would assume that like Python, because it's an object oriented programming language, it is totally compatible with agent based modeling, but that the interface will be not user friendly and that it will be much more powerful, much more effective, but that it will be harder to get started, especially if you are not already have like a logically functioning prototype. So that's very good. I can add that to my slides for the next time I run this agent based modeling series. I'll add that to the intro slide on, here are some options for agent based modeling and I'll put in there that people are asking about this. This is cool. This is interesting. I'm sure it's very powerful and very good, but I suspect it is not user friendly. I suspect it wouldn't, for example, have this sort of like automatic visualization where you can see the little agents moving around and that's very, very useful for bug testing, for like prototyping, for getting your head around like what has the code that I wrote actually done. But maybe I'm just a very visual person and someone who's a whiz with programming would find this a big distraction and waste of time. I don't know. Maybe. I do think, yeah. If you're interested, like contact me on Twitter. If you want. Jay has my complex. So it's just my first initial, my surname and the word complex all ran together and there are no spaces. You can, if you're interested to know more, I'll have a look at it and give you my impressions. And we can maybe have a wider discussion about agent based modeling in object oriented programming languages and pros and cons. Yeah. Sorry, I don't have any direct experience with it. Yeah. That looks like it. So thank you very much everybody for attending this workshop. Please do contact me via email or Twitter if you want to follow up on anything, if you want to start discussions, if you end up using agent based modeling in your research and you want to tell me how it worked or, you know, like that we're always happy to host, you know, to sort of platform or to share the results. If you publish a paper using agent based modeling, we're all cheerfully retweet that. Let me know. Otherwise, I will let you go and have a good one. It's been lovely.