 Yeah. So, scripts, what's the... Yeah, the general first question is why using scripts at all? And why not just stay in Jupyter and use Jupyter Notebooks for what you're doing? And yeah, I think the main reason why moving away from Jupyter... Well, why do you move away from Jupyter is essentially if you would need to run your Notebooks multiple times with multiple different parameter settings and so on. Or if you want to make them more modular and more easily used by other people, as this is no longer just one script to replicate some data from a paper, but this is becoming library functions that can be used for multiple different purposes and so on. So as soon as the complexity inside your Python Notebook starts to be that you need to be careful in which order you execute cells and so on. At that point, you kind of start to think about, okay, maybe I want to create scripts, I want to modularize my code. But the biggest thing for me at least is using different, well, multiple different parameter sets and potentially for more expensive computations so you can't just run them in a for loop. So basically getting someone, if someone sends you a Jupyter Notebook and says modify these few lines to make it work. That's not good. Yeah, maybe you may send it yourself. That's kind of the situation where I have a script that works for something and now I need to modify it. And even if I have it under things like version control, the file that I'm currently working with will be modified and it might not work for the older anymore and I need to have a copy. And then I do some some modifications in the one file but not in the other copy and it gets messy. So, especially things where you want to add flexibility on what's the inputs. We don't want to have a, well, please enter the file name for your file here in the Jupyter Notebook. These kind of things where it's a lot better if you have it as a script and can use input parameters for the script. So do you want to see, well, I guess that's what we just talked about here, why scripts. So how do we make these input parameters? Should we get straight to a demo or is there anything else to talk about beforehand? I think we can essentially take, well, the preparation for the first example or for the first exercise and just walk through it a little bit or walk through the preparatory steps a bit. So for this lesson, how much is type long and how much is demo and how much is exercise? For exercise one, I think the idea is that we'll show you how to download and put and get the data into your Jupyter instance. And then the rest of that exercise is essentially yourself doing it and converting it. Yeah, and I think we have good solutions for everything here. So you can also do it as a follow the solutions instead of figure it out yourself. Okay. And I think there's also a bit of a roundabout way of downloading the files here. So we tried very carefully to find a way that anyone works for everyone. Anyone can do scripts sort of from the command line without having to know how to use the command line in advance. So in reality, most of us would do this slightly differently. But what we're doing here is designed to be like a simple way of everyone can manage it. Okay. So the first thing is, So what's up here on my screen? Do you want to share your screen to start off? Can also do that. Just need to I'm switching to you. There we go. So essentially where we are at the moment is we have a Jupyter. In general, we have a Jupyter notebook and we want to get a Python script out of it. And Jupyter actually in itself already already gives the gives options to do that by safe and export notebook as and they can simply say executable script, which is a Python script in the end. There's one thing that you should keep in mind when doing that. Optimally, you remove any mad, any magics beforehand. So all these time it and similar commands should be removed. The export does allow to use them, but then you need to have additional imports that those files normally don't have. And it just makes things a lot more complicated. So as a general principle, remove all magics before you export a Jupyter notebook to a Python script. Yeah, okay. And then there are there is of course the other option that I think we would normally take we would normally just go to the command line and use and be convert, which is the same as what's happening with the always the menu item, but more convenient to someone who is more accustomed to the command line. Yeah. Okay, start doing good. You can actually start downloading the the files and back to my screen. Yep. Let's see what's the right order for my windows. There are multiple options what how you can get the downloaded file into your trip to instance, depending on your operating system and where you started your trip to lab and so on. It is it's easier or a little bit more complicated. So the easiest is download this. The one here. I'm downloading it with the web browser. This link here. So it says it's downloaded. And now I come to Jupyter lab. Yep. And to be sure that if you if you know where your trip to lab is you can download and you can download directly to the directory that also works. The other option way which is probably easier is if you click on the upload button. Yeah, to to to further right. That one. Okay. And then you can navigate to your downloads folder. And just select the file and upload it to your trip to lab instance. I know this is ridiculously complicated. If you know what you're doing. If you are not entirely sure or more beginner, it's the simplest way to be on the safe side that this is going to the right place. Yeah. But anyway, here I have it whether observations. Yeah. So the point is I can now open weather observations in Jupyter. Yeah. And, well, now I would give everyone five minutes to just try. Try the. Okay. Try this five to 10 minutes to try the try this exercise or from step. Yes, essentially from step to onwards because step one you have already done. Yeah. Okay. Oh, we go to five minute pause. Yeah, I think that would make sense. Yeah. So if this was trivial, trivial for you, I'm sorry, but well, believe me, it's probably not trivial for everyone. So, see you soon. Like, hello. So, um, yeah, assuming most people managed to get this converted and headed run. Yes, it doesn't produce any out any directly visible statements. Um, it's just producing an output file. It's producing a weather dot png file that is in the same folder. And that is essentially what you would see what you also see in the Jupyter notebook if you run it. Yeah. There was a good question from Twitch chat any preferences or experience working with VS code versus Jupiter lab in these kind of settings. No, admittedly, I have never used. But we can use code for Jupiter lab versus some other editor or environment. Well, I mean, all the conversion thing is in the end I would say it doesn't matter what editor you use. The problem here is that I have no idea how exactly you export or you convert Jupyter notebooks in VS code into Python files. Yeah, I'm pretty sure there are tools to do that. Yeah, I'm almost certain, but I haven't ever use it. So I don't know. Yeah, I mean, in practice, most of us would probably use be doing this from the command line using the nb convert command and then editing it in an editor instead of in Jupyter lab. But this is a simpler starting point, we think. Okay, so I'm coming back to my screen now. And what do we do now? Let's essentially this was the exercise. So command line arguments with sys.argv. Yeah, this is essentially just a demo. So in this in this script, you have a couple of different arguments that or a couple of different things that you could change. One thing is, it has a fixed start date. That you that is being inserted this first of June 2021, and the end date was the first of October 2021. It also has a fixed has fixed input and fixed output argument names. And this is all stuff that, well, here, these are the fixed dates. Okay, this is all things that you might want to change in your analysis for something or in your image generation. So you might want to change the start date to beginning of 2020. You might want to output, give the output file an actual name that reflects what's in there in the end, because at the moment it's weather. Yay, that doesn't say anything about what's actually in there and what plot this is supposed to be. However, it's kind of inconvenient to every time change the code itself, and it's prone to lead to errors where you change the one but didn't change the other. And in the end, you have no idea what your what which file was producing or what which code was what which file actually contains. So I'm changing this to command line arguments has the advantage of making the code more flexible and allowing you to essentially run it just from the command line with with different inputs. And the most basic approach here is the Swiss package from Python. The Swiss package gives you a bunch of tools that you have to do with the system. Like for example, what additional input arguments are given to the to Python up on starting it. So like here, can we see this example. So, you could essentially open the Python file that was generated in your in your trip to lab. Okay, yeah, I'll do that. Did I export it myself yet. I think I did it doesn't seem like it save notebook as no export. Save and export not because executable script. Okay, further down. Yeah. Oh, I did it not save it in the current directory. Because for some reason it saves it probably in your downloads again. That's. I don't know why it does that. I assume it's essentially X. Well, worst case uploaded from your downloads. Okay, so here we go. And I open it. Yeah. So if you put in the importer said it's a statement in the top so that you're actually importing it. Should I be doing this or is this the exercise is this type? This is this is either demonstration or type along. Okay. The exercise is doing it with a system that is more flexible and actually better for your scripts in the end. Okay, so people should be typing along. So I'm adding import sys, which gives us a thing like access to the sister this package and start date becomes sys.org the one. Agree is all the input arguments that are given given to the Python argument. So you normally normally know that the argue that Python lists start with an index of zero. But the index zero is essentially the Python file that is being executed. So in this case, it would be weather observations pie would all would always if this is called would always be sys RV zero. So we give it the first and second argument and give it an output file name. Oh, yeah. Okay. I'll put file. That should be further down here. It shows it up at the top. Oh, okay. And then all the way down us replace the other safe big safe big on by the weather being by your foot filing. Okay. There we go. Should I save it and try to run it? Yeah. Okay. How do we save? Yes. Save. So control s would have done that. Yeah. So to run it, I do it from the terminal. So I will make a new launcher with this blue plus and come down and terminal session here. And it starts and it's in the directory that I started Jupiter lab. So it should be here. Yeah. And if you scroll a little bit further down, there is an example command. Okay. In the, there's an example command that that we could use now to run it. Python weather observations dot pie. Oh, one. Oh, three, two, oh, two, one, two, one. Oh, should I run this? Yeah. Okay. The arguments that are passed in are separated by spaces. So whenever you have any, any arguments that have spaces, it becomes tricky. Yeah. Okay. We'll talk a little while but work. Yeah. Mainly probably because it downloads the data again. Yeah. At least in my, on a Linux and Mac computer, the LS command will show what's in the directory. And you also see it on the left in your Jupiter lab. Oh, yeah, I see spring. Should I click it to open? Yeah. Nice. So. So yeah. Not bad weather. At least temperature wise. Okay. So. So. Well, this works. We now can put in additional, well, put in the start date. We can put in an end date and a. Output file. Yeah. And is this better than modifying the script every time? I would definitely say it's better than modifying the script because if you modify the script, you are bound to have typos in there. You can't, you can't really. You can't really change additional things in the, in the script without having to recreate or create a new version of it. You can't save, save the file that created your, that created a certain certain figure. And like this you can essentially stay save that command and this command gave that figure. So I only need to need to kind of keep track of the commands I was using and not also keeping track of all the, all the versions that were being used. And I guess we can even write a script to run the script with different arguments and stuff like that. Yeah. But let's not get there yet. So what problems do you expect when using this approach? So what is this ugly? Well, it works okay for three arguments. It can be a bit hard to remember the order like this to output file come first or last. And what if there is say 10 arguments or something and I don't need all of them. Yeah. And I think the last is actually the most important. So if you have, if you have optional arguments, as long as you have fixed fixed arguments and all arguments are needed as inputs. Yeah. Then this works okay ish. You might get into the position problem. But I think the biggest issue is if you want to have optional arguments. And they're tools like our past doctor click. There are a lot of different libraries that can give you really nice input parsing. All of them essentially also give you some additional functionality like provide providing you from what you set up a help, help file. You know, all these tools where you have tool minus, minus help, and you get an information on how the tool is to be used. And all these libraries give you essentially something like that. So, yeah, our parts is the simplest of these built into Python. And it's built into Python so you don't need anything in addition to it. The others have a lot of additional functions normally. But we will stick with our parse for now because it normally does most things that you might need. Yeah. And personally, I always use art parse because it's built in and I don't have to worry about making additional dependencies if I wanted to be standalone. But so art parse. Yeah. So art parse is essentially structured that you create an argument parser. Should we see the example here? Here we go. So we make the argument parser object. Yeah, you don't have to type along here at the moment of this will be part of the exercise in the end. And this is just an example how it looks. So you essentially you create an argument parser. There's a comment in HackMD. So, yeah, right now just listen to us. You'll have time to actually do this in a bit. So, yeah. Okay. So we make the argument parser object. Yeah, you make the argument parser object and then you set the arguments to your argument parser. And arguments have a, well, I have two different types of arguments. There are positional arguments, which have a fixed position and that position is dependent on where or when they were added to the argument parser. So the first is at position one, the second is position two and so on. And there are optional arguments. Optional arguments are the things that in Linux you get with minus, minus something or minus value. And those you specify by minus, minus name, space or space and the argument that you want to put it. We have an example below. Like Python birthday.py. And it helps an optional argument here, even if it's automatic. Yeah. Help is an optional argument that doesn't take any value. And is automatically generated by our pars. With optional arguments, you can give a default value. And something that is really nice with, oh, and there's actually a mistake here. The type should be str. String is not Python that I need to fix. Okay. But the nice thing with specifying the type here is that you don't need to convert anything. All input arguments that you normally get with sysrv, for example, are all strings. So if that is supposed to be a number, you first have to convert it into a number. Otherwise, it will use it as a string and you can end up with sometimes with auto results. If there is an auto conversion in some way. But with our pars or the other libraries, you can only specify what type a certain argument is, and it will, at least for the for most simple types, it will convert it into. From the string, essentially. So do the conversion automatically. And then, once that you have to find the arguments in your function, you say, well, parser.ParsRx, which is essentially telling it, okay, take the sysrv input and follow all the rules that I've said with that arguments and produce the arguments in here. And then you can essentially access all the individual arguments by arcs. And their name or in the case of date, it would be the minus minus argument. I'm not entirely sure if you can say arcs.d as well. Richard, do you know that? Let's say that again. If you would say arcs.d for the date. I don't think that would. I think it doesn't work either. I've always used the full name. Yeah. But this is how it works. And that allows you to make the parsing more robust and just in general more, well, better. And it automatically generates this nice help message, which lets others know what they have to put in, which is also quite useful. Yeah, so should we go to the exercise? Yeah. We have until 50 of the hour. Yeah. If you do 15 minutes for exercises, then there's about five minutes to wrap up and see what advanced stuff you might want to read about later. Okay. Yeah. So, yeah, so this exercise to I'm putting into HackMD and see you in 15 minutes. Okay, we have a bit less than 15 minutes. Okay. See you later. Bye. Hello, we're back. So we have only a few minutes until we are we go to the break. So let's go straight to it. So what did we get from here? What's the point? Well, we essentially are now able to pass multiple arguments in a fashion that we can understand what each argument what each input is. We have an help. We have a help. And yeah, that that makes reading the command the command lines a lot easier and also using them. Even though these look long, it sort of makes sense. Mainly long because of the, because it's URLs that are passed. Yeah. Okay. And it really is not that hard. Like it's something that can be done to every script and things like click are even easier to use than our parse. But what's, where does this start getting difficult? So what about this line here? And it's where it's starting to get huge and where we need to think about something else. At some point, if you have too many more or less necessary arguments, you might think about trying to use some config files where you can store your general configuration and then overwrite some things with the input arguments. So, so what's our proposal here? There is this. Yeah, well format, or yeah, well, what it's not about the format but the idea. The idea is essentially having in having arguments and values in some form. Yammer or Jason are common forms how to how these things are being structured. So basically the idea is you could put all of the general information in one file. And then you run your program and you just need to give it this file. Yeah. And it knows. And there's an optional exercise here which we don't have time for with you wanted to try some of this. So that said, there's some extra stuff to read but that's basically it. So what's the summary? Summary is essentially we had we went from a Jupyter notebook to a Python file we made our first excursion into how the command line works. And we have now the possibility to write scripts that can be used for example on an HPC system where you can come out where you can run multiple different versions in a larger script. Again, that gives certain input arguments for you for each run. Yeah. Okay. So we're off to the break then until two minutes past the hour. So see you then.