 Hello, we're back is our audio good. Hello everyone, you hear us. Okay, so now we're demonstrating packaging a thing so that it can be installed. And of this I'm typing and Marijn is telling me what to type, so. Yes, I will take the lead. Let's get to it. So thank you, Richard. Yeah. So packaging, packaging basically means trying to organize your project in such a way that other people can quickly install it. So you're doing packaging well when somebody comes to you and says, well, how do I use your code in my project? You can tell them, oh, you just pip install it or you just have one line. The shorter your installation instructions are, the better your packaging is. That's a good point. Yeah. By the presents a few options of us to us, we'll explore a few of them. So what if you just have a single file, for example, with some functions in it, Richard, how would you share that or we have any issues already? A single file. I guess if it's that small, I might just leave it as it is and say, if you need it, then whatever clone to get repository. I have several projects like this, which are so small, I haven't bothered trying to package them and it's cloned and sort of run manually. So do I. If it's just a few functions in a file, you just say, well, installation issue, take this file, you put it in your project folder and you import. Yeah. You call it a module when it's just a single file. But what if it's multiple files? So if it's a single file, you can just import and then the name of the files we have seen, but say you have like 20 files. Yeah. If installation instructions as well, you have to do 20 import statements in order to use them. That's going to be very annoying very quickly. Yeah. So maybe we can create a few files or Richard, you can create a few. Yeah. So are you seeing my screen through the gypsy? Let's see. I can. I'd recommend opening that. So you can see. Okay. So are we doing, we're starting with the demo and I'm adding these files here. Actually, let's back up. So I'm starting and I'm making a brand new package here. So here I am. I'll change to my Git directory. What will we call this thing we're making? Let's call it a packaging project. So our module is going to be called a calculator, but we're going to name this folder something else. You will see why later. Okay. So say we have now, let's quickly create three files. Say we have an add and subtract. Yeah. So I will make these. I just basically copy paste them. Yeah. So just, these are simple things that we just have something to work with, right? So our project now has three files. Each file contains some code. So if I now would want to reuse these files in another project or give them to somebody else, as per some, I have to do three import segment, import subtracting, adding, import integrating. So if you want to now make it like a single import statement, for example, if you import numpy, you don't have to import everything. You can just import numpy and it works. This was Python calls a package. So each file, Python calls a module. And when we want to bundle them all together as one thing, we call it a package. And the way we do that is we just drop them into a new folder. So you can, if you want to bundle this up, you can just, oh, create a folder with a name. And the name of the folder will be the name of our package. It's gonna be what we will be importing. So what should it be called? So do calculator. Did I spell that right? Yeah. Okay. So yeah, now we have a calculator folder, okay? Because dump all the Python files in there. Okay, so I've moved them into the calculator folder. Yeah. So I'm running a lot of Linux commands here. You don't need to worry about that. We're saying what we're doing and arranging stuff. Could also do it in Windows. Just create folder, drop the files in there. Yeah, okay. So we're almost done. But now it's just three Python files in the folder. There's one thing we need to do to actually make it a Python package. And that is to create one special magic file inside our folder, inside our calculator folder. And this is called underscore, underscore init, underscore, underscore. So the underscore's indicate this is like super magic special file. Yeah. It's basically just the Python convention. Yes. You can actually leave it empty. So I've now already pasted stuff in there, which is good. Yeah. Just to note, I mean, just the presence of the file makes it a Python package. Yeah. Like maybe I would say having the init file tells Python that a directory is importable and can be considered like a package. Something like that. Yeah. Okay. But now let's go to your file because you pasted something in there. Let's show you people what you've pasted. The init. Yeah. But this code, there are, so this init is this magic file that indicates this is a package, but the init is also the entry point to the package. So if we, well, without the init, you could import calculator, but then if you want to actually use our files, we'd actually have to still manually import them. You have to say, okay, from, well, the function would be named calculator.adding.add, calculator.subtracting.subtract, which can be a little bit annoying. So one of the cool things you could do in the init is add import instructions right in your init file. So the init file will be run. Whenever you import your package, the init file will be run. So these import commands will run. So if I run import calculator like you see here, then this gives us, or it gives us calculate.add, because the calculator module, when you import calculator, it gives you this init file. And this init file has imported ads to attract an integral from these submodules. Yes. Without them, you would have to do calculator.adding.add. Yeah. Let's see. You also put a versions training there. That's also very useful. So now you would have a calculator.underscore.underscore version as well. And any variables and any functions that you declare inside this init function, there will also be part of the package. Okay, should we go? Okay. So now we can import a calculator. So we quickly try that. So if you just open up a Python, can we now import calculator? Okay. It actually works. So now we have it. Calculator. So this shows us that the calculator module itself is located at calculator init.py. Yes. See, init is our entry point here. Okay, and all our functions should be in there, right? While we are adding, yeah. But we don't need to manually import now calculator.edding, right? Our init file should have taken care of that and should all be there. So we have add directly in there. Okay. Okay. So now it's already better, right? So now if you want to tell somebody else, okay, how do I use a code? Well, here is this folder. And you put this folder of files, you put that into your project. And as long as this folder is there, you can import it. We can go even further, of course, and we can say, well, you can just pip install it. Okay. So that's what we're going to do next, try to get it pip installable. Let's see. Should I maybe scroll it down? Yeah. Maybe I should say, well, so for now the installation instructions are put, you put this folder inside your code, your own code folder, your own project you're working with. But this is of course not very nice. So if you have multiple projects that reuse the same code, you now have to basically copy, paste this same folder in all the locations where you want to use it. Of course, Python has a central location. So if you put this folder in Python's central location with Python called site packages, then it will be available for any time, right? This is where your NumPy lives. This is where you're not plot lib lives, your Jupyter lives. So let's see if we can make it easier to install the package. So when we say now, installing the package, basically it means you take this folder and you copy it to your site packages. And Python can do it for us and provides mechanisms to do this. Yeah. And the entry point to that is another type of special file which we call setup.py. This is the mechanism Python uses to make a package installable. So you can just tell the package, okay, install yourself into the central location that Python has so we can use it everywhere. Okay. So is this what I'm doing now? Yeah. So I'm making a setup.py with these contents. Let's see. Let's hope the paste works. Yes, it worked. Yeah. So what are the main features of this? So setup tools is somehow the common thing that's used. Yeah. It provides the setup function that we call below. Yeah. It's trying to open a readme file and we'll use that as the description of the package. I guess we'll need to add that readme for it to work. Oh yeah. That's a nice shortcut that we usually do. So since your project, probably if you put it on GitHub and stuff we'll have a readme file, you can use that as the long description of the package. Why not? So just open the open file. Okay. So we should also remember to make a readme file. So this is actually work. And this name is the name you use to install it via pip, which has to be unique. So that's why it says put your own name in there. The author. Yeah. So put your own name in there. It's not a general recommendation. You can just name it like however you want to name it. But since the name calculator is probably already taken by Pi, so later on. So we will add something unique to it just so we don't clash with existing packages. But you can name it everything you want. I'm leaving off the author email, which even though it says it's needed in practice, I've noticed it's not. Okay, there's a short description, a small example package. I'll leave that. Here we see the long description being added in. It's being told that the long description is marked down. URL. Well. Yeah, we don't have a website yet, but if you have one, you could put it there. Packages. Okay. That is a special one. So there could be inside your folder. We could have, we could bumble multiple packages together and say, so one of the bigger ones, I think if you have like a SciPy or something like the bigger ones, they, these are a project that actually consists of multiple packages. And so basically what the setup call needs, well, okay, what's the list of packages that you want bundled into one thing. So if you do install on this, what should I install? And you can usually leave it like this because setup tools provides this convenience function, find packages that will just look at what's there inside this folder, what packages are there. That's the packages you want. And I think 99% of the cases, that's exactly right. You want the packages that are inside your folder. That's it. Okay, sounds good. So install requires, I guess this is the dependencies. So we're saying that SciPy is needed. Yes. I think our integration function depends on SciPy. Yes. Just to illustrate, how do you denote package dependencies? So. And can we have classifier lists? These are like tags or something. You can find some big list of what they mean online and find the ones that are appropriate. It might be like scientific software, this or that, whatever. Exactly. What platforms does it run? Does it run on a Mac or? Yeah. I think here it says always independent. It will run wherever Python runs. Yeah. So it will run. And then Python requires. So I guess that's sort of obvious. Okay, so this is it. Okay, that looks good. So now, let's not forget to make a read me file, otherwise this, our long description will not work. Maybe mistake. Helculator. Okay, there we go. So now that we have the setup.py. Okay. So the setup.py actually operates now as a script. This is now our installation script, the setup.py. So we can run it, you can just try it. Python, if you go to the command line and say Python setup.py. It's now a thing. It's now a script that you can run. If you don't give it anything, it will just give you some help, I think. Yeah, okay. No commands. And I believe install is a command. Install is command and build. Yeah, so it has several things. Should I run this? Install is the most useful. Maybe I should activate my Anaconda environment first so I don't mess up my system. Okay. Yeah, do whatever you want. Yes. I mean, an important thing it provides us is install. So, and install does what I said before. So install will take the folder of the package and will copy it into your central location where your Python stores all its packages. It's just called site packages in full. Okay, so it seems like it worked. If I read using this, my home directory Anaconda live site packages. So installed, it says finished processing dependencies. So yeah, that worked. Yeah, it will also, I think, well, it will check the dependencies. It will check if you have all the dependencies. I think this will just give an error if you don't because this Python script does not have a, well, it's not pip, right? It doesn't have a complete dependency resolver. Oh, right, yeah. So when you installed with pip, it will, but if you just run it as a script. That's it. So now our installation structure is already better. So now our installation instructions is, you get this folder and you run our setup script inside the folder and you tell it to install itself. Now we're getting closer to having like good package code. I'm giving an example here in my home directory, I can't import it. If I source, if I activate Anaconda. Yeah, so it's been installed from the Anaconda site packages directory. Yeah, so. I'm going to be upstairs. Okay, so this is good. So so far, I can install something at least. Yeah, okay. It's going to be better, right? Can we make it even easier? Yeah. It's even easier if you don't, you don't even have to clone the git repository. If you just do pip install. Yeah. And we'll just do it. So this is how you would also install all our bigger packages. You don't always need to clone git repository. You would just say, pip install, give it a name. So can we? So there's this exercises one here that's test a local pip install. Should we tell what that means? So we actually, yeah. Or should we? Yeah. Maybe let's just, I guess we can demonstrate it. So here we installed it with python setup.py install. There's also pip install. So there's a dot here, which means the current directory. So I'm telling pip to install what is in here, which in practice is what I do most of the time. So I hardly ever use setup.py anymore. I used to- Well, you need it, but pip calls. So when you do pip install, what it does is run the setup.py script. Exactly. Okay. So- I have no idea what this is, but we see it was trying to install it. It said things like requirements are satisfied. Yeah. It's cool. Yeah. And then at some point it worked out, okay. But anyway, yeah. So if you do clone something from GitHub or something, pip install and the directory name will install it and then also try to get all of the necessary dependencies and install them. Yeah. Okay. Even do pip install and then give it the URL of your GitHub repository and build this. Oh yeah, exactly. So it's downloaded and then install it. So that's a good way actually of providing good installations instructions. If you don't want to upload it yet to the official pip repository, if you just have some work in progress, it's only on GitHub, you can do this pip install and then the URL and it will just fetch it from the URL and install. It's also a good way to have a single line installation instructions. Yeah. I don't know if this is exactly the correct GitHub URL scheme, but something like this. Yeah, it looks a little bit like that. Okay. So- Okay, that's good. But let's take it one step further. So we're now pip installable already, but we're not part yet of the official like pip, but what is it called? PyPy, PyPy repository of all packages. Yeah. So let's upload it to that. So maybe first create an account, a PyPy account, but let's not do it on the real, real PyPy because a little bit clumsy to upload our example package to the official repository. There's a test repository. There's test PyPy which you can use to do whatever you want. And we don't mess up the real repository. But in order to use it, you have to create an account there first. So maybe you can just follow the link in the lecture notes. So I can- I'm sure you already have one. And I've confirmed I already have a account there. So that's good. Okay. So now if we want to upload stuff, we're gonna upload stuff to the test PyPy and there's a little utility that will make that a bit easier for us. And it's called twine. I guess that's some multi-python reference. Maybe. I'm not sure it will be a multi-python reference. Okay. So do you have that twine command? And otherwise you can just pip install it. Yeah. So I'm in Conta, so I will- Yeah, twine. You need to install it. So this is installing to my base environment. Well. Good enough. Okay. But before that, there was this step. It's Python setup.py. So we need some- Yeah. So what twine does, twine will not try to parse or set up.py and do all that then. All that twine does was just upload zip files to PyPy. So we first need to create a zip file for twine to upload. And that is something our original setup scripts can do. So we've already seen setup install. But this is setup as dist. What that will do, as dist stands for source distribution, it will zip up your code. So what happened here? We see a dist directory. Yeah. Distributions. Calculator, RK, Darst, and then targz in the version number. Okay. That's our little zip file or targon zip file. Yeah. And then we can upload with twine. So now that we have that thing, we can tell twine, okay, please upload that zip file to PyPy. But not to the real PyPy. Let's say it, okay. So that's a dash R testpypy. So okay, we're gonna upload it to the testpypy. Not the real one. Are we sure this is the right thing? I don't know if you want to do dist star. I think you just want that one zip file that we created. Yeah. I don't think we want the egg. Yeah. Okay. So twine upload to the testpypy. Are we sure dash R is the right thing? Maybe I'll check the help first with the dash H option. Okay. And don't notice this is made with ArcParse. So it shows us this really nice help here. Repository. Okay. So I guess at least it won't upload to the proper one. So should I do it? Yeah, try. Okay. Okay. So it asks your username and password that you, that's your account on testpypy.org which are already made. Yeah. So it says it uploaded. All right. To there. And there's a URL. Maybe I'll copy that and let's paste it here. Calculator RKDarst. It tells us this nice pip install command. So telling it to install from testpypy, the name. This also tells us the version number and if we need this exact version. Oh, here's the readme that I entered here. That's taken from the long description field I think of your Python. So it's not taken from your readme file. We see a readme file here because we like put that into the long description field of our setup poll. Yeah. And here we see the homepage. This is example.org. So that was also defined in the setup.py file. Yeah, we left it at that point. And probably also, does it tell your name? No, it does not tell your name. Oh. The author, no. Here, maintainers RKDarst. Oh, there it is. Okay. And here's the classifiers. So, yeah. All right, let's see all the things. So that's really really nice, right? And it will be the same on the real pip repository if you upload it to the real one instead. Yeah. But it's always recommended to first upload, of course to the test one to see that it works. So. Okay. So we're going to pip install it based from this test repository. Let's see. So maybe I can... First have to uninstall it because we already have it installed. Fine. Yes, okay. So it says it's uninstalled. Right. Maybe move to another directory. So we... Yeah. I bring in my old terminal here. Oh, yeah. New terminal. What was the command? So if it were in the official repository, we could now just type pip install calculator Richard Darth. But we've uploaded it to the test repository. So we need to give pip also. By the way, use the test repository, not the real one. And it seemed like it worked. So, yeah. Nice. So now our installation instructions are that cometi just typed. So if you want to use this package, you just pip install this. Yeah. That is already pretty easy. Yeah. So what are the other considerations here? Is there some minimum size of code that you would use to do this? Or if it's small, would you sort of leave it on GitHub and give the manual URL? Well, for me, it's not so much a discussion of code science, but more how many people do I think will actually use this? So if it's something I just write, and I think, well, maybe my friend and his dog will use it and nobody else, I will just leave it on GitHub. But I have one project that is essentially one source file, but it's so useful and used by so many people that I did put it on pip, like on the proper pip. Because when it is there, when it's a file dependency of many projects, it's really useful to have it in pip because when it's in the pip repository, people can just list it as one of their dependencies and people will automatically install it. And this is a little bit hard. I mean, you can still do it when it's just on GitHub, but it's a little bit harder. But for me, it's a question, how many people are actually using this? And if it's a, yeah, maybe more than five or something, I say, okay, I will go through the trouble making actual release out of this. Okay. We see building a content package and sharing it. We could try to do that. How much time do we have? I think we are now at half an hour in. We're out of time basically if we want to do the panel discussion idea. So. That's actually a good idea. Yeah, but let's just say you can do the same thing with Gonda. And actually it's pretty easy when the instructor last year demonstrated this, I was like, well, okay, actually it's not so bad. Once you have. It's not so bad if it's already on pip. So now our package is on pip. And when it's on pip, you can make a content package that says, well, basically it's a pip package. So you just, the content recipe is, do we install it from pip? And then it can, then it's pretty easy. You have the instructions in there. Yeah. All right. So do we have any questions in the HackMD? Let's see. There's pip versus content. And yeah, they're basically different tools. So if you scroll to the previous lesson or earlier in the day, there's some discussions on the differences between these two. There's some comments on the pip install dash E option. So this is really interesting. This is something that I thought was quite useful. So it means that it installs it in an editable fashion. So when you edit the source code, the live version is immediately updated. Really it's not like it's the live version's updated, but the live version is linked directly to the source repository. So it's useful when you're developing things. So instead of copying the folder into the installation location, it will create a symbolic link between the two. That's even still. That I use that all the time when I'm working on a project. Yeah, that's a super useful command. If you're developing a package, then as a developer, you install it like this and then you can just work. Yeah. There's a question, can I install a package in a separate directory, not in the central location? And that's the environments that we did at the beginning. So these days, everything I do is in virtual environments. And I even have a shell alias for installing it. Yes, that's the best way to do that. Yeah. Yeah, so do you want to bring all the instructors here and see if we can get some debates going? Sure, let's do that.