 Hello, we are back. So next up is one quick talk library ecosystem where, well, it'll be a bit shorter than what's on the schedule, but the basic idea is we're looking at the big picture of like what we're doing here in the course and preparing for tomorrow. So in HackMD, there's two initial questions here. So what other Python packages do you use to write the name and a short description of them and then has anyone else ever used your work and will come back to this in a little bit? And in the meantime, I will switch to the course page, my screen. So here we go to library ecosystem. So there's this good quote I had from someone I was asking people, what do you know now about a computational research or scientific computing that you wish someone had told you when you first started? And the student answered, okay, so when you're a student like undergraduate student, you're expected to do everything yourself and it's how you evaluated. And if you reuse something, someone else done that's plagiarism, when you become a researcher, you have to be able to reuse what others have done. Otherwise you just can't do your work and you don't have much practice in this. So that's sort of the theme of this short talk here. So this whole ecosystem of different packages. So there's maybe two different types of things that you might use. One is well-maintained libraries that are used by others. So say NumPy, SciPy, Jupyter, these kinds of things. These you can really expect that they work. And then there's a whole bunch of other code which is written by others maybe for an article or something like that, but it's not really well-maintained and who knows if it even works well. So yeah, there's some common terms that you might have been hearing here. So library is a collection of code used by a program. If it's a package, then it's been made easily installable and reusable and is often published on things like the Python package index. So this is what's used by pip install to install things. Another public repository is the Anaconda cloud which has things like ContaForge. So when you do Conta install, it takes from there. And a dependency is a requirement of another program not included in that program. So actually tomorrow we talk about turning your code into packages. And also we talk about handling dependencies. So I believe at least Simo's talking about the handling dependencies and I forget who's talking about packaging. But those are actually pretty useful things to do even if you're just working on code yourself. So having it be a little bit less chaotic is really useful. So make sure you come back for that. Let's talk about the bigger ecosystem. So when people say sci-pi ecosystem, there is one big library called sci-pi and that wraps a bunch of, what was it that it wraps? GNU scientific library or? Well, a huge host of libraries. There's like a plus libraries for linear algebra, there's FFT libraries for Fourier transforms and all of the functions that use these kinds of things. So if you want to do a convolution or something like that underneath it, it uses FFT to create this thing. So all of the mathematical functionality uses some lower level libraries to do the calculations. And here, this is what we could call sort of the core packages of the sci-pi ecosystem. And it's called sci-pi ecosystem even though sci-pi is just one of the packages in it. Of course, first there's Python, there's NumPy that has the arrays, there's sci-pi that builds on NumPy. So NumPy holds the arrays and sci-pi has functions that work on those arrays. Map.lib does the plotting and then there's a lot of other plotting things that build on that as the back end. There's Pandas, which is the data frame, data structure for tidy data, which uses NumPy. And then there's IPython and Jupyter for interactive work. And I would want to say that you cannot stress enough how important this is for scientific computing and Python because if you think about Python, Python was invented in 1991, so over 30 years ago. So when people think of, okay, Python is a new thing or something, well, nowadays probably don't, but 10 years ago people were like, okay, Python is like an upcoming thing that's conquering the world. But it's really these packages that have enabled it to conquer the world. Like in the 90s, people weren't using Python for scientific computing because they weren't effective packages for this. But nowadays, because of these packages enable one to use Python as this kind of collect them all things. SwissArminite. Yeah, SwissArminite, definitely. Toolbox that you can use to do all kinds of things. That is really the core thing about Python and scientific computing. So I guess you can say Python is bad for scientific computing, but it has good libraries that make it good. Yeah, I would actually nowadays even add TensorFlow slash PyTorch to this list. Because so much stuff is now going on with machine learning that this is almost, yeah. This should, in my opinion, should be added. Yeah. That ecosystem list. Yeah. Okay, but I still think that's more special purpose than these, which is across every domain. But okay, let's go on. So there's a huge list of different libraries here. So our point isn't to read all of these things because you can read later just as well as I can read now. But it just goes to show that, like how big the different breadth of things are here. So yeah, but let's get to this interesting part here, which is connecting Python to other languages. So what, the way things like SciPy and NumPy work, they're not actually written in the Python that you know. They're written in C that uses a Python interface. And if you go and you search the Python docs for the C API, that's what these things really do. And through the C API, they can link to other languages like Fortran or whatever else when there's existing stuff in those languages. So we don't need to go into detail there, but there's two main terms you might hear sometime. One is extending Python. And that's writing your own modules in C or some language which Python can import. And the other is embedding Python, which is you have some other application in some like C or whatever. And you have Python go inside of that where your other application is the primary thing. So then Python becomes like a scripting language inside of your program. But we don't need to go any more details there. Just realize that it is there. There are also like these kinds of like very popular projects such as Python mentioned there as well. And Namba, for example, that basically like you've write code that reminds you of Python, it's similar to Python, but underneath it converts all of the Python stuff into C and then it runs it on C. And that's usually much faster, but then you also have to like deal with the added complication of compilation and stuff like that that comes with the static languages such as C. Yeah. Yeah. So basically if you do have some library written in another language, there's actually pretty easy ways to use it to extend Python without even having to know the Python C API. So the last part of this is evaluating packages for use. And there's some things you can consider here. Like is it actually released or is it code on someone's website? Are the dependencies actually recorded? Is the code stable enough? Like are the authors randomly changing it while you're using it? Are there automated tests? I'm sorry. And this is actually many of these things are considered in a code refinery course. Which we should think here, but it basically teaches you like how to make your code so that it's usable by others. Oh, actually here's the link to code refinery. But before you go taking some random code you see somewhere, it's worth thinking about this and understanding what the risk might be. Yeah. So I've already, well, this first one here, let's see what people have written in HackMD. And now about the risk, I will quickly mention that there's like nowadays, especially if you're working with something that goes into the internet, like something that connects to the internet, if you create a web API or something, there is a growing tendency in software world of these kinds of like attack by these kinds of like packages basically, like that somebody has access to some package and then they can bring systems down because they install a version of the package that has some vulnerability in it, or like they can do stuff like that. So usually you don't have to think about it when you're doing scientific computing, but still it's good idea to like remind yourself that whenever you're using something created by others, you are trusting them and you're extending your trust to them that their code does what it says it does. And of course it's in most cases, you can trust like science is all about trust, we trust our researchers to be good faith when they publish a research or something like that. We trust the publishers, we trust software creators, but you need to sometimes very far or be certain about it of which. So I wouldn't install a random name, get a package without like checking who is the author or what is inside the package and doing stuff like that. Yeah, so let's see. There's some good example packages here. Yeah, by the way, Tamu, if you're listening, we're almost ready for web APIs to start. Yeah, work X I've used a lot. I'll also mention here that like they are usually more than one package for one application. They might be like somebody has created a package, that package doesn't have a functionality that somebody else wants and then they create a fork of the original package or they create their own implementation of the thing. And it's usually not bad to have multiple things doing the same thing. Like a competition breeds creativity usually. So and using one of them doesn't make you like worse person if you choose the wrong option or something like that. If you back the wrong horse, that happens over time. If there is a wrong option. Yes, yes. That I think most of the times there is not really a wrong option there might be the non for this instant not really best option, but it's not really a wrong option as such. Yes, often. Yeah, like usually the main focus should be on getting what you want done, done. And in that sense, like even if you get it done using something that is not the most elegant or beautiful or like some perfect way might be better than just creating your own package or your own thing for it. Yeah. So I don't see Tamu here yet. We're ready to go to the next one. I have an idea. Let's look at this evaluating packages exercise together on my screen. So here we've got six different packages. And I propose we go in order from least good example or Yeah. So I'll open them in some order and quickly comment what you think of how they look. So first is this one, which intently I wrote long ago. Would you use this for anything? Like, what's your first impression and why? Honestly, my first impression would be that I there's no read me file. I necessarily wouldn't like, like know what's happening over here. So I would probably like look for the like basically no read me file. If I pointed to it by someone, that this is a good thing to use for this specific purpose, then I would probably head back to them and tell them, okay, if you can explain to me exactly how I have to install it after I use it and so then I might use it. But yeah, there is simply too little information too. Yeah. And also the time, like last update, last commit reveals that it's not necessarily like a leaving package anymore. So I can tell you really, well, there's some good stuff there, but you wouldn't want to use it. I wouldn't. Yeah. I wouldn't. So I guess that the underneath, like there's bound to be something. Here's another one I wrote. So at least there's a read me file now and look a release and a citation, but it's not packaged. It's not installable. I mean, if I knew I needed it, I might try to figure it out, but otherwise I would go on. Okay. Let's look at something that's good. Oh, here we go. So I see it's relatively more active. There's a read me file, even just looking at the table of contents, who's written it, requirements, how to install it. Okay. Look, I can install it from PIP, how it's used. I mean, this is already looking quite good, but okay, it's time to move on. So hopefully this was an interesting observation of things. So