 Thank you all for coming today. My name is Randy Barler. I work on the Fedora infrastructure team at Red Hat. I focus a lot on Bode, which is our update system. And today, we're going to talk about Python 3's AsyncIO library, which is a pretty neat library. I really like it. Just in case you're interested in following along on your own computer, this presentation is on GitLab. So I've got a link here. So feel free to go there and check it out. All the files, I'm basically not using slides. I'm going to be showing you just code. So all the code I'm showing you is inside there, and you can check it out and play with it on your own. And I welcome questions during the presentation. So if you feel like you're not understanding something, or if you think I'm not understanding something, feel free to speak up. So I've been thinking about this presentation for a little while. And one thing I wanted to do was sort of create a demo program that shows off what AsyncIO can do. And so I was trying to think through what kind of toy programs I could write. And you can do all kinds of things with sleep and prints and things. And I decided, oh, I had a problem, actually. My phone will not play MP3s right now. I really don't know why. It's an Android phone. If you push play, it doesn't make any sound. It says it's playing, but you don't hear anything. I don't know why. And so I like to listen to RSS, my podcast when I fly on a plane, and I want to download all these MP3s, but they won't play. So I decided to make my toy program for this talk be where we're going to do is we're going to go download my RSS feed. We're not really going to do that because of the conference Wi-Fi, so we're going to fake download it. And then the things we need to then convert all the MP3s to a different format so my phone will play them. So what I'm going to show you first is how you might do that in sort of a traditional synchronous code. So I have this script here pretty straightforward. I'm keeping the links in a file called links.txt. So the first thing we do is we go and just get the links out of the file. It's pretty simple. And then what we're going to do is we're going to do a giant for loop. This might be, is it better if I make it a little smaller? Is that better? OK. So we're going to go over each link. And we're going to start by downloading the link. We're going to start by printing that we're downloading the links. And then we're going to use Python requests and get it and make sure it was OK. I'm going to do some path formatting to find out what the file name was so I can write it out to that name of file. Going to print some more things to the screen. And then lastly, I'm going to try to find out the base of the name so I can tell ffmpeg to convert my mp3 to an org. You can see here that I'm calling subprocess. And then lastly, I'm actually going to have the script play the file so that you can hear it and make sure it works, because sometimes it's weird. And that's it. So we're just looping over the files, downloading them, converting them, and then playing them, and printing out some status. Feel like we're all pretty familiar with this code, right? Like it's not hard to follow. Any questions? OK. So I have a question for you, which is, what problems do you see with this approach? It's synchronous, so what is that? Yeah, so you have to wait for one download to finish before starting the next one. What else? Exactly. So you don't just wait to download one to start the next, but you also have to wait to process the one you just downloaded before you start doing the next one. And similarly, you have to also listen to it. Fortunately, I am only listening to two seconds of it. That's what this dash dash length two does. But still, yeah. Any other problems anyone sees? Say that again? Oh, right. OK. So yeah, if you hit an exception, in fact, I think we're going to raise an exception at one point. So if this happens right here, as soon as you hit that, the whole thing is going to exit. No, we could fix that by having something catch that exception. But I was trying not to make it too complicated for the demo. But yeah. Anything else? Well, actually, I'm going to listen to it. That's true. I think it would, I guess it probably would exit with an exit code. And I am doing check call. I actually did have some problems. I have a problem with MPV that I'll talk about later. But it isn't what it isn't that. I think you're right, though. So yeah, I guess I didn't have to listen to things, but actually did. Yeah. The reason I'm playing it is that I'm going to show a more complex form of this later, and playing it does something funny. OK. Well, I don't know of any other problems, any others that you guys can think of. Oh, OK. Oh, I didn't know that the microphone's not on. Can you? The microphone is that. I only open the notebook. Do I sound better now? OK. I didn't realize that it wasn't on. OK. OK, so before we, so I want you to keep that downloading thing in the back of your mind. So what we're going to do next is we're going to look at an extremely simple asynchronous program that has nothing to do with that. And then we're going to come back to that problem and make it asynchronous. OK. So this is a very simple asynchronous program. Do I have a pointer? Is this a pointer? Maybe? Oh. OK. So in Python 3.5, you get these new keywords, async and await. It is possible to do things like this in Python 3.4 as well, but this syntax is different. And I don't really intend to show that today. But just for your information, it is possible to do things like this in Python 3.4 as well. But the code that I'm showing here requires Python 3.5 or later. So what we're going to do here is we've got a function, an asynchronous function, called say, except a word. And it's going to wait for a random amount of time between zero and two and a half seconds. And then we're going to sleep. Then we're going to print that word. And I kind of want to explain a little bit about what these keywords are doing here, and then we'll show it run. So when you use this async in front of a function definition, what you're doing is you're creating something called a co-routine in Python. And a co-routine is sort of strange. When you run the co-routine, so if I write say, parentheses, and a word, it doesn't actually run right now. It returns a co-routine. And that co-routine can be handed to something to run later. And you can see us doing that down here. So our main function is going to build a list of tasks. And this is going to call say with a word for each word and let's make it asynchronous. At this point, we're not executing that function, even the traditional Python that looks like I'm calling it. But what I'm really doing is creating a co-routine that we'll call it later. Then what I'm going to do down here is ask async.io to this gather the tasks. That is a way to tell async.io that you want to run this group of things all at once. Just start it all off. Now to clarify, we're not working in true parallel here. We're just working asynchronously. And the difference is important. Because we still have the gil. We're still in Python. So we'll talk more about that in a second. But we're basically telling async.io to just do them all. That's true. Yeah. So this function returns something called a future. And a future is a object that you can await. And eventually, in the future, it will do something. So when we call this line, that's when all the tasks are going to run. And then the last part is that we're going to use async.io has this dot run function. And you can just hand it an asynchronous method. So that's what this line here is just essentially running our main program. And I'll show you how that works. And if you run it every time, if you remember, we were sort of waiting a different amount of time between the sleeps. So if you run it over and over, you'll see them in different orders. And actually, when I was playing with this earlier, I did it in order one time. And oh, there you go. So that's kind of fun. I think maybe about 1 in 24, you should expect to be in order. One thing I wanted to do, though, was put some more print statements in here to sort of show what is happening when. So we get some sense that, to remind you, this is not parallel, when we enter this function and we hit this await, we're actually handing the process control back to Python. And in this case, we're calling a sleep. So when the sleep happens, Python's like, OK, I'm going to go do something else. Because I'm not going to just sit here and block. So whenever you hit these awaits, Python will go to the next thing in your list of tasks. And it'll run the same. In this case, it'll just hit the next await. And so you're going to hit all four awaits before any of them finish. And then it's just a matter of which one slept longer, because we have that randomness. So to demonstrate that, I'm going to stick some print statements. And you know what's going to be funny is I'm definitely going to forget to put parentheses around the print statement, even though this is a talk about Python 3, because I've been a Python 2 programmer for like, I don't know, 12 or 15 years or something. And I cannot get used to that. It's so hard. OK, let's see. Let's actually do this. Word to sleep for live typing in front of people. That's great. OK, so we'll put the word and the wait time. I remembered the parentheses. That's good. And then we can, well, we could put a print there, but there already is one. So OK, let's run that. And what's really interesting here is that you can see that those new print statements we put all happen first, because each one of them runs until it hits the await, and then the control goes back to Python, and then Python just runs the next one. The other thing that you might notice is that it's always going to print those four in the order we told, right? So we said, let's make it asynchronous. We called it in that order. And each time you can see us hitting it in that order, but then it sleeps for different amounts of times. It finishes in a different order. So that's sort of the key there. Are there any questions at this point about anything that doesn't seem clear or that I could explain better? Yeah, the question is, do I have to use the sync for the main here? I think you do. I'm not actually 100% sure. You can clone the repo and try it. Oh, I guess we could try it right now. Let's try that. Let's do an experiment. I think you do, because I think the run expects a coroutine. However, due to my confidence, we can see what happens. Yeah. Yeah, so if you don't have that, you're going to get this exception that, well, if you remove the await, then I don't think you can run async.io.gatherer without waiting on it. What are your suggestions? Oh, I see. So the suggestion is I could kill main and do this, do this, and do this. That actually seems like it could work. What am I doing? OK. That's what happens when you confuse Vim's paste with your, OK. I have a feeling this will work, actually. Let's see. Oh, my. I suppose that the gather does not return exactly what that's expecting. So we're beyond the limits of my knowledge here. But when you read examples online, you do sort of see that first form that I was showing. And of course, I have now exited, so I can't undo and go back to it. That's fine. Perhaps. Yeah. No, because you can await co-routines, too. You cannot. Run a future. Yeah, you cannot run a future. Yeah, that seems. Once you run a co-routine, you get a future. Yeah. And then you await your future. Yeah, that's right. That's right. You look from the morning. Yeah, yeah, yeah. So you and future is a state of thought. Yeah, yeah. So this is actually highlighting some really fun things about all of this, is that it is like a really weird, different world. And as you go, you kind of run into a lot of problems like this. It actually took me a long time of trying things. And you sometimes end up realizing that you're awaiting the wrong thing than what you think you're awaiting. And then your program doesn't run, or you end up with deadlocks. So it's a lot of times it's just kind of experimental, like we were doing here. Like you just try things and try to learn that way. Does that mean five minutes left? That's fine. OK. Yeah, that just goes away. OK, cool. You have some space here. OK. I feel like, OK. Well, let's move on, because with only five minutes left. Yeah. OK. So this is a simple form of that. Remember the problem where we're going to download my MP3s and try to turn them into Augs? So I wanted to show you a simple form of that that uses imports that aren't used in the file. So you can ignore most of those. If you remember, we start off by reading our links. This is exactly the same. I'm going to introduce this new function called download MP3. It uses one non-standard library, or uses a library that's a third party called AIO-HTTP. It's sort of like requests, except that it's a synchronous. And you can see I'm creating a session here with it. Wonder, is it better if I make it a little smaller so you can see more code? Is that still readable? Yeah, OK. So we have this download MP3. And then we have our main, which this time I'm not going to try to change. And the main looks just like that last thing we saw, right? We're creating some tasks with our links, our coroutines. And then we're going to ask async.io.gather, which basically just says, kick them all off, do it. So let's focus on the download MP3 function. It's pretty similar to our first code at the beginning. So we're trying to find out what file name it's going to be called. This AIO-HTTP, like I said, is a lot like Python requests. It is not identical, actually. So it's not a drop in replacement. I can't remember. I think one really weird difference was this one has response.read to get the bytes. But the other one is response.content. So you can't just drop it in. But it is kind of a similar idea. It's a nice library. So we're going to build a session. And notice that we do that asynchronously. So even creating our session is asynchronous. And here, we're going to ask it to get. And I think all that really does is start the connection. And next, we're going to print that we're downloading it. And then we're going to open our file. And then we're going to do this. And this is the line that it's going to wait on. So as you download the file, it's going to hit that line. And it's going to sit for a long time while you download these files. 9 to 100 megabytes each. So we can't really do that on the Wi-Fi. And then lastly, it just prints out some statuses, saying that it saved it, and so forth. Any questions about this before I bring in some more complication to it? Oh, OK. What does a sync with do? So we're all pretty familiar with the with statement. So this is an asynchronous function when we're creating the session. But we often use the with statement to clean up things when you're done. So this is a way to combine the concepts of being asynchronous and also having a way to automatically clean up when you're done. And so we want to clean up the session. And we want to clean up the call to get. Because later, all we do is ask it for the read. But that doesn't mean it has closed the connection. So I think this ensures that we close our connection and then close the session as well. Yeah, the question is, what is the client session doing? And why is it asynchronous? I don't know. That may be a detail with how AIO-HDP works. I looked at their docs for only a couple of minutes to write this. And then it worked. And then I was like, their docs sort of showed this way. So to be honest, I don't actually know how they're doing it internally or why they want to do it this way. I could only really guess. Yeah, presumably there's something about creating a session that takes some time or can be done in the background. Yeah, I would probably recommend maybe that their docs explain this. I really didn't read their docs because Jeremy Klein doesn't like when you read docs. I didn't want to disappoint him. He's always on my case about not reading the docs. Yes? Yes, so to summarize, it seems that we're handing control back to Python while some kind of creation happens here, some kind of initialization. And when all that's done, it's going to come back to us. And then we can proceed. Jeremy? I did something asynchronous inside of that. The width would exit before the future event had been completed. So you're downloading something. That file hasn't even been able to get it. Right. So you have to say, return a future. And then it changed a bunch of callbacks with the final one being closed after. Yeah, so to summarize, Jeremy Klein's comment, if we didn't have asynchronous theorem, we did it with, we could run through this whole thing because there's a, we've run through the whole thing before we even downloaded any file. So this is sort of a way to help us wait and sort of chain things, including the cleanup. You had a thought? I actually, I just know why there is, I don't know why this particular particular function is usually asynchronous, but I know the difference between asynchronous and normal. Because I think in this case, it could be, even use normal, normal, normal with coroutine. But the, I think it works that way that the, when you have a complex manager, you have basically two special methods. One is a enter, and one is exit. Enter is generated in the start of the context and exit after the context, even if there was exception raised. But you cannot use, normally you cannot use coroutine in there. And the asynchronous created that you could use coroutines as an asynchronous. Okay. To summarize the comment is that they sync with allows you to have an enter and exit for your context manager be asynchronous. And so it's a way, it's a way to have a schedule the enter and exit on your context manager. I may be out of time. Okay. Some people maybe want to switch to another session. Yeah, if people want to go to another session, that's fine. Okay. Okay. Well, I, if you're interested, I encourage you to look at my code because I actually had a lot more. The next code was that it actually processes the MP3 and then actually plays it. And it actually also limits how many can go at once. And apparently it's way too much to do in 30 minutes. So thank you all for attending. And if you want to look online, I've got some more code than this. And you can play with it and try it out yourself. Yeah, thank you.