 Hello, hello, everybody. Thank you for being here. My name is Maria and I am a software engineer at sky scanner And today I'll talk about the synchronous programming and how it works in modern Python basically the idea of this talk is to Actually revisit the male mindstones that have been happening in versions of Python that arrived to The current state of the art in in our programming language Basically, they have been digging through some old peps in order to understand what had been the main cornerstones With the idea of not just exploring the API because probably you already know that or you know how to use the functions But maybe you're interested in knowing what's behind it or how things work or you're curious about What's internally free so this is a rather long journey in terms of time because it starts with something that Came out about a very long time ago with generators in the year 2001 in a very old version of Python 2.2 And the idea was back in day to support a more efficient way of computation like for instance They were created with the idea of lazy iteration in mind. So let's say you have to process large Bunch of objects. Let's say like a million numbers or a trillion numbers and one way of doing so you say I can put Everything into a list and then process the list and that would work But you will take a lot of memory. So or the same goes for any object So the idea was to okay Let's provide a mechanism in Python to be able to generate Only the object that I need in the given point in time and no more than that and save a lot of memory So the the point of this is that there were The idea is okay We say memory and we support like the iteration pattern and at the point in time the generator is going to give us The value we need for for that particular moment and it's going to suspend there And it's going to be there until we call it again with the next value that we need so a new keyword was introduced the Jill and This works like with two very simple concepts. You have any function But if that has function happens to have a Jill Statement in any part of its body is going to be a generator function Which means that when you call it Whatever it is on the body of function, it's not going to run instead. It's going to give you a generator object So they work like a factory when you call it you receive and a generator that was just constructed And then you start working with that object Advancing the element one at a time at it next point You get the element the generator is going to suspend there It's going to be like frozen waiting to be called again With the next value and there is nothing else to produce because it got exhausted and in that case is top iteration Exception is raised Which is the mechanism to signal that that iteration is over So now let's say that time after we want to support coroutines in Python and I'm not giving like an academic or like a formal definition of coroutines But just for the purpose of this talk we are going to have a more pragmatic understanding of coroutines. It will be something like You're able to suspend at any point in time But you also want to be able to resume at a later point in the program to be able to continue the execution with different values So if we were to implement this in Python like and we start to think okay, how can we have this working? Do we have to Let's say start from from scratch and implemented everything on top or is there something that we can already take advantage of? And the reason why I started with which generators is because okay They have something that we can use for advantage, which is the point of suspension Remember that a generator is suspended at the point in time So the interpreter is frozen in a part of the code and then it can be resumed later on So this I this step came along this is coroutine by enhanced generators because okay There is something that we can take advantage of so it's not that we have to do everything again So let's recap how far can we go with with generators with simplicity can we suspend? Yes, that's actually that was the whole point of using them in the first place But can we pass some data to the generator wise is suspended? I mean in its simple form like in the very basic form No, that's not possible What about exceptions come can we send an exception to a generator wines is suspended to signal that something has Going on in the program No, that was not possible So that's why the enhanced part came along and the interface of generators needed to be extended So more methods were added To fulfill their purpose for instance the same method with and the throw through handled exceptions And close to some cleanup in the in the coroutine or the generator But technically that's just a particular case of throw because it just handles an exception Which is generator exit, but the most important is is send because it's what changes the the semantics a little bit so now let's Stop for a second in this concept because okay We have generators and we start to think in coroutines because we can't create something that we are able to suspend and interact with And a point in time in the program So synthetically there are difference that hasn't been any changes in coroutines They they still use the gild keyword as they as the old generators did in the very beginning But semantically they are different because and I will come up with this later in a few slides But just keep in mind that even though they're they're written the same and they're even technically the same thing Conceptually they're they're meant with completely different purposes And this will also be an important step later for now Let's say how we can interact with coroutines at least in a very basic form because we're still in in a very old version of Python This is still to 2005 Python 2.5 and Now Jill not only produces values to the color, but you also can get some value as a result. So Now Jill for example result is what the color is going to get when you call the the generator or coroutine and Value in this case the variable value is going to be whatever was passed from the outside to the coroutine It's going to be captured. So I prepared a small example just to illustrate this Let's say how a coroutine that only iterates and it's counting some steps and it's printing whatever gots received So I want to create that coroutine and start sending values just to see how this now Jill works in both ways so the first thing I do is I will create the Coroutine it's post for a second wait so Once I created remember it gives me a generator object because it's like a factory And I call next the first time this is an important detail because if I don't call next I'm not able to send anything to it Or I will get a type error this is because the next remember that it moves the Execution of the coroutine or generator ups up to the next Jill statement So it needs to post there so I can send the first element, okay, so now the program of dance up to that point where it's Waiting for something and look at that line The the one is like you can see you can think like that line is split in like in two ways and Only half that statement Run the part that give me the value zero because it yielded the first step which is started at zero So now I can start interacting with the generator and let's say that I will send I know 100 And the second part of that statement the one that works in words Let's say towards from the outside to the generator is going to run now receive It's going to have the value 100 because it's how is and then the rest of the program is going to run So it's going to increment the step is going to print whatever I just sent which is the value 100 It's going to circle back to the beginning of the loop Where the last to the true statement and then it's going to find the next yield when it's going to pause again and suspend there I'm telling you do something about it and it yielded the Step now one and that becomes the result of send same as so it's it's similar to next actually now next is a particular case Case of send so calling next on a generator is like sending none But we are the idea that it's about works compatible and now we have the semantics And even if this is a deliberate like simple example just to illustrate how it works It marks the beginning of coroutines because now that we can suspend and interact with the coroutine We can do some asynchronous programming. We can suspend and do some non-blocking IO operation while Signaling the program that that that it's waiting and there's something else that can be run So Okay, and now if I send or if I throw an exception you can of course get what whatever is happened the exception is going to be thrown at the point where the Coroutine is suspended and in this case is going to fail Because it's not handled but you can add as part of your logic to handle the exception of course So no, okay This is great, but let's see if we can do we can do better because this is not entirely convenient And what if we want to have larger programs or we want to refactor coroutines? so This pep came along 380 which actually now we're in python 3 So we're getting closer to the current API. I say it's like six syntax for Delegated to a sub generators and this is actually quite important because it marks like two important Improvement first now generators can return values Which before it wasn't possible You could we have a return statement is a generator because it will be a syntax error And now is the Jill from syntax was introduced which will explain the semantics and how both things are related One has to do with the other So let's say I have another Simple generator or coroutine that only produces two values and I call it the first time I got one of course then second time I got the value two and then there's nothing else to produce and remember that stop iteration Was the exception that was signaling that that generator was exhausted that there was nothing else to produce Okay Now the same happens, but the return value comes in the exception because exception Are just Python objects after all and we can set any attributes to objects because they're dynamically added So the return value will come in the value attribute of the stop iteration exception And that way whoever is calling that coroutine can know what's not only that it finished and it completed But with with which value it completed So now we can say like Jill from Like in a very most simple form like whenever you have an interval And you go you'll Jill from that interval and then you you can do a for loop and then Jill every element of the interval and that would work But that is not the actual reason why they was introduced in Python There was actually much much better use of that and has to do with the true previous points like first this the syntax is going to Allow you to chain coroutines So remember the example I just gave when you send values or you throw exceptions to coroutine Well, you can do it that with multiple levels. So let's say like I have a Coroutine that calls another one that calls another one The the mechanism works the same and now it's important to know why Coroutines can return values because I can capture the return value with this syntax into a variable And even if it doesn't look like to big deal It's actually quite interesting because if you look at the syntax It kind of resembles like a like a thread like you thought you have something running there It's not a thread of course technically it's not a thread But the point is you have something there running that you don't know which order is going to run You don't know how it's going to run at some point is going to suspend and continue and Some point in the program is going to stop there and it's going to give me a final value And I can have that in multiple multiple levels. So this starts to improve things quite a lot and this marks something that we are probably more familiar with which is We can have multiple of these children from different coroutines and do like different operations like for instance reading from a database or Doing an HTTP request something that is not blocking and what the actual I or the actual operation in the database Is going to take is is taking place The the program or some from part of the program can know that and Okay, say this part of the goal is suspended. I can run something else in the meantime and Share you the order So let's see this with an example Let's say and now I have a generator that I call general that goes another generator inside of that so it yields from that generator and again is a simple example just to illustrate the the mechanism but It works all the same with multiple levels. I mean these works at all levels So it just prints out the a loop Similar to the first example and when one part completeness it will continue with the next one So, okay, like I create my Coroutine like at the beginning and I call next to it because I want to start sending values And of course I get the first one but then I send something and remember that I'm sending that to the to the first coroutine to the general not to One of the internals and if you see the print message you'll see that What it was printed was the first one the first one go that that string So what happens is I send something to general general pass it to the internal coroutine It did the processing it printed the value it yielded something that went back to the general And that became the result of the of the sense time. So now we can have a more Sophisticated code we can like re-factor and have smaller coroutines with more fine fine-grained responsibility And of course if I keep iterating that this will keep it will continue until the first one stops and then the second one will take place and It all works like if it were just one coroutine. I will just put all the code in in a single function So this is actually quite convenient because actually this pep has an example of the code You will have to write if you didn't have this functionality Built-in in Python. So it's actually saved saving a lot of code and a lot of trouble as well Because there are a lot of exceptions that you have to handle that stop iteration and with different conditions, etc So just to recap this syntax allow us to create even more powerful coroutines and chain many iterables all together Well structuring our code in a more convenient way And with this we can start thinking about like event loops and some objects that are going to handle Coroutines, but that that is not all Because there are still more things to do. I mean even if this is a great improvement, it's not everything like Remember that I mentioned in the in the beginning that there was a When we first start to think about coroutines We wanted to have like we will be in top of generators We say like okay syntactically they're the same they use Jill the yield statement or Jill from so and actually technically The coroutine is a kind of generator but semantically they're different so Generators were created with the idea of iteration in mind and Supporting the iteration pattern while coroutines were created with the idea of asynchronous programming in mind or Suspending some part of the code or non preemptive computation. So those are two different Use cases or completely different computing scenarios that fall into the same technical detail, which is that Down the line in Python. There's a generator running there. So and also the Jill from syntax works with any intervals Generators are intervals, but they're Also, all the kind of intervals are not generators are not coroutines for instance strings or least So and you can do Jill from a generator or a coroutine and that will make sense But syntactically that's nothing prevents you or stops you for writing like something like Jill from a string And or a when you were expecting a coroutine in that place and that would fail and since Python is dynamically type You probably wouldn't notice the error until late in the program so a more time came passed and This was okay. We need to address these limitations and in more modern version of pythons Again the syntax change notice that since the very beginning in 2001 when the Jill was introduced There were no new keywords in Python and now We're in the version of Python field 5 And the year 2015 or 16 and new syntaxes we introduce all all that long time There was pretty much no changes syntactically even though there were no constructions and the idea is with this pep 492 We can create coroutines with its own syntax So when you instead of putting depth you put a sync depth you're creating a native coroutines before it's not just a kind of Generator that has a flag in the virtual machine now there in the pep that are called native coroutines And instead of Jill from you would use a weight and this addresses the semantic or the problems that we had in the previous Examples for instance Now you it's harder to make mistakes, which is a good thing Remember that before you there was nothing preventing you for you doing Jill from a string when you were expecting a Coroutine but now you cannot do a weight string accidentally because it will fail Python will will throw a type error saying okay a weight only works with Coroutines or eatable or awaitable objects, which are objects that implement the magic method for a weight So with with these syntax now things start to look more Like in in the last version of Python 3.7 or even in the future version of Python And now we can take a look. Okay. Yeah, what what I think I owe is is actually doing Because okay now we create our coroutines with a new syntax And I think I owe is just an event loop which is an object is the one comes in the standard library But you can use any or event loop Whatever it's going to happen. It's like okay We define our coroutines and typically our coroutines do delegate to some other third-party libraries Like for instance, let's say I use async pg to connect to the database and do await our select or or a query and What happens is our the event loop is going to call a coroutine at some point in time It's going to run their call our call is going to delegate to the third-party library Which down the line is going to implement? The IO at that point when we call a weight It's going to return something to the band loop So that way it signals the band loop and it is the way the event loop knows that that coroutine is Suspended is waiting for the IO operation to complete So another one will take place and we'll run and the same will happen again and again and again And at some at all points the coroutine is just going to be updating the status of the Sorry the event loop is going to be updating the status of the coroutine So you're going to call send or throw respect Respectively as it happens with the exceptions in order so our coroutine can know okay I have to act with this value or I have to respond to this exception, etc So down the line there's nothing like fundamentally new or different that it's that is going on in Python The idea is like okay technically coroutines are still some type of Generators which this is for historical and technical reasons remember that semantically they are different and Things that look to be like completely different or completely new or different stuff are actually More simple Constructions that we have been using in Python for quite a long time and it did for a very old Time remember that for it for instance like every time we call away Well, it will call something that down the line it will end up at the Jill statement Just as a way of signaling that that has to suspend So this is probably the way I will I wanted to show the the or explain the way a synchronous programming works because it's not like Perhaps under understanding what's behind or demystifying what was actually Python doing every time we program synchronously in our program That's pretty much it. I think you enjoy. I really hope you enjoy the talk and the rest of the conference as well Thank you very much for listening We have three minutes for questions if you have any