 Hello, everyone. Our next speaker is John Sutherland, and he's going to take us on a tour of iteration on Python. Please give him a big hand. Thank you. Can you hear me okay? Okay, good. Good afternoon. Thanks for coming. I realize it's late on a Friday of a long week, and thanks for coming. So this is iteration, iteration, iteration. This is my name. Pretty good for a slide, I think. This is my internet name, where you can find me on Twitter, GitHub, Telegram, and I'll be tweeting links to the slides afterwards. I work at a place called Fanjul in Edinburgh, Scotland, where we make an online daily fancy sports thing, but I'm not going to be talking about that. As I already mentioned, this is iteration, iteration, iteration. So most of programming is looks, conditionals, and a few other bits and pieces, and everything else is kind of nice to have on top of those two things. And so what we're going to cover this afternoon is or what we're going to look at this afternoon is some basic loops recursion, look at reduce, iterators, iter tools, list comprehension, generate expressions, generators, coroutines, and then at the very end some stuff that you really shouldn't do. Unplug it and plug it in again? Okay. So we're going to use a contrived example of factorial for the first few examples. Just to demonstrate some of the basics, it's pretty boring but reasonably well understood, hopefully you can you'll be able to see the differences between the different methods. So for anyone who's unsure, can't remember from their high school maths classes, four factorial, so four bang here is four times three times two times one. So the while loop, so Python has while loops, which you don't see that often unless you want to create infinite loops, which I do all the time for fun. And here's factorial with a while loop. So we can see here the iteration state, which is the variable n, the calculation and the actual looping itself are all kind of bundled together in these three lines. There's quite a lot going on here, and you would never really write this, I don't think, not like this anyway. This is Python and we can do better. We had the for loop. So the for loop version of factorial is something like this. So we would iterate through a range from n to one. So in this case, we're actually going backwards with the minus one on the end there. And every time we're going to multiply our running our accumulator fact by i and then return that at the end. And this is nice. This is probably the best way to do this for this case. But the problem is our community logic and our iteration variable are all hooked up together. So we could do it with a recursion. So some of you may have noticed that four factorial is actually four times three factorial. So the definition of factorial is recursive. Sorry, and in the general case n factorial is n times n minus one factorial. And we can see that quite nicely in this code. So here factorial, the definition of factorial includes a call to factorial, which is recursion. So the if here is our escape clause, and this prevents us from looping or recurring forever. For larger values of n as well, we could blow up the stack and we'll get a runtime error. So we should probably use the for loop version rather than this recursion. And the last of our factorial examples is with reduce. So what we're going to do with reduce is we're going to create the numbers one to four and then we're going to use the reduce function and the multiply operator to collapse our list of one to four or one to n into a single value. And so since Python three reduced moved into func tools, but previous, it was a built-in. So yeah, and this works. And this is reasonably clever, but where I work and certainly if I was code reviewing, I would you begin to thumbs down on that one or a nice suggestion. Would you mind changing this to a for loop? So slightly more interesting than factorial are iterators. And iterators are created by calling itter on an iterable. And a lot of this was quite confusing and quite complicated, but hopefully we'll be able to follow it through. And so iterables implement one of two protocols, the iteration protocol, which is the dunder itter method, or the sequence protocol, which is dunder get item. And we're passing in integers starting at zero. And then iterators not iterables, iterators implement the iterator protocol, which is dunder next. So here we create the iterator of a list two, three, five into our iterator. And then we just successive calls to next yield the values two, three, five. And if we call next on an exhausted iterator, then we get a stop iteration at the end. In practice though, you would never actually need to catch that stop iteration because you would use this in a for loop. So this is a little bit more useful or a little bit more common. But actually just to go full steam ahead with the trivial examples. Here it's an iterable because it implements dunder item or dunder itter, sorry. And it's also an iterator because it implements dunder next. So iter tools, there's lots of cool things in iter tools. And you should have a look at them yourself. Just have a look in the docs, they're reasonably good. One of them is itertools.count, which just generates a forever sequence starting at zero. Or the value that we pass in. So it's a little bit like range, but instead of a stop value, we just have a start value. And it will go on forever. And we can also pass in the stop value and the step value as well. This basically goes on forever. A couple of the other ones in there, I slice, which we can pass a value and it will slice an iterator down to that length. And it's kind of like the slice operator with lists or tuples. Count, which we just saw there. And then cycle, which is reasonably interesting. So cycle will repeat the values in the iterator pass to it. So here we can see ABC is passed into cycle and we get the values ABC and then A again at the end. And it will continue ABC, ABC, ABC for eternity as far as I can tell. And in Python three, some of the built-ins like zip take and return iterators rather than previously where zip returned a list. Let's comprehendions. I decided to choose a really big font size for these title slides. And comprehensions didn't fit on the slide, which is why I've hyphenated it like this. I think it looks okay though. So let's comprehendions are a nice way to create lists from an existing iterator or iterable. And they're also what made me fall in love with Python. I mean, I'd seen a lot of this stuff before, but when I saw this comprehension, I thought, this is for me. So here we're generating a new list. So imagine we have some people that we pulled out of a database with Django or M or something similar like that. And we just want to get the people's names. So we're going to call P dot name for every P in people. And we get the names Ro, John, Lucy and Tom. We can also add an if clause to Alice comprehension. So here we're going to get the names of people who are not bosses. So my house, this is me and my son Tom. And finally, with list comprehensions, we can have multiple for loops, which are sort of multiple f's, which are, they're nested. And so here we can see on the top line there, two, three, four, five, six, seven, that's one plus one, one plus two, one plus three, and so on. And the second line is two plus one, two plus two, two plus three. And so these are all the permutations of two dice rules. So generator expressions. Generator expressions look very similar to list comprehensions, but create generators. And generators are also iterators and iterable. This is where it gets a little bit confusing. So if we change the square brackets to run brackets, we get generator object. And we don't actually get the names, the values out until we iterate over the generator. So we can pass the generator to list. So in this case, we can actually remove the run brackets from the generator because we're just going to put them straight into list. And we can see the values out. So this is the same as our list comprehension. They're also useful in for loops if you don't need the intermediate list. So rather than going through all of the, rather than putting everything in a list and iterating over the list, we can just use a generator expression like this. In likelihood though, you would probably just have a for loop over people and then do something with p name in the loop. So generators are very similar to iterators and their generator expressions generate generators or create generators. So generators, they use the yield keyword and they give us for free a dunder, a turn, a dunder next. As well as a nice side effect of returning to the yield or resuming at the yield the next time we call next. So here's another Fibonacci example. So the first time we call next on a new Fibonacci, we'll come down to the A and we will get a value back. And then we can call next again and we'll get another A and another A and another A. So if we remember our previous Fibonacci example, which is all of this carry on, the generators is quite a bit nicer to write. And it works the same way. So we create a Fibonacci and we keep calling next and we get our values 1, 1, 2, 3, 5, 8, 13, 21, I think. We have to be careful with these as well as infinite iterators. If we pass them to list we can end up in an infinite loop, which is where we would use iteratools.i slice. Yield from is the other use of the yield keyword in Python or the second of the three I mentioned and possibly only three. So yield from allows us to delegate to a sub-iterator. So in this change in Python or this was yield from was added in Python 3.3. But before 3.3 this is a reasonably common pattern to see, which is 4x and x is yield x. And this is fine, but in Python 3.3 we can do this yield from, which is maybe slightly nicer. That's all there is to yield from really. So we've seen a bunch of ways to iterate, create iterables, iterators, generators. And coroutines are somewhat related, but they're a little bit off of a tangent. And they also use the yield keyword, which is why I've included them here as well. And they're useful for creating pipelines of operations. And the way that we do it is that we move the yield to the right, that's my right and that's your right, the right hand of the assignment. And this allows us to send values into the coroutine. So previously we were yielding values out of a generator, now we're sending values into a generator. So we can see yield here on the right hand side of the item equals assignment. And we're going to be able to send values in here. So the advantage of being lasting on a Friday is that I can add these things I've learned this week, like f-strings to my slides at two in the morning. Well, to this morning actually. So I decided to add them here. So the way we use this, we call next to do what's called priming the coroutine. So this just advances us to the first yield and then controls return back to where we are and then we can start sending the values in. So we're going to send in roll to see Tom and our printer is going to add the value 012 to the front of our strings. So here we send roll and we get 0 roll and Omegle. And this is the kind of, this is the best I could think of for how to visually represent a coroutine. We're going to call this kind of coroutine a sync. And we mentioned pipelines and we'll see how this sort of works shortly. So a slightly different version here where we can pass in a predicate function. So this will print any item where the predicate on that thing is true. So the easiest way to understand what's going on here is we're going to create a predicate where we're going to check if the second letter is U. And it's only this because of my examples. So we can see that roll doesn't pass so her name isn't printed, Lucy does pass so her name is printed and then again Tom doesn't pass so he's not printed either. The problem with this though is that we've sort of tied up our filtering logic and our printing logic. So once again filter printer is also a sync and these are useful but not super useful. So we'll skip over this because this is a little bit wild but basically what we're doing, oh no sorry, we're not going to skip over this. So here we're going to pass the sync into this coroutine and then every time we send a value into this coroutine we're going to send it onto the sync. So you can see this is sort of becoming more of a pipe and less of a sync. And I've added this coroutine decor at the top here. Here is defined but basically what we're doing is we're just sort of automatically priming it with that call to next and then we return the primed coroutine. And there are reasons that you don't necessarily want to do this if you've got a lot of setup at the start of your coroutine. But for the rest of the slides this is a kind of useful thing to have and it removes some of the boilerplate that we'll see shortly. There is also a coroutine decorator in the async.io module but that's different to this and it does different things. So our filter that we've just seen is now a pipe rather than a sync. So there are now two arrows on either side. And how we use it is we can create a printer sync at the top there. We have this filter pipe which our predicate is to print people who can drive and we pass our printer into taxi drivers there. And then for people we're going to send all the people to our taxi drivers filter pipe which will then send them onto our printer pipe. So we can see here the two people that can drive are Roland John. So my wife and I can drive. So this is slightly more useful. We can maybe start to see how this might become useful. And it looks like this. So we now have a small pipeline so we can send values in on the left hand side there. And they come through the filter and then maybe come through the printer at the end. And we could start to build up some quite interesting pipelines. So the Unix utility T is the namesake of this coroutine. So we're going to pass in a list of syncs and we're just going to fan out our anything that's sent to T to all of our sync coroutines that get passed in. So the way that this might work is we could create a log which is our printer. We're going to have a boss filter and a driver filter which both sent to log and then we're going to create our a T coroutine which takes both boss and drive. We'll send our people in again and we can see here that Ro prints twice because she's both a boss and a driver. So she comes through both filters. John comes through the driver filter and Lucy comes through the boss filter. Why you would ever use this? I'm not entirely sure but it's sort of fun to think about and to write. And here's a visual representation of this. So values coming in on the left getting sent to both boss and driver and then potentially coming back to the printer at the end. So now for something a bit different which you shouldn't do. So our coroutine pipelines even when they're fairly simple tend to look a bit like this. And the thing at the end is always deeply nested and we know from from PIP 20 that the flat is better than nested. Other languages including bash and elixir and enclosure have nice constructs for dealing with this sort of thing where we can pipe things together. So here's a whole bunch of magic. We're going to wrap our coroutines in a partial. Then we're going to wrap the partial in this powerful company or object which overloads the power operator. And this could certainly be tidied up a bit. There's a lot of kind of weird stuff. This is not easy to follow and this is also not getting the thumbs up on code review. But it allows us to do this. And as weird as this looks this does work. So an example here we have this range which is not the actual range. It's when I had to define myself which will generate the values 10 to 15. So 10 11 12 13 14. We're going to filter out only the even values to get 10 12 14. And then we're going to send those on to the printer and it works. But yeah don't do this. So that's pretty much all I have. So we've had a look at looks recursion into tools comprehension generators coroutines and then that stuff at the end that you shouldn't do. But thanks very much for coming and for listening. Again that's my tour name and the slides should be on Twitter very soon. I just want to thank you and are there any questions. Thank you very much John questions. Thanks for the talk. I noticed you had a nested list comprehension in there but there wasn't a yellow sticky note on it saying you shouldn't do that. Or it wouldn't pass code review. Is that something that you think should pass code review or is it something that should be kind of spit out into maybe a loop and a list comprehension. OK so the question is I guess how complicated should list comprehension be. And my kind of rule for thumb is if you're doing more than two things don't do it unwrap it into a for loop. So if you are creating a triple on the left hand side from keys and values maybe or you are you have a forward and if which is preferred to the sort of functional filter now. Are you doing anything slightly more complicated or if you have to write it over two lines I would suggest unwrapping it into for loop and then just appending to a list. I've got one further question if that's OK. So the method you've shown of kind of composing these pipes with sinks and things like that seems quite a powerful way of transforming data. With these reusable components and but you don't tend to see the style of programming very often. And I wondered why you think that might be or even maybe this isn't Pythonic in some way. That's quite possible. So the question is why do we not see coroutines and pipelines more often. I think it's I think it's just weird. I don't think it's yes you can do it a lot more easily a lot more. It's a lot easier to understand in other ways. I think the first talk I saw on coroutines someone asked the first question was what was the original problem you were trying to solve by writing this code. And there isn't there wasn't really an answer. So don't do it I guess. Any other questions. Hi. Could you bring up the slide with the dunder power again and maybe talk a bit why about you recommended not to use that. So why not use this. Yeah what was bad about that. There's there's too much going on. I don't I don't necessarily suggest you write this style of code anyway. But there's there are partials. There are nested functions. There are star args star star quarks. There are there's overloading built in methods just to do a little trick which is the star star to create the pipeline. Yeah you can do this stuff. It doesn't mean you should do this stuff. OK so the same would be for dunder model or matrix multiplication. It's just the whole idea of changing them is a bit complex. Yeah yeah I think so. Any more questions. I have a question myself. I find myself something sometimes wondering if I should use a lambda the partial or a local definition of a function. Do you have a preference for that. It depends. I've seen good uses of all all three. I don't I don't know that there's a good answer to that. I think it depends what looks good. What the rest of the style in the code base looks like. If landers are generally avoided generally avoid them. I think partial is reasonably unknown because it lives in func tools which is reasonably unknown. So I would probably avoid that a lot of people would probably need to look up the docs on that. So yeah again it depends. OK thanks. OK that's it then I think so please give them a big applause.