 Yes, this talk I call iteration inside out, and it's an explanation of Python's iteration protocol. It assumes that you do not particularly know much more than how to write a for loop. That's the main assumption. So, yes, I'm Naomi Cedar. I am at the moment the chair of the Python Software Foundation. That has nothing to do with this talk, but I put it there anyway. I am the author of the Quick Python book, third edition just came out. I do have a couple of copies with me that I will give away to people who ask nicely. So come see me later on, and I will be happy to talk to you about that. And I also lead a small Python team for a company called Dick Blick Art Materials. We're the largest supplier of fine art supplies, that is paints, brushes, canvases in the United States. It's not a huge market, but we're one of the biggies, so I enjoy that job, actually. So I'm going to start with a quote from Dave Beasley. I don't know if you've had a chance to see, he gave a remote talk for a PyCon Pakistan, which among other things features him doing a trombone solo, highly recommend that. But in this talk, which is called the iterations of evolution, he goes way, way, way back in Python history. He describes the for loop and the iteration protocol as Python's most powerful useful feature. And I happen to agree, it's really an interesting thing that is enormously powerful and enormously taken for granted. So when you think about it, we have repetitive chunks of data all around us. It's one of the things, after all, that computers are good at is repeating something over and over and over and over and over again many more times and much faster than we can do. So just picking some random examples of things that I have run across not that long ago. You could have a series of temperature readings for the month. You could have dictionary keys, maybe for a dictionary mapping member IDs to members. You could have a CSV of a million or two or 20 million products. You could have the text of Moby Dick to a very big book, a lot of things in sequence. You could have the results set for a database query finding your daily sales even there. So there are just a lot of different things. And these things don't, on the face of them, have a whole lot in common. They're different types. The objects involved are different. The elements contained in those objects are of different types. Sometimes it might be just a simple type. Sometimes it might be itself a compound type of various other pieces, whatever. They're all kind of different. But the situation that we want to get at them is all the same. That is their large series of data where we want to do the same thing one after the other all the way through. So in Python, obviously, we would use a for loop. And that seems pretty obvious, except that the way that Python does for loops was not always an obvious thing. In fact, it used to be something that had to be called out as being different. So going back in history, this is from the Python 1.1 documentation, which is nearly 25 years ago. And here's what it says about for loops. I kind of like the tone of this whole thing. It's like Python in for loops, like for loops and you. And the four statement in Python differs a bit from what you may be used to in see your Pascal. Those are the days, Pascal. Rather than always iterating over an arithmetic progression of numbers like in Pascal, for those of us who suffered through Pascal, you remember, that's a for loop, just gave you numbers. That was it. Or leaving the user completely free in the iteration test and step as in C. And I'll talk about that in a second. Python's for statement iterates over the items of any sequence, e.g. a list or string, in the order they appear in the sequence. So in the early days of Python, this was weird. This was something that was specifically called out in the documentation. So for contrast, some of you may have done C. I mean, some of us have survived C programming. This is a C for loop. And for those of you who aren't familiar, you can see, though, that there's a lot going on in the line with the four on it. In the parens there, we're declaring a variable. We're initializing a variable to be able to control. We have a condition for when the loop continues. And we are incrementing I, the loop control variable. That's all packaged in that four line. Now, if you haven't done C before, I can tell you that what's required there is that you have two semicolons, basically. All the rest is up for grabs in that four line. And in fact, really what's going on is that this is just a repackaging of this kind of while loop. You see, we've got the same pieces. There are loop control variable. We initialize loop control variable. We've got a condition to see whether the loop continues. And then at the bottom, we increment the loop control variable. It's the same thing. In both cases, you are free, if you want, to not put those things in or to put in something completely different. It doesn't matter. You can do whatever you want, but that's the way that it works. And in both of those cases, the loop control variable, the I, is just an integer variable that gets incremented and it's used as an index into the array. There is no particular connection between the two. In fact, when you notice, this is controlled by a variable called list.len. And of course, I didn't realize for C, it's a little bit of anachronism or a misnomer to even use the word list, but I put it in for you. But that has to be determined completely separately. That has nothing intrinsically to do with the array. So that's the way that things used to be. Now, you can, and here's where I'm going to start playing around a little bit with code for your amusement, for my excitement, but you can. And people, when they used to start doing Python, you'd see this a lot. They would do a for loop and they would say like for I in range, when, oops, A list, and then we would do something. And that's kind of reproducing what the C sort of loop would be. And as I say, people beginning Python coming from a language like C would do this a lot. I remember I think I did it when I started. So that gets you the loop. It works. But in fact, even that isn't exactly what C is doing. Because to get our list of index values, our 0, 1, 2, 3, to go through these elements in the list, we're actually using the range to get ourselves a sequence and then we're going over the sequence. So it's not even exactly the same as what C does. And it's certainly not the way that most people would write a for loop that they would think of as being quote, Pythonic. Normally, what we would do is more something like this. And that, you'll notice, throws away that whole going by index. And instead, we're going by one item in a sequence. OK, now this shouldn't be new to any of you. This is just kind of setting up our main premise here that Python's iteration is a somewhat different thing than a lot of the traditional things. I will say that these days, there are many newer languages that do sort of the same thing, or pretty much the same thing. A lot of the languages that didn't use to have something like this now do. Even in C, they have some macro jickering that makes you come up with something that kind of sort of similar in a way. So there are lots of different options for doing this. But for a while, Python was pretty much one of the few that did things this way. And all of this kind of raises the question, how does it do that? So when we go back and we look at that example, let's just pop back there, how does that for loop know where it is? I mean, the whole key to a for loop is you go through an order one after another after another. So how does it know what's next? There's nothing in the for loop that we just did that really explicitly says that. And how is it that, basically, you can use for loops on all of these wide varieties of container types, and it all just seems to work? Or better yet, suppose I want to make something that I can stick in a for loop, how do I do that? So that's what I want to talk about very quickly. So key thing is for Python, this whole iteration using for relies on a protocol, not on individual types. And this has been the case since Python 2.2. And I'm old. I mean, I was around at the end of the Python 2.1 few months, whatever it was. It wasn't the current version for very long. But so this is about as long as I've been doing Python. This has been kind of codified as a protocol. And particularly, if you're newer to the language and you hear people talking about Python doing duck typing and you don't really have a clear example in your head what this means, that is what this means, is that Python's iteration protocol is an example of duck typing. And if you're not familiar with duck typing, this is a case where if it walks like iteration and it quacks like iteration, it's iteration. So this is a protocol. And for this protocol, it turns out you need two things. And I realize this is sort of looking like we're defining one thing by variations of the same thing. What's this? What you need for iteration is you need an iterable object. I'll tell you about those next. And there is also need for an iterator object. And I'm going to talk about that after I do the iterable. But in fact, most of the time, Python actually takes care of the iterator for you. So it's not something that you need to worry about a lot most of the time. So before I realized I was going to go on way, way over and I had to sort of squeeze things down, I was going to go through the whole Python glossary thing on iterable, but instead, nope, you got the TLDR version. So basically an iterable is something that can return its members one at a time in order, like a list, string, tuple, anything like that. Or it's so and basically that means one of two things. It can be a class with a Dunder iter method that returns an iterator, which again, I haven't talked about yet, so put that on hold. Or it can be any class that you make that has a Dunder get item method that follows sequence semantics. And by that we mean you start at the beginning from zero and count up using integers to get to wherever you want to be in the sequence. So it's one of those two things. Can be both, but basically those are the two things. And what happens is when you do a for loop, if you hand it an iterable, it's going to create an iterator sort of anonymously behind the scenes to manage the for loop for you. So that's why you almost never need to worry about it because it's done automatically. So again, to restate, if you use the iter function, which is a built-in function, on an object, in order for it to be an iterable, it needs to return an iterator. So it's either got a get item that we can use square brackets and an index to, or it's got an iter. So let's actually talk about doing that. Now there are a couple of different ways that you can check to see if you've actually got an iterator. So you can look and see if it's got an iter method. So like if we've got a list, I can check and see if my list has a property under iter. And since it's a list, it does. So you can check that. But that's only one of the possibilities. The other possibility is, does it have a get item that follows sequence semantics? Well, you can check and see if it's got a get item. But it still might not be following sequence semantics. Dictionaries, for example, have get items, but they don't take integer indexes. So that wouldn't work. So that's a little bit trickier to decide. Basically, though, remember, we're programming Python. So we have the philosophy it's easier to ask for forgiveness than permission. So what you can do is you can just try calling iter on an object. And either it will give you a type error, or it'll give you an iterator. If it's an iterator, then yes, you have an iterable object. So what I want to do, first step, is I want to make the minimum possible iterable. Now, this is kind of modeled on an iterator that is in the collections library. But basically, I want to make a thing that repeats itself, trying to keep this as simple as possible. So basically, if I make a repeater object and I tell it that what I want it to repeat is hello, and I give it the number four, I can loop over it, and it will give me four hello's. Simple enough, right? So let's see what that's like. So I'm going to, and as I say, here's the exciting part, where we'll see if I can actually manage to write this class without having something horrible go wrong. So let's see here. We need an init. And I want to give it the value that it's going to be echoing to me. And I'm going to give it the limit as in the number of times I want it to repeat. So here, this is just a little bit, see, slough. I already did it. So value equals limit. OK, that's good enough for my init. Now the other part I needed was get it, right? So let's see what that might look like. Self, and I'll call that index. Oh, thank you. That's very helpful. I know I did that when I was practicing, and it took me like two or three minutes to go back and figure out what I did. It kept saying, this is not an iterable. Yes, it is. Oh, it isn't. So yeah, so I've got that. So here, we need to do something that will satisfy sequence semantics. So I'm going to do, we need to start from zero. If it's less than zero, this is really not going to work out well for us in this case. And we don't want to go over our limit. Otherwise, why would we bother to set the limit there? So if it's, ah, thank you. That was the other thing I kept screwing up. I was just testing you. There we go. OK. So if that's the case, I'm going to return self.value. And what happens if we go over? It's the other question that's not quite solved. So what we do there is, if you want out of my four element repetition, you want item 99, I'm going to say that's not a good index. And I'm going to throw you an index error. OK. So we got that. And I don't think we screwed that up. So we actually have then, this is a class for an iterable. At least I'm saying that. Let's see if it actually is. So I'm going to make an instance of my repeater class here. I want it to say hello four times. And then I want to see if it actually does have a dunder-dunder iter method. OK. It doesn't, because I didn't define one. OK. Fair enough. Does it have a get item with sequence semantics? Well, we could mess around with that a little bit. Basically the easiest thing to do is just to see if I can get something from index zero. And I do. So OK. So we've got something that kind of looks like a sequence object. Let's see how that actually goes. So can I get an iterator from it? This is sort of the real test. If I can't get an iterator from it, then this whole talk is over, right? So maybe that's a hint. I don't know. But let's see. So yeah, you can get an iterator from it. Now remember, what I got there was really pretty simple. So if it's an iterator, it should work in a for loop, right? So there we go. So that little tiny class is all that you need to have it be an iterable. And let's comprehension. Oops, let me do that again. There we go. There we go. OK, so that works too. So basically what happens behind the scenes is I say every time that repeat, my little object gets passed to a for loop, the iter method is called on it, and an anonymous iterator is created behind the scenes. And that's what manages how many hellos we get. OK. And when you get to that point where you're off the end of your object, when you hit that index that's one too far, the iterator knows to pick up the index error and stops the iteration. And then every time we do that, we get a fresh iterator. Otherwise, we would only be able to do the loop once. And I'll talk about that in a second. So again, this was the minimum case we had. The get item was the bit that we actually needed to make an iterable. And the index error was the part we needed so that our iteration would stop. You can for amusement, and I've done this before. I don't recommend doing it very often, though. You can leave off that part and just have it go forever. One hopes you have some other strategy for getting out of the loop then, short of turning off your computer and walking away in disgust. At least that's why you had to do it in the bad old days. So that's it. It's really very easy to make an iterable. But that's kind of just taken that question we raised at the beginning and just kind of kicked it on down the road. It hasn't really answered, how does it know which item is next? Because clearly, the iterable is just responding to being given an index and it feeds back an item. It doesn't know what's next. The loop itself doesn't seem to know what's next. A loop doesn't even care what you're giving it as long as it follows the protocol. So it's the iterator part that is actually keeping track of what's next. And anything that does that for us is an iterator. And the key thing for an iterator is that it has a DunderNext method. Or for reasons that I've never really managed to make up a good explanation for, in Python 2.x, it's just next without the Dunder. I don't know why that happened. But obviously, it seemed like a good idea at the time. But it's got a next that gives you the items in turn. So that's what an iterator is. It's one of these things that will manage where you are in the sequence and keep on giving you the next. When you get to the end, it doesn't raise an index error like our iterable did. Instead, it raises a stop iteration exception. That it knows means no more loop. And if you keep trying with that same iterator, it'll keep on telling you stop iteration. It's over. Basically, iterators are one-use objects. You can make one that wouldn't be that you've but this is the way that they work. The other thing that an iterator needs is it needs a DunderInter method. And that DunderInter method returns the object itself, which means that all iterators can be used as iterables. So it's like, huh? We'll talk a little bit about this in a second. The thing with iterators, though, again, is one-use and done. They don't magically refresh. You can't reuse the same iterator. You're always going to get a new object. So I want to do the same thing, but I want to do it with an iterator now so that you can see how that works. So let's back up here. We need to do two things. We need to have a next. And we need to have a DunderInter method. So again, let's sort of do this. Help me out here. So first thing I want to do is, again, a little bit of housekeeping. We'll do the same sort of thing. We'll do that. So obviously, that's just the same business. And we want to do one more thing, because we're going to actually need to keep track of where we are in this sequence. So I'm going to call it count. And I'm going to start counting how many objects we've iterated at zero. So I'm just kind of setting this up as part of my DunderInter. So then, what was the next thing we needed? Audience participation? Next. Well, or either which upper. But yes, next is more fun, so we'll do that one. OK. And again, we can kind of whatever. I mean, this is a little bit simpler. I do not mean to do that. Stop that there. So we want to be sure we're not past our limit, basically. So if we're still in the iteration business, remember, we started counting at zero, but we don't want to go past our limit. So if we're still able to do that, we kind of need to do two things. We need to type properly is one thing. First, we want to keep track that, oh, yeah. We've probably used up one of our iterations. So we'll do that. And then the other thing we would do, of course, is return self.value. And if we are at the limit or over, if we've gone past where we should be, I'm sorry about that. Pardon me? The makeup is in your face. Oh, I'm sorry. I'm not hearing you. Well, yeah, that's probably the wise thing to do. Oh, I see. I lost the whole thing. That would have been embarrassing. That would have not been good. OK, else? So let's see here. We want to raise stop iteration. There we go. OK. There we go. Let's move ahead. Oh, let me advance this quickly. OK. We're running out of time. I know I'll get in trouble if I go too far. So basically, I need to back up here. And I'm going to make a repeat iterator. Does it have an iter method? Well, you can guess that, yes, it does have it under iter method. And I will quickly do this. Pardon me? Get back to repeat iteration. Pardon me? Get back to repeat iteration. Oh, I didn't actually. The definition of the salary has a state of the next. Oh, yeah. There's a quickie script right there. Oh, thank you. Well spotted. All right. There we go. OK, now I'm going to do this. And we should be good. So there, we got one repeat. All right. I'm going to move ahead here. And this is the last question, because we're running out of time. I didn't go nearly fast enough. But if I do a for loop here, this is a trick question. What happens now? We should get three, because I've already called it on at once. Oops. Oh, dear. Oh, that's because in my hurry, I forgot to add the iter method. There we go. So yeah, we're running out of time. So I clearly mistimed this. There we go. Make that. Make that. And then if we do that now, we have four, because I didn't exhaust it. Now if I were to do this again, if I were to run this again, nothing happens, right? Because it's exhausted. And in fact, if I then want to, let's fake this, I'm going to do this one last thing. I'm going to call next on it. Next six probably won't work. What happens now? Right. So for those of you who know, yes, you get a stop iteration thing. It's only when you call next that you're going to see that. So in any case, this is a little bit beyond what I was going to do, but the slides are there. And I will post them. And you can look at the Jupyter Notebook, and you can contact me. And thank you very much for your time. Thank you. Yes.