 In this video, we are going to talk about yet another data type to manage the memory efficiently. It is called the generator data type. So generators are kind of similar to the map and filter objects that we saw in previous videos. All of them have together that they are so-called iterators in an abstract sense. But generators are a bit more flexible. So in the previous video, we saw that the filter object is capable of only doing one thing. It basically filters out objects according to some rule. The map object is basically giving us back objects that are mapped to some other objects and nothing else, right? So now we are going to see the same example from the previous videos again, but we are going to solve it in a different way. And in order to do so, I will first build up the idea of a list comprehension first and then we are going to build up a new concept called a generator expression, which is related to a list comprehension. And that creates a new object of a data type called generator. So let's start by creating a new file and let's call it generators. And let's use the same example as we do before. So we use a numbers list with the numbers from 1 to 12 and it's going to be the numbers 7, 11, 8, 5, 3, 12, 2 and 6. And then we have 9, 10, 1 and 4. So the outcome should be 292, I can already tell you because that is the outcome in the previous videos, but let's also quickly review what is the task here. So the task is to first go ahead and transform all of the numbers in the given list here. According to the rule, y is equal to or will be set to x to the power of 2 plus 1, so it will look like that. And then after we transform all the elements, we will filter out the odd ones. And then last we will simply sum up the remaining ones, okay? In our very first approach, where we wanted to solve that, we used two for loops, the first for loop to do the transformation part, the mapping part and the second for loop to do the filtering. And each of these loops had or needed an empty list to start with where we could then build up the result by appending the resulting objects into the list. And in total, we had three different list objects in memory. And then in the previous two videos, we saw how when we use the map and filter buildings of Python, then we can do so without using two temporary list objects in a memory efficient way. So now we are going to see a different way of how to also achieve the same outcome. So first, I'm going to build up the idea or I'm going to review the idea of a list comprehension. We have seen a comprehension a couple of times, but this list comprehension will be a little bit trickier, so therefore I will slowly build it up and then we will develop that into a generator expression that we have not yet seen before. So how could we go on here? So first of all, let's go ahead and observe the following. We want to transform all the numbers first, and then we want to throw off the odd ones. And previously we did that with two for loops, two independent for loops. However, we could also do that with just one for loop. So let's do the following. Let's create a variable. Let's call it transformed events, and this is going to be an empty list first. So just like in the very first video on the map filter readers paradigm, I had two empty lists at first, one called transformed, one called events, which we use to store the results of steps one and two. Now we are going to introduce a new empty list, transformed events, which is going to store the results of steps one and two together. So we are summarizing the two steps into one. So we are going to write a for loop, of course. So for number in numbers. And now I have to do the transformation. So how can I do the transformation? Well, let's simply do it by writing transformed will be set to number to the power of two plus one. Okay, just as before, nothing special. Then in the same for loop, we will simply do the following. We will say if transformed, model or divided by two, double equals zero, we know that the transformed numbers even, then let's simply go ahead and append that to the transformed events list. Okay. And then also maybe let's give out the list just like that for transformed events. Let's put it right here. And we see that we can solve steps one and steps two using only one list object. And then to find the final result, we can use the list here and simply sum it up to get 292. Let's quickly go over to Python tutor and see what that looks like in memory. So let's go ahead and copy paste everything over and let's run the cell. So first we get a global numbers list that is no different from all the previous versions. Second we create an empty list called transformed events. And then we are going ahead and we are going to run the for loop. And the for loop as we will see will slowly build up the final list. And then at some point the list is done. And in the very last step, the built in thumb function simply sums up the temporary list here. Okay. So that is compared to our very first solution in the video where we talked about the map filter, a first look at the map filter reduce paradigm. We had two lists, two temporary lists, so three in total. Now we only have two lists in total. So we are saving ourselves one list by putting together the mapping and the filtering part into one for loop. Okay. So that is already an improvement. We are saving one, one list. However, we already saw in the previous video when we use the map and filter built-ins that we can get away without having a temporary list at all. And now we want to get to this point there as well, right? So otherwise we would have a disadvantage here. This version is definitely worse off than the previous two versions when we use the map and filter built-ins. So let's continue. And before we come to the memory efficient solution, let's first go ahead and translate this cell here into a list comprehension. So the difference in tactically speaking is this here is a code cell that has several lines of code. So it has one line here. It has a second, a for loop here. And it has several lines of code that have to be executed one after the other. A list comprehension is an idea that we saw a couple of times before. And the main idea behind a list comprehension is that we want to derive new list objects out of an existing one. So derive emphasis on new, a new list or new list objects out of existing ones. That is probably the main purpose of a list comprehension. So how can we do that? Well, just like we did in the very first video with a code example in this course where we averaged all the events in the list, let's go ahead and write a bracket notation here. So we are going to create a new list. So we indicate that by simply using brackets. And now we have to translate the for statement and the if statement into a list comprehension. So we do so by copying over the for loop without the colon right here. And for now, you remember that in a list comprehension in the beginning, in the first part, we have to write some expression that determines what is the number that is going to survive, right? So if I write number singular, then I basically get a copy. So that would be a shallow copy of the existing numbers list. If for example, I write here, let's say the word hello as a string. I get exactly 12 hellos white wealth hellos. Well, they are 12 not elements in numbers. So the for loop in the list comprehension is going to run 12 times. And 12 times we are going to store simply hello here. Okay. Obviously that is not the solution. I just wanted to show you that whatever I put here is the thing that's going to survive. So maybe what we want to do is we want to keep number to the power of two plus one. Okay. So these are all the numbers squared and we add one to them. So that is however, not all the numbers we want to keep. We want to filter out the odd numbers. Okay. So how can we do that? Well, obviously we have to write some if clause here. Okay. So what, what do we have to do? Well, could we go ahead and copy paste the if clause from the upper code cell? The answer is not entirely. Why not? Well, the condition up here uses a variable called transformed that is calculated two lines before. So transformed down here does not exist within the list comprehension. So therefore we have to get rid of it. Okay. And we have to replace it with an expression. And to be sure that the arithmetic works out, I will put there a pair of parentheses for grouping. And now I will go ahead and I will take number to the power of two plus one, which is the same as we have in the front here. I will copy paste that here. And if I execute that, I also get a list object back. And it turns out that this is actually the same list as we get back above here. Right. Okay. So that's the list comprehension. So now to make it a bit shorter, let's replace number singular with just n so that it's a bit easier to read numbers global, of course, has to has to remain. So what this list comprehension does is it derives a new list object out of the existing one, which is numbers. And the rule is we keep every, for every number in numbers, we keep n squared plus one, but only if n squared plus one is even, that is basically what this is saying. Okay. So this is almost a mathematical notation that we see here. It is still Python code, of course, but in many other programming languages, there is no nice syntax to mimic mathematical notation in Python. However, we can do that. Okay. So this here creates a new list object. And now the question is, could we go ahead and simply sum that up? And the answer is, of course, yes. So let's use the sum function and let's pass our list comprehension as the argument to the sum function. So when I, when I execute the cell, what is going to happen first is the execution, the evaluation comes from the inside out. So in other words, the list comprehension is being executed first, a new list object is being created. And after the list object is created, it is immediately passed as the first argument to the sum function. So the problem with this is, is the following. This looks maybe nicer than this. You could argue some of you would even argue that we should just keep it like that because this is more explicit. But in terms of memory usage, both versions are pretty much the same. Okay. So let's copy paste that over. Let's copy paste that into the result part down here. Let's get rid of all the code up here. And let's see what that looks like in memory. So first we create our list object. And next we are going to create a list or we are going to run a list comprehension as we see. And the problem is now that after the list comprehension is done, as we see here in the diagram, a temporary list object is created. So even though we don't have a name for that, we don't assign the list comprehension to some variable, even though there is no name for that, there is still a list object. Okay. So in the split second before the sum function is being started, there is a second list object in memory. So what we learned from that is that the final outcome shows us that there is no second list object. However, at some point, as the program was being run, there was a second list object, okay, just right before the end. So in other words, memory-wise, this is not an improvement, okay? So we are simply using a different way of expressing that we are creating a list object. So up here, we are starting with an empty list and we are appending it in a stepwise fashion. Down here, we are using the list comprehension to directly derive a new list object out of the existing one. But in both cases, there is a temporary list object. Up here, it has a name. Down here, it has no name, but this doesn't matter. The memory usage is still there. Okay, so that is a list comprehension. So maybe let's go ahead and copy-paste the list comprehension out of that. So just to review that, that's the list comprehension. It creates a new list object. So now comes the new thing that I want you to learn in this video. And the thing is called, so-called generator expression. So you may wonder, what is a generator expression? Well, it is an expression that when it's evaluated, it generates a new object, or it creates a new object that is of type generator. That is why we call it a generator expression. So first of all, you may wonder, what is the syntax? So how do we write down a generator expression? Well, there's a very easy solution. We are simply going ahead and we are going to replace the brackets with parentheses. So one thing that the parentheses here do not mean, they do not have anything to do with a tuple. So that is misconception number one. And misconception number two is, the parentheses are grouping operator. It's also not grouping. So the parentheses here have yet another meaning. And the meaning the parentheses have here is that of a generator expression. Okay, so what do we conclude from here? So if you understand what is a list comprehension, the way to go from a list comprehension to the corresponding generator expression is by simply replacing the brackets with parentheses. That's it. And if I execute the cell, we get back a new object, which is called a generator object at some memory location. Okay, so let's do the following. Let's go ahead and store the object in a variable. Let's call it gen appropriately because it's a generator. Let's see what is the type of gen? Well, the type is generator. And so what can gen do? What can a generator do? The answer is pretty much the same as the map and the filter objects. The only thing a generator can do is it can give me the next object in a line of many. And according to some rule, whatever the rule is, and that's it. So let's go ahead and ask the gen object with the next function. Hey, gen object, give me your next number. And it gives me the number 50, which happens to be this 50 here, right? So the first, so this is after transformation and filtering the first element. If I call next gen again, I get a number 122, 26, 10, 82 and two. And remember what happens when the generator is exhausted? So now we have already reached the last element. So if I execute the code set one more time, I again get the stop iteration exception, which is really not an error. So Python uses or misuses an error message here for something that is really not an error. This is just a signal telling you the generator is exhausted. Okay, so what else could you do now? Well, from now on, there's nothing you can do with the generator. You have to create a new generator. So how can you create a new generator? Well, simply run the generator expression one more time and you get a new generator. If at any point you want to, what I call materialize the generator, what you could do is you could simply pass it to the list constructor. And then what happens is the gen, the gen object here is simply a rule in memory that knows how to calculate the next element in a series of many elements, but it has not yet done so. However, if I go ahead and use the list constructor, what the list constructor does behind the scenes is, it basically calls next all the time until it has pulled out all the numbers and then it takes all the numbers and puts them in the list. And I call that to materialize the elements. So maybe the term materialize is a bit weird for you because nothing is really material in a computer program. It's just energy flowing. So it's once and zero is basically just energy flowing or not flowing. But before that, when I only have the generator, I don't have any once and zeroes in memory that model something. I only have once and zeroes that basically know how to calculate something, but I don't have once and zeroes that mean the numbers. After I pass the generator, for example, to the list constructor, there are once and zeroes in your computer's memory for each of the numbers. That is what I mean with materialization, okay? So in other words, the generator itself takes basically no memory, but once we materialize the generator, then it covers all the memory that it needs, right? So this is now just as good as above here. So you may wonder, what is a generator then good for? Well, one thing it is good for, let's create a new generator. So now we have a new generator here. A generator is abstractly speaking, a so-called iterator, and an iterator is something we can loop over. Don't mix that up with iterable. There will be a video coming soon that compares the concepts of an iterator with the concept of an iterable, two different concepts. However, the concept of an iterator means it's basically a special case of an iterable. So you can also loop over an iterator. You can loop over both of them, but you always want to remember any object I can loop over is a so-called iterable, and I will make these two definitions more precise in an upcoming video. So for now, as we saw, we can loop over the generator, so it's kind of like an iterable. So what we can do is we can simply pass it to the sum function. Let's execute that, and I get back 292. So what that means is I can now go ahead and calculate the entire result without any second list. So let's do that, actually. Let's go back into Python tutor, and maybe let's do result one up here with the old way, and let's do down here result two, and let's replace the brackets here with parentheses. Okay, so now we have double parentheses, double opening parentheses, and double closing parentheses. The inner parentheses are part of the generator expression syntax. The outer parentheses are the call operator. So you just have to get used to the fact that parentheses have many different meanings in Python. And here we have the outer parentheses are always call operator, and the inner brackets and parentheses are just syntax to create either lists or generators. So let's go ahead and run the example one more time. So first we create a list called numbers, a materialized list, and then we run the list comprehension. And the result of the list comprehension is going to be that we have a temporary list. So for an instant of a second, maybe a nanosecond or so, we use the memory here, right? We have a second list object. And if I go ahead, the result of summing up the list, the second list object is of course correct here. It leads to the correct result. And now comes the generator expression, as you see. And the generator expression also loops over all the numbers and transforms them and filters them. However, at the end, the generator expression does not create a second list object. And it also leads to the same result. And the reason why it leads to the same result is because the built-in sum function internally uses this logic of a running total. So what the sum function really does is it talks to the generator and it asks the generator, hey, give me your next number, and it gives it, and then it adds it onto the running total, and then it goes back to the generator and says, give me your next number. And at some point, the generator will say, I'm out of numbers, and when the generator does so, then the sum says, well, I'm done. That's the sum. So using a generator is a memory-efficient way of also solving this result here, of also solving this task here, okay? So this is an alternative to map and filter. So map and filter are very precise in what they can do. Generators are more flexible. And there will be also a future video on so-called generator functions. Generator functions are basically function-like syntax that allows you to customize the logic that is behind a generator even more. But at the end of the day, the result of both a generator expression but also a generator function is that we get back a so-called generator object. And the generator object is any object that, or is an object that only can do one thing, give me the next element in line, okay? So let's do one last thing before we end this video. So let's go ahead and make this a bit shorter. So remember, I copy-paste the expression, the generator expression down here. And whenever I run the cell, I get a new generator, a new generator object. And now, as we saw in Python Tutor just now, I can pass that to the sum function just like that. And now I have double parentheses and double parentheses here. And it gives me the correct result, of course. However, the one thing that is now important is under one circumstances, you can do that. Whenever a generator expression is the only argument to a function, and that is the case here, I don't have a second argument. So whenever a generator expression is the only argument to a function, what you may do is, you may get rid of the inner parentheses which is equivalent to getting rid of the outer parentheses. Okay? And if you run that, you also get the same result. Okay, so maybe let's keep that in here in the document with double parentheses and also single parentheses. And that is basically the shortest version you can do. So in other words, this one line here, that I mark here, solves the entire problem that previously we solved with the map and the filter buildings. Using the map and filter buildings, you remember that the final expression was a bit, yeah, there was just more code. We had to do two steps. First, the mapping step, then the filtering step. And here using the generator, we can do both steps at once. So that is what I mean when I say a generator expression or a generator in general is more flexible in what it does. However, there is one big conclusion from this video. A generator is really just a rule in memory that knows how to calculate the next object in a series of many without having calculated them yet. So in that regard, it's just a rule. It's basically no different than a map and a filter. And all three of them have in common that we call them iterators. So iterator is the abstract concept, just like a sequence is the abstract concept to a list and a tuple, for example. So list and tuple are concrete data types. Sequence is the abstract data type. And here, map, filter and generator are the concrete data types and iterator is the abstract data type. Okay, so that is it for this video and I will see you soon.