 In this video we are going to revisit the example of the previous video, and we are going to implement a memory-efficient solution. So, let's go ahead and create a new file, let's call it MFRRevisited. So, the example is the same as in the previous video, so we are given a list of numbers. And we are going to use the same numbers that we always use in this course, so 7, 11, 8, 5, 3, 12, 2, 6, 9, 10, 1 and 4. So, hopefully I didn't mess up the numbers because otherwise we will not get the same result of 292 as we did in the previous video. So, let's see if I got it correct. So, let's also briefly revisit the task. So, the task is simply to first transform all the numbers. And the transformation rule is simply going to be the new number is going to be the old number raised to the power of 2 plus 1. So, I will write that using a mathematical notation y will be the new y, the mapped number will be x squared plus 1, the old number. Then we are going to filter out the odd ones. And last, we are going to sum up all the remaining numbers. So, that is the task. So, previously we solved the task using a temporary list object or two temporary list objects. And in this video we are going to avoid that, we want to be memory efficient. So, let's go ahead and see how Python can help us do that. So, at first we are still going to do the mapping step. And mapping step is all the operations in our calculation that take some number or some element in our original data set and map it to some other element. That is of course step number 1 in the task. So, we are going to map every number to its square plus 1. In order to do that, in this video we are going to define a helper function. We simply call it transform for now. The function takes one element as an argument. So, let's simply say element as the argument. And now it is going to return the element raised to the power of 2 plus 1. And you will see in a bit why I have to define a function here. So, that is the mapping step. So, given some element we just square it and add one. So, now comes the big idea. Now we are going to use a built-in into Python called the map built-in. And the name map is of course derived from the mapping step. From the map filter reduce paradigm there is the mapping step. And in Python there is a built-in function called map. And this function implements or helps us implement the mapping step in a memory efficient way. So, let's do that. So, what does the map function do? So, the map function takes as its input first a function. So, maybe let's go to the Python documentation so that you can also read that up simultaneously. So, the map built-in is right here. And we see that the map built-in takes as its first argument a function. And then it takes further intervals. So, one interval and then the dot-a-dot simply says there can be further arguments also going to be intervals. And we have seen in a previous video of how we can implement a function ourselves that takes any number of arguments using the star notation for packing and unpacking, if you remember. But for now we only have one interval which is our numbers list and one function. And the function that map takes is going to be a function that has to take one argument itself and it has to provide back the element that is the thing that is being mapped to. So, what we are going to do is we are going to give the map built-in a reference to the transform function. So, note how once the function is defined, I can simply reference the function by its name. So, transform is just a normal variable that I can use to reference the function object. And now I am going to pass a reference to the function object as the first argument to the map built-in. The second argument is going to be our numbers list. And now let's see what happens if I execute that. I get back a map at some memory address. So, what can we do with that? So, first of all let's go ahead and store that in a variable let's call it transformer. So, there is a transform function and now there is a transformer. And the transformer is the thing that transforms the elements. That's why I call it transformer. So, the type of transformer is going to be map. It's a map. But sometimes you hear programmers say that it's a mapping element from this set of things to this set of things. But map is a type as we see. It's a type, a concrete data type built into Python. And now what can a map do? Well, now comes the thing. It can only do one thing. It can only, using the function next, the built-in function called next, it can give you the next element in line. So, by saying next and we give it transformer as the argument, I get the number 50. The number 50 happens to be 7 to the power of 2 plus 1. Let's see if I execute next transformer a second time. Now I get back 122. That is simply 11 to the power of 2 plus 1. Now I do it one more time. I get back 65 and you get the idea it's 8 to the power of 2 which is 64 plus 1. So, what is the map object? So, there's another concept that I will talk extensively about in a future video which is called the iterator. So, iterator is an abstract data type. And just like we saw before, a sequence is an abstract data type and a list is a concrete data type that implements the sequence idea. The map data type is a concrete data type and it implements the idea behind an iterator. And in iterator, formally speaking, if any object in Python that can do one thing and one thing only, it can provide you the next element in line. So, in other words, the transformer object here, it cannot even go backwards. So, if I go ahead and let's say there is no previous function, there is only a next function and also, if I go ahead and I ask Python, hey, what is the length of transformer, I will get an error message because the transformer object does not even know how long it is. So, it is not a sized object. You remember the term sized is also an abstract property that sequences have, for example, but the transformer object is an iterator and iterators are not sized. So, in other words, how I think of that, casually speaking, is transformer is really a rule in memory that knows how to calculate something but has not yet calculated that. It's like a postponed calculation. It's like a rule. It's a box in memory, an object in memory that models a rule how to calculate something without having it done yet. And how can we make the rule calculate the elements where one way is the way that I just showed you by simply using the next function. The next function goes to the rule and says, hey, rule, can you give me the next element that you can calculate? And the rule cannot go backward. It can only go forward. Instead of using the next function, we could also go ahead and use the list constructor and I could give the list constructor the transformer object. And now I get back a list of all the transformed numbers. However, note one thing. This list is shorter than the numbers list. Why is it shorter? Well, let's see what is the first number we get here. The first number is the number 10. So why is the first number 10? Well, the last time I called the next function with transform as the argument, I got back 26. So 26 is obviously the mapped object that is mapped to the number 5 because 5 to the power of 2 plus 1 is 26. So 3 squared plus 1 gives me 10. And 12 squared plus 1 gives me 145. So therefore, the list constructor now, basically behind the scenes, we remember that the list constructor takes any iterable and an iterator is also an iterable. That is already maybe a confusing thing, that an iterator and an iterable, they are two different concepts. But we will talk about that in a future video in detail. So for now, let's simply view it this way. The next function will simply get us the next object and the list constructor will simply go ahead and call next, next, next as long as there is no next. So the list constructor is like, maybe let's write it down, automating next. It automatically calls next as often until the end is reached. And the object itself, as we just saw, does not even know when the end is. So the only thing that transformer object knows is, hey, I don't have any more element to give you. So let's go ahead and see what would happen now if we went ahead and we called next on transformer again. Now I get a so-called stop iteration exception. So I get a red error message. And this error message is really not an error. This is just a signal from Python to you that the iterator, so the transformer object, is now out of elements. So now, from now on, as long as I call that, I get back nothing. And let's say, one more thing, if I go ahead and call the list constructor one more time with transformer as the object, I simply get back an empty list because the transformer object does not know how to produce anything anymore. So what can we do now with transformer? Well, with the transformer object, you cannot do anything anymore. It is basically exhausted. That's the technical term. It is exhausted. You cannot do anything anymore. So let's remove all these cells again. And let's create a new transformer object by calling the built-in map function one more time. And this basically gives us a new transformer. And now if I call next again, I will get the first mapped element. However, I don't want to do that now. I want to postpone that. That's the whole idea of being memory efficient. I want to postpone all the calculations until I really have to do them. So let's continue with the second step, the filtering step. And the filtering step for that, I will also write a function. And I will call the function, say, ifEven. And the function also takes one element as its argument. And now it is going to do the following. It will check if element modulo divided by two is zero. Then the function is going to return through otherwise, so else, the function is going to return false. So this function is given an element, a number here. And it tells me yes or no if the number is even or not. So now let's make this function a bit shorter. So the first way to make it shorter is get rid of the else clause and unindent the return false. This is called the early exit pattern. And now there's even a nicer way to do that. So note how this function gives us back either through or false, one of the two. So this expression here, the condition element divided by two double equals zero itself also evaluates into either through or false. We can drive it out by simply going ahead and say, let's say if I had two modulo divided by two double equals zero, I get back a true. If I divide three by two, I get back a false. So we see that the condition itself generates already booleans. And this is going to be true when the number is even. So in other words, what I could do, I could get rid of both the return statements here. I can get rid of the if clause, the if statement. And I can simply return the result of this expression. That's it. The result, this is going to return either through or false. And true indicates the number is even. So that is our is even function. And now, similarly to how we used the map built in, we're now going to use a built in that is called the filter built in right here. And the filter built in also takes a function and one interval as an argument. And basically how this works is as follows. If I go ahead and I create a filter and I give it as its function, the is even function, and let's say I for now give it the numbers list, I get back a filter object. So what is a filter object? So filter, let's maybe, what is a good name for that? Let's simply call it evens. So evens is now not a list, but evens is now of type filter. So filter is another concrete data type that abstractly speaking is also an iterator. So what do the map and the filter data types have in common? Well, they have in common that they only are good for one operation, getting the next element in line. So if I now go ahead and say next filter, and not next filter, but next even, of course, I get back the number eight. Why the number eight? Well, if we go through this list here, the original list of numbers, it says seven, 11, they are both odd, and eight is the first even number. So next evens gives me eight. So what would be the next number it would give me back? It would be the number 12, obviously. So let's try if it works, and indeed it does. So the filter object is good for one thing. It will give us, according to some rule in memory, it will give us back the next element in line, and the rule is going to be the next element must be even. That's it. So now if I go ahead and execute this a couple of times, I get back all the even numbers, one by one. And at some point I get a stop iteration exception because the evens iterator is exhausted. So same idea as above. Now we are going to do one thing different here. Instead of drawing the numbers from the numbers list, what we are going to do is we are going to draw the numbers from the transformer object. So note how here I created, let's do it over again so that we can be sure that the object is fresh. It is not exhausted, not used so far. The transformer object is now only a rule in memory that knows how to calculate something. And the rule is simply take some number from numbers and transform it according to this rule. And give me the next number according to this rule. Now the filter object down here, as it source, so to say, is not using the numbers list but the transformer. So we are building a rule that as a source has another rule. So in other words, we are going to ask the filter object, hey, give me your next thing. And the filter object says, well, I'm going to ask the transformer object to give me the next thing. And then the transformer object goes into the numbers list and the numbers list will give us the next number. And then the number first goes through the transformer to be transformed. And then the transformed number is going to be filtered right after each other. So in other words, if I now go ahead and I say, next events, I am not going to get the number 8 anymore. But I'm possibly getting the number 7 squared plus 1, which is 50. So let's see that. And indeed, I get back the number 50. So I'm doing one by one calculation, so to say. This is like some people call that a pipeline. So whenever I execute this cell here, next events, the filter object is going to get its next number and it's going to ask transformer. And transformer is going to ask for its next number, which gets the next number from numbers. So we are pulling out, in other words, we are pulling out the numbers one by one. We are transforming them, filtering them, and then we have to do something else with them. And the something else we want to do with them is the following. This is the easiest part in the whole chain here. So let's get rid of the next here. The easiest part in the whole chain is simply the reduction step as before. So this is no different from before. And the reduction step, we can simply use the sum function. So the sum function, according to the documentation, takes, as we see, any interval, anything we can loop over. And now we are going to loop over the events object and even the filter object. So now this is giving me the same answer as in the previous video, luckily, so the numbers above are correct, 292. However, we are doing that in a memory-efficient way. So let's quickly go ahead and copy-paste that over to Python Tutor so that we see the difference also in a memory diagram. So here are the numbers. Now I have to copy-paste a little bit more. And in the next video, we are going to see how we can even get rid of the functions that I now need to copy-paste. So let's go ahead and copy-paste over this here. Let's also copy-paste over the filter function and let's maybe put the functions on top, so the filter function goes on top here. Then we create the events object and then last but not least, the result as the sum of events. Okay. So let's go ahead and visualize that. So first, we are going to create the numbers list in the global scope. That is exactly the same as before and we must do that because somehow we have to be given some raw data. Now we are going to create two objects which model the rules. They are just plain ordinary function objects, nothing special about them. And when it comes to memory consumption, we can assume that they are not consuming a lot of memory. Okay. So we only care about the big list that possibly has, I don't know, 10 billion numbers in it, but we don't really care about the size of the function objects here. Now next we get a transformer object, which is, it says here a map instance. So it's an object of type map, which is simply a rule that uses the functions above to pull out numbers one by one. Then we create the even filter object and last but not least, we are going to run all of them through the into or putting all of them into the sum built in. And this is simply going to calculate as we see on a one by one. This is actually a very nice visualization. So now we are going to run the sum function and the sum function first calls the transform function with an element of seven. So the first number, this is going to return 50, which is a square plus one. Now the is even function is given the number 50 and it's going to return true because the number 50 is even. And now we go over all the numbers one by one as we see. So it's a couple of steps I have to click through. I will make that a bit faster, but we see that there is no second list object and no third list object. So all the numbers are processed one by one. And what the sum function does is the sum function internally works just like the very first Python example in this course. It's basically running total. That is usually how program does addition for more than two numbers by doing a running total. And at the end we see a result of 292 and there is no other list object. And again, those function object and map and filter objects here we can disregard them. They don't really have big memory consumption. So that is the first step that you have to get used to if you want to work with big amounts of data using objects that model rules that know how to calculate the next object in line without calculating it yet. And then in this case the reduction step is the thing that drives all the elements. The reduction step is the thing that makes the rules materialize into real numbers. But we're doing that on a one by one basis and then in this case the sum function simply takes the transformed and filtered numbers on a one by one basis and simply adds them on top of a running total and then calculates the result. So super memory efficient. So I know that for beginner this may be a bit hard to swallow but I think it's not too hard. I would suggest compare it with the previous video again to see the differences in memory. We don't have a second list here as we see. And in the next video we are going to talk about the same example one more time and we are going to make it even more concise by getting rid of the function objects that we have to define here. So I will see you in the next video.