 In this video, we are going to review the same example of the two previous videos and we are going to rewrite the memory-efficient version using the map and filter built-ins in a nicer way without having to define a function. So we are going to create a new file and let's call it MFR for map, filter, reduce, revisit it again. And let's go ahead and create the same example one more time. So it's a list of numbers and it's going to be the list 7, 11, 8, 5, 3, 12, 2, 6, 9, 10, and 1, 4. And I'm not going to write down the task or maybe I write a short version of the task. So the task is first transformation. So transform, so maybe let's write it down. The task is to first transform all the numbers according to the rule y is x squared plus 1, just like that. Then we are going to filter out the odds. And last but not least, we are going to sum up the remaining numbers. Okay, so same example as before. And now let's do that without having to write a function. So previously I wrote a function to do the transformation step and it looked like that. Def, transform, function took an element as its argument. And then we simply wrote return element raised to the power of 2 plus 1. However, I'm not going to define the function. You see I'm not going to execute the cell. We're not going to use that function. This is just to compare. So how can we model a function without defining a function? So maybe you ask yourselves why would I want to do that? Why do I not just stick with a function? So remember that in order to use the map built in, we first have to pass some function that does the transformation. And then we have to pass, in this case, an interval, the numbers list that contains all the elements that are to be transformed. So in other words, for syntactical reasons, what we need is we need to have a function right here or syntactical reasons. However, what is the main purpose of defining functions using a def statement? Well, the main purpose of doing so is to reuse the function. So are we going to reuse the function in this example? And the answer is not really. We're only going to use the function exactly once and we are going to define the function. We have to define the function so far because the map built in needs to have a function object as its first argument. So wouldn't it be nice if you could somehow pass in the logic, the transformation logic without defining a function? And we can do that using a so-called lambda expression. So in chapter two, there is a video where I talk about how to define anonymous functions or functions that do not have a name. And we did that using lambda expressions. And maybe back then you were wondering why would I ever need that? Why would I want to define a function that has no name? Well, one situation where this is quite useful is in a place like here in the map built in and also later in the filter built in, they need a function for syntactical reasons, but we are not going to reuse the function anyways. So we don't want to give the function name because we are not going to reuse it. So what we do instead is we will go and define a lambda expression. And I will briefly review how lambda expressions work. So lambda expressions, they are written lambda. That is where the name comes from, lambda expression. Then we are going to mention all the parameters the function should take. So for now let's call it element. I'm going to use exactly the same names as above. Then we are going to write colon. So there's no parentheses here. The real function definition has a parameter list up here using parentheses. The lambda expression has no parentheses. We have a colon. And then after the colon, we are going to define one expression that is also going to be the return value. However, we don't need the return statement. So we are simply going ahead and we will say element to the power of 2 plus 1. This, if I execute that, as we see below the cell, will give me back some function object. So it's code in memory that can be executed, which is nothing but a function object. So really what I'm doing here is I'm creating a box in memory, an object that is of type function, and it contains all the function code. However, it does not have a variable pointing to it. There is no name in the global scope that can reference the function. But other than that, the object that contains the function's body is pretty much the same. It's exactly the same actually, except for a couple of technicalities. But it is really the same. So now what we often do is, when we use a lambda expression, lambda expressions, they are a simplified version of a function. So note how we can only have one expression that is automatically going to be the return value. So if you have a function that has to take more than one line in order to calculate something, it needs like two or three lines to calculate some result, then probably you're still better off using a function definition using the dev statement. Only functions that have basically one line that we can immediately return, these are functions that are good enough to be transformed into a lambda expression. And then what we do is to keep it more concise. Instead of calling the parameter element here, we're simply going to call it x. So we are going to say lambda is given... Oh, there was a slide here. So lambda is given some x as the input and it's going to return x squared plus one. So that's it. So that's the transformation. It's also basically the transformation we see right up here. It's x squared plus one. But the overall result would not be y here, but it's simply the return value of that. So now you may wonder how can you call such a function? Well, in the previous video in chapter two where I talked about anonymous functions, what I did is I put the lambda expression in parentheses. We have to do that for grouping purposes. And then I'm going to call the function using the call operator, which happens to be also parentheses. And let's say I give it two and I get back five here. So the two is transformed, it's squared, and then we add one to it. However, if I don't want to call the function, there is no need for putting parentheses here. So this is a function object. So this is an expression resulting in a function object that could then be called. And now what we are going to do is we are going ahead and we will copy paste that and we will place it right here where it says function. Okay, let's also get rid of the cell with the dev statement, which we never executed. And now we execute the map cell here. And this is going to give us back a transformer object. So let's call it transformer as before. And the transformer object, what can we do with it? Well, we can of course call the next function with it. And we'll see we'll get the next number, 50. So seven to the power of two plus one gives 50. We could also, as we saw in the previous video, use the list constructor and we get the list of all the numbers transformed except for the first one here. Why not? Well, the first one was already pulled out of the transformer object. And we remember from last video that the transformer object can only give us the next element in line. It can never go back. Okay, that's an important idea. So if I execute this cell again, I will get back an empty list because the transformer object is what we say exhausted. Okay, so what do we do with an exhausted object? Well, there's nothing you can do with it. You can basically throw it away. So let's get rid of this code cell and let's get ourselves a new transformer object. Okay, we have to create a new transformer object. And now we're doing the same thing here now for the filtering step. So for the filtering step, remember that I wrote a function is even, which also took an element and the function has an if statement in it in its original version. And it says if element modulo divided by two has no rest, then return true. Otherwise, so else return false. And then we made this function a bit shorter. Why do I need to make this function shorter? Well, as I just mentioned, a lambda expression can only consist of one expression that becomes the return value. So therefore we have a statement here. So the if statement is as we can guess by the name. It's a statement. It's not an expression. So therefore we have to make this function such that it consists of only one expression. So what we can do is, first of all, we can get rid of the else and unintended return false. This is what we call the early exit pattern. And then what we also mentioned in the last video was the condition of the if statement itself returns either true or false. So this condition is exactly true when we want to return true. So therefore what we did is we got rid of the bow of the two lines that say return. We get rid of the colon at the end. And also instead of the if statement, we're simply going to say return. So we're going to return the result of this one expression here, which is either true or false. And now in order to transform that into a lambda expression, what we are going to do is the following. I'm going to write lambda as a buff. I'm going to write element for now. And we simply go ahead and copy paste this expression. So that's how we can transform this function from its longer version into a single lambda expression. Also let's get rid of this function. We don't need it. I told you that I don't want to use any function objects here. And also let's get rid of the element here and let's call it X. And every function has its internal scope. And because lambda expressions also create function objects, these function objects also have the internal local scope. Therefore, the variable the parameter X here and the parameter X here, they are different parameters. They live in different scopes when we execute these functions. Therefore, we don't have to worry that we are reusing X here. So if you want to call this function, we can do our trick and we can wrap everything in a pair of parentheses for grouping. And then we can add another pair of parentheses for calling the function. And let's say I give it as a value to number two and it returns true. Why true? Well, because the number two is even, right? If I give it a number three, which is an odd number, I get back false. So we see that this function will always give us back true or false. So let's get rid of the calling operator here and get rid of the grouping parentheses. And now let's go ahead and put that lambda expression as the first argument inside the filter built-in. And now as the second parameter or the second argument that we passed to the filter built-in, we're going to simply pass, as in the previous video, the transformer object. Okay? And we get a filter object. And now what we are going to do is we are simply going ahead and we will put that in the same code cell. And we will call that even just like that. Okay? So let's get rid of all the other code cells. Let's go back into this code cell and now we want to finish up the example. We want to sum up all the remaining numbers and we will simply do that by saying sum up the events. And if I execute that, it says 292. Right? So here I don't have any function object that has a name. So what we can do here is we can go to Python tutor and one more time for this example copy paste over the code that solves the problem and you can then compare that with the previous two videos in how that is different in terms of memory usage. So now we are going to create first our numbers list and then we are going ahead and we will create a transformer object which is basically a rule in memory that knows how to calculate the next object in line without, that is important, without having calculated it yet. And then we create an events object which is a filter object which is also a rule that knows how to calculate the next object in line without having calculated it yet. And then last but not least, we are going to execute the sum function and the sum function is the reduction step and the reduction step is basically on a one by one basis asking the events object, hey, can you give me your next element, your next number? and the transform object itself will go back to the transformer object and will ask it, hey, can you give me your next number? And the transformer object will go back to the original list of numbers and it will pull out the next number in line. So therefore what we see here, the numbers in the list here, they will individually go through the first transformer then to the filter. So each number here will on a one by one basis go through this, we call it a pipeline. And only the even numbers after the transformation will come out of this pipeline. So if, for example, the number 8, the number 8 squared plus 1 gives me 65 and the number 65 is obviously odd. So therefore the number 8 will go through the pipeline but it will not come out here. And then down here we can imagine so to say the sum function waiting for all the transformed and even numbers that come out of this pipeline and then it sums up all the remaining numbers onto a running total. So internally the sum function works with a running total just as we did in the very first Python example of this course. So let's execute that and we see that now it says lambda here. So Python 2.0 shows that we are executing a lambda expression and we see that the numbers are transformed on a one by one basis and filtered on a one by one basis and of course it takes some time but we still have to go through all the numbers and transform them and so on. However we see at no point in time a temporary list object and at the end the end result will simply be result is 292. No second list, no third list as in the very first version of this example. And in comparison to the previous the second version of this example we don't even have any function object here. The function object is created by what we say slide so to say. So it is created once and then put right into the map and filter objects as the argument. Okay. So what else is there to be said? Well what you could do is if you still want to save yourself these two temporary variables here this is something you would not need to do because otherwise whenever you create a variable in Python this is a very cheap operation that will take no memory but let's assume you want to do that you could rewrite that just like that. We write the sum function and the sum function takes as its argument the filter object and the filter object here with the transformer takes as its argument another map expression here another map call a call to the map function a little bit nicer to read what we could do is we could go ahead and go ahead and break a couple of lines here so that we see the internal structure of how this will be constructed and now we see that this expression here in this cell will be evaluated from the inside out so first we take the numbers and we transform them on a one by one basis that is important. This is still the same as above using no variables for transformer and events after the transformation we filter that and then at the very end we are summing that up so you have to read this expression kind of from the inside out first mapping then filtering then reduction with a sum so let's do that and we get back to the same result 292 so that third video now basically concludes the discussion on the map filter paradigm there is an introduction to the more topics that follow in chapter 8 in the book materials so make sure you understand the map filter reading paradigm the first video in this video and the last two videos shows you how you could solve the problem without learning anything from chapter 8 so with everything you learned before chapter 8 and then the last video and this video show you how you can use Python built-ins to compare the map and filter built-ins to do the same exercise again in a memory efficient way and this is where you want to aim for if you want to go into the field of data science because eventually you want to solve interesting problems and that means you want to work on big data sets and therefore you have to know how you can design your calculations in a memory efficient way so that is the main reason why chapter 8 is in the materials of this course because this course allows you to become a data scientist to work with real-life data big data sets and therefore we should concern ourselves with how the memory works in detail even though Python takes away a lot of the load from managing the memory but some details are very worthwhile to know and knowing about map and filter I think is very much worthwhile so in the next couple of videos we will continue the discussion of what is called iterators the abstract concept that is behind the map and the filter objects so iterators are all objects in Python that function like rules like rules that can give me the next thing without having or that know how to calculate the next object without having calculated it yet so that is the big paradigm or the big topic of chapter 8 so I will see you in the next video