 Welcome back to the introduction to Python and programming course. This is chapter 7 part 2 So in the first part of chapter 7, we looked at all kinds of sequences. What are sequences? Well in Python a sequence is any data type that fulfills four properties and the properties are that we must be able to iterate or loop over the object We also have to do that in order So in other words the data type must be reversible and in other words that means we have a forward order as well a sequence is finite and Lastly any sequence is also abstractly speaking a container so it contains references to other objects What was the big or one big down side in the first part of chapter 7? Well, for example, we looked at the list type and the list type was characterized by holding references to all the object that it holds and all of the objects exist in memory simultaneously and Now you may wonder How can we model a sequence without? Expressing or without storing all the elements of the sequence in memory explicitly Well, we have seen one example already the range built in allows us to go over a range of integer numbers in one by one and we have seen that in this situation we only get the integer objects one at a time and We don't have them in memory simultaneously, which is a big improvement memory-wise and We have seen when we worked with numbers So for example when in chapter 2 we defined a function called average events the function was given a parameter called numbers and then This was a list of any numbers. It could also be floats and Then what the function did is? internally it went over all the numbers and Rounded them to get integer objects and then it went over the integer objects and Filtered them to only keep the even numbers and then finally we we took the average and We saw when when you go back to the recording of chapter 2 you will see that In Python 2. So we visualized one of these functions and when we caught the function We had not only the original numbers list in memory But we had also up to three copies of the original list and it was very bad so that means when we did the calculation back then we used a lot of memory and Yeah, the question is can we do better and the answer is of course? Yes, we can and we will do so by introducing Now formally the so-called map filled of reduced paradigm We have already seen the map filled of reduced paradigm briefly in the end of chapter 4 and Also in this chapter in the second part of chapter 7 we will look at data types that allow us to model data That doesn't exist in memory Okay, so let's look at an example Let's look at the first of the three steps the mapping step and we do so with our Familiar example would just take the list of numbers from 1 to 12 unordered And now we look at the first of the three steps mapping. So what is mapping? Well a map is regarded as a transformation That takes the elements of a sequence the x or so to say one by one and Transforms them into y's by some mathematical functions So let's see one example So here we have a function called transform it takes one element And what it does is it just returns the square of the element plus one we have seen exactly This transformation already before in chapter 4 So let's define this function and now the question is how can I? Apply this transform function to all the elements in the numbers list So let's see a naive way of doing that You know uses one of the list methods we saw in the first part of chapter 7 the append method So what we do here in this code cell is we create a new list Transform numbers which is empty in the beginning and then we loop over the numbers list and Within the loop we call every element in the old numbers list just old and then we call the transform Function pass it to the old store the result as new and Then we go ahead and we append the transform element to the transform numbers list So if I run this code cell after that The the list transform numbers will contain all the individual numbers So what that means is in memory and Here in the memory diagrams. I will Try to be brief. So I won't draw everything. I will just show you the big picture so to say so we started with a list and we know a list is an so-called array and This is of type list and the list contains references and Two int objects and I will just Draw them in a small way here. This is our original list called numbers And now what we did is we created a new list That was empty in the beginning and we call this list transformed Numbers and then as we went over the for loop we took the individual element that all existed simultaneously in memory and We transformed them and we created new elements that we put Into this new list here And at the end of the day This transformed list transformed numbers list. I will zoom out a bit here. This transformed numbers list is Contains just as many elements as first numbers list. So now we have doubled our memory Okay, that is mapping And now the question is Can we do that in a nicer way and the answers of course? Yes And how do we do that? We use a python's map built-in so we will find it in the documentation of all the built-in functions and if we look for M map here and So let's see what what does map do it takes At least two arguments possibly more and the first argument is a reference to a function So in other words, it's something callable And then the second argument must be an iterable it doesn't say it must be a list or something It just says in abstract terms. It must be an interval object And then it it says the documentation says return an iterator That they play that applies function to every item in iterable So as of now, we don't know yet. What is an iterator? I will introduce this word At the towards the end of this chapter But iterator is also an abstract concept just like iterable So let's continue in the presentation So as I said map takes two arguments. The first one is a callable So we we just passing the reference to the function. So by the way, as you may remember the function transform is Basically here we have an object of type function and we have the name transform and this also exists in memory so functions are also objects in memory and Yeah, so this also exists and this is why we can pass it as an argument So I'm passing the reference which means here the reference to this a function object That's what I pass to the map built in and also of course a reference to the numbers list and whatever I get back From that the iterator. I store in a variable called transformer Okay, so let's see what is transformer and It just tells me it's a map and what we see there if basically the first word here means that's the type so it's an object of type map and This here in the hexadecimal notation here. That's basically The location in memory where this map object is stored. So this is we remember that the angle brackets here They by convention mean that I cannot copy paste this back into a code cell. This is not a literal so in other words I Cannot it's not a literal just like for example a numeric literal. I Can create map objects only with the the map built-in function So now the question is Well, what is transformer transformer is a map. So basically what I got here in my memory diagram Is I got a new object? I draw it down here. It also has something in it and this is of type map and Now I have to write a bit nicer. Of course. This is a Transformer So this is transform the function and this is transformer the map object So this is what we have done in memory And now the question is what can we do? With the this object, so let's for example, we can use the next built-in function So let's check in the documentation what the next built-in function does So the next built-in function as we can read it takes as the first argument and Interpol an iterator so not an interval but an iterator and then It the documentation says retrieves the next item in the iterator Okay, so Basically what that means is if I now execute this code so I get The first number that is transformed the 50 so I'm asking to transform And what transform does is it will give me back? Exactly one integer object and then it is up to me as the programmer to decide what I will do with the 50 and In this iteration, I didn't store it So the 50 is just given back as output and it is basically immediately forgotten I didn't I did not store the 50. I didn't capture it for the return value And now if I execute this last code cell again, I Get back the next number the 122 and if I go back in the slides These are exactly the numbers 50 122. These are the exactly the same numbers that are in the transformed numbers list However, I'll now just get them back on a one by one basis. So I'm basically getting back I'm producing these objects here on a one by one basis And now let's see what what happens if I go get to the end So I execute a cell a couple of more times and I see I go over all the numbers one by one And then eventually when I reach the end I get something that is called a stop iteration error and the stop iteration Stop iteration error indicates to us that now the map object the transformer object doesn't know how to give me the next integer Why well because it's it just ran out of integers and now I can keep running this cell again and again And I will always get a stop iteration So the map object now the transformer object from now on is useless So all it does it allows us to go over the numbers on a one by one basis once and once we have done that This is we basically have to create a new map object Of course, we could do that I can go back up and I run the cell again. I create a new transformer brought object and now I Can go over all the numbers again Until I hit the stop iteration error again Okay, so that's basically like a for loop so in other words What the map object gives me back or the map function gives me back It gives me back a map object and what is a map object a map object is basically like a for loop Just that it has not yet run It's like a lazy for loop. It only runs one at a time when I make it run and Sometimes we say as programmers that I have to drive the object. I have to Drive basically through the loop so to say manually one by one So what can I do with it? Well, I can use the map object at a map built in here Just as I used on the slide before and I can wrap it with the list constructor And what this will do is this will create a new map object That knows how to calculate the numbers one by one and then the list constructor the list constructor Basically does what I just did manually it just goes over the map object and gets out the individual numbers and then stores it in a list And now it and saves the list as transform numbers So basically what I just did is I regenerated this list here using a map object This is what I did. I regenerated this and whenever we go from an object that Basically resembles a rule in memory that allows us to calculate individual items or objects and We go from such a rule to an object that actually has all the objects in it that are to be calculated Then I call this to materialize. So in the first situation I don't have all the simultaneous objects in memory, but then I materialize All the individual int objects in this case into a list So materializing is the term that we want to use here And what can I do with transform numbers? Well transform numbers is of course just like a list because we just created a new list that's mapping and now as see The the other two steps so now filtering The second step in the map filter reduce paradigm and we will see But this step works basically just like the first step. So here I define a function I call it is even and the function again takes one element and then it checks if the element is Even we modulo divide by two and check if there is no rest and if there is no rest We know that the number is even and in the case where the numbers even I return true and in the other case I return false. So this is also an example of the early exit pattern Okay, so this function takes an element and returns a true or false How can I apply? This function to all the elements in a sequence. Well Let's do it first in our old naive way Well, at first I take I create a new list called even numbers that is empty in the beginning and then I loop over The transform numbers. So this is the materialized and transform numbers and for every number I Pass the number to the is even function and again this function gives me back a true or false So this is why I put this function as the condition of an if statement into this line here And in the case I get back a true I have an even function an even number and in this case I append the number to the evens number list Okay, so I run this code and after that and have an even numbers list that is now shorter But it's still materialized. So what happens here effectively? It's now I still have here the list object and this now has a couple Also a couple of integers But it's now shorter, right? So that's now even numbers. So let's write here. Maybe even numbers and Of course, there is a reference going here So now in our computer's memory, we have the original numbers list up here We have to transform numbers list here and we have the even numbers list here So we have three lists in in total and in the worst case scenario Which means if all the numbers in the original list are even then I would have Exactly three times the memory usage as compared to the original list on its own and that's bad, right? I mean, of course, it's the worst case scenario because this list is most likely going to be smaller But in the worst case this list is not smaller now the question is how can I Do that with an object that is similar to the to the map object Well, we just use a The filter built-in function and the filter built-in function works just as the map built-in function We pass it a reference to a function first. So is even And then I pass it in this case a sequence a list however, as we will see in the documentation Where is it? It takes an iterable So it doesn't matter if we give it a list or it's all a tuple or something else as long as we can iterate Over the argument. That's enough So See how nicely you are now able to read the Python documentation because again the Python documentation focuses more on the behavior of an object and not on the on the type so now I execute this code as sell install the results and events And now what is evens? Well, evens is a filter object and that works exactly Like the transformer object down here. I don't throw it in the memory diagram But what we know is the filter object is just one object in memory that spits out one by one the elements so Transform numbers just for a reference. That's the list of numbers that exists materialized in memory And we see that we pass transform numbers to the filter objects So let's now go over the filter object with the next built-in So I get the next even number and the next even number is of course the 50 next even number would be 122 so I get back to 122 and our next number 65 this one is skipped. So I get back to 26 so the filter object just loops over the Over the sequence we gave it one by one and it only gives us back the numbers that fulfill the search criterion which was is even so Also for this object, I can loop over it until the end now 82 the next one will be 2 So the 101 is skipped and now there is technically one more element the 17 in it. However, this will not get Given back to us because it's not even so when I now Hit enter again. I get a stop iteration error because the filter object is now out of out of elements Okay, so now we have seen two ways to do transformations mappings and Filtering's without, you know having a copy of the data in memory So that's a huge improvement and this is important to know When you want to work with big data and Whenever we are given such a filter object or a map object and we want to really materialize all the elements into memory Simultaneously, we can always just wrap the function With the list constructor Now we get back the filtered list and this would be The list here called even numbers and all the elements exist simultaneously Now why is and this why is this notation nice compared to the for loops? Well one way That makes it particularly nice is I can now stack together the map and the filter so In order to do both for loops in just one line of code I can write this expression here and evaluate it and I get back the same result So here starting from the original numbers. I first Do the mapping and then I do the filtering second and then once the filtering is done the list Constructor here does the materializing of the individual objects. So maybe This one line may be hard to read and whenever you find yourself, you know in a With with a line of code that is maybe a little bit too nested. So you cannot really see right away What's going on you can of course? Write it in a bit nicer way. So one nice way would be to do it like this So let me just like this so and then usually Whenever we write something in this way you would for for stylistic reasons put a comma here and put the Parenthood the closing parentheses on the next line And so now we can already see that we evaluate this function calls from the inside out So, you know at the end of days just a composition from of callables. So first we do this Then we do this and then we do this and this of course is also legal in Python. So White space as we know is mostly ignored by Python Only at some places for example in for loops or if statements We do need to indent by four spaces or any number of spaces. So white space sometimes means something in Python But as we see here In this case most of the white space most of the line breaks is actually ignored So this is exactly the same as not doing the line breaks and now Why is this a way of writing nice? Well first I can express two loops in just one line of code But second I could also just go ahead and switch the two for loops, right? So Let me also make this a little bit nicer so If I go ahead and write it like this I get back a different result and the second result The second code cell is actually the exact same numbers that we saw first in chapter in chapter four and When we also when we first had a first little example of the map to the produce a reduced paradigm So I tend to reuse my examples. So That's why I have the example here that we already saw in chapter four and Again here. What is nice here? Well, I can just basically switch the loops and I can do that in one line of code and everything is very compact and these are stylistic and You know a code maintenance issues, but there's also another issue, which is of course speed As we have seen the follow-up is definitely smaller. It's definitely Worse because it needs more memory here. We don't need this intermediate memory and also what we maybe maybe I should Do that again. So what happens? Let's let's look at the new memory diagram to see what Basically one or basically both of these code cells do well first we create a list of numbers again and this list holds 12 numbers 12 int objects and This is called numbers and then what happens let's say I go ahead and Assign here this here to the to the Variable called result. So what now happens is as this gets evaluated what happens is the right-hand side of the assignment statement gets evaluated first and Then only the result is stored. So what that means is what happens first is the filter object is created So let's put it somewhere like here. We have the filter and then the filter object All it does Maybe I use a different color It is given a reference to the function So how do we do this? That's basically let's put two more objects here. So we have the function So here's a function which is called Transform and then we have another function which is called So funk The other function is called is even so that's what we have in memory and now what What happens to this filter here the filter is basically all it does is it really holds like a reference to this is even a thing Function and also a reference to this object, but it's not yet executed So in other words the filter object only knows how to produce new numbers out of these numbers That's the source basically with this function But it doesn't do anything right so and then what happens next in memory is the map object is Produced so let's let's put that here So we have no idea what's inside here. It's only ones and zeros. So this would be the map and What the map knows? The map audit knows is Maybe lose a different color here the map is attached To the transform function because that's what it what it's given as the first argument and it's also attached To the filter so in other words Let's maybe write In a descriptive way here source and here is the f and I mean with source That's where the data is coming from so this would be also the source so to say and this would also be the function That is applied so this is now an intermediate state in memory So this is what happens if the innermost line that the the lines of code that I just marked here When they have been evaluated this is the state in memory that we have So see nowhere do we have any? resulting Numbers so in other words So far nothing happened and what we see here is what we also call a lazy evaluation So in other words the objects that know how to calculate things the rules so to say they exist in memory But they haven't worked yet So they are lazy they are waiting for an instruction and then we pass The entire expression here to the list constructor and the list constructor is what we would call eager So the list constructor says well, I have to create a list. I'm sorry. You have to give me all your numbers and so the list constructor goes ahead and Creates a list and let's maybe also put the type here list and also here list So the list constructor creates an empty list and then what the list constructor does is the following It talks to the map and it says because we passed this map object. We passed as the argument to the list constructor So the list object as the map object. Hey, give me your next element, please And then what does the map do the map says? Oh, I don't know. I don't have any elements However, I know how to calculate them. So what the map does it looks at its source Which is the filter object and then the map object says hey filter object. Can you maybe give me an element? And then the filter object says, okay, wait a minute. I don't have an element in me but I know how to calculate one and then the filter object goes ahead to the original list and Says hey list. Can you please give me your next element? Which is the first one and then the list goes ahead and gives the reference to the first element to the filter and then the filter goes ahead and applies the filter function and Because the first number here is actually a seven The filter will say or I don't like this number. I cannot return it because the filter says I should not return the seven because it's not even And then the filter object says hey list object Can you give me your next number and then the list object says well no problem? I give you the next number and At some point the list object will give back the filter object an even number and when that is an even number then Is even function will confirm to the filter object well the numbers even you can spit it out And then what the filter object does it it gives the even number to the map and then the map says okay, now I got an element to work with so the map goes ahead and checks back with the transform function and to to know how to transform it so the map object sends the Even number the individual even number to the transform function the transform function Creates a new function because it's mapping the element gives it back to the map and then the map finally gives the The filtered the even and also the transformed number to the new list object and then the list constructor Just places it here and then after that the list constructor goes back to the map and says give me your next element And then the map says what I don't have one so it asks its source and then the filter object also says what I don't have one and it asks this list and Then in this way on a one by one basis the objects that are in the list up here They move wired a filter through the map into the new list object so and then at some point we have a list object here, which Possibly has less elements than the first one And then what happens is once this list this list here is done once the list constructor has created the list then in the code the left-hand side of the assignment statement is evaluated and We write the name result in the list of all names and Make it reference the list and then what happens is quite interesting because we don't have a reference to these two objects What happens is they just go away They are just deleted removed because we don't have a reference So in other words the garbage collector just collects them right and then what we do is what we did in this example We went from this list to this list right away with no intermediate list. That's the important thing So the memory consumption that is needed to go from this list to this list is minimal Okay This is basically all the the paradigm is about it's about saving memory and Being able to work with big amounts of data later on. Okay, and now comes the third step The third step would be the reduction step reducing and What reducing means we have also seen this before this means Then that Let's look at the original picture again so we now have this list called even numbers and We now want to do something with the individual numbers We want to reduce those elements in the list into one summary statistics. So to say that's what the reduction step does so here What I do is I pass The the number here this expression That I actually had before so it's actually this is the exact same Expression as this one here as the inner one that I just passed to the list constructor This now gets passed for example to the sum function to the built-in sum function and we can also of course Make this look a little nicer Like this and now the only thing that has changed is that instead of list here We write some and what does the sum do well? It just sums up all the elements in other words maybe In order to stay maybe to stay in this example here that we just did Here we use the list constructor to create this list here out of the original list But what happens now in the code here is we go from this list via a filter object and a map object To an individual number so we don't create a list, but we eventually spit out a single number Okay So this is what happening what's happening here and then let's see two other examples of a reduction For example, okay, creating the minimum or the maximum. This would also be a reduction So whatever operation we use to reduce Sequence of numbers into a single number. This would be a reduction and some min and max are typical examples Okay, so again, I just summarized everything you see here does everything we have seen before in chapter four Just without any intermediate list As efficient as possible Okay And now let's see how can we write reduction functions on our own? So we have written the transform function and the is even function. So we were able to Define our own mapping and our own filtering and now I want to define my own Reduction and I don't want to use the built-in reduction functions some min and max But so what we do here just for illustration purposes to understand how reduction works We will Reimplement the built-in some function on our own and We will call this function Some odd for alternative some however at the end of the day this function Will do exactly what the built-in function does so this is not necessary to or it will never be necessary To write this function in production code But this is just for illustration purposes because I think Summing something up is something easy to understand. It's the easiest kind of reduction that there is So let's see how does the reduction? What is reduction? How does it work? Well reduction is a is a function that takes two arguments So before transfer mapping and filtering only take one argument but here a reduction function takes two numbers and The first element at the first argument is called some so far and the next one is called next number So what we see here, and then what do we return which we return the sum of the two? So what that is in a way? This is kind of like a running total so we have an intermediate total and we add on the next number and then we return the Cut the old running total plus the numbers over we return the new running total in other words This is a function that at the end of the day as we will see Will incrementally update the running total just as we saw in the very first example in chapter one So let's see how can we use that? so What I do here is I put together the filter and the map and I store the entire object into an object called evens transformed. So maybe We will also Do that in a memory diagram So, okay, we have our original numbers list again. So here we have and so on type is list and That is numbers So now what happens is? We Go ahead and we create a new filter object So the filter object, let's put it for example here. So this is the filter and the filter Has and also maybe let's put it back in the two functions Is even so where do we do that? Let's put it here. We have is even here and transform. So is even is a function a function object and Then we have a second function object Which is called transform and now what happens is the filter object we create is Attached to the numbers list again. That's the source and The filtering is basically connected to the is even here That's the f that goes to the map to the filter and then we create a second object and This will be again of type map and the map Let's use another color again. We'll use the filter as the source and Transform function as the f and now What we do is we store that in a variable called Evens transformed. So I will write here evens Transformed and I don't have to create a new object because the object. I'm referencing is already created. It's this map So this is the situation we have in memory And then in addition to that I will now create a new function Let's put that here and this function is called some odd That's our own Implementation of the or re-implementation of the sum of the built-in sum This is the memory situation we have and now I forgot to Execute the evens transformed and now what can we do? How do we use the some the some odd function? Well, here is just an intermediate step only for deductible purposes. So in order to use a Function like this that starts with a sum so far and always adds on a next number This needs to have a a starting value, right? And the starting value could of course be zero if we want to sum something up But we can also take for example the first number that comes out of even transformed So here the first number that comes out of even transformed we store in Then in in the variable called result and then what we do is we loop over Even transformed and what the for loop does is it basically as even transformed the map object to give out the numbers one by one However, because we've already taken out the first number the follow will only start with the second number and Then inside the for loop I print out something for deductible purposes as we seen before and Then within the for loop what I do is I call our sum odd function and I pass in as the first argument the result and As the so the result is just in the first iteration of the loop the first number and then as the second argument I pass in the current number in the in the loop in the current iteration and whatever the sum odd returns Which is the you know the running total so far Will be stored in result and then when we loop here when we go back up here and go through the second iteration the loop Then result is not just the first number But it's the sum of the first two numbers and then we pass in the sum of the first two numbers into result And then we add on the third number that we get from the loop and We go on like this. So this follow is just is basically looping over the filtered and transformed numbers Without the first element because we just got it out before and if I do that what happens is The you know the the result Is basically calculated so we can also see the total result is also 370 370 is the result We have seen before if you don't believe it, then we can just go back and We see here is a 370. So The our alternative in the implementation works. We also get back the same result here and The important point to understand is how does the sum function work? Well, the sum function works such that it always is given two arguments The first one is an argument that is some so far and the second one is the next number We want to add on in other words as we calculate the sum We only have to be we only have to track two numbers the sum so far and the next number And this is the trick that we apply so that we don't have to store all the numbers simultaneously we just always Just only store an intermediate sum and plus the next number we get from the map object And we just add it on top. So something is really so all sums all summing is in computers Also for programming is always a running total under the hood It's always we because we can only add two numbers and if we want to add ten numbers We have to add two numbers in a pairwise fashion until we added up all ten of them So that's how summation works in a reduction or that's what a reduction step is and the reason and the reason why I show This alternative implementation is that this is the generic way of how to implement a reduction function So any reduction function will always take two arguments and Return the argument that will then be passed in as the first argument in the next iteration of the loop. This is reduction Okay, and now there is unfortunately no reduce Construct in the in python so at least in core python so what we do is from the standard library there is a module called func tools and From that we would we import a function called reduce and then what we can do is We can use the reduce function just as we use the map and the filter built in functions So first in this line of code first We filter the numbers Then the filtered numbers get transformed and then the transform numbers are reduced to the sum So we can also write this in maybe a nicer way Okay, and then if I execute this code Of course, I get back 370 Okay, so the only difference between the built-in reduction function sum and our own reduction function here is that we use the more generic way and With the approach we use we can use we can basically code up any reduction function we want so any function That is capable of taking a sequence of numbers and reducing them into a single Statistic and how do we do that? We all we need to do is we need to find a logic as to how to implement the reduction Norge logic into a function that takes two arguments. This is the generic way of doing reduction so this may be out of the Out of the three Steps map filter reduce the reduction step Maybe the hardest to understand first when we want to do it in a generic way But really what reduction is just think of summing minimizing and maximizing Okay, so This is an introduction to map filter reduce and now throughout The next couple of sections we want to fine-tune the map filter reduce and look at Other data types that can also allow us to work in in a similar fashion and But first let's look at lumped expressions and lumped expressions Is the last subsection of the map filter reduce paradigm? It helps us to make the three steps that we just looked at even more concise Now what are lumped expressions? Lumped expressions if you have read chapter 2 you already know what lumped expressions are I Skipped lumped expressions in the presentation of chapter 2 because chapter 2 was already quite lengthy and Back then I did not have an application for lumped expressions And but now comes the application But why are lumped expressions part of chapter 2 well lumped expressions at the end of the day It's just a syntax to create a function object that does not have a name referencing it So let's look at an example So what I do here is I Provide a lumped or expression That does the exact same thing as our sum alt function So what what does it look like? We will also Look at the memory implications of course So what is the lumped expression lumped there is a keyword? So it's just like death for example just as we define a function object with death We write lumped and then what we do is We list the parameters of the function separated by comma and We end the parameter list with a colon. So here's the column. So that's the information That usually goes into the first line when we define a function now. What's missing here? Well, what's missing is obviously the name So when we define a function we give the function a name and then we provide the list of parameters We specify for it and here we only provide the parameters without the name for the function So in other words what a lumped expressions intuitively Well, they allow us to create anonymous function objects functions without a name. So you may wonder The whole point of having functions is to reuse them, right? That's what we learned in chapter 2 so we define functions to give names descriptive names to Fragments of code that we that we write and the main purpose of functions is not only to give names to code But also to reuse the code and now I'm creating a function that does not have a name So how can you reuse something? That does not have a name and the answer is the Lumpter expression syntax gives us a way of creating a function in a context Where we don't want to reuse the function. So whenever we need a function syntactically for syntactic reasons But we don't want to reuse the function. So we know ahead of time We only want to apply a function once and then we will not reuse it Then maybe a lumped expression is a better way As compared to a dev statement because then we don't have the side effect of creating a function object that sticks around even though We won't ever use it again and then How does the lumped syntax continue well after the colon? We write the actual code from the function's body But the the rule is We must only write there one expression So when we define functions with the dev statement, we can put as many lines of code in the body of the function Here in the lumped syntax We can only Write functions that consist of one line of code basically So here I'll just say some so far plus the next number and then also note in the original Definition let's go back Where I reduced where I defined the sum all function here. I have a return statement So that's so I return the this expression in the lumped syntax We don't see the return and the reason for that is because Because lumped expressions only allow allow us to write one expression in the function's body This expression will also be the return value So, um, yeah, that is just the return value here So lumped the the lumped expression syntax is just a syntax to write to create functions without a name That are very short We cannot we cannot write long functions with it. So let's create one What happens? So I show you in memory what happens I just evaluated this code cell So what this does is It creates An object with code in it Of type function. I just write funk for short, but it's type function and now With the def statement what we would do is we would write the function's name here and make a reference But however here I don't have a reference because I didn't create a That function with a def statement the function does not have a name So what would now happen after the the code cell is executed? But what happens is This function gets basically forgotten right away. Why because we don't capture it in a variable okay so The garbage collector at some point gets rid of it So now usually when we write lambda expressions We don't want to use such long variable names even though The best practice when you write functions is to use parameters and also variables inside the function that are self-describing With in the context of a lumped expression the the default is we just use Short names. So in this case, I write the exact same function by just Renaming the parameters into x and y. So if I execute this What that would what that would mean is another a new function object will be created in memory And we don't write a name here so now maybe Just to show to you that I can actually call the function How can I call this function? well, let's Copy paste the function and now I go ahead and I do a trick I put The entire expression within parentheses That's the grouping operator And now I put another pair of parentheses and what's that that is the call operator And now I want to add let's say two numbers. Let's say one and two And I execute this and I get back three So what have I done now? I have created a function object So as we say on the fly and I have immediately called it with two arguments And then I get a return value from the function call And after I got the return value the function object itself is gone It goes away because we don't have in reference to the function object So in a way What do you see here? If I can create functions without a name but still call them even though they don't have a name So and in what context do I use them? Well, I use them of course in the map filter reduce context to basically As we saw here in the Where is it? Here is one of the earlier pictures And we have here a function called if even and we have another function called transform and down here I have a function sum alt and these functions all exist and they all have names And now what I want to do is I want to create the same diagram without names to the function objects Okay, so let's do that So or in other words What how do I so how do I rewrite the entire example without giving names to the functions? Well, here's the first example So I take here. This is the entire example. This is the entire logic of the example that I used In this first part here of this lecture today So I create the list numbers New again just to show you that This fourth line is also here. So if it's a complete example and then in the second line I write filter But note that I don't pass in a reference to is evens here instead I pass in a lambda expression An anonymous function and the function takes of course only one argument This is the element And then I return either true or false because a filter function has to return either true or false How do I do that? Well, I just write there Um a boolean expression. So I divide X the only argument to the function Model or divide this by two and check if there is no rest And if there is no rest then I know that X must be even in which case I just return The the value to which the the focused code now evaluates to which is of course true Right. So just to show you that If let's say I have the number six and I model or divide by two and I compare it to zero I get back a true and if I Model or divide the seven by two I get back a false. So this is true or false. However, we have also learned That numbers evaluate Or behave as if they are true or false in a boolean context And referred we refer to that as truthy or falsey in chapter three So we could also exploit this but here I'm a bit more explicit So I just compare the result of the arithmetic operation to zero and that is then a boolean expression So I get back a true or false here and I don't need to return keyword here because the only expression in the lambda Functions body here will be returned Okay And then the second argument to filter is of course numbers So this is basically exactly the same as if I wrote this Is even that's exactly the same But now I don't have in this new formulation I don't need to create a function with a name And pass this name to this filter function and then never ever use is even again So in this situation, I only create the lambda function on the fly Use it for filtering and then immediately the function will be forgotten And then in the next line So the result of filters I store as evens and in the next line I go ahead and I use evens here And in the map object or in the map function and the first argument to the map function is of course also a function that accepts one argument And we also call that x here, but it's a different x than above And then I return x squared plus one. So I just do the transformation And then in the last step I just want to sum everything up So I call the built-in sum function and I pass it transformed So in other words The numbers list here this create this creates 13 objects in memory one list pointing to 12 int objects And then the second line and the third line. They only create One object each and the objects are of type filter and map and these are rules in memory That somehow know how to filter and transform the numbers, but they don't they don't do that So up until everything that is now above the empty line No calculations have been executed So no filtering has happened and no mapping has happened And then once I pass in transform to sum so up until here We would say all of this here is lazy. It hasn't done anything This is an example of lazy evaluation. Nothing has happened And then comes the sum function the sum function is what we said What we described as eager to some function eagerly drives basically The lazy objects up here to go over all the numbers filtered and transformed one by one and sums them up And we know we remember in the background summing a sequence of numbers is basically just an iteratively updated running total And when I execute this I get 370 so this is now um Basically two for loops. No three for loops. Sorry. So uh eat the Except for the first line, of course every line Implicitly has a for loop behind So this is like three for loops that I'm writing here without writing for But it's not the forward we care about it's about the implications of memory. So this is memory efficient, but the question is is it Let's look at Another formulation of this. So what what is different here? Well, because I know and that's just Maybe it's cheating a bit, but just because I know the numbers up here are the numbers from 1 to 12 They are just unordered But I know that it's the numbers 1 to 12 and no number is missing what I do down here is I I replace numbers By the range object from 1 to 12 So the left index is including the right index here is excluding. So these are the numbers from 1 to 12 in order So, okay, um, I go from unordered numbers to ordered numbers But everything else is the same and what have we learned about range already in chapter 4? We talked about how the range object is a rule in memory that can give us the numbers one by one Without holding references to all the numbers simultaneously So this is basically a memory a memory less way of going over to the numbers 1 to 12 it's as as efficient as it can get in memory And then the second and third line They also don't cause any harm in memory. So nothing will be materialized in memory and then the sum function just eagerly drives the entire Object or the all the three rules that are stacked together. They are basically driven by the sum built in However, the sum built in is also memory efficient because it only keeps track of two numbers at a time Namely the the running total so far Plus the next number in line So in other words, this is as memory efficient as it gets and if I execute this I of course get 370. So now I promised you that in chapter 7. I promised you now in chapter 2. Actually, I already Showed you when we first Looked at the average even function I promised you that I will show you a way of how to do all the arithmetic without having any without using any memory And that's the way to go So now you may wonder at what point in your programming career should you start doing this? I think This is Not so trivial I think the example that I've chosen can be easily understood But if you want to apply the map filter reduce paradigm in practice, it's not so trivial So in your first a couple of coding exercises regarding this you will probably just use a list or a tuple or whatever And it's more about getting the code to work and understanding all the looping concepts We have looked at but then eventually When you want to scale up whatever code you write to big amounts of data Then you should think about replacing four loops With the map filter paradigm. You're the map filter functions so in other words, I think by now most students in this course they are Super familiar with the fourth statement or the for loop. They don't like the why loop so much Even though the why loop is also rather trivial But the for loop is usually your go-to thing when you want to Repeat some code, but at the end of the day I think a good practice is is to get rid of four loops and when we are in chapter nine and talk about array and data frame data Then It's also there's also this best practice that anyone everyone keeps telling you that when you work with Arrays of data and data frames of data You should not use the word for the fourth statement because it's inefficient and usually it's inefficient for two reasons and Both memory-wise but also computationally the speed for of the fourth statement is also not so good as here for the filter and map Functions so at the end of the day This is what you should aim for in the long run But in the short run when you are still studying I think this could be a little bit too hard But maybe you have understood it now everything and you can just Start working with these concepts and now also just to show you How these concepts can be visualized so whenever We want to visualize something. I will show you How we can do that in a nice way in python tutor. So here python is the very first Implementation the naive implementation the implementation that you would have probably done If I had told you to solve the problem. So here we define two functions is even and transform We generate a list called numbers. It's a list object. So it contains the numbers 1 to 12 materialized Then we create an empty list events then we loop over the numbers then we append to events And then in the next line we create a list transform which is empty in the beginning We loop over events. We append to turn the transform numbers and then at the end we sum up So we already see it's 116 steps. So I go over it Rather quickly. So first let's create the function objects Then we create the materialized list and now we create the empty list events and now what we do is We go over the for loop here And then eventually We see that the list slowly builds up and this is of course the filtered list So the list slowly builds up and in the worst case scenario if we are given only even numbers This list would be as big as the the first list and now we go to the second, um, you know loop So now the transform the transform list builds up And um Yeah, and then so maybe just shortly before the end now We have three lists in memory and in the worst case scenario The second and the third list are just as long as the first one And then we go over the last list and we reduce it to one To one number, which is the 370 which we have seen before so that's the memory Implication so even with pys and tutor you can actually see what goes on in memory and you can actually tell that The way we solve this problem here is not memory efficient and then to contrast this Here is the um Last example that I showed you in the presentation And now let's see what happens in memory here Well, in the extreme case, I just used the range built in without Materializing it so there's no list around it. So we just have the range object here And then we build up a new object called events, which is of type filter. So we have a new object events And then in the third line, we execute we now create a a map object called transformed. So it's also a map here And so far the interesting thing is nothing has happened in memory This is like we we don't have a single number in memory at this moment, right? We only have the the range object, which will be the primary source for the data We will have the filter object that knows how to filter the source And then we have the map object which knows how to Transform the filtered numbers, but at the end of days so far in memory. We don't have a single number And now comes the the eager Some function that the reduction step which basically drives all of the calculations and now we see here An own namespace a local scope Which goes by the name lambda. So what that means is Whenever a um a lump the expression is executed Then at the end of the day, this is a function that is executed Which is why we see a function scope here, but the function does not have a name With you know, we don't we didn't name the function here is even or transform and because of that this function is anonymous And python 2.0 shows this with just this the lambda symbol here So now we can go over this and we see how the loops go here, but we see that so far Nothing has really materialized, but this is actually a wide lie. So Yeah, no, I'm sorry. This is actually absolutely correct So nothing has materialized so far so The only thing that some does it keeps track of the running total and the next number in line which gets from the From the rules up here So at at most we have two numbers in memory simultaneously And then when we reach the end We will of course also get 370. So we we just saw a way Where with at most two numbers in memory we can calculate the result to be 370 Okay, so let's continue in the presentation. So Let's go on or maybe We okay, maybe sometimes you also write that in one just one line here one expression, but then what you also could do Is we could also make this look a little nicer No, this was the wrong parentheses so could maybe write it like this and Or even to be a bit more extreme So this maybe Is the most extreme version of writing this so we see from the inside out We take the numbers one to one through 13 We filter them we map them and then at the end we sum them up. This is all only one expression We don't need any temporary variable here even though Using temporary variables here is really not bad because we we've seen that temporary variables or any variables in python are just references So they are super cheap to create. Everything is just fine But also we get back a 370 here Okay and now That was the map filter reuse paradigm And now we look at something that we have also seen before in the first chapter Which is called a list comprehension And list comprehension are one way of why python oftentimes looks very similar to math. So Let's see an example We start with the numbers list again And now We want to do both the transformation and the filtering in one step Can we do that in one follow we could actually And we have done that in chapter one So we loop now over the numbers and then in every iteration of the loop I filter for the even numbers and only the even number gets transformed here And the transformed number gets appended to the list evens transformed So let's do that And then evens transformed contains all the even and transformed numbers And these are the same numbers that we saw in chapter four already and now um How can I rewrite this well? I can rewrite Um the entire for loop. That's an explicit for loop here with the fourth statement I can rewrite this as a list comprehension And what the list comprehension really is Whenever you find yourself in a situation Where semantically speaking We want to express the idea that we start with a list And we want to derive a new list out of an existing list then we do that So we write uh the brackets here But within the brackets the literal notation for the list. We don't just write um, we don't just write the Individual elements, but we write a an expression that allows us to calculate the individual elements and basically What this means is we have a four number in numbers and then we have the if here and so We just use the same logic and we all we do is we write the transformation the number squared plus one We take this and put it in the front and here I say n squared plus one to make it a bit shorter And then comes the for loop and we write it as for n in numbers And then comes the if a statement the filter and we write it as if n Motto divided by two equals zero and if I execute this I also get the same list So, um, this is how we get a list comprehension out of a loop We just use whatever the expression is that we want to append to the new list And we take that to the front and then we write the for loop here And then if there is an if statement that does not have to be one So there's also is also possibly the case that we don't want to filter something out Then we'll just leave away the if here this also works without here, but this is like the full version and whenever this may be Confusing we will see an example of where we Have two nested for loops shortly One thing I can already say is whatever is the logic in an explicitly written for loop Is the same order in in the list comprehension people sometimes get confused as you will see later But it's really just copy pasting the explicit for loop down within the list comprehension But now this has two advantages one is this here is a little bit faster than this implementation Why is that well primarily because when we create an empty list what python does in memory It creates let's say There should be around eight slots And then what python does is or what we do here is So we create an empty list and then we append a single number at a time and let's say what? Let's say the following happens so we have we at We put in more references here And let's say now the the array that holds all the references is now full And now let's say the for loop is not over yet What happens is the dot append will be executed one more time And then what python does when it runs out of space in the list It does not go ahead and make the list larger. This does not work like this unfortunately So what python does instead? It goes ahead And creates a totally new list with a longer array and will then go ahead and Create new boxes or down here or basically In this case, it will actually go ahead and just keep those boxes And then also create new boxes right and then python will automatically get rid of the the smaller array And will only keep the the bigger one and the reason for that is python in the beginning just does not know How many data will go into the list and it has to make a guess and the guess will most likely be wrong so because of that Because python in maybe has to like Copy all the elements that are already in the list again because the list is too short This is a big reason why the first line of code the first code set here may be Slower in practice That's the first reason why I don't like this Code cell and the second one is Really when When we try to express the idea that there is an old list and I want to derive a new list out of the old list Then I think this way of writing it expresses this idea a lot better because then we say Uh, but here the here what I'm expressing is Well, somehow I need an empty list and then I repetitively execute some code I have no idea what's really going on here But here I really express the idea that we create a new list And the the list is derived by some rule From the old list code number. So it's uh, this code is a lot, uh, easy a lot nicer to read in the long run And it's faster And then of course, what can I do with list comprehensions? I can just use a list comprehension and put them for example within Uh, any place where I would otherwise use a list object So, uh, if I want to sum up a list, I would just call some with a list passed in but I could also Write in there a list comprehension and execute this and I get back to same 370 So let's see another example. Um, and this is called working with Cartesian products And this will show you how you can how how we work with nested for loops So what is a Cartesian product to begin with? Imagine You have two independent sets of objects So first of all here, we have a set x which has the axis x 1 x 2 and x 3 And then we have a second set of things the y's called y 1 y y 2 y 3 a Cartesian product Is any transformation that we apply To the Uh, to any combination of axis and y's so we see that we apply a transformation to x 1 x 2 But also to y 1 and x 2 and to all the combinations And whenever you will go ahead and we have a set of things and another set of things And we do something for every combination for every two table That consists of one element from the one set and the other elements from the other set Then we call this a Cartesian product And of course, I have here a Cartesian product derived from two sets But the the idea of a Cartesian product also extends to more than just two sets So we can have a Cartesian product derived of you know from five different lists or so on And let's see an example So the example would be this Give me a set so you the task is to calculate a set Which is derived from the axis and y's With the following function uppercase pi which is Basically like the summation sign that we know from statistics The only thing that's different is the pi the the uppercase pi means multiplication So that means I want to multiply all the All the numbers here basically so I want to do the following transformation. I want to divide x by y And I want to add one to it and then I I'm then I end up with a table of nine numbers and these nine numbers I want to multiply So this will then look like this So for example, let's take the numbers 10 20 30 and 40 50 60 here So how do I calculate this 1.25 here? Well, this is just one plus 10 over 40 And this next number would be one plus 20 over 40. So which is 1.5 and then the The set would be defined as the multiplication of all of them. So the product of all the nine numbers here So and whenever you find yourself So what what do we see here? We also see map filter map filter reduce here, right? So what is the filter? Well, there is no filter. So I mean not having a filter is also Of course legal in the map filter reduce paradigm What is the map? Well, the map is given here by this formula And what is the reduction? Well, the reduction is the product of the nine elements here So before we have seen three kinds of reductions. We have seen the sum We have seen the min and the max and now we see the product as the reduction So how do we implement this with a discomprehension? Let's first go ahead and create two lists And then I want to go ahead and create the Cartesian product So how do we create the Cartesian product here? Well, a first naive implementation would be to start with an empty list And then numerate over the list first and we call every element in first the numerator And then we within the for loop we have a second for loop a nested for loop and we loop over all the numbers in second We call that the denominator and then we calculate the quotient by you know Just numerator divided by the denominator and then the actual number that we want to you know, multiply by is One plus the quotient Yeah, so I should really In line with the formula maybe say One plus the quotient And then what do I do with one plus the quotient? Well, I append it to the Cartesian product list And I get back The Cartesian product and these are all the numbers that are in the table And now the next step Would be so this is what step is that that is the mapping step And again, there is no filtering. I just want to map here And then what is the next step? It's the reduction step. So I want to multiply them all together Oh, no, let's do something first. How can I write this in a nicer way? How can I write this with a list comprehension? Well, as we have seen before I just write opening brackets and closing brackets the the list literal I have here the transformation So it's x over y plus one and again, this should maybe be One plus x over y And the parentheses are not really needed but It looks a little nicer here. I think it's easier to read And then the question is how do we put the force inside? Well, we have two for loops here So we just write for x in first for y in second. So this is what I meant by Sometimes people get confused when you write this comprehension. So people often ask the question So in what order should the force go? Well, if you have a hard time to figure that out just write out in your head the for loop explicitly Just as we do up here and then you just use the same order within the list comprehension And of course We would use x and ys to keep everything a bit more concise and if I execute this code cell then I get the exact same list. So this is basically nicer and This also conveys the idea that I start with two sets So to say or two lists of objects and then I derive out of them the Cartesian product With some transformation And then of course if you switch the two force you get a different result, right? So you have the order matters But of course if we switch the order up here The order of the two if they force loops up here, then we would just You know divide in the wrong order, so to say so the denominator becomes becomes numerator and vice versa So what can we do now with this list comprehension? Well, we want to multiply them together So we need a product function and remember in the Towards the end of the first part of chapter 7 when we talked about tuples and about unpacking and packing in particular We came up with this example a function called product Which takes one argument called arcs And what is arcs? Well arcs is going to be a tuple And how do we know that because of the asterisk here? So what the asterisk is? It means we can pass in as many arguments as we want to product and they are all Collected packed so to say into one tuple and then what we do is We loop over we take the first element and tuple and loop over the rest and we multiply them all together So we the first number is the first element just is the first result And then we iteratively multiply on all the numbers And of course the loop starts with the second number in arcs But you should look at this in detail in the first part of chapter 7 and also in the exercises There is one exercise that extensively deals with this function and still will enhance this function pretty much so What can we do with that? Well, I can of course You know pass in the list comprehension to the product function. However, the product function Is built so let's maybe put in another another Fail here code cell to see how does the product function work? Well the product function for example It would take just one number and the product of just one number is the number itself And if I go ahead and put in let's say two more numbers Then the product of those numbers Should be 500 here and indeed it's 500 So because of the star in the definition in the parameter definition here, I can pass in as many arguments as I want By position only so we will see in a similar thing in chapter 8 But here we can pass in the arguments only by position and in order to do that Coming from a list object, which I have here with the list comprehension I have to unpack the list and how to unpack a list. We also saw that in the first part of chapter 7 We unpack with the asterisk. So the asterisk means here unpacking and here it means packing And yeah, so if you don't know How this works you should review some of the materials in part 1 of chapter 7 But I told you back then that I like to use these asterisks because the packing and unpacking Syntax really makes code look A nicer and it's also a faster way to code at the end of the day So let's calculate the product By unpacking the nine numbers into the product function and we get back the product of 20.58 And now I can of course I could this is just an example To illustrate some point further. We could of course also use a lambda expression with the reduce function to also do the same and actually There is a big difference here so maybe Maybe let's do Let's let's try live coding So what I mean by that is we go to Python tutor and because I did not prepare this so let's Copy paste the product function here And then see something so first we put in The example with unpacking the list comprehension or maybe Let me let let me wait. I just remember that This example is not yet done. We will look at this example Later, so in let's say 50 minutes when we've learned about another data type so that I can contrast three different data types But just remember that Yeah, we can also use here the reduce the reduce built-in So and that's the other data type. I wanted to talk about generator expressions. So to summarize the chapter the section before We can use list comprehensions to derive lists out of lists that are given to express the idea that we derive a list out of lists and then we can use list comprehensions syntactically wherever we can just use lists just as I unpack a normal list here I could unpack the list comprehension and now comes something very similar to list comprehension namely generator expressions and The difference between a list comprehension and a generator expression is just that a general expression is just as a list comprehension Just it's lazy. So we know that lists by now they materialize all their elements into memory And that is sometimes what we want. Sometimes it's not what we want And sometimes we want our a code to to evaluate lazily to not Create to do not put all the numbers in this case into our memory And in order to write lazy code, we use a generator expression. So what is a generator expression? Let's look at the an example the numbers list and here is again Our transformed and filtered numbers That is this is just from five slides ago. So I get back the same numbers as we know from a chapter four onwards And now here I have obviously a list comprehension which we see with deep records So how can I make the list comprehension into a so-called generator expression? Well, it's very easy I just replace the brackets with parentheses and now you will meet you would think oh, wait a minute What does the parentheses mean parentheses usually for us? We have seen them in many ways and your first response here would probably be That the parentheses are something like the grouping operator And if this were the grouping operator, it would be absolutely pointless because why would you do? Why would you put parentheses around one entire expression? This is basically useless, but in this situation the parentheses have yet another meaning Namely they create a Generator expression. So let's Let's execute this cell and I get back a so-called generator object. Now. What is that? Well, it has it has to have something in common with what we talk about in this Part two of chapter seven all the time and what is the main message of this? This part two of the chapter seven here. Well, the main message is that we can Create rules in memory that know how to calculate stuff without calculating it And that is what a generator expression is. So in other words I can go ahead And for example pass the generator expression the generator expression is just a rule So what this means is we generate these numbers here One by one And that's what I mean with the list comprehension is is eager the list comprehension materializes all the numbers in memory And the generator expression It creates a generator object which knows how to create these numbers But does not yet create them and how can we Basically pull out all the numbers from the generator expression. Well, we of course Use the list constructor again. So we wrap the list constructor around the generator expression And I get back the list with the same numbers Okay so This is basically again as I said, it's just a list. It's basically like a list just lazy And now one thing that is nice Whenever a generator expression Is the only argument to a function Then what we can do is we can get rid of one Pair of parentheses here. So we have the outer parentheses What are they? These are the call operators. So we are calling the list constructor Then comes the second pair of parentheses That is the syntax to to create the generator expression and here the third The third pair of parentheses is just grouping Here the n squared, which we don't really have to because the The exponential exponentiation operator goes before the addition anyways, but I do that for better readability So now the second parentheses we can just get rid of And also the second to last we can get rid of and now I Run this again and I get the same output. And so this is nice. So I can just write in that means I can Basically write a generator expression as the only argument to a function all the time and we will do that you will see that done Very often and this makes python look very nice. So what we know is how can we read this now? This basically says that we have an object here inside That is a rule a rule that knows how to calculate objects And then the list constructor out here that pulls all the object out and materializes all the objects And instead of for example the list Constructor we can use for example A reduction function like the sum function. So let's call sum And it gives us back 370 again So, uh, what's the difference between Having let's say a list comprehension in here and not having a list comprehension Well, let's let's now maybe go to python tutor because now this This really makes sense to to see the difference so I copy paste that in I copy paste The same expression without the brackets And then somehow we also need numbers, of course because python tutor will not know otherwise What is numbers? So and now let's visualize this code So now at first we create the list numbers or maybe Let's edit this code. Let's not do this, but let's just write it as Range 1 to 13. So let's also be lazy here. I like being lazy because it saves lots of memory So in the first step, we create a range object that knows how to create the numbers 1 to 12 without creating them And now I write sum and I put in a list comprehension and the list comprehension does the entire map reduce So it filters the numbers It filters for even numbers and it transforms them And then the entire numbers get passed to the sum of a function and then the sums are calculated So what happens? So what happens is First the list comprehension runs this makes sense And then at some point A list materializes and that's already a hint that something that is something we don't want So what that means is in order for the sum function to work In the last step before the sum function is being started A fully materialized list object exists in memory And then of course the sum function will return 370. Okay And now let's look at the other sum function the second call where I replace the list comprehension by a generator expression And now what happens? So we go over the numbers 1 to 12 again in a lazy fashion And let's see if there is a list created no So see the difference. So when we run the second argument No list object is created no list object materializes, but if I go back a bit Um, the first call to sum with the list Comprehension inside this creates a list this materializes a list. So we have to be careful now If we really want to store memory Or save memory Then it's always better to use generator expressions in place of list comprehensions wherever possible So in this situation here, it's possible. So now with the list comprehension I get 370 and if I get rid of the brackets To make it a generator expression. I also get 370 Okay, but the second time so this version is a lot better in memory So now the question you wonder maybe how in the how in the world Do you know this? Well, now I told you this and now you know, there are not many more rules Not not many more um situations where you can by accident Do this but you have to know at some uh places where you know, there could A list being be materialized and whenever you write a list comprehension that there will be a list somewhere in memory So that means some at at one instant of a second Or one instance of a moment is maybe the better term In your computer program, you will need lots of memory to hold all the reference to all the int objects in this case Simultaneously if you use a generator expression, this does not happen So you you only go over the numbers one by one and you add them up in a running total way And then at most you have two numbers in memory. So this version is super memory efficient and with a list comprehension It's not memory efficient But sometimes you need to have a list right and whenever you need a list Then you can of course use the list comprehension So let's look at the generators a bit more in detail So let's create a new generator expression and store it with gen. So let's do that So what this does in memory Is it does the following Let's create a new memory diagram here and what happens is This will of course also assume that we have our numbers list somewhere in memory and then We create a new object gen and what is gen? We don't know what it what's inside. It's some ones and zeros inside But it's of type generator. I I abbreviate it with gen And in this case, we also call it we give it the name gen And so what is the generator object? Well just like We had with the map and the filter objects before the gen object takes numbers as its source and Basically what the generator is it's a rule So now we have created a ruling memory that somehow Does in the in this situation the same calculations as the map and the filter So now this is like the two objects We had previously the map and the filter together in one But we can have any rule in here right any rule that we Create in a generator expression can be you know put inside the generator here And so so far nothing has happened And now what we can do is so now first we have a generator object Which is stored in memory at this location here It's of type generator And now what can I do? I can use pythons built in next function to draw out The next number so for example the 65 So basically I go ahead in memory and say hey Can you give me the next number that you know about and then the rule says well? I assure and then the rule talks to the list and says give me your next number Which is the first and then it does the filtering and the the transforming and then it creates the 65 And then what happens in this situation here? It gives it back as output And because we don't store it away what happens is The object gets immediately forgotten And then if I go ahead and call next again with gen Then the generator is being asked hey give me please your next number And then a generator talks to its source and gets the next number and the next number in this scenario would be 145 of course also an integer And then because we don't store it away The integer is returned to us in jupyter notebook as the output And then because we don't store it away It's immediately forgotten and goes away and if we ask the generator give me the next object Then we would get next one in this case the five and now Let's guess what's going to happen if the generator runs out of of objects to produce Let's see what happens We get a stop iteration error. We are familiar with that. It's the same behavior as the map and the filter objects Okay, and the technical term that we would use here is we would say that the generator is now exhausted That's the the official terminology here Okay, and now let's go back to the Cartesian products example and also use a generator expression here And then we will quickly go to a python tutor one more time to see a second area or a second situation Where we have to be a little bit careful so As we do the example and instead of three lists that contain the numbers 10 through 60 The materialized versions I use the range built in to be memory efficient And now what I can do? I can pass in The generator expression to the product function and unpack it remember that in the first the first time I showed you this example at this stage we had the We had a I can actually copy paste it and reproduce it here. We had a list comprehension. So the parentheses were Like this So we had brackets here. So what we do here is we have a list comprehension That does all the calculations for the Cartesian product and then it materializes all the numbers in memory And then at the end before the function is run The list with all the numbers which are nine numbers gets unpacked by the unpacking operator here And then the product function on the other hand Also has the asterisk which does the packing. So the unpacked numbers from the list from the list comprehension get packed into what? Well, the the answer is of course into a tuple because that's how functions work And then the function works and runs And it gives us back to result 20.58 and then we could of course now get think and that's the second situation I mean where you have to be careful So you've now by now understood that When we use the brackets, there will be a materialized list So we need lots of memory. It's bad And then your your first intuition would be to replace the brackets with parentheses to use a generator expression Because I just showed you that a generator expression is a rule That gives us the numbers on a one by one basis However, and that's a bit of unfortunately you can already see that I wrote a note here So that I must not forget to tell you this This is not memory efficient And why not because I'm using the unpacking here. So whenever you have a generator and you unpack it Well, what happens is you have to you know unpacking means you you throw out all of the numbers in it And they get stored within a tuple inside the function. But the thing is They are stored Inside the function as a tuple materialized. So this unfortunately Means that using a generating expression instead of a list comprehension does not automatically lead to a memory efficient solution We also get back 20.58, but this is not memory efficient. So now let's see. How can I make this memory efficient? And here comes The reduced function again from the func tools module in the standard library And Now it pays off If you understood what the reduced function does and it also pays off That you understood what the lambda expression does Because now we are doing everything in a memory efficient way And the reason is That in order for the product function to work It has to you know pack all the values that are unpacked into it into one materials tuple But the reduced function the reduced function takes an anonymous function here the lambda expression Which takes two arguments, which is the in this case the product so far Plus the next number and then it multiplies the product so far times the next number And yeah, this way we don't need all the numbers. We we we basically calculate a so-called running product in memory And now the question is where does the reduced function get its data from? Well, the second argument is the source and the source is now also a generator expression Because you don't want to use As the second argument here a list comprehension because we learned that a list comprehension will Materialize all the numbers never never what you do no matter what you do. So we cannot use a Or we must not use a list comprehension here So we have to use the generator expression and that means that the reduced function now Now lazily or the reduced function actually is eager but it asks the generator give me your next object It gets the next object and it multiplies the objects together And on a one by one basis, so it keeps on multiplying on so to say the next number Produced by the generator and this way we don't have to have all the numbers Simultaneously in memory and this also gives us back 20.558 and now maybe it's time to Copy paste this Where is it? Here it is copy paste The three versions here So we see all of that together. Maybe in Python tutor Maybe I should also Correct my spelling result and This has been a long lecture. So okay. So this is now the second version and let's also Use the third version to compare all of them how they work in memory. Okay and for this to run we also need from the func tools module in the standard library the reduce function and then Ideally we should also we must also not forget Here that we define first and second And I define it with the help of the range built in so that we have super memory efficient here And now let's see if I made a mistake. It seems to work So let's quickly go over this and I will just summarize everything I just said with a nice image now So we we import reduce we create first and second Which is memory less because it's the range object Then I create the reduce the product function with the packing operator And packing means we can pass into product as many arguments as we want And then within the function arcs is a tuple Of all the arguments I passed in In other words arcs is materialized And then notice run three different versions. So first the first version is I derive the the nine numbers to be Multiplied with a list comprehension and I unpack this. So let's go ahead and do this. So we see the list comprehension starts to run And then Now I was too fast. So the list comprehension starts to run and now we have For an instant of a second I have a list A list of all the numbers Unfortunately, these numbers now all exist simultaneously. So a materialized list and now What happens next in the next? moment is This list the the numbers in the list gets unpacked into the product function But the product function packs them together into arcs So now we see that now with the list scope goes away and the product scope will appear And the product scope now has one name called arcs and this references a tuple Which now has all the numbers but nevertheless all the numbers are materialized So the the list by the way is now gone It but it must have been somewhere in memory. Now the product function will just loop over Will just loop over everything here And produce the result that was the first version So this was memory efficient at two situations first it created a list and then it created a tuple both materialized Now we look at the second version the formulation with the generator expression Will we unpack the generator expression? So let's go over over it the generator expression runs It has of course its own scope And then at some point the generator expression will be done And what then happens the product function gets called and it unpacked Sorry, it packs the individual numbers provided by the by the generator expression They they first get unpacked as arguments and then they get packed together in a tuple So they all also exist simultaneously in memory. That's also bad It's a little improvement before that we had a list and ended tuple and now we only have a tuple And now let's look at the now the calculations run again And it's gone and now let's do the third way The third way is of course the best way to reduce a function And what it does is it first produces The generator expression at the as a second argument and then the the reduce a function One on a one by one basis pulls out the numbers from the generator and passes them to the reduction function Which is a lumped up expression and the lumped up expression multiplies on on a running product so to say the individual numbers and we will see this and So let's go back one more time and now I start running the third expression here the third line and we will see that hopefully at no point in time Do I have A materialized tuple or list or anything so so far. I haven't seen any yellow and we are right at the end So we have seen the third way indeed. It does not need a tuple and it also does not need a list So the third way is the only way um, that is memory efficient And so you may wonder why could I not write a nice function like the product function? Why do we have to use this weird? lambda expression as a reduction function With two arguments and so on but it really pays off to understand that once you start to work on big data sets Okay So now Let's continue The presentation And now with everything we have learned so far What I want to do Is I want to take I want to take a look at our very first example in in this course averaging even numbers that we also looked at in chapter two When we when we looked into what our functions And we defined a function called average events That was pretty well back then But it was not memory efficient and now just to reiterate the point here How you can use all of what you learned in this chapter And put it together to reformulate your function from chapter two in a way that is super memory efficient So, uh, what do we do here? Average events still takes numbers Numbers used to be a list. So in the um a doc string We used to write that numbers is supposed to be a list But now we we we know more about python So we know that the only thing the only two properties that we need For the function to work is we need that the first argument the numbers argument Is iterable and it also has to be finite because if we want to if we want to Calculate the average of something then whatever you know data we want to calculate the average of better be finite We cannot you know average up an infinite stream of numbers So these are the two properties that we rely on and that's also duck typing at work in a way, right? So I don't say uh, give me a list or something I just say give me something I can iterate over and that that is finite And then the scalar we also have but we can ignore it for now And then what happens in the first line? We have here on the right hand side a generator expression This used to be a list Or a list comprehension to be precise But we now know that we don't want to Use a list here or a list comprehension because we know that lists always materialize and we don't want to materialize So I write a generator expression That loops over the numbers one by one and just rounds the numbers So remember that averaging evens average evens only makes sense for integers So if a user provides a you know a data stream of floats, then we round the floats to get integers And then we have and we store this generator expression in a variable called integers And integers is then a generator object and then we write a A reduced step we are using the reduced function From the func tools module in the standard library and now it's a little bit tricky here. And this is actually I would say now a little bit advanced, but it's still on a conceptual level easy to understand. So what we do is let's look at The second part here. So we reduce takes two arguments first a function which we provided with a lambda expression And then the second argument is the source of the of the reduce for the reduced step Which is a another generator expression and this inner this other generator expression here derives from the first generator expression And then we loop over the individual numbers and we filter them And then instead of returning the numbers as n we return a tuple n comma one And we use the comma one the second the second element in this tuple to Implement the counting because you know to get an average we have to sum something up and then count divide by the count and so we have to also keep track of some count somehow and we cannot use the the len function because Unfortunately generator objects don't know how long they are because generator objects. They only know one thing which is How to get the next object, but they don't know how many objects there are And so because of that with this trick with a with a tuple here We You know have the actual number we want to sum up for averaging and then we add up the ones and for every You know for every number that remains so for every even number We just add up one over time and that means we just take the count And then in the last step I just You know divide the total by the count and multiply by the scalar And yeah, this is it so up until here the function is lazy and only the reduced function will eagerly drive all the generator expressions And if we look at this implementation now I can of course call average events with our familiar numbers list here get 7.0 We can of course also put in The numbers as as floats it also works. We could also go ahead and of course As seen in chapter 2 provide a scalar of 2 to get 14 it also works And now maybe let's do a little experiment Let's import the random function and now what I what do I do? I pass to the average even function another generator expression and the generator expression Is basically A big loop that loops 10 million times And then produces in every iteration of the loop a random number a random integer Between 0 and 100 And we know that the numbers between 0 and 100 If they are equally likely we should have an expected value of around 50 when we Run, you know when we draw many many random numbers between 0 and 100 So let's call this function and I can already tell you that if we wanted to materialize a list that is this big This will most likely You know, this could actually crash my laptop because my laptop does not have so much space But I mean this wouldn't actually because 10 million is not so much But we can imagine that I can increase this number easily by a little bit To get my laptop to crash So in order to calculate the average now the cell is already running To calculate the average of something that does not fit into my computer's memory. How do I do that? Well, the trick is and now you can guess it by now We just loop over the things we want to average one by one So that they fit in memory And now we see that the average of all the events between random literal numbers numbers between 0 and 100 Is around 50 so This makes sense. So what's an application of this? Well, let's say you are a company that is a big corporate and you have lots of sales data and you want for example to Calculate, you know the average amount of a sale Let's say you are like a retailer like Lidl or Aldi or something And you have many small sales, but you have really many and you want to have and What you want to know? What is the average? you know The average basket that is sold Then How how how would you how how would you do that? Well, probably, you know a company like Lidl or Aldi has so much sales that it doesn't fit into a computer's memory But by going over the numbers one by one, we can calculate the average of something We could also calculate the standard deviation and variation of something um That um, you know doesn't fit in memory. So we can easily see that A company with big data can can make use of that Okay so Let's uh talk about something simpler So um reduction functions as we have seen are Examples for that are like the sum the min the max function But also the product function for example that we show what that we have seen Let's look at a different kind of reduction functions So called boolean reducers and that's a Seems to be a fancy term, but it's very easy. Remember in chapter four. We had um one example Well, we had the numbers list again, of course, and we asked the question Is the square of at least one of the numbers in it? Uh above 100 or I can reformulate this question I can ask the question is the square Of all numbers in the list below 100 And how can I formulate this in a in a nicer way without the for loop? So in chapter four we we did that with a for loop Here what we do is we use the all built in So all is a reducer a reduction function just just like some And what we do here is we pass in a generator expression that loops over the numbers And for every x for every number in in the list Takes the square and compares it to 100. So this expression here is a boolean expression It returns a true or false and then uh the all function the all built in um function Evaluates to true if and only if All of the elements it iterates over Evaluate to true Who would have thought so? The the answer here is false. Why is it false because the 11 for example the 11 squared would be above 121 100 would be 121 And then remember we in chapter three we talked about short circuiting So this is also implemented here behind the scenes. So um the all function is smart enough To stop going over the numbers after it hits the 11 because the 11 Gives me back a false and then the all function says well once I have encountered one false It doesn't matter what the rest is. I'll just stop and return false So the so the all function is quite efficient here And similarly we have a function called any and I can ask the question Are there any access in number? Is there any number in numbers whose square is larger than 100 and the answer would of course be true And if I play a little bit with the threshold here Let's say 1000 I get back false and any also implements short circuiting behind the scenes Okay, so boolean reducers That was an easy topic and now comes the last section of part two and chapter seven And this is basically just repetition of everything we learned so far But of course as always We want to give abstract nice abstract technical terms to our concepts so We have seen What are rules in memory? We have seen three types of rules So here in our in in the current diagram We have a generator and a generator is a rule that knows how to calculate stuff We have seen two other types that do the same thing in it Of course with a different rule, but they have the same logic. It's the map object and also the filter object And now the question is What do all these objects that model rules in memory have in common? well They have in common that all they know is they only know how to calculate the next object in line They have no understanding for example of how many objects there are You know a list. I told you a sequence Has the has the concept of of length we call this concept sized. That's the technical term And a sequence can be asked with the built in length function how many elements are in the sequence But the rules that we have here They they are not that smart The rules that we have seen in this chapter. They are super dumb The only thing they can do it literally the only thing that they can do is we can ask them with the next function Give me the next object in line And they just give us the next object and once they are out of out of items. They give us Um and stop iteration around that's it and all types all data types in python That behave like this see that behavior is again the important thing and not the type All the data there are all the data types that behave like this that behave like rules in memory. We call them iterators Right iterators and don't confuse that with another technical term called iterables An iterable is anything that we can loop over So a sequence would be an iterable But an iterator is anything that only knows how to give us the next thing So it's something it's it's similar, but it's something totally different actually. So let's look at an example Our numbers list again And how do we get iterators we use the built in iter function and what we do here is I use the built in iter function And I pass in the numbers list and I get back an object and the object I get back I store with the variable iterator one. So let's do that maybe One last time We draw a memory diagram so What does that look like in memory? Well, of course As always we have a list It contains all the data So list And then we have the int objects And so on the list has a name numbers and now I pass Numbers to the iter built in function and I get back an object And we also do that a second time for illustration purposes I get back Iterator two now by calling it here again And now the question is what is the iterator? What is iterator one? It's of type list iterator. So what we did here is Let's call it it And we can also call it here iterator and this would be iterator one And then we have a second object I also write in it And it's also type iterator list iterator to be precise, but I just abbreviated it with iterator And this goes by the name iterator two We do that Now the question is what are the iterators? Well, I can already tell you that they are rules in memory that are somehow Connected to the list that they are built with So somehow the both both iterators Are connected to to the numbers list So now what can we do with an iterator? I told you that they can only do one thing We can only call the built in next function with them. So here In the next code cell creates a tuple Which consists of three elements that are drawn from iterator one and iterator one as I said is smart enough to only give me The next object in line. So what will I get back here? Well, I get back the numbers seven eleven and eight Okay, that's it. So in other words This is kind of like a lazy for loop. It's a rule It's a rule in memory that knows how to calculate the next thing and the thing is here It doesn't have to calculate anything. It just has to loop. So it's like a lazy for loop And now we can um Do another code cell here and I say next for iterator one and next for iterator two And now I get back what I get back the five, which is the fourth element And I get back to seven again, which is here the first element again So what have I what what does that mean? That means I now have two rules that are for loops that loop over the same object simultaneously And they don't get confused So in other words, I'm actually looping over numbers in parallel at the moment And this works in python and why does it work? Because um because it works because of the for loop It's not concerned with this object with the list object here at all The for loop only talks to those iterators as we will see And the iterators all that they do is They are really dumb. The only thing that they know Is they know the last object that they gave you back They in other words all they store In in the boxes here all the iterator store Is the index of the last element returned not even the object Just the index of the object that they just produced And now we can ask the iterators and we get back the objects on a one by one basis Okay, and now We uh, I I will show you another type of iterator Uh of type sip. There is a function Uh built-in in python. Let's maybe quickly see if we find it So if we scroll down here It it's sip and now we also understand Um the the documentation here may be a little bit better after studying chapter seven. So the sip built in What does it take? It takes star Star intervals. So what does star intervals mean? Well star here is the so-called packing operator Which means I can pass as many Objects as an argument to sip as I want And by the name by the argument name, which is intervals I can already tell What the sip function wants me to pass in It wants me to pass in intervals And then let's read documentation. What does he do? It says make an iterator That aggregates elements from each of the of the intervals Okay, let's see what what what that means Well by the name sip we can already We can already tell that the behavior will be like a zipper that you know from your jacket So we sip together two things Just like we sip together a jacket So let's create a zipper object So here zipper is an object of type sip. So what did we do in the memory diagram? All I did I created A new object Which is of type sip And uh, yeah, it's just sipping stuff, right and we called it zipper That's the reference and now what does what does a sip do well sip is connected to This one and this rule and the first rule this is we call that source one And the second iterator here. We call that source two So whenever we go ahead And we ask zipper now. Hey zipper Give me your next object in line. What zipper does is it talks to the first object the first its first source And says hey, can you give me your next object in line? And then the iterator goes to the list and says well or basically the iterator remembers the last index That it got and then it gets the next index and gives it back to sip And the same goes for source two at the same time sip asks the sip iterator two Hey, can you give me your next object? And then the the iterator two object says well, yes, I can it remembers its last index So it goes back to the list and obtains the element in the next index and gives back the element and then all the zipper does It takes both elements that it gets from the two iterators Sticks them together in one tuple and gives it back So let's see that So how what can I do well for example instead of just Well, maybe once we do it manually. So we can of course call next with zipper And I get back three and eleven. So why three and eleven? Let's see so three Is the the fifth number and eleven is the second number So it's the next number that is in line with the two for loops that are going on simultaneously right now That's what zipper does and of course We could also loop over it So what we we now get is we get the remaining objects. However Note that we stop when the first source When the first iterator here Is exhausted at the four at which the second iterator is at nine. So in other words The second iterator is not yet exhausted. It's just here. So we looped over We looped over numbers twice in parallel And there was an offset between the two loops And once the first for loop was done The second loop had to be done because the zipper function works like that and zipper function will always Only go and as long as it gets one object from all from all the sources, right? In the moment one of the sources to zipper. I exhausted then the zipper itself is exhausted Okay And now what do we see here? Well, what we see here is that I can actually loop over zipper But now wait a minute zipper. I told you it's an iterator Why because it knows how to give me the next object in line What we see here is that I can also loop over zipper That means zipper must also be iterable iterable So it's both iterator and iterable And this is true in general Every iterator iterator is always an iterable And you can read in the chapter why this has to be the case. I don't want to go into detail here, but What we learned from this is that iterable means We have an object that we can loop over And how how does the looping work in python now? We finally understand how looping works in python looping works in a way where where python creates iterators That only remember the last index and then via the iterator the loop is basically driven so in other words An iterable means an iterable in other words is any object That can give that can create an iterator And how do we create an iterator out of an iterable while they're calling the iterables in function? so That's um, you know another abstract concept, but it's actually not not that hard an iterator in layman's terms is just a rule in memory that knows how to compute things without computing them and then What we can do Now iterator one is exhausted right after the for loop that we just executed iterator one is exhausted So what does that mean? Well, that means if I call next with iterator one, it will raise and stop iteration exception. It's exhausted And now what happens if I call next with iterator two? Well iterator two obviously was not yet exhausted So it will give me back some more elements for example the 10 For example the one for example the four and then what happens finally iterator two is also exhausted So now we have stop iterations on both iterators and now just to Repeat this It does not matter how often I execute this code again In both cases I will get only I will get nothing but stop iteration exceptions because the iterators are exhausted And as I told you they are dumb once they are exhausted. They don't know how to go back So they are really dumb objects. All they know is they know where they were They only the only thing they remember is the last index that they returned And when we ask next on them, then they will just give us back the next at the next index and once the You know the object here is out of indexes So we reach the end then the the iterator does not know how to go back And then we can basically you know remove the iterator because it is useless for us And now now that we have seen how iteration works in python Let's quickly look at the fourth statement again I told you in chapter four that the fourth statement is really syntactic sugar for the while statement And if you want to know how I expressed that you should look that up in in chapter four However in chapter four, I was actually not telling the entire truth to you So let's look again at what the for loop actually is so an example We create now an interval. It's a list of numbers from zero to four And now I use as the variable names the technical term. So I use I I name it iterable and now in order to Loop and we said we said anything in python is an interval that we can loop over So what do we do? Well, we just write a fourth statement for element in iterable And then just to illustrate a point we print all the elements on one line And now I want to show you quickly What is the four statement really? So what's what does it really do the fourth statement is really syntactic sugar for this here That's what python does when we use the for loop So python will first Look at the interval that we want to loop over with the for loop And ask hey you iterable Can you give me An iterator that knows how to loop over you that knows how to get the next object in line The next element That's what the first line does And then what happens here? We go into an indefinite loop while true And then we have a try except else And in try what we do is we try To obtain the next element from the iterator with the next function And we know we have seen that once the iterator is exhausted. It will raise a stop iteration error. So what do we do? Well, we accept For the stop iteration error and whenever that occurs we know that from then on it will always occur So in this situation, we just break out of the while loop And if we don't have a stop iteration exception, that means we are not at the end of the loop That means we just execute the for loops body right so in the else clause We will see the for loops body And then to be super precise What what I forgot here python is actually kind enough To delete or to remove the reference the variable called iterator again at the end So this is what python does whenever we execute the for loop So let's do that and we get the same output, of course So this is how looping works And once we have covered chapter 10 on object orientation And we know some basics about object orientations we can in advanced with some advanced syntax on python Create our own objects our own data types That we can iterate over or use as an iterator Okay Now this is really the last section in this chapter. This is just to use everything we've learned And categorize the built-in sorted and reversed. So what I sorted in reverse we have seen them before So let's start with our numbers example again And numbers is now a list And the sorted built-in takes as we saw in the documentation Any iterable and returns a new list With the the elements of the iterable sorted So I get back a new list sorted. However, the numbers list again Remains unchanged So sorted only looks so sorted only iterates Over the elements and sorts them, but it does not touch the numbers list So in other words Sorted will always materialize a new list On the contrary The reverse built-in if I call reversed and I give it numbers I get back a list reverse iterator and now we know what that is It's just an object a rule in memory That knows how to loop over the list in this case in backwards fashion in reverse order However, we only get back one object So we don't get back a materialized list here We only get back an object that we can use to loop over and if we want to materialize this we have to wrap it With the list constructor and then of course I get the list reversed and just remember it's obvious I guess but Sorting in reverse order and reversing are two different things So I just want to show you how the built-ins they return You know what the built-ins return may be different So the sorted built-in returns a new list and the reversed built-in returns an iterator However, however, because iterators are always iterable We can of course loop over the reversed object Which is why and and we have seen this before We can write for number in reversed numbers and Print out the numbers in reverse in one line Okay And numbers of course is also unchanged here because again numbers at the list of the list type is a mutable data type So in order to change numbers in place, we would use the list functions that are covered extensively in part one of chapter seven And here we don't change the list in place Okay, so this is uh it this was the second part of chapter seven I think this may be the first Part in the course That may not be so super trivial because we do all kinds of fancy things But at the end of the day what you learned in today's lecture Was that how can we work with big amounts of data Without having all the data in memory. That's the big big storyline and this prepares you For you know working with big data sets of course and running analysis in an efficient way And now we have learned Lots of information on what sequential data is In chapter nine when we talk about arrays and data frames These kind of data structures They are also kind of like sequential data, but they are not purely sequential data Which is why they are in a separate chapter and also chapter seven With almost I think five and a half hours of lecture time Is also not just Is already too much. I don't want to put more in this chapter So that's what you've learned and we have learned some more Abstract behavior. So the behavior being an iterator So the the behavior of just knowing how to loop without looping And we've seen several different data types in python that make this work So we've seen a the for example a generator expression that evaluates to a generator object Which is a rule. We have seen for example the reversed which returns a generator Or an iterator so iterator is just the more abstract term but a generator is basically An example of an iterator And then we have seen the map object the filter object And so on and then also we have seen The the entire map filter reduced paradigm In this part here and we have seen other nice things like the list comprehension We have seen how list comprehensions Always create lists that are materialized and if we want to work in a memory efficient way, that's not what we want And we also have seen some situations where we have to be careful Where some objects are materialized in memory without us knowing or seeing it So we have to know the syntax in detail And other than that I think I'll leave you with that. This is enough material and See you then in chapter 8