 Hello, welcome to SS Unitex, so we will decide and this is continuation of PySpark tutorial. So in this video we are going to see about the pivot and unpivot transformation. So as per the today's agenda, first we will see about the pivot transformation, then we will see about the unpivot transformation. So what is the pivot transformation? So let's assume we are having a input data frame and here we are having one of the column that is a country. So in the pivot transformation, while we are going to apply this. So in the output of the pivot transformation, it will be picking the unique value from that particular country column and will be converting those unique values into the columns. So let's assume if we are having unique values like India, UK, US and Canada. So those four values will be converted into four different columns. So that is the pivot transformation. In the unpivot transformation, it is a reverse of pivot transformation. So it will be going to pick all those four countries from those different columns and will be converting into a single column. So that is the unpivot transformation. So let me quickly go inside the browser and we will try to see in practical. I am going to create one of the data frame that is DF and it is having total three columns with product, amount and country. So our requirement is we just want to convert all these countries into different columns. So how we can do that? So for this, we have to use the pivot transformation. Product will remain same, but the amount will be some of the amount and this country will be going to convert into multiple columns. So for this, first we have to use the DF dot. We have to specify the group by. So inside the group by, we have to specify those columns, which we don't want to attach. So in our case, we don't want to attach the product column. Then we have to use the pivot. So inside the pivot, we have to specify the column, which we want to convert into multiple columns. So this is the country and the third column. We have to use the sum of amount. So this will be the third column. Now let me put this into another data frame that is DF one and let me use the display of this DF one. So we'll see the output of this. So here the command is running and now we can see it is having the product. Then we can see Canada, China, Mexico and US. So these four countries are here and we can also see the sum of these values that we can see. Now next, let's assume we don't want to keep all these four countries. We just want to keep only Canada, China and US. So how we can do that? So first we have to create the country as a variable and inside this, we have to specify an array. And in array, we have to use those countries, those we want to convert. So in our case, I am going to use Canada, China and US. So once we have done this, next, we have to use the DF that is our main data frame. Dot group by will remain same. Inside this, we have to specify the product column. So we can go with the product. Next, here we have to specify pivot. So inside the pivot transformation, the first column, we have to specify actual column. So this is the actual column. And second, we have to specify the list of countries which we want to convert. So in our case, I have created this variable that is country. So we can specify here. And next, we can go with the sum of amount. So amount is another column. And let me put this in another data frame that is DF2. And let me use display of this DF2 and execute it. So it should be going to return total four columns. And this time we can see Canada, China and US only. So either we can specify in a variable and then we can use this here, or you can also specify this list of array values directly here. So whatever the way that you want to choose, you can use that. Now here, let me try to see a data frame that is DF1. That we have created where we have all four values. So that we can see like Canada, China, Mexico and US. So now the requirement is we just want to unpivot of this. So how we can do the unpivot? So for doing the unpivot, we have to use the DF1 data frame dot. Here we have to use the unpivot transformation. And inside that, first we have to specify the column which we don't want to touch. So that is the product column. Now next, we have to specify list of countries which we want to convert. So in our case, we want to convert Canada. Next will be China. And next will be Mexico. And last will be US. So these four countries, we want to convert. And what will be the column name that we have to specify here? So the column name will be country. And then the actual value column, so that will be amount. So we can use the amount here. Let me put this into another data frame. And let me use the display of this data frame 3 and execute it. So because we have not used this into single code. And similarly for the amount as well. So these two should be in single code to double code. So now we can see here like this is converted like the product will be the first column. Then the country. So inside the country it is converted and it is now having Canada, China, Mexico and US. Similarly, we can also see the amount. So under the amount, we can see nulls. And if we have noticed at the starting, we don't have any null. So if we don't want to see any null, then we can simply filter out or we can delete the value which are having null in amount. So that we can do.