 Hello, welcome to SSUnitex Societies side and today we are going to see a very important concept inside the Azure Data Factory that is how we can get the latest file from folder and that folder inside your Azure Blob Stories. So this is very important and sometime in the interview interview also going to ask this question. So let me quickly go and try to understand the requirement first and after that we'll be implementing the same. So we are under the Azure Blob Stories of the input folder and here we have all these files. So in the real time you will be having all the files with the same structure and you will be getting the file like every half an hour or every one hour. So on that basis you will be getting all those files. So the requirement is every time you should be going to pick the latest file which we have received and after that process that file. So in this example we are going to pick the latest file among all these and after that copy that file into the output folder of the same Stories. So how we can do that? So for doing that let me quickly go into the Azure Data Factory and here first let me try to create a new pipeline and let me call this pipeline as get latest file like this. Now first as I told you we are having a folder and under that we have all these files. So for getting the information from this we are required to use the get meta data activity. So first we need to use the get meta data activity and this get meta data activity is asking for a data set. So we have already created the data set for the same. So go to on the settings and under the settings we need to select that data set. So the data set which is ds underscore input. So let me try to open this. So it is pointing to the input folder. As you can see it is pointing to the input folder. So once we will execute this then we will see how many files we have under that. So here we can see the field list. So under the field list let me try to add a new. So here we can see the child item. So child item will be going to return all the file names and let me quickly try to click on this plus symbol. So it will be going to have the last modified. So we need to check whether this will work or not. So let me try to debug it and we will be going to see the output of this. So this is executed successfully let me check the output of this. So what we can see here the last modify is returning the last modify of the folder. So it is not going to return the last modify of the files which is under the child items. So here we cannot use the last modify directly. So this is the tricky part of this scenario. So here under the child items we can see we have the name which is containing the file name and after that we have the type what is that. So either that is folder or the file so that we can see the file. Let me try to close this and here we are not required to have the last modified. So we can remove that. So only child items are required. Now we need to look through for all these files. So for that we have to use the for each activity. So under the for each activity let me connect with the get metadata activity. So in this for each activity go to the setting and here we need to select. So what we need to select? We need to select the output of the get metadata of the child item so that we can see here we can select directly and click on OK. So this for each will be executing for all the files so that we can see 1 2 3 4 5 6. So it will be executing total 6 times. Now next what we need to do? Let me go under this for each activity. So under the for each activity again we need to use the get metadata activity and that get metadata activity will be going to get the last modify as per the file and file name that we are getting under the for each activity. So here what we need to do? We need to point for each file. So let me go into the setting and here we have already created a new data set which is the ds underscore input dynamic file. So it is asking a file name value. So from where we will get this file name value. Let me try to open this data set first. So I can show you that. So here we have created one of the parameter of the file name. So this parameter we are passing over here like the file name. So when we are using this data set it is asking the file name should be supplied on this. So file name we are getting from the for each container. So let me go here for each container and which property that is a name property. Don't worry for now. I will show you later like from where we are getting this name property. Let me click on OK. So here what we want to do? We want to get the file name and we also see the item name and this is the file name mainly and the last modify of this file. So here we can see the last modify. Now let me try to execute this. So here the get metadata activity one will be executing. So this will be going to return all the files which is available under the input folder that we can verify from here like the name of this and total six files. So total six we can see after that we can see the for each. So for each will be going to have total six items. So it will be executing six times. So six times it will be executing the get metadata. So once we click on this then we will see the item name as the file name and the last modified at the last modified. So let me try to copy this. So it will be going to use later on this and let me try to put this into the notepad. So if we can go in the upside. So it will be going to have another file. So total six files we have with us and whatever the last modify of those file will have. So what next we need to do? So first let me try to create a variable and that variable will be having the file name and the last modify. So like two variables we need to create here. So let me try to go into the variable and here this will be for last modified and that is string and default value I am going to pass 1900 0101 like this. Here let me call this as file name variable and default value I am not going to pass anything. Now here let me go under this. So once we have this next we need to check whether your variable value of the last modify is greater than or less than of your actual last modify of the file. So that we can get from the get metadata activity but for checking we need to use the if condition activity. So let me connect the if condition activity from this get metadata activity. So in the condition what we need to do here simply go into the expressions and under the expression here first we need to check greater then we need to supply the first one. So let me try to use the format date time function and this format date time from the get metadata to which is this one output and output of which property that is the last modify that we have seen here. So let me try to copy this. So this property is having the last modify of this. So dot this property. Now if this value is greater than again we need to use the format date time function and then we need to supply the variable value the variable which we have created. So here we can see the last modify. So if actual modified value of the last modify for any file is greater than the variable last modify value then we need to add this variable value as the last module of that file. So what we are doing here we want to check between all these files and doing the comparison and we will be going to pick the latest file from here and after that we will be proceeding accordingly. So let me try to click on okay. So if this condition will true then we need to use the set variable activity. So under the set variable activity first let me try to add the last modify and this last modify value that should be equals to the output of the last modify to the get metadata 2. So that we can see here. So it will be going to supply into that variable. So in the first execution it will be going to have 1900 so it will be true. So it will directly jump under the if condition activity and under the if condition activity it will be going to assign under that variable of the last modify and at the same time let me use one more set variable activity and this set variable activity I am going to supply the file name as well. So file name is also label under the get metadata 2 of the item name. So that we can see here because we have seen the item name property is holding the name of the file. Now let me click on okay. Go back to here and here let me try to execute and we will see. So what it will be do? It will be doing the first get metadata 1. So it will return all the 6 file then for each it will be going to loop through with all the file and under that the get metadata 2. So it will be going to execute one by one according to the files that we are getting through the for each container and after that under the if condition activity we are checking if that is true. So the true condition we are checking the actual last modify of the file and the variable which we have created. If that value is true then we are assigning last modified for that file into that variable and next execution it is going to check with the first file. So let me go back here and try to check the outputs here. So as it got executed successfully here we can see the set variable activity last it got executed. So the value we can see 982022. So this is the last we have set into the last modified variable. Let me go here. We can compare with all the modified here. So we can find this is the last modified and same thing we are getting over here as well. Let me close. So this is we have done let me quickly go and try to use the copy data activity and this copy data will be executing after the for each container. So once everything will be done then we need to copy the data file from input to output location. So for that we have already created the dynamic file data set and here let me quickly go and try to supply the value. So this time this value should be coming from the file name variable. Let me click on OK. So that is done. Let me quickly go into the sync and for the sync we have already created the data set. So we can select the output data set here executing. So once it will be executed we will be going to see the last modified file should be copied from input to output location. So we need to wait a little bit on this so we can check and confirm. So in between let me go here and try to go into the output folder of this SSU testing. So we need to wait and here the copy data is in progress. So we can wait until it will not be completed. So as we can see it has copied. Let me go back here and here we can see the item details dot CSV. So the last modified that we can see today's date and time that we can see. So we have successfully copied this from source to sync location and if you can see the input folder. So under the input folder we can see this is the last modify among all these. So that's why it has copied from here to there. Let me recap what we have done here over here. So first we have used the get metadata activity to get all the files and after doing that we are using the for each container the for each container will be going to execute total file is six. So it will be executing for six time and under the activity we can see the get metadata two. So this get metadata two is required because we have already seen the last modified is not working in the get metadata one. So here we are going to supply the file name from the for each container and after that we'll be getting the last modify for that file and then we are checking through a variable if the variable value is less than the last modify of the file then we need to assign the current file last modify value to that variable. Similarly we are going to assign the file name as well at the same time. So after doing that we are going to copy that and the file name that we'll be getting from the variable where we have assigned under the if condition activity. So this is the flow that you have to remember. So thank you so much for watching this video. See you in the next video.