 Hello and welcome to this tutorial that is tutorial 9 where we will be learning how to handle passive microwave observations. So till the last tutorial that is tutorial 8, we learnt how to deal with active microwave data. Typically, we try to analyze data from the Doppler weather radar from next rad and from the MOSDAQ website. So here our focus will be on data from passive microwave remote sensing. So let me take your attention to one particular slide that I had shown in the previous lectures that is how different satellite remote sensing missions help us to learn about hydrology in particular. There are missions that give us information about the ground water, we learnt about gray satellite and then there are missions that give us more information about surface water wherein we learnt about the fundamental principles of radar ultimetry. And we also saw how to deal with data from satellites like Jason and Searle. So here in this particular tutorial, we will be trying to understand the data pertaining to soil moisture information that is obtained from satellites through the microwave region of the electromagnetic spectrum. So why soil moisture? Because you know it is a very crucial variable for numerical weather prediction and for climate models as well as for surface hydrology for vegetation monitoring. So at L band frequency that is L band of the microwave region, at L band frequency the sensitivity to soil moisture is very high. So just to reiterate in today's tutorial, we will be trying to look at which satellite missions give us information about soil moisture and in particular we will be trying to focus on the CMOS mission. So a little bit about CMOS before we try to understand its data products. So CMOS is abbreviation for soil moisture and ocean salinity. It is a satellite which is part of the European Space Agency that is ESA living planet program and it is intended to provide us with new insights into the earth's water cycle. CMOS is also known as ESA's water mission and it helps to improve our understanding of the earth's water cycle. In addition for this tutorial we will be using the CMOS level 3 data, CMOS level 3 radiometric soil moisture retrievals obtained from the 43 kilometer mean spatial scale CMOS observations. So you can obtain this data from the website, we will come to that shortly but let us explore the instruments onboard the CMOS satellite. So if you go to the ESA website, European Space Agency website, it has details about the microwave imaging radiometer with aperture synthesis that is the instrument onboard CMOS which is a passive microwave 2D interferometric radiometer operating in the L band, L band of the microwave region and you can explore the site which gives you more details about the instrument and about how to access the data and you know a lot of details are available even the CMOS toolbox is available, details about that are available. So if you are more interested to know about this mission, I would urge you to visit the earth.esa.int website to get more details. So as part of this tutorial, let us try to understand how to access the data first. So what we do is to access the data, you have to visit this particular website, catds.fr. So this archive as you see here the server gives us data from CMOS mission and you get data at different temporal aggregations. You can get one day soil moisture, 3 day soil moisture, 10 day soil moisture which has been aggregated temporarily to 1 day, 3 day and 10 day. So if you just browse you can see L3 stands for level 3 soil moisture and also you have the data that has been arranged over land and ocean, over land and over ocean and as of now you can see that it is showing a locked symbol which means you need to have your own user ID and password created before you can access, get access to these data products. Usually what happens is you click on the registration at the top corner of the window and once you click on that a window like this opens up and then you get to complete the registration and then you are allowed to log into the site. So you can see that now I already have a user ID. So I have logged into the site and then the interface what I see is this. So if I click on any product say the 1 day soil moisture, the data details are shown, the starting date, ending date is shown for which data products are made available. I can click on any product say over land, over ocean and over land and ocean. So if I continue clicking on the same product, it leads me to the data object selection you know. I can see the estimated size of the files and there are certain filter options that I am able to use. For example, I can decide on the starting and ending date and these options these filters are available for any data set from the website. You feel free to explore. Let me try to look at the root zone soil moisture data which is also made available. You can see the filters in front of you. Let us try to specify a starting date and ending date for which you need the data. So as an example, I can go to 2020, Jan 1 and then I can give ending date as say 2020, August. Now after that the orbit type information is available. Ascending is 6 am local time and descending is 6 pm local time. The product type information is available, reprocessed data are made available and operational data are made available. So I click on operational and then I can select the processor version. Now coming on to the geographical selection. That is over which area do you want the data to be downloaded. So I can pan in and then create a geographical bounding box to mark the selection. So what happens is all the satellite data that is available for the dates that I have specified that is from the starting date to the ending date, it gets shown to you how many files are available for this particular bounding box that is region. So you just draw a rectangle marking the bounding box and the coordinates appear to you. So for this exercise I have tried to see the files over Ganga basin which is one of the largest river basins which lies in China, Nepal, India and Bangladesh. You can see the number of objects, number of files are given, its size is given and you also get to select the format to archive or compress the data. So I will click on zip and then one archive file per order. So now the summary of my order gets displayed and I am going to submit the order. So you get to view the order submitted and also you can view the summary of your order in the summary order window towards the left side there. So the order that I have just submitted is made visible in the user workspace. Once it is available a simple double click will let you download. So the download window will appear and the compressed files can be saved to the desired location. So now let us see how to import and use these files in Python. So as before the smalls data are made available to us in the next CDF format. So it is going to have an extension say .nc. So what I have done is I have gone to the website. I have downloaded the root zone soil moisture data over Ganga basin and created a bounding box and selected the area and I have given the starting date and ending date. So all the smalls level 3 root zone soil moisture data for that particular date and for that particular study region is going to be downloaded. I have downloaded and kept it in one particular region of my system and now I am going to import it into Python and see how the data looks like and how I can extract the relevant variables in Python. So as before I have created an empty Jupyter notebook and I have named it as tutorial 9 followed by name of this course. So as before I need to import all the necessary libraries inside. Now in case you do not have a few libraries installed, I would urge you to first visit the command prompt, install these libraries we have already seen how to do that in a few tutorials and then once the same are installed you then import it from here. So for this particular tutorial I need net CDF4 because as I mentioned the files have .nc they are in net CDF format and then for plotting I need matplotlib. So let us try to open one single file to begin with. I am going to use the data set. So to create a net CDF file from Python you simply need to call the data set constructor. This is also the method for opening an existing net CDF file. So what I am doing is I am using the data set and then I am trying to specify the path where the file is being saved. Now because this data has a very lengthy name I want to make sure that I do not make any mistakes. So let me type the name of the file, remember to start with we are just trying to observe how one single file will look like okay. Now there are a means by which you can you know automatically read the strings but right now I will try to keep it simple, I will try to type the name by myself. But it is very difficult to do it if you have a series of files say you want to do time series analysis and you have 100 or 200 files of these in a folder. We will see how to work with multiple files shortly but for now I am going to type the name of the file and I have specified the path that is where the file is located in my system alright. Now let us see how the data looks like okay. What is contained in the data? You can see the latitude, longitude information is given, it is the data creation time is given, the projection is given, datum is specified as WG is 84 and towards the end the variables are given that is the different variables that are contained in the net CDF file are also given. Now say I want to access a specific variable, what I do is I can click on data.variables and then I get to see what all are the different variables contained inside the net CDF file. So here if you notice closely root zone soil moisture is named as RZSM okay, it is saved with a particular name. For example latitude within codes you can see it is saved as LAT and longitude is saved within codes as LON. So what happens is whenever you are importing a file into Python and say you want to extract specific variables let it be root zone soil moisture or longitude or latitude, you need to first find out what is the name with which these variables are saved inside the file and then you use commands to extract those specific variables okay. So now that we know how each variable is named like, let us try to visualize the root zone soil moisture data. So among the n number of variables present we will be using just three that is latitude, longitude and root zone soil moisture. As I mentioned before we are going to now extract the three variables that are required for visualizing the root zone soil moisture data. So I am going to type latitude equal to data.variables within square brackets and codes LAT okay. Name in net CDF file for latitude is LAT LAT and similarly the name of longitude in the net CDF file is LON long and the name of root zone soil moisture in the file I am going to use it to extract the variable RZSN. So now we have extracted the latitude, longitude and root zone soil moisture data. Let us see how it looks like okay. For example, if I want to see what values are contained in latitude I just type latitude and then the coordinates are visible to me. Similarly I can see the values contained in the array longitude and I can also see the values that are contained in root zone soil moisture which has been extracted to soil moisture underscore RZ. The values you see in front of you. Let us try to visualize the same and as before by now I hope that you are familiar with the basic commands used to visualize raster data in Python. So we will use the PLT.IM show function to plot and for the color bar we have PLT.color bar okay. So let me directly use these functions PLT.color bar. So this is how the plot looks like. You can see the legend it is from 0 to 0.7 and as before we can make the figure more legible by adding the title, by adding the x label and by adding the y label which by now I am hoping that you are familiar with okay. So now what we did is we have downloaded the data, we have extracted a single file. I have looked at the variables that are contained within the net CDF file and for plotting what we have done is we have extracted just three variables what we will be requiring that is latitude, longitude and the soil moisture information. So it is root zone soil moisture data that is being shown here. Now as I mentioned earlier say you have 100 or 200 files located in a folder, you cannot go on type the name, the lengthy name of each and every file is not it. You need to automate the process or you need to use a certain function that is going to do this easily. So as the next step let us try to work with multiple files, multiple net CDF files in Python. So as I mentioned earlier for this I am going to call glob and numpy okay gluby glob and numpy. So as mentioned before in case you do not have this installed any of the libraries whatever I mentioned in the tutorials in case you do not have this installed please go to the command prompt to install it and then you import the same in Python. So because I have already installed these I am directly going to import globe and import a numpy as NP. So the globe module this finds all the path names matching a specified pattern okay. Let me re-iterate the glob module GLOB this finds all the path names that are matching a specified pattern. For example if I want to read all the net CDF files which are having .nc extension I can directly use globe.globe and then specify the pattern within brackets that is .nc okay. Now this is a useful part of the Python standard library wherein globe is short form for global okay. I am going to set recursive as true now let us see whether the function works. So I say I want to see the list of paths you can see all the files which are having .nc net CDF extension as .nc will be displayed okay. So that I do not have to go and then type the name the lengthy name one by one okay. So what we will do is we will try to find out how many files are there okay. I use the len command which stands for length it shows me that there are 217 files that I have downloaded for the Ganga basin for the period from Jan 1 2020 to end of August 2020. So now let us try to read all the variables contained in all these files okay. For that let me first create a 0 array using np.zeros function and using latitude, longitude and the list of paths and I am going to create a multi dimensional array okay that is why the size of that is specified it is basically a 0 array okay a 0 array is created. Now what we will do is once we have created the 0 array I want to use a simple for loop to extract the root zone soil moisture from each and every net CDF file and to write it to the 0 array okay that is why the size is specified in such a way that I have np.zeros bracket length of latitude, length of longitude and length of list of paths that is nothing but 217 in this case. So let us try to use a simple for loop as I mentioned earlier why we are using it so that we can extract the root zone soil moisture into this 0 array okay. So I am going to type for i in range 1,217 you do not have to memorize 217 you can even write length of list of paths okay I am going to use data set and let me try to extract the soil moisture. So I am going to write value dot variables within bracket r z sm that stands for root zone soil moisture. So what I am trying to do I am trying to enter these values from each of the net CDF files into the empty array, empty 0 array that I created with specified dimensions okay. So now I am going to type that so every time soil moisture information is read from a file it should be assigned to the empty array multidimensional array that is DAT that okay. So if you go through you will find that at many places there are no data value okay for n number of reasons. So if at all you do not want these values to interfere with your programming what to do is we can easily represent the no data values as NAN that is not a number okay just simple lines to make it happen. So for this I am going to use two simple lines going to write whenever the value is less than or equal to 0 assign NAN to it by using NP dot NAN okay NAN is not a number. So just to see the same okay so now you see that wherever there are no values it is replaced by NAN alright. So what we did right now is we downloaded the data saved it in a particular location and then we learnt how to extract variables from a single net CDF file and here we also saw how to read the n number of net CDF files which are available at a particular location okay and then how to extract the variable of interest which in this case is root zone soil moisture into the empty multidimensional array okay. Now let me take your attention to slightly different topic which is known as trend analysis. So here as part of this tutorial we will be trying to perform man-kendall trend test sometimes it is called as the MK test. It is used to analyze time series data to understand whether they are consistently increasing or consistently decreasing okay monotonic trends and it is a nonparametric test which means it works for all the distributions that is the data does not have to meet the assumptions of normality but the data should have no serial correlation okay because in hydrology we have time series of data and normally we try to do trend analysis to find whether or not the data is having increasing or decreasing trend. So just some simple commands on how to conduct trend analysis in python will be shown. Typically all man-kendall test functions have similar input parameters the input functions required are you know firstly you have to give the vector that is the data and the significance level alpha the lag that is number of first significant lags and the period okay. So these are the inputs that are required for the function to perform man-kendall trend test. So the man-kendall test they return a named tuple which contains trend, H value, p value of the significance test, kendall stow, man-kendall score, variance and send slope. So to perform the man-kendall test we need to install pi man-kendall as shown in the screen and to install the same the following command can be used as is displayed here. Let us see how this is achieved in python. We can see how to conduct trend analysis using the root zone soil moisture data that we have downloaded. Again you know only a few months of data are being downloaded for this particular tutorial and that is for representation purposes okay. So let us see how to achieve this in python. So what we are trying to do is we are trying to carry out the trend analysis using the man-kendall test mk test okay using the pi man-kendall which is a library which helps us to perform the mk test. So what I try to do first is I try to import the library. So I have already installed the same, now I am importing the same. So as an example what I will do is I am going to pick a single pixel okay. I am going to pick a single pixel and then I am going to extract the time series corresponding to that particular pixel. So if you remember we have downloaded the data for the entire Ganga basin. In that I am going to pick a single pixel and then look at the values for that single pixel over the entire time for which I have downloaded the data for. So let me say 10 comma 10 okay 10th row 10th column all the values I am picking that is what the colon indicates and then mk.original underscore test and picks 10, 10 that is the value for a single pixel I want to see whether there is any trend. So you see the output gets displayed as trend equal to no trend, H is false, the value of P is given, Z is given, kendall's tau is given, sense slope is given, intercept is given. So all these details you get as output when you are trying to use this function and also let me re-iterate typically trend analysis requires lengthy time series of data. So whatever you see here is for representation purposes only and this is for one single pixel for which the values you see are displayed in front of you okay this is for one single pixel and the time series it is just a few months okay alright. So alternatively you know if you are interested let me try to introduce you to something on a spider you know the same set of commands as you used in Jupiter notebook it can be carried out through spider which is nothing but a free and open source scientific environment written in Python for data analysts. So if you just go to the Anaconda Navigator you can click on launch button and spider gets launched and this is the screen interface that you see in front of you. So what you do is if you want you can go to tools and preferences and change the directory okay the current working directory can be changed. So spider it is included by default in the Anaconda Python distribution wherein as I mentioned earlier you can directly launch the spider and an interface like this shows up and if you want you can change the current working directory to any folder any directory okay. The location I can even point to the location where the files have been downloaded and kept okay. And as before even in spider we have to install the packages before trying to import it and as I mentioned before these packages are necessary to handle the next CDF files in Python. So in the previous tutorials we have already installed the next CDF for library. So you see these are the next CDF files that are downloaded by you. So as before you can type the name of the next CDF file okay and then when you right click it tells you whether or not to run the command. So always install the libraries before you are trying to import it and you know it is not mandatory that you work with spider but if you are more interested to explore I just thought to introduce this section wherein you are aware that such facility is made available. So the syntax and commands that you use in the Jupyter notebook as well as spider are same okay just if you are interested and you can see three windows here. So what I am trying to do is in this interface I am trying to type the name of the file the lengthy name of the file and to make it run once I finish typing the name of the file I just need to right click okay let me expand I just need to highlight the line and then right click and the run cell option comes up okay run cell alright this is just for just to make you aware that spider is also available. So in this particular tutorial we learnt about small satellite data how to download the same and we also learnt how to download the data, how to import the net CDF files into Python and also we got to know the necessary libraries which are used to open the net CDF file and to extract the required variables one by one and we also saw how to handle when multiple number of files are available. Finally we touched upon how to carry out trend analysis so let me hope that you found this section helpful and I shall meet you in the next class thank you.