 Hello and welcome to Tutorial 10, Handling Passive Microwave Observations. So just to give you a brief overview, till now from the previous tutorials we have been learning one by one about the different satellite missions capable of examining the water levels and then we understood how to deal with data from JSON and several satellites. Moving on, we also understood about small satellite that gives a soil moisture information and also the grey satellites which give us information about the terrestrial water storage. So in today's tutorial, we shall learn about the missions dedicated to the study of precipitation like TRMM, GPM and tropics to name a few. So just to give you a brief overview, precipitation it depicts very high spatial and temporal variability and shown in front of you are some distrometer, radar and rain gauges that help us to measure precipitation. So surface gauge observations like rain gauges are usually relied on to measure precipitation just like what you see here. So the rain gauge which are commonly used to access precipitation at the surface, it measures the depth of rainfall that gets accumulated over a period of time. And we have several different types of rain gauges like tipping bucket gauges or optical gauges etc to name a few. Of course each of these have their own advantages and disadvantages that is the gauge, the gauges suffer from site locations or for example there may be some areas which has a denser network of rain gauges whereas some other areas which are difficult to access like the mountainous terrains, they may have irregular distribution of networks and also they may be subjected to instrument errors. So gauges have their own advantages and disadvantages. Now what you see towards the left side and in the middle is nothing but the GPM distrometer and the radar. So radars and distrometers have also been used to examine the composition of precipitation and also to understand the underlying physical processes. But then you know if I need to plan a global network of Doppler weather radars that is of course limited by lack of accessibility and yes funding is also an important factor. So coming on to distrometers what you see towards the left side. So these distrometers they detect the rain drops and they measure its size and this helps in the micro physics parameterizations which are typically used in numerical cloud models. Just I am mentioning these terminologies to make you familiar with a few existing methods. Now about radars we have already seen as part of a previous tutorial. It scans the atmosphere horizontally and then it helps capture the three dimensional structure of precipitation. So as part of previous tutorials we have seen in detail how to access the data and how to analyze the Doppler weather radar data from next rad and the MOSDAC website. Now coming on to the automatic weather station you know that is the AWS that is another means which is an integrated system of components that help us to measure about weather parameters like temperature, wind speed, wind direction, solar radiation, precipitation etc. So with this background let me share a walkthrough of India's network of automatic weather stations that is AWS which was established to strengthen the surface observation network. So what you see here is the details from the India Meteorological Department website. So data from AWS are having several applications in operational meteorology and agro-meteorology and here you see at Colaba what are the parameters of temperature, relative humidity and rainfall and the time is also given. So you can zoom in and click to a particular location which displays the details and what you see here is the automatic rain gauge network over India that is ARG. Again you are free to zoom to a particular location and click on it so that the details are getting displayed. Now this particular site allows you to download the data once you specify the state and district and you know you get information like cumulative rainfall, air temperature, relative humidity etc. So just for the sake of example I can type in any state, any district, click on the data that I need and then I can use the options towards the right side that is there is a download option, there is a view graph option and there is a view data option you know. So just for the sake of example let us try to view the data, it shows cumulative rainfall at the top and then we have air temperature as you scroll you can see details about relative humidity. This website also allows you to download the data but of course you need a user name and password created before the same can be accessed. Now coming on to the weather radar network of India. So we covered a little bit as part of a previous tutorial. So if I go to the India Meteorological Department website mosam.imd.gov.in, I can see the network of radars over India and I can also click on a particular station say I click on Mumbai and then towards your right side you see the radar images being displayed. I can even get to create an animation, radar animation is possible say for example I click on the radar in say Delhi or Cochin any place I can zoom in and when I click on the particular location it allows me to see the radar images. So displayed here are the plots if you remember we learnt how to create these plots in python. You can see the reflectivity values in DBZ are given different colors and it also allows you to as I mentioned earlier to visualize the radar images say for the last 3 hours as an animation. So you can see the colors represent reflectivity in DBZ just to zoom out overall the network and more details about each network, each radar you can get by visiting the IMD website. Now let us try to move towards satellite based estimation of global precipitation. So till now we have seen different instruments that can be placed at the surface which helps us measure precipitation. Now coming to satellite based estimation of global precipitation shown here is the precipitation from iMerc from the following website gpm.nasat.gov.in. See satellite systems as such they provide invaluable means of measuring global precipitation at regular intervals. Now the number of sensors that are made available in meteorology has advanced markedly. So broadly we classify the sensors operating in visible infrared red region on geostationary and low earth orbits, passive microwave sensors on low earth orbit and active microwave on low earth orbit satellite. So these are the broad sensors that are capable of giving us information about precipitation. Now often to increase the accuracy the data collected from say visible and infrared as well as passive microwave sensors are blended integrated together so that you get better coverage and better resolution of precipitation. And such multi-satellite precipitation source is depicted that is one sample image is being shown from iMerc which stands for integrated multi-satellite retrievals for GPM. GPM is a global precipitation measurement mission. We will talk about that shortly. But just to summarize there are multi-satellite precipitation products available that merge or blend the information captured individually in different regions of the electromagnetic spectrum. When I say different regions I am specifically referring to visible infrared region and of course the sensors that operate in the passive region and the active region that is passive microwave sensors and active microwave sensors. Now with this background we have an overview of different means of measuring precipitation. So we know that rain gauges give us point based measurement of precipitation. So wherever you have a rain gauge it gives you the depth of precipitation that has occurred in that particular region. Now satellites scan the atmosphere vertically it is looking down so it is scanning vertically and Doppler weather radars scan horizontally to measure the precipitation. Now say assume you have a time series of rainfall from AWS station any AWS station or rain gauge. So how do you analyze this time series from a point location in python? So let us try to get an idea of how it is done. So with this background now we have an overview of different means of measuring precipitation. So let us try to analyze this using python now. So I am assuming that by now you understand that rain gauges give you point based measurement of rainfall and satellites they scan the atmosphere vertically and Doppler weather radars they scan the atmosphere horizontally. So before we try to look at data from the satellites let us try to analyze the time series of data from a point source say it can be any AWS or ARG stations. So as before I have opened an empty Jupiter notebook and I have named it with tutorial 10 followed by name of the course. So first step is we are going to import all the necessary libraries required for us to perform this tutorial. So I am going to import numpy, scipy and matplotlib.pyplot import numpy as np, import scipy and again import matplotlib.pyplot as plt. So by now I am assuming that you must be familiar with what each of these libraries offer for image processing. So for this tutorial let us try to work with hypothetical values of time, rainfall, latitude and longitude. So instead of going to the site zooming in directly clicking on a site and downloading the data we are to keep it generic, we are trying to create hypothetical values of latitude, longitude, time and variable that is rainfall. And for this we will use certain functions, let us see them. So firstly I am going to introduce you to np.lenspace, it returns evenly spaced numbers over a specified interval, np.lenspace and to create hypothetical values say 50 hypothetical rainfall values I am going to use np.random.rand because it indicates random integers. Similarly I have created 50 hypothetical values of latitude and 50 hypothetical values of longitude. So these are the data assume you get from a station, a single AWS or a single ARG station. So what we will do is we will try to see how the bar plot of rainfall values look like. Now to plot the bar plot we need to use plt.bar function and remember I have imported matplotlib.piplot as plt and hence I am using plt.bar directly. So it creates a bar plot with bars positioned at x with the given alignment and to make the figure more legible you can add x label and y label as well. So here I am going to add x label as time and y label as rainfall. If I run I can see the bar plot of rainfall values and say you need to add the units to rainfall and time. I can maybe change the x label as time and minutes and rainfall in millimeters. So on the x axis you have time and on the y axis we have rainfall, so bar plot. Now let us try to estimate the cumulative rainfall values for which I am going to use np.cum sum that is cum sum it returns the cumulative sum of elements along a given axis, cumulative underscore rainfall. If I just type on it I will get a series of rainfall values, cumulative rainfall values that have been estimated using np.cum sum. Now let us see how it looks like. So for that I am going to use plt.clf and plt.plot. Now plt.clf actually clears the entire current figure with all its axis but leaves the window opened such that it may be reused for other plots. So now I am going to plot the cumulative rainfall values what you see here, remember these are hypothetical values but you may as well repeat this exercise with real value from a real station. Now let us try to create a spatial plot by interpolating the values you get from points. By points I mean assume you have a density of AWS or ARG stations and you have individual time series for all of these stations. Now you want to interpolate the rainfall values and create one spatial plot. So this particular section deals with that let us see how to create an empty grid and how to interpolate the values to create a spatial plot. So in the first line what I have done is I have used np.mesh grid. Now mesh grid it returns the coordinate matrices from coordinate vectors and it is very useful to evaluate functions on a grid. And then I am going to use grid data, it interpolates the surface at the curie points. See if you make any mistakes while typing the function names the moment you press run the program is going to prompt you to correct the errors. Small spelling mistakes let me correct and if I run it again I can see what grid data contains. So I have used mesh grid and grid data, see rainfall at latitude and longitude needs to be interpolated with respect to x and y. Now let us visually try to plot and see how the plot looks like. So what are we doing? You have independent time series from a network of ARG or AWS stations assume and you are using these values to interpolate the rainfall and create a spatial plot. So again I am going to use plt.clf for clearing the entire current figure. And then I am going to use plt.contourf as the name suggests it plots contours. And then of course I can add the color bar and to make the figure more legible I can add the x label, I can add the y label because these are hypothetical values I am going to add a fancy name to the x label and y label. So let us see how the figure looks like. So this is how the figure looks like. I have interpolated the rainfall values from the individual locations that are marked by the magenta dots. So I have interpolated and created a spatial value of rainfall and you know as I mentioned earlier you can easily repeat this exercise with point based rainfall values from A, AWS or ARG network. So in the beginning we introduced a little bit about the sensors that are capable to measure precipitation and then I mentioned that they can be broadly classified into sensors that operate in the visible infrared region, passive microwave sensors and active microwave sensors. So shown here are the missions a few of them that dedicatedly offer data to examine precipitation from space. See passive microwave radiometers they offer a direct method of measuring precipitation because they are sensitive to precipitation sized particles or what you call as hydrometeors. So shown here are the three names of tropical rainfall measuring mission TRMM, global precipitation measurement mission GPM and tropics which is going to be launched in 2022. So tropics stands for time resolved observations of precipitation structure and storm intensity with a constellation of small sats that a small sats stands for small satellites. So please be mindful that there are satellites like AMSRE that stands for advanced microwave scanning radiometer for earth observing system and SSMI which measure geophysical parameters related to water. But as part of this tutorial our discussions are going to be focused on these three missions that is as you see in the screen in front of you TRMM, GPM and tropics. Now seen towards your right side is an anaglyph you know it is best viewed with 3D glasses. So this anaglyph shows the footprint size which is not to scale of various TRMM microwave imager channels. Now TRMM microwave imager is an instrument on board the TRMM satellite and it operates in 5 different frequencies as shown here 10.65 gigahertz, 21.3, 19.35, 37 and 85.5 gigahertz and when TRMM microwave imager looks down at the earth using these frequencies the footprint size that is the major and minor axis of the ellipsoidal footprint ellipse. It is going to differ and the differing dimensions not to scale are shown as an anaglyph. So the footprint sizes are shown just to give you an idea that these missions they operate in different frequencies within the microwave region and each of these frequencies result in a differing spatial resolution. But nevertheless to summarize there are different missions, different instruments on board different missions that operate in different frequencies within the microwave region. So GMI also uses a set of frequencies and showed us topics. So just to introduce you to the instruments on board TRMM. Now as you see here there are 5 instruments of which we have something known as a TRMM microwave imager TMI we have already discussed this. So 4 of the frequencies have dual polarization and there is one frequency 22.235 gigahertz which is at single polarization and we have the precipitation radar PR which is you know one of a kind. It consists of an electronically scanning radar that operates at 13.8 gigahertz. The aim is it gives you 3 dimensional rainfall distribution over both land and ocean. Similarly you know you have LIS that stands for lightning imaging sensor it is a very small but then it is very sophisticated instrument to study about lightning over the tropical region. And then we have series that is cloud and earth radiant energy sensor which is used to study about energy at the top of the atmosphere okay and of course we have the VIRS that is visible infrared radiometer and this operates at 5 different frequencies and the aim is to provide high resolution observations on the type of cloud that is cloud type and cloud top temperatures okay. Now let me give you a brief overview about GPM which is nothing but a successor to the TRNM mission because TRNM functioned till the year 2015 and then GPM the successor mission was launched and it consists of a core observatory satellite that is used as a reference standard and this core satellite uses two science instruments okay one is the GPM microwave imager GMI and the next is dual frequency precipitation radar that is DPR okay. So here we have a radiometer as well as a radar okay as part of GPM. So by now let me assume that you are familiar on how to access the data sources one by one how to create a user account if necessary and then proceed with downloading of the data by specifying a bounding box or giving the coordinates corresponding to the region of interest because over the series of previous tutorials we have been showing how to access each data source and how to download the data. So for this particular tutorial I am going to use pre downloaded sample data sets of GMI that is GPM microwave imager and now we shall see how to begin working with the net CDF files of GMI okay alright. So as before I need to import all the required libraries. So for net CDF file which has an extension .nc I need the library net CDF4 I am going to import that as NC and then I am going to use NC.dataset and specify the name of the file that I need to be opened in Python okay. So as mentioned before these files have pretty lengthy names the nomenclature is different for each mission. So in order to be careful I am going to type the name of the file so that there are no syntax errors later on okay .v05 that is version 5 .hdf5 .nc4 one file of the set that I have downloaded. Now to see what NC underscore file contains I can simply click on it and run and it gives me the entire set of details okay. I can see the generation date, time and other details say the bounding coordinates okay. I can see if I scroll down I can see the bounding coordinates there you go and most importantly I can see the variables that are contained in this net CDF file okay. Say for example I need to see in detail what all are the variables inside this file. I can just type NC underscore file .variables and click on run so that all the variable names are going to be displayed here. So I can see something known as ice water path and then I do have surface precipitation and also the units millimeter per hour okay and the latitude and longitude are specified as n lat and n long okay right. So all the details about each of the variables that are contained within this net CDF file you can see in this particular screen okay. So now that we know the variable names say I need to extract the latitude and longitude and rainfall values okay. So I am going to use NC underscore file .variables and specify the name of the variable that I want to extract so that is n long that is longitude. Similarly I need to extract say the latitude values again I am going to repeat NC underscore file .variables within square brackets the name of the variable that is n lat latitudes and say I want to extract the surface precipitation which you just saw in millimeter per hour. So again I can specify the exact variable name okay surface precipitation what you see is the deprecation warning that we have addressed as part of an earlier tutorial. So when I click on precipitation you can see that the data is saved in the form of a mass grid array, mass array okay. I can even see visualize how the data looks like okay. So I am going to use plt.imshow function to visualize the precipitation data. See sometimes when you plot the global precipitation data there may be a requirement for you to rotate the figure okay I am not going to go into those details now. So this small section is for you to familiarize yourself with how to open an hcdf file from GMI that is GPM microwave imager and how to read it in python, how to see the variable names and how to extract each variable name if required and then how to plot it okay. Now you know as I mentioned earlier there may be a need that especially as meteorologists or as hydrologists you may deal with n number of net cdf files okay then it is impossible for you to keep on typing the name of each and every individual file you need to automatically read it in python and then extract the variables to do the analysis. So what we will do is we will try how to process multiple GMI files okay. So now I want to read and process multiple net cdf files from GPM microwave imager. So as before I need to import the necessary library. So here I am going to use import numpy as np okay. So wherever precipitation is 0 I want to replace it with nan, nan stands for not a number. So that it does not interfere with the calculations. I can even use np.nan mean to get the mean value of precipitation and say I want to estimate the variance standard deviation of precipitation even that is possible. So np.nan var helps us to calculate the variance of precipitation okay and to see how the data looks like okay it is in the form of a mass deray float 32 is the data type okay. I can visualize the mean value of precipitation and the variance of precipitation alright. As I mentioned earlier I want to read multiple net cdf files of GMI. So as mentioned earlier I have pre downloaded this set of image which is being used for this particular tutorial. So I am going to import glob net cdf4 and numpy and I want to use glob.globe within brackets np okay recursive equal to true I am going to set it as true. We have used this line of command as part of a previous tutorial. So as before I can see the complete set of files that are available in the particular location okay which is having .nc4 as extension. So typically there are 12 files here it may be possible that you have many more files that will take of course some time to process. So to keep things simple and generic I am working with just 12 files. So what I am trying to do is I am trying to open each of the files I want to extract the latitude, longitude information and the precipitation to be more specific the surface precipitation and millimeter per hour from each of these files I want to extract it okay. So as before I am going to open it in that one and that one dot variables I am going to specify the variable to be extracted in square brackets and single quotes nlon, nlact and surface precipitation. Now I am going to create a array of zeros okay the dimension should be from length of longitude le is length, length of latitude and length of list of paths which is 12 that is number of files okay. Going to ignore the deprecation warning now and then I can click on long and then a mass array appears wherein the list of longitude values are displayed similarly I can display the list of latitude values and of course the precipitation values in each of the files okay. I am going to use a simple for loop now this was discussed as part of previous tutorial. So I am not going into the line by line explanation I am going to directly use a simple for loop to extract the values from these 12 files without the need for me to specify the file names individually for each file okay. Because I have read it to list underscore of underscore paths here i is in the range of 112 that is 12 files you do not have to memorize the number of files in a particular folder or directory because you can just give length len within brackets list of files okay. So again I am going to extract the variables contained in the files for representation purpose I am going to extract the surface precipitation values okay just a couple of errors. Now once I fix it I can display the result the output okay. So what are these? These are the precipitation values to be more specific the surface precipitation values displayed as a mass array. Now if for computation purposes say you want to replace wherever there is precipitation 0 you want to replace it with nan not a number that is also possible you can use np.nan okay and as before you can also estimate the mean value of precipitation and variance of precipitation alright. The only difference is earlier you are trying to estimate these for one single net CDF file and here you have a number of net CDF files in a particular location file or directory and you are trying to access it automatically and you are trying to extract the precipitation values. Now say I want to create a time series containing the spatial mean of each month okay it will be a good practice to use np.zeros and then fill in that within a loop with the spatial mean of each month. So again I am going to use a simple for loop where i is in the range of 1 to 12, 12 being the total number of files that is present in the directory. So each time I am going to read the values and then estimate the mean using np.nan mean and now the spatial mean of each month is saved in ts underscore monthly and I can directly plot it using plt.plot function plt.plot. So now as part of this particular tutorial we understood about how to measure precipitation and what are the different ways in which precipitation can be measured using distrometers using drain gauges using Doppler weather radars and then we came to the different regions of the electromagnetic spectrum which can give us information about precipitation such as the visible infrared, passive microwaves and the active microwaves. And then we showed how a single net CDF file of GMI that is GPM microwave imager can be brought into python red and then we can also extract the required variables. Finally we have seen that if instead of one you have multiple net CDF files say from any location for this particular tutorial the example of GMI was given. So if you have multiple net CDF files how to open it in python and how to extract the required variables that was discussed and ending with how to plot a time series of spatial mean values, monthly mean values of precipitation. So let me hope that you enjoyed this section of the tutorial I will see you in the next class. Thank you.