 Do you know how to turn it on? No. That's probably good. Hello. Good morning. First, I would like to thank Fred and Glenn for giving me this opportunity to attend the workshop. And I would like to talk to you about sub-seasonal predictability of precipitation. So the outline that I'm going to follow is, first I would like to give an overview of the sub-seasonal to seasonal project. Maybe some of you are not familiar with it. Then I'm going to discuss the predictive skill of precipitation in the sub-seasonal range. And the community is defining sub-seasonal range as lead times between two, from two weeks to eight weeks. And this problem I have been studying. First, one of the questions that I want to investigate is how forecast skill varies with precipitation intensity. And first I'm going to show you results for the winter season in the United States. And later I'm going to show you some results for South America. And then conclusions. So if we think about when numerical weather prediction started in the 50s, there has been tremendous progress in the forecast skill in the short to medium range. So from one day up to 10 days or one week. So this graph here, actually this paper by Bauer et al. summarizes very well the progress that has been made in numerical weather prediction in the medium range. This figure here shows here's year and here's some measure of forecast skill. And we have several curves. Each of these curves are for some lead time like the blue is for three days, red for five, seven and ten days. The top of the curve is for the scale in the northern hemisphere and the bottom for the southern hemisphere. So what we see is that over time forecast skill has been increasing significantly. And also the difference between the northern and the southern hemisphere has become smaller and smaller. That means we have better observations in the southern hemisphere and also models have been improving in the southern hemisphere. So this is on one end of the spectrum like from short to medium range on longer time scales like seasonal time scales. One thing that I wanted to point out here is that the predictability on short lead times up to medium range, they are heavily dependent on the initial conditions. Now, on longer time scales like seasonal time scales, we have seen seasonal forecast become operational. This example here from the IRI, the Institute for International Research Institute, we see that now several centers around the world they are producing seasonal forecast. The predictability on these long timescales comes primarily from boundary conditions and ANSO is one of the main sources of potential predictability on these timescales. So between short and medium range and seasonal timescales, there is a big gap in what we know about predictability in the subsesional timescale. And that's what I wanted to point out here. So in the subsesional time range between 10 days and up to a few months, what we call subsesional predictability, we don't know if we can explore predictability on these timescales. And these timescales, they are very important because if we can develop accurate and reliable forecasts on these subsesional scales, there can be huge benefits for society because management decisions and also emergency preparedness, they can really take advantage of forecasts on these timescales. And so many applications can be made having these forecasts in mind. So based on these facts, several organizations like the World Weather Research Program, the World Climate Research Program and Thorpex, they implemented this project called a subsesional to seasonal prediction research project or S2S. And one of the goals is to improve forecast scale and understanding on subsesional to seasonal timescales, especially with emphasis on high-impact weather, so extreme weather events, and also to promote exploitation of applications from the forecast on these timescales. So the focus of this project, the S2S, is from two weeks up to a season lead times. Now, together with this idea for this project, the S2S research committee has realized that there are some sources of potential predictability on these timescales. One of them is the Madden-Julian Oscillation, the MGO. Also, stratospheric initial conditions are very important to exploit subsesional forecast. Land eyes and snow initial conditions, it seems that they play an important role in the subsesional forecast and also sea surface temperatures. Together with this S2S project, there are several, a few sub-projects. They all focus in some specific goals. There is one subsesional S2S sub-project devoted to understand more teleconnections and how they impact subsesional, S2S predictability. The MGO is also another sub-project. One focusing on the monsoons, one for Africa, extremes, and verification and products. One of the sub-projects that I'm participating is on the extremes. This is led by Frederic Vitard from ECMWF. In this S2S project, we are interested in assessing the predictability of extreme events such as heat waves, cold waves, floods, extreme heavy precipitation, etc. Also, subsesional predictions of tropical storms and we are doing this on case studies and things like that. So now, if you're not familiar with the S2S project, I would like to discuss a few points here because I think they are important in how we can investigate S2S predictability. So if you go to the ECMWF website in the S2S project, you can find this table. Let me explain a few things here. There are currently 11 models participating in this S2S project. One thing that is a little bit complicated in this project is that each center is developing forecasts in a different framework. So some centers, they forecast up to 60 days, other centers up to 30 days, and things like that. For example, in ECMWF, the forecasts run out to 46 days lead time. They have different resolutions, but in the S2S database, they all have been standardized to one degree latitude-longitude resolution. Now, in these columns, we are seeing here is for near real-time, as the S2S forecasts become available, they are in the database. On these columns here, they are for re-forecast. The near real-time data also varies quite significantly. Some centers, they produce forecasts every day. They initialize the model and run out to 30 days or 60 days, 40 days. Other models, they have less frequent initializations. The ECMWF runs two forecasts. They initialize twice a week, Mondays and Thursdays, and they run out for up to 46 days. For ECMWF, in near real-time, there are 51 members in the ensemble. And you can see that other centers, they have very small ensemble size. Some, they have only four members, 25, so it's quite variable. Now, one thing that we need to do also is calibration in the forecast and also have large samples. So each center developed re-forecasts. And for the re-forecasts, it's the same thing. So they are quite variable the way they do. For the ECMWF, the re-forecasts have 11 members in the ensemble size. Other centers, like the ANSEP, they have only four members in the re-forecasts. So this poses some issues of how you can investigate some problems. So what I'm interested is in extreme or heavy precipitation. And precipitation is very different from the other fields, like the circulation or even temperature. So precipitation is highly variable in space and time. So ideally we want forecasts with a very large ensemble, so that ensemble members, so that we can spend all the possible outcomes. The other thing is that we need, for precipitation, in order to validate the forecast and understand more about sub-seasonal predictability, we need very large sample sizes. So working with the near real-time is quite difficult because especially if we don't have initializations every day, if you have only twice a week, the sample size becomes very small. So here in this talk, I'm going to show you results based on the re-forecast. For the ECMWF, we have 20 years, twice initializations each week. They run out for 46 days, and we have 11 members in the re-forecasts. I have also looked at the ANSEP model. It seems that ECMWF, at least for precipitation in North America, in the United States, ECMWF is better than the ANSEP. Not much, but it's better. Now working with the other models to understand sub-seasonal predictability of precipitation becomes challenging because of these small sample sizes. So why understand more about predictability of extreme events? This is quite obvious. In this graph here, it shows the number of weather fatalities in 2016. And if you look at the red bars, so here in the bottom is the type of weather events. So if you look, for example, at floods and winter storms, many fatalities occur in the United States every year. So this is just to motivate that we can have tremendous benefits if we can explore sub-seasonal predictability of extreme precipitation. So the questions that I want to investigate are listed here. So first of all, I think it's important if we determine if there is predictive skill or what is the current predictive skill of precipitation in the sub-seasonal range. And because we are dealing with extreme events and also very long lead times, the forecasts have to be probabilistic in nature. Then the second question is how does the predictive skill of precipitation vary as a function of precipitation intensity? One outcome of this could be to determine if heavy precipitation is more or less predictable than light to moderate precipitation. And I'm going to show you results for the United States. But also what is important is to understand how predictability of heavy precipitation varies on different climatic regimes. And I'll show some results later for this. So in the first part of this talk, I'm going to show, as I said, exam this S2S probabilistic forecast skill of precipitation in the Contiguous United States during winter time. So we're going to look at forecasts from November 1st up to March 31st. And this plot here shows the climatology. So during the winter season, precipitation is very heavy in the western United States, especially along the coast or on the western states, California, Oregon, and Washington. Then you have a minimum here in the central part. And then in the eastern part of the US, precipitation is very heavy. So first I'm going to show you several steps in the methodologies so that we can understand more about the results. So I'm going to use only the ECMWF model. The forecast range goes out to 46 days. The spatial resolution that we are using is a one-degree latitude launch too. There are 11 members, one control and 10 perturbations, two initializations per week, always on Mondays and Thursdays. The period is November 1st to March 31st from 1995 to 2015. I'm using the re-forecast. For verification, I'm using the CPC, the Climate Prediction Center, Unified Precipitation Graded Data. And then there are a couple of intermediate steps. So first I interpolate the model forecast to the same grid of the verification. Then I estimate the daily mean model bias. The model bias is going to be a function of space, latitude launch too, and also the lead time. And I remove the model bias from the forecast for each day, each grid point, et cetera, and each lead time. Now, because we are in this sub-seasonal project, we're interested in exploiting predictability. And after, well, precipitation is always very variable in space and time. So here, instead of looking at a daily precipitation, I'm averaging into a weekly mean precipitation. And that helps to reduce noise in the data. Then for each grid point, I'm going to fit a gamma probability distribution function. And I do that for the validation data set and also for the forecast. Now, for the forecast, since we don't have a free run from the ECMWF, we don't have a climatological run from the ECMWF, I use the one-week lead to fit the gamma PDF. So here is a schematic of the PDF for precipitation in this time period from November to March. Instead of just looking at one precipitation intensity, I'm actually computing the percentiles in the PDF from the 10th, 20th, 30th, et cetera, up to the 90th percentile. So I have nine different categories of precipitation. In this first set of categories of precipitation, I look for the probability forecast of precipitation being less than a given percentile. For example, less than the 40th percentile, 30th, et cetera, up to the 10th percentile. That's what is in orange. And then the other set of probability forecasts, when the probability forecast is greater than, for example, the median or the 50th percentile, greater than the 60th, 70th, et cetera, up to 90th percentile. So we go from moderate precipitation up to the tails of the PDF, to very heavy precipitation. Now, for the forecast, the probability forecast is computed as the number of members forecasting that given event divided by the total number of members. So 11 members here. The validation, as I said, is from November to March 1995 to 2015. Each season, from November to March, there are 44 initializations. Each initialization, the forecast run out to one to six weeks lead time. So that's the lead range that I'm going to show results. So in total, for each grid point and each lead time, there are 880 forecasts that I'm going to validate. And here you can have an idea if we have a very short, a very small sample size in the reforecast or if we have very small ensemble members. So that poses some complications. Because in the end, we end up with a very small sample size to verify the forecast. Now, there are many ways to verify the forecast. For probabilistic forecast, I'm using this methodology here. So if we think about a given precipitation category, which depends on the percentile that I showed earlier, and lead time, we're going to have pairs of forecast probabilities here indicated by y. So the probabilities, of course, are between 0 and 1. And for validation, we have the outcome. So this OK is 0 if the event doesn't occur, and 1 if it occurs. Then to validate this probabilistic forecast, we use the bryer score. The bryer score is the mean squared difference between the forecast probability and the outcome. So we want bryer score to be as small as possible, like an RMS error. Now, the bryer skew score is computed as a function of the bryer score here normalized by the reference. In the reference, we usually use the climatological probability. So we compute this from the observations with the CPC unified grid data set. So the climatological forecast, for that given day, you always use the climatological probability. So the first question is, what is the current probabilistic forecast skew in the sub-seasonal range, at least from the ECMWF model? So here is the bryer skew score, and the formula is there. First, looking at the bryer skew score, for forecasts of precipitation exceeding the 50th percentile or above the median. So each panel here, it goes from one week lead time, two weeks, three and four weeks lead time. And here is the bryer skew score. So as expected, for up to one week lead time, the skew is quite reasonable. So the colors here, they represent improvement from the climatological forecast. So most of, over the United States, we see improvements ranging from 30 to 40 percent better than climatology. In some places, it's less, so the improvement is only 10 to 20 or so percent. As the lead time increases to two weeks, three, four, the skew drops. Now if we look at another precipitation category, like when forecast, when the precipitation exceeds the 70th percentile, we see some interesting changes here. First, for one week lead time, actually the skew is better in this MWF model. So you see, in many places in this category, you see lots of yellows and light greens. For 70th percentile, you start to see improvements on the order of 50 to 60 percent over climatology. And also, not only for one week, but also in two weeks, you see, you still have a actually very reasonable skew into these weekly times. And if we increase the percentile or for this precipitation category, probably forecast when the precipitation exceeds the 90th percentile, for one week, you can also see that in many places here, the skew improves to 60, 70, in some places even higher. And the same thing for two, three, four weeks lead time. Another way of looking at these results is aggregating over different sectors in the U.S. So I split the contiguous United States into three sectors, Western, Central, and Eastern. And for each sector, I computed the percentage of that sector with bias skew score greater than a given threshold. And that's the type of curve that we get. So let me explain this. So this is when the precipitation is above the median value, the 50th percentile. Here in the bottom is the lead time from one week up to six weeks. The color represents the percentage of that sector that had bias skew score greater than the threshold. The threshold is here in the vertical. So for example, in the Eastern U.S., for one week lead time and bias skew score greater than 10 percent, almost from 80 to 100 percent of that sector had skew better than that threshold. So if you increase the threshold from 10 percent to 20 or 30, the percentage of that sector that had that given threshold, bias skew score, drops. So you see that it goes to only 10 or 20 or 30 percent of that sector. And as the lead time increases, the percentage of that sector with that bias skew score also drops. And the same for the other sectors in the Western and Central U.S. So it's interesting to see that in the Eastern U.S., the predictive skew of precipitation is not that great after two or three weeks lead time for precipitation exceeding the median value. Now if we increase the precipitation threshold, or in other words, when precipitation exceeds the 70th percentile, we get a very different curve. Now, for example, again in the Eastern U.S., the percentage of that sector that has a bias skew score greater than a given threshold increases to longer lead times. For example, up to three weeks, let's say three weeks, for a bias skew score between 10 and 15 or 20 percent, 20 to 30 percent of that sector has that given bias skew score. So these curves, they increase upwards and also to the right. Now if we increase the precipitation threshold to 90th percentile, we can see that there's still some skew on lead times from three to six weeks, which is encouraging. So that's good news. But the bias skew score is just one single measure that can give you an idea about the skew in the forecast. There are other measures to measure the quality of the forecast. One of them is called the reliability diagram. So in the reliability diagram, we have to compute some other statistic parameters. One of them is called the conditional average observation. So we are looking at the probability of the outcome or that event occurring for a given probability forecast. So you have to compute this statistical measure. But basically this tells you how well each forecast is calibrated. Now the other parameter is called the refinement distribution and you have to compute this n sub i is the number of times each forecast is used in this data set. So the refinement distribution tells you if the model can discern different outcomes. And here are some schematic possibilities of what can happen. So if the model is well calibrated, ideally this conditional average observation is going to follow along this one to one line. Now if you have over forecasting or wet bias in the model, they deviate from the one to one line. And you can have many different possibilities in the conditional average from the model. And the refinement distribution, if you have this type of distribution, indicates that the model has low confidence. And if you have this other type, it says that it has intermediate confidence and this type of the model is highly confident in the probabilities. This is what we get for the reliability diagrams. Now one complication is how to visualize this because for each grid point in the US I compute the reliability diagram. So in order to get an idea how the model, what is the reliability in the ECMWF forecast model, I average all the reliability diagrams over the US where the bright skill score was positive. So just to get an idea how the reliability diagram looks like in grid points where there was some skill. So here we are looking at the reliability diagram for one week lead time, two, three and four weeks lead time. So for one week lead time we see that the model, the conditional frequency doesn't fall along the one to one line. So the model, actually the conditional frequency, it is above the one to one line for probability forecast from zero to point two. Now for probability forecast greater than that then it deviates from the one to one, but below the one to one line. So it indicates the model has some conditional bias in the forecast. Now as the lead time increases you can see that this line gets even further and further from the one to one line. By three weeks the model starts to show very poor resolution in the forecast. So that's for precipitation exceeding 50th percentile. Now for 70th percentile it kind of looks like the same with some small difference, but the main point is that as the lead time increases, so the bias skew score can be large like 20 percent or so average over the grid points where there is skew. The model doesn't have a resolution, so some calibration has to be performed. So one of the conclusions from this is that is there skew in probability forecast of precipitation in the sub-seasonal range? So probably yes, but we need to do additional calibrations in the forecast. So the next question is how does the predictive skew of precipitation vary as a function of precipitation intensity? Since I computed this probabilistic forecast for nine categories, we have nine different percentiles, I can actually plot the bias skew score, sorry, I can plot this forecast validation matrix as a function of the percentiles of the precipitation. So here I'm looking at how the bias score varies as a function of the precipitation intensity or the precipitation percentiles. So here in the bottom is the percentile of precipitation and here's the briar score. And I separate the curves for one week lead time and then in this other plot it shows the bias skew score for two to six weeks. Now there are a few points here that I want to highlight. So first of all you see that I mentioned that you want the bias score as small as possible because it's kind of like an iron mass metric. So we can see that the bias score for one week is less than the bias score for two to six weeks. So this curve is below those curves. The other point is that if we see that it has this negative slope, so if we think about, for example, forecasts above the 50th percentile, like here in the middle, as we increase the percentile, the probabilities become smaller and smaller in the forecast and also in the outcome. So it's natural that it has to decrease because you're summing smaller and smaller numbers. So it has this negative, it has this kind of slope, the same for one week and two weeks. The other thing that is interesting is that if you look at two to six weeks, it's almost linear. The bias score varies as a function of precipitation percentile almost linearly. Now, for one week, it's also decreasing but it has this slight change for some percentiles, for 20 percentile. Now, when we compute the bias skew score, which again is given by this formula, so you're going to normalize or divide by the bias skew score, the bias score from the reference. The reference is the climatological probability. So it turns out that for one week, we have this kind of shape in the bias skew score, which is quite different than the bias skew score for two to six weeks, lead times. So for bias skew score, for one week lead time, it's going to maximize precipitation percentiles between 70 and 80 percent here and it goes to a minimum for low percentiles. Now, for two to six weeks, it has this kind of a parabola shape. So the bias skew score is smaller for precipitation percentiles for case of precipitation above the 50 percentile. Now, as we increase the threshold for the precipitation percentile, the bias skew score is actually higher. Now, these results, they were for the United States. Now, what I showed you is for the wintertime, for the wintertime and over the U.S., right? So these results may vary over a different climatic regime. So one thing that I did is compute the same thing for South America. So now, we are looking at, I'm going to show you results also from November to March, but it's the southern hemisphere in summertime. So we go from mid-latitudes, exotropics, mid-latitudes in the U.S., different precipitation regimes to South America, where it's highly convective in the summer monsoon. So here is the bias skew score for precipitation exceeding the 50th percentile, 70th percentile in this column, and 90th percentile. So for one week lead time, we see, in some places, we see high scores, good skew in the forecast for 50th percentile and 70th percentile. In some places here, especially in the eastern part of South America, in Brazil, Argentina, Uruguay, the score, the bias skew score, is between 30 and 40 percent better than climatology. And as the lead time increases to two to three and four weeks, it starts to decrease. Now, let me see. I also computed the reliability diagrams of South America for each grid point. And one thing that is very challenging is that the reliability in the forecast is quite variable in space. So you may pick one grid point and look at the reliability diagram, and it's quite reasonable. Then you look at the grid point, just the next grid point, and you have a very poor calibration in the forecast. So it indicates that it's quite challenging. We have a lot of work to explore these probabilistic forecasts in the sub-seasonal range. Now, just to compare the same results, here is the bias score. Again, the same plot. Here is the presentation percentile and the bias score for one week lead time, two to six weeks lead time. It behaves almost similarly. There are some differences. This slight tilt for one week lead time is not as apparent as over the US. So you get kind of a different shape. So some conclusions. Probabilistic forecast of precipitation over the US. So the ECMWF shows a high skew up to one week, which is kind of expected. The bright skew score can be up to 40% better than climatology. For precipitation exceeding the 50th percentile. But if we increase the percentiles, the bright skew score is much higher, actually. It shows a skew out to two to four weeks higher for heavier precipitation, but also it shows conditional biases and indicates that we have to do better calibration in forecast. There is a distinct behavior in forecast skew versus precipitation intensity. And as I showed, you have these different behavior in the bright skew score and the bright score. Now, just a few words about how I'm planning to use the open IFS. In another paper, we did some predictability studies using Worf. And one of the results from that study is that, as I mentioned before, the MJO is considered a very important potential source of predictability in the subsystem range. Now, one thing that I did in this study is to look at one case study when the MJO was occurring and look at how forecast errors grow within the MJO. And one of the conclusions is that the forecast errors on scales not directly associated with the MJO, they grow very, very fast and they can actually propagate to the exotropics and impact the forecast skew in the exotropics, for example, in precipitation forecast in the U.S. Now, Worf is a regional model and all the results that I showed here, for the ECMWF, they may be model-dependent, so we need to look at other models. So, in this predictability study of the MJO, I'm planning to use the open IFS to conduct more investigation of how forecast errors, when the MJO is occurring, how they grow in time. So that's what I'm planning to do with open IFS. So thanks for your attention. It's nice overview.