 Good morning everyone and welcome to the second day for to the workshop on S2S science which is being held as part of the 2021 ASP summer colloquium on science and applications. So today's opening speaker is Andrew Robertson who is a senior research scientist and heads the IRI climate group. He's also an adjunct professor in the department of earth environmental sciences and teaches as such. Most importantly he's a co-chair for the WMO S2S project and is also heavily involved with SUVEX which is the American extension of the S2S efforts using additional models. Andrew welcome and we're looking very forward to your talk. Thank you very much Judith for the kind invitation and for the nice introduction. So sharing my screen. Can you see that on full screen now? Yeah yes it's perfect. Okay great. So what I thought I would do is to talk about some of our real-time sub-seasonal forecasting that we've been doing at the IRI and some of my collaborators here are Jinghuan, Angel Munoz and Boha Singh and so the outline will be to show you some of these calibrated probabilistic sub-seasonal rainfall and temperature forecasts based on SUVEX models. So as Judith was saying we are involved in the SUVEX project too and the nice thing about those models for us at IRI is that they are available in real-time those forecasts. So I'll talk about that and so it has to do with calibration, multimodal ensembling and crucially probabilistic forecasts you know what can we get out of what can we do for sub-seasonal scales for those. And then secondly I'll show you some of our tools that we've been developing for looking at forecasts in terms of weather regimes and that'll be over the North Pacific and Western U.S. So to set the context I just put this one in I was looking it's actually up on my wall slide if it's this graphic it's one of IRI's first seasonal forecasts issued in maybe issued in December 1997 so it's for the 97-98 El Nino event. This is when the IRI forecast division was still at Scripps before it consolidated to Lamont and what you can see here is I mean already at that time the IRI knew that what it needed to do if you're making forecasts that are going to be useful to people is to make forecasts that are in a probability format. So you're giving some you're giving the potential user some estimate of the confidence of your forecast and so it was done in terms of terciles so you can see here in in eastern Australia here there was a 60 probability being forecast of being in the below normal tercile category 25 in the near normal and only 15 in the above normal. So the IRI has been doing this for a long time you know since since the late 90s and still doing this in fact I mean at the time that was that was all done you know forecasters sat around a bench as before my time and you know really drew those lines on on the maps but since became fully objective and automated and the question is well can you do something for sub-seasonal forecasts as well uh and actually I was under meeting I think it was in 2016 and I think it was Gilbert Brunet who said to me oh you know IRI you should do this for sub-seasonal forecasts you've been doing it for seasonal these years what about what about sub-seasonal so that so I thought yeah well good point you know and so we sort of embarked on on a project with no no funding to to to make that a reality and we since have a set of listen in what we call our map rooms so forecast map rooms and you'll be able to access this make a pdf copy available afterwards but you can you can find our map rooms and if you navigate to forecast and sub-x forecast you'll find all of these and so we have things like probably precipitation median probability or in terms of both weekly and bi-weekly averages going out to four weeks ahead and this based on sub-x models so for example and we have been doing this in real time and so you know that's the IRI thing it's going to make forecasts in real time and so we have been doing this since October 2017 every Friday issuing these for the upcoming week so the weeks then go from Saturday through through the following Friday and here's an example of the one from last Friday for weeks three and four precipitation so that's for the the period of 14th to the 27th of August and precipitation on the left and temperature on the right so what we chose to do was actually just formatted in exactly the same way as the seasonal forecasts were formatted so that that would be something that users are familiar with and maybe they can you know get a grasp on what these forecasts are saying and then we can see well you know is what's the information content of these forecasts is it similar to seasonal ones how to stack up against that and so for this latter part of August you can see that is it the forecast the dominant teresal category is is the above normal one we still have this kind of land linear issue well we have a kind of land nearest influence so there's a there's a 60 70 chance of being above normal category and a lot of regions you'll see it's just white so that's equi-probable forecasts these are like our calibrated multi-model I'll tell you what that is in a second but but this is so you'll see there's some regions where there's some color on the map where we where the forecast has some has some sharpness in the other case it's just flat looks like the paleontological distribution where it's where it's white and then likewise for temperature on the on the right and then you can see that maybe it's a bit more there's a bit more color over land we've shown it over over the ocean as well for precipitation so one more thing is that that's just in terms of the those teresal categories but one thing we've been pushing in terms of you know what can be useful to a user it may not be you know those those teresal categories there's no one size fits all so what we've been pushing is this this idea of a flexible format where the the user can pick their own quantile that they're interested in and see well what's the probability of exceeding a particular quantile and so i've shown here for the age of age of region what's the probability of exceeding the 80th 80th percentile of the the distribution and you can see here that so over southeast Asia there there's some pretty high probabilities exceeding that much more than the point two that would be the climatological distribution and then what you can do in this this map is really just click on any point on the map and i clicked on a point over Sulawesi indonesian island here and this is the this is what you can query the pdf at a point there and so the dotted line here uh well let's look at the bottom first there's the the climatological pdf for that point peaking at about 20 millimeters per week but then you can see here in the green line that's the forecast pdf and so it shifted shifted to the right a wet a wetter forecast so the the pdf has been maybe compressed slightly and shifted to the right and then the user could query this in terms of well what's the probability of exceeding 40 millimeters of rain per week and that's exceeded that's that's gone up from a probability of something like you know point point one up to a probability of point four so you know you could get some some guidance on on your likelihood of floods for example or you know droughts likewise if you looked at the other table of pdf so how was this made uh as i said it's made uh using using output from sub-x models and which we have archived in in the iri data library and in order to make this product what we did was we found three models that all issued forecasts on the same day so all issue on on wednesday when they wednesday starts the cfsv2 is issuing every day so that's that one's easy and then we've got the the noah gefs model and the ezreal fin model that we've taken as well and what we do is we do the calibration grid point by grid point uh on on a you know one degree grid of this gpcp uh precipitation grid and then for each point start and lead we fit a regression model for each model separately we calibrate each model separately uh we we make a forecast of the probability and then we we make a multi model average of those of those probabilities to get the forecast of precipitation uh at that point or likewise for temperature and then the way that we do this how do we do the what's the regression model it's a logistic regression model so in in order to to forecast the probability we use logistic regression and the predictor for this regression is actually the ensemble mean of the of the forecast model so we're not using the individual ensemble members we use the ensemble mean of of the forecast each model and we actually add another another predictor here a second predictor which is the quantile that we're predicting for and this is what this is the extended bit in the logistic regression and it allows us to make probability forecasts that are consistent for for different different thresholds so uh this is another snapshot of that map room and if you look uh what we have also is to look back if you look choose one that's an older forecast this for example this is from the 7th of august 2020 where we have observations for that for that two week period we can look back and see well what was the observed percentile so plotted on this map for the 22nd august to the 4th of september 2020 that's the observed percentile against the the the hind cast period of 1999 to 2014 and so i'll draw your attention to a region where the forecast worked well which was say over pakistan here if we look in the pakistan here we can see that the the it was heavy rainfall at that time was around about point that the 90th percentile uh if you compare that that bi-weekly average with the ones from uh each year from 1999 to 2014 and we had indeed forecast uh a high probability in the above normal uh category so that's one one example where the forecast worked relatively well but you can find others where it didn't work as well so for example over indonesia in this case i guess we were just around uh the uh upper upper the 66 percentile but uh plenty of places where you can see that well we did quite well over china but uh other places maybe not so good so you can query this map room you can go back to 2017 you know and i'm looking if you haven't if you have an event to look how well this particular mme camped with the forecast so that's one way to verify another of course is to use a skill score and so we have map rooms for the the the rank probability skill score and so you can you can look look at that by month and by lead time so this we have weeks two and three and three and four on the right here so these are for august starts and you can see that we do actually very well we have very high skill scores here getting up to point two i mean that's actually quite good for a rpss score so this is a region where two two particular drivers come together s2s drivers of skill and so an mjo so they're both contributing here to the skill over over the maritime continent it's the it's the dry season here but you still have good skill over india weeks two and three over northern india you see there's some skill in in these forecasts but it's mostly gone by weeks three and four i just showed here an example of a particular event a heavy rainfall event over the state of bihar in northern india in 2015 this if you if you plot an anomaly for that that weekly period of the the 6 to 12 july we we had we had positive rainfall anomalies over in this kind of diagonal band with dry anomalies to the south and if we look back at the forecast made what we call the week one from the beginning of that week or if you go back to week four you can see that the models do capture this kind of interseesal oscillation pattern in that in that case it was really sort of a random case that i selected but i guess we got got lucky with that one and you could sort of track that back to a very high intensity mjo event over in phase seven during that period over the western pacific and if you look at a phase composite over over phase seven you can see this this dipole of the iso over over northern india that led to this over india led to this ability to make a forecast at this time so i mean these are really you know forecast forecast of opportunity and i think the point to be made is if you use a if you use this kind of calibration to make forecasts if you calibrate your forecast then in times where there's where where there's no signal in the forecast or there's no skill you will just get the climatological pdf you should just get the climatological pdf coming out of it the probabilities of exceedance will just be the climatological ones whereas if there is you know some active episode only then will that climate that forecast pdf deviate from the climatological one so that that's the goal of of of calibration to enable such forecasts of opportunity to be done really automatically and i'm just showing here that we have these also for you know one week two and uh there's much more skill obviously week one but you have to make the point that there can be useful skill say at week two which might not be the or even week one for that matter that might not be in you know what we think of as the s2s forecast range but these are formatted in a different way from what we would think of a weather forecast where you're looking at the forecast each day this is using weekly averages which can be you know useful to certain decision makers to look at well in week one in terms of the coming week what's the probability of being above below normal etc or week two how does this compare with the that skill that i'm showing you how does it compare with iris skill for seasonal forecasts what i'm showing here is for the august to december kind of period second half of the calendar year on the right here this is this is iris seasonal forecast skill again using this rpss on the left what we're getting it at week three and four lead time so you can see fairly fairly comparable i guess you'd say to what we're getting in our seasonal forecast so i guess this is good news that you know this is just an experimental you know s2s is a is a new kid on the block and already you could say you know it has similar skill to what we're getting in seasonal forecast after all these years of doing that uh if you look over india well maybe we're doing even better here the the seasonal ones at the bottom here but basically no skill in in this period was we can see that there's some skill in the week three to four range so i can see i'm already kind of running behind in my my presentation trying to speed up a little bit this is the other thing that i wanted to show you uh using these daily circulation pattern weather regimes uh as a kind of forecast guidance tool so what we did was to define these these uh these weather regimes from from for march to march to sorry from october to march using uh z 500 uh over this this region north pacific north america and we we use four regimes here from reanalysis data a west coast ridge there's a there's a negative nao pattern in the atlantic uh we've got a we've got a pacific trough here on the bottom left on the bottom right pacific ridge and uh they have some you know some surface weather expressions associated with them but the idea here was to say well can we project our forecast onto these four circulation patterns and get a a low order depiction of our forecast and uh what we did was we used the cfsb2 model because we have a forecast every day from that we use a lagged ensemble uh average for that and what we have done is every day to take the forecast in real time from cfsb2 and project it onto those four patterns from reanalysis data and and show uh the the closest one so uh what you can see here is we do this every day and this is a forecast going out to 45 days and it was closest to west coast ridge in the first few days then transitioned to pacific ridge for a few days uh going into pacific trough etc and then the saturation here is uh the the probability really of being of that assignment to a particular observed regime so we're not looking at the regimes in the model itself what we're doing is we're projecting the the model's own forecast of your potential height onto those patterns to get a lower order view so we did that for the whole season and this is what comes out updating it every day and then if we look back we can see uh if you just look up uh basically along the bottom this is the regime uh the the regime sequence uh that that was observed in reanalysis data and so you can see we we have some sort of interseasonal uh transitioned between various regimes here uh in the in the dark blue in the pacific ridge regime and if you look up you can see uh how advanced how far in advance uh this was captured in the forecasts and we can see that you know these things the the bands sort of extend around vertically around 10 to 15 days so the skill uh up to about that far in advance if you go further in advance it loses its skill uh we would like to see uh you know vertical bands uh going further up and so there were no sort of strong episodes where that was the case in this winter although you can find it in other winters another thing we did was well cfsv2 actually goes out you know get to seven months so why why how about if we uh just show uh not only the the first 45 days but also show uh monthly averages out to week six so this is like a seamless depiction where we're showing averages uh two to three days four to seven week two weeks three and four and you can see that what the forecast was showing in the in the in the seasonal range was really going into these uh ridge regimes but it couldn't make up its mind between west coast ridge and and and pacific ridge so west coast ridge the the red and the blue but it was definitely moving into you know because of um but it definitely showing this this this this this pattern if we look to see the number of days that were were observed versus versus forecasted then we can see that in in a month three we got a bit we got too much west coast ridge and about the rise a man of pacific ridge so the the forecast was fairly accurate in in that respect but it couldn't tell uh which one exactly so uh just to summarize uh we've been making these routine calibrated probabilistic sub seasonal rainfall and temperature forecasts uh every week but based on subjects in real time uh i've shown you those map rooms and then there's weather regime tools as a forecast guidance so forecaster can really look at that and see well how well how well does the model do over the season uh in in real time how well has it been doing up to now uh what's it forecasting in this very low order sense in terms of uh low order circulation patterns illustrates really the episodic behavior of the S2S scales uh we generally see we lose skill after about two weeks uh but some of the skill then comes back in the seasonal range so thank you very much i'll be happy to try to answer any questions that you might have thank you very much andrew so if you have a question please raise your hand or post it in the chat i have a question um it is i really enjoyed your analysis about the regimes and the weather types and um how the predictability of the regimes um is how their systematic model bias is preferring some regimes over others and i'm wondering um where would you start um in model development to address some of these issues you're diagnosing in model development i couldn't possibly tell you i mean that's that's up to you model developers to tell me that i mean it's uh we can i guess we can you know we can give some uh depiction of what the what the models do and if when you you know have a a land in your event uh they they move you know we can see that they uh very much forecasting these these ridges but they may not be uh may not be in the exactly the right place but if you look in the seasonal forecast uh for the for the western us we did quite do quite a good job you know the model did a good job of capturing that uh on the seasonal range on the sub seasonal range here uh not not quite not quite so clear so i think it's a it's a really it's a hard question to answer uh you know what model developers should look for but maybe you know some of the things are to look at well what this can can tell you uh you know what the it can highlight particular drivers you know it can highlight you can uh you know relate this this behavior to to end so uh some of the we've looked back in in previous uh winters looking at this and you can relate some of the episodes to the mjo as well so you know if if something was captured um maybe you can relate it or not captured you can relate it maybe to whether or not the the model captured the mjo and whether or not it captured its teleconnections and as that gets to some along the lines of what you're asking thanks so much uh will has a question well why don't you mute yourself hi hi andrew thanks for the great talk um i was noticing in your extended logistic regression you're only utilizing the ensemble mean information and nothing with the dynamic ensemble spread i was wondering if you could talk about what motivated that choice and if there isn't actually any skill in the ensemble spread that would assist in the calibration yeah so that's a good question thanks uh in in our work on the seasonal forecasts we've never found any any information in the ensemble spread i think you know the spread skill relationships are you know they they they have value at the weather scale or the seasonal scale there's not really any evidence that there's any any value in that information maybe in the sub seasonal range i think it's still an open question as to whether there's some spread skill relationship that you could use you know if you have a tighter pdf that you you could really you know make a more confident forecast in that case so i i think they call it what heteroscedastic if you add here uh you know you can add another term for the the the spread of the of the ensemble that could be included in here and i think some of my our i colleagues have actually tried that we haven't managed to get any more skill coming out of out of doing that uh that's far thank you any more questions if not yeah i have one i guess i'm not coming out of the chat that's okay you don't need to put it in the chat you can just unmute yourself yeah okay yeah and i had sort of sort of a practical question actually i'd be wondering about a bit i was curious what your take was do you think that users are on balance obviously they're individual users but on balance are they better served with these routine probabilistic forecasts or maybe more of a sporadic approach where it's sort of a null forecast until there's a quasi deterministic forecast of opportunity so the point i would make is that if these forecasts are are well calibrated then you should have a null forecast maybe most of the time so then you will just be issuing and then you know the user goes to this site and they will see what's the probability of exceeding the the 80th percentile it'll just be given you know by the the climatological probability of exceeding the 80th percentile it'll only be you know when something's really going on that you will that this green curve will deviate from the from the from the black one so you know the calibration needs to be done properly obviously for that for that to be the case but i think that you know the rpss i would say that our forecasts are generally in that in that ballpark i'm not sure with you know if if that's if that's probably but i think that gets to what you're sort of asking because it's always it's always issued but it only deviates when something's going on when we have information to to uh to the models have some information to give matt fosdada are you happy uh well this is actually very like i said this is it's almost a philosophical question it's probably better over beer if we ever meet sometime again in the world yeah yeah that's a good answer here yeah i really like this idea of issuing a climatological forecast unless something is going on i think that's a really nice way of dealing with this windows of opportunity that only occurs sometimes i also forgot to say um um andrew is also leading one of the tutorials the pie weather regime and pie cpt tutorial together with uh angel monos angel is um uh uh on pto this week but um they really worked a lot with the students and made everything available so thank you so much if i could just add to that you know it's been great to see what the students have done and they have actually been looking at these weather regime patterns during the summer season as well as the other seasons and i think that's something i only show them for the winter but you know it's a question is such an approach also be useful in in the other seasons so maybe we'll hear some of that tomorrow afternoon or morning yep exactly thanks okay thank you again