 David Lundweiler, and the title there is a very long sentence because it's on the prospects and challenges of weather and climate modeling at convection resolving resolution To my opinion, it's trying to make complex thoughts and views a Little bit more easier to understand Please give David a warm welcoming applause and let's kick off Welcome also from my side Well, it's really terrifying to stand here. So please excuse me. I'm extremely nervous, but you've probably heard this a few times now Before I begin I would like to thank a lot of people. So first of all all the one volunteers that make this conference possible It's been amazing so far. I've seen so many interesting talks. I've had so many interesting discussions and Second of all, I would like to thank my my co-authors for this talk They're my team I work with them on a daily basis, but for this talk in particular I would like to thank Christoph share Oliver for and hones forked who Contributed some slide which allows me allowed me to stay a few days In the last week with my family instead of preparing slides. So who am I? My name is David I'm an atmospheric scientist by training I currently work at ETH in Zurich and they're in a group which is interested in In the summer climate over Europe and I work on the interface between atmospheric science of climate modeling computational science and computer science so This is basically the summary slide from the first half of the last talk So the climate system is warming that is unequivocal and the human influence on the climate system is clear So now that we've established that the climate is warming in the last talk We it's time to go a bit beyond and and look what's coming in the future So we know the climate system is warming and we're very interested in how a future world might look like But then at the same time there is this this famous saying Making prediction is hard especially about the future and One way to do that is by using numerical models so trying to put the earth system into equation and Use climate models to make projections But we we make our models to answer very specific questions or very specific test very specific hypothesis and we don't Want we don't want to create The earth in a virtual world. So this is kind of important to remember So all the models are wrong, but some models are useful, especially if you have the right questions So this is another summary slide from from the last talks is this very famous IPCC slide From the summary for policy makers and essentially tells us that we have a choice Depending on the amount of carbon emissions We can either live in a world which looks which has a temperature change like the plot on the left Or we can keep a business as usual scenario and we can live in the world That looks a bit more like the world on the right So this is a plot of surface temperature change and it contains a lot of the very useful Robust features that have been addressed and Investigated in the past for instance, we see that the continents are warming faster than the ocean We can see that the poles are warming faster than the tropics and everywhere. We have this these strange Blast off black dots is Where all the models that contributed to this plot agree in sign and where the signal to noise is very high So another way to look at it is By looking at the Global mean temperature change. So on on the bottom panel You see past observations and in the big panel we again look at the scenarios So we have a business as usual scenario on the top and we have a mitigation scenario on the on the bottom This would be roughly the two-degree world. I've showed you before and In the straight red line, you can see the mean Change predicted by a large number of models. I think it's around 40 and The color shading indicates the range these models span. So this is This is kind of the uncertainty range that our models project and a lot of effort has been made in the past to understand this range to try to see where this range comes from and Of course, we would like we would also like to to reduce that uncertainty range So we know a bit better where we're heading and this in this talk at the I Want to look at some ideas how we can We think we can reduce this range by improving our climate models because we think that Someone a large part of these uncertainties comes due is due to uncertainties in the response of clouds And if we improve the Representation of clouds we might be able to reduce these uncertainties So kind of an outline an outline. I first would like to talk about clouds and climate sensitivity Then I have to give you a very brief introduction into weather and climate models how they work in a nutshell Then I would like to talk a bit about new climate models, which are able to mitigate to Not to mitigate to Work around the some of the uncertainties that are in the current generation model of models Then I would like to talk a bit about challenges in Computing weather and climate so computing these new models and finally I want to give a short Outlook on what we can expect in the last in the next five years Or the next couple of years. So let's begin with with the clouds You remember that I told you that a lot of the uncertainties comes from the representation of clouds and clouds can react to warming in the large Various ways and here I've selected a few So the clouds can either contribute to the warming by that would be a positive feedback So as the temperature gets warmer their contribution makes the climate system even the warmer But they can also have an opposite effect and they can meet that they can moderate the warming That doesn't mean at all that the climate will get cooler, but that the increase will be a little bit smaller so For instance in the future world We could have more high clouds and that would then contribute to a Positive feedback we could have more we could have more low clouds which would have the opposite effect we could If the clouds could get higher which would again have a positive feedback They could get less icy and more watery which would have a negative in feedback or they could Change their position. For instance, if they move closer towards the poles, this could again have a positive Feedback so you can see that we have a whole zoo of ways the clouds can react and here I've chosen just a few so there are more of them and Our problem in the current generation of climate models is that clouds are not explicitly resolved, but we parametrized them So it means we use Physically-based semi empirical models to represent them in models In climate models and the way to illustrate this this so we have We have a model world and we can change that world and here and one way You can change it is you can try to simplify it to make it easier to understand and here what the authors have done They've stripped the earth of all the continents so now you have only a nurse consisting of water and the notion But nevertheless these planets that features that are very similar to our climate system so we can see We get there is a an intertropical conversion an intertropical conversion sound there are Hadley cells There are extra tropical cyclones, but in essence that the system you have to investigate is much easier to understand and now you have this this idealized world and you can run it for a couple of years and Then you can warn the entire system by Four degrees and it will be up here and you can run it again for a couple of years And then you can take the difference between these two simulations And this is what is plotted here and is plotted for for the four different models And you can see that these mod these four different models here in the in the vertical they React very differently so the change in cloud-reactive effects the patterns look different and the change in precipitation also looks different so and We we are our hypothesis is that these differences are Mostly because clouds are not represented explicitly and that's the parameterizations involved here Have a substantial amount of uncertainty So to to to explain quickly what parameterization is I want to give a very short introduction into what climate models are So what does a climate model do? It has two important jobs The first job is it distributes heat horizontally so we have differential heating around The the the tropics that there's this warmer than at the poles and the first job is to kind of mix these two states and The way it does this is the following so this is a high-resolution simulation at 13 kilometers which was Denimation is by by Oliver stepp or other and you can this is basically now how a Climate model looks like from the top. So this is not the satellite picture This is entirely virtual world and you can see that we have many of the features So here for instance the tropic we have that the strong convection That the funders the deep thunderstorms going on here in the extra tropics We have that extra tropical cyclone so that the low pressure system coming towards towards Europe you many of You know these very well at some points. We will see tropical storms. So Here would be a tropical storm and it is a lot kind of these these eddies then mix The the the gradient between the tropics and the poles. So this is one job The other job is to mix in the vertical, you notice a similar plot from the last talk So in a greenhouse atmosphere the atmosphere warms your surface and Somehow a lot of the energy from the surface has to get back in the atmosphere and this usually this work goes through the Water phase so through clouds But still these these clouds are very badly represented. So we have one He's trying to heat energy in mass transport which has been represented very well since the 70s So these are my knob and it's Makarinsky and these people they've done an awesome job But on on the vertical transport. We're still a bit unsure so Some equations what the climate model does it basically solves a number of Couple differential equations, so it doesn't matter too much what this is but here on top you can for instance see the Navy Stokes Equations some might recognize this below here. So it's the equation of state or the ideal gas law And this is from the atmospheric part of a climate model. This is the climate model Cosmo Which I'm using it is amongst other things also used by the German weather service to make a do weather prediction And now Unfortunately, so we want to solve these these equations so we get the nice motions you saw in the animation, right? And sadly we don't know an analytic solution to these equations So our only way is to solve them numerically using numerical methods on a grid And this is very very expensive So we have a numerical mesh surrounding the earth in in three dimensions and we solve the set of five equations at each grid point and Then once we have the we have Resolved the equations everywhere. We have a new state and from then from that we can then progress in time So we do kind of the classical time-stepping approach So now depending of the on the computational power you have available You can vary the number of grid points you throw at the problem or you can vary the resolution of the grid And now we will have processes will which will be very well represented on that grid And you have processes which are either not representable or which have a length scale That is too small to be represented on that grid so typical global climate simulations today have a Grid spacing between a hundred and twenty five kilometers So imagine you have these two grids by grid points here, and you've a process that falls right in between That's then hard to capture by the grid So we have resolved processes and we have so-called sub grid scale processes and from there we have a lot So we have for instance chemistry Cloud micro physics radiation Sensible and latent heat fluxes which you are very hard to represent them in in on a grid And then we have some processes which we add as as a new model Module so we have for instance surface model and ocean model which also has similar Kind of sets of equations and then we have a lot of processes that have length scales Which are too small to be represented on the grid which is for instance here the deep convection at the shallow convection So the the big thunderstorms in the tropics or even in the extra tropics and next I would like a bit to look at At that in a bit more detail So here we have a very very wonderful Photograph of a thunderstorm Over Lake Constance. This was in April and 2006 and it's a very beautiful image because we can see a lot of the processes that are going on so for instance oops Here we have the up and down drafts region. So where the the air Purcells rise very fast with up to 10 meters per second over here We have what we what we know as the anvil region. So there's a lot of ice In there and here directly below. You see something very interesting This is namely kind of the hydrometer was falling out of the cloud and informing these these gust fronts near the ground And the fact why we actually can see them in this image is because first for this case here They consist mostly of ice particles normally this rain and the ice particles makes us Makes them a captureable by a camera So now imagine we want to we want to resolve this in a model So we overlay a grid and the grid now has a grid spacing of 10 kilometers so we have information at these grid points, right and At these grid points and now what the physical parametrization tries to do it tries to capture All the processes that are going here on here in between And this is an extremely difficult task and it has been worked on since the 70s and Some of the most brilliant scientists I know have worked on this problem And we still even today have to have the uncertainties that they've shown you So we're proposing to take another route Maybe a simple route We propose to simply increase the resolution of the grid Because now we have a few points Representing the cloud and we actually can start to resolve the motions Associated with these processes we can go even a bit further and then very well capture many of the features of this cloud so We've done this We as I said we use the cosmo weather and climate model and we've done this for a simulation over Europe. So we have done three Simulations one at 50 kilometer one at 12 kilometer and one at 2 kilometer and We were able to do this 2 kilometer simulation because we use a very special version of cosmo Which is a version which is able to make use of the capability of GPU accelerators this was an effort led by by Oliver Fuhrer and We used then this model to run a simulation which has about 1500 times 1500 times 60 grid points and we Simulated a period of about 10 years on 144 notes we can run one year of simulation in about two and a half Days of feudal time because of this GPU prototype so I can now show you a lot of statistics But this is boring, so I'll just start I show you a few cases from this So some of you might remember this storm. I was Kirill And it affected it was a very strong windstorm that affected Germany in January 2007 and in this representation here The white shading indicates clouds the colored shading indicates Precipitation with the blue colors having very light rather light precipitation and the red colors having a bit Stronger precipitation and just to orient yourself that the wind basically blows along these these white lines And now at this point in time The storm is located here over northern Germany and it exposes all the typical features of extra tropical Storms namely here is a very nice a warm front with precipitation along the front and here We can see a lot of cold front with a bit of precipitation along the cold front, so we have usually Precipitation up to five millimeters per hour Now we can increase the resolution of this model what you can see is first of all we get a lot more detail But then we get also a bit more precipitation here along the cold front But kind of qualitatively the two simulations still look a bit the same now We're trying to disable this convection this parameterization of deep convection and increase the The grid spacing of the model to about two kilometers and what you can see is that the picture Changes quite dramatically So we have physical features going on which we couldn't identify before for instance here You can see these very narrow bands of rather intense Precipitation so these are cold frontal rain bands and if you for instance look closely it's it's really fascinating so the wind blows here along these white lines and The rain bands are actually not perpendicular to them, but they're they have an are oriented in an oblique angle and You can also see that they're a bit broken up And if you compare this for instance to this radar image of Which was captured from an extra tropical cyclone over the Pacific you can see that now actually These models start to look like the observations So if you compare again these two this looks like a lot more like the observations Okay Switching seasons going to the summer now This the same kind of plot and we're looking now at the day in July 2006 so we have thunderstorms and rain showers in summer and On the left side you can again see the 12 kilometer simulation and on the right side You can see the two kilometer simulation where we treat the thunderstorms explicitly by the equations of motion and on the left side all the the rain that you're see comes out of these parametrizations So if you for instance now focus a bit on the left side and you focus on the precipitation You can see that it covers a very wide areas and that we have mostly greenish and bluish colors appearing And if we watch closely You can see that it peaks around noon so now around 12 o'clock Now let's switch to the right side Of course the animation has now ended We can actually now identify The individual bubbles so we can see the individual thunderstorms in the simulation because they're treated explicitly and some of the thunderstorms They have very small areas of very intense precipitation and this is a Very different physical behavior of these simulations and if you look when the rain peaks You can see that it peaks right about now So in the afternoon and if you think back when do you experience? Do you experience thunderstorms in summer? It's usually in the late afternoon. It's usually in the beginning of the night so now the timing of these of of of the Precipitation in these simulations agrees much better So as a summary For for the first part we want to switch off the parametrization of deep convection Because then we can formulate our models much closer to physical first principles This yields an improved representation of summer convection and precipitation extremes This has been very well investigated so here you can see a couple of papers on that But the downside is these models are computationally extremely expensive In fact, if you want to double the resolution of a climate model you need about a factor 8 in computational effort So to go from the 10 kilometer range to the 10 kilometer range We need to somehow find a factor 10,000 in computational effort and I'd like to To show a bit what we've tried in this direction so Computing climate models is extremely expensive and what the way we do it is we use what we call super computer So what makes the super computer super is the network? So we have a bunch of nodes We have a bunch of sockets that are occupied either by CPUs or GPUs or whatever But the important part is that their network together with a very strong and fast network and then we can Slice our problem into smaller chunks and distribute them to the individual What we call them nodes so so yeah, CPUs or GPUs And this is the the computer. I'm using it's called pit state It's located in Lugano at the supercomputing center there and it consists of about 5,300 Hybrid nodes so each node contains an Intel has full multi-core CPU and an NVIDIA Tesla GPU and This it is one of the larger machines of this world so some might know the top 500 lists or currently it's number three and What's more important does that it is also one of the greenest Computers in the world. So if you measure for instance co2 footprint So it is a user it is a user machine. So you to gain access you basically write the proposal And which then gets sent into review and if the judge is like you you get some compute time on it But the important part is the thing is that 90% of the compute power of this machine comes from hybrid nodes So they contain CPUs and GPUs and we think that the GPUs are chips Which are very suited to compute weather and climate models And this is because Graphics cards so graphics processing units or GPU accelerators, whatever you like to call them They're substantially parallel and they're throughput oriented But more importantly they provide a much higher memory bandwidth and they're I and they're able to hide Some of the memory access latency between a lot of threats And this is important for weather and climate models because in fact many of the operations in these models are Limited by memory bandwidth and not by the available flops. So not by the available floating point operations Just to quickly explain what I mean by memory bandwidth. We look at one of these operations We do in the most expensive parts of Of the climate models on the dynamic Dynamical core and in there many operations are characterized by the fact that they need information from their neighbors So to update the grid point here the orange grid point we need Data from the left and the neighbor on the left and the neighbor on the right for for this easy case here And if we write it down you can see here that we have a of i plus one and then a of i and then a of i minus one And so we have to transfer a lot of data to update the single grid points And the way to express it is the arithmetic intensity So it says how many flops we can do or how many data How much data we need to do a flop and the easiest way to assess how much is by simply counting So you cannot count all the floating point operations here one two three four five So we have five flops and then we count all the data accesses which is one two three four five six seven Okay, so we have seven times eight bytes in single precision so we have about We do about zero point one flops per transferred byte Now we want to know whether we're memory or flops limited and for that we use a very simple analytical model Which was established by Williams at all from from Berkeley. It's very insightful because it essentially says that Detainable double-point floating Point performance can be separated into two regimes So one is limited by flops and the other one is limited by memory bandwidth So if our automatic intensity is above one if we do more flops than per byte We're in this regimes here, and then we are limited by the mat by the peak and Floating point performance or if we do if the automatic intensity is low We're here in the memory bandwidth bound regime and now we can go back and look at our 0.1 flops per byte and we find that we are here on the left side of the curve And this is typical for for the stencil operations, which we find in the dynamical course of weather and climate models Now if we increase the the memory bandwidth available, of course Our time to solution will go down and this is why we think GPUs are good for our models So we ported our entire code The entire thing is very well documented here in this paper By Oli and so Cosmo is about 300,000 lines of Fortran code The dynamics so that the part treating the equations that have shown you a bereaved in C++ and we use At a domain specific language that some of us wrote called Stella Stella is nice because it abstracts the hardware architectures from the operations, so we have backends now for multicore and GPUs are written in CUDA and It allows us very low-level aggressive performance optimizations while so for instance changing the loop order while still maintaining a higher syntax The same syntax at the higher level of the code The physics were ported using compiler directive So was personally more involved in this part But the essential thing is that after about 10 person years of work We had the entire time stepping algorithm ported and validated And this now avoids expensive data movements between CPU and GPU each time step So what did we gain from using this new architecture and one way to look at it is through a strong scaling experiment, so we take Given experiment a given problem, which is in our case just a given domain size and then we throw an Increasing amount of resources at the problem and what we expect is the time-to-solution to go down Proportionally with the number of nodes we throw at the problem. So this is what we show here So on the y-axis you have time-per-time step, which is a measure for time-to-solution and on the x-axis you have when you go to the Your right we decrease the amount of Amount of nodes or we increase the amount of grid points we compute per node So again here you can see two regimes here on on your right. You can see the linear scaling regime. So here the time-to-solution decreases about proportionally with With the resources you throw at the problem But then down here we see a saturation regime and this can be mostly understood I think by by little slaw for those who are interested So now what do we gain? So here at the point where where we think we're optimal We gain about the factor six in time-to-solution or what's even more important for us We gained about a factor eight in resources so now we have a very nice and efficient model and The question is now How far can we push this? I've shown you regional simulations over Europe, but actually The problem of climate sensitivity, so the uncertainty range I showed you at the beginning is a global phenomenon and we wanted to know how far we can get So we now do the other part We do so-called weak scaling of the problem and by weak scaling We mean that we vary the number of nodes No, let me begin again. We vary the size of our computational domain and we vary the Nodes we throw at the problem proportionally So what we expect is the time-to-solution to be constant and this is not a given but in essence it means that for for this experiment here where you where we simulated Over the alpine arc we use about ten nodes for a European domain I've showed you before we use about the hundred nodes and the question now is will this entire model scale to the planet? So that would be something around 5000 nodes And now the question we want to answer is first does it scale second? What's the time-to-solution? So are we there and we want to establish a baseline at one kilometer and finally? We want to assess the efficiency of the Cosmo model so we did this entire exercise and We find now we again here have the time per time step on the y-axis Which is a measure for the time-to-solution and here? We have the number of nodes to the entire size of the machine Which is about 5000 nodes and we can see that the time-to-solution actually Stays constant and that's what we expected due to this to the numerics that are in the dynamical core so okay, we now have a Model that can scale at very high resolutions to to global domains and we can do some experiments with it For instance, we can think we can test whether we are fast enough to do global simulations So to this end We did an experiment of the idealized baroclinic wave. It's a very famous test case for for climate models it's by Yablonov skin Williamson from 2006 and We did it we there we Simulate the evolution of a baroclinic wave and the baroclinic wave And essentially you can think of it as the extra tropical cyclones have showed you before and what what you know kind of us autumn storms And because we have a regional model we now need to apply some tricks For instance, we glued in the West East direction. We made the domain periodic. So we glued them together and We have a small problem at the poles Namely because we have a regional model here You can see that the distance in the grid gets very very small So we get at pole ultimately we get the singularity and we have to address this So what we do is we simply cut the domain at 80 degrees south and north Because here we do a computational experiment But we nevertheless we cover about 98 point four percent of your surface. So this is simply the sign of Zero of 80 and We then use the setup to do a simulation which is about 10 days long and here we come to the Solution of of after 10 days. So here you can see this barrel the Baroclinic wave after 10 days so we can see three Low pressure system following each other and then in one of the low pressure systems We can see here this cold front or here the cold front which gets wrapped around the core of the low pressure system And if you look closely, we can see these very funny things here These are small eddies, which are embedded into the mean flow. So here we have a zoom of Of the small eddy And this is actually a new feature So this case has been run over and over by hundreds of group non hundreds but by a lot of groups But only at at script basings of 20 20 or so kilometers now that we can go to one kilometer We actually find new phenomena in very established and old cases Okay, now coming to the time to solution. So when we do Climate simulations, you can do a lot of experiments But two very famous ones or two that you do a lot is the first one is you do a 30-year-long time slice experiment So you can for instance Simulate the last 30 years until today and then you can simulate 30 years at the end of the century. So for instance 2070 until 2100 and then compare the two so this is a very famous type of experiment The other one would be kind of century scale simulations where you simulate hundreds of years and of course To do century scale simulations you would then need to be about 10 times faster So to do these 30-year-long simulations, you need to be able to to simulate the few months per day and If you now go to our benchmarks, we can see that here the 0.23 SYPD, so this is simulated years per day So these are a few months and we can reach that today at the kilo at the grid spacing of about two kilometers When we scale to the to the to the full machine At one kilometer. We need about another factor 5 to 7 to get to this this window and and of course to these century scale simulations We need another factor 10 to do century scale simulations at One kilometer so we have a big challenge to go and of course These simulations come at the cost and we measured The power they required so the energy in megawatts per simulation year and then extrapolate it from there to the 30-year-long simulation and you can see the for instance that one kilometer we use about 18 gigawatt hours and You can then using the carbon intensity of the Swiss energy mix you can then Convert this into a co2 to solution which would here be for a one kilometer simulation would be about 2200 tons of co2 equivalent just to compare this annual see full annual Footprint of a person is between zero point one and ten Tons of co2 depending where you live on the earth and of course on your lifestyle So we have a big number here, but that we have to bring down We have a lot a lot of ways how we try to do that so we can look at the energy mix We can look at the arithmetic or the algorithms in the model We can look at the math. We can look at compilers. We can look at the computer architectures We can look at the chip architectures. So there are a lot of questions Still open and a lot of potential to address these problems But nevertheless we can all also say that we can do global simulations or with with two kilometer resolution So the nice animations I showed you they also apply But we wanted first we wanted to establish a baseline and we wanted to know how efficient we are on today's computers and to this end we Proposed to use a new metric which is called the memory usage efficiency and it draws from two main ideas The first idea is that if you're limited by memory bandwidth Of course you have to get as close as possible to that upper bound that I showed you in there from from the roofline model So the first term in this in this metric is the measured bandwidth So what you measure during execution of your model and then you did in the nominator We have the achievable bandwidth and we can get that through a set of highly optimized micro benchmarks Then the other point is that the other idea is that you've maybe heard of the expression floating points are free And this comes from the fact that floating point operations on today's Architectures, they're actually much cheaper than data movements So they're actually about a thousand times cheaper in terms of time No, a hundred times in terms of time and about a thousand times in terms of energy So that's been measured a few years ago by Petron Sholf So that the second idea is that in order to reduce the energy so this factor thousand here We have to reduce the overall amount of data transfers and we can Simply count the number of data transfers we do in our model, but then comes the hard part We have to know the lower bound so how many Transfers are actually needed and to this end Torsten Carlos and Greg developed a poor performance model It's a bit hard to explain In two minutes, but in essence what it does is you have an operation as the one I showed you before And then you look at what data do you require for that operation and what do you write down? And then you map that these data requirements you map to your to your to your architecture so for instance to do the different cache levels and And now then you can compare it to what you what you actually do and at the important part is This performance model doesn't only look at one individual operation at a time But the important is if you do an operation it will have predecessors and we will have followers So they can share data and they can share data and their performance model looks at the kind of at the algorithm Or the implementation of the algorithm how it is written out and then gives us a lower bound of the required data transfers So we obtained that for our model the number itself is a bit a non-telling non-informative, but we can see that We perform quite well So for the first term so it means that the our implementation fits the GPU architecture Quite well, but we can do maybe do a bit better with regard to the memory bandwidth And now we hope that other groups will take up this measure and compare so we can learn from each other All right, that was mostly it Here are my conclusions. I hope that I was able to Show to you that we have some very interesting cool fun physics to to address And that we have some very interesting computational science problems to solve if we want to attack these big questions such as The climate sensitivity and there is a lot of potential for improvement And there's a lot of interesting science to be done And I hope maybe some of you would be interested and talk to me or Tell me that everything I've done is wrong and that you could do it all of it much more easily And that would be amazing. So thanks a lot for your attention Fantastic you on how you do your work. I can understand some people are quite jealous about the computer power is using isn't it Some questions please maybe one moment. Sorry if you want to post question in German That's totally okay. Well, then try and translate them into English so everybody can understand So if you want to ask in German go ahead Okay Hello, yeah, thank you for the excellent talk Did you ever try to dynamically adapt the grid size like it is done in mechanical engineering to Make it more precise or is that not possible in that case? We ourselves didn't do it. We had another Task, but it is a very active research topic in my community. So much less in climate, but a bit more in Weather prediction because there You have to be much more even more efficient than we are in climate and what you can for instance do is Let's say you're interested in in tropical cyclones and in hurricanes and whenever you detect a cyclone Hurricane you can dynamically adjust your Your grid For climate. This is a bit less interesting in my personal opinion because The energy balance of the oceans is mostly governed by these low level low level very shallow clouds and To resolve those clouds, you need the grid spacing around one kilometer and below So if you want to cover these clouds, you need an extremely high resolution everywhere anyway So maybe we should focus on that first before we go into these these refinements Thank you. Can I have a question from the web? There are no questions. Okay. Lots of others here number two, please would Observing the clouds help with Describing them or simulating that I'm flying there with aircrafts or UAVs or something like that Yeah, we do that a lot not not me personally There are a lot of measuring Campaigns the people flying through clouds with aircrafts people trying to model each of the processes So how cloud drops form how they grow how they collect each other and there are Since actually decades we do with the observation campaigns So one of the most interesting months that will happen. It's maybe right now or in a few months It's the narvel to campaign Is involved is also that the Max Planck Institute in Hamburg and they will do some very interesting coupling They will do some very interesting new experiments, but at the same time They will also apply these high resolution one kilometer or below Models at the same time and try to to leverage the knowledge gained from both of these fields So, yeah, we do it and of course we also use satellites There's a lot of observation of clouds, but the problem is extremely hard Good number seven So I'm from a somewhat difficult Different field. I used to do some medical imaging stuff on GPGP you and I was rather surprised when you said that The memory bandwidth was the most decisive thing about your usage of GPUs, which is really unusual because Usually used that for for great computational density per byte So is that mostly about I don't know having access to? Cheap really fast memory at a large scale or have there been like attempts to develop a six that is a application specific hardware for for your kind of computations and climate modeling Well The application well why we use them really depends on the characteristics of your application and You remember like you remember that the roof line model, right? and you can see that the increase in Kind of the difference between the two lines was about as big for the Increase in memory bandwidth then the increase in peak performance But it's a very different view you have of course because I guess you do a lot of matrix matrix multiplies and these Operations, but they're actually useful for both types of problems now I mean why we why they were chosen. I don't know Well, I know but I wasn't the person that made the decision This was supposed to see a CSES and they did a very thorough analysis And but one reason is that to build supercomputers you usually can leverage what is there in the consumer market, right? And you can you can essentially leverage what what what is already out there and try to put it together in a new way Rolling your own chip. I've seen there. I've seen that there is a talk about that is very expensive But nevertheless there have been Attempts using for instance FPGAs to try and co-design codes and chips and This is something that we will certainly have to look at in the future, but it has not been done yet operationally So the people I was talking with don't know for your adult from from Berkeley. There are some very interesting papers on that We have enough time questions. I see some other people number three, please Thanks for the very interesting talk and currently clouds or radiative transfers more than 2d And do you also plan to experiment with 3d radiative transfer like the 10 stream algorithm or something similar? Yeah, well right now the Rr rr radiative transfer code is it the Delta 2 stream approach? It is a bit the question on the processes you're interested in right if you're interested in a very Timely high-resolved radiative transfer question, you of course want to go there But radiative transfer is the other expensive part in a model So it's in one case it's the dynamics and then the other that the next one is radiative transfer and actually today we don't run Radiative transfer every time step, but we currently run it in the two kilometer model about every quarter of an hour and We think it's just enough not to affect the results of our simulations too much So first we'd have to to I would like to increase the the frequency We call that module before going to to to to 3d radiative transfer, but Yeah, it's a trade-off and Currently we think we're it's it's okay still But if you want to go to one meter alias there is a lot of work for instance using also Ray tracing and these ideas There's a lot of work going on in the community Okay, we still have 10 minutes. I see question on the on the web. Yes. Yes The Fortran code you were talking about has been around for quite a few years. What about the parallelization code if the architecture changes? Okay So You basically you have a number of ways you can approach this this porting problem, right? And this depends a lot on the politics going on in your community What is this community trying to be and where does it try to go? So for instance there is there is one idea that in order to port large code large scientific codes to different architectures You simply you have kind of the master branch of your code and you try to keep that as clean as nice as well tested as possible And then whenever a new architectures comes around you simply take that quality branch it off You let a shitload of PhD students on it that ported and then you have a new new codes The other approach is what what we were trying to do So Cosmo is what we call a community or consortium model So it's about I think about 15 weather services around the world. I want to use it It's about more than 160 universities and I don't know how many individuals up from the top of my head So they all have different machines. They all have different architectures and We chose a way which where we try or all he did he's the mastermind the Way that we have we try to contain The single syntax at the high level so we don't we want to make a minimal amount of changes between architectures, so this is why they use this this stellar they develop the stellar library in order to make low-level optimizations to the code and That's all the odd then we use the compiler directive which were supposed were supposed to be able to not change the code that much in Practice this is not Possible for every small detail, but for large parts of the code So there are these two approaches and just to tell you if you're interested about it Here is a list of literature and I think this paper describes it in detail Something to drink I have a couple of questions still we have some time number eight pleaser in the back Sorry, we can't see you really though Thank you for the talk You stated that most of your calculations were for climate models Do you think it feasible to apply these to operational short time weather models? Okay, so to develop the GPU version Was actually a lot of it was driven by mental Swiss, which is that the Swiss weather service and a lot of the developers were paid by mental Swiss and The initiative I think a bit of it was also born there because today They run this code that I showed you the baroclinic way from they run that operationally they do I think around 32 kilometer simulations and one one kilometer say simulation operationally daily and Currently the code is reintegrated into the the master branch and I'm I know that several other Weather services want to pick it up because for them it's much less about Large domains, but for them it's about the energy use So you you You spy babe by switching your architecture you save a lot of energy and if you have to pay the energy bill yourself The investment into the engineers will be compensated quite fast And number four in the back Thank you. I have four questions First for yes, firstly Yes first The time-to-resolution seems to be very high. What's the practical limit in that and secondly, what's the? Main time between failures for machine like that like like once you run and the third question What's the minimum resolution that you expect to be useful will there be a qualitative plateau eventually and Forstley and this is a little different Well, how about the the competition between models and the parametrization How much does politics play a role in that? I mean in the sense of international politics and the competition between different Climate models coming from different nation states. Okay. I will answer the first three the last one and it will take a long time You we can talk that about that in between so Yeah, minimum time-to-solution so it is the latency of an architecture so Latency of the memory access so That will there will be some physical limits on how fast you can Access a memory So that's why we think it will be very hard to go to to ten or twenty simulation years But they because today we apparently can't see an architecture that the latency is short enough The second question, what was it? Can you help me again? Yes, what's the mean time between failures for super computer? I don't know the number from the top of my head from my experience from doing these simulations is I would say Each month so we did this in monthly chunks and they would say it was about ten percent and it was mostly Disc access that failed Metal Swiss has two computers so whenever one fails they switch over the entire thing to another one So they've backup systems. I don't know in percent, but I think they're at least they're very robust and The third question was again dirt. Sorry. I'm so nervous I can't remember qualitative plateau in resolution. Oh, yeah So the ultimate resolution that we want to go through to is ten to the minus two meters So zero point one centimeter because that's the that's the that's kind of a The the inertial sub range of turbulence. I See nice nice crowd arriving here and the meanwhile you have a question at number five. I'm sorry cool intervention Thank you for your thank you for the talk Thank you for the talk so my my question is on the on the grid size so with Smaller grid size you have to decrease the time stepping as well So and and you're using a weather model as far as I understand to do climate modeling So in now your problem is that the time stepping is getting small and your new numerical errors at up and in fact if you do that over 30 years your your Results will be some kind of Full of numerical errors. So why not do a very good Simulation for one year do a parametrization and use that parametrization for the climate models again and do adaptive grid Blah blah. Yeah Okay, do we have enough time Okay, yeah, one minute in one minute So we will solve it all so in in numerical schemes you have explicit schemes on your wing plus Placid schemes and in the explicit seems yes, you have to decrease that your time step with the With the grid spacing but the cool thing is that the methods are only local so the the perfect linear scaling that I showed you is because the methods only involve local communication and many Implicit methods have global communication So if you think of it in terms of the energy question again The energy consumption is a lot higher then coming to the numerical errors Of course the numerical error is given by the grid spacing and a bit less by the small time steps because First of all We have a lot of forcing. We have the Sun which is a very strong forcing. We have continents that force we have mountains that impose forcing we have Very large dynamical systems that impose forcing. So these the forcing When you compare the strong forcing with the error the forcing will dominate then the other thing is that we have something called Chaotic systems or the Lawrence papers if you know what I'm talking about. So very small errors Can that cause very large differences in the simulation after a couple of days and We have by now we have some methods how to address this chaos problem Which we and the solution is we do a lot of simulation and perturb them a bit So in the end they will lead they will have a climate which is in the mean very similar but the initial state The state at a certain time to step is very different from each other So this is a lot of work by Tim Palmer and his group if you're interested I can recommend some papers and that Well, David. Thank you very much for this fantastic lecture We're glad to have you there to take a look at how this is changing