 Thank you very much for the introduction, for the invitation. Just briefly, I'd like to acknowledge some of the co-authors on this talk, some of my collaborators on this project. So Laura Condon is a graduate student. Laura and John are graduate students in my group. And Stefan and Ian are postdocs who used to be in my group, who've moved on to real jobs. Ian's staff at the USBOR and Stefan is at Bonn University. So the motivation for my talk today, and sort of for a lot of my work, is trying to understand dynamics of the... It's no matter how much you check your slides ahead of time, there's always errors on it. So the idea is to... And this is a... This is really weird. I wonder if this is a memory problem, because we just checked the slides during the break, and of course the slide gremlins never sleep. So the idea is that we have the coupled hydrologic cycle, which is not anything really new to anybody in this room. And this is important for a lot of different reasons. Climate change impacts, biogeochemical cycles, nutrient cycles. And what we tend to do is take all of these couple processes, and we model them in disconnected fashion. We have, for example, the groundwater vitisone model, or land surface models. And we have fluxes, surface water modeling. We have fluxes that are abstracted, and these things break down by disciplinary boundaries, often by time scales, a lot of the same things that we've talked about. Now my particular interests are understanding interactions between the land and energy budget. And if we write these in a really simplified way, this is not how we solve them, but how we typically write them, at the change in terrestrial hydrology and the energy balance, water and energy balance. What we immediately notice is that latent heat flux is one of the key variables that's shared between these two systems, and this is one of the key interactions. This is, of course, not at all new. The atmospheric community has published extensively about this starting in the mid-80s. And if we look at some examples from the atmospheric community on how these interactions can be important, this is some work from Ned Patton, who's just down the road, or just across foothills. And this is some work that he did in 2005, which I really like as an illustration. So what Ned did was he had a very finely resolved atmospheric model. This is 30 kilometers by 2 kilometers at high spatial resolution. This is actually a 3D model, and he's averaging into the page. And he has very simple experiments where he has a uniform soil, so uniform soil moisture across the 30 kilometer transect, and then alternating wet-dry patterns here. That very systematic, very simple setup. And this is water vapor mixing ratio in the atmosphere. And what we immediately see is that under uniform conditions, sure, there is some variability because of the turbulent nature of the atmosphere. But generally, if we interpret the PBL height or the height of the planetary or atmospheric boundary layer from this, we get a pretty uniform atmospheric boundary layer height. Whereas in the wet-dry patches, we see the dry patches heat up more quickly. And we have upwelling here. This is temperature differential, gives you a vertical velocity differential, which is just a convection cell in the atmospheric boundary layer. And this gives you variable boundary layer height. And driven by soil moisture conditions, is driven by this connection through evapotranspiration, latent heat flux in the water energy balances. So the idea is if we try to couple these systems, where do things kind of intersect? And what is the point of interaction? And that really falls into two categories, into land service models and groundwater models. And unfortunately, neither really up for the task of interactions as traditionally formulated. So land service models really simplify subsurface hydrology. They're originally developed as a bottom boundary layer for global circulation models, general circulation models, global climate models. They're really vertically oriented. They're these resistor type models. And they really have about a two meter subsurface. And while there are versions that are evolving with the groundwater box or another storage component, they're really not built in any way that there's topographic, either surface or subsurface lateral communication. And they're built in such a way that contaminant transport's not possible. So a lot of these biogeochemical cycling are really contaminant transport type questions, which are very familiar to the hydrogeology crowd. And they're really not formulated in such a way that makes this possible. Now, I would actually argue that groundwater models are worse. We tend to calculate evaporation completely operationally. It's an imposed flux. There's no feedbacks. It's often even very hard to design feedbacks because of the ways that we abstract the land surface. And the groundwater community has been incredibly slow to bridge scales and bridge high performance computing. And there are very few HPC or parallel groundwater models out there. And so really this divide has been somewhat of a struggling point. I would say on top of that that groundwater is often underappreciated in the whole scheme of things. We have a range of paths and scales in groundwater, and water may move really quickly. I'm going to move really quickly here as well. Groundwater may move relatively slowly in a physical sense, but pressure propagates several orders of magnitude faster. This is pretty well known, and this contributes to a wide range of scales. There's a lot of classical work. Jim Kirchner, this is a figure from one of his slides. Bill Alley, Roy Haggerty, this is a figure from one of his slides. And it's pretty well known that topography and land surface processes have a really strong influence. And so when we think about models that are typically used to describe these systems and calibrated models that are typically used to describe these systems, a lot of them miss fundamental types of behavior, particularly in scaling. This is a plot of spectral power and wavelength, which is just one of our frequency. And this is some data, chloride data. This is from Jim's Nature article in 2000. This is the rainfall chloride spectra, which basically has no coherent signal. And then this is the outflow chloride spectra of these same watersheds. And what we see is a really clear fractal signature where we have a one-to-one relationship and log spectral power, log wavelength. And if we look at our common, this is the Invection Dispersion Equation, which is the most commonly applied transport equation in groundwater. And we look at the exponential hyperrexone type model. Even though these are best fits, best fits of calibrated models, they clearly don't get the fundamental behavior. And if we look at a shorter time, this is some of Roy Haggerty's work. This is residence time distribution in the channel, log scale. And then this is time in seconds. This is the power, power, law, tail in Roy's data. And again, the exponential hyperrexone model misses the fundamental behavior. So the question is, if we use integrated models, can we get similar behavior? And this is some of my not so recent work anymore. But what we see in integrated models set up under similar experiments, and I don't have a lot of time to talk about the details of this work, but I'm going to talk about some research letters. We see the same in terms of residence, this is mass fraction and time, this is log time in years, log mass fractions is essentially like a PDF of residence times within a system. This is the same power, law type behavior that Roy sees. And then if we take the spectral transform of this, we see the same power spectra that Jim sees in his data. And so this gives us hope that we can use integrated models to get some of these reflections. So there is some, I'd say a growing body, and I expect you to read and memorize every single one of these citations, of course. But there's a growing body of work that seeks to pull these systems together. And I've kind of broken them out in like hydrology atmosphere, hydrology land surface, and integrated hydrology. And in this body of work, my group uses and develops a code called Parflow. And I think that there's a really exciting dynamic groups doing integrated hydrologic modeling work. And I think we're a really interesting time, and we can think about hydrogeosphere, or catheter, open geosys, or the Penn State model, or the integrated hydrologic model. We happen to use and develop Parflow within the group. But there's a lot of really, I think really terrific models that are out there. And what we do, and which is what most of these codes do, is we solve saturated three-dimensional Richard's equation everywhere in the subsurface. So we don't delineate between groundwater, veto zone. We solve 3D Richard's equation, compressible Richard's equation, everywhere. And where we get groundwater, due to geology, due to topography, we get groundwater where we have veto zones, we have veto zones. And then coupled to that, we solve overland flow and surface runoff using a combination of either continuity and manning in either the kinematic or diffusive formats. So we have a fully integrated system where we're solving the PDEs for surface water flow and subsurface flow in a completely globally implicit fashion. So we solve both these equations simultaneously at every time step. Then we include with that land surface processes using the NCAR model CLM. We're using it in common land model format, but it's pretty similar to the community land model, of the earth system modeling framework. And this has a number of canopy vegetation processes, snow processes, land energy budget, radiation balance. All of these things are within CLM, but we strip out the hydrology and replace it with power flow. And this is fully coupled. It's fully mass conservative. And then one of the pieces I'm going to talk about a little bit more with power flow is that it's completely parallel. I think every time I turn my head, the volume is up and down here. I'll try to change the microphone orientation a little bit. So we can think about this cartoon as a way to understand this modeling system. And so we have topography, we have atmospheric forcing, we drive this model with observed meteorology, pressure, temperature, solar, precipitation, and then we populate the land surface with vegetation and land cover. And what happens is the model evolves and develops naturally. We have areas with a very deep veto zone. We might have areas with a very shallow veto zone where vegetation may be free out of fiddic or groundwater dependent. And then anywhere where we have ponded water, either through access saturation or access infiltration, we get automatic topographic routing. We have groundwater. We can look at flow divides. We can do contaminant transport in the system where we can use Lagrangian techniques to trace out flow lines. We might see that say the flow divide doesn't necessarily match the topographic divide, which is more common in groundwater systems. And these are really a nice platform for testing these sorts of hypotheses about connections within the hydrologic cycle. So the idea that we have, or at least the conceptual model that we have, is that we should see some of these interactions in really specific ways. So first if we look at another cartoon of our system, we think about on the hill slope, we have water table, we have overland flow. In terms of groundwater's impact on the land energy budget, we have areas where groundwater is very shallow. And we categorize this kind of as like a riparian or river zone, convergent zones where land surface processes are not going to be so moisture limited. Because groundwater is always so close to the land surface that we essentially have no water limitations on land energy processes. We can then jump up to the top of the hill slope where we hypothesize that the water table depth is so deep water is so far from the land surface that land surface processes are decoupled. They're not able to access groundwater, they're not influenced by groundwater, so they're not really getting any benefit. And essentially these things act like two coupled disconnected or uncoupled systems. And there may be some flux of water from the land surface, some recharge flux down to the subsurface or not. But that's the extent of the coupling. So then we hypothesize that there should be this critical zone where we have this really tight interaction between land surface processes water table depth. And if we look at thinking about this sort of another way because we can think about this as a coherent 3D system or we can think about this as a series of columns we can think about this as columns that should break into water table depth where we have a really close groundwater land surface. We have a very far groundwater land surface which we disconnect. And then we have this critical zone where we have small changes in groundwater depth that give us big changes in land energy flux. So we can test this with a couple of models. And this is some work that appeared in water sources research in 2008 that Stefan and I did and there we were modeling basically a research watershed in Oklahoma called the Lulawashita, which has been the subject of a lot of study both by us and other groups. And the Southern Great Plains in the U.S. are really historically ripe area for land atmosphere interactions. And here we have at the end of a, this is a dynamic run we're forcing it with one hour observing meteorology. We spin the model up we compare to observations. We don't actually calibrate in this. We compare to observations and it compares very favorably. So then we start to look at understanding dynamics within the system. And one of the things that we did was we looked at annual average latent heat flux plotted here versus annual average water table depth. So every dot on this plot is a spatial location where we can have an annual average latent heat flux energy component to evapotranspiration or and an annual average water table depth. We immediately break this out by land cover, right? Shrubs crops and trees. And what we see is that we see some differences due to different vegetation types, which we'd expect. And then we see some differences. This is log scale from shallow and deep groundwater. The first observation we might make is that these differences are about the same order. Different vegetation types gives us about 15 watts per meter squared. Different between shallow and deep groundwater gives us about 25 watts per meter squared. Pretty similar in terms of impact. What we also see is that we have a very flat region here which we hypothesize is our shallow groundwater where we have changes in water table depth here that give us zero change in latent heat flux because we can't add more water to the system. This system is operating on atmospheric demand. So if there's more water available, there's still no more atmospheric demand for that water and so this won't change our land energy flux. We also see in deep ground waters, we see again a region where changes in water table depth give us no change in latent heat flux. We hypothesize that this is our disconnected region where it doesn't matter if the water table is 10 or 100 meters, 10, 20, 40 meters, it's still disconnected from the land surface and so changes in water table depth don't impact changes in land surface energy processes. Then we find this region where we see this really tight slope between latent heat flux and water table depth and we hypothesize that this is our critical zone. We see really small changes in water table depth give us huge changes in latent heat flux. So we can do this again except for climate change impact studies and this is some work that appeared in Nature Geoscience a few years ago and what we see there, these are now latent heat flux differences between different perturbed climate scenarios. We have a hot scenario, two-degree increase in all cases. We have a hot wet scenario where we also perturb precipitation and we have a hot dry scenario where we perturb precipitation down by 20% and increase temperature. So this is hot wet, hot and hot dry compared to our baseline which is water year 99 which is pretty climatological. What we see is that we get these three regions except now it's a temperature controlled region where it's shallow water table depths all the cases basically within riparian areas or within the river channels still have similar access to groundwater. So they're not water limited and their changes in energy fluxes or their changes in ET are all controlled by the increase in temperature. We have cases where all three are disconnected again deep water table now they break out by precipitation or hot wet is an increase in precipitation and hot dry is a decrease in precipitation and the hot case is the same precipitation as the control so it dives to zero and then we find a really important groundwater controlled region within our critical zone. So we see this impacting not only sort of understanding current but also understanding changes or perturbations in the hydrologic system. So as if that weren't enough we've now twice coupled power flow with two different regional climate models we did this first in 2007 with the ARPS or advanced regional prediction system. We've done this again this came out actually not in 2010 but in 2011 and month of weather review with WARF which is the weather research and forecasting and it's been talked about in earlier talks. So now we have a complete interaction between power flow the NOAA land surface model within WARF and the atmosphere within WARF so we have a full accounting for all of these interactions and the reason that we want to do that and this is some work from AWR in 2007 is that we've seen these interactions be important for particular systems at particular times. This is more work in the La Washita this is actually an atmospheric fully coupled atmospheric groundwater simulation where we're actually at the beginning of the second day this is a transect x direction in kilometers is just a transect through the middle of our domain the two river valleys highlighted here we have water table depth plotted here in meters across this transect we have soil moisture here plotted in meter plotted in percent across the transect we have soil temperature plotted here and then we have boundary layer depth in the atmosphere plotted here so what we see is that areas that are shallower groundwater have increased soil moisture that makes sense we're in these river valleys what happens at this particular point in time in the simulation is that these river valleys heat up more slowly than the hill tops so we have a differential of about 4 kelvin between the river valleys and the hill tops this creates this is vertical this is atmospheric wind here this creates differentials in the direction this is zero of our vertical wind gradient we have downward in the river valleys upward on the hill tops this is exactly what Patton showed we have a convection cell and this convection cell alters our evolution of the planetary boundary layer this is driven by groundwater these moisture differences which drive temperature differences which drive vertical velocity differences which drive the evolution of our atmospheric system were ultimately driven by groundwater storage so we can look at another example to sort of dive into this this is with pf4 and here we have a really simple this was our test case which we used for a lot of different things within the paper where we injected moisture into the atmosphere so we could control the amount of moisture we injected in the atmosphere we could check and make it rain look at the distribution of moisture within the coupled system and we can use this as a mass balance test now in this particular system we have a gradual slope we have open boundary conditions in the atmosphere we have no flow except for surface we have par flow here and warf here and so if we look at the rainfall pattern this is definitely Hawaiian type rain this is 2.4 meters we made it rain like crazy to stress the model we really wanted to really really stress the model we see we have area very heavy rainfall where we've injected moisture and a bullseye and no rainfall elsewhere in the domain as we look at the evolution of some moisture within this system we have at early times saturation fall is rainfall as if it would in plain warp this is just no different than any other land surface model but at later times we rain hard enough that we actually generate saturation excess that saturation excess is routed down slope and eventually flows out of the domain now what this does is if we look at contours of latent heat flux our area of original rainfall at the end of our simulation we rain for a certain amount of time then we turn off the rainfall is not actually our area of maximum latent heat flux our area of maximum latent heat flux is where water was routed not where the rainfall originally happened so this is a nice very synthetic idealized case that demonstrates how these interactions can be important sort of in two ways okay so getting to sort of the high performance computing part of this now this is some work out of a large European effort called the TR32 or Transregio 32 which is centered at Bonn University but has collaborators all over Germany and this is some photosynthetic this is some remote sensing data that's basically a tower remote sensing product and they're using photosynthetic activity which they're sensing remotely as a surrogate for ET and so if we look at these patterns really what's interesting to me is that we see this range of ET values and as we zoom in we see that these have a large variance across a number of different scales so this got Stefan and I very excited and we said well can we start to model this directly and one of the first things that we did was we were interested in deploying power flow at sort of petaflop scales so we have access to Eugene which is a one petaflop almost 300,000 core supercomputer at the Forson Center in Munich and we built a fully coupled 3D non-linear test problem and we looked at scaled parallel efficiency and what scaled parallel efficiency is is a measure of your scalability of your code so called weak scaling where you're simultaneously increasing the problem size and the number of processors such that your problem size should grow with the number of processors and if your wall clock time is the same on out you should have 100% scale parallel efficiency so what we see is we run out to an 8 billion cell problem at 1,384 processors we see roughly 60% scale parallel efficiency which is excellent this is leadership class this is as well as many code scale and this demonstrates to us that we have an efficient code that we can scale out towards the petaflop scale the sorts of things that we can do with this is we can look at effective heterogeneity this is a one meter by two and a half centimeter resolution simulation at 32.15 square kilometers so we built a synthetic 5.6 by 5.6 square kilometer watershed at one meter by two and a half centimeter resolution we forced it with observed meteorology we had imposed strongly anisotropic subsurface heterogeneity and what we see is the same range of variance that is seen in these high resolution photosynthetic remote sensing type exercises the idea was that my talks used to finish here this used to be right so we can do this and blah blah blah but one of the things that we really started thinking about a lot and it's been workshops on this and if we're running an 8 billion cell high resolution watershed simulation we do the math this is continental scale at still relatively high resolution it's a kilometer resolution and so why aren't we doing this well we decided to do this and this is not a unique idea there's been synthesis workshop that concluded this was important Eric's paper recently came out this is a great charge for high resolution high resolution and land surface modeling is a grand challenge and Eric has the nice this is a figure from the Amazon this is basically complex topography and we see all of the different scales of river networks so the idea is that why shouldn't we do this so this is what we did this summer we built a one kilometer resolution 6.3 million square kilometer power flow model of most of the continental US so we have the entire Mississippi we have the entire Colorado watersheds and really this is it's actually not all that big a problem it's only 158 million unknowns and really well I mean we're regularly running 8 billion so this is orders of magnitude smaller than you know what we could run and this has come online relatively due to a couple new data sets one the Hydro Sheds data set is a fantastic resource for this but the big question for us is always what to populate with the subsurface and Tom Gleason at UBC he's just taking a faculty position published a hydraulic conductivity data set in physical research letters and was kind enough to share this data with us so we could use this to populate our subsurface and so we're solving a fully integrated 3D Richards equation shallow water equations land surface processes all in one shot so this is our soil map this is Tom Gleason's hydraulic conductivity map and if we look at preliminary results these are return flow tests we'll talk more about this this is the whole domain at one kilometer resolution we don't see really any detail at on this particular screen because it doesn't have a resolution to show but if we start to zoom in and we zoom in on the south plat this is the watershed that we sit in right now what we notice is that we see river networks we see the same range of scale of behavior that we see in these high resolution topographic examples that Eric and colleagues had in their WRR paper so this will be the second moment of truth of the video's work we can look at a runoff test over the whole domain so this is a runoff test over the whole domain but we're actually just looking at the south plat so we rain on it then we rain on again we see the domain starts completely dry we impose no explicit river networks everything evolves due to topography so we see the river networks are maintained we see that we get a pretty wide range of behaviors you'd expect and we can zoom in on the junction between the north south plat and Missouri so that's all happening right here this is just the same simulation this is a movie all done with visit by the way which is which is again zoomed in we solve the whole thing at once but we're just sort of zooming in for the movies we see we get pulses of water that move through the channels the channels are behaving properly the entire model is behaving properly and topographically seems pretty reasonable so then we did a couple of return flow tests the return flow tests we essentially set the water table at a uniform depth from the land surface so it's out of equilibrium and then we let the water table converge this is a return flow test we did two one with homogeneous subsurface everywhere and then one with Tom Gleason's heterogeneous subsurface and this is at the end of 2000 hours of return flow tests and we see that we're defining the topographic convergence again hard to see if we don't zoom in but if we start to zoom in we see that we're picking up the groundwater contributions or the base flow contributions to these different networks and we see real cross scale type behavior now if we look at with Gleason's heterogeneous subsurface so we now have in orders of magnitude of parameter variability in subsurface and we zoom in on the same snapshot we see that we have effects of subsurface properties and of topography and if we even zoom in further we start changing scales what we see is that we're getting cross scale behavior we have high perm areas that respond quickly due to topography lower perm areas that respond more slowly and that we're getting some very interesting type of behavior this is just sort of a we're running it now with distributed forcing from the HRL we're spinning it up so now we're forcing it with observed meteorology we have preliminary this is ground temperature and latent heat flux and so far so good things look pretty reasonable so in conclusion I think it's an incredibly exciting time and we have this a lot of real interest in these large scale simulations a lot of interest in coupling models a lot of interest in a lot of action in terms of groups that are developing really nice integrated models and I think it's very exciting and I think that we see these same feedbacks that are historically talked about in the land atmosphere literature also being driven by groundwater and we can tackle these grand challenge problems using in hydrology using integrated models in high performance computing and so you know kind of a parting thought as we increase resolution and think about these big questions we have these big challenges and I think that we really need these innovative solutions so thank you very much for your attention happy to entertain any questions