 and actually just get started. So as I mentioned, my name's John Swartz. I'm a postdoc working with Pala Pasalakwa at UT Austin. I am both doing research with her as well as co-managing or directing the research coordination network, advancing the analysis of high-resolution typography, which is an effort led by Pala, Chris Crosby, who you see here as John Swartz number three, and Nancy Glenn, who might be joining us later in the clinic. So with that said, we have a little too many people for, oh, we have another John Swartz entering the waiting room. Oh, we'll admit a little bit. I just admitted two more. Sorry, this is just getting comical. But we're going to just kind of dive into the clinic material. So Pala will be helping me moderate this. And so if you have any questions or comments, please put them in the chat and she will interject and raise those questions. And if it's worth something having a broader discussion for the entire audience, then we'll go ahead and have you unmute yourself, and we can have a little bit of discussion there. The format of the clinic is going to be, we're going to be bouncing back and forth between a presentation I've created, as well as the different notebooks and workflows on the Google Colab that we're going to be going through. And John, I was going to say that, hi everyone. And you can certainly feel free to raise your hand, unmute yourself, and show your video if you want to ask a question and voice it. We're not a lot of people. So I think, differently from this morning from the keynote, we can have a more fluid kind of format. So don't be afraid to to interrupt. I mean, I think for John, it's going to be hard to see this thing. So Chris and I will monitor the chat and monitor if you're raising your hand, but also just feel free to interrupt. I mean, what's important is that you get the most out of this. So don't be afraid of, you know, interrupting, speaking up and just it's going to make it way more fun. Yes, thanks, Paula. I'm sorry, I was just taking a screenshot of the eight John's forces that we have. I did. This is going to end up on Instagram later. But yeah, so again, we do have, you know, a good enough kind of number of people that we can have much more of a discussion than if we were doing a clinic with like 80 or 100. So please feel free to jump in there, like Paula mentioned. So with that said, we are, let's see, we're recording. Let's go ahead and jump things off. So let's see. All right. Can everyone see that? Can all the John's see that? Excellent. Okay. So thank you everyone for signing up for this clinic and joining us in the virtual CSDMS. I'm sad that we couldn't all meet in person, but hopefully next year we'll still see a lot of each other. So as you know, this is the clinic for exploring surface processes and landscape connectivity through the use of high resolution topography and exploring the integration of data from high resolution topography in numerical modeling. As I mentioned, I'll be your host along with Palo Paso L'Aqua, but this is also a project that's co-led by Nancy Glenn and Chris Crosby. So this is all happening under the auspices of our research coordination network, which is funded by NSF EarthCube that's labeled the advancing the analysis of high resolution topography or a squared HRT. And this is an initiative funded by EarthCube that's focusing on assessing the current state of high resolution topography usage and developing best practices and linkages between models and observations within our community of practitioners. If you want to be involved in this effort, we have a working group on data model integration that some of you may know about that will be starting up with a series of online webinars as well as leading towards a white paper on the state of the science of data model integration. We also will be starting out a summer learning series focused on common workflows and data handling techniques involving high resolution topography that Chris and I are currently working on. And if you want to be involved with any of these initiatives or you have any other ideas or just want to know more about what we're doing, please contact us or meet specifically. So in overview of what we're going to be doing today, we're going to first go through an introduction to high resolution topography, its applications as well as a broad sense of landscape and hydrologic connectivity. We're then going to move into a hands-on exercise looking at basic raster and terrain analysis and uses of high resolution topography using a series of Python workflows and notebooks. We're going to go back then to look at some details and examples of the use of high resolution data within numerical modeling, followed by an exercise of actually running our own hydrodynamic models using an open source platform called a NuGetup. And then we're going to wrap up the day with just a broad discussion about kind of what we've seen here or any sort of questions or comments that people may have or ideas of how we can kind of move the community in some of these themes forward. So broadly, we want to start off with a question of what is high resolution topography? This doesn't really have a good definition. A lot of people will say they know it when they see it, but generally you can define it as technology that can acquire meter to submeter scale datasets. So this encompasses a wide variety of platforms at this point. So you have LIDAR, you have photogrammetry such as drone structure promotion, you have synthetic aperture radar, or you have multidimethymetry if you're working in a subacquoise environment. What you get with using these sorts of techniques and platforms is accurate characterization of small scale processes and landscape structure or vegetation structure. But with this accuracy and with this detail comes challenges in data handling and computation because datasets over large spatial areas can become quite complex and quite large in terms of size. So we're going to start by just moving through the two most basic terrestrial high resolution topography methods, which is LIDAR and structure promotion. So starting with LIDAR, just to put it in a very simple way, what we're doing with LIDAR is we're measuring the distance that a laser travels in two-way travel time, and this is a laser that can be sourced from a spaceporn, airborne, or terrestrial platform. LIDAR gives us a vertical accuracy typically in the five to 15 centimeter range and horizontal accuracy in a 20 centimeter to one meter range, depending on the point density of our acquisition. Now the quality of LIDAR can be highly variable as anyone who's used it a lot can know. So you can occasionally have location uncertainty, you can have errors associated with steep terrain slopes, you can have errors associated with dense vegetation, you can have errors associated with your flight footprint. A lot of different things go into LIDAR acquisition and processing, and so understanding these workflows is critical to assessing your data quality. One of the advantages that LIDAR has over other technologies, which is why it's so widely used, is that we're not simply getting a single elevation value for a single beam or return, instead we're getting a profile that looks like the above right diagram, where a laser sent from a LIDAR platform is passing through the canopy before finally hitting the ground. And so if we look at the time series of those returned energies, we can assign different variables to these different returns and let us differentiate and subtract out vegetation structure as well as the final ground return, which gives us those nice beautiful digital terrain models or bare earth DEMs that we love to use. So LIDAR does not give you a DEM from the platform, it doesn't give you a DEM out of the box, you have to process it and what it does give you are point clouds. Now point clouds are an interesting subject in their own right, but it's kind of just useful to look at what a point cloud is and kind of understand what its structure is and how we might use it. And so if we were running this in person, like if we were unfortunately subject to the pandemic of COVID right now, we'd be sitting in this building right here, which is CSDMS on CU Boulder's east campus. This is a point cloud that's being visualized in Open Topography's online point cloud browser system, which is a really cool platform I suggest you use. And you can see that by visualizing the terrain in this way, we can see both the kind of ground, we can see the trees, we can see building structure, we can see the front range up in the distance, but we have all these details of the landscape, not just the kind of bare earth terrain. And so by handling point clouds derived from LiDAR data or other platforms, we have the ability to kind of query different aspects of the Earth's surface in a really interesting and useful way. So we have a point cloud from a given LiDAR survey, how do we get to those DEMs or those rasters that are often what our end product is that we then want to use in our analysis and use in our science. So the very kind of basic workflow is we had a choir and QC a survey. So this is an example from the USDA Forest Service looking at different flight paths and the different density of both pulses, outgoing pulses from the LiDAR instrument, as well as returns, seeing that the actual return density can be quite variable. We'll take those returns, which form our point cloud, and we'll classify those. We'll classify which points correspond to the ground surface, which points correspond to the vegetation surface, which points correspond to water or buildings or what have you, however many kind of classes you're doing. You'll take those returns and those points and then you'll grid them based on some return density. So again, as we saw in this upper left image here, the return density for a given survey can be quite variable. And so we need to make some sort of grid resolution choice that honors our data and doesn't either kind of captures things in a too sparse or too dense manner so we can optimize what resolution we're getting for our eventual DEM. What I'm showing here are just a series of algorithms from the points to grid, algorithm that Open Topography uses in its data server for serving up DEMs from the LiDAR point clouds that they have archived. And then finally, after following this workflow, we get our beautiful raster, in this case a bare earth raster of a meandering river system somewhere in the American West. So LiDAR is not just about kind of terrestrial systems, however. And I'm just going to put this out here because I think it's a really interesting and important point, especially as the technology becomes more mature and its uses become more ubiquitous. But we have the ability through certain LiDAR platforms to acquire total bathymetric LiDAR, which allows us to basically see through the water column if water conditions are correct. And so what we can do here then is capture the elevation structure or the landscape structure of the sub aerial and sub aquaist environments at the same time about using different surveys or different types of acquisition technology. And so this is an example from the Gulf of Mexico looking at the San Luis past tidal inlet showing the structure of the barrier islands and then this beautiful detail and imaging. But we have another John Swartz. Just beautiful detail and imaging of this flood tide delta located down in the Gulf. And so this data is starting to become more publicly available. It's starting to be acquired more widely. And so the potential advantage of the coastal and fluvial systems is pretty exciting to think about. So I just wanted to kind of put a plug in there for that. Okay, so the next type of technology that we most commonly use and is becoming really, really widely applied is structure from motion photogrammetry. So this is that technology that we know from things like either terrestrial gigapan systems or more commonly now drone systems. But it basically relies on building a 3D model by matching corresponding scale invariant features using multiple angles or offsets of just imagery or just photography. And so in the example here we can see a series of four photos from a desert environment looking at the same scale invariant feature that's highlighted in the red box. And we can build them 3D models or point clouds of these surfaces without any a priori knowledge of where we are, what our camera angles are, or what our locations are, because we these scale invariant features that we then match and then use basically off the shelf commercial software to build very dense point clouds and digital surface models. This can be done really cheaply. It can be done quickly. And so as a result, it's really exploded in usage and it's starting to kind of overtake lidar in terms of just how the average user can kind of go out and acquire their own surveys. So as I mentioned, we can acquire really nice 3D models pretty quickly and we can also do some really amazing science with them. And so as an example, this is some work by Kendra Johnson looking at fault zone topography. So going out there and mapping this fault surface using structure for motion technique. And so what you can see in her work is that you can build and you can both acquire and build ultra high quality structure for motion DEMs with extreme photo density. But then if we compare it to pre existing lidar surveys, once we have our ground control points in there are giving us almost lidar quality of those bare earth terrain models from a pretty inexpensive drone survey. So this is your own user acquired drone versus a like kind of spec aerial lidar survey, getting pretty good results. So that's great. But what's the cat with that? If it's so wonderful, why aren't we using this for everything instead of lidar? One is it can be really difficult to acquire really large spatial areas using some of these techniques, particularly the drone based ones, versus aerial lidar surveys. Additionally, and this is a huge one, they cannot see through vegetation, right? What makes lidar so powerful is that we get multiple returns for every single pulse that let us differentiate between what's vegetation, what's the ground, what's manmade structures and so on. Whereas with structure for motion, all we're building is a 3D surface, a very detailed 3D surface, but one that encapsulates and incorporates all those different elements that we're seeing in a photograph. We can also have some issues with lens and camera distortions, such as I'm illustrating here on the right, that when propagated over really large areas in your DEM can lead to some sorts of doming or banding of the features, although there are ways of processing this out. But one of the major things compared to lidar that can make using structure for motion challenging is that right now there is a lack of standardization in acquisition and processing data quality. We don't have good standards for how we handle metadata associated with structure for motion. We don't have good standards for what data you report or how you handle and store the data like what you would lidar. And so as a result, when we want to compare, like for instance, repeat surveys like pre and post storm or pre and post fault rupture, not having these details of the processing can make making those sorts of comparisons difficult in a really concrete way. So I mentioned a few times that both lidar and structure for motion digital terrain or bare earth models versus digital surface models. And it's worth just kind of spelling out in a little more detail what that actually means. So DTMs and DSMs are probably the most commonly encountered topography data that we'll see. So the DTM is that classic bare earth surface. So this is using lidar stripping with a vegetation and building a really, really nice terrain model of what the landscape looks like. If you didn't have any of those pesky trees or bushes in the way. With a DSM, that's what we're looking at the upper right here. This is just the highest point structure. So this is just the highest elevation that could correspond to the terrain. It could correspond to a building could correspond to the tree canopy and so on. What with lidar, we can build both of those. But with structure for motion, we're typically limited to digital surface models, although there are some interesting processing workflows you can do to try to build a train model. And both have utility depending on the science. So we don't always want to just jump to these digital terrain models because often that can be neglecting really interesting structure of the landscape involving the vegetation that might be really important for the question we're trying to ask. So just to illustrate the difference between a kind of surface model and a terrain model using an example from north of Houston. So this is a really densely vegetated lowland depositional plane setting. On the left here, you can see a high resolution just visible imagery satellite image from GUI. And the right is a one meter lidar image stripping all that away. So we can see that with using things like DTMs and lidar derived elevation models, we can see kind of an unprecedented level of detail of the landscape. So on the left, you might be able to point out where some of the stream locations are, you might be able to point out where some of these smaller creeks and drainage are. But the main signal that you see is just that you have kind of different patterns of vegetation. When we remove all that, we can see just beautiful detail of all these meandering small scale stream systems that are responsible for kind of really kind of plumbing the landscape and a lot of these lowland terrains. So what are some common applications and uses of high resolution topography? We've touched on a few of them so far, but I just want to go kind of go through some of the ones that are most commonly used and kind of the most exciting in my opinion. So one is we can use them you can use high resolution topography to really characterize the structure of the landscape. So like in the previous image, we can strip out all that vegetation and we can understand you know what the structure of these drainage networks really are. We can also use it to characterize small scale processes and surface characteristics that might be difficult to capture in a synoptic way otherwise. So this is like the observation of hill slope failure or creek, the observation of very small scale drainage systems in flood plants, things like that. We can use high resolution topography as an input to surface process modeling. So we can use it to derive attributes that are necessary for fully parameterizing our model or we can use it as kind of raw inputs into the model itself. We can also use it to get detailed information about the vegetation canopy structure. So linking different types of vegetation or different kind of vegetation characteristics to different portions of the landscape which is actually touches upon some of the keynotes that we saw earlier today. And then finally, what's becoming more and more common is to use these techniques to do really high precision change detection. So on the right is a bit of work by Jonathan Mason on the Trinity River in Texas looking at the change of the plan form geometry and volumes of point bars of the river between a large flood that occurred between 2011 and 2015 and doing quantitative change detection of how much volume was lost and built on each point bar going downstream in the river. So when we talk about the landscape structure, we've been looking and trying to characterize this since the start of geomorphology and geology, right? And one of the biggest revolutions that occurred was the use of or the the ability to kind of use widely available topography data worldwide such as the SRTM data sets which are in 30 meter spatial resolution. Now if we kind of zoom into an area here and this is all in West Texas by the way, we can see that in the upper right we have that 30 meter DEM from the SRTM data and we have a 1 meter DEM from the USGS that's a LiDAR derived DEM. And when you're just looking in terms of these, they look fairly similar. Maybe the SRTM is a little bit hazier, maybe the LiDAR is a little sharper, but in general you can kind of see similarities in terms of the landscape structure. What high resolution topography really allows you to do is look at the small scale structure in detail of what's kind of dominating this landscape. So I've zoomed in now on that area and applied a hill shape to that DEM. And we can see now all these kind of different features that are really really interesting to think about and think about their role and what's controlling the processes of this landscape. So as I mentioned, one of the kind of uses or utilities of high resolution topography is not just in having a really, you know, high quality big data set that you can do all these sort of computations or metrics on, but it's also just letting you see and understand the landscape better than you could previously. So if we start annotating this landscape here, we can see all these sorts of interesting observations or features that might lead to the formulations of questions that we want to then go out and test either in a field campaign or through an American model. So for instance, here we can see the asymmetrical divides of these drainage basins where we have kind of really steep channel heads eating away into this plateau. This plateau is dominated by all these funky depressions that I don't have no idea what they are. Within the drainage network here that's eating away into this plateau, we can see that it's interrupted all over the place by dozens and dozens of small scale dam structures, which are agricultural stock tanks or dams that are in place at a local level and have almost no regulation or documentation at all. So when we look at the landscape like this, we don't just say, oh, we know now where the drainage networks are, we know what the drainage divides are, but we can start asking questions about what is the implication of the fact that every single channel head in this landscape has been interrupted by a dam? What is the implication for these asymmetrical divides? High resolution photography gives us the ability to both kind of better quantitatively query the landscape and do computations and analysis on, but also formulate ideas or hypotheses that we've got then go out and test about what processes are actually active. Now with networks, one of the biggest advances in things like network delineation and flood modeling has come from high resolution depography because especially in lowland systems, we need this high resolution depography to really resolve these networks and resolve the landscape. So here I put just together four panels of 30 meter SRTM data, the 10 meter national elevation data set, which until recently was the gold standard continental US data set that you had publicly available, three meter lidar also from the national elevation data set and then one meter lidar. So you can see that in each step of this resolution, we're getting more and more detail into the structure of the landscape. And that has pretty significant implications to our understanding of both the kind of network structure as well as just the amount of relief that exists here. So you can see for instance here, each of those data sets plotted along this yellow profile, and you can see that all the other data sets aside from the one meter are drastically under measuring and under representing the amount of relief that actually exists within this floodplain. And that amount of relief has a lot of implications both for flood routing and propagation, but also things like how fast is this landscape dissecting? What are the potentials for evulsion and so forth. So with data sets like this, we've actually been able to really improve our characterization of flood mapping. So in this example here, we have the female 100 year flood zone. And then we have an improved flood zone based on the recognition using high resolution topography of these small scale gullies and structures that are actually much more common in the landscape than we would have previously thought. And just to illustrate that further, here's the 10 meter national elevation data set of a floodplain in Texas. And here's the three meter lidar of the same area, again showing that without this high resolution topography, we're missing a lot of the fundamental structure of the landscape that has a lot of implications both for surface processes, but also things like ecosystem service and habitat distribution. So not to neglect what we can get from the vegetation. As I mentioned before, the high resolution topography specifically the lidar allows for that characterization of the vegetation structure, but also the type and the density. And so we can start using these techniques to relate our landscape structure like the drainage density of a floodplain or so on to what kind of the ecological systems and biospheres are across the different portions of the landscape. Also, interestingly, we'll see some more of this later on, we can use this data to more accurately characterize and parameterize things like land cover and surface attributes, which might be potential inputs into numerical modeling. And finally, as we already saw with the Trinity River, we're seeing a lot of use of these now for rapid surfacial change detection. So here's an example of looking at kind of really detailed structure of a landslide displacement using drone structure for motion imagery. And so you can see here then looking at a pre and post the landslide event, a lot of small scale deformation and features that help us understand how landslides occur and also understand kind of actual magnitude of displacement in a horizontal way that would just never be possible without this sort of very high resolution techniques. Okay, so now that we've basked in the glory of high resolution depography and we've seen a lot of what it can do and what some of the common applications are, let's give it a try ourselves. Let's start playing with some data. So I'm going to exit the presentation. And in the email I sent to everyone earlier, you should have seen a link to the GitHub that I put together for this for this clinic. And if you are following along with me here, so I'm looking at the main screen of GitHub right now, you'll see a bunch of data to folder, so if we'll do a look later, and then we'll see a series of notebooks here. Now, for the first one- John, we'll still see the slide, actually. They might be still loading. Let me let me let me pull that. Can everyone see that now? It's coming. Yeah, we got it. All good? Okay. So now now that we can all see the GitHub here and I this you should have seen this link in the email. We're going to click on this first notebook, which is labeled CSDMS 2020 HRT part one. So we'll open that. It's a little big, so it might take a while to load. There we go. And so you should be confronted with the screens. We're just looking at the Python notebook within GitHub. And what we want to do, which is going to allow us to actually have some fun with this, is click the small button above the title labeled opening co-lab. And I know that we might have some issues with this for people who are outside of the United States. If you do, I sincerely apologize, but these notebooks are going to be available to everyone for you to try to incorporate and build in your local environment. Just within the confines of the clinic, we weren't able to figure out a way of doing this on everyone's kind of home installation. If people want to use them so you can either put messages in the chat if it's not loading for whatever reason or there's also an option for using the actions at the bottom. So you can always give a thumbs up. And so that is a question from Nicole, John. Yes. Nice. So Nicole, here, let me go back to the GitHub and show you. So if you're on the main GitHub folder, you'll see part one notebook. So you click on that notebook. It's labeled part one. It'll load. And then do you see the small button that's labeled opening co-lab? You'll click on that. So now hopefully everyone is able to open the notebook and then click on opening co-lab and opening co-lab. And so this is a really, really cool platform that allows us to basically run and handle everything in a Google virtual environment. So we don't have to deal with Python installations and versioning and getting all the libraries and dependencies installed in everyone's personal machine. So it's a really cool tool for clinics and teaching. So once you have the notebook open within co-lab, again, hopefully everyone's able to do this. I again apologize if for some reason we have some regional issues. But once we have the notebook open in co-lab, we'll go ahead and connect to a hosted runtime. So in the upper right corner, you'll see a little button that says connect and then just click connect and then click to hosted runtime. And what this is doing is asking Google co-lab to provide us with a virtual, essentially a virtual machine that we'll use to do our computations. So once you've done that, you'll see that it's working because you'll have a little status indicator in the upper right that shows your RAM and disk space. And they actually give you a pretty good number of resources, which will come in handy in the second part of this clinic for sure. So once you open the notebook, you click on open in Google co-lab and you connected to a runtime, we can go ahead and actually get started running through this workflow and kind of just having some fun with some of the data. So the first part of this is going to be basic handling and analysis of high resolution topography data. And then we're going to showcase some kind of more advanced techniques or the potential for using Python workflows to do terrain analysis and network analysis using high resolution DEMs. Most of what we're going to be doing today is going to rely on an open source toolbox called white box tools that was developed by John Lindsay of the University of Guelph. This is an incredibly, incredibly powerful open source tool set that's really, really fun to play with. And so if you're someone who's kind of terrain inclined or interested in using Python tools and landscape analysis, please check out their GitHub and give some of these tools a shot. And but we're going to showcase some of them today. Okay. So now we're all open in Google co-lab. We have our environment set up. We can go ahead and we can start actually then running some of these cells. And so I've structured this and try to document it in a way that should be fairly easy to follow. But please just say something in the chat or raise your hand if you have any questions or you're not quite getting what step we're at here. So we're going to start by cloning the GitHub, which has a lot of the files that we need in it and installing our environments and importing all our necessary library. So if you're not familiar with Python or Google co-lab, you can execute a cell, which I've already pre-populated and wrote out all the routines that we're going to do either by hitting shift enter or pressing the little play button. That's to the left hand side of every single cell. So when you press the little play button, you'll get a warning saying the notebook is not authored by Google, which is true because it's authored by me. We'll click run anyway. And what we're doing then now is... So did you want us to clear all the outputs first or no? Oh, yes. Thank you, Paola. Yeah, yeah, absolutely. So sorry. I jumped the gun here a little bit. If you could, please. And you can do it even after you hit the run on that cell. Could you click edit underneath the notebook title right here and hit just clear all output? We have to do this because for some reason, the notebook has all the previous output populated in it, which is not what we want. Just ruins the surprise. Thank you, Paola. Yeah, so please go ahead and just clear the output. Again, like you saw, it didn't affect anything that we're doing if you had already clicked run on the cell. But I can just say what we're doing a little bit here. So what we did was we cloned that GitHub directory, which had a lot of our necessary files. And then we installed in our virtual environment all the different toolboxes and libraries that we're going to use in our training analysis. And you'll know that it's done because we'll have... We'll show a little ready sign at the bottom. So once you see that, you've downloaded and you've installed all the necessary libraries for the exercise we're going to do. So once we have the libraries installed, we also have to import them. So we've installed in our virtual environment all these different things like NumPy, Matplotlib, Seaborn, Rasteria, which are all just basically ways of you're handling data sets or plotting data sets or visualizing data sets. But now we just need to import them as tools or commands that we can then use to actually query our data. So we'll go ahead and just run that cell again. And you'll see once that stops spinning that it's done. So now we've installed all our libraries. We've imported all our libraries. And now we have to actually pull in all the data that we're going to use in this exercise. So one of the disadvantages of GitHub is that you can't have large file sizes. And if anyone's ever worked with LiDAR data before, you know that even small spatial areas can be pretty large. And so what I've done here in the cell is just created a little downloader that will pull from my personal Google Drive all the geo tips of different topographic data sets that we're going to use in our analysis today. So go ahead and click play there. And you'll start seeing them in the bottom here at the output, the script that says that you're downloading that data and then it done when it's done downloading. And while we're doing that, we can actually see where these things are going. If you open the file little file tab to the left, you can see we have our CSDMS clinic 2020 folder. We can see data. We can see the San Luis because we're going to work in the San Luis mountains. And then we can see all the data sets that we've either imported from GitHub or a downloading from Google Drive actually populated right here. And so this is pretty fun because it also allows you then to actually take any of the output that we're running in co lab, even though it's fully virtual and transfer it to your own Google Drive or just download it to your home PC if you want to save any of these outputs or continue playing with them. Okay, so we've downloaded all of the necessary DEMs. Now we just need to set our data directory for our future analysis. And you see you can see then we've just printed the what's in that directory, which are the same files that we're just looking at the left hand side here, which are again art of the DEMs that we're doing with our analysis. So we can sell our libraries, we've imported them, we've imported all the necessary data sets for our workflows today. And let's just have a look at where we're going to be working. As I mentioned, it's going to be in the San Luis mountains, which if you're not familiar, are down here. So if we exercise this cell, I've just put it in a little raster, a little folio module that just lets us have a kind of interactive environment for looking at exploring an area just north of Taos here. And so we zoom in here, you can see the Rio Grande River to the left. And you can see that we're going to be investigating this area of mountainous watersheds transferring into the Lugia Valley over to the left. So just kind of a fun little display. This is just kind of barely scratching the surface of the power of things like folium and some of the other visualization techniques, but I want just to kind of showcase their utility. Okay. Pelle, can you still hear me okay? I'm getting a low system resource message that my audio quality might suffer. Good? Yeah, it's, it's, I was hearing a little bit of noise, but it's still coming across clear. So no issues. Excellent. John, you might want to see if you can make your screen a little bigger. Yeah, that's right. It's true, depending on the fonts a little bit bigger. Yeah, just zoom in, John, like we did that time. Yeah, I forgot. Depending on your monitor size. That's true. That's is that better for everyone? Yeah, that looks better for me, at least on my screen. Anybody else with issues with the font or these looks good, you can give us thumbs up or whatever. Yeah, with things like that, everyone, please let me know. So I can correct if something could be better or, you know, we have any issues we need to work out. Okay, I'm going to read the font size. Okay, everyone can kind of see the outputs. Okay. I think, I think the situation silence is good. Okay, so we've had some fun looking at where we are. We're in the San Luis mountains here just east of the Rio Grande in New Mexico. So let's then start actually loading and plotting our different raster data sets. So here we're going to import white box tools for the first time, and we're going to set its working directory, and then we're actually going to start opening up our different data rasters. So let's run that cell to download white box. We're installing it, and then we're just, again, setting its data directory to that same directory that we had put all our DEMs in. So we're going to now look at what those DEMs actually look like. So we saw from the from the fully in plot that we're going to be looking at a mountainous area, but one that's kind of transitioning to a relatively flatter terrain as we move towards the Rio Grande. So let's see how those look using both a one meter DEM as well as a 30 meter SRTM DEM. So let's first load or open one of our DEMs. So this is just a routine that's using a rasterio to open a geotiff DEM. So a DEM that exists in a TIFF format, which is a very common format, and we're reading it into a NumPy array. And if you're not familiar with Python or NumPy, basically this is just letting us treat our raster as a numeric array, which allows us to kind of perform computations on it in a really efficient way. So we can see that we've opened our high resolution topography DEM, which was originally as a TIFF and now it's an array, and we can see the size of the array there. So this is essentially the pixel extent. We'll do the same thing with our 30 meter SRTM digital service model. So you can see that even though the extents of the SRTM data and the LIDAR data are exactly the same spatially, the number of cells associated with each array are drastically different because the resolutions are very different, right? We have a one meter by one meter DEM from the LIDAR, and we have a 30 meter by 30 meter DEM for the SRTM. So let's now plot both our LIDAR and our SRTM data sets and see how they look for this region. So plotting here, plotting can sometimes take a little bit of time depending on the size of the data set, but you should be left then, within the left, a DEM from our one meter LIDAR and a DEM from our 30 meter SRTM. So DEMs without a hill shade can be a little hard to kind of see detailed structure on as some of you may know, but it's interesting that between these two, they look very similar. Again, the 30 meter might be a little fuzzier, a little hazier compared to the one meter, but overall the kind of gross structure of the landscape looks pretty similar between these two data sets. So let's see then what the actual distribution of elevations of these data sets look like. Is there a difference between the LIDAR and the SRTM data? So what we're going to do now is we're going to take those arrays that we read in from our TIFFs and we're just going to plot histograms of them. So we're going to just look at the distribution of elevation data across this spatial extent. So on the top here, you can see the DEM distribution from the LIDAR and in the bottom you can see the SRTM. So you can see that despite the difference in resolution, despite the kind of platform of acquisition, despite the kind of data quality, the overall structure of the landscape between these two is almost identical. We're kind of capturing the broad distribution in a very, very similar way despite the differences in where these data are from. So that's kind of interesting. Keep that in mind for when we move into the next section here. So next we're going to actually start taking those DEMs or those rasters and pouring calculations on them. And the first thing that we want to do is just create a hillshade to help our visualization. As I mentioned and as you saw above, DEMs without a hillshade kind of don't look like very much. And so we just want to better visualize that. Now, if you're not familiar with what a hillshade is, we're essentially going to basically specify an azimuth and angle of illumination that basically will create artificial shading and shadows that let us better visualize what's high, what's low, and what's steep relief and so on. So let's then calculate using white box some hillshades for each of these DEMs. So this cell here that we've executed has created a hillshade for our lidar data and a hillshade for our SRTM data and then read both of those back into arrays again to allow for easier plotting. So now that we've calculated our hillshade rasters, let's visualize them the same way that we did our DEMs. Okay, now we can see some pretty significant differences, right? The DEMs by themselves didn't really look that distinctive, but now that we've applied a hillshade, we can see that the lidar is capturing a lot more of that small scale detail and variation within the landscape as compared to the SRTM data. But that also then makes it interesting that the distributions of elevations were so similar. So let's zoom in then on a smaller portion of this landscape and do some more calculations and attribute analysis. Again, though, I just want to emphasize that the overall structure between these is really similar, which is though surprising, the cell size from the lidar is 1,900th what the SRTM data is. But if we zoom in there, we can see what some of the implications of that kind of smaller scale cell size is, where if we look at the headwaters of our small catchment, the SRTM data and the lidar data are both kind of capturing the broad structure in terms of where our ridge positions are, where we have highs and where we have lows. But we're missing a lot of really, really important detail about the channel network in these headwaters. We're missing important details about the transition between channel processes and hill slope processes. We're missing detail about potential mass movements and failures. And that's really what the power of hybrid systemography is, is letting you understand and kind of characterize and quantify those sorts of smaller scale processes, even if the bulk structure of a landscape is fairly consistent. So let's then subset that larger DEM into a smaller area that will be able to use and run some more kind of computationally expensive algorithms on. We have to do this just to conserve RAM, because we don't actually get that much of it from Google Colab. So this was our kind of original study extent. And we're going to kind of zoom in on the area of this red box here and use that for all our future calculations and work in this exercise. So let's create some hill shades for the lidar and the SRTM data from this smaller region. So we're creating the hill shades again with white box tools. So now you can see in that smaller region just what the structure of the smaller catchment looks like. So again, we can see the kind of alluvial plane that's being dissected by all these gullies in streams coming out of the mountains. And again, over to the west would be the Rio Grande River, which all these are flowing into. Okay, so let's then start playing with some of the more advanced things that we can do with raster data and high resolution topography using tools like white box tools. So one of the kind of fun things to do with topography data, especially in kind of really dissected or heavily channelized environments is detrending DEMs to look at the kind of relative topography or relative relief. And so if any of you have ever seen some of the really kind of cool images or visualizations of river valleys or manning river systems that have been detrended so that the point bars and the channels really kind of pop, this is what this technique is. And I want to showcase and show you how actually easy it is to do on your own. And it's just kind of fun to play with. So the first thing that we're going to do is we're going to detrend to DEM and calculate some DEM derivatives. So again, detrending is essentially fitting a trend surface through some sparse sampling of a digital elevation model, creating a new surface from that and then subtracting the two surfaces. So in this example here, if we have a kind of sloped environment like this, we would sample what that plane is. And then by subtracting out that plane from the original data, we would flatten it out by whatever that trend was, which will allow us to better kind of visualize what was positive, what was negative versus that original plane. So let's try that on our dataset by creating a second order polynomial surface with zero being a flat plane for a one meter LIDAR DEM. So we're going to use a white box tool to build that second order polynomial. And then we're going to read that back into an array and then use those arrays to kind of subtract everything out again using white box tools. I also just want to kind of emphasize here the power of doing a lot of this analysis and a lot of these workflows in something like Python is that it doesn't have as much overhead as using things within like QJS or ArcGIS would. So for instance here, we just created a polynomial trend surface sampling from a DEM and then writing that all out in about 19 seconds, where the same operation in ArcGIS could take two to five minutes. Let's then subtract that trend surface that we just created from our original DEM, done very quickly again, and then let's now plot our original DEM, the hillshade, the trend surface that we created and what the resulting detrended DEM actually looks like. The plotting actually takes longer than the computation itself, which is kind of funny. Okay, cool. So again, we have our original DEM here. We have the hillshade. We have the trend surface or that second order polynomial that we created. And then we have the detrended DEM, which again is the subtraction of this trend surface from the DEM. So you can see then in terms of this kind of residual topography removing out the overall kind of slope of the landscape, we have pretty significant differences from the western side of our study area to the eastern side, which we might have guessed from the hillshade, right? So in the west, overall, things are a lot lower elevation. We have more large flat areas. We have more kind of just deep dissection of a relatively plainer surface. Whereas to the east, we have much more rugged terrain, much higher highs and much lower lows. And again, this corresponds with coming out of the fringes of the San Luis Mountains into the Rio Grande Valley itself. So let's then play with some of the other things that we can kind of do very quickly. So here, we're going to calculate again, using white box tools, we're going to calculate slope, curvature and aspect from our one meter DEM. Now, one of the things that we often want to do with rasters and with high resolution topography is actually look at not just the kind of elevation structure, but kind of landscape characteristics or attributes, such as which direction are different portions of the landscape facing? How steep are they? What's the curvature of them? Because this information helps us understand things like surface roughness. It helps us understand potential channelization, potential hill slope processes, things like that. So let's then plot those different aspects or those different characteristics of the landscape that we just calculated and kind of see what they actually look like for a very high resolution DEM of a mountainous area. So now we're actually seeing some color, right? So we have our hill shade again in the upper left just showing us the kind of structure of the landscape. You can see the aspect here showing us kind of the different facing slopes in very, very graphic detail. We have our slope and then we have what's the derivative of slope of curvature. Now we'll get to curvature later in the channel analysis portion, but what's really important to point out with all of these is that while we are able to see kind of very high definition structure of the landscape through things like hill shades, if these attributes which are going to help us actually move the needle in terms of auditing some of our terrain classifications are really understanding kind of the evolution of connectivity in these landscapes. And to kind of make that point, let's load in our original SRTM data set and see how attributes of that compare. Now if you remember from our just from looking at the DEM, the overall structure of the landscape in terms of elevation between the SRTM data and the LIDAR data set are almost identical, right? But now when we start looking at some of these kind of different metrics or parameters of the landscape such as aspect or slope or curvature, these are all plotted on the same exact scale, you see really, really big differences start to emerge because now we're running into the issues of both how the data is acquired, but also things like the actual just cell size and resolution of features that we're able to see. So let's compare the slope and aspect from the LIDAR and the SRTM data side by side to really kind of drive that point out. Well this is running, John. I was going to check this, everybody following along, you're all seeing these images. If you don't just let us know or if you want John to move slower or move faster, we'll do this for a bunch more time so it would be good to get a sense if this is a good speed or no. Yeah, please feel free to ask any questions or if you have any comments. Again, we're a small enough group that we can have a little more informally. Oh good, excellent. Awesome, thanks for the feedback. What's the scale on the plots? The length scales? Oh, I'm sorry. So the domain that we're looking at here is roughly 3,500 meters by 6,500 meters. Sorry, I should have plotted a length far here. Okay, are those just pixel numbers on the plot? Yes, exactly. So because we're plotting, because we're plotting numpy arrays, they're just pixel numbers. So they're kind of dimensionless like that, which is why for the same spatial area and the same size, we have very different scales and that's again because in the SRTM, each pixel is 30 meters by 30 meters, whereas the LIDAR is one by one. So the LIDAR, since it's one by one, LIDAR is in straight meters. Exactly, exactly. Right. And so you can see then comparing these that again, the overall structure was very similar between them, but the detail of the landscape in terms of a lot of these parameters like slope, it's starting to get really, really big deviations in terms of like the distribution of slopes across the landscape or even how we can kind of characterize the aspect. This starts to matter when we're trying to do kind of larger scale analyses of landscapes, where we might only have things like SRTM or more global datasets, because we might then kind of fool ourselves and think that we can more accurately characterize a lot of these parameters that we might actually be able to. And I'll show an example of that right now. So what we're going to do is we're actually then going to quantify how our landscape looks in terms of something like slope and compare LIDAR to SRTM. So again, we're going to plot our slope raster into a NumPy array, and we're going to do the same thing for SRTM. And the reason why I'm doing this, by the way, before we plot these diagrams is because all rasters in GeoFit format, or most of them, have no data values, and a no data value needs to be converted into a not a number value and a NumPy array to allow you actually to make a histogram out of it. So that's the only reason we're doing this intermediate step here. So now that we've kind of created our NumPy arrays from LIDAR and SRTM, let's plot the slope distributions for both of these. And remember, the elevation histograms were identical, but the slope distributions for these are extremely different. We all have that peak around five five degrees slope, but the SRTM or the lower resolution data is completely missing the second modality about 30 to 35 meters, or 30 to 35 degrees. It's just absent within the SRTM data. This matters a lot for again, the characterization of a lot of things like the divide between whole slope and channel processes or where we might potentially have channelization in the landscape or surface roughness. The ability to kind of better characterize the actual kind of distribution of the landscape and characterize a lot of these kind of relatively small scale features that are spatially small scale but are really important for potentially driving surface processes dynamics. And so I just want to kind of emphasize that point through here. All right. So now that we've kind of played with rasters a little bit and played with how we can kind of visualize some data, how we can actually start quantifying different measures of the landscape, we're going to jump back into the presentation. But before I do that, does anyone have any questions about kind of what we've gone through so far or what we're seeing here or what kind of difference between a high resolution data and the lower resolution data? Cool. All right then. Let's hop back in for just a little bit of background on kind of network processes. And then we're going to actually play with that within the co-lab. Yeah, go ahead, Nicole. So this is super cool. And this would be super great for teaching. And I'm wondering if I'm guessing this is going to go away after this clinic because you don't know. It's going to be free and publicly available forever. Oh, okay. So even though the data are sitting in your own Google Drive, you're going to leave them there for us? Yes. Well, that's very generous of you. Thank you. Sorry, I'm just going to put on my CSDMS half for a second. You should think about including these in the CSDMS, the educational part where they have the different labs. I think this would be super helpful if it's not there already. That's a great idea, Nicole. Great idea. Thank you. And Nicole will also post these on open topography so they're accessible there as well. Oh, perfect. Yeah, that's great. Excellent. Thank you. Cool. Any other questions or comments? That's a great idea about the repository. We'll make sure to do that. Yeah, it is. John, I was going to say, I think you're all used perfectly fine. When you're running some of the heavier operations, it gets a little bit more affected. You can always stop your video. I think it's fine. I can hear perfectly well, but there's always the option of the video. Great. Alrighty. So then we'll just kind of keep pressing ahead because we got a few more kind of really exciting things to get through, especially in the co-lab. But the other part of this clinic and one of the implications of the ability to better characterize the landscape that I want to focus on because it's an interest of mine, it's an interest of policy, it's an interest of our research group, and I suppose it's an interest of all of you, since you took the time out of your day to come here, is how we can use these tools and techniques to better assess and understand things like landscape connectivity. So what we do is we focus on hydrologic connectivity and how hydrologic connectivity controls processes. So in general terms, hydrologic connectivity is the framework that sediments, solutes, and biota can move through and influence or be influenced by landscape function development. And it's a really interesting question for a lot of different people, but how do these different kind of domains interact and on what spatial and temporal type of scale? So for instance, how does a river couple to a floodplain? How do different habitats grow or decay through time? What are the kind of patterns of these of these changes? And so there's a diagram right here by Ellen Wol, that I have always found very, very helpful for kind of thinking about, but essentially we can look at inquiry connectivity by trying to understand if we have certain boundary conditions, such as network configuration, geometry of the valley or geometry of the channel, and we vary an input through it, such as the amount of sediment that's being generated in these channels, how much precipitation is raining in a watershed? What are the outputs that we generate over time? How much sediment do we get out of that kind of those boundary conditions? How much water do we get out? What is the pattern of where that sediment and water is going? We can use kind of this connectivity framework to ask a lot of questions on either geomorphology or sediment transport or ecological service. So what influences connectivity? I mean, that's a really, really loaded question, right? Because the answer is tons of stuff does. Here I've just kind of plotted some different ways of thinking about connectivity that have been used in the past. So for instance, again, here we have a plot by Ellen Wol looking at how differences in stream gradient in a mountain catchment might lead to kind of hydrologic disconnection or lack of connectivity amongst different stream segments because of things like waterfalls, things like dams. On the right, we have things like process coupling or process connectivity. So this is work by Alicia Czendrowski looking at how different processes that are active on distributary delta system influence and feedback into each other. So how does depth, tide, and wind influence things like temperature or nitrates or turbidity, how does flow modulate all of these? We can look at all these different processes and understand their interaction with each other just the same way that we can look at these network dynamics and think about how the net changes in the network might influence connectivity. So what we're going to focus on is looking at tools or techniques of characterizing landscaping and help us better understand things like hydrologic connectivity, which is one of the main ways that surface processes and ecological systems are mediated or have feedbacks between each other. So again, just two examples of these processes in action are some work by Matt Hyatt looking at, for instance, channel island processes or exchange in delta systems. So when we have water and sediment coming through a main channel, how does it actually get on to the delta islands in the levees in a way that's necessary to start building land and allowing vegetation to establish itself? And vice versa, how does sediment and nutrients on these islands get back into the channel? Or in another system using a totally different technique that we didn't talk about yet called an interferometric synthetic aperture radar, how can we look at the evolution of water levels in wetlands as a function of discharge in the main channel as shown in these dash lines here? So how with time is water getting from that channel into the wetlands and bringing necessary fluids and solutes that these ecosystems need to sustain themselves? What we're going to focus on today and what's a pretty significant researchers of mine is the network structure across different domains. And so when I speak about domains, I want to point out that we often kind of split and characterize the landscape into things like dendritic upstream catchments, so mountain catchments like the ones that we've been studying in New Mexico, the kind of transfer zone of these meandering or braided streams, and then a final distributary such like these delta systems here. But across all these different domains, they all have a network structure that is both being acted upon but also acting to modulate the flow of material within these landscapes. So a very kind of typical channel network extraction or workflow that we all I'm sure most of us are familiar with and it's the one that you'd be taught in an RGS tutorial is something like if you have a DEM, you can create a channel network or a stream network in watershed by doing something like taking a DEM, hydrologically connecting it, correcting it to remove pits and little depressions, running a steepest descent flow algorithm over it to actually track how many upstream cells are contributing to it, and then getting things like these channel networks in blue, which relate roughly to where your stream network is, and then also maybe extracting out things like in yellow, your main trunk streams. Now this is a really great technique and it's super helpful because it can also then let you do things like do more kind of quantitative geomorphology in the landscape. So looking at, for instance, within one of these stream networks, what is the actual kind of profile of relief of the main stream itself versus the bounding slopes? How do these kind of different systems evolve? Like what is the kind of diffusivity processes that are modulating what the bounding ridge elevation is versus the erosional processes that are carving out and forming this convex profile of the stream network? Networks though are dynamic in space and time, right? So in that earlier plot by Ellen, we had channel networks as being kind of a fixed geometry, but that's not necessarily true. And the scale in which they can change is actually pretty interesting. So on the left here, we have just an example of looking at the dynamics of channel networks and channel heads pre and post wildfires. So looking at very, very rapid change in the landscape structure in response to a specific forcing event. So using high resolution topography to look at how the position or even occurrence of upstream portions of this channel network in these mountains changed because of the modification of the landscape post fire. On a longer time scale, you can see the same processes play out, although due to very different reasons in terms of geologic scale divide migration and river capture. So again, similar to some of that work that we saw in the keynotes this morning, these networks that we often kind of think about as being constants do change on both really rapid human time scales as well as geologic time scales. And the changing nature of these networks has pretty significant implications for a number of processes and number of disciplines such as ecology. Okay. So one of the things that we're going to look at today is things like channel floodplain connectivity. So this is just a nice little diagram showing a kind of, you know, conceptual schematic of how for a long given river reach certain areas of the river with a given discharge condition might start losing material and water to the floodplain. In other areas, they might start gaining water and material from the floodplain. And by better characterizing what is controlling these inputs and these outputs, we might be able to better understand the kind of dynamics of things like channel floodplain connectivity on human and geologic time scales. So we're going to actually play with a version of this model later on in the clinic, but we can look and use, we can look at and use both high resolution topography with numerical modeling to look at how things like small scale topography drive things like that channel floodplain connectivity. So here's work from our colleague and groupmate Shazad Rahman here at UT Austin doing very detailed numerical hydrodynamic modeling of the kind of fluxes in and out of the main river channel of the Trinity relative to the floodplain using knowledge gain from understanding of the small scale topography as well as a really kind of cool numerical modeling. We're going to actually run this model for ourselves in a very different domain, but I just want to kind of showcase the potential for using these different tools together to answer some of these questions about connectivity. But one of the first steps before we can kind of get to that level of numerical modeling or testing some of these hypotheses is we have to just kind of characterize the landscape and see what we have in terms of network structure. So as I mentioned, one of the most common ways of doing this is through that kind of steepest descent flow routing. And I just have plotted an example of here of how that often can lead us astray particularly in relatively flat loan environments such as deltas or within alluvial valleys. So you can see here that kind of classic channel network delineation flow line versus where the river channel itself actually is. Additionally on the right here we can see examples of how man-made objects such as drainage ditches and erodes can lead to really spurious results in terms of actually kind of delineating or tracing where our channel network actually is. Now these are really important because we want to do really accurate characterization of what processes are active in the channel network right now as well as how it might change in the future. We have to understand where our channel network actually is relative to our overall landscape. So there are other ways of doing this and so this is a lot of the work that Palo's group has focused on and what we can do is we can use things like curvature-based approaches such as GeoNet which is another open-source toolset that we have available for the community to use. And what this does instead of that steepest descent flow accumulation is it uses a curvature to essentially threshold different types of objects within the landscape and allows you to do more of an automated approach to landscape analysis. So for instance here we have some synthetic topography that's showing us the height here then we have Laplacian curvature and geometric curvature. Now the implications of that are different types of features such as a natural stream channel versus a road versus a drainage ditch can have very different curvature signals that allow us to then filter our elevation data if we have high enough resolution topography to actually differentiate between what's a natural channel, what's a road, and even what's a natural channel versus what's a drainage ditch. And so it allows us to better kind of characterize the natural channel network of our landscape without running into errors associated with some of these kind of artificial features. So just as an example of that here we can see just a QQ plot of geometric curvature showing that on different sides of that of that distribution we have where we have likely channelized regions and where we have likely valley regions and where we have more hill slope regions and then we can use that to kind of classify within the landscape where we have likely channels and here's just the comparison of where we would predict channels to be versus where was the field survey location of those channels as well as where those channel heads are actually present in the landscape and see that by doing this more curvature-based approach we can actually get a pretty good idea of what the landscape structure actually looks like relative to kind of classical low flow accumulation techniques. Okay so that said we're going to go back to the co-lab so let me just kind of stop here and transfer back over to github. Okay can everyone still see that now? Yep they're all back. Excellent okay so hopefully then nothing happened in the meantime where you weren't disconnected or you didn't close the notebook or anything like that. If you did that's fine we can just rerun everything. We're now going to actually look at running some of those sorts of schemes and some of this kind of channel network analysis on the dataset that we've been playing with previously. So as I mentioned before the kind of classical approach to channel network analysis is the steepest descent flow accumulation routing and so these are just some examples of how those algorithms work but basically you rely on using a flow direction associated with calculated from the relative elevations of cells in a matrix to calculate where flow would go to the next cell and then propagating that through your model and so let's kind of experiment with some of these kind of classical approaches and see how they look within our landscape and compare our high resolution data to our low resolution SRTM data. So let's first calculate a flow accumulation raster but using both this classic d8 model which means that you're looking at the eight cells surrounding a pixel as well as a de-infinity model which allows you to kind of differentiate or have flow going across cell boundaries and so we're going to calculate both of those techniques and we're going to visualize them and see kind of what the differences are and then we're going to show what these look like and compare to a geonet based curvature approach just to kind of highlight what these kind of different things get you especially by using high resolution topography. So we're running the cell here which is again using white box tools to calculate all these different kind of derivative rasters such as flow accumulation. So we've now calculated two different types of flow accumulation as well as smoothing a raster to create a hydraulic hydrologically connect a conditioned raster and we've done it all in 36 seconds which is pretty remarkable again kind of the power of using some of these more kind of lightweight efficient processing tools. We're going to do the same for SRTM data and now let's just like look and see what some of these flow this flow accumulation or flow routing schemes look like on our DEM. So we're going to plot now the DEA and the de-infinity results of flow a flow flow routing for the one meter. So again we have our original DEM here we have our hill shade we have the DEA and the de-infinity flow accumulation. So you can see that for the one meter data both of these actually look very very similar and we'll look at them in a little more detail because there actually are differences but overall both do a fairly good job but what we've generated here is basically a stream network or a channel network of likely channelized areas based on the accumulation of upstream cells flowing to them and so we can start seeing that and querying the actual kind of network and landscape structure here. So let's do the same for SRTM and see how the SRTM looks and compare it to the lidar. Okay so again original DEM here's our hill shade here's the D8 flow accumulation and here's the de-infinity flow accumulation. Again at this scale both look pretty similar but I think you'll start seeing that if we compare the lidar and the SRTM pretty significant differences right. Again as you'd expect based on both the cell size and also how we're not quite capturing a lot of these kind of small scale characteristics especially in the head of water regions. So now we can see this is using the same accumulation thresholds by the way what the SRTM looks like versus what the lidar looks like for both of these methods and you can see that in general the structure looks pretty good in a broad swath right. If you just wanted to understand where a drainage basin was again on a large portion of the landscape using the slow resolution data would do a fairly good job but where it's going to miss a lot of really important features especially for a lot of the problems that we care about as people in the Earth's surface processes is what's going on in a lot of these kind of smaller head water regions as well as the dynamics associated with the transitions between hill slopes and channel divides. So we now have our flow cumulation rasters but we can then take those and actually extract the streams out of them using some sort of threshold. So again this is just measuring how many cells are flowing each next cell along a stream profile. Here I'm defining saying a threshold of a thousand cells what counts as a stream and what doesn't count as a stream. So I've run those and now I've kind of made some pre some predetermined plots just in the interest of time and computation here for us to look at along a small region of the up slope area. So here we have our hill slope of the one meter DEM and here is the result of the stream routing scheme using D infinity on that with a thousand cell threshold. So again you can see that overall is doing actually quite a good job right we can see the stream looking up here we can see these channels going up towards the divides we can see where they kind of peter out close to where the channel heads would be and that kind of transition between the channel and hill slope processes if we look at D8 you can see that it's compared to the D infinity giving results that don't look as realistic or have potentially some spurious things we have a lot more linear linear features being shown here we have a lot more kind of cross cutting things again that's because compared to D infinity D8 has a limited range of areas that I can say a flow line is going whereas the infinity has an infinity number of regions which is why that kind of gives you more realistic results if we plot the two of them together again especially given the fact that we're looking at one meter DEM in a really small region of a headwater catchment overall they're pretty much in agreement with each other but the D infinity is giving us a little bit more realistic results let's see what the SRTM looks like on top of that so I plotted here in black the SRTM flow accumulation and yellow is from the LIDAR but our background image is the SRTM data and so again overall it's not actually that bad right we're capturing in general where the divides are we're capturing in general where our stream is located but what we're missing are kind of the all these smaller scale spatial details in terms of like well how much of the stream is where where are the kind of transitions between the hill slope and the channels and also there's an interesting discussion that we're not going to get into here about how at coarser resolutions you would actually calculate the length of your streams to be a lot smaller than they actually are in real life but again almost just entirely due to the data resolution okay so what about that lower relief overview of valley so if you remember in the previous exercise we showed with our detrended DEM that in the eastern side of our domain we have the really kind of steep and rugged mountains and they transition to a relatively flatter landscape that's being dissected as it drains into the Rio Grande so what does that mean in terms of flow accumulation a lot of data sets and a lot of flow accumulation schemes and channel delineation schemes they all perform pretty well in mountainous areas where you have a relatively good difference between what's a high and what's a low but they start breaking down when you move to relatively lower relief flatter settings and so you can see the result of that here so again in black we have our d8 channel network in yellow we have our d infinity but you can see significant difference like deviations in terms of where we expect channelization to be occurring and where our streams are actually going associated with things like this curve curve feature here which is actually a road so you can see that both of these methods are in some cases picking up where the actual natural streams are very well in other cases they're simply following along these man-made features here and so this is one of the biggest challenges with doing channel delineation channel network extraction in relatively low relief flat settings and it only gets worse actually in areas like Texas and the Gulf Coast which is often where we need this sort of data and the sort of accurate characterization the most so that's where curvature and automated feature extraction can help now geonet is a little too computationally expensive to run within the colab right now so i've run these previously or rather our group may out did and i'm just going to show you what the results of the geonet workflow are and how they compare to the ones that we've calculated here but again geonet is completely open source and is able to be used by anyone so if you're interested in it please email one of us and we can send you some literature as well as link to the github for you to play with yourself so we can use things like geonet and uh these hybrid curvature approaches to see if we can better characterize these lower relief landscapes so let's return to the the headwaters first and so again we have our hillshade here and now we have the curvature attribute that we calculated previously of our of our one meter dm and so you can see now that where we have these these high curvature values shown here in yellow uh in white rather are what we're going to be able to threshold and classify as likely channelized regions so we have our hillshade we have our curvature here's our da again in black and d infinity and blue and then in yellow here we have our likely channel skeletons from the geonet approach you can see that between these is actually some pretty big differences but you can also see that in this kind of quick and dirty first order pass we have disconnections between where we're kind of classifying a channel because geonet and these curvature based thresholding approaches don't rely on continuity from an upstream area they're relying on thresholding of what these curvature maps look like and then we're modifying them based on the flow accumulation rasters that we're creating but they don't necessarily have to be continuous because they're not relying on the accumulation of upstream cell area which makes them a lot more useful in things like these flat areas so if we go back out into the alluvial plane it by the Rio Grande again here we have the hillshade here and we have curvature and then we have da and geonet so you can see that geonet is not really being fooled by the roads in this region instead it's mostly picking up and classifying as likely channel network just the natural channels itself and again that's because we can set different thresholds of curvature that allow us to differentiate between a natural stream network and a man-made feature such as a road where we have a few areas where maybe it is picking up on some things but again this is the first pass and we're doing this for real we do a lot more kind of refinement and parameterization to better tune that out but just in this kind of quick and dirty scheme we can see that we're getting really different results which lead to a pretty different view of the landscape especially when we want to do things like query what kind of different process domains are actually active on this channel network so that's just kind of little example of geonet I'm sorry you can't run it yourself in the cold it definitely suggests you play with it and kind of see what it can do for yourself so now that we have an accurate channel network we can do a lot of different things with it right depending on the sort of question that we want to ask so I just had a fun little example here of showing what we can do if we have a stream network and a DEM together and so one thing that you may have heard about is a technique called a height above nearest range which is basically a way of saying if you have a landscape and you have areas that are drainages or streams what how high is the rest of the landscape relative to the nearest stream or drainage so it's a way of basically doing rapid prediction of potential flooding and potential inundation and so we're going to actually do using white box tools a very a very rough very simple version of hand modeling because we have now an accurate channel network and we have an accurate DEM and we can just kind of see what some of the routes are going to get this is just more for fun but it's kind of showcasing what you can do once you have this sort of data in hand and once you have a decent visualization or representation of the channel network with the landscape so what I just did here was I create use white box tools to create my elevation above the stream and now I can see here a lot of our stream network ooh that didn't that didn't plot as well as I'd like to we can cut down the cut down the vertical max here yeah so you can start seeing emerge here as you kind of like tune it down I keep doing that to myself but essentially like we we basically just really really rapidly built a raster that shows the height of every single portion of the landscape above the nearest portion of that stream network and we did it all in just about 10 or 15 seconds of calculation again just something just for fun but just kind of showcasing what we can do once we have kind of this sort of data ah there we go so again we have the stream network here and now we have our kind of height above nearest stream raster again showing the kind of big divide between relatively rugged regions of the mountains and the kind of more low relief areas we're getting closer to the Rio Grande but again this has a lot of implications then right in terms of like what sort of process domains there might be as well as kind of potential connectivity of the landscape where you have less of impedance of getting out with one stream to the next okay so that's that for part one of the work of the hands-on portion we're going to go back for just a little bit of an overview of numerical modeling and an introduction to Inuga and then we're just going to play with numerical models for the rest of the day so before I hop back into the presentation does anyone have any questions or comments I see Paolo's having a good discussion no no I was gonna yeah no we yeah I was answering some questions here in the background about the the disconnected channels you can get them connected so there's an additional module in in Geonet that is based on geodesics that applies cost minimization for connecting channels which has the advantage with respect to steepest descent that if you have disruptions those are not going to stop you because it's a global minimization but this kind of thing is very interesting because you can appreciate where there's disruptions which might be roads maybe landslides and and so forth when you calculate the height above nearest drainage how are you interpolating in between and and past the channels so the the height above nearest drainage is computed from the the centerline to which you drain so it's your relative height the relative height of each pixel with respect to the centerline of the channel that according to the flow directions you're draining too does it make sense Mike of the one you're draining too okay yeah the one you're draining too which is where channel density is very important we use hands as a as a way to do kind of quick and dirty inundation mapping in a sense kind of fast simple computationally efficient for firefighters and that's where the density plays a big role because of course if you have coarser resolution you're just draining to those main ones and you're kind of missing the smaller scale channels that might flood first but yeah it's a very easy calculation you just do flow directions you see where you're draining and then you get that relative height and it will give you a first sense of where things might flood first john you're doing awesome you need a glass of water a little coffee I had a quick question this is abby here so you said you could differentiate between natural channels and things like ditches so what I that made me think about all the man-made navigation channels and then the Mississippi Delta I was curious if you'd be able to differentiate those from like the natural distributaries using this yes absolutely and actually that's a that's an object of interest of one of our students in palace group who's actively working on that right now as well as looking at historical maps of what the delta looked like before those channels were actually created in the first place yeah I will also say that any landscape like the one you're describing abby they're very challenging anytime you put artificial features gets very hard because a geonet is really based on that curvature distribution and so you know one thing is the natural landscape in which we can interpret these shifts in curvature as a signature of natural processes but once you put artificial features in it they go into the same pdf so you're kind of losing the signature of the natural processes and so if you have a landscape where the artificial features are really dominating it's going to be much harder to interpret that pdf that threshold in the probability density function as a signature of natural processes so we do have ways of doing it I would say that we also fail in various locations and it would really depend on how strong the the signature of the natural processes still is urban landscapes and flat landscapes are definitely much harder than a nice steep water sure yeah and then that's a really good point I mean because I showcased it a little bit when we when we did when we were looking at the kind of like flatter western portion of the San Luis landscape but um that's I mean that's still amazing relative relief compared to an area like the Gulf Coast which you know coming from Louisiana you're right right the other yeah go ahead John sorry no no it's just it's it often when you're when you're trying to do this or doing projects involved kind of actually need to do things like that you want to kind of try to automate it and do things like feature extraction as much as possible but there's also always just an element of you need to look at your landscape and see what what people have been doing to it right the the other thing that was going to bring up is that another thing that some of these tools like that Geonet can do is that you actually don't need to fill the pits which you know from an hydrological perspective the pressures are actually really really really important and so lots of these methods will require that you fill in all those impressions but we don't actually need to do that because the geodesics will just go through that which is which is nice there was a question from Nicole as to whether we what was it oh if we stop the notebooks that we just run and we just close the tab you just close the you can just close the tab we're gonna we're gonna open the next notebook in a little bit and then I'll start a new kernel entirely young go ahead yes please my name is younger from Florida and I really confused me the terms and you mentioned hydrological connectivity but hydrologic connectivity is more like more dynamic like it can change within a single storm right yes I mean the connectivity you mentioned is more like it's close to the like a structural connectivity or that is correct you are absolutely correct that is correct yeah yeah and so what I'm showcasing in part one of the clinic here is the framework that we can use to observe what potential structural connectivity is in the next step we're going to look at potential hydrological connectivity through an American model so I'm just showcasing that we can use these techniques in this data to get a handle on what the kind of boundary conditions or the structure or landscape is but you're absolutely right the hydrological connectivity will vary immensely depending on things like discharge, precipitation, tides, sea level all that sort of stuff which is more the process-based or functional yeah and I think and that brings up actually another interesting yeah it's a very good point and it brings up another thing that is that is interesting to note that apart from the main channels which you know they're gonna carry water at all times what you know what's something like geonet or when you get to LiDAR what you're seeing is the signature of this section so it doesn't necessarily mean that those pathways are actually connecting water that would be activated during large events but they're not getting water at all times which is really feeding into this idea of structural connectivity in terms of having that path which may or may not be active at all times right right right right absolutely yeah I'm just showcasing the use of these tools and workflows to give us an idea of what structural connectivity the landscape is as a way then of querying and building an understanding of hydrologic connectivity which is what we do a lot of our work on and then one more quick comment or question and I agree you mentioned that you are going to use the hydrologic model to reproduce the processes right I mean the connectivity processes is it true to examine I mean you are using model right yeah yeah right and then I agree so I believe the modeling seems like the only way to reproduce the hydrological processes and investigate the connectivity dynamics right yeah some some papers mentioned about the field scale plus scale connectivity they they made some observations and it looks like very expensive in that map yeah I think it depends on the application the the network that John was showing the the paper with my former PhD student Alicia that was actually a time series that was all time series connected in the you know collected in the field over three months but you're right there's you can do it sometimes you can do it all the time and especially for sediment in that case was with a lot of loggers which are overall fairly inexpensive and then wind and tides were from you know USGS gauges and so forth but yeah if you're going into for example durability then good luck right collector everywhere so we do sometimes a mix sometimes we do infer process connectivity from the from observations particularly for hydrodynamics or doing smaller scale for you know other processes like sediment transport yet it's it certainly there are limitations and though we'll have you know more of these cheap sensors so you can just deploy all the way from remote funds and as well yeah there was another question that popped up let's see um I have one so cool hello and John I'm wondering said I mean you've worked some in Bangladesh where we only have SRTM but the scale of the features is so large you know how far can you take these these tools because there certainly are still small scale features that you're not going to pick up in the SRTM but you know but the main channels are 10 kilometers cross yeah I mean that's that's a really that's a really good point how I mean you could speak to the the Bangladesh project in a little more detail but I mean as always you have to think about the scale of the feature that you're trying to resolve relative to your technique right so in a case like Bangladesh you know if your main river channels are large enough that you know a technique like SRTM can kind of resolve and characterize them easily for a specific question that's been more than good enough right it's like for instance when we were looking at the kind of histograms of elevation values if all I care about is just the kind of distribution or shape of elevation in this mountainous regime SRTM does basically the same thing that lidar does for me it's only when I need to kind of answer more specific questions that require more small scale features but I need to move to a different data set so as always it's kind of a fit-for-purpose thing right and right and I would say Mike that you know Dave Tarboton and I have talked about this many times and we usually consider like the transition between GeoNet and he's taught them at around five meter resolution if you're about five meter if you're like a 10 meter resolution you can still use GeoNet but the advantages are less but what you bring up is also a very interesting point in this conversation that essentially we are having in model data integration because even in a case like Bangladesh right if you had for example from drones or you know from small lidar applications some snapshot of what the high resolution you know topography would look like in a subset of the system then something that in my group we're really interested in is how do you then go about developing sub-grade scale parameterizations for those smaller scale features and how you're going to embed them into the larger scale model so lots of the work that we've done is kind of like at the boundary between these two things and try to use pockets maybe of information that is finding a resolution and then think about how you're actually going to put that into the larger scale model which is a super interesting topic right okay I think let's let's move on to the last portion of it you have some time with the actual modeling which I which I hope everyone will enjoy but this yeah I'm really happy we're having this discussion because this these are a lot of the kind of topics and questions that you know we talk about a lot but are also being focused on the RCN okay all right so last last part of the presentation and then it's just going to be kind of playing with in our sandbox so the rest of the rest of it um so as we were just talking about right a big question is so we have high resolution topography data and we have these beautiful data sets so what what is the utility of them in things like numerical modeling so when we have high resolution topography we can use it as an opportunity to better integrate and validate numerical modeling and forecasting but there's a lot of challenges that come with that like some of the ones we were just talking about so we have challenges with computation in terms of how can we run you know a really large domain model using a really small grid size without just taking forever or costing way too much money but there's also a question of just because something's high resolution is it useful if we're trying to answer a question about a process or about a kind of domain that's occurring on a large enough scale do we necessarily need to always use some of that high resolution topography and so what we're really trying to do is have better coordination between these kind of different data communities we're talking about sorry do you still want to say something that's all good John go ahead okay um better coordination between you know the data driven communities and the modeling communities hence the research coordination network that we're trying to do here as an example of that you know here we have a study looking at um kind of numerical modeling of uh flow within a bay or an inlet and we have really really detailed multi beam symmetry at basically one meter resolution and when we have grid resolution we're running our delt 3d models at 50 to a thousand meters how do we kind of connect or span those two different domains and what are some of the kind of ways that we can do that so in my view when we think about how high resolution data is used in modeling there's essentially three different kind of views or ways one is ideation or conception so it's the idea that if we can observe a phenomena or process or feature within the landscape using very high resolution formula a question or format hypothesis that will then be tested with the model even if the model is not using that data directly per se secondly we have actual input so this is using like a high resolution topography data as a direct input into the model whether it's from attributes like vegetation density or surface roughness or is the actual kind of base grid that we're going to run a flow accumulation model on or things like validation so if we're running a model such as uh you know beach uh beach erosion in response to a hurricane we can use things like pre and post slide or pre and post structure for motion data to test how well our model did in terms of predicting patterns of erosion and deposition so I'm just going to run through some examples of kind of work that I've seen that kind of touches upon these kind of three different concepts so one is this kind of recognition of small scale features within a landscape kind of like what we were just talking about in our test example of the San Luis Mountains so here's a paper by Jonathan Suba looking at the recognition of these lower leaf flood plain channels within river systems in Indiana and recognizing that the fact that these exist even as though you can kind of process scale them or handle them using lower resolution data or even most modeling techniques has pretty significant implications for predicting patterns of flooding and patterns of peak kind of discharge within the river system in terms of things like input we have examples like this by Saggy Cohen looking at how using a very very high resolution grid a one meter raster in this case and actually just running a numerical model directly on that raster helps you better understand patterns of flood validation in response to specific events going a different way not just using the kind of raster as your kind of base elevation surface and going back to a little bit about what Paolo is talking about we can use information from our high resolution data sets to help us parameterize and kind of inform the structure of a domain that we're then going to run a model on so this is a really cool example looking at a flood basically flood propagation but rather than trying to say I'm going to run my flood propagation model over a really really you know small cell size raster I'm going to use the lidar data in point cloud format to look at basically the standard deviation of heights within each cell as a function of the elevation surface and then take that as an input of a roughness coefficient that I can then distribute across a coarser model domain so essentially using very high resolution data to parameterize at a sub grid level of model domain and getting much more accurate results relative to you know actual measured in nation extent in terms of things like validation we have some recent work by Katie Barnhardt that's super super cool if you haven't read this suite of papers I highly recommend it but basically using high resolution topography data from upstate New York and to characterize two different watersheds that kind of touch upon different types of the lithology and using these the high resolution topography of the validation water set to essentially validate numerical models trying to replicate what the calibration watershed is seeing in terms of like validation as I mentioned as well we have things like you know direct measurements of pre and post beach berm extent here in the Gulf of Mexico in response to a hurricane and then predicting how well a model like x beach did relative to those measured differences from our high resolution data so those are just kind of a few of the different examples of how we've seen a high resolution topography used in data sets but now I want to kind of get into what we're going to be doing for the rest of our afternoon or as long as you want to stay with me of how we can actually run our own numeric models and kind of inform them with our high resolution topography data going back to the same always mountains so to illustrate a modeling workflow we're going to be using a new go which is a hydrodynamic modeling code from the australian national university in geoscience australia hence a new go which is a very very powerful open source solver that allows us to solve a depth integrated finite volume shallow water equations across an unstructured triangular mesh so this was originally developed for tsunami inundation modeling but it's recently been adapted for more flood and flow modeling as well as sediment transport modeling as in the also actually erosion modeling the use of this unstructured triangular mesh is very cool because it allows us to kind of specify different resolutions and parameter domains within a single study domain so we can for instance model maybe in very high resolution the area directly around our river channel but then have it start kind of coarsening out as we get further and further into the floodplain but also change the roughness coefficients of each of those domains as well so just some examples of how a new guy has been used in the past before we actually get to playing with it so here's some work by maryl perignon who was recently of cu boulder here using a new guy to model a flood propagation of the 2013 front range flood so taking a high resolution topography data as an input as well as storm uh rain gauge measurements of the intensity of storms and transferring that precipitation to discharge about the stream network and modeling what the floodway of that new guy would predict was versus what was measured in real life so in this instance here we have the hydrograph showing uh in the thin black line the usgs gauges measured hydrograph and then black the modeled a new guy output coming through the system from this high resolution modeling domain and showing actually pretty good correlation between the two but just like kind of shift in the timeline between them what they do to you know things like roughness parameterization within the landscape so before we start doing it for ourselves we're just going to kind of talk through some of the basics of a new guy and how we set up a model domain so we specify an overall domain here so again this would be like this first rectangular area which is just our overall study region and then we can specify internal sub regions such as i've shown here on this mesh that's a little bit denser we can choose a resolution for our triangular regular mesh for each region and that resolution is not the same as a raster resolution where if i say it's 25 square meters every cell is 25 square meters the resolution just represents the maximum allowed area for each triangular node they can actually be smaller we'll we'll see what that looks like in a little bit we can specify a manning's friction coefficient for each of these regions and we can also specify boundary conditions of our domain so for instance when it when the flow model flow hits one of the boundaries does it reflect off does it just go right through and so on and then we can specify things like inlet operators outlet operators it's a pretty powerful pretty flexible modeling platform that we we use a lot in our group now so just showcasing again some of the details of that kind of mesh structure so you can see this is again some of the work from shazad it's work on the trinity river showing the population of the mesh with very a very high resolution domain about the channel of the river itself and then corresponding out as we move into the flood plane but also showing that it's since it's an irregular mesh not every triangle triangular node within the domain has to be the same cell size as each other you're just specifying a maximum cell size all right so a little that little bit of an overview into both just kind of how we're seeing numerical modeling using high resolution topography as well as a bit about anuga let's uh let's play with it let's actually become numerical modelers for another 15 minutes or however long you guys want to stick with me so to do that i'm going to share my screen back again we're going to go back to the github and we are going to open the notebook that's labeled part two so can everyone see that fine yeah it'll good okay and remember to clean it up yes so we're going to open we're going to open the notebook and we're going to hit open in colab again and the reason why you wanted to close the previous tab before doing this right now it only runs in python 2 and everything else we were doing is running in python 3 and that's also one of the reasons why i had everyone do this in colab because trying to get multiple environments and pro python versions on everyone's personal machines would be uh you know exercise and frustration okay so now that we have our python notebook opened in colab we're going to connect to it run time again we're going to get our warning that python 2 is difficult in colab if we were trying to do this clinic next year probably wouldn't work okay so we're connected we're connected to an environment and here we are so we're going to be looking at modeling flooding in the san louis mountains in a region that's just in the northernmost extent of the study area that we were characterizing in part one of the notebook um so you can see here if people clear the outputs sorry john no thank you for reminding me yep uh that's not that that was just an embedded image it's not an output but we're going to now experiment with the newga over this domain that's again just to the northern portion of the watershed that we previously characterized so you can see in yellow here that channel network that we delineated previously and we're going to basically run a flood through the domain shown in the shaded box and we're going to measure and monitor the evolution of the flood wave at a series of virtual gauges which i've specified that are shown here in blue so the overall domain that we're going to run is about you know 1250 meters by 650 meters um also just a note this this exercise was really inspired by a previous clinic that steven roberts of an u ran in cst ms 2018 uh his clinic was all about a newga and went to a ton more detail so what you see here is interesting you want to learn more about it please visit his clinic and visit the main kind of a nougar repository and just kind of get a feel for this but it's a really really cool platform it's pretty powerful on what it can do okay so similar to last time we're going to clone our git lab our github and bring in all our necessary libraries and install them so it's going to run here we're going to get a little uh finished message when it's all when it's all good i know we only have 11 minutes left i promise you will be able to at least run one of these model realizations in the time we have left while this is loading does anybody have any questions science is good yeah we're almost done installing our environment cool okay so ready next we'll just import a newga all right now we're going to set up our model domain so very similar to the workflow showing you in the uh in the powerpoint presentation so we're going to define our overall bounding region through an extent csp that i provided we're going to specify a region about the stream channel that we're going to be looking at we're going to specify the dm again using our one meter dm and then we're going to set a resolution for both of these regions again 100 meters for the kind of broader extent and 25 meters for the area around the channel so i've set we set those domain boundaries and then we create our our mesh cool so now you should see a bigger pop-up that shows again the dense low high resolution mesh about the stream and coarsening to the outer edges of the domain so now that we have a mesh we're going to set up the initial conditions for model runs we're going to just set an elevation yes um we're going to set our elevation um from again that dm we're going to set a friction coefficient um and then we're going to set just like initial stage condition in this case we're going to say everything's dry when the flooding starts which is actually not a unreasonable approximation for an area like this all right so we've set our initial conditions we've sampled using our dm and that mesh elevations to each triangular nodes and now each triangular node has an associated elevation value we're going to set up what the kind of boundary conditions are whether and then we're just going to say they're all transmissive so if water hits the boundary it can lead up we're going to set up our initial discharge conditions in this case 250 cubic meters per second and we're going to say we're going to run it for about 200 seconds within the model okay so let's run the model and see how it looks you're going to see the time separate calling out every yield step so you can see that for each yield step it's taking about two seconds to run so this overall model run will take about um 80 80 to 100 seconds which is actually pretty fast for this sort of model and then we're actually also going to then create a plot of the evolution of the flow through time along this channel i have a quick question yes waiting so with the triangular mesh did we create i'm just we're kind of rushing through did we create kind of two scales for the channel and the flood plain like you showed in that example yes we did so so we specified those sorry right i see it there and i'm like when did that happen yeah so we so in this in this first portion we see we set a base resolution of 100 meters and then a channel resolution of 25 meters okay and they're just two different polygons and thank you right and jennifer that's one of the the main advantage is i would say of an uga in terms of you can actually design these meshes in an extremely simple way it's really down to just delineating which domain you want finer which domain you want coarser and it's very very easy to do yeah very cool yeah so it lets you it lets you really kind of nicely set up all your boundary conditions and use information from you you know field data and observational data to say like you know this region is going to correspond with for instance this vegetation type which has you know this potential roughness this region is going to be you know our channel and kind of do all that sort of or all that sort of stuff in a really simple way which is really nice and yeah hannah this you're absolutely right this is very similar in terms of how it's been applied to geoclaw which i think they're running workshop on on friday but it's very good point okay so our first model run is done and so now we're just going to now we're just going to plot it that part just takes a few more seconds but we're almost there i promise okay so hopefully everyone's model run completed successfully hopefully now you have a plotter window with a nice little play bar so let's play it and see what happens oh so hopefully you're able to see the floodway propagating down the stream so you can see the scale bar here is five meters max of five meters depth which nowhere nowhere in the smile really getting close to you but there's a few things and we'll look at more detail in a second but i want to point out is that you can see that even though we specified a relatively coarse mesh size 25 square meters for the area around the channel the actual effective resolution along these areas is actually a little bit higher because again that's a maximum area so you can see that like for instance once we get over to the west side of the domain the pattern of flow is starting to be really affected by a lot of these kind of smaller secondary channels that are located at higher elevations relative to the main channel and so we can actually look at that in dm so you can see here within the dm of our our set of the hill she applied to it once we get to the western side of things we actually have multiple channels present within the small scale stream valley and so that's reflected in these patterns of inundation because you can see that the flow is actually starting to branching and bifurcate to take advantage of these preferential flow paths that exist within the dm even though they're so small scale you wouldn't necessarily think they're being picked up by the scale of the mesh that we chose okay so like i said we actually plotted you know some virtual gauges along here and sampled um what the flow depths were or the stage was and so let's actually plot those and see what the timing of the flood wave actually looked like and what that's happening to the inundation depths along those regions okay so there we have our six gauges corresponding to these six gauges along the base of the stream valley ooh the numbers there are plotting i apologize for the scale it's for some reason why i put like yeah uh plotting it in a very very weird way but we can we can get the point across where we can see gauge zero which is closest to where we're setting our discharge to start sees the arrival of the flood wave basically immediately gauge one sees it a little bit later gauge two a little bit later and so on gauge five is the last to see it because it's the furthest downstream because again we're modeling a dry bed that then has a constant discharge being applied to it and we're seeing that bore of water come down the system so you and you can see that once that water arrives because we're maintaining constant discharge conditions at the inlet it basically stabilizes that some value and everything just kind of keeps being uh what transported out of the domain to the west this is not necessarily the case if we were running this domain or this model under more realistic conditions where we would evolve it for a much longer period of time and with an actual hydrograph from an event as the inputs we'd be actually varying our discharge conditions with time we're specifying just for the sake of playing with this model just a constant discharge condition uh and a relatively small uh amount of time that we're letting the model evolve okay so let's try to run it in the model again with a differing friction value so the manning's particularly specified earlier was about 0.015 which is already kind of unrealistically uh high let's see what happens when we kick it up to 0.03 we're going to keep everything else the same so we're going to run this and what we're going this this cell here what this is doing is again just specifying our base and channel resolutions specifying all our domain conditions and then also specifying the name of the file that we're going to save the second model run to now what's also really cool about anuga is that all the model output is packaged into something called an sww file which is essentially a netcdf file so if any of you are familiar with netcdf uh there it's a very common file format that actually has a lot of kind of pre-existing tools and workflows to kind of sample and process data out of so by having everything within this sort of wrapper um it lets us kind of store and then manipulate model runs out for the fact pretty easily okay so we're going to keep everything the same we're only going to change the friction and now that's what our mesh looks like um let's run the model again and see what it looks like with that kind of different friction and then we're going to query those same virtual gauges and see what difference just changing the friction factor made in terms of the timing of the flood as well as the kind of inundation depths that we're experiencing well this is running does anyone have any questions any questions from the from the johns the panel johns i'm seeing you can see too it's kind of fun numerical modeling doesn't have to be you know scary or or intimidating um we can use pretty complex models in a pretty simple way just to start playing with and try and testing different controls for boundary conditions and different inputs um and so they're really a great tool for kind of experimenting with what kind of different characterization of landscape means for you know effects like flooding or surface processes or hydrologic connectivity um and this is just kind of a fun way of illustrating that uh hey am I here I do have one question yes um I was wondering when you're running this are you manually changing the refinement when you're playing with the parameters so then it will run faster and then are you or is there like an adaptive mesh refinement uh kind of section in a new go where you're able to just uh let it auto select for you or something like that like and we say the mesh refinement you mean like the resolution we're specifying for it yes uh we're manually selecting it here yeah so you but you specify the domains in this case okay thank you okay so now we have our output from the second model run so we can run that it looks fairly similar but one thing that you might notice is that it appears to be propagating across the landscape a little slower and the maximum depth seem to be a little bit higher but let's see from our gauges whether or not that's actually true okay so now look this is kind of this is different from last time I read it but there's an interesting screen where the pattern of inundation in terms of the flow depth as well as the timing of when that flood arrived are different based on what we changed with the friction so by increasing the friction factor of the domain of our channel we've basically slowed the propagation of the flood bore as well as increased the inundation extent so it's kind of fun to think about how you know this might relate to a real world problem and for instance again in these dry watersheds if for instance this is a watershed that hadn't been occupied for some time and had become heavily vegetated you might actually have a pretty rough friction factor relative to a continuously occupied you know sub-aqueous channel and that might then mean pretty big differences in terms of what your kind of predicted maximum inundation extent and depths might be for this landscape again we can play with the effect of those parameters using a relatively simple tool like this then we just have one final one final run so this is again keeping everything constant but now we're modeling an incredibly unrealistic and fairly catastrophic flood across this domain this case we're going to run a flood with 5,000 cubic meters per second of water for 200 seconds and we're just going to see how that compares to the previous ones and then we can just pause the final pages of it there so this though after this run this is essentially what I had prepared for everyone for the for the clinic and for this portion of the workshop you know I'm happy to keep playing with this or answer any questions what I was also hoping to have and I know we're over time now by two minutes um is a little bit of a discussion with everyone about just what what you guys think I mean how do you use high resolution topography data with your own work how to use modeling what are some sort of the opportunities that you see for better integration um and and kind of path forward for the community to take um I think we can just kind of open it up now to more of a kind of free discussion if anyone doesn't have any more specific comments or questions about what we're doing of the nougat here again this exercise is more just to kind of illustrate um how we can kind of use complex models in a simple way to kind of start you know playing with different aspects of the landscape and different aspects of connectivity but it's also just kind of fun to to vary some of these parameters and kind of see what happens like I'm doing a lot of this in a very unrealistic very simplistic way versus what you'd actually do in a you know a more well characterized study uh I have a question about what you're doing with the nougat and defining those domains between channel and floodplain um I'm sure it's somewhere up in the code and I'm just not seeing it but how exactly are the the boundaries between those different domains being defined for the different mesh resolutions and like how uh how much certainty is there in that right so the boundaries between the different domains are being set by me uh in the form of a of a CSV file for each domain that essentially has the extents of what I want that domain to be um so if we run back up while the code is still running uh let's let's see here so you see when we first imported everything we imported a bounding polygon and a channel polygon which corresponds to this extent CSV and a channel CSV so these CSV files essentially just hold a set of x y points uh that specify the shape of the polygon within the domain that we're going to call either a bounding polygon which is the overall area or the channel polygon and then we also then just specify what the base resolution is and what the channel resolution is and also that we're defining these regions as what we set as the bounding polygon the channel polygon and then when we set up our domain using the annuga code that's where we then say you know our maximum area is going to correspond to the base resolution our interior regions are what we kind of made from our CSV files that we put in um and then that's what then gives us when we set up the domain right here in this domain dot set uh what our kind of irregular mesh is and then this is where we just specify you know things like elevation um and then sample our elevation file from that mesh that we built but everything up until this initial condition set was all about setting up the domain and specifying what regions of our mesh we want to be what and the resolution of them that's a thank you i mean it's an interesting question because you could say how do i get that right in terms and in some cases you may not know what the boundary is and i would say you know at that point you would use a higher cube information you'll gather you can from for example cross-actions on the lighter and you'll get imagery if you have field data and if you have completely you know if you don't know then you can always run multiple simulations and kind of vary that boundary in terms of where you change your friction coefficient and perhaps if you field observations get a sense of what's the right uh parameterization for that coefficient because i'm guessing that that's all you're thinking Vanessa that might be that in some cases it's not that obvious as to where you're going to switch that roughness coefficient then you may want to you may want to try different ones i mean i think in our experience with our work what usually happens is that we've either gone to the field or we have imagery or we have some other type of remote sensing we can compare to and if if you don't know any of this then some form of Monte Carlo approach of like where you vary that boundary would would probably be what i would do yeah yeah that's exactly what i was thinking like low relief landscapes wear that right what that you know and the interesting thing is that it's also going to in some cases for example in some of the the places we work on it's actually also very seasonal right you kind of have cases in which you know during the winter and there isn't a lot of vegetation and then perhaps you know that flood plain coefficient is not that different and then when everything explodes in the summer you know Louisiana coast it's one of those cases where we want to switch it so it's also something that temporally you may want to have you know you may want to represent seasons as well which would change what that boundary is yeah and and i yeah i apologize i think i misinterpreted what you i thought you were asking where we're specifying within the code not a more you know journal question of how are you no i think it was both and i just i just took off on that because it's an interesting question yeah for sure because i mean in this example that we're running it's mostly complete arbitrary it's maybe a little bit guided by the lidar but i've just chosen this is where the channel is this is everything else um like palo was saying like when we've used this in actual you know studies you'd go out to the field multiple on ends and you can see you know in the winter when all the vegetation the point fires dies off versus in the summer when everything's kind of growing you really kind of get a feel for how that varies but it varies you know pretty substantially but that's again where things like you know multiple wider surveys and field surveys can really help you kind of delineate these you know potential both geomorphic domains but also these kind of potential uh you know like land classification domains which might help you then parameterize something for a numerical model like this right so our kind of giant catastrophic flood should be done running and you can see that the overall inundation is much much greater than the previous rounds but it's still pretty well confined to this valley because we're looking at a pretty kind of entrenched nest landscape if we are running this on landscape like the Gulf of Mexico which is very low relief this would look quite different and a lot more interesting in terms of the patternism inundation of all these kind of small secondary channels that can become activated at different discharge levels however when I tried running that uh within co-lab it quickly used all the ram and crash which is why we're not doing that today but it's fun to play with um and then we can kind of plot our final final figure of gauges and yeah just confirming what we see with the visualizations where the massive flood wave is just you know head and shoulders above the maximum inundation depth being measured at the other kind of modeled discharge values and also interestingly over the course of the um the model run time in most of our locations we're never actually approaching a steady state which means that it's still kind of evolving and still actually kind of creeping up and filling up a lot of the photography available to it so that is just a very very quick brief introduction to Anuga and American modeling and some of the concepts of how we might think about kind of characterizing landscape and incorporating those kind of framers and characterizations into uh you know testing some hypotheses with the model um so with that I'm gonna stop my screen sharing here um sorry Paula can you hear me yes okay sorry I got a message on zoom saying that my speaker is not working which doesn't seem to be the case there's just a little bit of static here and there but it's not a problem all right cool so with that guys that is really the uh the end of the clinic so I wanted to thank everyone for you know taking the time especially since we went over by a few minutes uh to to join us today I hope you had some fun I hope you learn learn some things um again I want to just kind of take the opportunity to open this up for any sorts of discussion or comments or questions or anything that any of you might have um we're we're a we're a small enough group now I think we we're just down about 15 people that any sort of questions or discussions we can have we can have much more about an open discussion now yeah and I would say also that you know if you have to run that's you know that's perfectly understandable and one thing to keep in mind is that this is one of the themes of our earth kubar sian so if you speak your interest and you want to be involved there's going to be way more to come especially over the summer so feel free to email any of us and just uh get part of the group thank you all so much yeah and like pala said you know if you have to run we have to run but I again we would be interested in you know hearing from you if if you're working on kind of you know similar you know questions or thinking about how you might use this in your own work uh you know it's we'd love to have a discussion about that right and also if you have like if you have tried to do this right because this is really the intersection of modeling and data and you know whether you encounter challenges either in one or the other right how you go about analyzing later data what are those challenges worrying how you analyze them or things like that I guess just to share um I've tried doing some of this in GIS and been pretty frustrated and know that there's like I could do code and and like have freedom from clicking and that would be wonderful and just to have these examples showing what libraries you're using and you know some little snippets here and there just I feel like it's so much more accessible suddenly um so I just yeah really appreciate like one open source is amazing and then you know having you guys spend all this time putting this together in this way is um you know amazing resource for the John John's work I don't want to take about it it's it well thank you I'm happy that this this is helpful for you I mean I hope we just kind of showcase like we barely scratch the surface right this is a pretty pretty like you know basic introduction on this but I'm really happy that you see you know things or elements that you can kind of go into more detail on because it's really a lot of fun to do yeah sometimes it's just overwhelming like all the different ways to possibly do it and just to see like somebody's thought about this a lot and this is what they're doing and like I can at least start here yeah now that's that's what we've encountered usually you know with the ladder and and that's how I think often sometimes people just will just build another code just because sometimes it's also hard to understand what people have done is it's not well documented I think that's changing a lot but what we've always found with with our researchers that there's always the need for some sort of like stamped or approved workflow that you're like all right if you have to start you have to do this this is one way of doing it right and I think that we can always improve that type of information or documentation and kind of facilitate that exchange within the community which is going to help people that get started but also people that just want to you know don't spend time reinventing the wheel and just using what's been done and then use your time to do something new yeah I could see it's an investment and I just again want to thank you yeah for making this investment and sharing thank you for joining us on that note I just think it's you know especially if you're thinking about how to kind of use this sort of stuff or this is for everyone or you don't share with other people I also think it's this is a kind of fun example of how we can use things like Google Colab it's like a really good sandbox for kind of playing with a lot of this different stuff does because I know you know from personal experience when I was learning like how easily you can kind of get yourself into trouble trying to manage versions and repositories and libraries on your own machine and you can start breaking things up quite understanding why things are breaking whereas in a virtual environment like Colab you can kind of just keep everything within a still contained environment and it breaks you can just start over yeah I'd be curious what what you see where is the limit of that like this seems good for trying things and then at some point it feels a little insecure right and so I can tell you from making this what some of the limits are and the main one is RAM it doesn't give you that much memory and so with some for instance like going back to like topographic analysis the domain that we are working on was about a 160 180 megabyte raster which is actually a pretty decent size but it's a pretty small one by lidar standards and that was pretty you know pretty handleable for as you can see we can we could run flow accumulation across that we calculate curvature we do all that sort of stuff in just a matter of 10 to 20 seconds so the computation time is not the problem what is is that larger datasets particularly rasters and arrays have to be loaded into memory to be computed and once you get above like 500 to a gigabyte all your all your memory is gone basically in like 30 to 40 seconds and the in the runtime will crash so there's ways around that you could do things in a more tile approach like we had tons of different tiles you could just you know operate on all of them sequentially but using like really big datasets that's where this thing really starts falling apart just because we don't have the the memory access that you would really need to do something like that but other than that I mean as you saw we were actually saving all our outputs in you know proper geotip format so if you were doing this if you wanted to just get into play with this sort of stuff and use it in your own work which what I've done in the past actually is you can just download those geotips right back to your home directory or into your you know box folder or google drive and then just use them like you would any other output because like as far as co-labs concerned you're not doing anything special in turn it's not like a you know a weird format or a weird environment you're just using a virtual machine so all the outputs and everything that we're doing can be saved locally and used just like you would anything else so it's pretty powerful in that regard and it's certainly a perfect teaching tool that's helpful thank you yeah I imagine that the ram yeah I could see the little bar there telling us when we were hitting limits and yeah I know that these files can get humongous yeah and you don't get any warning either once the bar hits once it peeks out it just it's over yeah and this is raster so if we were to be in point cloud this would be way worse although interestingly white box tools we didn't play with it here but white box tools actually has the ability to handle point clouds as well so you could you could actually execute a full lidar workflow using these open source tools of like taking a point cloud turning it into a raster and so on it's very cool I have a question that's maybe a little bit out there or maybe just not really known but um with what you were showing us about all the different you know the resolution that you can get from this really or all the different you know landscape detail that you can see from this high resolution data um I guess I can see how that's you know something that's really important to have a handle on when the question that you're asking is something that you want to kind of solve on a human time scale um but in terms of like you know controlling things like channel initiation like how important are these little details when you're looking at million year time scales I mean is that I guess what kind of the angle I'm coming from this with this from is that I'm you know doing this work with trying to like reconstruct a paleo surface uh which is obviously going to be pretty low resolution I don't really have a choice in that but I'm just wondering you know what what kind of different results would I expect if I had access to all of these little details no that's that's a great question I mean and that's that kind of touches on some of the things we were talking about with like how we would use this data within modeling and I feel that for a lot of these things like for instance if you wanted to kind of run over a really large scale landscape evolution model on millennial scales of like running stream power and how stream power starts affecting drainage divide migration and capture and whatnot I don't necessarily know if you'd have a huge difference between using a very coarse resolution or a very fine resolution what I think the fine resolution really gets you is more in that kind of ideation or concept stage in terms of trying to understand what scale of parameterization or what scale feature do you need to kind of incorporate and are other parameters that you would need or your model would benefit from having included that even if you're not kind of using it the same spatial scale you know that those kind of smaller scale processes exist and are important in the landscape on these timescales and so on a longer time scale you need to think about a way of modeling them or incorporating them even if you're not running them at that same kind of high resolution if that makes sense I mean it's a super interesting question I would say that some of those stream captures and things that do happen at the headwater stream right and so detecting those channel heads plays a role now it is true that of course you have limitations so it is interesting to me it's such a you know it's a big question because it's always the and it kind of came up in in John's slides as well if you have high resolution topography is that appropriate all the time and it's not necessarily going to be the case right it's not that because the data is going to go to one millimeter resolution that you have to run the model one millimeter resolution by having the ability of you know even if you're going over geologic timescales just representing what that subgrade component looks like then perhaps you can run the model and then re-embed those features in future scenarios as well it is your question going to make me think about Greg and Kate Barnhardt work what they did that sat in that landscape that had nuclear waste right and they were trying to understand the evolution of these nuclear waste and where it would go so they had to run over longer timescales and of course they couldn't necessarily get back what the specific distribution of these features was not specifically the actual location but they could get a distribution of the elevation and kind of get a sense of where that subgrade plate at all so it it's super interesting and it does depend on your question and I think if we have the ability of getting that data then we can we can address where we want to include that and and not right and as a follow-up on that like I think just thinking about some of the things that I've done in my own work or things that I've noticed that you know I'm really curious about or they're interested in for instance like using stream power to you know model erosion of a landscape or relative erosion off you know heavily relies on the slope right of the channel in a lot of landscapes your kind of calculated slope is drastically different depending on the resolution you're using to capture that so for instance like the even in using 10 meter resolution data versus one meter resolution data in the old coast looking for the smaller stream values the 10 meter data under represents the length of those stream networks by an order of sometimes 20 to 50 kilometers and thus under represents or over represents the steepness by about two or three times simply because the large the lower resolution data is not capturing the small scale meandering of that's actually present in a lot of these river and stream networks so there's a lot of kind of really interesting implications for that right it's like because we get into that scale like what what scale of do we need to really accurately characterize the landscape and we saw that some things in many people's resolution is a good job but then when we go to the smaller scale we can see you know even if we're in a you know relatively steep mountainous environment those streams can actually have pretty significant meandering in the gravel bread river systems and that meandering is really really efficient at reducing the overall gradient of the stream and so then you can maybe think about what sort of kind of sensitivity tests or uncertainty do you need to propagate into your model to account for the kind of range of distributions that you might expect based on what you see in small scale but again like Paul said even if you're not using that small scale data itself like to evolve over over time the other thing I was thinking about is really the theme of these CSDMS meeting right because I think for those that really work at the interaction between topography and vegetation then I think those fine scale features are really going to be important from an ecosystem perspective where you have that you know smaller scale features and patterns and patches of vegetations does actually play a big role so again it depends on the question but I think for the for the theme of these years CSDMS I could see the fine scale structures really playing an important role when it comes down to ecosystem because talking about connectivity you know it's all about vegetation patches and size and distribution and that plays we know already from our work that plays a huge role when you look at transport yeah that's that's all really good stuff to think about so thank you for those thoughts any other comments general questions concerns no we're just gonna see this part of our first yeah exactly right John cool well then if that's that's everything if there's nothing from the Johns I think we can we can go ahead and call it an afternoon again thank you everyone so much for joining us especially in this virtual forum I'm glad that people found it useful and got everyone kind of stayed engaged with it yeah looking forward to hopefully seeing everyone next year at CSDMS or RGU or something else well thanks everyone and I think CSDMS sent an email about the impromptu happy hour so those of you that want to jump in I think there's a zoom get together which is informal over the two hours so I think from now to now and a half from now otherwise we'll see you tomorrow morning Kenyans well thank you all bye John is we Chris is weird to see you John all right catch up y'all later nice job bye Chris bye