 Will you hear me all right? All right. So this is basically, this talk is gonna be a little bit different from some of the others. It's really gonna be kind of a broad overview of image processing in Earth science. So as far as my background, I'm pretty much a remote sensing person, mostly focusing on land cover changes, things like that. I use mostly satellite imagery, but also I'm beginning to use a lot more aerial photography, especially low altitude aerial photography from things like UAVs or drones. But when I was putting, actually, let me just go to the next slide here. For the interpretation of Earth science, I mean it's basically Earth science is all the sciences that have to do with planet Earth. That's too much to really address in 15 or 20 minutes that I have here. So I'm gonna focus on what I do, which is mostly focusing on the Earth's surface and a little bit into the lower part of the atmosphere where the weather happens pretty much, or more specifically the biosphere where things live. And when I was putting this talk together, I was trying to think of what aspects of Earth science kind of differentiate it from some of the other sciences, and one is the distance that Kevin mentioned. We also work a lot with multiband and hyperspectral type data, which I guess other people do too. I guess more people do than I expected, really, since a lot of the image processing toolboxes that I have experience with seem to be focusing a lot on RGB or kind of lower dimensionality data sets. We also have to deal with environmental effects, so things like atmosphere and terrain. A lot of other people are working in laboratory environments where it's quite controlled. As far as the platforms go, pretty much anything that you can imagine. I mean, we're working with satellites that are up tens of thousands of kilometers, some of the geostationary weather satellites, all the way down to handheld cameras. People are using more and more. They're using these camera traps. They call them for photographing animals, or people are using UAVs a lot. More creative things are like balloons and kites and poles. As far as the imaging sensors, I think, I was used to think they were kind of complex, but listening to some of the talks earlier, our image sensors are big and expensive, but they're really pretty simple. We're just the multi-spectral, hyperspectral sensors working mostly in the visible. We do a little bit of ultraviolet sensing through the near infrared, mid-infrared, thermal infrared, into the microwave part of the electromagnetic spectrum. Most of our sensors are passive, so they're using the sun basically for illumination. There are some active sensors, radar sensors and LiDAR. We also use cameras, some very sophisticated cameras for aerial photography, but also a lot of point-and-shoot cameras. And I'm doing a good bit of work trying to see how far we can go with using consumer-grade cameras to basically use them as scientific instruments, and just trying to test those limits and push them. I have acoustic sensors here. I don't work with acoustic data like Sonar data. A lot of people don't even consider that as part of a remote sensing-type instrument. I think of course it is, but in any case, I will not be talking much about that. So I want to talk a little about the processing steps or the tasks that we do in Earth science, again, focusing more on my type of Earth science. And I have it broken into three categories, extracting information, image corrections and adjustments, and then this catch-all with a few others called other. As far as extracting information, some of this we were talking about yesterday in the tutorial with classifications and regressions. Traditionally it's been, you know, in our field it's been parametric statistics. That's kind of what I'd learned. These days, non-parametric methods are catching on. We're really kind of learning from the machine learning folks, which is really nice to see. I think we're a little slow to uptake it. It's still, even though a lot of the algorithms are quite old, or 10 or 15, 20 years old, for us it's pretty new. And I just make a note here, and I'll repeat, I'll come back to this later, about how difficult it is to use some of these approaches with a higher resolution imagery that we're acquiring now. We also do feature recognition and matching, and I differentiate feature recognition from classification, when I talk about classification, I'm talking about wall-to-wall classifications that we're saying this patch pixels are forest, this patch pixels is water, and things like that. Feature recognition would be trying to identify or count individual features. And basically what we're trying to do is eventually replace a human interpreter, which usually there'd be a human interpreter outlining and trying to find certain features on a photograph, and we're trying to kind of reduce that workload. And a lot of that is coming from the computer vision folks. As far as image corrections and adjustments, it's mostly geometric and radiometric corrections with the geometric corrections. What we're trying to do is warp an image, so it does fit a map, basically. So it fits a map base. So it is spatially explicit. And to do that, we need to remove the distortions from the terrain, but if the Earth was flat, it wouldn't be a problem, it's just kind of an imaging plane on a plane and it would be simple, but the terrain makes it a little bit more complex. And a lot of that has been automated now. I mean, when I first started doing this, if we could get images that were automatically corrected within hundreds of meters, we were pretty happy, but these days it's within several meters. So just using the telemetry from the satellite and ground control points and elevation models and things like that, we can do pretty well with automated methods. With the radiometric corrections, what we're trying to do is convert pixels to physical values. And most of the instruments are recording in radiance, but the physical value that we really want is ground reflectance. And that's complicated because we have the atmosphere and we also have wacky illumination geometry because of terrain, but as we get finer resolution, we also have, just with the features that we're imaging have three-dimensional structure, which is causing a lot of, makes it a more difficult problem. We also do mosaic-ing where we're trying to put together two or hundreds sometimes, especially with these low-altitude aerial photographs, trying to put them into seamless mosaics. And we're using, for a lot of that, it's structured from motion algorithms. But one of the issues that we have is normalization because especially with satellite imagery, the way that imaging sensors work, we might have one image from one season, another image from another season, when you put them together, the atmosphere is different, the sun angles are different, the vegetation growth cycle could be somewhat different. So it becomes very difficult to kind of normalize and make these seamless mosaics. So it's another kind of issue that we have to deal with. This other category is, the first one is feature transformations and derived products. Texture, someone was talking about texture this morning. There's several texture metrics that we use. Texture often has a lot of information about helping us better define and classify features. We do feature transformations, principle component analysis, for example, both to reduce the data dimensionality, but also some of the components have a very strong correlation with certain variables that we're interested in, for example, vegetation greenness. So that's another reason we would use these transformations to pull off to kind of highlight certain features. Interactive visualizations, I think some people already talked about some of that. And it's, I think, I don't do much of this, but I think it's fascinating what people are doing with this multi-dimensional data, making it, basically translating these data sets so a human can look at it and really make some sense out of it. I think that's extremely important and they're working with massive data sets when they're doing this. And I added hardware control, that's kind of a new thing. I think, again, with these UAVs or drones, being able to follow features and also both the drone or a camera gimbal, I think is going to be really interesting for a lot of wildlife studies and some other applications as well. Now the software that we use is probably just as varied and as broad as Earth Science itself. There's specialized software for specific domains. There's a specific software for specific sensors, for specific types of sensors. So it is quite broad. I tend to do most of my processing on a desktop, I always have, although a lot of the development trajectories I guess are going toward server clustered cloud type applications. That seems to be a somewhat of a sensible way to go for some people, but a lot of the people that I work with, I work a lot in the conservation sector overseas where a lot of the people don't even have access to the internet, so that's not going to help them so much, but there are a lot of mobile applications as well that are being developed both for data input, data analysis, data visualization, so that's kind of interesting to see some of the innovative stuff coming out in mobile devices. When we're trying to solve or do some image processing, we have access to the libraries just like everybody else, I guess packages, plugins, applications, and it's never really straightforward right from the get go of what the best way to implement a workflow, but it is nice that we have a pretty broad variety of these packages and libraries and applications to pick from. Open source software is quite strong. There's an open source geospatial foundation that was formed a little, I don't know, 10, 15 years ago, maybe now, so they've done quite a bit to kind of get geospatial software, open source geospatial software out and available, but there are still some gaps and that is filled with proprietary software that tends to be extremely expensive, at least for a lot of conservation-type organizations and us at the museum, it seems quite expensive, but I think open source is closing some of those gaps. And I just note that we have challenges writing and maintaining multi-platform software. That's something that our group tries to do. We try to produce a lot of applications for the masses so people can access some of the algorithms that are maybe more difficult to access by writing applications and it's not always easy. That's something I'd like to get out of this actually, this is next, or tomorrow. As far as the file formats, one of the big differences, well, I think a couple, one is maybe the multi-dimensionality, but again, it sounds like a lot of people are using multi-band type hyper spectral images, but we have coordinate reference systems that we need to store with the data, so a lot of the image processing packages just do the image processing, it doesn't work with the geospatial data, so that is one thing that kind of sets the worth science apart. These are some common formats. We do have HDF, that's quite reasonably common. A lot of the data coming out of NASA is available in HDF. There's also a lot of proprietary formats which can be a bit of a pain to work with. In addition to the geo-referenced imagery where the image looks more like a map, we also use geo-tagged, which would just be photographs that have a GPS coordinate assigned to them, so just basically the location of where the camera was, so if you had a GPS camera to take a picture, it tells you those coordinates of where the camera was when the picture was taken, and we've had some issues just trying to deal with the metadata, copying it and processing an image and updating the metadata and things like that. So to wrap up, I just wanted to talk about what we do well, where we have some issues, and then some barriers. This first bullet is access to public domain imagery, and it doesn't have to do with image processing directly, but it's something that I think is worth mentioning because the USGS has done a fantastic job making just massive amounts of data available in the public domain, and other countries now are following their lead, so they're the Europeans, the Japanese, and some others are starting to release their satellite imagery or spatial, geospatial imagery either in the public domain or at least making it freely available, which is great. I think we do pixel by pixel classification pretty well, that's pretty much what we've been doing forever. I think we've got the statistical models down reasonably well. I mean, you still see new things coming out in the peer reviewed literature that they say was an incremental improvement over something else, but I think in general we do that quite well. I think as far as the image corrections and adjustments for the geometry and the radiometry, we do that reasonably well. I think the limits there are more of the data that we have available. I think we do large image processing on the cloud, that's, I say we do it well, I think there's still a lot of room for improvement, but there are some resources that are available to us that pretty much give us access to the entire satellite archive that goes back to I think 1972, so it's just kind of neat that we have that capability, although as far as the processing algorithms and how to deal with those massive amounts of data, there's still a lot of work that needs to be done. What needs improvement? In my view, I think a lot of it just has to do with being able to deal with the higher resolution images. When I started remote sensing way back when, an 80 meter by 80 meter pixel was the good, the high resolution. If you're classifying forest, forest falls in an 80 by 80 meter pixel. These days we're down to half meter or even smaller and then a forest is shadows, it sticks, it's leaves, it's ground, it's all this other stuff. So we need to tease that out. So the way we do that most of the time is segmentation and the segmentation algorithms are improving, but the problem is that we, I haven't seen a good segmentation implementation that works in very large images. You need to pretty much put the whole thing in memory and if you're working with tens of gigabytes, that's a problem. And there are some solutions where they'll tile it and then segment each tile and try to stitch it together and those work a little bit but the implementations I've seen haven't been great. So I think that's one area where we can improve. I also have data and sensor fusion methods. So this is something that's been, we've been doing it now for, I don't know, probably 20, 25 years fusing optical and radar data to basically get more information than you would be able to get from a single data source. But I don't think we've made much headway because there's been a really, there hasn't been easy to get radar data, but just in the last few years have been a few radar data sets that have been released to the public domain. So there's this kind of flood, floodgates for radar data as I've opened and people are interested in this but they're using methods that were developed a long time ago. So I think this is an area where we'll see some improvements over the next few years because it's a lot of people are beginning to be aware of the potential and I think people step up and try to figure out ways just to do that. I think real-time imaging and feature recognition on cameras themselves, somebody had mentioned this earlier, so if there's a camera being able to identify the feature from the camera rather than taking it back in the lab and processing it would be quite useful for some of these UAV and camera trap type applications. And just to wrap up, some of the barriers that I see in our science, and I think this is with a lot of fields, it's just to disconnect between the state of the art and the state of the practice. I mean, a lot of the work that people are doing here is really a state of the art. It's the cutting edge, but most of you folks are at the cutting edge of research, but I'm more of a user and then I'm trying to help other people that have fewer resources than I have. And there's still, a lot of people doing remote sensing are still using the same techniques that I learned 30, 35 years ago. So trying to reduce that gap I think is important and I'm not sure the best way to do it, but if people have ideas it would be great to hear. I think another issue is with licensing, both with data and software. So a lot of the commercial data that is available, when you pay for it, not buying the data, you're just buying a license to access it. And those licenses are usually very restrictive so you can't share it or anything. So even though there's a ton of data out there, very high resolution data out there, we don't have easy access to a lot of it. And with software, even open source software, there's so many licenses and incompatibilities that I don't know if you run into this with psychic image where you wanna use somebody else's open source software library but the way it's licensed doesn't fit into your licensing. So it's just another constraint that I think if we could just figure out ways around some of that it would be great. And lastly, I just have software wrappers versus native libraries and I put this in because this is something that we're struggling with right now. We're trying to develop some Python applications and there are a lot of libraries out there that they're not written in Python, so they're written in C, so we have these wrappers. But if you're developing an application for multi-platforms and for the masses with a graphic interface and all that stuff, it just makes it so much more complicated if it's not written in Python. So again, that's something that I'm hoping to get some feedback from folks here, either informally talking or tomorrow at the hack of Fast or whatever it's called. So that's all I have, just sort of a quick overview of first science image processing.