 Thank you for having me on. That was a very interesting talk, Kim. I'm going to be talking about our work at the National Arboretum in Canberra using drones to make 3D models of the arboretum growing. I'm the director of the Australian Plant Phenomics Facility at ANU and a lot of this work was done when I was working for the ARC Center Plant Energy Biology as a research fellow and I should first off say thanks to all the people listed there that have been essential to this project. So I wasn't quite sure what level of understanding people would have on this. Drones are part of a, as Kim pointed out, they're part of a key emerging tool set for next-gen monitoring in the environment and these make up sensors, high-resolution imaging, LiDAR and a whole bunch of other things that let us measure the environment in ways we never could before. And since I've been working in the phenomics field really we're interested in how you monitor the environment along with genetics so you can measure the phenotype because those are the things together that define how ecosystems develop and in a crop context you know the the phenotype is essentially what we want for increasing food yields. So a bit of background on UAVs, I think a bit of this has been covered so I'll go through it quite quickly, but you basically have a lot of choices if you're starting a drone program from the sort of low-cost X rotors like these quad and X rotors like the DJI Phantom. These things are changing so fast so I used to give this talk and say that drones were expensive and confusing and then I said that they were cheap but still a bit confusing and now they're just cheap and easy to use. The DJI Mavic Pro came out this year it's a very tiny drone but it still has a 12 megapixel camera on it you can throw it in a backpack and you can still do pretty decent monitoring with it at the low end of the things. These sorts of drones typically cost in the one to $2,000 range although the bigger ones like this one here cost more in the $8,000 range and you get something like 15 to 20 minutes flight time and up to 10 kilogram payload for the really big ones. You can also use fixed wing drones so these give you a longer flight time something the 30 to 60 minute range but you do have to have takeoff and landing issues and they can have higher payloads but they do tip to fly faster which impacts how you can monitor with them. A new system that's being developed by people that we work with that Pro UAV is this vertical takeoff and landing system and we really think this is going to be an amazing tool because it gives you both the benefits of a fixed wing and some sort of a copter because with the helicopter part of it it can take off straight so it doesn't need a landing zone or runway and if something goes wrong and the gas motors fail it will catch itself and land so you can carry quite a heavy payload and you can put really expensive equipment on it because it's quite robust. On the camera side as Kim pointed out you have a lot of options ranging from RGB cameras the unborn cameras on the Phantom work quite well all the way up to DSLRs but you do need to watch about what cameras are using and isolate them from vibration or you get rolling shutter issues where the camera is trying to take a photo but the sensor is not writing all the data at once and so when there's vibration you end up with bad data. The sort of next step up unless you're building your one yourself like Kim pointed out is the multi-spectral cameras that can get you NDVI and the main thing that people use now are either the Micasense, Sequoia or RedEdge and those are in the three to five thousand dollar range. There are a lot of other sensors so you can put hyperspectral cameras on your drones or thermal and you can see you can get bands in the 400 to 1300 nanometer range but you need a big quadcopter because they're heavy they're very expensive and the data is quite hard to process so there really is a range of stuff available from really easy to use up to quite challenging depending on what output you need. The typical outputs you get from 3D reconstruction software which is a lot of what I'll be talking about are orthomosaic images so these are sort of essentially satellite layers that you can put into Google Maps or MapBox and then also DMs and GeoTiffs and some of the software can actually give you somewhat classified outputs or remove the ground so you only get trees can provide you with RGB multi-spectral or hyperspectral indices and also 3D point clouds and then you can make a 3D model of your environment as well if that's what you're interested in. There are a lot of options on the 3D reconstruction software front and I'll just go through a few. It's important to know that it requires a pretty beefy PC to run these things so you're looking at probably 1500 Australian dollars to get a PC because you need a fair bit of RAM you need a pretty good processor and then you need a graphics card. Pix4D Mapper Pro which is the one that we've been using because it was about the only one available when we started working. It ranges from $2,000 to about 9,000 depending on what license you get and you have to pay about a thousand dollars a year for support. It also has a challenge that you can only run on Windows so if you want to run it on a server or on the Amazon cloud for much bigger data processing and for automating stuff you have to get the Pro license. The other one people use a lot is Aggie Soft Photo Scan. I haven't used it personally but it's been well recommended. The cheap version if you just want 3D models is about $60 US and about 550 academic and then pricing goes up from there and those also will run on the cloud if you buy the Pro version. Mosaic Mill is another software package. It's the last quote I got from them was about 4,500 euros. It comes in a lot of flavors but I haven't used it. There's a free software package called VisualSFM which is FSM which seems to work pretty well because it's free and somewhat open-source software. It doesn't have all the bells and whistles the expensive ones do but if you have more time than you do money for your project it's probably a good thing to explore. There are also a lot of online options and these are great for testing so if you either just do occasional flights and don't want to invest in the software in the PC or you want really fast processing or you're preparing a grant proposal so you just need to get some initial data websites like these ones lets you just upload your images and give you a point cloud and other data quite quickly. A disclaimer this is not a complete list and all of these prices change rapidly and frequently so check the vendors for pricing and don't take my word for it there's plenty of other information out there but you see from these sort of software you can take a flight like you see on the left side here and then you get a 3D reconstruction on the forest and I'll talk more about that in a minute. So the the site where we've been flying drones is at the National Arboretum in Canberra and this is a really great site because it's just five kilometers from ANU and has fast Wi-Fi. The last site I worked at was in southern Utah and it took about five hours to drive each direction so when something broke it was you know good day and a half just to get there and back and having a field site where you have Wi-Fi access from your desk and you can drive out there in five minutes it really makes it easy to test new and emerging technologies until they're ready for deploying elsewhere and also the forest was only planted about seven years ago so we have an opportunity to monitor this this forest and build a three-dimensional model of the entire Arboretum growing in into the future which has basically never been possible before. So we initially at the ANU research site we installed a 20 node wireless mesh sensor network, camber weather stations for baseline data, some gigapixel cameras these are cameras that take hundreds or thousands of high-resolution images that you squish together to get a multi-billion resolution picture. We've done some lidar scans both on the ground from various sorts and then we've done near monthly UAV flights and once you sequence the trees out there because tree sequencing is getting down to you know in the $10 to $20 per tree range you can really measure phenotype environment and genetics in a way that gives you an amazing density of data about a space that we were never able to access before. So here's a camber on the southeast side of Australia the Arboretum is over on the west side of camber south of Black Mountain if you've been there this is our main field area this is a picture of what it looks like from one of the cameras up on the hill so this is a 500 megapixel image that we generate every hour and you can zoom all the way into that image to see the forest out on the far side there this is what that forest looks like when you fly over with the drone and then from the drone data you can bring that into Pix4D as I'll show in a second and get the three dimensional model of the trees. So the drone monitoring program the the goal was to test and develop a time series drone monitoring program so we could get 3d models of the trees growing we get time-lapse georectified image layers 3d point cloud and then phenotypes we can measure like tree height area is measured by top-down pixels and color data over time and this forest we're studying is it's 12 forests a spotted gum and iron bark was planted about 2012 and it's four hectares so it's a really a perfect size for drone monitoring because you can fly the whole site in about 15 minutes it's not too hard to process and it's it's very amenable to that sort of monitoring we've really come a long way on this so in April 2013 I flew the first drone over it which was a cell phone duct tape to a drone that I'd made at home and now we're using a really nice solid matrix probe by DJI with Daryl from Pro UAV who's been flying it for us and really good cameras and so we're really getting a lot of solid data but just in the last four years this tech five years this technology has changed so much the typical workflow in this is the Pix4D is a software that we use you it pulls in all the images you can see here that were taken from the drone and then you can see on the left all those green lines going down are key points that is detected in all the images and there's really a lot of black magic that happens behind the scenes in these software packages and you can see on the right those are some little tiny piece of ground that the software has detected as being the same in about 30 different photos and it uses that to calculate the actual position of each photo relative to the ground those are called control points from that it creates the 3d point cloud as you can see here and that gives you this sort of ghostly 3d model of of your entire forest you can see that there's some data missing from the bottom because the trees are the the drones are just looking down from the top and can't see around the edges of the trees the total processing time to do this is very hardware dependent so it ranges from like three to 12 hours but it allows you to go from from pictures to point clouds to three-dimensional models of the trees you can see here it's important to realize how groundbreaking this technology is because you know previously if you wanted to measure the location and height of every tree it would have taken you an incredibly long time there are 2,000 trees in this forest and we can do this now in about a morning but there are still challenges as Gerr was pointing out with working up with working with and serving up the data and so we developed we implemented a point cloud viewer called poetry on our website tradecapture.org you can go there and see some of the point clouds that we have online and we wrote a software package called forest utils that runs on Python that lets us pull out the locations and height of every tree and and the point cloud data associated with that tree so this this assumes an open canopy if you have a closed canopy things are a bit harder unless you have GPS locations for your trees but now once we have the the tree locations when the canopy closes we can keep tracking them and it outputs tree height top-down area location RGB colors and a point count which is a measure of how many how many points were generated for that tree and it also spits out a CSV map that you can just CSV file that you can just stick on Google maps or anywhere you want that has all of your tree locations so that that's the program we ran we probably flown 30 flights over the last since since mid 2015 and I want to talk a little bit about data management because it's this is really crucial and if you're you're planning to do anything more than just occasional flights you really need to come up with a good data management plan and you want to do this before you start your surveys and it's important to consider the entire workflow right because it's not just who captures it or what happens to it but the entire process if you think of who does the surveys if you have more than one company or more than one person that data has to get to you somehow it has to get processed you have to figure out where it goes on your computer and track it you know until people like turn have made us nice tools for having our data go seamlessly online you have to manage all this stuff yourself until it gets to the point where you publish it you have to figure out smart ways for naming things and you know that you may get a data set and then add to it and then add to it again and then process it and then you need some sort of workflow that tracks how that how that is taking place and doing that across thousands of images or hundreds of flights can really be a challenge and if people upload data you need to make sure that they've told you how much data they had so that you can have all of it so you don't spend a week processing their their data and then find out that they hadn't finished uploading it and you needed to add another hundred images and run the entire thing again often you run into problems like we stole all of our data on a large data server at the research school biology but we process it on a computer that's local and has an SSD so we have to move the data gets uploaded into one folder by whoever whoever took it gets moved to another folder which is the storage folder and then gets copied to this computer to process and then has to be copied back with the the new data in as well and so that that makes it quite hard to track things over time and you know if experiments fail or new data comes in you need to have a workflow for how you know where things are and what what the status of them is so it's really best info in force rigorous note-taking even just having people you know whatever tech is running the project writing down what they're doing as they do it can be really happy handy also shared Google Docs and notepad plus plus so you can put just files within each folder can be really useful so here's an example of the naming structure that we settled on and basically the idea was to make when someone looks in a folder to make it reasonably human usable so we have the year location site who captured it in the status in this case it was the National Forest and National Arboretum the ANU Forest plot actually this was pro UAV that captured that one and we wrote that it was done and uploaded so this seems like a great idea but of course if you have any sort of nested folders like you might want to name your dataset National Arboretum the file names get rapidly too long for windows in your entire process fails so this makes it a challenge because you need metadata but you have to have someone who actually is maintaining it and as I said shared Google Docs are good but this isn't really a cell problem and also you know everyone always ends up with files like this because before we implemented a data management plan things were just going on to my laptop and then getting copied into random places I think I think a lot of people get flummoxed by data management because it's really hard but it's not easy for anyone and it's easy to think that you don't know what you're doing but I'm not sure that anybody really does and you won't get things right the first time and you have to start with the plan and keep working at it and just acknowledge that it's not going to work the way you expected as soon as you as soon as you start to implement it and then you need to go back and change it because in reality our data data management typically looks something like that and we want to move it more towards the vision but it isn't actually there but if you need to address these problems because they end up making your data unusable when we have these large scale huge time series datasets some other challenges with processing drone data are that they're for this new these new kinds of three-dimensional data so for things like NDVI or some of the metrics that Kim was mentioning that there's a lot of known information about that because people have been working with that sort of data for a long time but a lot of data like a three-dimensional point cloud of a tree it's hard to know how you tie data values like that to biologically meaningful things and it's also challenging getting back to what Gerr was talking about with the the provenance of the data and tracking what's been processed you know you can you can process the same project in three different versions of pick 4d and get three at different outputs and there's also about a hundred different ways you can vary the settings so at one point recently we decided to test every single setting we could think of in picks 4d and you can see an output from that table here and using exactly the same images you can get a stitching time ranging from five minutes to 55 minutes and point cloud sizes ranging from 44 million points down to 11 million points and it's easy to think for example that more points means more data but if you happen to be taking pictures on a windy day and your tree is moving around you probably want your data doesn't have the resolution of a leaf it just has the resolution of the height and structure of a tree so it may be that either the less points is actually a better measurement of that tree volume or somewhere in between but it's really hard to ground truth these things because there isn't any way to go out and measure that volume of the tree anyway else it's also important to choose the right tool for the job so drones are really great people are using them for good reason with lots of things but they're best suited for smaller areas like maybe less than 20 hectares and a good example of this is that because we've been flying the arboretum so much we would all wanted to do a full arboretum survey so that we could get a 3d point cloud and model the entire arboretum and measure the height of every tree in the arboretum I think it's about 35 thousand trees that they have out there so we finally got the funding to do this we got Daryl and to fly it and it ended up being a huge project it took many weeks of planning there were four or five flight days required you have to have multiple staff on site because when you're flying an area that large you can't fly over people so it's easy to follow castle regulations when you're just a remote forest but when you're trying to find fall fly over a 250 hectare area that's open to the public it becomes much harder we ended up with 8800 RGB images and more than 12,000 images from the sequoia it ultimately took about two months of manual processing because we had to break everything into smaller subsets because the full cloud couldn't run on any machines none of the online folks can handle more than about 500 images so we couldn't just throw it at one of the cloud services and we ended up with a point cloud that had 584 million data points so it is important to consider what workflow you will use so this is something we know we could probably do this once a year max but if you add up the time cost of it it becomes prohibitively expensive to do this at this point with the technology available and for something like that it might be better to fly a plane over the Arboretum for instance it's also if you're thinking of setting up a monitoring plan you need to consider weather and distance to the side and accessibility and time of day because if you want to fly your all of your plots at the same time of day say around noon you have to drive between sites you can't do that on the same day and sometimes it turns out that just putting a camera in so you can get consistent data even though it's not as maybe high resolution might be a better option or using satellite data and again for example a lot of the local agricultural monitoring it turns out it's cheaper to do it by helicopter because you might want to fly five flights fly five sites in the same day and with a helicopter you can do that in about an hour it might cost you a couple thousand dollars but that's cheaper than trying to drive around to five sites over the course of a week and find a drone and you can put a much heavier payload on a helicopter so this is the point this is the point cloud we got out of the Arboretum again if you go to trade capture you can you can go around and explore it it looks pretty good but you can see there are some artifacts where the the stitches didn't align up perfectly because we're doing had to do it in pieces so get on choosing the right tool for the job larger scale surveys like are not necessarily best to do with a drone and there was an interesting article from frontiers in the ecology the environment last year where they looked at doing a UAV surveys in Alaska and it turned out that it was $1,700 per site to use a UAS but only $400 per site to use a plane mainly because the the sites were quite far apart so it's important consideration now in the future I think what we all want is self-driving small drone swarms that fly over our forest or field daily we have all this data upload and process in the cloud and get near real-time analytics to the farmer or the end user and I've been wanting someone to release something like this for years and finally this year a country called X aircraft in China and a group in Sydney called Revolution AG are starting to release the system so here's an agriculture UAV it can carry pesticides or water or nutrients fertilizer and they can fly themselves so of course there are regulatory issues around this but we're getting close to the point where we can have a swarm of drones that's just monitoring our sites continuously and in the future as micro drones get get better and higher quality they may be more feasible and safer particularly for field monitoring where maybe there aren't going to be people there because you can imagine a small a small city you pay $10,000 and get 500 of these drones and they take off every day and fly around your forest and come back they're all solar powered they live on a tower somewhere that would be a really amazing processing solution to give you continuous three-dimensional data of your of your research site so some thoughts on developing a drone program drones and sensors are crucial component for field monitoring and phenyl typing but it is hard to do there's a lot of technology available so you want to focus on the deliverables who are your stakeholders or customers and what do they need to know what would be actionable information for them it's easy to say oh yeah let's just get a drone and then realize afterwards that you're not you haven't really figured out what kind of data you're gonna deliver deliver and then working back from that you can figure out how big the area is what the full costing is and so on I think I covered a lot of this and then also you want to start small and get your pipeline working so you don't want to start offline the 250 hectares of the arboretum you want to have done the smaller for us first you want to make sure the UAV you buy is the correct one and whether or not it might just be cheaper to subcontract and when you process the data you need to decide if you're gonna outsource it or do it yourself and if you need to develop novel tools are there groups already doing this stuff and how can you share this with others so that we all have to good tools to use for processing sorts of data and again don't underestimate how hard data management is and have a plan in place at the start take we're running out of time so I'll run through this really quickly to I just want to say that all of this data is incredibly it's new and it's incredibly hard to manage because it's three dimensions and it's got multiple layers and time series and we don't have tools for this that that we used to like the tools that we normally use for data visualization management are not usable for the tool sets that we have now so we need a sort of MATLAB Excel or GIS for time series three-dimensional hyperspectral data and this doesn't exist yet so there's a couple groups that the NCI that are developing point cloud viewers there's a we're working with the vis lab to make a time-lapse virtual reality and windows based point cloud viewer and if you if you search for Adam steer at NCI he's working on a NCI back one which would be really great because then groups like turn and us could dump all of our data in the same spot on NCI and we'd have these real-time tools for pulling point clouds out on the fly and just the last thing I'll go through this quick because I know we're out of time one thing that I wanted to do once I start getting three-dimensional data is be able to visualize all the sensor and point cloud data on the landscape where it was collected because that that's a way to me that helps helps you really see the the data in the context of of where it was collected and and the place that you're monitoring so I collaborated with some groups with some students in the computer science department and we made a virtual reality three-dimensional model of the National Arboretum using all the drone data as well as our sensor data essentially what you can do is you can take three-dimensional modeling software developed by Hollywood and use it to reproduce your landscape but rather than using the whatever data they're using for a movie you can use your data from your drone flights to generate the three-dimensional models and so this is an example of the the project for the National Arboretum here the drone the drone flight data there's the three-dimensional force that we get out of it this is the virtual reality version of it where we've taken the digital elevation model to have a generate the landscape and then put the trees in at the locations that the drone data measured when you interact with the trees they show you their metadata so in this case we're showing height and area of each tree and then you can also map onto the landscape and play back in time series the the different data types that we're collecting with the mesh sensors so this is a really great tool for pulling all of these dense data layers together and visualize them in one spot to help us make sense of this incredibly complex data and I think we should all consider this is just the beginning right so when I was a kid I used to play Atari and this is what it looked like and now this is you know the same dragon games that my kids play and we're at the Atari stage in VR and in our ability to measure the world continuously in 3D and in 10 years VR and AR are going to be indistinguishable from reality so the question is what do we do with these tools and how do we create the next generation of interfaces that facilitate ecosystem research and how we build monitoring and monitoring programs that make best use of all these data types so we can really model our environment in three dimensions and solve the grand challenges of this century. There are lots of people to thank but I think I'm out of time so thanks