 Think Tech Hawaii, civil engagement lives here. Good afternoon ladies and gentlemen, my name is Kalani Donas Rivera and today I am your special guest host on where the drone leads. So today I am a member of the University of Hawaii Drone Technology, the program manager and today we will be speaking about our project here at the University of Hawaii and how we are reaching out and making a name for ourselves nationally and internationally. So before I move on, let's move on to our main host, Mr. Ted Rolston. Thank you Kalani for coming on the show and David Harris, let me introduce you in a moment and Austin a little bit too and thanks very much Kalani for stepping in in this quite emergent situation here and taking over as guest host. We had kind of a harrowing experience coming down here to the studio as a result of which I want to have to take off at our first break and go back and take care of some not aviation problems but some land transportation problems that could have ended up in some crunch metal. Fortunately didn't and thanks to you guys for pointing out where the handbrake was and we're not fine but anyway, I normally host this show and certainly welcome you guys on board. In fact anybody who's here can take this show anytime you want. Even when you're far away at Cornell you can do this show by Skype. Kalani is a veteran on the show, you came on a couple of years ago. Yes sir, this is my third year on the team. I started off as a sophomore in the image processing subsystem which Austin will soon go over and each year the next year I stepped up to the aircraft subsystem leader and this year my senior year in mechanical engineering at the University of Hawaii I am the program manager or the big boss if you want to call it that. I don't like to call myself that but okay, the Ayatollah so to speak. That's what he is. All we should do is introduce the basic subject of the concept of a competition you're going after for search and rescue with drones becoming the means by which you get there which is why you have to have signal processing, why you have to have image processing, why you have to have an airplane, why you have to have a VIP system to tie it all together. So why don't we turn to David and ask him to kind of lay out the big picture. You've been on this thing for about three years now. Yes sir, this is my third year on the team and we do a plane design for search and rescue. Airplane design. Yes, plane design, airplane design for search and rescue, specifically designed for the AUVSI competition that takes place every year in Maryland. It's designed to fully simulate. So we go from Manila to Maryland to this competition. To go to this competition it's at an Air Force Base and they couldn't get permission to do it here because they tried and also it would have been very expensive to send all the teams out there. So we compete every year, it's a worldwide competition so international and national teams. Last year we placed sixth internationally and we were third in the nation for our airplane and the whole competition is designed to simulate a full search and rescue mission, whether that be going to find the person, searching the area for the person, delivering a water bottle to that person. By drone. By drone. And getting harder every year. And it always gets harder. Right, okay, so your success last year, you can't rest on those laurels very far because they've changed the competition, made it more intense and so this forces the team to dig in harder. That is true, each year is a different animal, they throw different challenges our way and it's our responsibility to respond to those challenges and develop, as engineers, the best method of solving these problems in a cost-effective manner while not compromising for performance. That's such an incredible learning experience because it's something that you won't, it's really hard to get in an industry actually to see it all come together in that way. And let's turn to Austin. You're a programmer on the first year on the program and you're kind of at the core of this thing, the image processing to find that victim. Tell us what your perceptions are on how this whole coming together and how it's working out. Well, it seems like it's been going pretty well. We've definitely had our technical challenges. I've personally been working on image recognition, trying to detect targets on the field and we're using artificial and machine learning is the current term for that. It's been a challenge, we've been using TensorFlow and it's definitely been a learning experience. Okay, so talk about what the signal processing or the image processing is all about. What exactly are you trying to extract from the surface pictures you're taking with a plain old camera? Well, at the competition we're going to have targets cut out of plywood in various shapes, stars, circles, squares. They've got colors painted on them and then a letter or a number in a different color painted on it also. And our objective is to identify the GPS coordinates of these targets, the color, the shape, the letter and number on it and the color of that letter and number. So you've got basically from that complex picture that the camera takes with grass and runways and people and cars and targets, you've got to have the automatic extraction on that target that's important and ignore everything else. Yes. Find a location of it and push that forward to the judges. That's what's going on here, right? Yeah. We have all of that fully automated and that's kind of a challenge. Yeah. Well, that's okay. But that's really cool because that's a really high level of performance as expected. This isn't just flying a drone or the camera on it and then looking at the results after the flight. You've got to do this in real time. Yes. And this is kind of what drones are all about anyway. This is what they should be doing. And just imagine if it's hard this year, what's going to happen next year in the competition? Especially because we're losing our big boss. Yeah. And you should talk a little bit about that. Where are you going after this? So I want to point out three programs such as this one. It opens the doors to different opportunities in life. I've been on this team, as Ted mentioned, for three years and I've learned to love UAVs, especially the controls aspect of this. And 310 and his mentorship and our project, it has opened the door for me to attend graduate school. So I will be leaving the Aina, if you will. You're leaving. You're leaving Hawaii. Yeah, I am leaving. And how far are you going? I'm going from this little island in the Pacific all the way to the northeast of the United States. So a little town called Ithaca, New York, and I'll be attending graduate school at Cornell University for my... So you're in the PhD program at Cornell via the VIP program here? Yes, sir. I think that's why they chose me. Kalani, congratulations. That's impressive that you're saying that and if that's the truth of that, that the VIP program has that much value. That's something you can look up to and you can look up to as well. So this was cool. At this point in time, let me just ask if I can thank you once again for coming on being the host here. The program's still rolling and the two of you guys will get you back again some other time. I've got to run and take care of an automobile problem that's a real problem right now. So we're going to take the first break here. I'm going to disconnect, back out, and turn it over to you, sir, for the show. Yes, sir. Thanks for coming on. Thank you. And David and Austin. Thanks for having me. I'm Pete McGinnis-Mark and every Monday at one o'clock, I'm the host of Think Tech Hawaii's Research in Monart. And at that program, we bring to you a whole range of new scientific results from the university ranging from everything from exploring the solar system to looking at the Earth from space, going underwater, talking about earthquakes and volcanoes, and other things which have a direct relevance not only to Hawaii but also to our economy. So please try and join me one o'clock on a Monday afternoon to Think Tech Hawaii's Research in Monart. And see you then. Hi, everyone. I'm Andrea Gabrieli, the host for Young Talent's Making Way here on Think Tech Hawaii. We talk every Tuesday at 11 a.m. about things that matter to tech, matter to science, to the people of Hawaii with some extraordinary guests, the students of our schools who are participating in science fair. So Young Talent's Making Way every Tuesday at 11 a.m. only on Think Tech Hawaii. Mahalo. Hey, aloha, everybody. Thanks for joining us on Think Tech Hawaii. I'm your host, Andrew Lanning, the security guy. I host a program called Security Matters Hawaii. And I hope you'll join us on Fridays. We are at 10 a.m. And we're going to be talking about those security things that really should be important to you. And maybe get behind the scenes on some things that you may not know about the industry or about products or even about your habits. Security's all about people, processes, and products. We hope to bring that to you in an informative and hopefully a useful way. So again, 10 a.m. on Fridays. Security Matters Hawaii on Think Tech Hawaii. Join me. Thank you. We're back from the break. Unfortunately, our main host for where the drill needs it, Ted Rausen, had to take a break. But I am here as the guest host. I'm back again. Hello. And in the first segment of this show, we mentioned VIP. And for those of you who don't know what that stands for, you might think it's a very important person. But that's not what VIP stands for. In our case, VIP is a worldwide consortium established by Georgia Tech University. And it stands for Vertically Integrated Project. And what that means is that it's a program, a learning program in which students can be involved in projects at the university level and the undergraduate level, from freshman level, all the way to seniors. And what that allows is that that student carries on the project from the freshman year, carries over to the sophomore year. And as they grasp more and more knowledge in their subject, they can contribute that to that very same project and then have that be their capstone, project, or contribution to the world once they graduate from the undergraduate program. So at the University of Hawaii, we have a number of VIP projects. And our team, our group, University of Hawaii Drone Technologies is one of them. So actually, I want to give a brief history on UHGT. UHGT was founded roughly four years ago by two electrical engineering undergraduate students, if I'm not mistaken. And since then, their goal was to expand the capabilities and applications of drones or unmanned aerial vehicles in society. And what we have been pinpointing is the area of search and rescue. So in this past year, there has been a series of hurricanes, natural disasters, and people have been displaced or in harm's way. And that has been a challenge because with so many people in harm's way, we need to search and rescue and address their needs. So one way that we can do that is by implementing that technologies in UAVs or drones, which we have been doing for the past three years. And I'll pass it on to David to give a brief overview of how we've contributed to search and rescue. Yeah, thank you, Connie. And so for our system, we do basically three pieces. And so we have the mechanical side, which is our aircraft subsystem. And the aircraft subsystem is in charge of taking the actual airplane and modifying a pre-existing, something that you can buy online. We shop online. We find an airframe that we like. We buy that airframe. And then the aircraft subsystem, the mechanical engineers on the team or those just interested in the aircraft subsystem, they are in charge of taking that airframe and modifying it to fit the needs of the electronics and the image processing subsystem. So those are our other two subsystems. But the aircraft subsystem does stuff like they install a box in the bottom that lets us drop a water bottle on a location. Accurately, they design the model for the autonomy software that lets us drop the water bottle to get it perfectly on location. They also are in charge of protecting the airframe. So for example, this year, our mechanical subsystem, our aircraft subsystem, designed a landing skid with the help of Ted Ralston, designed a landing skid that allows us to protect the airframe on landing on hard ground. We really struggled with certain parts of the plane breaking because we wasn't protected enough from stock because we've increased the weight of the airplane so much with all the systems that we have on board. Our electronic subsystem, they are in charge of all of the electronics on the plane. That's making sure everything gets powered correctly. That's making sure there's no issues with interference on the plane. And then they're also in charge of all of our radio systems. So this year, our electronic subsystem, one of our members, or a group of our members, designed our custom power distribution board, which includes power over ethernet and lots of different voltages and ways to power the pieces of the plane. And it's been future-proofed so that we don't need to worry about developing another one in the future. We should be set for whenever we need to move on. And he's a great example of why the VIP nature of this team is so important. Because the member who was in charge of designing that has been on the team for three years, and he learned from the person last year who designed a PDB. And we really encourage people to join the team early. And so if anyone's listening is interested in anything that I'm talking about, when you get to UH, talk to Dr. Sharoma or Dr. Trimble, who are our faculty advisors for this project. And they are both important members to our team in the fact that they are what allows us to even exist. They are professors, and they're very informative on everything that we do. And they really encourage us to go as far as we can. And they help us get to that point. The last system, which is what we're going to be highlighting today, is our image processing system, which we talked a little bit about earlier. But we're going to go into a little bit more detail. And the image processing system is basically in charge of taking the images and actually processing them. Austin here is a integral member of that subsystem. And can you give me a little bit more detail on what you specifically do and what the workflow is for the image processing team? Sure. Well, the overall workflow, I guess, starts with the plane. We have a camera on the plane that takes pictures of the target directly down. And then we have a Raspberry Pi that transmits those pictures down to our ground station laptop over a Wi-Fi link. Once it's on the ground station, we have a couple programs running. One of them builds a database of GPS coordinates. And then the other one processes these images. First, it runs these images through an OpenCV blob detection algorithm that pulls out what it believes are targets that kind of stand out from the background. Once we have those blobs, we run them through a TensorFlow machine learning algorithm that tries to identify the shape and also filters out some of the false positives that OpenCV gives us. After that, we do color recognition. That's also done with OpenCV. We try to figure out the color of the shape and also the letter. And then we send that to another script running on the same computer that communicates with the judge's network and transmits these images and all the data that we've determined to the judge's network. So Austin, you made all that sound extremely easy. As a program manager, I know I've seen you guys struggle and push through obstacles. So can you provide a brief description of what machine learning is and why it's so time consuming to do so? Well, machine learning, generally, is a very complex subject. And I cannot say I'm an expert on it in any way. But we're trying to use TensorFlow, which has been developed by Google. And we're also using their inception algorithm. Actually, it's a neural net. And they've developed this for object recognition. It was originally developed to compete, I guess, in this ImageNet challenge, where they try to categorize 1,000 different classes of images. And to give you an idea of how effective Google's inception model is, they actually beat a blogger not only in speed, but also in accuracy. A blogger tried to categorize these various categories of images. Some of them are simple, like a car or a cat or something. And others are fairly specific scientific categories of animals that I personally would not be able to identify. We've been using transfer learning to retrain this algorithm, this neural net, to identify the categories that we're trying to, the categories of the targets, the shapes. Well, that is incredible. So I know David just gave a brief overview of each subsystem. So the hard part about this project is that we have each individual subsystem addressing different components of the mission. The mission consists of a waypoint navigation in which our aircraft, our UAV, navigates autonomously to preprogrammed waypoints. And from that, it does a payload delivery section in which it delivers the payload. In this case, an 8-ounce water bottle to the target that has been provided. And then from that, we were given a search area in which we're given an area. And there's targets in that area. And it's our job to run a search area pattern and to locate and identify those targets. So that's where image processing comes into play. Payload delivery is where the aircraft subsystem comes into play. And the electronics makes everything happen. So the hard part about all of this is integrating it and making sure each individual subsystem works as a whole. So this past semester, we've done extensive testing down at the Mars, the Kauai Nui Mars. And David, can you speak about our testing and the different challenges we face? So for our testing, we have to do a lot of troubleshooting, let's say. So when we get out of the Mars, when we're in our lab, we really don't get to see exactly if everything's going to work in the air. We can test as much as we want on the ground. But as soon as we get up in the air, that's when we actually find the problems. So just a brief story of what happens when you don't prep sufficiently on the ground. So as you know, a plane has a bunch of control surfaces that allow it to fly in the air. And one of those allows it to control its altitude. So that's what tilts up to let it level off or fly up or down. And that control surface, it turns out, if that control surface is backwards in your autonomy software, when you try and take off, the plane does a nice loop-de-loop straight into the ground. And this kind of stuff happens at the field. And it's one of the things about UHDT that I think is exceptional is that we are able to turn around from stuff like that, have a catastrophic failure where the airframe is destroyed at the airfield, and then we go and get it back up and running in two days. If we don't crash at the airfield, then we are taking off, we're flying, and we're doing full mission tests. We're taking off, or we are not taking off. The plane is taking off because it flies itself. The plane takes itself off. It gets up in the air. It goes, and it does waypoint navigation. We tell it where to go, and we see how accurately it gets to those locations. Then we approach the drop location. We find where exactly they want us to drop. And we drop whatever our payload is. This year it's a water bottle. Last year it was also a water bottle, but we had to protect the water bottle. This year the water bottle has to break. We drop the water bottle on a specific location. We've been getting it within 30 feet of that location. Then we have to go and search the area, and the image processing system. Receives images down from the plane. We run it on our ground station laptop. And at the field, we're running all of this. We're doing the entirety of the mission at the field. Well, after we tested each individual system at the field, we've been doing the entirety of the system at the field. For the last month or two, we've been running full mission tests at the field at the Kauai Nui Marsh. So we spent the entirety of this semester integrating each of these individual subsystems and testing them until they work effectively. We will be attending competition in June 15 in Maryland, Patuxent River, Naval Air Base, Maryland for competition. And through our testing, we are confident that we will perform better than in previous years and make Hauai proud and show that the UH is not a force to be reckoned with. And so I want to go and provide an overview of this project. I didn't mention how many members we have on this project. We actually have a total of 22 members in our project spanning from multiple disciplines, electrical engineering, computer engineering, mechanical engineering. And that is something incredibly beautiful because in industry, this is something that is expected of you because you are not be working with people only from your major or people in your classes. You need to learn how to get along and communicate with people of other disciplines, other schools of thought to be able to make the overall system work. And I think that's something that's extremely beautiful about this project. And its scales are transferable not only for this project but out in industry. And that's something that I know employers are striving for. And this VIP project is an excellent example of that. And I know that we have focused on this competition, but we hope to soon take these capabilities, these search and rescue capabilities out into the real world and to our local community. And David has a really good idea about how to apply them in the years to come. So I have been, after discussing this with members in the community, I have been given examples of situations that we can use this in. And there is funding slash opportunities to be able to use this for a variety of means. The first person who approached me interested in using the system was the ex-fire chief of Kauai. He came up, he saw the system that we had and he was very interested in using this system to actually do search and rescue, working with the lifeguards to be able to help with the ever-present problem of people drowning off the coast of Kauai. Our system would be capable of flying over, identifying people who are in the water, identifying people that are too far out and need help. And he's approached us thinking that we might be able to help him do that. And next year we plan to continue that farther. We're going to try and see if we can actually do something like that next year. We've also been approached by the Coast Guard, who are very interested in someone who is able to drop, with high accuracy, radios and radio batteries. When they do search and rescue missions that are in the mountains or in a forest, not necessarily in Hawaii, the example that was given to me was, say, in Alaska. They know someone is lost or a plane went down. They don't know exactly where, but a plane wouldn't be able to find them. So you have to go by foot. But if you're going by foot, they send out their search teams, but they can only go so far before their batteries run out. So having a system that is as cheap and usable as ours to be able to deliver batteries and radios to locations where their search and rescue workers are working would be ideal for allowing them to actually be able to find the people. Because they have to turn around. As soon as they can get radio connection, it's not worth continuing. They have to turn around in order to save their own lives rather than having more people be lost in the woods. And the last group who has expressed interest in our system is the Marines. They have a specific need of they want to be able to deliver something onto a location, let's say, or from a boat onto an island. They need to be able to deliver something to that location with high accuracy and be able to drop something onto location and then be able to fly back nearly unnoticeable. The beautiful thing about our system is that if we turn off our radios, we're made of styrofoam. Our airframe is made of EPO foam. And foam is invisible to radar. It's invisible to most sensing methods because of the size of our aircraft. So above being able to accomplish the task, we also have the advantage that we are invisible. If we fly high enough, we are unsensible. Currently, that's, of course, going to change. But currently with the system, the Marines also expressed interest in our capabilities. And that's not even including all the people interested in just the image processing system alone. With the development that we've done, people have approached us just looking to get a piece of that image processing within our ability to identify stuff from the air. So that is incredible. So I know when you think of drones, you think of either incredible camera shots or a really expensive toy to play with. But I think as a team, as an organization, we are testament that these drones can be used for so much more. It can be used for saving people's lives, for doing research, for expanding what we know of our terrain through image processing and whatnot. And I want to express that if you're interested in this project, as David mentioned, contact Dr. Wayne Sharoma or Dr. Trimble at the University of Hawaii at Manoa. Or if not, feel free to run into David. He's always around, homes hall, the engineering building, and if you see him, just say, hey, I want to join your team. And I want to point out you don't need to have any previous experience in RC or drones at all. I came onto this team not knowing anything about drones, and through my three years of involvement, I've been exposed to so many things. And as I mentioned before, that has opened the door for me to explore even more. So on that note, thank you for tuning in to Where the Drone Leads Today. I apologize. I'm not Ted Rauston, and I am only the guest host for this segment. And once again, thanks for tuning in.