 Can you hear me out there? I want to see Marines in the back. Give me a thumbs up if you can hear me. Roger, thank you. Okay. Please, gentlemen, take your seats. Thank you. Reminds me of my days in the basic school when we wouldn't make a move unless somebody told us to take our seats. First of all, my name is Frank Kelly and I am extraordinarily proud to be able to be the chair for this panel, focusing on unmanned systems, but more importantly, focusing on the human machine team. Okay, I don't think that you're going to find another panel in Sierra in space that is going to talk about human machine teaming an incredibly important element to what we see as our future in unmanned systems. So one thing that I want to tell you right now, here's the good news. We want you to use your devices, okay? You are not going to hear me say turn them off because you never do anyway, right? Nobody ever does. What I want you to do is I want you to take a look at this new app. Yes, we're leading with our faces here today, okay? I want you to take a look at this app and we're going to experiment a little bit today. Folks at Sierra in space have asked us to utilize this app to where if you feel like you can't get to a microphone fast enough or you would like to remain anonymous, we're going to use this app to help us ask some questions. And we've got some folks, some great folks that will be queuing up the questions for us. And really, again, I can't say enough about the folks here at Sierra in space. They actually came to us and asked us if we would like to participate. We seized the opportunity because in our particular shop for the Assistant Secretary of the Navy for research development and acquisition, a new deputy Assistant Secretary of the Navy for unmanned systems to be able to participate this early because the evolution of our office is unique and we are really proud to participate. I've got a couple of people here and I know that everybody else is probably going to introduce their folks, but they're right here, right here in the front. We've got Commander Carol Appoint, who's my chief of staff, Dorothy Engelhardt, who helps run programs. We've got my future's guy here, a guy by the name of Christel Mastrow, and Andy Vansiak, away in the back. Andy probably didn't get to sleep at all last night because he is absolutely... Anxiety is probably...anxiety ridden that he wants to make sure that this thing gets pulled off. I can see some other folks in the audience, by the way, that have really supported us and really good to see you out there. Greg, thanks for being here. I am going to assume that you are here because you have an interest in unmanned systems, not that you didn't have anything better to do with your time. We need you to engage. We want you to participate and your questions, quite frankly, are going to help us form where we go in the next couple of months as we build our unmanned future for the Navy. So we've got a distinguished panel. I've got a distinguished panel today and I'm actually going to start at the end with Colonel Jenkins. One of the things that you'll know about Colonel Jenkins, one thing that we really pride when we talk about Marines is you've got a complete MAGTF officer there. Colonel Jenkins is a 1990 graduate of Cornell University and I only know of one other Marine that is a graduate of Cornell University and that is our assistant commandant, General John Paxton. He is the smartest man I know. So Jenks, up to you to prove me wrong about that if you want. He's a WTI, a weapons tactics instructor. We send our best and brightest to WTI. He's got several Mew deployments with 2.2 and 2.4, served several tours in the Med. He's been in aid, which is no easy task. Let me tell you that. He's been a WTI instructor. He was a CO of HMLA 269 and also MAG 29. He's been to AWS and also the Army War College. He's got a master's in international relations and he also went to the Army War College. Currently, he is the director of science and technology at the Marine Corps Warfooting Lab and Jenks. I'm really happy that you're here. Thanks for being here today. From the Coast Guard, our sister sea service, we have Admiral Neto. He is the assistant commandant for capabilities. No small job, I'm sure, especially for a one star. He's from Maine and he's a 1989 graduate of the Coast Guard Academy. He has had tours as the chief of inspection, the senior investigating officer, the CEO of the safety and maintenance chief, the office of design and engineering and standards, and he's been the head of delegation at the International Maritime Organization. So to me, that sounds like you are the face of the United States Coast Guard globally, which is, again, no small task. And it's no surprise that he was the United States Coast Guardsman, engineer of the year in 1998. It's really a pleasure to have you here as well, sir. General Vinnie, colonnades, okay? And if there's anybody here that we can say is sort of like the godfather of the panel. It could be general colonnades, okay? He is the assistant deputy commandant at INL working for Lieutenant General Dana. He is a warrior and a complete MAGTF officer as well. He hails from New Jersey. He's a 1983 graduate of Indiana University and also a graduate of probably one of the most famous and challenging basic school classes that we've ever had. It was Gulf Company 783. A pleasure to have you here as well, sir. He's a Motor T officer. Served time with the Wing as an engineer. He was an Opso during Desert Shield, Desert Storm. Served in Operation Enduring Freedom, Operation Iraqi Freedom. He was the commander of CSSD 78, a service support group for Mew, 2-4 Mew, and also the commander of CLR 27. There's a number of firsts that general colonnades has remarkably stood up for our Marine Corps, and they all involve surrounding SOCOM and a remarkable feat to be the first Marine and also the first Marine J-4 as well. So, sir, thank you for being here as well. Admiral Bob Garrier. I work closely with Admiral Garrier. He has suggested that I call him Bob. I just can't do that. You're Admiral Garrier to me. We work closely with his shop, and he's got a number of his folks. As a matter of fact, when I look at this front row right here, with the exception of Steve Kwan, our AP guy over here, it is Dazin on Man and N99. And to be quite honest, he's not surprising to me at all. He's a 1983 graduate of the United States Naval Academy with Merritt. He's got a master's in international affairs, a master's in marine affairs, and a master's in public admin from the Monroe School, the Kennedy School of Government. And he specialized in negotiations and conflict resolution. And I can think of nobody more qualified to spend time in the Pentagon at this time, sir, than you. So I didn't know that about you before, and now it has all become so much clearer to me that I know that you have those special skills. His commands are absolutely incredible. Commander of three strike groups, and carrier strike group seven, Ronald Reagan strike group, and also Nimitz strike group, carrier strike group 11. Absolutely great to have you here as well. You're also the author of several books, and you can tell that this is a man who values leadership and values leadership in teaching our young naval officers on how to do their tasks. So that's a responsibility that everybody wishes they could take on, but few have actually done it. Our last guest is Dr. Peter Singer, and he's a strategist and senior fellow at the New American Foundation. Do a Google search if you want on Dr. Singer. And you will find out if there's a list of 100 great thinkers in America today, Dr. Singer is on that. The Smithsonian Institute, National Portrait Valley, 100 leading innovators, Defense News is one of the most, 100 most influential people in defense, and Defense News top 100 global thinkers. Wall Street, the Wall Street Journal, which I'm sure many of you read, states that Dr. Singer is the premier futurist. Absolutely remarkable in terms of the way that Dr. Singer thinks and the way that he compels us to be thinking as well. But the one thing that I do want to thank you, sir, and that's the time that you take to appear with what is primarily a military panel, the time that you've taken to appear down at smaller venues, down at Quantico, the work that you've done for us at the Pentagon, hopefully not over the top, but there's many ways to be a patriot and the way that you make us think, I think, is a cult of patriotism. So ladies and gentlemen, I'm going to tell you right now that you've got a remarkable panel. Please take advantage of the panel and ask some challenging questions. So what I'm going to do is I'm going to tee up, this is the order that we're going to go in, but I'm going to tee up some slides for you to stand up, but it was about a year ago that the Secretary of the Navy at this particular conference at Sierra and Space decided that he was going to create two new organizations within the Department of the Navy, and it was a new Deputy Assistant Secretary of the Navy, and that's our job, the one that I'm quite proud to be in charge of up in the Pentagon on the Secretary of the Navy staff, but also N99, not NAV Resource Sponsor. And when you have a Resource Sponsor, that means that you as a Department are very serious about what you're about to do, and that's what Admiral Garrier is doing. The things that are really, really important about that, though, are the fact that the Secretary told us to use our imaginations, and you don't necessarily get that, get permission from your highest leadership to use your imagination, and I will say that both new organizations and with our, with the people that we work with and for within the Deputy, within the Secretary of the Navy and for the Resource Sponsor side of the House, we're all doing that a great job. Now my boss is Mr. Sean Stackley. Mr. Stackley is incredibly involved. We were given some tasks in terms of developing a roadmap, developing goals for the for the Department in terms of where we were going to go with our unmanned systems, and Mr. Stackley is deeply, deeply engaged in all phases of that task. Next. Okay. Some of the themes that you're going to hear today, but these are the themes that we've had a chance to talk about in our office, and we get a chance to speak. We've emphasized this human machine teaming. That's obviously the thrust of what we're going to be talking about today. It's not a one-for-one replacement. We had a chance to do some media events. One of the things we wanted to emphasize to the media is that we are not trying to replace Marines and sailors with unmanned systems. That is not the goal. We need to optimize our use of unmanned systems and think about autonomy as well when you think about optimizing our use. To what degree are we going to allow these systems to be fully autonomous or remotely operated by Marines and sailors? And one other thing that I'd like you to think about is we need to get out of this this paradigm that of domain, that only aviators can talk about unmanned aviation systems and that only submariners can talk about things that dwell underneath the water. The biggest thing that we like to emphasize is that unmanned systems are going to fundamentally change the way the Department of the Navy operates in the future. You could probably provide some examples of where it already has. I love this picture. But our future, the future that we envision, is where Marines and sailors are working side by side with unmanned systems. And I think Jinx has some footage a little bit later on that will go into a little bit more detail on that. Just going more on the not a one for one replacement. We're not trying to take those things, those missions that we used to do with Marines and sailors and now going to just do those things with unmanned systems. We're going to open up our eyes and think about new ways, new missions, new ways to operate. Next. Some of those new ways are going to be in swarms and we can utilize the elements of war fighting like mass and surprise with considerably larger numbers than we would ever thought about previously. And that would be, I'm not talking about tens of things. I'm talking about thousands of things being able to mass and surprise. Also being able to fight in areas where we may not have fought in the past. Underground. Maybe on the surface We really need to think completely differently about the way we've waged war and where we've waged it before. Next. I like to slide. This is just one way to think about the domains. It's not about the sky and the surface. It's not about a jungle environment. It's not about an Arctic environment. It's about the new missions and utilizing those environments to maximize the capability that we're going to have with unmanned systems in the future. Next. Throughout our history we've had a variety of ways to modify and modernize our warfare and this just shows a progression of where we are today. It's a cultural issue and even just mentioning that it's a cultural issue is something that's tough for everybody in our department. It's a human thing to challenge our culture. But this is one of the things that we have got to realize that has to be done. We do need to change the culture. Next. Thank you. Hey, I'll tell you what. You're going to hear this today too. Trust is a critical element of what we do. It's trust in the systems. It's trust in the people that are going to develop these systems. It's trust in the people that are going to actually operate these systems. But almost more importantly, it's trust in the system itself. And someday we might be able to replace that particular image of a marine and the dog with an automated system. We need to be prepared for what these relationships are. These war fighting relationships. These relationships between marines and sailors and the machines that they're going to fight with and alongside are going to go. Okay. Now, I've already introduced all our panel members. I'm not going to introduce them again. Colonel Jenkins. So typically when I'm in a position like you, Jinx right now and I'm the junior guy on the board and we've emphasized that quite a bit as I'm sure has become conspicuously clear. My response to that is first to fight and in your case, first to brief. So Jinx, take it away. Thanks, sir. So, sir, first of all and to the Navy League, thanks for having me. I really appreciate the opportunity today. As I was thinking about this, I really focused on the word evolution. And to me, evolution implies change and hopefully progress. And so with that in mind, I wanted to start with some very recent past of what we've been looking at and have really focused more on where we think we're going with respect to unmanned systems and with respect to autonomy. So as an opening gambit, this is what we see as a state of unmanned systems today. As Daz and Kelly said, they're not a one for one replacement but unfortunately by and large they require an operator to constantly manage the system and run the system and also be ready to assume control of the system or of the task at any time. And so you've got a one to one ratio and they're really not multiplying force at this point. They are creating standoff. They're creating force protection for those operators. But at the end of the day what we found is when the bullets start flying the marine either becomes so absorbed in driving the robot that he loses SA to what's happening around him or he drops the controller and becomes a rifleman again. And so we want to move past that. Next slide. Expeditionary Force 21 is the operating concept that the Marine Corps involves. And what we really see this supporting and where we need unmanned systems to go is systems that are taskable in the same manner in the same way that a squad leader or company commander would task humans that work for him or are in support of him. They need to be functional and sustainable in a distributed manner. And then they need to to really again work as a team and be functional in a dense urban environment which is probably the most challenging environment for these things, especially when we talk access to GPS signals and things like that. So at the lab we've done a couple of things to try and start moving away from that remotely operated vehicle and towards autonomous systems that will accomplish tasks as part of a team with people. So next slide. The first one we're talking about is GUT Spot, you see that earlier. What this is basically no more than pre-programmed responses. So the robot knows go into the building, take a look around, report back, and then squat and hold. Great if nothing unexpected happens, but we know in war that's likely not to be the case. We can expand that in time and space with something like GUT. So a vehicle that I can program with a route that I can program to go to certain points along a route and on-load and off-load supplies or Marines more autonomous, but again it doesn't handle a changing situation well and it needs to be caught at either end. So our ultimate goal is something like this. A multi-unmanned system platform that allows us to give the machines mission type orders. Hey, screen this named area of interest and tell me when you see person X or vehicle Y and then report back to me. These systems will conduct their processing autonomously and so they're really only interfacing with a human just like a subordinate fire team leader going back to the squad leader when they have something to report or need new orders. Go ahead and go to the next slide. All that being said, Mr. Kelly mentioned trust already and I think it's something we can talk a lot about as part of the panel. In my experience starting to think about these things over the past year, the trust factor has mostly been about how much trust do we withhold. So we talk about the killer robots. Do I give that robot a life or death decision without a person in the loop to make that decision for them somewhere along the way? And where is that point? Do I suffer from the irony of automation and that I've got to constantly monitor the system to be ready to take over or am I willing to let it operate more freely? And in our doctrine of maneuver warfare trust is absolutely essential. So for marine leaders who interact with these things they're going to need to have a basic understanding of the hardware and software that makes up the system and work with it often enough to understand how it's going to respond just like they would with their man or with their human teammates. I did want to address though the flip side of the coin and some of you have probably seen this. This is a picture of a robot that was used in the Georgia Tech experiment. And what this experiment proved was that you can take people and have them follow a robot to a meeting. Have that robot demonstrate incompetence make wrong turns, break down etc. And then when you put the people under pressure like a simulated fire they will still follow that robot past clearly marked exit signs on darkened hallways filled with furniture because it's a robot and must know better. And I think there's risk on that side as well especially as we're early in automation that we assume it's a machine, it's perfect and so we have to ensure we continue to teach our leaders at all levels critical thought because you have cyber threats you have RF interference you have just general wear and tear on these systems out in an expeditionary environment and you have the quality of the information coming back and so that information should be evaluated by the leaders in the same manner that they would if it was coming from an unmanned system. So that's all I have. I look forward to your questions. Okay, and Jinx just one thing I just, the work that that you guys have been doing down at the war fighting lab and the number of events that you guys have been holding lately have been spectacular. A little bit later on maybe you could talk a little bit more about that but I tell you as a former Marine I'm really really impressed with what's gone down and I know that General Alford probably doesn't really care that I'm impressed with what he's doing down there but I think you guys should be extraordinarily proud of that. You guys are leading the way to be perfectly honest. Thank you sir, you know it's a bunch of captains and majors doing all that work. Yeah I do know that and good on you for not taking credit for it by the way, okay good to go. Well, Neto sir, so you know what we've, you know what we're already starting to build a trend on here and that is there's obviously some technical challenges and with your engineering background I'm sure that those might even seem to you to be like the easier ones, look forward to your remarks sir. Sir you give me more credit than I deserve I was not a very good student, low grade C so the reason I'm no longer an engineer now that promoted me is to get me out of that work. Well, you know that would have been nice to know before we picked you. Yes sir. No money back guarantee. Thank you very much to Sierra Space Navy League for hosting this great event and thank you sir for inviting me to participate in this esteemed panel. Yeah, we don't have an office of unmanned systems in the Coast Guard yet so this is what you get. But I will go through quickly and just touch on some things where we are taking advantage of machines and have our people working with them and taking advantage of what that provides us. You know, you look on the left there that's how we used to do life saving back in the day. You have people on huts sitting along the beach maybe walk along the beach and their ability to detect people in distress was limited by how far they could see and what they could hear. And then if they did hear something they went and pulled the boat out of the shed there and or tried to row out and find them and save the people. We had great success doing that, we saved many many lives but here's how we do it today. On the right we have a command center, man 24 7, all kinds of sensor data in that command center and they can actually take the search out of search and rescue. We have equipment lining 40,000 miles of coastline that can help detect when someone actually triggers their radio we can find them with lines of bearing, DFN and then launch our our RBM there, our response boat medium which creates about 40 knots and can help them if they are in distress. So we have made some progress. Next slide please. On the count of drug side there you see up in the that's how they used to do it, right? We sail our boats out to the entrance to the port and look for bad guys coming and trying to smuggle things into port and we'd hail them and hopefully they'd stop and we'd go over and do an inspection. Then as things progressed on the lower left that's when I got involved we would go out on these ships and we still have those in service actually since the mid-60s. We would go out and patrol in a box and give us some coordinates out there in the middle of the ocean and you patrol in your box and look for bad guys which I'm sure a lot of you have done. You look for stuff, you see somebody out there you hop in your motor surf boat and cruise over at six knots hopefully they stop for you, maybe they don't and then you climb aboard and see what you find. Today on the right you see our national security cutter has all kinds of capabilities there using lots of different streams of intel. Cruises out, we have two different small boats on board that we can use depending on what the situation may warrant. They cruise out at 60 knots and it doesn't want to stop yet. We now launch the helo that they carry which has a gunner in the back and is authorized to take warning shots and then disabling fire to shoot out the engines so we can get them. So we're going to get a few pounds of marijuana and come back a few pounds of weed if you were lucky. Today I can say that the Stratton came back last year after a four month patrol with a few other 33 tons of cocaine pure cocaine on deck from one patrol. We're going to manage some of these technologies and have an impact. Aviation will spend a lot of time with a lot of aviators in the room no more about these aircraft than I do but suffice to say they've come a long way from when we had our hover boat up there on the left to our C-130J and our Minotaur system and just freeing up the aviators time on board to be able to do other more important things and just staring at the screen. Next slide please. So I want to think about this problem set a couple things I've observed which I know more esteemed panels probably add to this but one thing I think is that can't lose sight of the fact that it's not just purely engineering scientists that comes up with some of these ideas it's really a I think an intersection of the arts and the sciences with the engineering or the arts and humanities with the sciences and engineering Leonardo da Vinci coming up with this helicopter concept 450 years before it was developed Einstein when he was trying to work on general theory he hit a hard whip out of the island and started playing Mozart so there's this certain creativity that comes from these people that are extremely genius but they also have a soft side if you will. Second I would point out that with the industrial revolution and how man took advantage of machines there I think some of that still stands true today and a couple things we look at what we need to do and we break it into discrete tasks and then we take a look at those tasks and see which of those we can use machines to help us I don't think that's changed today when I look around and I see that that's still that model holds true today it's just do we use a machine to perhaps replace the human in that task or are they simply enhancing the effectiveness of the human in doing those tasks in the lower left I just would want to make a point that we're not the only ones that are using machines the adversary is also very keen on this that is a picture it's a very crude machine but a very effective one that's a self-propelled semi-submersible which some of the transnational crime organizations can pack with about 6 to 8 tons of cocaine and they make it very hard for us to detect and lastly I think this evolution does take time and as we do that I want to say that it's my belief that it's very unusual to have some genius sitting in the corner that has a sudden brainstorming session by himself that comes up with a great idea it's more often a team of folks academia private industry government military all working together which then we this tapestry if you will of this creativity that really brings us to new and exciting places and lastly I'd say that as General McClelley touched on the trust factor for me is something that I just look at my kids and you know I give them any new thing I buy in the store you know iPhone or I whatever I come home and I'm looking for the manual the first thing I do is I go I got to look at the manual figure out how to use this thing whereas I look at my kids I just hand it to them and they immediately know how to use that and to me that's the gets back to the trust but how comfortable am I with this these new machines and this ability versus what my kids and their kids might be able to do the limiting factor may not be our ability to come up with the machine limiting factor may be our ability to accept its use which I think we touched on so I have for you I look forward to your questions and the other discussion thank you very much sir thanks and a a great point you know so here we are you know some older folks sitting up on the the panel for you and twice or maybe even three times we've talked about trust and one thing maybe that we'll be able to talk on later on is as we at least within the department of the Navy go out and canvas our civilians and our sailors and you know our Marines in terms of you know what are they thinking about with unmanned systems we get a couple of different groups that break out there and in some ways it's surprising in some ways that it's not but I will tell you one thing that it's our younger Marines and sailors and our younger civilians that come in as engineers to work in our warfare centers are appalled when they don't find it they expect to come in to a Navy, a Marine Corps and the Coast Guard and see it in many ways that's why some of them join because they felt that that we would be on the on the leading edge here and so in many ways it's our younger folks that are driving that and so that might be something we can explore a little bit later on as well so general colonies as we said before is the assistant deputy commandant and INL working in LP is that correct sir Roger now he too will probably tell you that he's got a bunch of young young Marines and civilians that really pushing the envelope as well in the LP and I would say that it's one thing that I like to remark on and that is a lot of this is not going to happen without leadership I do know that within INL General Dana has been pushing a lot General Dana picks his team so he picks general colonies out there to lead LP which I think is really really important and the other thing that I think that they've done in INL as well is sort of push the notion that you don't have to be within a domain that's not your domain of expertise in order to push where we go with unmanned systems so sir over to you Frank thanks Frank was referring to our class in 1983 we were in the same TBS class at Quantico and Frank was a lot smarter in a lot better shape than I was back then so we had a very good class four of us made general officers out of our class so I was the surprise candidate you're still here sir I don't know about that so what I want to do is we like framing things and putting points of reference my boss he talked about Kelly talked about General Dana his words he's liquored up on innovation every day and and so is our common and so both deep thinkers and both wanting to push where everybody's uncomfortable and so we're partnering within INAL where General Dana's in charge of installation logistics really to G4 for the Marine Corps we formed a team and the team is set up by Colonel Springer and some great civilians and Marines but that's kind of their night job you know they're really they're doing other things but the truth is if you read General McChrystal's book team of teams or lab working with all the other services and everything that's happening and so many things are happening trying to fuse that and come together as opposed to working against each other is kind of what General Dana's doing with this logistics innovation team so in 2013 Mr. Wheeler from the Federal Director of Federal Communications set pretty much frame the revolution in networks from printing press telegraph railroad and digital network that digital network is kind of what we're living in now and I would say that Steve Case who wrote a book kind of the founder of AOL wrote a book third wave he further breaks down the digital network and he said the digital network if you think about it in 1972 when the first email was sent and people were excited about that and people knew about it and it was sent over the internet working system the net which we call the internet and so the first part of the internet was really building the infrastructure that's the Cisco's, the sprints the system itself how we were going to communicate over it and then the second part of it Steve Case says really was the software development the iPhones, the iPads the communications that were out there and that's kind of what we're I would say coming out of now and then the third phase is really the internet of things that now everything is connected and those sensors that are being developed every day you can't go by either television commercial or radio commercial or something of some other sensor that you can use in your home or at work that is dealing with artificial intelligence and making decisions for you whether it's the temperature in your house or security type of things and that's kind of where we are now and that's moving really fast I would say the slide so in it when we're talking about the topic the human interface with machines so we always interface with machines you look at the Model T and crank it to get it going back then now we have systems on cars that park themselves with the human in the vehicle once the human pushes the button then we have which is out there now the autonomous type of systems that are driving themselves that are making decisions we have aviation technology that we're experimenting right now with flying helicopters that literally will fly to helicopter and make decisions dealing with the weather landing zone winds and all that kind of stuff autonomous cars are here I think the technology in this area is really the biggest explosion is going to be both in medical and education I think it's going to turn the world upside down on that you know I sit here right now my wife is a juvenile diabetic and I'm sitting here monitoring her blood sugar right now and so that she has an insulin pump and she has manually turned it on but pretty much that's going to be done once FDA okays it she's going to have a pancrease replacement basically and you think about when you google medical right now, when you're sick we figure out what's wrong hey I googled it and this is what I think I have right and sometimes we surprise them in that and that trust when we have that, imagine that computer or that software is going to tell us what's wrong with us very predictive you know high blood pressure or whatever you're coming down with this so I really think that's going to be and that's coming really really fast I think there's a if you look at this is where we're going we have an empty VR 7 ton and a marine core and you talk about autonomous vehicles you know will we be driving empty VRs in the future probably not probably we'll be programming the vehicles or the vehicles actually will be at the census we will tell it to go do something and it'll figure out how to do it if you look at the the com suite there on the side we have humans actually monitoring and looking at data and trying to make decisions on that and if you ever worked in a command center there's no such thing as one single operations picture as we get more sensors and all you usually have more and more things come in it's kind of the art of command having this information coming in and trying to make a decision on it with artificial intelligence and everything else a lot of that stuff will be consolidated and basically come up probably with choices for commanders this is what happening and you can make these three choices to affect these outcomes with the information that's coming in and hopefully it won't be a single outcome it'll give the commander and we'll talk about that kind of reason why we don't want to turn everything over to a machine that we need that human interface and then if you look at the next one the fixed displays and you talk about augmented reality and battle space I would say we're there now too we have in the Marine Corps infantry and merging trainer where we put marines in it's a real environment but it smells, sounds it replicates the battle things they're going to see because your sensors are the most important thing in time of war, the human sensors I'm talking about but they're able to train in an environment that they're going to and simulate that and that's kind of with the technology it's amazing what we can replicate if you think of the advanced simulators now and everything else you can be anywhere in the world and you talk on logistics whether it's Desert Storm or any of the recent conflicts it's amazing how dependent we are on fuel ammunition food and the size that we as U.S. just the volume I would say that we require with a large force you don't see that but the number of ships number of trucks refuelers and everything else it's daunting and if you kind of look in the past whether it's the Battle of the Bulge Military, Napoleon or you pick anything of these how the adversary lost or whoever lost often time it was a logistics failure that they couldn't keep up with their logistics and so we can't have those mountains anymore we have to be smarter in it and so you kind of see the Amazon drone type of thing and that's what we need whether it's a drone or a vehicle or submersible we have to be smarter and more flexible and also we need volume to do it depending on what area we need cheap things to do it not as logistics we don't want to have this multi-million dollar UAS system delivering a pallet of MREs we want to have that UAS system that if it gets shot down we don't care in the sense that we have so many of them that we can mass them, send them and they're somewhat expendable so some of the concerns is when you have the machines doing everything you lose some expertise if you think about your children voice to text and stuff like that I don't know if anybody can write and as it gets more sophisticated the machines are actually doing things for you that we used to do ourselves so you lose somewhat of an expertise also providing solutions if the machine provides I said three examples of a course of action if you ever think about it now you go on Google Maps and you really don't question it you just say go there and if you ask how you got there you have no idea because you just went by Google Maps and the machine made the decision for you and so there's a concern that it will take human brains pretty powerful and if you take it totally out of decision cycle you might be you might be making the wrong decisions but and then just the loss of, believe it or not the social interaction and the loss of adaptability you got to be an adaptive thinker if you stop thinking and let the machine do all the thinking once again you're going to be kind of controlled by that machine if you will in its decision we need to use it but we need to keep the human brain a part of the solution and slide that's it yeah I look forward to your questions and thanks again Frank for the invite first of all I think there was a story on NPR this morning about truck drivers that can now as they get tired just crawl in the back of their cab and the truck will continue to drive so think about that the next time you're on I-95 but the other thing that I think you brought up sir that's a really good point I'm going to write this into our lexicon of domains for unmanned systems and that's the IT realm general colonies one of his jobs is in the basement of the Pentagon literally watching the world you're the watch officer for the world and our IT systems or information systems how we make decisions is critically important and that's one that we can't forget and you finished off with thinking I'm just going to tell you right now I think that Admiral Gary probably has a thing about how we think and he comes by it honestly so thanks sir go ahead thanks Frank so I think as we talk today this morning about human machine teaming I think context is really important so just a couple of slides here to help frame the discussion and I definitely want to get to some questions certainly so I'll move through these so I think context the path that we're on as you can see here in the slide there's lots of continuums that you can consider about how you think about the various the march of time moving through here's one of them how we think about information from analog, digital information, the current age and one can put forward are we now entering the collaboration age and so it's how you interface with machines, with technology in a more collaborative fashion hence the name and the green there that's why I want you to focus on those two parts you already heard today this morning about trust and I'd offer why trust it keeps on coming up again and again well if you find ourselves again context admits this tide that's rising tide of technology and you think about how do we best leverage this in our existing forces in our profession security defense we're heavily invested in platforms ships, submarines, aircraft carriers that are spectacularly capable how do you leverage the greatest effect technology and it comes down to trust I didn't think of this phrase but I love it I mean the line is that culture eats strategies lunch you've heard that before but I want to bring that up because it's about trust in building a culture that says ah there's something there that we can leverage machines, unmanned systems and factor them into every aspect of our current profession of how we do things in the Navy how we can help out sailors and Marines in our tasks so that's the purpose of this slide and also to offer a context of the march of information and what's at our fingertips now but it's also cautionary in that if we don't trust these things we'll never implement them or fully employ them hence not realize the full potential while others may do so that are not as heavily invested in things from the 20th century as we move forward so let's talk next next slide here about how we've been using unmanned systems a very very simple slide here that just talks about initially kind of platform centric the first orb there an extension of what you currently have you know and that may be very obvious in fact it is obvious is that unmanned system help you see over a hill or is it something that you launch from an existing platform you control it so again along with that human assisted human controlled the state of art of technology at the time so it was very natural and you look at that middle piece about well maybe there's more we can do maybe you can have more of them create distribution networks and you plug into it cover more area there's lots of jobs for us to do out there lots of spaces to connect to be aware to maintain situational awareness so that's the second role of unmanned systems in the broad category and these all haven't necessarily evolved in sequence I think generally left to right but you can find examples of all three of those that exist now and then the third part on the right this notion of human machine teaming we talk about the subject of our panel you know and as technology increases the capability of systems unmanned systems to have higher levels of autonomy because autonomy is just that autonomy comes in degrees to the point of artificial intelligence a higher level they can do more things that are not as directly connected to the human but the human is still in control in command and there's something to that there's something to be gained something we need to factor into our force so this is a way of thinking about it and each of you experiences in your backgrounds can come up with examples of each one of those things but I'd offer if we have to deconstruct how we do business how we think about our job, our profession our fleet, our sailors and marines doing their jobs you have to kind of deconstruct it to its most basic level and then think about how we can add value trusting these systems because if we don't do that we'll pursue excursions and it will not be as fulsome as it could be all along you see those continuums on the bottom which leads me to the last point and that's what's the main thing what do we do and I'll offer the main thing and this slide shows you and I'll put it together for you so I hope it isn't too confusing it's to understand and then act Boyd came up with this back in the 40s before too to observe, orient, aside, act repeat, adapt do it better, faster than your adversary it was crafted in a fighter cockpit mentality if you will for how an individual, a fighter understood his environment observing and orienting making decisions and then acting and doing so quicker than their opponent so in this construct here you've got input and output input on the top, the green and on the bottom, the R's there, the output the effects, sensors effectors we say and what's that part in the middle that's that orient and decide piece that's the human machine piece but I'd offer human machine teaming can take place all across those so the arrows, the gray arrows on the left and right side it takes place in all domains as Frank had mentioned in his opening comments about we shouldn't be so domain centric yes, things take place in domains of course they do but that's not the main thing and war fighting and security takes place on various platforms and ship submarines, aircraft carriers vehicles on land, expeditionary forces those are things that's the with what it's more about the how and how fast and not so much the where the domain and the with what of course you have those things but your ability to traverse what you're seeing, understanding figuring it out doing it quicker deciding to act to the point of forecasting now that's human machine teaming that's fighting at machine speed if you will that's that combination it's not all machine, it's not all human it's that thing we sense intuitively that you put them together the right way and our view clearly is keeping the human in command but increasingly leveraging what the machines can do for you what machines do very well in deterministic situations and then leveraging creativity wisdom if you will that humans have and together that's a pretty powerful combination that's what we think about when we talk about human machine teaming and then the need to deconstruct how we fight, how we orient and then see the applications for unmanned systems throughout Sir, thanks, I'll tell you one thing when you get a chance to start a new organization it can be hard it's really good when leadership sets up two organizations because then you can form a bond early and at least you have somebody to kind of commiserate with and Sir, I want to thank you there's not much commiseration because we're just quite frankly too busy but it's been an honor I think I have said it and I think others have said it though working with Admiral Garrier is both inspirational and aspirational he really compels us to think harder, think longer think in the future and Sir, thank you and Dr. Singer one of the things I neglected to mention is that you have degrees from both Harvard and Princeton is that correct? I hope it is, thank you, okay you do now have those degrees but thank you very much but I'll tell you one thing that I really appreciate and that is without ever knowing any of that your ability to honestly fit in with the room full of J.O.'s or room full of S.E.S.'s Admiral's General's Captain's PFC's is remarkable and there's a reason why you're the only civilian on this panel Sir and we look forward to hearing from you so thank you very much it's an incredible honor to be up on the stage with these leaders but it's a double honor one of the things you didn't mention in my bio is my grandfather a Navy captain who served in World War II and Korea and some of my most cherished memories are going to visit my grandparents and in his study leafing through C-Power magazine and the like so this is just a real special to me to be here there's so many exciting programs that we've heard about on stage or you could go downstairs and see on the convention floor that I thought the most valuable way as the civilian outsider to aid the conversation is to lay out a few of the key challenges that we face in the years ahead and since actually you're already flipping ahead for some reason you can go back to the slide that we were on go back please is since we were limited to three slides as you've seen there's three there so I'm going to identify three of the challenges you'll note though and this relates to the theme that we've heard across the conversation is that none of these three challenges is about the technology itself it's about the human side and not the human individual team member but rather the human organizations and ideas that I believe could prove most of vaccine so now let's jump ahead to the next slide there okay the first challenge is what I would call the old and new problem or more directly what I jokingly call the Pontiac Aztec of war phenomena so when it comes to talking about defense systems we focus on weapons programs that start out promising all sorts of things but they end up being overpriced over engineered and over promised a lot like how the Pontiac Aztec was supposed to be everything from a sports car to a minivan to it would take you camping and the like now notice I did not name any defense systems but many of you were thinking in your head about certain parallel programs my point though is when you go to this bottom graph there it lays out the top ten pentagon programs of record by what we're spending now but you'll notice a couple things that stand out about it the first is even though that's what we're spending the most on right now we still have a massive amount of spending to come on these top ten programs of record the second thing that you'll notice is that there is no use in that program that is one side of the team is missing from what we're spending the most on now the issue isn't that in the years ahead we won't be spending more on unmanned systems but in some way shape or form they are competitive with existing programs and systems and so as the pentagon wrestles with you know a flat budget environment the new becomes directly threatening to the old and in bureaucracies the old is often at an advantage it's more likely to have internal constituencies be they program offices or cultural tribes officers who identify with that system they also tend to have stronger support from members of congress who want to protect existing factories and existing jobs in their districts rather than the promise of some future program office next slide the second problem I call the sea plane problem there's very few that are outright resistant to a new technology or tool but we often fool ourselves we like to say that we are embracing the new but tend to do so only in a way that supports and sustains the old it's akin to if we were gathering at a navy league convention say in the 1920s and you would likely have heard from battleship captains who would say no no no no I'm not against planes in fact I've taken very valuable deck space on my battleship and apportioned it to sea planes I'm embracing the aviation revolution what I'm getting at here is when we're thinking about the new or when we're thinking about machine human teaming how does it break the existing approach and if it doesn't break the existing approach in at least seven ways and what we buy how it changes the way we train how it affects personnel system including promotions what doctrine we choose what tactics we use to how we overall organize bureaucratically if it doesn't change these either individually or more probably all seven of them then is it just like what happened in terms of putting a sea plane on a battleship third challenge next slide synergy there's so many different amazing programs going on and again you've seen them downstairs in the book Ghostfleet which is available on paperback for $11 next week please tweet about it we document at least 21 different examples of man machine teaming autonomous weapons programs that we're working on today but we document them and they take place in all the different domains as you laid out of conflict be it undersea warfare aerial to new domains like cyberspace outer space the key is that each of these is a historic level shift within that domain but the fact that they're happening simultaneously makes it all the more demanding and so to me the doctrinal question is not merely are we working enough on each and what should we be doing but how do we bring them together not just how do we bring human and machine together but how do we bring cross domain teaming together my sense is whoever gets that right will be the winner or loser in terms of the next major conflict right now we're wrestling with finding the equivalent of say the blitzkrieg or the fleet problem exercises or the marines wrestling with amphibious warfare and close air support and the banana wars in Latin America in the 1920s and 30s and I think it's important to end on those parallels that I gave they point to how neither budget nor operational pace are limiters from learning innovation and adapt it they figured it out in the 1920s and 30s we can figure it out today thank you sir thanks very much now we have microphones in the middle of the room and I'll tell you what we're going to do is we're going to establish a priority of questions if there's folks that have questions in the room and you'd like to go to the mic I'm going to take that one first I see folks getting up but they're heading in the wrong direction not a good sign you're a good man hold your okay we also have the app okay and we've got some good questions already from the app and I really encourage you to ask your questions so it looks like we've got some folks queuing up over here to my left and go ahead sir question about the culture that's been raised a lot Admiral you said that culture each strategy is launched every day and I've noticed that our culture for some of the reasons that some of our panelists have brought up is very platform-centric I want my destroyer or my aircraft carrier or whatever to be the fighting knight in the fighting armor that it is so our culture and our military culture of we are warriors well you all are warriors I'm just an engineer but we are warriors and we're going to do the fair thing and we want to make sure that we minimize civilian casualties and all of that all of these things kind of push us into a particular area of unmanned weapons and then on the other hand we have adversaries who if you look at their culture and their doctrine it's all about asymmetric warfare and having the land rise up and be the enemy of the invader and these sorts of kind of classic Maoist sorts of doctrines so their culture is very much I want these unmanned systems because they pull into my culture my doctrine my strategy kind of bridge that mismatch that we have going on so great question thank you very much that was job number one when we first stood up N99 again as director of unmanned warfare systems for the Navy it became very clear to me that my relevance our relevance the office's relevance getting unmanned spread in the Navy the department of the Navy Navy Marine Corps team both wasn't about me being a cheerleader and asserting it with a loud voice it was all about bringing along all the other tribes as well and so job one was to make the point that there's something that's bigger than all of us and if you had to deconstruct it again hence the point of my first couple of slides you had to say well what does a winning team look like you know and if you can see farther understand quicker act faster and adapt continuously those four things that's a pretty darn good war fighting winning team and those things cut across domains they cut across platforms they cut across communities and that was job number one because you're right I couldn't agree with you more if you're completely obsessed and it's understandable you invest in platforms and systems billions of dollars and the training associated with mastering those systems but to what end in the end it's a team problem set that were involved in here so that was the first job to get everyone to agree to that and I'm here to say and I'm very happy to say that there was agreement that there's something larger than each of those tribes and unmanned plays a role in each one of those so couldn't agree more to get the most out of this you have to go beyond what you know and see where it can help thank you thank you I have a question for Colonel Jenkins maybe Admiral Gehrer and others want to get at this too I wanted to get back to what you said about developing the trust between the troops and their potential unmanned teammates I was wondering specifically if there are ways of training that you're looking at that could get at that or new ways of introducing the equipment or whether it's just a matter of establishing the reliability of the systems and developing that trust over time thanks so I think I would say first of all that there's no better experience than actually putting them in the hands of Marines and I think that's where you get to Dr. Singer's point is that we we're still fighting those established programs in terms of providing exposure I think it emphasizes again what General Colini said and what the Commandant said yesterday these things have to be small, smart, and cheap because I have to be with them every day in some way shape or form and it can't be this boutique item that I roll out once a year and interface with I've got to be constantly working with it and so it's got to be disposable to a degree and inexpensive enough and rugged enough that I can use it on a regular basis and then I think eventually you'll get to the point where you could do simulation because the software is the brain eventually getting that software in support of my maneuver in a simulation environment really allows me to train with the machine as long as I keep the software's matched up especially as I start getting into the AI and the machine learning types of things where I need to make sure that all of my systems learn everything that the other systems are learning so I hope that gets your question to some degree at least I'd like to add if I could so how do you build that trust and it's certainly it's getting splashing systems getting them out there in the hands of the Marines and the sailors and to figure out how they work and develop those tactics techniques and procedures part of our portfolio in the N99 office is rapid prototyping and experimentation working with these systems in an operational context with technologies that are right getting them in the hands we haven't figured out yet how to use all these things but building that trust is seeing the ultimate operational application for it or hey, this thing fills a gap this thing fills a need it's helping us do something better and so that builds up trust especially when it's a solution set for something that's been dexing you for a while so that's a big part of the trust piece is again getting things to fly getting things to swim seeing how they work develop those con-ops I think there's an example when you go to a plant and a robot is doing something in the manufacturing line you also have a human in the line and some of these companies are actually then what they're doing is switching tasks now a robot does the human task and the human does the robot task and it's just that in itself and if you were using an example of a Marine rifle squad or something like that and you had a robot doing something with the rifle squad I mean with that individual I mean with the robot with the individuals and then you'll develop a trust what they can and cannot do so I think the example of training is a good example I'm sure you want to be mindful of in these early introductory exercises a huge supporter of them though is what Clayton Christensen calls the innovators dilemma and it's the fact that the first generation of a technology will invariably be the worst of that technology and second, usually it's worse than whatever technology it's to replace and so when you have human organizations that take a long time to adopt you sort of have to it's tough for the disconnect between their slowness versus how a technology might go faster so if you're thinking of the parallel if you had limited your envisioning of tanks to well, the early tanks go slower than people walk and that's the only way we're going to envision their use versus the assumption that hey, what happens if they actually can go faster or if you're thinking of some early submarines that have little range you have to be willing to not let the flaws of the prototype steer you away from its long-term possibilities and I think if I can build on that Peter and you, Admiral Gehrer it's finding that right niche, I mean we programmatically we tend to A, gold plate and then B, wait till it's perfect and we've gone through DT and OT before I put it in the hands of Marines I think what we see over and over again is these young Marines and sailors are better at seeing the potential than some of us old folks are and so if you can put it in their hands and create a feedback loop that allows them to get back into the discussion, hey why don't you make it do it this way your development and your evolution, if you will, of that system can be a lot greater because of that interface. Okay, so you three folks at that microphone we are going to get to you but we've got to use this app, okay, we've just got to use it I'm going to, so what I want the panelists to be thinking about is access, we've got folks in here from industry from academia and they're probably going to want to talk to you, I'm going to read it as it was written access to the right people within the department of the Navy and Department of Homeland Security if I can just throw that in there, can be challenging for many technology firms how does an innovative organization initiate collaboration with the Navy and how can they benefit and Admiral Garrier, can I start with you? Certainly thanks. Again this conference is a good example a good starting point, we're certainly reaching out not a week goes by when I'm meeting with representatives from industry, academia other government entities as well this whole notion of how we partner because the solution set is clearly not held by any one entity I'd offer our government labs is a very good portal if you will or a pass through to interface I work with a Frank's office on a daily basis we work with our warfare centers that are developing technologies to meet needs they reach out to industry continuously that is a very good avenue, that is an active avenue that are again our warfare centers that are currently engaged in projects and meeting needs, so again the dialogue is one area warfare centers directly through I offer those two avenues as rich thanks sir, appreciate it okay sir. Thank you we've seen a lot of interest in the third offset with undersea warfare so I was wondering with the unmanned systems how are we planning to use them for anti-submarine warfare are these going to be just new platforms that we can put sonar sensors and other sensors on and to attack or is there a new concept of operations that we can use unmanned systems to conduct anti-submarine warfare I'll certainly take that question I think right now our approach is broad it's based on the understanding that even as I talked about in my opening remarks that there are unmanned systems that we can use that augment an existing platform that help a platform do its existing job there are unmanned systems that can serve in distributed netted manners that we can then work with and then there's a higher end as technology has enabled it and various technologies support this especially as we look at energy systems and the density of energy systems becoming greater you can put more into a smaller package in a larger area and also as we develop autonomy you can think about unmanned systems that are off operating more independently so in those three categories you've got different functional approaches and then I'd offer there's even sizes of how things small, medium and large if you will so it's a very much a broad approach that seeks at meeting gaps in the process when it comes to unmanned systems undersea systems specifically but all of them is that if we use them right we're going to increase our capability and our capacity it's not unmanned for unmanned sake it's unmanned where we need it as an augment to our existing structure and there's many many applications for that Hi, Mark Parmerler with Defense Systems I know you mentioned not to focus on kind of the domain but that's exactly what I'm going to do here the Air Force uses kind of that or they're developing the loyal wingman program for the air domain I was wondering if any one of you could kind of address some specifics in terms of from a maritime perspective air, surface and undersea in ways that this machine teaming can penetrate some of these contested environments summarize that I mean if I could just summarize I'll take a quick stab at it the totality of contested environments if you will and the broad application of how unmanned systems can help us help us work that problem set is that a fair characterization of your question Yes, you are kind of focusing on this sort of A2AD environment what are some examples of how this man machine teaming can help to penetrate those contested environments So I think so one way is you look at the totality of the problem typically you look at areas that are contested and seldom what I say are all three areas, domains contested simultaneously so if you could think about how you look at access in one domain and using it to help unlock another that is an approach I'd offer that approach values a system of systems where you're leveraging again going back to Frank's opening comments about it's not all about just the one domain anymore it's about the general connectedness of many things so we think a lot about systems of systems and I can tell you how you think about platforms populating those systems man platforms unmanned platforms they can serve as many nodes so when you start thinking in this larger framework about how you unlock areas that are here too far were contested you're thinking about them in very one dimensional ways so that's an approach unmanned systems can help populate nodes manned unmanned nodes in larger systems of systems maybe I'll add in a couple thoughts on the future of warfare so when we've been looking at the idea of the challenge of say an integrated air defense with particular the new wrinkle on it is the long range and the ability to fight our bases we've been right now we are envisioning in the air domain the unmanned system we just moved it to the air-to-air refueling side basically we had the situation of short range tactical air at the very same time an adversary is developing greater reach so it's a boxing match where they've got a long punch we just gave ourselves little alligator arms and we're exploring the idea of unmanned systems as a work around to that then you had the older idea of the unmanned system as the sensor that it will allow you to detect in particular that the challenge is not just an integrated air defense it's that it's mobile as well that's the further wrinkle onto it so that you can use the unmanned as a sensor to be able to find it what's interesting to me is the third and again that sensor might be something that penetrates the enemy air defense in terms of a squizzet jet fighter size program or it might be a swarm of teeny tiny ones what's interesting to me is the third category which is where I believe we will head which is using the unmanned system and the team to a trick to destroy that defense and it might be either it's carrying the weapon systems itself even the one that's in the air to air role it is going to have to be armed unless you want it to be unable to defend itself and therefore putting the human pilots that depend on it to get back to the carrier at risk that's why we armed the man tankers and we will arm the unmanned tankers but also, so it's not just armed in terms of taking out the enemy the other thing that unmanned systems offer is their disposability so it's the envisioning of the way for example the Israelis originally use unmanned systems against Syrian air defenses back in the 1980s where the drone's job was solely to make the air defense to turn on the missiles, the SA-2s turn on their radars reveal itself and even more so they used missiles shooting at it we wanted the enemy to shoot down drones and so we may look at unmanned systems the part of the team, it may be the disposable part of the team, the team member that you want to get killed it's a totally different way of rethinking it but again it points to new opportunities in doctrine great, thank you fantastic, thanks and I don't know if Jonathan Lashenko is still here we had a question from him on the app it's a release authority and I think that Dr. Singer's question answer, excuse me, responded to that in some way so Jonathan hopefully you found that helpful now it looks like a lieutenant commander yes sir you have been extraordinarily patient I'm not going to say anything about the gentlemanly manners of the two guys in front of you but we are going to hear your question thank you thank you for being here today I have heard each of you talk a little bit about the cultural buy in from people to trust these unmanned systems or any technology and I've heard some of you talk about your young children who already buy into it and I can tell you I'm not even that old but I have people who work for me who are they are disappointed in our lack of innovation, our lack of being on the leading edge and they are doing their own workarounds to complete the same mission that I'm doing so what they lack though is access or understanding of the leadership or the power to change that so I'm wondering from you, you are the leadership so what are you doing within your respective service to to tap into the ingenuity or the motivation of the young people who are already bought in but don't have access to the power to change that so okay, yes sir so exactly right so we are doing an innovation challenge we are going to reward them come up on the net and solve we are going to give them a couple of issues to solve absolutely right it's a very collaborative environment we believe jobs that if you don't cannibalize yourself, somebody else will and so exactly right and it's not just technology it's processes and ideas and how to use things better we as Americans, it's fixated on technology so you think about the ID we come up with sophisticated jamming devices and they come up with fishing wire and so we do a billion dollars thing and they do a 10 cent of our stuff but those innovation ideas and stuff like that of how to counter that, how to come through it you're exactly right and you have to have an environment that accepts it and also that culture in the military we're not necessarily built that way to have somebody you know at PFC come up and go hey I have a better idea how to do this and then you welcome that PFC's idea and then bring it so we have, I don't want to get into how we're doing it because it'll take too much time but we're doing exactly what you're talking about I can add if you like you create the culture, you create the conditions that encourage this information flow you know CNO's design he talks about high velocity learning creating a high velocity learning environment at every level so what is that it's a systematic way of applying what you understand and then you look at it you come up with a hypothesis you test it, you measure your results you come back and you make an adjustment and you repeat and that's at every level so if you value that from the top and he has, it's part of his design it's one of the four main elements of his design and at every level we're going back and looking at our own internal processes and saying hey what's my hypothesis how do I think about this problem set what am I doing am I making progress on those areas is it what I thought it was going to be how should I adapt the quality of that is every level from junior to senior so that's what we're doing now in the Navy right now and again it sets up an environment where you're interested in listening and Frank I'd add we're not doing enough, we can be doing a lot more and thanks for bringing it up and keep bringing it up until we do better hold us accountable I applaud you for raising the question great question and I think it extends to many other things as well I know a person I look at I just came from boot camp they're different than when I was that age but they're so smart, they bring so much we have to learn how to tap into that if we don't, we're going to fail, they're going to go elsewhere and we don't want them to go elsewhere we have to keep them in the service we have to be better at this, no question thank you sir the other thing I say without deflecting from us up here is the lower level leadership needs to do a good risk assessment and then allow that innovation because at the end of the day I'll change the entire organization all at once when I get the great idea but if I can demonstrate it and then bring it up the chain and here's the data that shows how much better this idea is or this piece of gear is it makes it an easier sell and as Admiral Gehrer said hopefully you're working for leadership who encourages that type of exploration and enables you to do that risk assessment to make that happen but I think that's part of it too is really at every level don't rely on those of us sitting up here to make the changes but kind of start working to make the change yourself a bite at a time okay, great question thank you and we are over we are over time and we are over for this event but I really appreciate everybody for being here yeah I also want to thank Navy League for letting us do this there's an appetite out there for more of this we look forward to discussing with you in the future thanks again so I appreciate it great, thanks thank you don't fall off the stage yeah sir, thanks hello Tony yep, hey, great thanks hey, hey good to see you