 What about you? You're talking to people and then you're off to have somebody else. You can put it up now. You can just put up the first slide. Is he down? Is he right? Thank you. It's a little dust on the camera here. I'm very polite. Is he just going to do our hair? There we are. Impossible. Thank you. Thank you. You can't do anything with mine. Oh man, I love that. Look good in here. Two minutes. Well good morning everyone. I am Shirley Ann Jackson. I'm sorry, we just need one minute. One minute. Because the camera needs to warm up. Oh the camera needs to warm up. I thought you said we had to warm up. So you have to do so much. You never do that. Well good morning everyone. Thank you for coming and welcome. I'm Shirley Ann Jackson, president of Rensselaer Polytechnic Institute. I'll be moderating the session this morning. Today we're going to consider an essential question. How can we prepare for a new kind of war? And indeed even what is the meaning of warfare today? Now clearly the context for conflicts is changing in the multi-polar world that we inhabit today. Now the United States was seen as emerging from the Cold War as perhaps the only superpower. But today China clearly is growing and has been growing in influence and importance both as an economic and a military power. And we see a newly assertive Russia. But we also live in an asymmetrical world with extremes of wealth and poverty. And asymmetrical warfare is a future, a feature of that world. One in which non-state actors such as ISIS are able to target rich and powerful nations. And indeed the tools created by the fourth industrial revolution which merge the physical, the digital and the biological worlds can be turned easily and sometimes inexpensively into weapons. Potentially allowing even small groups the opportunity of launching broad scale attacks. And we see proliferating gray zone conflicts that fall short of open warfare. We see it in tensions in the South China Sea. We see it in the Ukraine. We see it in covert operations including drone strikes. Of course in proxy conflicts. In a new kind of guerrilla warfare that extends beyond geographic borders. And we see it of course in cyber warfare where the infrastructure that in many places is controlled by companies and private hands become targets as well. And so this morning we are very fortunate to have a distinguished group of panelists who will help us to consider ways to prepare for a new kind of war. So let me introduce them. First is Dr. Mary Cummings. She is professor in the Duke University Pratt School of Engineering. And the Duke Institute of Brain Sciences. And director of the Duke Humans and Autonomy Lab and Duke Robotics. Her research interests include human unmanned vehicle interaction. And human autonomous system collaboration. And the ethical, social and policy implications thereof. Dr. Cummings also was one of the very first female fighter pilots in the United States Navy. Good morning. We also welcome Sir Lawrence Friedman. Emeritus professor of war studies at King's College London. An expert in among other subjects the role of armed forces in contemporary international affairs. As well as strategic theory. He is the author and editor of many books on subjects that include deterrence, strategy, the Cold War and US involvement in the Middle East. Sir Lawrence was awarded the commander of the British Empire title in 1996. And the night commander of St. Michael and St. George in 2003. I did my research. We welcome as well Jean-Marie Gano, president and chief executive officer of the International Crisis Group. An independent non-profit, non-governmental organization committed to preventing and resolving deadly conflict. Which is headquartered in Brussels. As the United Nations under secretary general for peacekeeping operations between 2000 and 2008. He led the biggest expansion of peacekeeping in the history of the United Nations. He has served as director of policy planning in the French Ministry of Foreign Affairs. An ambassador to the Western European Union as well. As director of the Center for Conflict Resolution at Columbia University in the United States. We also have her excellency Jeanine Henness Plauchart. Who has served as minister of defense for the Netherlands since 2012. From 2004 to 2010, Minister Henness was a member of the European Parliament for the VVD or the People's Party for Freedom and Democracy. In 2010, Minister Henness became a member of the Dutch House of Representatives for the VVD. With a focus on issues that include security and disaster and crisis responses. So there's some broad questions that we hope to talk about this morning. But we will return to them, but I will outline them here at the beginning. And they are three. The first is will multilateral institutions such as the UN and NATO still be effective in helping the world to resolve conflicts in this new age of warfare. If not, what kind of institutions do we need? Now embedded in that is the question of whether the current multilateral framework and multilateral organizations and institutions are able to evolve along lines they may need to. The second question is how do nation states address gray zone conflicts with other nations and with non-state actors when sometimes they themselves may be backing gray zone operations. And third, what framework or frameworks can provide stability in the cyber realm? Do international treaties make sense? Do we need an international treaty on cyberspace such as the United Nations Convention on the Law of the Sea? And so we want to come back to those and so I would ask you to think about those as we carry forward in our discussion. And these are the broader questions around which we would like to engage our audience this morning as well. So Dr. Cummings, let me begin with you. How will advances in robotics and artificial intelligence transform warfare? And what safeguards and conflict resolutions frameworks will be needed in a world in which the fighter pilot may be an intelligent and indeed autonomous robot? So I think it's interesting to even put that as a potential uncertainty. I think it is certain that we will see more and more missions in many militaries turn over to what we think of as robots. Israel has said that they are going to go completely unmanned in their aviation world, meaning fighter pilots in Israel will become robots, at least by what they're saying today. And even in the United States there have been generals who have said that they think the last fighter pilot in America has been born. And the reality is that for the typical fighter pilot mission and bomber mission that we think of today, the aircraft absolutely do these missions better than humans. And we actually crossed that threshold a few years ago. It is actually safer, and by safer we mean pilot air. It is safer for the U.S. Air Force today to send a drone on a fighter bomber mission than it is to send a human mission. And so it's a done deal. I think we will see more and more missions in the air turned over to robots for sure. But I think the question, and we see this now in the debate in the United States over the F-35, when we start talking about advanced technology, is there a place where we are, I call it the cool factor, the American military is in love with robots. We want to spend a lot more money on robots, a ridiculous amount of money on a platform like the F-35, when in fact, if we think about asymmetrical warfare, ISIS can go out now and print drones with a 3D printer, can print thousands of drones with a 3D printer at very low cost and arm them with conventional weapons and or biologic weapons, for example, and basically result in much more devastation than an F-35 in a surgical strike could cause. And so we are in this strange place where the barrier to entry to drone technology is so low that everyone can have one, and if the Chinese go out and print a million copies of a drone, a very small drone, put those up against an F-35, then they go into the engine and you basically obviate with a very expensive platform. And so I think the military and the militaries around the world are struggling with what it means to have this advanced artificial intelligence technology in large aircraft, but yet at the same time they can be embedded in small aircraft that everyone has access to. So let me follow on with that, particularly those of you who have been involved in the government, because an embedded question is who decides? So one part of the issue with the new and available technologies has to do with what is possible, and what is possible in a very destructive sense, and what can compromise, as you point out, very expensive weapons systems. But another is, and it's come up in the context of drones, but not in as a direct away, if we have truly intelligent autonomous fighters that are drones, who decides how much collateral damage is appropriate or not? First of all, as I placed an order of, I don't know how many F-35s, I just wonder if you could advise me whether I should continue or not. But the price is dropping, as I understood last week from Locate Martin, but I would like to continue this debate later on. I mean clearly, I just double check my notes because there is clearly an ethical dimension to this question. So we have a very hot debate in the Netherlands on the use of autonomous weapons. And what I always said is that it's important that the deployment of such weapons must always involve meaningful human control. So I'm clearly, I'm convinced about the advantages in the future by using such weapons. And it's there to stay, so there's no way back. And definitely, I mean, we might face, for example, self-learning systems that are able to modify their own rules of conduct. And then again, this ethical dimension is even more important. So the issue of meaningful human control by deploying such weapons is now being debated within the UN in Geneva. We are obviously as a government and many others are involved in this debate, but this is going to be the key for the future in my view, meaningful human control. I'm going to come back to that. But Sir Lawrence, I want to pose a question to you. Now you have defined strategy in a very beautiful way, I think. And I'm going to quote you, you can tell me if I'm wrong. Quote, it is about getting more out of a situation than the starting balance of power would suggest. It is the art of creating power, unquote. Now tools such as social media that have been employed skillfully by ISIS and by nation states are transforming the art of quote unquote creating power. So is traditional military might losing its meaning as a measure of power and as a projection of power? And is it losing some of its meaning as a tool of deterrence given the initial part of our discussion here? No, I think military power, violence, force is a way of controlling territory people and the extent to which people still resort to it. Quite micro levels around the world indicates that that's not going away. Now part of that is how people think about their identities, about their enemies, about how they think a conflict may develop, what they think matters, what their grievances are. And social media obviously plays a role in that. But I don't think we should think that's new. I mean if you look back at any of the strategists of the past, what they might have called the psychological element was always there. It was always important. And I also think we've got to be careful about not exaggerating it as well. Let me just make two points. First historically people were saying the same things in 1979 after the Iranian Revolution when cassettes going around mosques were the way by which Khomeini built up his support. And this was considered in its own way remarkable at the time. Look at the role of radio in the interwar years which was the major instrument of totalitarianism. And in fact one of the things that happened afterwards is that instrument was broken down. So this aspect of media has always been around and attempts to influence minds has always been around. Secondly I think we've got to be careful about assuming the ease with which minds can be changed. There's an awful lot put on what people call information operations or more nicely public affairs or something like that which is just another word for propaganda. And it works if it's got fertile ground. But I think the idea that the sort of precision guided thoughts that you can lob into people's minds so that they having thought one thing one day suddenly start to see the world in a whole different way. Is a bit optimistic if things are going wrong in their lives or something that they've been promised doesn't happen. And then somebody's coming in with an explanation of why that might work. Then that has an effect. But I think just to give the example the Russians have given enormous credit for this. But actually most of the time people assume even when the Russians are telling the truth now that they're lying. So if we just take the example of MH17 which is obviously subject close to the minister's heart. This was an issue that could have been dealt with very quickly when Russian backed separatists or even Russians themselves shot it down in the summer of 2014. But because they lied about it because they tried to find any explanation other than the actual one it's still an issue it's still going on. I think one has to be very careful about not assuming that there are really clever people out there who know how to manipulate minds against the actual experience and evidence of other people. So Ambassador Gano. So there are two things that have been talked about here. One has to do with the advance of autonomous technologies. And the other has to do with the fertility of ground of mental ground in terms of the influencing of people by now social media. But as Sir Lawrence points out people have used whatever the media at hand happened to be at the time. So you know you've worked to resolve the conflict in Syria as deputy joint special envoy of the United Nations and the League of Arab States on Syria. So clearly Syria remains an ongoing humanitarian tragedy. But what lessons should we take from Syria with respect to the questions of the interplay of technology and what that means for peacekeeping and conflict resolution. And then what lessons can we take about the kinds of conditions you see in creating their fertile ground for influence through social media. Well Syria is a tragic example of the kind of conflicts that we have now. It is multi layered because there is a conflict between the Syrians. There is a conflict between the regional powers. There is a conflict or a tension a confrontation between global powers. And if you ignore any of those levels you're unlikely to succeed. In that sense it's quite typical sadly of many conflicts of today's world. But it's typical also in its asymmetries I would say. Asymmetries of interest. That is in Europe before there was a refugee crisis there was much talk about Syria. But frankly in Europe as well as in the United States actually more rhetoric than reality. Because people may feel some kind of sympathy for the Syrians. But between that sympathy and taking it as a key core interest there is a huge distance. And actually the more horrors you see the greater the distance. Because you have a sense that this is not your world. So there is a big difference of interest between people who talk about a conflict from a distance. And those who are directly affected by it. There is a difference of interest for Russia and for obviously the regional powers. Syria is a direct interest. Syria is the real neighborhood of Russia. It was not in the neighborhood so to speak of Europe until the refugees came in. It's asymmetric in the means employed. Of course there are jet fighter aircraft, there are drones. All the technologies that we have been discussing have been in play in Syria. Also acquisition of intelligence. All that is there. And then you have people with Kalashnikov and rather rudimentary weapons. But who have a rather direct interest in the conflict. There is an asymmetry of commitment there that is massive. And I think Syria reminds us that yes as Laurence Friedman said technology changes. But in the end as you said it's about control of people and territory. And we should not confuse the means with the end. And what matters in the end is the politics of it. And Syria is going to be decided much more by politics, by the various asymmetries of interest than by some magic weapon which would really transform the conflict. I think that's an important point because I think whenever there is a parent and I'm in the whole science and technology arena. But whenever there is a major advance in technology and of course a foundation of our meeting has to do with the fourth industrial revolution that brings the various technologies together in rather unique ways. But in the end if I'm hearing you correctly it's still about fundamental human conflict disagreement ability to resolve differences, territorial claims and control of the people they are in. And so is that a key point to take away from your remarks? Yeah and I would add that these asymmetries they are a big part of what we call terrorism because confronted to drones, to high tech, to weapons where human beings are not at risk. It would be foolish from a rational standpoint to go after those weapons. You have to go after the people who are the political infrastructure so to speak behind the weapons. So the notion that terrorism is some kind of irrational madness is very wrong. It's very much a logical consequence of the technological asymmetries of our world. So is terrorism a strategy then, Sir Lawrence? Yeah it's not a very good one. It's a strategy if you haven't got anything better to do. I think a lot of it comes back and reinforce Sean Murray's point to control of territory. If you're doing it by an occasional atrocity you're not controlling anything. You're creating mayhem, you're upsetting people you're putting them to expense, you're having all sorts of effects but they're not necessarily giving you victory. So it's very limited in that sense. The most successful terrorist groups run a campaign so it's not just one outrage, it's an outrage every day. So you've just changed the whole mentality of the victim community. And by that time it's turned into something else it's become guerrilla warfare and eventually may become even more as ISIS was trying to do. So it's sort of an entry level strategy if you like and often they never get beyond that. Can I just make one point because I think it's very relevant to how the Western world thinks of force and maybe Missy we want to come in on this as well which is the limits of Western power if it is only coming in from the air or by drones because if your view is we don't want to put our people at risk therefore we don't want to put them on the ground then actually your ability to influence conflicts is limited and that I think has become a real problem because public opinion for very good reasons is very resistant to that but if that's your view then you're going to have to accept that you're always going to have to find somebody else to work with on the ground and they will have their own agendas or else actually what you're doing is hurting but you're not in any sense controlling an event. Doctor come, oh please, sure a minute. Boots on the ground is what you refer to. There's one way of putting it. And I understand what you're saying at the very same time I mean we have this fight against ISIL in Iraq for example and it's a very wise decision not to send in boots on the ground because we want ownership within the country itself so we will have to continue to work with the Iraqi security forces so this is also an argument sometimes not to run the risk yourself but you know to deal with the Iraqi security forces in a different way. Now on Russia, the threat posed by Russia I fully agree with you that not all the aspects of the threat posed by Russia are new in history I mean we recognize a lot from the past at the very same time one can deny that we are facing some new aspects for example the cyber threat it's a new phenomenon and it's deeply disturbing us and it's also a true danger to our democratic order so I agree with you that we should not overestimate but at the same time I would argue that we should not underestimate the threat because some of the aspects are as I said deeply disturbing They are but I think it's important I think these things too first as with terrorism stuff is going on and it hurts if we don't stop it and can't find ways to prevent it so as a politician I'm sure you very much need to address that and it's true with cyber there's very few big crises that aren't accompanied these days by a denial of service attack or something like that going on but even cyber attacks don't control territory they're supplements they're not the thing in itself and I think one of the problems we have is when a new technology comes along we want to see how it can produce a knockout blow be decisive in itself and that's not what is going to happen it's ancillary, it's a part of it it's a part of the story but it's not the whole thing in itself but Sir Lawrence and then I would invite Dr. Cummings to weigh in on this I think we don't want to overestimate what we think of the use of cyber warfare but we don't want to underestimate it either because I think denial of service attacks are one thing but if one thinks more broadly of command and control systems if one thinks more broadly of the internet of things and that one can use a cyber pathway to disarm a complete defense system and or cause a complete or even if it's not complete a significant shutdown of an electrical grid which then obviates many many things these are non-trivial things that emerge from the cyberspace absolutely I'm not denying that for one second but I think there's a tendency you can see coming back from when people started to talk about this it was immediately a cyber pill harbour everything's going to come down whereas actually there has been a lot of from the Balkans wars onwards a lot of attacks using cyber capabilities so absolutely it's important to recognize it to find ways both of stopping it and using it but it has to fit in with other things and if you can't follow it up then the fact that you've caused damage means a lot of people are very upset but you haven't actually moved anything on politically and I think you always have to ask when talking about warfare is why is it going on what are the purposes, what are the grievances, what are the issues rather than just how did different people hurt each other I totally agree with you but if we go back to proxy wars where in fact you have nation states that might work through other groups and so they do have a political interest and agenda we can't overlook that but Dr. Cummings what do you think of this question about war from the air and the degree to which it can be effective or not or is there a question about a symmetry of commitment that it represents? So there's a couple of different points that I want to touch on you know as a former fighter pilot who is on the tip of the spear war from the air is what we like to do because it's safe for us safer anyway and it's distant and in fact this idea of putting distance between the war fighter and the war we've been doing this ever since the bow and arrow and in fact this is why drones have become the chosen form of warfare both for the United States but it's human nature it will propagate across every country and almost already has and so we want to back up the distance of killing because it makes us safer and it removes us from the battle I will actually tell you though that you know there's been a lot of debate about whether that's a good thing or a bad thing but when I was a fighter pilot I was given a mission I would go over a country it would be my decision using the imagery and the intelligence that I had and basically I had to make that decision and I have a lot of peers that made wrong decisions killed a lot of civilians killed a lot of their own forces I have a friend who was the F-18 pilot who killed four Canadians in the Iraq war and so he has never recovered he's an alcoholic domestic violence is very common for people war fighters who have been engaged in that and so how does drone warfare change that? Drone warfare makes decisions really by committee it's inconceivable to me being old school to think that I would be talking to a lawyer in real time who is seeing what I'm seeing which is what's happening with drone pilots now theater commanders, pilots, lawyers they're all making a joint decision and so even though we might think that it's distasteful to back up killing by the remote distance in fact we are getting much better decisions today not that they're perfect but then we were in the olden days and I really shrugged to say that about myself so I think that the campaign by air it's not going to stop because it's human nature I think that there's some benefits to it if you can actually get commitment from a partner a collaborator to get the boots on the ground I do think that there's some benefits to it but I did want to come back to this terrorism and this idea of terrorism and autonomous vehicles because I think that when we were embarking on a whole new world of when we start to think about not just drones not just military drones but civilian drones which in fact there are far more numbers of civilian drones by orders of magnitude in the world today than we have military drones we could argue that commercial aircraft they're fly by wire so they are just drones with a babysitter in the cockpit but now that we start to see driverless cars coming and in the United States and in other countries there's a big push for something called vehicle-to-vehicle technology where basically all vehicles are connected into the internet of things and I think what's interesting about the world that we're embarking on I don't want to be alarmist but the reality is that when we go to an internet of things of vehicles particularly driverless cars we will now have a potential worldwide connectivity of terrorism terrorists can get into the network and start hacking driverless cars and we've seen it in countless areas what's the barrier to entry when a terrorist can get into a network and hack a truck and the truck doesn't even have to have explosives on it you hack five trucks in the Washington D.C. area in beltway traffic and you put them in the right places you'll have traffic shut down for a while and that would enable terrorists to do a lot more activity in and around that so I do think that we tend to be in the United States anyway we are very much in love of technology and we want to push technology and we could argue that there's a great marketplace and commercial innovation and I'm in agreement with all that but I do think this idea of cyber security is really important and there's a huge connectivity between civilian cyber security and military cyber security that I think is almost virtually absent that we need to be thinking about today so Dr. Cummings you've actually illustrated very beautifully what I think is something that I try to get people to think more about which has to do with what I refer to as intersecting vulnerabilities with cascading consequences if there are triggering events and you just described one such example so I would argue that the thing I would ask you all to comment on is whether there is enough thought about intersecting vulnerabilities I don't think there is and I think one key point that comes from what you're saying is that offense is much cheaper than defense in those matters which means that it's highly destabilizing and we have to think through how societies are going to have to reorganize themselves because building a kind of comprehensive defense from the top down will not work if the diffusion of legal capacity among almost every citizen is there the whole idea of the state having the monopoly on the legitimate monopoly on the use of force all that is on the threat is going to disappear so how do you react to that? You don't want just to have the cyber equivalent of militias but you want to create something that will allow a much more decentralized defense if there is a decentralized offense and that means new types of political organizations So Minister Hennis you have to think about this every day of the week every day but it's true listening to you all of you nowadays it's not just about territorial control it's about networks getting into control of the networks and using it for your own benefit and cyberspace clearly is not well governed and regulated as you speak so there is some work to do you mentioned the convention the law of the sea I think it's in a very nice example and to copy this for cyberspace is not going to be easy but it should be made abundantly clear what states, businesses but also individuals can and cannot do in cyberspace again it's clearly we run the risk of cyberspace being the better space in the future so we have to and work is being done and we have to continue to work on the development of internationally agreed norms, standards and principles but reality is as well that international cooperation in cyberspace is clearly still far too weak to ensure that we will make progress on the short term now the Netherlands hosted a cyber conference on cyberspace in the year 2015 next year it is in India I know that the international community is working on it in order to support practical cooperation to promote capacity building and knowledge exchange to discuss norms for responsible behavior in cyberspace but we're still at the beginning and progress is what we need and unless if we do not succeed then again it's going to be the better space of the future two things, first most people in war are killed by artillery and guns that's how they die and the same way that they've died in wars for many a decade the reality of if you look at the images from Ukraine or Syria it's people firing weapons that are not even that much more advanced versions of weapons that we use in the first and second world wars so we always have to be careful not just to assume that the coming thing is transforming war the viciousness and brutality of war is quite basic the second point which slightly goes in the other direction is the interaction between warfare and criminality if I'm worrying about what's going on with cyber I'm more worried about criminal gangs to be honest than terrorists but the as Jean-Marie will know the interaction between criminality and militias is enormous how do they keep their wars going for so long they're trafficking they're dealing with people they're dealing with drugs metals whatever so one of the problems we've got is at the micro level the things that sustain war are the things that make about greed how things people can make their money so what the transformation element in this and where it does link in with cyber is the opportunities created for making money which is slightly different from acquiring territory and there I think you can imagine how people will want to use cyber for most terrorists it's about terror and terror can come from high explosives it's still the weapon of choice you would agree with me that cyber is a serious threat multiplier possibly what you've got with cyber if you want to use it as a weapon is knowing whether it will work because until you try it you can't necessarily be sure about whether or not your target has defended or if you've understood what cyber allows you to do is to test defense and those who wish to cause harm terrorists let us say we only have to get it right once and terrorism is fundamentally about just that terrorizing people isn't it easier put an explosive device on a truck and set it off all I'm saying is the danger is not we mustn't be complacent about cyber it is important it's a part of every conflict it's very when it's developing all the time but keep in mind that the people that we're dealing with often have a very much more basic understanding of what causes terror and it's being caught up in a very large explosion let me put something back to the minister for a second to what extent do you worry about nation states carrying on a kind of grey zone type of warfare through the use of militias that can be disclaimed or even criminal networks is that of concern to you it's a great concern to me you mentioned yourself to what happened with the annexation of the Crimea in Ukraine the South Chinese Sea it's already happening in a way it's a blast from the past but also something that will intensify and will definitely again be of great deeply disturbing to the world order it's going to increase I know you have something you want to weigh in on but I'm going to ask you to fold your comments over this question which is what of the multilateral institutions how do they have to evolve how do they remain relevant in light of what we're talking about or as Sir Lawrence is talking about perhaps things are as they've always been these institutions are still relevant I'm not sure that everyone believes that but perhaps the two of you could make a few comments I am biased having served in one of them of course but I would say that the multilateral institution they are as strong as their components and today if there is a crisis of multilateral institutions it's because there's a crisis of the states that are behind them and there's a crisis of the states and that's where the conversation connects with your question if there's a crisis of the states behind them it is because those states now they face a number of competitors non-state actors including criminal actors and the states themselves they're not quite clear what they stand for because politics is also in crisis which by the way is an explanation why propaganda operations can be so successful because if you're not sure what you stand for you are very vulnerable to all sorts of manipulations and so in a way our weaknesses are much bigger problem than the attacks so to answer your question I think that for multilateral institutions to have a future they need to adapt so that actors not criminal actors obviously legitimate actors can acquire a voice and that is something that is being tried at the moment but we haven't yet really found the way because behind that there's a whole question of legitimacy and the legitimacy still sits with sovereign states but sovereign states which have less and less control over their own affairs let alone the affairs of the world so to introduce those new legitimacies in a global discussion is something that we are searching for without really having yet the answer I would say Minister I mean much of this has to do with a lack of political will as we speak I mean this is the key in my view it's all about political will now to encourage this integration which is very destructive but it's happening and we cannot deny that the world order is multi-layered and that we have to deal with it and even if our multilateral institutions are under pressure because their components or the states are under pressure I fully agree with that still I come to the conclusion that those multilateral institutions remain the best avenue for effective international cooperation that we have to continue to struggle but also find the solution and empower those institutions because imagine a world without the UN or without the European Union or without NATO life would be really truly complex if it comes to how to run and deal and govern a global order so I think we have quite an audience this morning and so we would like to have the opportunity to hear from some of you with questions that you may have for the group and perhaps wisdom you may have for the group would anyone like to begin please and briefly identify yourself Hi, I'm Johan Andresen I'm the chair of the council on ethics for the Norwegian sovereign fund now we're supposed to recommend that companies that produce weaponry or significant parts that under normal use violate human principles or warfare we're supposed to recommend that those companies be divested from the fund now we at this point there are no companies that we can identify that actually produce fully autonomous weapons so we haven't come to the conclusion yet that those actually do violate those principles but I have a bad feeling that we're getting close and how close are we Professor Cummings because there's been tests done previously in October I think or there was 103 predicts let out and they actually behaved like an intelligent swarm with self-learning healing self-healing and well these were surveillance drones but I mean as you said they can easily be equipped as a weapon so how close are we really to to go beyond that line where somebody is actually producing it and somebody like us to actually make a decision on whether those companies should be excluded from this fund so I think that's a great question and it's a very very difficult question because it's multi-dimensional first of all what do we mean by fully autonomous and where is the goal setting in that so arguably we could say we have fully autonomous defense of weapons today so are we drawing the line between defense and offense are we saying it's okay to have fully autonomous defense of weapons but it's not okay to have fully autonomous offensive weapons and does that mean that for example we can take a tactical Tomahawk missile right now and change its coordinates in flight such that it seeks out a building on its own we could say that we have a fully autonomous weapon yes it's a set of coordinates but the way that the missile's reason are actually legitimately autonomy so are you going to get rid of all your you know going to divest from anybody who has anything to do with the tactical Tomahawk missile so are we saying okay well that's not what we mean we had a specific set of coordinates so now what we mean is that I'm going to say okay I want you to I'm going to show a picture of a bad guy to a missile and then I'm going to tell that missile after that bad guy okay that's what we mean by autonomy well there is no capability that exists today but I am concerned because as a former fighter pilot having seen many people make many mistakes humans under stress in combat do make mistakes their history is replete with examples of this it is possible and I'm not saying I'm advocating for fully autonomous weapons right now no one has the capability to do that in any stretch of my imagination in my world right now but I can conceive of a day where it is better to show a picture of a person to a missile a very bad person and that missile could actually do a much better job at targeting that individual than any other human could and if we are saying that technology in some cases and reduce human error then is that the right decision to do a ban on all weapons that have any kind of autonomy in there and so that's why I think it's very it's a very tricky area because we need to understand what are the capabilities of human what are the capabilities of these systems and should we be putting blanket bans on these technologies and my answer to you is no that's not to say that we shouldn't put measures in place in the interim but the future is the future and we cannot predict with any certainty what it holds it does raise the question of when there are artificially intelligent reasoning systems where is the real point of control because I think that's behind that there's really a question of legitimacy because you say for instance a drone that will have the picture of a very bad person that really is a great illustration of the blurring of the line between war and peace because traditionally a very bad person you bring that person to a tribunal and you decided what should happen to that person so the question there is do you have stock of a company that is controlled by a state which has a legitimate process of deciding which is a bad person which is not a bad person you decide that you sell weapons to a dictatorship or you don't sell weapons to a dictatorship I mean you enter a whole new set of questions there because it's no more really about it's the individualization of war that in a way changes the nature of war and conventions imply that there is a decision making process and there's a point of responsibility that can be assigned for human control that I address but in the end it's a question of political accountability as well I mean efforts to control particular weapons going back to the crossbow through the dum dum bullet they don't work because there's ways around them and in the end it's often about use rather than capability if somebody has using these weapons for a particular purpose however much autonomy they've got once they're on the way the responsibility lies with the person the government that has used them and I think keeping that political accountability in mind is important secondly we're taught I mean these ideas that assumes that attribution well that assumes attribution but it can that comes back to what are you doing it for if you're relying on people sort of knowing who's responsible to get your effects the second problem with with a lot of this again comes down to the problem well it's just in this control because we're all the time talking about two different levels of warfare there's this high tech, very precise very controlled idea of autonomy showing a picture I'm a reality of war most of the time in most places we're just shooting at whatever you can see high tech cyber attacks yet somebody's getting a truck and driving it into a crowd in Berlin there's these two different levels and they're very very different final point which goes back to the first point about not just a question of the weapon but the purpose we've just seen really in Syria the first sophisticated campaign run by non-western country now we were priding ourselves in the west the great thing about precision weapons is we could avoid collateral damage we could avoid civilians, go for the combatants, avoid the non combatants but the same precision weapons can be used to attack non combatants if that's what you want to do if you want to hit a hospital rather than avoid a hospital you can now hit a hospital with greater accuracy and you can try with those who take who make the decision saying that's what we want to do in order to coerce a population to leave which is what the objective was so let's see if we can get another question please could you use the microphone and say who you are excuse me I'm a little bit concerned by the way the conversation is going it often conflates the past with the future and even when we're talking about autonomous weapons we're talking for example if that's just going to be a drone that picks out an individual target but an autonomous weapons system is not a specific weapons system it's a feature set that can be added to any weapons system so we're talking about high powered munitions could also be autonomous and we are entering into a realm where these systems intentionally or unintentionally could start new conflicts escalate existing conflicts or just have failures as complex adaptive systems so that's just a little bit on the lethal autonomy question and I think really what comes up here is not so much whether we can stop the systems it's really how strong should meaningful human control be and should meaningful human control be that there's somebody there dispatching that weapon before the weapon gets dispatched not just that it's been delegated well in advance and I think that's what the argument is about it's not about whether autonomy will not exist and one other very quick thing on what we're talking about in terms of cyber attacks I understand that traditionally warfare has been about territory and it has been about the power invested in those territory but I think we're talking about something very different when you have a cyber capability of bringing down a society that is totally dependent upon IT infrastructure and I don't think we understand what would be the ramifications about that it may be as simple as just getting one of your adversaries out of the way for a while while you took territory from another adversary no I think you've made the very point that I think we need to spend more time on because that is where I see that is an example of the intersecting vulnerabilities and this question about the meaningful control is the point that the minister has been making repeatedly so thank you for reinforcing that I think those are very important points to bear in mind is there another question or comment so each one of you gets a final statement oh really? what do you want me to say no I mean there are many things we have to take into account and as I said we should not underestimate the new phenomenon like cyber threat or the outer space or the deep oceans because the government goods they're of great interest to many of us and it's going to bother us for the years to come so nothing to add really so why? we've argued the point on cyber it's not a multilateral institution point there's a difference between the role of the UN and the role of NATO I think if you didn't have the UN you'd have to recreate it in some way like the UN was a recreation of the League of Nations and if we didn't have NATO we'd find it very hard to recreate it the thing that worries me more than most other things at the moment is not appreciating the value of having that issue of alliance sorted in Europe because alliances forming alliances is a pre-war activity having alliances in place for such a long time takes that issue out of crisis management not because it solves problems it just exists and the notion that it's obsolete and therefore you could do without it and nothing much would change is just so wrong so alliances are very different the point about alliances is deterrence it's not about so pre-existing alliances have I mean they can be dangerous as well but in this particular case it has value for the ambassador well I would say that warfare has always been at the intersection of political institutions and technology from the arrow to the nuclear bomb to cyber warfare from the knights to the mass mobilization of the 20th century to the drones of the 21st century and it works both ways technology influences war and influences political institutions political institutions have an impact on technology I think what we are going to see now is that under the influence of technology political institutions are gradually going to have to change because the diffusion of little power that over time will be a great feature of our world will require a diffusion of power in the political institutions so the thing that kind of concerns me the most is something we have not even talked about here today which is the fact that in robotics particularly in drones right now I believe we are at a nexus where commercial drone capabilities are starting to far exceed those of militaries around the world so for example Google I believe and Facebook potentially have drone technologies that far exceed what any intelligence agency in any government has and I think that there is a lot to be thought about and said when corporate nations corporate states start to have superior capabilities to nation states so again it's hard to drop that bomb on you we haven't talked about it actually what you said is very important it's interesting because there was a discussion that I was part of yesterday having to do with political risk that perhaps corporations need to have a foreign policy and so it's interesting that you use the terminology corporate states because there is the question of so much of critical infrastructure and technology being in private hands and how then does that intersect with traditional governmental roles and governmental responsibilities that's where the question about standards and norms what conventions pertain who's involved in those and what exactly and then human motivation for conflict may never change but when the connectivity becomes ever greater the delivery systems become ever greater and you have this crossover between what is in the corporate or the private realm in the traditional governmental realm and new ways of carrying out proxy wars and even new definitions of them and they can then impact how one resolves conflicts you have what we discussed this morning so I want to thank everyone for being here and I especially want to thank our panelists