 I appreciate your coming out. I think this will be a really interesting event. It's a little bit of a different topic than we do most times. The topic is smart governance, how public policy can keep pace with technological developments. And when I was thinking about this event, I thought about how we're entering or we've entered an era of data abundance, right? And the product of all being networked and connected and having digital devices is an immense quantity of data on all kinds of activities from all kinds of sources, whether it's sensors or smartphones or geolocation or wireless networks. And this data abundance will change the security environment. It will change the business environment, and it's changing the social environment, but it will change the security environment. So the increasing availability of data is a challenge for governments and policymakers in finding new tools for governance and for security will be necessary. And that's what we're going to discuss today. Let me give you a brief overview of the event. We will start, as soon as I finish speaking, with the Honorable Secretary General of the Netherlands Ministry of Security and Justice, Mr. Peter Klu, who will give keynote remarks. We'll follow this with a panel that I'll moderate with Professor Frans, Director of Scientific Research and Documentation, also with the Ministry of Security and Justice. Dr. Robert Griffin, the Acting Deputy Undersecretary for Science and Technology. Many of us know him from his previous life, of course, and very happy he could be here. Dr. Chris Kirchoff, we have so many doctors on this panel. It's like being in a hospital. Dr. Chris Kirchoff, who is the Senior Advisor to the Counselor to the President, aka John Podesta, and was one of the authors of the recent report on big data, also works at JCS, has a long career at DOD, and so I'm very grateful that Chris could be here. And finally, Dr. Rob DeWick. I told you everyone was a doctor, who's the Director for the Hague Center for Strategic Studies. So what we'll do is we'll have opening the speech, followed by a panel, and then reception after this. We should run till about five o'clock. We'll see how long it takes, and then please stay and have a drink with us when we're done. But Peter, if I could invite you up to speak. Thank you. Well, thank you for your introduction this afternoon. I believe we could have, get even more money for the tickets because there's a lot of people who come extra this afternoon, but it stayed on the price we had initially thought about it. Ladies and gentlemen, very welcome to this symposium. I'm as a Secretary General of the Ministry of Security and Justice in the Netherlands. We are here to establish the relationship between the Netherlands and the US. We try to get some new arrangements, and we already did, with my delegation. And the third thing is that we are talking, and we are trying to get new insights, and also can give some answers and some information about things we are doing on cyber, on forensic, et cetera. I will give you a brief introduction. And I heard already Rob DeWijk is, of course, for the international studies in The Hague. But he is here especially because he is leading a combination from universities to ministries and about 150 companies in the Hague security data where we cope together on innovation, et cetera, on security. We try to get also in contact with people and to do some new projects, and et cetera. The Ministry of Security and Justice only has been there for about three years. And the reason is that it is a very big ministry at this moment. I think about 40% of the whole Dutch governmental staff is within the Ministry of Security and Justice. Because of the police, part of the ministry has become part of the Ministry of Justice. So we are now security and justice. But also, it's just being there three years. And it's, let's say, a merger, a group of companies. And we try to get some cement, et cetera, between the companies to see how we can cope with criminality, et cetera. Under the ministry, we have the National Police since 2013, the National Police. Before that, there were 26 regional police forces. That was very difficult for us. The prosecution and the judges apart are under the umbrella of my ministry. Then we have the prisons. We have counterterrorism, cyber, and so on. And that's the whole department. We had marvelous talks and meetings the last three days here in the US. We are very happy to be here and to see another day coming. The weather is very good. We had some space in the cargo department of the plane. So we took some nice weather with us. And I think we succeeded already in that. That we are very happy that this nice weather. When we look what's happening, these are the times of very rapid changes. Counterterrorism has turned into a cyber part and a foreign fight apart. And it was 10 years ago, of course, you know, totally a issue. We try, we see that also in the Netherlands, we see the same issues. So we have to turn the Dutch government and all the buddies who are working on this area, we have to change that. We have also to get the system, the juridical system, more up to date. I think it's just this part. I think we're about 10 to 15 years behind in ICT. That's not on cyber. That's not on forensic. But on all the other things we are doing between the prosecution, police, and judges. We are far behind. So we have an investment program. Only for the Netherlands, I think, almost, it will be about a billion euro for the next few years. So we had a large plan because we have to fit to get caught on criminals, to exchange information between the Netherlands and other countries. E-justice, what we call it in Europe at this moment, we are developing in a rapid way. So we have to do that very rapidly because otherwise the criminals will be ahead of us. The key word, I would say, is innovation. The key word is innovation. But what we saw is that we had an innovation that everyone did a part innovation. What we now did in the last two years is that we try to get focus on the innovation. And we have some people here with Klaassen who is now in charge within our ministry on the innovation because we want to have focused innovation and not just innovation to put some money on the street. And when you think about innovation, how will it be successful? And then you can see that, as you may know, if you have some experience there, you have to create a culture of innovation. Let me give one example of what we don't want and what we indeed want. When I was head of the National Police in the board for a year before I started one and a half year ago as Secretary General, then within the police force in the 26 forces, they all had their own innovation. So one functionality, let's say, number plate recognition, we had 43 projects. Nothing would be on a general scale set out. So we had just thrown away some money and they called it innovation. That's not what I call innovation. Innovation is when you try something new, you get the universities with it. You try to get also companies to fit in and then you try to see if there are two, three, four ways. We probably could, on one functionality, could get a step further. And that's what we call innovation. And that we have to do that and we can't have 43 projects because then we have thrown away the money. So a culture of innovation is important. That needs also a staff that's open to innovation. So the senior directors have to be open for innovation. Not saying, oh, we saw that already 10 years ago and then we put it away. So we try to get people who are thinking and acting out of the box. That is a little bit strange sometimes within the Ministry of Justice, but we have to do that and we are doing it at this very moment. In the US, of course, you have the companies like Apple, Google, et cetera. And those are IT companies but you don't have them to tell how they're getting on with innovation. So we have to learn from that kind of companies. So we have to be open on the ideas from society, on the ideas from other countries. That's the reason that we are here and in a month we are in China, et cetera. So we have to be open for new ideas, not closing discussion, just opening discussion and keeping on those discussion and making projects with a lot of people and parties. That also means that making a network on the innovation is very important. And the network is what I called the Hague Security Delta where and for universities but also two ministries and also more than 150 companies are working together on the innovation on security. And that's the reason that we try to get that type of networks because then we can act, we can learn, we can provide new projects and we can provide results also to society. And we are happy to be founder of those, the Hague Security Delta, what we call because it's only in the security and just this cluster. We did a good job last March when we had the Nuclear Security Summit in the Netherlands. We made a lot of progress on about, I believe about 50 innovation. We worked on a year and we applied it during the NSS and that was very interesting to see that and we made also an innovation room to let people, journalists, people from other countries so they could see what was happening. It's an interesting, if you want to know more about the Wijk, Bud Klassen and some others are here in this room. I'm convinced that we have to get progress on these innovations because we are sometimes behind the criminals. We see in the Netherlands and the UK has the same issue that we have some criminals who try to laundry money in Spain and we don't get hold on that issue. But we try to get the money away there, assets, we try to get the assets, but we have to then exchange more information between the countries on the theoretical level and we are now at the moment to do that. And the innovation where we are working on makes us work smarter, more quickly, safely, sustainably and often with less money, less people. But more based on IT. So we are now working on a ministry, white innovation agenda which includes one list of strategic subjects we need to know more about. Second is a list of subjects which could be solved by innovations and we want also to see which steps, that's the third thing we can take to encourage a culture of innovation. That's the interesting part and we hope to have also this afternoon more information about it. We will present the innovation agenda just after the holidays, after the summer, in the end of this year and that will help us to develop a comprehensive innovation policy for the ministry with our two ministers from migration and justice. This policy will help us perform in the most important social tasks more quickly and efficiently. I'm convinced about it. And that has to lead to a safer society founded on a strong, effective legal order. I thank you very much. Well, thank you for those remarks and I think to create a little space, I'm gonna stand up here at the podium. One of the things that I'd like to do is begin perhaps by asking the panelists a few questions and we'll open it up to you but maybe we could start with each of them giving, they all have a different perspective, they all have different responsibilities. Maybe they could, as they answer the questions, I'm gonna start with work in their additional remarks and think about what it is we want to discuss here today. Let me start by asking them, so we've heard a speech about innovation agenda and what are the big innovations we ought to be thinking about? Should we start with you? We'll just go up and down the row and that will make it easy. Thank you. The question about innovation is important because innovation, excuse the analogies, sort of like pornography, it's sort of the idea of the whole way to know when you see it. Part of what I see is that you're starting to see, I'm sorry, can you hear me in the back now? The joke was brilliant at the beginning. But it's only good once, so I can't repeat it. Bob, your remarks have become suddenly much more interesting. Yeah, or less depending on where you are in the room. I think the issue of innovation is a challenge for us. We have an opportunity to look at what I call convergence. So as we start to think about data and we start to think about interoperability and we start to think about communications and we start to think about gear, you can start to see where this issue of how we can operationalize data and bring it to the street and really change society, really change how first responders can respond as far as the security's concerned. But you have to stop thinking about these issues as separate siloed areas of investigation. Data is only as good as how we use it. There's a proliferation of data. Data is all around us and it's going to continue to grow. That's both a challenge and an opportunity. If I were to use, if I was to geolocate all of the data that we could have in DC, the map would be black. There'd be so much data that you couldn't use it. So part of what we have to think about is not just innovation but how we use tools differently that we already have. And that's a challenge, that's not always easy. But it's part of the opportunity that I see with both data and security. Oh, great, thank you. Sure. Chris, you of course worked on the White House big data report. And so maybe you could tell us a little bit about that and where you think it came out, what you think it sort of left up in the air for things we have to do in the future. I'm sure it's very exciting to have a report. You have one copy with me here where you go to whitehouse.gov. Slash big data, you can read it for yourself or actually watch another supplementary information. I mean, it was an interesting process to watch the 90 day study come together. As you all know, the 90 days is a very quick time period to produce a comprehensive report. But nevertheless, in that amount of time, a very talented team in the White House came together with some folks across the government, including the secretary of energy, the president of science advisor, the secretary of commerce and some others and had a phenomenal series of interactions with stakeholders of all kinds. We met with everyone from European privacy officials to members of industry to other folks across the federal government and even local and state governments that do interesting things with data. We are unquestionably in a kind of golden age of innovation so far as data is concerned. I'm on this particular topic just coming away from the study, frankly amazed at all the different places that big data will come to figure. As the report notes, there are many, many opportunities for the government, for the commercial sector to use big data to enhance how we live and work. There are also a number of challenges from a public policy standpoint that come along with the use of big data. The report in particular looks at potentially discriminatory uses of big data, which was an unexpected finding to the group. We went in not at all imagining that we would find that big data effectively eclipses in many of the regulatory frameworks that the US context had been used to regulate health, employment, education in many other areas of our lives for a long time. But that said, the report, I think, does catalog a number of really interesting opportunities and hopefully later in the panel I'll get to talk a bit more about those. Maybe I can just take a pause here from going on the row and say, Bob, you have real operational experience. And so maybe you could tell us in a few words how you think the access to data will change the safety procedures, law enforcement, fire and rescue. What should we look for in the future? What do you think will change? Yeah, there we go. I push the button. Thank you for the signal in the back. I'll call out what work that's being done in New York City on the Center for Urban Strategies and Progress at NYU. If you get an opportunity to check out the website, go to cusp.nyu.edu. It's pretty interesting, because what they're doing is they're starting to look at all of the subsystems within the system, so it's within the city. So they're looking at New York as basically a laboratory of how to pull together urban studies and informatics. So some of the projects that they're looking at is how could you start to think about combining your first responded data for your emergency medical systems with your transportation system data with your hospital access and availability and the type of call that you have in order to be able to send, not just the closest ambulance, but send that ambulance to the hospital that has the subject matter experts that can deal with that person who is in need. So you start to think about how you could combine multiple systems together in order to address that medical issue. You start to think about the ability of the police to work with the transportation officials. Who was in the DC area when we had the whole issue of the sniper? Remember the situation where we had a sniper loose in the area, basically terrorized the entire community. Police were looking for white vans, that proverbial white van. Imagine a situation though where there was a report of a bank robbery or a sniper situation and the police could work with the transportation systems and you could shrink the light cycles of your red lights from 15 seconds to 30 seconds to even a minute. Without people even knowing it, you could shrink the perimeter that the police would have to look for the bad guys. So if you start to think about our ability to use existing systems that we have in order to meet some of these pressing public policy issues or public safety issues, there's huge potentials. And this isn't wild sci-fi. These are just systems that already exist but we haven't figured out how to work them together for the best needs of the community. Sure, well it's CUSP, C-U-S-P. So it's a center for urban strategies and progress and I do a lot of work with the geo-visualization. The big companies, you think about Esri and you think about Google, but you think about some of the academic communities. Purdue is doing a lot of work, Rutgers is doing a lot of work, USC, Stanford, and CUSP is really interested because they've set a whole center up and the mayor put a lot of money into the center and it's looking specifically at the city. It's doing some neat stuff like measuring effluence to see if it can't be a predetermination for outbreak of diseases, to see if they can't start to measure the fact that there's gonna be a particularly bad flu season. Not very, you know, I don't get scatological, it's not lovely work to think about, but you think about the systems that it takes to run a community and how we can pull that information off to really understand the health of the community. It's huge potential and that's what the promise of big data can provide us. Thanks and one of the things I hope we'll have time to come back to is something I know Chris focused on in the report which is the implications for privacy in particular of all this and before we do that, I wanted to ask Franz to perhaps, the question, if you could remember was, you know, what are the big innovations we should be thinking about in this field? I fully agree with the situation you described where big data are very important if you see the beautiful little institute at the United Nations level, global pulse where big data are brought in line with development aid questions, remote monitoring instead of asking people questions, you know, when it rains, nobody answers the question and now you have the data available, very small unit, beautiful work for that field. If you see what Hopkins did with the Google prediction, although there was critique on the methodology in my office in the Netherlands, we tried to forecast certain things until now we use looking back data from, let's say, statistics Netherlands, but you can also try to get these data by looking into the same way as dengue and the Google influenza has been treated, which is not only cheaper, but much faster. For me, that's a very big innovation and if I may add one other thing, when cyber and bio-social studies go together, it is even getting more interesting because we know, and particularly because of a number of Americans who have been working in this field for a longer time than Europeans, that what causes terrorists and organized crime and killers and psychopaths and even, as they call them, CU people, callous, unemotional people, there are definitely also biological aspects related to that. If you have these data, you can add that. That can also, I think, be very helpful in predicting certain things, but I agree privacy is then really, really a big issue. Some of you may have heard me say this before, but I know one of the companies that develops algorithms for data mining and data services and I didn't know this. You can buy Twitter data in bulk and so people tweet really strange things like they'll tweet red dots on arm, right? And you can use it to predict unemployment and disease outbreaks and social unrest. So this bulk data is a very powerful tool, but I think we will have to come back to the issue of privacy. Rob, let me turn to you before we do that. Do you have a question? Or? I do have a question. And the question is when we talk about innovations that we should be thinking about for governance and for security, what is it that you're looking at? What is it you would put at the top of your list? Oh, we look at many things, especially clusters of issues, but let me quickly explain what the high security delta is because most of the people in the room might not know it. It was mentioned a couple of times by the Secretary General. Secretary General also said that in the Netherlands, we should create this culture of innovation and I fully agree. And I think this is also my task as general director of the high security delta to assist him in creating this culture of innovation. This is not something that comes natural. For example, if you make the comparison between the Ministry of Defense in the Netherlands and the Ministry of Security and Justice, defense has a very, very long tradition in this and a very close cooperation with industry and knowledge institutes. Now this is something we are now also organizing for national security and that is quite interesting and we have two aims that is drive economic growth and produce innovations, security innovations for acceptable money, for acceptable costs. Now what are we aiming at? There are a couple of classes and what we try to do is we focus, we link this to our national security strategy. So we defined what the security of the Netherlands is and how it could be negatively affected. And based on that, we came up with six focal areas that is national security in the first place and that deals, for example, with questions how to arrange your commander control, how to do it, what kind of procedures do you want to have because we also came to the conclusion that technology is important but it's not everything. You also should have the concept, right? There should be a mentality of, which allows you to do national security disaster management, disaster relief in an effective way because if you don't have the concept and you do not think in an operational way, then you simply do not have the desired results. Cyber security is a very important one, I think, all over the world, critical infrastructure, which is also, of course, closely linked to the whole issue of cyber security, urban security, which is becoming more and more important. It's a broad area but this, for example, linked to the issue of camera surveillance. Forensics, very important. I think that the Netherlands is indeed world class in these fields and, of course, education. We came to the conclusion that, especially in the field of cyber security, we lack expertise. Everybody is dealing with the topic. Many people have an opinion about cyber security but there is actually very little real expertise. So we started a cyber security academy in order to fill the gap. Let me say, if you have questions, go ahead and interrupt me. I have loads of questions so I'm gonna keep going until they're exhausted. But the question's not the panel. But maybe what we'll do is we'll leap to the privacy issue because I don't know, it'd be interesting to get a survey here between the Europeans and the Americans but I really was troubled by the recent decision about the right to forget and I think that might be a cultural difference between the US and Americans. It would be interesting to see. So maybe we can go down the panel again and have people say, we've talked about big data, we've talked about the benefits, we've talked about new technology and facial recognition. You have a question, I'll get to you next, yeah. And then what are the issues for privacy? How do we need to think about privacy in an environment where technology will give real benefits to security but it could also create risks that we might not wanna take. I don't know who wants to start. Chris, do you wanna start and then we can go around. I'd be happy to, I'd like to thank Bronze also for introducing a new diagnostic term to my vocabulary, which is careless and emotional. So I think I might be able to use that term even in the absence of big data analysis in certain contexts. Privacy is a really interesting and central topic when it comes to data and data collection and the 90 day study was done alongside a report conducted by the President's Council of Advisors in Science and Technology that surveyed the technological underpinnings of the big data revolution and that report is also worthwhile reading and I commend it to you. That report in particular, which our report builds on similar to a few trends that are just undeniable and that is simply the proliferation of sensors in the devices that we use and have in our homes, also in our workplaces and our cities. The way in which computing and processing are dropping rapidly in cost and the way in which data storage is dropping rapidly in cost and what this leads us to is a world in which increasingly data collection is functionally ubiquitous. It's simply everywhere and much of that data that's collected is effectively permanent and there are very logical and good incentives that are driving this and that has to do with the way that data analytics works. We may be collecting data today in a number of different contexts that we discover new uses for tomorrow because the techniques of data analysis that are applicable to that data that have not actually yet been invented. So in that kind of a world, it's very difficult to do anything other than legitimately want to collect a lot of different data. So that is a very different world than a lot of the foundational privacy regulation has grown up in particularly the very ubiquitous notice and content regime which is embedded in all of those little user agreements that I'm sure all of us, all of you diligently read like I do before clicking okay on whatever app I just downloaded. So the big data report does ask the question of if we are entering a world in which data collection is functionally ubiquitous and data storage is effectively permanent. Does it make sense to increasingly organize the regulation of privacy around the use of data rather than merely its collection? Franz, do you want to do it next? A little bit earlier, we were discussing is the this concept of privacy by design. So that where you when you are, you know, collecting or downloading or getting the data because indeed you're completely right. It is totally different than 20 years ago and you make your databases or your data spaces you organize it in such a way that there are disclosure rules very clearly that automatically let's say stop you like the automated pilot in a Boeing 747. And I think that is something that we could work on and I know that your organization is working on that too. And there is one other thing however that I would like to share with you that is from surveys and I know we have to say surveys are old fashioned data collection type. We know that young people in particular in Europe, I don't know exactly how it is in the United States are less interested in privacy than maybe people of my generation. And that is something we have to take into account. Does that mean that they are really less interested in privacy or is it only because till now they have not experienced bad things. And I don't know whether the report that you have made whether you pay attention to that too because if that is a different mindset a different set of values then I think it is relevant that we know how to handle. What's a very difficult subject to say the least because on the one hand people don't care about privacy. They use search engines, they use Facebook, they use Twitter and on the web there is almost no privacy anymore. But on the other hand if you look at the Snowden case people feel very uneasy when data are collected by governments and this leads to the conclusion that this is probably about mistrust or lack of trust between the people and those who govern them. And I think that is a real issue because people do not give, well, they allow data collection. They think that this might be a good thing. They collaborate on this, Facebook. And on the other hand they do not want that our intelligence services collect all the data needed to protect them and there is always traditionally a balance between privacy and security. What you now see is that the whole debate goes in my view in the wrong direction. More privacy and less security. This is already the case and not only in the US but also in my country where it is very difficult for example now to change the law in order to allow cable signals. That is a crucial element of the whole debate and it is not really debated in the Netherlands in public. How to deal with it? I don't know yet. I think time will change the whole debate. But so far this could have great consequences. What we already know is that for example criminal terrorists they don't use the phone anymore and they don't use the internet anymore not at least in the way they did it before. Before they knew that they will survey it on the internet. So it has all kinds of negative consequences and we should take this into account and we should change the narrative and we should explain it more and it comes back to what I said earlier about the expertise. There is very little expertise. People do not know what mass surveillance is. They do not know what for example metadata are. They simply don't know and how it might or might not affect their privacy. Thank you. Bob can I ask one additional thing which is given your remarks on the operational benefits and now we're talking about the privacy implications of that greater data. Could you also maybe talk a little bit about what are the requirements for transparency? What does it citizens need to know that their governments are doing and after Bob if any of you wanna jump in then we've got a couple questions. Actually in some respects I think the discussion about privacy limits the discourse that we should have been having and we continue to need to have and that's just where the balance is between Homeland Security and our civil liberties are. I mean this is a discussion that I really felt we should have been having even right after 9-11 as we started to think about the department standing up. I know that sounds very esoteric but really that's what we start to talk about. It's the trade-offs of things like our privacy for security or even a sense of security and it's part of the conversation we need to have. So when we think about transparency and we think about a meaningful dialogue within our communities I think there's two points that technology, I'm gonna talk about technology for a second. You know technology has, the use of technology has two sides. I mean there's positives and there's negatives. So even something like social media which has been an amazing development has its downsides. So if you think about the Tottenham riots in the UK social media was used to spoof the police. So the police were following the social media and they were protecting the liquor store on green and seventh while the clothing store blocks over was getting sacked. That doesn't mean the technology's bad. It's the use of technology and I think one of the things that we have to do within the Department of Homeland Security particularly in science and technology directorate is engage in conversations with the community about what are the acceptable uses of technology within the country to protect both civil liberties, privacy and the civil liberties but still try to meet the need for a secure homeland. I'll give you for instance, I think the issue of the framing of how we use drones. We called them UAVs. I knew the debate was really changed when they referred to as drones as a Star Trek fan or a Star Wars fan. Drone is a bad thing, right? But you look at it and you think okay, here's a technology that could help our first responders in the field. Anybody who's ever been out in the field trying to find Alzheimer's victim at three o'clock in the morning, if you could use a UAV to do that, you could save lives and you could save potentially a family a lot of heartache. But we don't engage in the conversations early enough in the development of technologies and I think that's one of the areas that we could think more broadly about. I'm gonna hit two other points though if I could. I have two kids, 18 and 17 and they're brilliant. They absorb data in ways that I can't. So I say hey guys, have you done your homework and I walk into the rooms and they're online playing games with friends who they've had for more than 10 years who they've never met. They're texting, they're listening to music, the TV's on and I look in and say shut all that stuff off and do your homework and they look at me like I got three heads saying we are. They're able to take in information and process it in ways that frankly I just can't do as an old person. So I'm not sure that we really can appreciate some of the issues or thought processes that the younger generations are going through as they start to think about themselves and data in privacy but I think we have to have that conversation with them. The other concern I have about this whole issue of big data and I'll toss it out because maybe we can have some more discussion is that if you haven't heard the term apophenia, you should look it up. It's seen fake patterns or imagining patterns in data that don't exist. So if I draw a dash in two circles, what do you see? There's a smiley face, right? Everybody says oh, it's a smiley face. No, not really, it's two circles in a dash. We also have to be really careful about the whole issue of crowd fallacy. There's a lot of thought that social media and mining big data is going to level itself with the wisdom of the crowd. Well, the crowd's not always wise. And one of the things we have to really think about is how can we bring science to understand where the limitations of the data is but also how do we address the validity of some of the data, particularly if we're gonna be making really important life threatening decisions. So those are two issues that I'll toss in the table of areas that I think we really need to think about as well. Anyone else wanna jump in on transparency? Well, maybe. Transparency in terms of a thing that I do not find that often in the literature until now, the use that we can make of big data in the field of evaluation. I mean, the United States is of course the country for evaluation with your golden years in the 60s, et cetera. Europe is catching up, but the link between the big data that we have, the locks that we have, there is hardly any need now to do before and after measurements and try to recollect the situation how it was four years ago because that's all locked. And the world of evaluators is not yet into this. Of course, there are downsides, but in general, if you have the data and you can do your studies over the years for very little money, that can be an input to evaluation, both exposed as also ex-ante that I think is really helping the community. And I think in general, we can say that we should be happy that there is a serious level of evaluations going on, giving all the public policy and the money that is involved in public policy. So that could be a new field of utilization for the new field of big data. Well, transparency is not about providing as much information as possible because I'm not really impressed by young kids who are able to do everything at once. And the reason is that there is a big difference between information and knowledge. Information is not knowledge. And what you see is that especially young people, but let's say the internet generation, although I'm working with the internet for, God knows how many years already, and I bought my first computer 40 years ago, so I don't know what the internet generation is and I know more about computers than my kids who are in their 20s, so I don't know. It's, no, it's very easily said, so the internet and the young people, but I'm also a professor at the university and what I notice is that the knowledge is not what it was anymore. And the reason is that people, that this generation simply cannot distinguish anymore between right and wrong, between opinions and facts. That's very important because they don't, they are not able to give the information a place. They can't judge the information anymore. And I think that is really underestimated. If you have, if you talk about transparency, then you should know what kind of interpretations one must give to the information. And if you can't do that, and large groups within the society within an effort or increasingly complex society cannot do that anymore. Now if you talk about issues like privacy and about the very complex issues of Intel Collection or the whole NSA debate, you only can judge what's going on if you know something about it and you can place the information about privacy, the NSA, our intelligence services in a broader context. Otherwise, you reached a dead end street and that is highly dangerous. And we, I think that is the key issue we should discuss. I will say the European Commission, I think it's funding a project that will create a tool, a software tool that will be able to distinguish truth from fiction on social network countries. So it'll be interesting to see if it works. My solution is to everything is usually there's a software solution. So Chris, you've been quiet. Anything you wanna add to this or should we go to the questions? Sure, I'll throw one example on the table that maybe connects some things that Rob has been discussing in terms of the legitimate exercise of government power and some of the privacy issues that have been on the table. And that is actually Rob Griffin's department, the Department of Homeland Security is featured in the third chapter of the report on the uses of big data in the public sector. There's a really interesting pilot that DHS has embarked on called Neptune. And it's a data lake, it's a kind of database that's able to ingest an enormous amount of different kinds of information. But what's really interesting is the way in which it ingests this data and then allows people all across DHS to use it. So first of all, it's important to put on the table that the Department of Homeland Security is a department in the government that has a full-fledged privacy office as well as a full-fledged civil liberties office that our staff with experts who have a lot of in-house expertise when it comes to law, the law and responsible use of data. And they got together and constructed in this big data pilot and created a massive table of meta-tagging. This is the ability of databases to add fields of data that capture what kind of information is being collected. The authorities and context in which it was collected and then the uses to which it can be put. Such that you can have really individualized controls over pieces of data. So for instance, if you're an agent at the US border and you're in primary inspection and somebody comes up to you and asks to be admitted to the United States, this database can only show you particular pieces of information that are relevant to your initial determination of entry. Whereas investigators in secondary inspection would be able to see quite a bit more DHS, investigative agents could see even more and intelligence agents could still see more. So what this allows in the public sector and could potentially be a best practice that's much more widely applicable than just in DHS is the very precise control and responsible use of data in a way that actually enhances privacy and civil liberties. So you may actually see a situation in which big data and data analytics properly implemented can actually enhance and protect privacy to greater degrees. Sorry, thank you. Lorelai Kelly, I'm at the Open Technology Institute at New America Foundation. Thank you, Mr. Lewis for all your work. I love, I'm only a wonk would say this. I love your writing for so many years. My question, I walked off a plane at National Airport the other day and I ran into a billboard that said are criminal hackers ruining your business? Here's one problem, in California hackers are artists and in Washington DC they're criminals. I have a real issue with how lots of governance issues with technology are framed always as national security issues. I worked in Congress for 10 years and just saw the migration of complex problem solving go into the military committees for example. As a national security wonk, I think it's a tenuous place for a democracy to always be categorizing technology and national security together as governance. In fact, I would be really interested and I look at Congress and technology. One of the issues that I think we really need to grapple with with the big data revolution is how do we turn this revolution into a policy making incentive system for evidence based decision making? I know that this is really important especially on issues of climate disruption for a place like the Netherlands. But we don't have a lot of examples out there like what you're talking about, supply chain tracking for legislative policy where experts can be in the room at the right time. Who's doing this? Can we do it? And how can we start putting these best practices together? Because right now, public interest does not have a chance in the US Congress. Don't all speak at once. Good. Well, I think it's an interesting problem you put on the table but I am not in the position nor that I know enough about the American situation to say something about that. But you heard the reference to Global Pulse which is an organization, United Nations which really tries to use it as an agent for democratization. They use it also by the remote monitoring. So no longer, you know, do the traditional monitoring but remote monitoring. That is actually power to the people. There is nobody in between. It's the people who give the responses by their behavior who use their apps and that is one of the goals of this organization to use this information to gather with other evidence from big evaluations. I think the same could be done for security and justice but that is a new way to go I think. I'm not sure that this is not answering your question but only for a little bit maybe. That's a good place to start, thank you. There's a couple other places I'd point to. One is that I think as we look at how the environmental movement has used geo-visualization to explain the impacts of climate change and whether you believe a climate change or not it's really not important. You can look at the maps and you can make your own determination. When I think about community resiliency and public safety though, a lot of the work being done at the local level I think actually is pretty sophisticated. We've been running a couple of pilots with California and in fact we have a next generation and command system pilot that brings together lots of community information it's been in use by Cal Fire in the last year was just used in the big wildland firefighters in San Diego County but it brings in things like social media, it's designed to be able to get better more actionable information not just to the first responders but also to the residents. Gives them a sense of where the danger is, as far as their homes and as far as they are concerned. It allows the policy makers to start to think about what type of resources that they have to move into place. So the headquarters of Cal Fire in Sacramento is actually able to look real time at what the conditions are and what the impacts are and what the potential impacts of things like weather change, time of day, those type of issues that affect fire flow and then I'm gonna take it up a level. When Cal Fire has a better understanding of how they need to start to shift resources from the north to the south to deal with that wildland fire, FEMA has the ability to start to think about what kind of resources that they have to make available to California to help shelter maybe a million, a million and a half people. I mean you start to think about all of the systems that go into taking care of those people who had to evacuate and fight that fire, you can start to see where the power of data and visualization can really make a difference in the community. So a lot of the work that you see that I personally see as being really beneficial to the country starts at the local level. What I don't think we've done well is figure out how we start to build that as the building block of data that Homeland Security really needs in order to think about the safety of the country writ large. So rather than say top down the way we've done a lot of things, it's gotta be bottom up because that's how the communities grow. So a couple of errors you can think about. I do share the discomfort with labeling everything on national security problem. So one of my goals in life is not to put the word security after any problem, climate security and food security. Food security is not overeating as far as I'm concerned. But please go ahead and if you could identify yourself. Joe Coulter used to be with Echo Storm as the back office of DHS at the NOC and AMOC and back office at DOD for full motion video ISR. And I've seen all this social media stuff and I'd say where is the world's wealth? It's in rural areas, production, energy, mining, ag. And yet we talk about a system that's vertical. All the data just goes one direction up. It doesn't go out. My friend Jim Sisko wrote that book called 90% of all the money spent on intelligence was worthless to the warfighter or first responder because all the data went up. It didn't go out. Expertise is distributed. Expertise is not centralized. In fact, rural areas around the world, the expertise is distributed. So I always think people that Snowden told everybody about metadata collection but it's actual metadata distribution that is really important, not metadata collection. Because if you're gonna evaluate a training program and do nothing about it, why'd you evaluate it? So my question to you is we're great in this country of building bubbles. We built the housing bubble, the internet bubble, the China bubble, and now we have the cyber bubble. And we think it's only about software. But guess what? If the Chinese have the chip at the door and it's part of your system, it doesn't matter what you do above the chip. So what are you doing to control your supply chain to make sure your clean chips of your mission critical systems? How do you do that when the big boys, if you ask them a question, where's their supply chain? They can't tell you. You're gonna skip either. I have a question for another one's colleagues after that, so go ahead, go ahead. Yeah, I came to the White House from the Department of Defense, so I'm sure Rob can follow my answer as well. But obviously, security is a paramount challenge in the age that we're living in, and the security of supply chains is one of the hardest dimensions of that. What's interesting is some of the research that's going on in places like DARPA that have asked the question, okay, if technological systems are growing so complex in terms of the components that come into them, it's simply impossible to verify them completely. Is there a way that we can actually design a network to work where we could actually run it on a dirty network, a network that we know is not secure and still achieve for ourselves and our users the degree of operational security that we need? And some of those emerging research approaches are quite exciting and might perhaps offer partial solutions to the tremendous challenges we face. This is gonna be completely unsatisfying answer, but I'll give it my best. I don't know if Dr. Doug Maughan is still here. I didn't see Doug. Okay, he's in the back, he does a lot of the cyber, does a majority of cyber work for science and technology, so I'm gonna throw him under the bus because he's not here. When you're done, Doug can give you a much better answer to this than I can. You raise a couple of really interesting points, and it's also a challenge in this whole area is that you've gotta think beyond software, you gotta think beyond hardware, you gotta think about the people as well. I was always fascinated with one of the studies I read coming out of the Pentagon where they were checking their systems and they freely distributed a bunch of thumb drives in the parking lot. And the number of thumb drives that were then plugged into systems because people thought they were being nice trying to return them to their owners, but they actually corrupted their systems was pretty staggering. So you've got these multiple elements that you need to think about. Part of the challenges with cybersecurity too is that so much of our critical infrastructure is owned by the private sector. So it needs to be a cooperative relationship between government, the people, and industry. And we really need to think about how we incentivize both good practices and better cybersecurity in areas where the government just can't mandate what needs to be done. Part of it's gonna be market driven, but part of it's gonna be regulatory part of it's gotta be cooperative as well. So I think that's part of the challenge. The other point that you're getting to is a realization that all of our critical infrastructure is interrelated. And this is a problem we all have with this issue of big data in privacy is that the technology in data is so interwoven with all elements of our lives now that you can't possibly pull the thread on most of it. I mean, I suppose you could just shut it off and go live in a cabin in Wyoming, but that's not what most of us wanna choose to do. So we're gonna have to think differently about a system, particularly a system like the internet that wasn't designed for regulation and control as we start to think about how it grew up. I didn't have one of the initial apples, but I had a whole bunch of old computers. I wish I had now, because I'd have something I could give to the Smithsonian. But that whole area is changing so quickly and technology is changing so quickly. I'm not being cogent. It's an area of cyber security that is going to take more thought and overlays than what we've been able to give it. And part of it too is that we're gonna have to make some hard decisions about potentially tidying up on some regulations, also as consumers pushing a lot of the business community to try to do the right things where they may not have been as willing to do in the past. That's a hard lesson. Let me ask our, another one's colleagues just on supply chain. We've been talking about innovation and how it can benefit governance. To a large degree, innovation is dependent on technology and on new technology. And technology is dependent on a global supply chain. So there's risk. How do you think about managing that risk? What do you do about supply chain? Yeah, please. First, we should not overestimate technology. It is important and it brought us many things, but if you make a comparison, for example, with military, that's quite interesting. It was always predicted that new weapons systems would change the art of war completely. This didn't happen. Why not? Because people adapt. And the same holds true for the whole debate on metadata, big data in general. You asked the question, it only goes up, but what comes down? It should come down in the form of analysis that can be trusted. It should be made available. Only the government has access to those data, make an analysis and make it available to the public. I think that is extremely useful. That should, for example, that is something that intelligence services should do. In the Netherlands, they do it. And I think that is absolutely great. So don't overestimate it. And if people feel uneasy because of they think that they are on the surveillance all the time, they will simply change their behavior. That is what they do. They will circumvent. I'm not too worried about it, but we should take this into account. This is what happens if you put so much emphasis on new technologies and on the collection of big data, then people don't do it anymore. That's exactly the reason why I'm not on Twitter, why I'm not on Facebook. And I ask my people to be very careful what they put on the service of our computer system. Yeah, Steve Winters, Washington-based researcher. Granting the point about the cabinet in Wyoming, I do hear a concern frequently raised, and apparently this has become an acrimonious current debate in Silicon Valley now with the letter of Cory Doctorow to Eric Schmidt. Mainly, while the privacy of the individuals in the society seems to be coming less and less, the privacy of the companies with their own data, such as Google, seems to be coming more and more, and the governments seem to be spending more time classifying more and more material, making it not available. So some young people, and Doctorow, not so young, see a lack of parity here. If, you know, where is the principle, you show me yours, I'll show you mine. And certainly the types of sentences and given to people who've hacked into and released data that they weren't entitled to release are extraordinarily high, again, showing somebody thinks this privacy is extremely valuable, but not for the individuals. And of course, you can talk about miracle and everybody else and Feinstein in this regard. So this is something I think needs to be addressed because people are worried about it and I often hear it expressed. Sean, on that point, there's a really interesting set of policies that the Obama administration has pushed forward now for a number of years on the whole topic of open data and open Gov. And if you go to data.gov, you'll see the fruits of a lot of that work. The default has also changed thanks to an executive order. So it used to be the case that government would create data and then data would kind of stay within the government. The default, as mandated by the new executive order, is for data to be created in a machine-readable format and the assumption is unless there's a privacy or a security reason for that data to be released to the public. And this is a very different way of looking at data. It's a way that treats data like a national resource rather than like something that's simply for the exclusive use of government. So there's actually now in some departments kind of data entrepreneurs whose job it is to receive requests from outside the agency and then to look for data that agency may hold that could be valuable to somebody in some way. And a world in which data is increasingly able to be applied to a variety of things, this is certainly, I think, a national asset. I usually think of Netherlands as an innovative country. Maybe you don't feel that way on the inside, but from the outside it looks that way. So what are you doing about data and the things Chris is talking about? It's an important initiative to make data available. We have also an open data policy to start with Statistics Netherlands, one of the biggest organizations which combines registered data going back to 15 years so we don't have a national census anymore because we have a virtual census. So we combine all these data and they are open. They have a website where you can make your own calculations. So want to find out whether a second marriage is okay for you given your age or your third marriage, you just do it. I recently gave at a party, I made it a little awkward or somebody was marrying the third time with a wife which was 15 years younger and I calculated this mortality risk which of course was not completely, it was a little bit of a joke, but that's open data, it's all there, hundreds of variables. I hope you didn't tell him. I told him at the end, yes, yes for sure. And it was a good friend so don't worry. That is one, with regard to data from crime injustice, of course you have to be a little bit more careful. You don't want to give the callous, unemotional, faculty pass the data as opposed to they could read it. But trans over time are available at the website of my institute and again you can calculate the reoffending risk on the basis of our reoffending index. Of course, dear colleagues, one big thing, these data are not specified with name and where they live, they are completely anonymized. But that is a way to maybe win some trust by being open. The policy on big data is in the making at this moment. So that's why I find reports so interesting because you bring all these things together. I used to work at a place called the National Opinion Research Center, NORC. Yeah, and we had the state, we had anonymized census data and you could waste endless hours typing in ridiculous correlations, you know, like did the weight of pigs in Iowa predict the smoking patterns in Chicago and things like this. And often I thought, I don't really need to ever do research again, I can just do ridiculous correlations of huge data. May I say one thing back on that? I think that one of the findings in the research literature is that innovations often come from clashes of disciplines, some lonely intellectuals doing the things that you were doing and then suddenly there is something new, so who knows? That's very touching. We had one more. Hi, my name is Talan Söpp, I'm from the Embassy of Estonia. And as in the Netherlands, I think Estonia has also profited quite a lot in using innovative solutions and all kinds of e-solutions. And we've talked now quite a lot about public interest. But at the same time, there is also individual interest. Just as an example, in Estonia's case, the main policy for the government, towards individuals, is that you ask certain things just once. I mean, your name doesn't change, your birth of date doesn't change. So by this, you can make it easier for people to interact with the government and people will be more willing to interact, actually. But being here as an observer in the States, creating this kind of trust environment is quite difficult. This mistrust towards the government is re-inherent. So as a kind of joke, how can the government create conditions or the environment for people to give more data without people being afraid that the government will take their guns away? Thanks. And maybe I can add something to that, which is for our European colleagues. What do you think about similar attitudes in Europe? Do you have the same kind of distrust, is it different, or is it very much the same? But why don't we start with Americans to answer the question about black helicopters, and then we'll go to another one. No one wants to touch it, yeah. Boy, this could be a career-ending answer. I think we'll shoot it. Yeah, yeah, thanks. You know, the country is based on distrust of the government, and I don't think it's changed since its founding. So where are you getting at is part of the, I think, a very actually healthy relationship between the people and its government. You know, you hear adages about, you know, that government's best, that government's least, and those that try to change freedom for security get neither. I think there's a few things that we have to do. I think that we have to, as a government at all level, need to be transparent on what we use the data for and what the community good is that people can get from it. Because I think if people can understand that, particularly at the local level, that the better the data, the better our opportunity to provide them the basic services that they need to be able to live their lives the way they choose in as safe an environment as you possibly can. I think the open standards that you talked about, I think the availability of data and understanding of what we're doing with the data, particularly at the federal level, becomes really important. I will give you a story from my background. I was the fire chief in Loudoun County, which is where Dulles Airport in West was on 9-11. We sent lots of units in to deal with the Pentagon after the attack. After 9-11, the federal government came to me and said, chief, can you give me the list of your critical infrastructures? I said, sure, okay, we could do that. So in six months afterwards, I called them back up and said, could you tell me, could you share with me what you do with that data? And they said, no, it's all been classified. And I said, but you wouldn't have known that data unless I gave it to you. And they said, we still can't share it with you. I said, okay. I mean, it's part of the concern that we have to have is that we can't over-classify data. I mean, we have to be concerned about national security. It's one of the primary goals of government. I don't care who you are. If you think about all of the thinkers, even Adam Smith, who saw no role in government other than the market, still said government had some role in protecting the security and safety of people. I was going somewhere else with this and it's gone. I'm sorry, it's been sort of a long day already. But what I think we have to be really careful of is that you can't over-classify it to the point where we lose the public threat on why we're doing is important and what service it provides back to the people. Because again, that becomes the basis of the most important debate that we should be having, whether it's in town hall or in the halls of Congress. What do we want to trade off? What do we really want to collect and what we don't so that we understand that if we're not gonna collect some type of data and something occurs, that you can't go back and point the finger. You know, and it's sort of that guccia mentality that gets us into this really bad to-do loop sometimes with government about trying to be absolutely perfect rather than be honest about where our threats and risks are and what the reality is of the fact that it can be hard to be safe. And sometimes we lose that discussion. Sorry, that wasn't as cogent as I wish it was, but you understand the thought process. No, and I brought up an interesting point that isn't always obvious, I think, to non-American audiences, which is the federal nature of the United States. And so when you were talking, I was thinking about people don't trust the federal government, but when you go down the chain, state governments, you don't see the same level of distrust. And when you get to local governments, very intense politics, but I don't know, I don't feel like the same level of distrust is there. So one thing people don't always recognize is the division of power in the US among the states, the federal government. And one of the sources of distrust is the federal government playing a bigger role. So the system was based to be inefficient. Let's be really honest. And as a disclosure, I teach state and local government at Georgetown. So if I get too professorial, there's 89,500 different forms of government in the United States, predominantly at the local level. When you start to think about special districts, school committees, counties, towns, municipalities, the preponderance of government is at the local level and they are all different, okay? So the federal government in the states were designed to be in conflict by the constitution if you really think about it. It's a balance of power. So now when people say, why can't government be efficient? Because fundamentally, the constitution was designed as a balance of power, not to create an inefficient government. And sometimes that gets lost in this whole discussion. There's this whole issue of state rights and certainly this idea of local governments. The other thing that you find fascinating looking at polling and polling data is the fact that everybody else's congressman stinks except for mine. So there are some really bipolar issues with this country as far as government, the expectation of government, what type of services they want to deliver, and certainly with taxes, that makes it a real challenge sometimes to govern. But there are enormous differences even within the country about expectations of what the role of government should be. And it's not ubiquitous. There's 89,500, I want to say 34 is the last time I taught the classes about a month and a half ago. 40,000 of them were in the district. Yeah. That's really, and it could be everything from a mosquito district in Idaho to the state of California. I mean, so it really runs the gamut, but each one of them is proud of their role and very protective and parochial of their powers. Anyone else want to jump in on that one? I just say that it's very clear with the technological trends that face us, that it's time, as the report calls for, a national conversation on a lot of these issues because a lot of the issues that are faced, particularly in health and education, are going to affect all of our lives at every level of government and in our interactions with private firms as well. And they certainly merit a very rich public discussion. And in the Netherlands, I used to have the Wassener agreement and there was a lot more comedy between COM, ITY, between government and the public. Is that still the case? Is this sort of that the problem? Do you have the same sort of distrust? We have a political scientist here. They know everything. Not everything. No, but there is a lot of mistrust and things have changed in the Netherlands, and not only in the Netherlands. If you look at the polls and the results of elections, then in most countries, approximately 30% of the people is willing to vote for extreme parties, either to the left or to the right. And this is a clear signal of mistrust because if you trust the government and if you trust the elite because it's a matter of trusting the elite, then this would not have happened. And it's perfectly explainable because globalization reach a new phase. People fear for their jobs. The economy is not doing that well. Economy is linked to security. So it will make people very uneasy and then people will look for the right answers and they cannot be provided by the existing leadership. So this is not really something new. We have had periods in history before, but it's particularly bad and it will not go away very quickly. And this has, of course, to do with the monumental change that they are taking place now globally. And this will affect for quite a long time. At least that is what people believe. The feelings of large parts of the population. And this, of course, is linked to the whole debate. We just had about big data, about surveillance, what have you. This will be a critical issue and this will be one of the major issues the political class has to deal with and they should come up with an answer. If not, you will have an erosion of the democratic system if we like it or not. Sorry if I pushed the button, yeah. 2.8 billion people currently live in cities and developing nations versus 900 million in developed nations. This is according to the ANSI workshop that was just held in DC in November. Global urban growth is expected to add an additional two billion new residents by 2050. An additional two billion. So you start to think about some of the issues that you were talking about, which I'm in complete agreement, with this enormous change in what we think about as far as community and society globally. This issue of big data and understanding communities and understanding the health of systems and being able to make sense of what it's trying to show us is going to be with us for a long and it's really gonna be critically important as we start to think about what the next 20 to, sorry, that 40 to 50 years is gonna bring as far as our children and grandchildren. Okay, we're only two questions away from a drink so I promise no more jokes. We had one question here. And please identify yourself, yeah. I'm Samira Daniels with Ramsey Decision Theoretics. I have a little bit of an academic interest in analytics which actually dates back to the 80s. You raised this issue of information versus knowledge as a generational issue. And I see this, and while you're right, I think that it's also a cultural issue in the intelligence community, across intelligence communities. And I think that this battle of information and knowledge has been waged for a long time within the agencies. And I think that that, I've followed Loc Johnson's scholarship. I don't know if you know him, but he's at University of Georgia. And I think that if unless this is handled organizationally in a more robust way because there are just so many different cultures, informational cultures, it's gonna be compounded with this whole issue of, and I'll pursue it. I don't wanna take too much time. But I was just wondering if you sort of see that at all, that these stovepipes and then the competition for funds and all of this falls into this whole issue. Oh, definitely. Thank you very much for the question because what you see is that stovepiping is more common now among the people, but also in large organization, including ministries. And that has something to do, of course, with the complexity of the issues. It is very difficult to get an understanding of what is going on. Now, this is highly dangerous because what you see is that we move the technology push because more and more important and then technology gets its own dynamics. In my view, this is wrong. It should be technology push, a pool, but you can only have technology pool if you know what you are talking about. So you need knowledge, you need concepts, you need to organize your knowledge. If you don't do that, then you only get technology push and that is dangerous. And this should also be part of this culture of security innovation about which the Secretary General talked. If you don't do it, it's technology driven and it will lead nowhere. What would you want to see? It's a setup, isn't it? Yeah. That's a basketball term. What would you want to see? What would you do? You could pick one thing. Chris, you just wrote a huge report. What would that... My first observation would be to make the starting starters in government, the same as the starting starters in Google, but the Ford and MacArthur Foundation actually just produced a very thoughtful report that looked at the whole question of how can we have more technologists than our public service? The barriers to entry obviously are profound given the tremendous draw that their services bring in the private sector. But that said, this is a big data technology is usually scalable technology. And by that I mean a very small number of people with a comparatively small amount of resources in governmental terms can do amazing things. So I'm actually quite hopeful that each department in their own way, as they see what the relevance of the technology is to their mission, how it can perhaps help them carry out their mission better, will adopt it and will adopt it strategically in a planned kind of way and in a way that hopefully does measurably enhance the delivery of public service. Maybe serious education may help. That doesn't imply always accredited education because unfortunately from the time that I was chief inspector of education in the Netherlands, I know that's still there, that there's a kind of commerce in accreditation. If you ask me what a serious good education takes a little bit longer given the drink that you are offering us. So that I think could be because if you have that and you have open data, you have the material that is hanging around that it is more than low hanging fruit that can empower society that I think would be on my agenda but that is much broader of course than security and justice. And I'm actually not sure whether the theory is correct, which is sometimes the case with theories. No, I fully agree. And do not ignore the soft side, the conceptual side. If you do it, then you have innovation for the sake of innovation, which will lead nowhere. You know, I think part of the challenge is that what we have assumed to be to be sort of state actors or state capacities is changing so dramatically. We haven't talked about nanotechnologies or 3D printing or what that could potentially mean in the whole issue of bio. We certainly haven't scratched deeply enough into the issue of cybersecurity and the difference between what a couple of guys in the basement can do versus the state. I think one of the areas that government in particular in S&T within DHS really has got to do a better job of understanding is the intricacies and interrelationship between our important subsystems. Charles Perot, if you haven't read him, talks about the issue of complexity. And basically he says that complex systems fail complexly. And we have to be very, very careful about that. It seems logical, but let's talk about Japan for a second. They had a good plan on how to deal with a nuclear accident. They had a good plan on how to deal with an earthquake. They actually had a plan on how to deal with an eight foot tsunami. What they didn't have was a plan that dealt with a 12 foot tsunami after an earthquake that hit a nuclear plant. So we have to start to think differently about the interconnectivities of our society. And it's both soft and hard. I mean, we've got to figure out different ways that we can communicate some of the threats, some of the risks and some of the expectations of the community because they have got to be resilient if we are going to be resilient as a community. So that's sort of what I would think about as part of the dream package of work that we do. Mr. Secretary General, do you have any final words you'd like to share with us? Here's the microphone. Well, I think that we only raised as a specific type, but not deep enough. But we have one and a half hour. So I think they did a good job on the panel. But we have made some remarks on how we get to innovation that can't be a technology push. It has to be a pull. It has to be about thinking. It has to be soft and the hard side. And of course, the privacy issue is especially in the Netherlands, very interesting but also tough issue for the next few years. I thank you very much and also the members. Great. Well, we do have a reception with drinks and some appetizers and hors d'oeuvres out in the room behind you. But before you head towards them, please join me in thanking the panel.