 Live from Cambridge, Massachusetts, extracting the signal from the noise, it's the Q, covering the MIT Chief Data Officer and Information Quality Symposium. Now your hosts, Dave Vellante and Paul Gillan. Welcome back to MIT and Cambridge, Massachusetts everybody, this is Dave Vellante with Paul Gillan and we're here at MIT IQ, the Information Quality Symposium. Symposium that's evolved into a Chief Data Officer event and the Cube is Silicon Angles live studio, we go out to the events, we extract the signal from the noise, Mark Krisco was here, he's the Deputy Director of Enterprise Information for the Office of the Undersecretary of Defense. Mark, welcome to the Cube. Oh, thank you. So that's the shortened version of your title. Yes, sir. What is it exactly that you do? What I do is I work for the Undersecretary of Defense Acquisition Technology and Logistics, Mr. Frank Kendall and we're the organization that supplies him the information for his decision making. He has senior leaders as well as working with his services and components on giving him the information. So you're a practitioner in information management, information quality, acquisition, so subject matter expert as well as that. I always think our trade is a bit of jack of all trades, master of none because we work from a transformation perspective, we work from a subject matter expert perspective, acquisition in our case, as well as technology and information management, so we cover a pretty large bandwidth. Well, I'm always intrigued by, you know, somebody who's got subject matter expert and you're applying it to a particular, you know, generalized discipline like information quality. So can you talk a little bit about that dynamic? As a subject matter expert in acquisition, what does that mean specifically in relation to sort of information quality? Well, let me backtrack a little bit from an experience perspective. I always found that was the key is information management in and of itself can be its own abstract term. The key is how do you bring value from your subject matter expertise perspective and link the two together to bring value to the organization. So it's building the bridge from acquisition and information management because data is the fuel that fuels the decision making and working in the process. It's critically important to understand what we are trying to achieve, what the undersecretary defense is trying to achieve with data, not just data management for data management sake. So let's start there. I mean, what are those overriding objectives that the organization is trying to achieve? Well, certainly in Mr. Kendall's case, it is he articulates his vision via better buying power and that is the mechanism which he highlights where he wants to improve both the acquisition process, the acquisition program management and the execution of the department. I mean, we have a critical role in supporting the nation's capabilities of delivering capabilities to the warfighter and protecting the nation. He always wants to do that in the cliche term, a better, faster, cheaper. And how do you do that? He has been very key on having data as a key of that foundation. So go ahead, Paul. So make that real. I mean, if we talk about better, faster, cheaper, yeah, yeah, yeah. But can you give us an example of how data quality enables that result? Well, data quality, the question is, and we have a stock three questions that we ask everyone. What are you trying to achieve? Because the use cases can be very broad. What do you want to see in your information and whether it be in acquisition oversight program management so we don't have cost overrun, so we have early warnings as associated with that, whether it be logistics to ensure that we have an appropriate logistics portfolio for the programs, or whether it be Palm, or as we would talk about it as the resourcing financial resources that we've adequately resourced programs to move forward. That's where data is the key. And from his lens as the senior, the defense acquisition executive, the senior most official associated with acquisition, it's his role that we fuel them that he can work through those issues as he sees with his organization. And how can bad data derail one of those processes and make that real? Well, bad data or undefined data, let's talk about it in both time, because you can have both bad data and data you really don't understand the meaning to. And I think it's pretty important that we put some due diligence to understand so we can better advise him as he moves forward. How can it derail? Your decisions are maybe not as good as they could be, but then again it is a real balancing effort of saying how much data do you use in those decisions and are you getting the right data and do you have a sense of what the condition of the data is. I always like to focus on we're going to move with information that we have and part of our role is to give a sense of is it good, is it bad, is it good enough and let the decision makers use that as best they can. So we're really in a support role for them. So in this example your decisions might not be as good as they could be, you might be not leveraging your buying power, you might be paying too much, might be choosing the wrong supplier, are those good examples? Well, I think they're good examples, but I think sometimes we make always the best decision we can with the information. We go to war, I think Secretary Romsfeld when he was secretary said you go to war with the army you have, not the army you wish you have. We make decisions based upon the data we have and data is an imperfect science and I think that's the one condition that we must always recognize is if we understand the degree of imperfection or the lack of information, we still are making good decisions, it's not that we're making bad decisions, we're making good decisions, it's just our information is less than perfect. But you said earlier you could make better decisions, I'm trying to understand what that means, that in your world that means you would optimize your pricing, you would- You could optimize pricing, you could monitor the program, you could be sure it's adequately resourced, you could engage with your vendors more completely and more specifically. Get more value out of the engagement, so in many situations information management has been about, the value of information management has been about protecting the organization against either data loss or exposures to data, legal issues, etc. It sounds like in your world the focus of the value of information management has always been on the asset piece, not the liability piece, is that clear? Right, I think that's fair and I think that comes from the perspective of being a subject matter expert in acquisition, you always have to focus on the value first. Those conditions of security and management of information are just as core to ours and I think that's why you have to cut across the range of it because both of them, which one do you trade on? More value without security or just security without value, you have to do both simultaneously because having information to give you a sense of size of the organization, the Under Secretary oversees about $1.6 trillion in terms of buying power and he's representing 150,000 acquisition professionals in the Department of Defense. That's 37 services and components, it's mind boggling. Yes, so in understanding that we're worldwide in coverage, we may not have big numbers kind of in commercial user perspective, whether it be Facebook or LinkedIn or those types of numbers, but in governmental institutions I know none bigger, none more complicated, but he worries about over 1,500 programs in some way, shape, or form and he looks at over about 150 of them on an annual basis, monthly and quarterly. He has a full-time job monitoring that portfolio. I could see, not that I would ever even dream about running something that large, but if I did, I would just say, just give me half a percentage point and I'm going to drop a lot of money to the taxpayers' bottom line. Well, there's a but in that question, which is how do you balance, I mean, because there's an obvious sort of thing to attack, that big nut. Let's shave half a point off of every transaction that we do. How do you balance that with the value piece of the equation? So I think an interesting thing, and I certainly can't speak for the Under Secretary of Defense, but I can send the signal of what's outside his door, he says, in God we trust all others bring data. So he has the sign outside his door of what he's based it upon. And in many ways, this administration, a bit of the previous administration, has invested in that shift to information, to fuel that. Because the danger is in organizations as large, you pick on one, but you miss the entire spectrum of what you could. And we are a big organization and he wants to power the logistics community, the acquisition community, the engineering community. They are big in institutions in and of themselves, and he wants to lead those efforts. And one of the key points have been through data and information. Do you think that the sentiment that you just expressed is somewhat administration specific, or is it here to stay? I mean, I think of smaller example, baseball, and moneyball, and saver metrics, and some teams are using, but every team uses it, of course. Some teams more than others, and we live in Boston. They're clearly violating all the edicts that they said they wouldn't violate a couple years ago, as they fail. But it seems like now, in this age that we live in, this is here to stay. Do you believe that, or do you think that it still will be sort of administrative specific? Maybe somebody will come in as president and say, I got a feel here. What are your thoughts on that? Well, we could probably tackle that question in two forms. One, I think information management is here to stay, and information and data fueling the processes are clearly here to stay. The institutions are moving that way in terms of academia. We're bringing skill sets up. Now, each administration is going to put their perspective on what they believe is important, and that's usually left for them to do that. I don't care to venture into that, but I can always see that whomever I work for as a career civil servant is that we would be of service to them and providing them information for where we are, and it has really been an emerging community, both of data science, data management, and value. I think we've seen that in commercial industry. I think that's why we have forums like this and MIT that are available. I'm sorry, I don't want to cut you off. You started your career nearly 25 years ago, buying F-18s, as I understand. Yes, sir. What have you seen happen to the quality of data and your ability to really mine that quality for value from that quality data over that period of time? So over 25 years, I think I would go back and you touch on the F-18 experience of being a major weapon system buyer. But before that, I worked in retail. And I've seen data emerge over my entire career, both commercially and federally, to be cores. And we've had many starts and runs at getting better. And I would argue that we've probably gotten better over the entire three decades of doing this, both commercially. I remember working for Lord and Taylor, I like to tell this little vignette. 30 years ago, they were still handwriting sales checks. They were having data, it was just slower, manually. You had a lot of people to work on it. One of my first roles was to implement registers. I mean, at that point, small micro computers to ring things up. I could remember 30 years ago implementing stocks by location, which were now automated in CRTs. I think we've been on this journey and I think those of us working in this profession find ourselves at an unique point of time of implementing the information age, as we talked about when we went into school or learned about. We are just at that nexus of the transformation from an industrial society to an information society. Mark, you mentioned data science before. That term is interesting. A lot of people, when we have this discussion, say, well, data scientists have been around forever. We just didn't call them data scientists. That's kind of the cool thing. How has that so-called data science role evolved within your organization? Well, I think I could sense that it has evolved in the ability to obtain even more data today because electronics are certainly a part of it. And it's opened up different issues of access control. I would argue we've probably had data science for a long time. We've gave people a bunch of sets of data, said take a look at it, study it, come back and tell us what you think. Make sense out of it. Yeah, make sense out of this because we don't know what's occurring either in the organization, the program. Pick a value chain associated with it. I think today it's all becoming closer and consolidating down in some ways where we can gain access. And I think everything from this administration and this president trying to open up data and making that more accessible. I also think the technology communities have changed. We now talk about things and open APIs and APIs of where we confuse data. So I think our speed and our abilities have come up. Now we have to bring our organizations up. How do we manage in that world? I think that is going to be the biggest challenge. How will we manage? We've got, we talk a lot about big data these days. Now we've been talking primarily with you about operational efficiency, using data to make the operation run better, looking more at the analytical side and how you use it for predictions and to see bigger trends, bigger picture. What is your agency doing in that respect? If anything right now. Well, Mr. Kendall again, as that individual annually for the past two years, he's issued the state of the acquisition and is using data as a part of that and having analysts and that is not my role to be the Uber analyst in this role, but we do provide the data to the analysts to look at that. And generally and simply speaking, he looks at it in terms of process. He looks in terms of how his organization is performing and how his programs are performing to give a sense to the American people to Congress of where are we in this journey? We should be always striving every day to get better, not just one fell swoop of one year from now will be better. It's every day we do that. Part of the themes that we hear in the commercial world, we're talking about data science before, we're talking now about getting better, improving. One of the techniques that people are trying to use to do that is this notion of the citizen data scientists. It used to be and still is, you have a big data warehouse, you've got some analysts, you're kind of scrubbing the data, looking at the data, if I want a single version of the truth, we've all heard this. And you have a few experts that you have to go through. They build a model or a cube or whatever you call it and you get an answer out a couple months later. It's kind of an inflexible approach and now we see all these new techniques to speed that up, lower the cost, et cetera, beautiful. It's still hard for an individual that's on the front lines of decision making to get access to the right data that he or she needs. Do you see that notion of what I called citizen data scientist, you know, coming to fruition at any time in our lifetime? Well, I think it is. And I think depending on how you would like, I'll answer that question in two ways. All our procurement data is already in the public domain. You can access it, you can analyze it, you can have your opinion on that data. Doesn't mean your opinion's right or wrong, it means you have access to the data. In other cases and internally because we deal with different sets of data in different conditions, pre-decisional, secret data, sensitive data, we've worked internally with the components of saying, how do we unlock that for ourselves? Maybe not in the complete public forum, or if it is, we better know how we're managing it because that's a large portfolio. We could tip many organizations. So we're learning how to manage in that environment better. About three or four weeks ago, we sponsored a study with RAN to look at how do we manage information? Do we have the proper things in place that we understand what to do, managerially, not just technically? And we could do better at that. And it's shed some lights on our organizationally that we're going to have to think about it, not just locking it up from a technical perspective where we've been strong points on. Is there any thoughts about sort of crowdsourcing some of this process? I mean, one of the benefits of standardizing your data and publishing it, making it transparent, is that others can then take it and do their own analyses and maybe help you to get better. Is there any initiatives like that underway? The answer is in a department as big as ours, absolutely. And I could think of two examples, one on specific acquisitions where organizations like DARPA had crowdsourced design patterns and saying, here's what I want to achieve. Let's crowdsource design patterns and let's move forward with the acquisition. I could also see that our data, as we make it available, we do have a crowdsourcing mentality because it's not only us obtaining access to it but providing it back to the Army, Navy and Air Force and saying, do with it what you need. We also have that mentality within the department is let's collect once used many so we can get those different perspectives. And that's what Mr. Kendall would oversee and correct that. So thoughts on this event, some of the takeaways maybe that you, actually you just got here. So that's sort of a really good question. But I was here last year. But how about you were here last year, right? And sort of takeaways in general from this event has evolved over the last nine years and sort of expectations, things that you want to take away or learn this year. I'm really keen on the value proposition associated with data. And I made this comment last year when I participated that this is one of the forums that not only focused on the technology and the management aspects of information and the technology, but the value of it. And I learned an awful lot last year and my participation this year is to further encourage those types of behavior because it's not just about the technology community, it's about the managerial community working in this space. And I find that very key here. Same question we asked James Mung actually about what commercial CDOs can learn from your experience. They're tackling now many of the problems that your agency has been tackling for many years. Well, and I think that's vice versa too. They've been tackling many issues that we're starting to tackle too. I am aware that the Secretary of Defense had met with the commercial industry out there in Silicon Valley to further dialogues associated with that. Depending on our use case, they have a very different use case than we do in terms of commercialization of data. We have a very different use case. I think we're going to have to deal with threats in our information infrastructures for quite a while. And I think we can learn from each other. I see various discussions occurring today, but things will emerge for us to better manage in the future, both commercially as well as a federal government. All right, Mark Crisco, thanks very much for coming on theCUBE. We appreciate your flexibility and hope you enjoy the rest of the event and really appreciate your insights. All right, thank you very much. All right, right there, everybody. We'll be back with our next guest. This is MIT IQ. We're here in Cambridge, Masters of theCUBE. We'll be right back.