 Hello, hi Alex Williams here of Silicon Angle here live at IBM Edge the Cube here with Dave McQueenie who is a How would I describe who you are and what you do you tell me Dave? Sure I work at IBM research and I'm responsible for all of our research on software and business analytics and our research labs around the world Great and we're joined by Dave lawyer. Hi. Great to have you. Yeah, good to be here Dave what would be the what what what kind of questions would you ask Dave about? about About IBM Watson IBM Watson. I well first of all congratulations. Thank you I mean an absolutely brilliant piece of theater to bring that to to reality pretty good computer science, too Don't you think? I absolutely absolutely so Can you talk about? first of all the fundamentals of it What what why did you choose this really difficult thing? What were the business reasons that you were looking at this? Describe a little bit about what's news for those people who don't know about it. Sure. I would really go back to The chess match we built a machine. Yeah, absolutely played chess. I'm a chess player. Okay And as you know at that time computer chess playing programs were not strong enough to beat human grandmasters Absolutely, but we felt that Computing had advanced to the point where the kinds of analytical decisions they could make and the kinds of complex reasoning That they were capable of could have a lot more applications and then people typically thought was possible So we were looking for a way to demonstrate a computer showing something Sort of like human reasoning or sort of like human strategy now, of course The machine doesn't play chess the way a human does any more than a 747 flies by flapping its wings But it accomplishes the same goal in a different way And so by showing that a machine trained by grandmasters, none of whom as humans could beat Kasparov But their training amplified by the computer's processing power could in fact beat Kasparov Fired a lot of people's imaginations about what they could do with computing And so after that we kept looking for something that was the next big grand challenge as we call it And we felt that it had to be something with human language because in chess You have very discrete rules and we have very precise mathematical formulations And while the game is is incredibly complex and incredibly subtle the rules are all pretty well prescribed Right and pretty simple to describe in fact I think that's the appeal of it is such a simple set of rules generate such a delightful complexity of game playing But human language isn't like that human language doesn't have rules In fact the rules can change over time and what's considered funny or cute or clever can can really be quite subtle And so we were looking for a way to demonstrate the power of computing in the realm of human language And it was really my predecessor Charles Lickl who at the time ran our software research Who seized upon the idea of perhaps we could make a machine that would play Jeopardy And he was influenced by the 74 game winning streak that Ken Jennings had He was the longest winning streak of any Jeopardy player and Charles came back to the lab and asked the team Do you think that the open domain question answering projects we work on with universities and with DARPA and other agencies Are they up to a challenge like Jeopardy and most of the researchers told Charles he was crazy You know maybe the old man's lost it, you know One researcher hung around a little bit afterwards and he came up to Charles and said With enough dedicated brain power and a couple years of strategic patience on your part as our boss I think we might be able to make progress Well, that was a guy named Dave Frucci now IBM fellow Dave Frucci who led our team who built Watson Who in fact in four or five years was able to take all the 20 or 30 years of research that the world had done on these topics Pull them all together and combine them in a unique way to solve this question answering task at a level Sufficient to beat human experts at Jeopardy. Not Jeopardy. Fantastic. So having done that Why are you using this for and let me preface that by saying I did some analysis I know that you're using it in the medical field. Yes, and I was doing some analysis on what could be the business Business the the the social impact of of this and it's pretty enormous The potential if you could give this to human beings like you or me to be able to ask these questions Of Siri or whatever. Yeah interface that you have The amount that you could eliminate of cost from the medical system is in the billions and billions Yes, I think that's fair to say. Yeah, I think I think you're absolutely right So so can you talk about how you get this from the you know the From from this great research project into real life? Sure. Well for starters We did the research project with the intent of having an impact on our clients business and our clients mission as the government clients like to call it And so we didn't do it purely to play Jeopardy, but that was a good first step And it was a way of saying to the world. Here's this grand challenge of human language understanding, which you know As any good grand challenge will never actually achieve the machine didn't actually understand it It just understood variations and it knew which ones were the most likely to be correct in a certain context And so that just like the machine playing chess looked like human reasoning and strategy It was it was solving the same problem a slightly different way And so we were always aiming at practical applications But playing on the Jeopardy show was actually in some ways harder than Some other domains because the language is deliberately cute and twisted and it's meant for entertainment And a lot of our computer science academic friends at the beginning told us that it was just too hard And they were the same ones that came back later and and complimented us on not only making good entertainment and good You know firing of the imagination of the community, but some good computer science too So if you look at medicine the first thing that people started responding to was are you going to try to replace doctors with a Machine and we said absolutely not in fact the way this is played out is very very interesting The human has the training and the experience and the intuition, but think of a human medical doctor How can they reread all the medical literature that's published? There's no way any human could ever keep up with the medical literature and yet each of us would expect if we went to see our Physician they'd be fully up to date And so the idea that a human and a machine could engage in a dialogue that might have five or six steps Diagnosing a complex medical condition And so the example I gave during my talk was someone comes in with a set of symptoms And we add the medical history and the family history and the medications. We're adding different dimensions of evidence And it leads the machine and the human to change their likely diagnosis as they gather more and more and more data But at every step in this reasoning process that's really guided by the human The machine can essentially reread all of the relevant medical literature at every single step in this diagnostic diagnosis Which no human could ever do and if you think about this if you were a human physician The most valuable thing the machine could do for you is tell you about a treatment or show you a publication That was relevant to your case, but you didn't know about and so your first thought is going to be I'm not sure I trust the machine to be diagnosing my patients And so Watson doesn't just come and say I think the answer is this Watson says here's the list of answers that I think Probabilistically are correct ranked ordered me. I can give you five of them or ten of them Fifty of them and for each one of these potential responses Watson gives the human expert an evidence profile basically what are all the documents in my corpus? That led me to believe this was the correct answer So the human doctor can get this list if he or she sees something that they didn't expect that's brought new information to them They can immediately satisfy their curiosity and why did the system think that by clicking on the evidence profile? And it'll take them right to the evidence we assembled and they can make their own judgment and that'll change their hypothesis And maybe they'll ask the patient another question Maybe they'll order some more tests and so you can see that this process winds its way from a little bit of information To eventually try to encompass all the relevant medical literature and then to focus down on what we hope is the best diagnosis And so it should give us the ability with the machine acting In lockstep with the human as a partner in a sense amplifying the human much as The deep blue chess machine amplified the expertise of grandmasters like Joel Benjamin Joel worked with us on the chess machine And Joel couldn't beat Caspar off right Joel's training amplified by deep blue Could I just think yeah, so I was just making a follow-up on one thing So let me push on that a little bit because what you're saying is the doctor is the center of this And you're trying to amplify the doctor. Yes, okay Can I push on that and say that's hot? That's part of the problem, but as a consumer My problem is that I don't trust my doctor that much. Yes, I don't trust the system very much to actually be giving me Really the the right advice Which doctor should I go and see? What are the options that I have? Couldn't you be thinking about this earlier in the cycle? Well and and giving advice to to us Maybe not exactly the same depth, but allowing us to Pick out where we should go and and you know eliminate a lot of waste perhaps there's a trap underneath all of that though and Even the folks who built the machine commented, you know because I asked some of the researchers I said when you see a question being asked of Watson that it's never been asked before Do you have an intuition about whether it's going to know the answer or not? And they said Dave quite honestly the system has grown in capabilities that it's reasoning about facts that we don't know It's you know They are in principle knowable by us, but we don't happen to have read that Shakespeare play or listen to that popular song And they said a long time ago. We lost some of our intuition about Instinctively knowing when the machine would get the correct answer and when it didn't and the answer they gave me was fascinating Which is one of the historical jeopardy questions was If you're standing in a room the direction you would look to see the Waynes coding and The correct jeopardy response is what is down now? And when the creators of the system watch the system ask that question for the first time They just said there's no way but it's gonna know that and it got the right answer Right, and so they actually they went beyond their intuition, but here's the trap that I'm talking about What the system as impressive as it is we tend to overestimate it and infuse it with insight that it Doesn't have that what the system is doing is it's? Reasoning about evidence that it has in its corpus. It's answering questions Where the answers must be contained somewhere within the information we've given it and by the way It doesn't have any sense of universal truth or universal falsehood It depends on us to prepare a corpus. That's well vetted It uses robust statistical techniques that can tolerate some incorrect answers and not overwhelm the correct answers And and as you were asking that question it felt a little bit like we were going to place more trust in it to have Opinions about things so I think it could help consumers with the beginning of a medical Discover I'd be happy to have yeah the probability of But you can't take the next step because you're not a physician right? I was doing the first of six steps. Yeah, it's an interesting I think it's an interesting it's a fair and interesting question because I did the Calculations on the potential benefit and the potential benefit of doing the first step Overlones the benefit yes, yes, and the other thing to consider that's interesting is it's very tempting to compare this to Traditional search engines now remember search engines are search engines are vital tools and we all use them every day And we think they're Indispensable tools, but at the end of the day, what does a search engine do it takes your query it cuts it up into words And it basically gives you back a list of all the websites for example that contain those words And search engines compete on giving you that list in the most intelligent order and using other related information The best search engines do that remarkably well, but even the best search engine Cannot give you a response document that is that contains the right content But has none of your query terms of it and we often see in a case of Watson a query And that the response that comes back as a top-rank response actually has no words in common with the query But it has meaning in common with the query And so it's actually a very powerful extension of a search engine to include some element of the meaning contained in Right a structure contained in the query because we use that and in fact It's computationally quite a bit more expensive than the inverted word index technique the search engines use because we not only Pick apart all the queries and we parse them and identify text We do it to the entire corporal which is incredibly expensive computationally to do Over the past several weeks and months I've been to several events where there's discussions about emerging technologies And it seems like you know Watson, you know when it when when you when it first Renounced it and then you did jeopardy. It seemed quite far out. You know, this is way in the future But now we're starting to see lots of interesting advancements Developing in memory technology for instance, and I think it gets to this point you're talking about Dave where you know Who do I trust but really it seems there is also a question about about Where the knowledge is and who has it and it's not you know we've been talking a lot about data scientists over the past two or three years and and The difference that I that I seem to be seeing is that with it data scientists What that means like a few people had a lot of knowledge and they shared it with a few people Yes, right and it was and it's very expensive to do that now with it seems like especially with with more data Being available in memory. You don't have to like you don't have to tear it in any ways Or maybe you do it like intelligently like I heard yes about it today Yes, but now the application has the knowledge so far greater. There's a far greater mass of People who actually understand it. Yes, that's true and it's interesting because if I'm talking about medicine There's a vast body of knowledge that's essentially available publicly. It's a journal publications and Textbooks and there would be copyright issues or perhaps licensing issues for the database owners But there's a set of medical information that is generally available through some means that you could assemble And there'd be very little benefit in having multiple systems that replicated that because it would be the same for everyone Then you get down to an individual doctor or an individual institution Whether that was a health care institution or a business institution in the health care business They have their own unique insight and so the interesting combination is The broad background knowledge that everyone has access to but then when you use it and when you use it you bring extra Additional insight and when I'm responding to your queries I bring in your unique point of view and your queries your unique point of view So you guys actually might get different answers to the same query if the query turns on unique knowledge That one of you has and the other one of you Doesn't and so that's a really interesting thing about where does the data come from and how do we decide what to trust? And and what data makes us as an enterprise or an institution? Better or smarter than another one and it becomes seems to me like there It becomes an issue about time as well And like what is the relativity of that time yes to the query and especially as you see Technologies emerge like you know Database is on top of Hadoop like H base where it's like a time base Yes database where it's like I always thought it was so fascinating about you know blogging and When I first learned about it started, you know writing and using the medium how every post was a unique URL Yes, now we're into a whole new age where the uniqueness of one point in time is defined Yes, and we can start looking at that in a contextual It's a fascinating point you make because in the presentation I gave today I made a list of what did the Watson system that played Jeopardy do and then what are the requirements that our Watson? Business division has to do for a production Watson system and there's a lot of a lot of work that they have to do to take the research Prototype that was pretty much coddled by its 25 PhD mommies And turn it into something that you could actually deliver to a customer site And one of the things that the game playing system didn't have to do was to understand that its corpus was time dependent Because we asked the Jeopardy producers they write all the game boards for the whole season ahead of time And they locked them in a vault and we said guys we don't want to know what the boards are we're going to play We don't want to see them. We don't have an access to them We want you to lock them in the vault and we only want to see them for the first time when we play on TV, but we knew that they were done and they were locked in the vault And so at that point you everything that had happened that could possibly in a question We knew yeah, so we didn't have to have our corpus be time dependent, but in any practical application It's very data changes things that were once wrong or now right and vice versa and new news comes along And so one of the significant enhancements our product colleagues have done to Watson is to start teaching it that The the truth that's contained in its corpus really just changes truth is probably the wrong word can change over time So so what's the next big challenge? You know we get that a lot And before you know, I don't know we've had some suggestions that were clever and humorous like Building a system to play iron chef And it turns out there's a whole science of what flavors go together and how to use the mystery ingredient and people get all excited about the Robotics to flip the you know the omelets and things but that was a humorous, but sort of an interesting query if you follow down the axis of emulating human brain Functions we started out with kind of strategy and deductive reasoning and pure logic in the chess game Then we moved on to this more complex expression of multifaceted ideas and subtle meanings You know if you stayed along that axis it might be along the dimension of Sort of emotion or feeling and trying to understand not just whether there was evidence or not, but How was that evidence presented? Can you give me any kind of additional context about how the person felt about it? Did they believe what they were saying did they not so if you render this thing sort of as a step by step You'd go from pure logic to pure logic including language to pure logic including language And maybe something about emotion or feeling and I don't know how to do that And I don't know what the grand challenge would be but we'd love to think of it so it means like Alan Turing said the definition of perfect Artificial intelligence is that you couldn't tell so maybe you could make a therapist Who and ask whether you he was real or not? They couldn't tell the difference Yes, yes, I mean we've laughed a lot of times because Dave Ferrucci Who's our IBM fellow who led the project has been on panels at various meetings You know with With other people who have asked him all these questions and was there gonna be you know some kind of a magical event in human History when the machine achieves consciousness You know so these are interesting things to think about I think we're actually a long way from truly Understanding what the language said but within a context and understanding the variations and picking the variations that are likely to be Right and then presenting that to the human with the probability You know not asserting the answer but saying I'm 60% sure that I understood this and here's why That's a fairly long way from true understanding, but it's a fairly long way from what we have now It's a big swoosh to anyone doing a process where a lot of really valuable information is encoded in human language Either in the form of speech or text, but interestingly getting to that point about perfection Right and not knowing if this is the a therapist is real or is not real I find it increasingly so that it's hard to tell sometimes when you receive Messages through Twitter for instance if it's what the identity is of the actual message itself And who is actually Delivering that message or what is actually delivering is it a machine is it an individual individual? It seems like that's a place where we are we're like we're kind of manipulating the meaning of identity to some extent as a kind of a first Past or is what yeah? Well, that's getting that's getting a little bit into the weird zone for me We're actually much more practical about this and we're just trying to say you know if we have a body of Information that that we think contains answers to some questions that matter for a business process or a mission process Can we take a query? Can we use the evidence contained that corpus to assist a human decision maker in of all of that information? They could they could in principle understand at all, but they don't have time to do can we make all of that information? Accessible to them Essentially in real time could we cross that bridge and that's a huge bridge to cross I think Because basically what it would say is that each human expert would be you know as good as they could possibly be as if they had Read the entire literature every time they asked they were asked a question, which they certainly can't do today so what other we've talked about The medical area yes, and I can see that it might help in some of the financial areas as well but are there any other areas that you know that We as consumers would sure I'll give you an example that That came from one of my customers in the federal government It was one of the social service agencies and they were processing a very long queue of aid applications from citizens and some of these aid applications had very time-sensitive properties someone was going to lose their home or someone might might die from a critical illness and they had a very long pipeline in the queue of Essentially these text you know textually handwritten some cases typed in You know paragraphs or pages of text that a human expert had to read and adjudicate What benefits would be offered and the question they asked us was are these techniques for Analyzing text good enough to say of the three-month queue of applications Can you find the ones that have a time critical? If someone loses their home due to a fourth closure, you can't reverse that process very How can you optimize the aid that you're giving to have right? Could you prioritize that queue where you pulled out the ones that had an irreversible time trigger in them right? You'd still give them to a human expert to finally adjudicate But they felt that the you know that's an example of the kinds of things that come up And and the interesting thing about that one is and if you look at the Jeopardy game You need to be able to train the system on what are correct responses and what are incorrect responses? And that's another test for a good application for Watson is is there historical data You could use to train the system because it's doing statistical weighting using machine learning techniques So it needs to be trained to find what right answers look like in different dimensions of evidence And the fact that the agency in question had been processing grant applications For some for a very long time and had the record of all the adjudications that they had made That was the perfect training in which ones do they accept which ones do they reject which ones were time critical? And so we thought that was an interesting one because it had not only the right kind of benefit The technology was capable and the training data should be available right what about things like? You know saying which transplant should we should get a transplant or who should get the medical? optimizing the value That one's a little tougher because you get into these very complex value decisions And I think in a lot of those cases the time criticality is well known and probably Representable by more quantitative means so that that one feels like it would be a little bit outside the comfort zone For this technology, but the other place we've been looking at just within our own business is Call centers and contact centers. You know we have many for ourselves. We run many for our clients and you know Processing FAQs and making each of the folks that answers the help desk call as smart as the smartest one again with access It's a lot like the medical doctor example, right? They're doing a bit of a differential diagnosis right and we think it should have a great benefit there You've probably seen the the show from British TV with the IT guys at the tape recorder We saw some interesting presentations this morning about innovations and storage and like where IBM is going this And there were some interesting Aspects that remind me a little bit of what you're talking about here for instance with archiving in such and in particular and also in terms of just the capability to you know Store vast amounts of data in places and being able to be able to still understand it Over time right how are those factors playing into your own research? Apparently and the evolution of Watson. Yeah, so so Watson depends on having a Body of information a corpus as we often call it that's been assembled and curated by some means And for the research prototype that played jeopardy that was all assembled and curated manually So again another one of the challenges that our product division colleagues have is as much as possible We want to make the evolution of that time-dependent corpus Happen without as much human intervention Programmatically right so that the system could actually go and get new data sources and go through some kind of a semi-automated or semi-automated vetting process to decide to bring that into the corpus and these are interesting challenges because Some of these are truly research questions because it hasn't been done before and you have to answer the question of is it possible before you Answer the question of how hard is it to actually execute that? But the more data that's available the more you would want in the corpus of a system like Watson Which then begs the question of what's not just what's the process Watson uses to reason over its corpus What's the process used to bring things into that corpus over time? How would you know which sources to trust? How would you know how often to update them things of that nature are remaining actual research questions for us to work on? So if I look at the Watson research team now that our product division colleagues are making a deliverable version of the technology Like you saw on Jeopardy and they're extending it for their own uses The research team is looking at these other next next round of true research questions like that So and I presume what one of those fears you might have is that you would find organizations that might deliberately want to for example a Company with a product that wanted to push that product into the medical field could could use the system as itself to to actually Change Change the perception use social media use other things Well, we see that in publicly accessible websites today people trying to influence recommendations in most of the use cases We've looked at for Watson It's been more under the control of someone with a mission So a health insurance company or a hospital or a financial services company And so even if they automate some of the ingest process that process will still be under their control We're not you could put a system up totally open and let Things fall into the corpus. However, they would fall. We've never considered a use case like that partly for the reason that you suggest is that there has to be some Editorial or policy control over what goes into the corpus because the responses you're going to get from the system are no better Than that and I think most of the enterprises we've talked to would want to retain control over what kinds of things went in there and under what Policies they went in there. Yeah, there's been some interesting controversies around What access that actually is? Legitimate to systems such as that. So, I mean there's been some real concern. I mean IBM for instance has There's a story about restricting Siri because there have been Concerns at that Siri, you know in answering questions and collecting this data could actually learn a lot about You know information that a company holds near to it, you know near to its you know vest and You know and so that seems to be an issue that comes into play when you're developing these next generation technologies like that This this question is not a new question as you probably well know I can remember years ago We wanted to show off the performance of some of our database systems And so we built a system that ingested all the US patents and then let us work the index in reverse So basically saying instead of saying here's a patent and it references these other patents We wanted to invert that indexing and say here's a patent What are all the patents that reference it and so at the time now? This is ten years ago that was whizzy for large-scale database technology because the patent database is so big And we got a lot of our clients really interesting We just put it up for free out of our research site in Alamedin for customers to see what our database is capable of and A lot of our clients came to us and said you know this is a fabulous tool We'd like to buy a copy of it and we said oh well you can use it on the web for free And they said no we're not Comfortable that the queries our researchers are making about patents are things we want either out in public or in the hands Of another company even one that we trust like you guys IBM We'd like to have a copy of this system to use inside of our fence because we want to control and so questions about you know What queries do we put out as consumers on the search engines or things like that are interesting and valid questions about you know privacy and Care of corporate information It seems to be logical that there would be kind of a whole range of personal assistance out there that have various functions Yes, and that as long as the company offering the service is clear about Where the data will go and what they do with it and what you can expect from stewardship Then you can make a personal decision just as a company could make a personal a corporate decision about that So to me the main thing is that people are straightforward about saying when I type in a query Where does it go? Are you going to save it and who's going to have access and once I know the answer to those questions? I'll know how to use that tool in a way that I'm comfortable with So it's interesting question So broadening the question what other areas you've talked about the Watson area Can you talk about some of the other areas that you're responsible for? Sure, when you were talking about large data sources You actually took took me to a slightly different position, which is about cyber security, right now This is another area where It's clear to us today and from our cybersecurity research team at IBM That with the advanced persistent threats that are present now that a lot of cyber security has become a big data problem Because what's happening is adversaries are getting pieces of code inside of an enterprise by various means not all technical some of them Social engineering and other things And once those pieces of code are inside the enterprise they can remain dormant for 1690 120 days and the reason that we think they do that is that they are Making a guess is how long you're keeping your log files because if you want to go back and do forensics after an attack happens You want to go back to the point of insertion and say where did that come from? How did it get here? What did it do? And if they it's a little scary that they have that much strategic patience. Well reminds me of Jeff Jonas's presentation Yes. Yes. In fact Jonas's concepts apply exactly to cyber security and so we've been doing a lot of work on You know experimenting on our own infrastructure to keep the log files for much longer and being able to go back and look at Historical events and we think that's going to be an important change in the way cyber security is done Well, David, this has been a fascinating discussion. Thank you very much for joining us today here Look forward to keeping in touch and hearing more about what happens at Watson And maybe perhaps we can talk some more about cyber security at a creature point Dave. Thank you very much very good Thank you joining us. Thank you very much indeed. We're here live at IBM Edge. I'm Alex Williams. We'll be right back