 Hello and welcome. My name is Shannon Kemp and I'm the Executive Editor of DataVercity. We'd like to thank you for joining the current installment of the Monthly DataVercity Smart Data Webinar Series with host Adrienne Bowles. Today, Adrienne will discuss advances in natural language processing. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we will be collecting them via the Q&A in the bottom right-hand corner of your screen. Or, if you like to tweet, we encourage you to share highlights or questions via Twitter using hashtag smartdata. If you'd like to chat with us and with each other, we certainly encourage you to do so. Just click the chat icon in the top right for that feature. As always, we will send a follow-up email within two business days containing links to the slides, the recording of the session, and additional information requested throughout the webinar. Now let me introduce our series speaker for today, Adrienne Bowles. Adrienne is an industry analyst and recovering academic providing research and advisory services for buyers, sellers, and investors in emerging technology markets. His coverage areas include cognitive computing, big data analytics, the Internet of Things, and cloud computing. Adrienne co-authored cognitive computing and big data analytics, Wiley 2015, and is currently writing a book on the business and societal impact of these emerging technologies. Adrienne earned his BA in psychology and MS in computer science from SUNY Binghamton, and his Ph.D. in computer science from Northwestern University. And with that, I will give the floor to Adrienne to get today's webinar started. Hello and welcome. Thank you, Shannon. It's always fun to be here and even more fun when I know that the microphone is working. Thanks to everybody who's joining. As Shannon said, we'd love to get questions. Try to batch them up in general until the end, but if there's something pressing, maybe Shannon will break in and I'll try to address it. The topic today advances in natural language processing is actually a very timely one. There's been a lot in the news recently and people are starting to get used to some aspects of natural language processing. But I think there's still a fair amount of confusion. And so my goal today is to, oops, let's see if, okay. So while we were preparing, I couldn't stop moving the slides. And now I can't get it to move. Let's try this one. Shannon. I don't know what to say. That's all from your computer. Yep. Hold on a sec. Okay. There we go. All right. Hopefully that will be the last of the glitches. Anyway, so what I want to do is set some context. We're not going to get into the actual technology really deeply because it gets pretty far into statistical modeling. I'll try and show why that's important, but if somebody is interested in more of the details under the covers by all means follow up with me. What I want to do is give you a framework for natural language processing, which includes natural language understanding or NLU and natural language generation. So for each of those, we want to go in and take a look at the technology market and some applications. And let me do one quick thing. Shannon, I think you're going to lose a screen for one sec, but hold on because this is a, give me one sec. What do you see right now? I'm sorry to have to ask you to take a look at this because it is. I see you're going through your menu bar. There we go. Yep. Okay. Let's try this one more time. All right. Lovely. There we go. That is not it. Okay. Well, we're going to have to just go with this for now. Okay. So let's put those in context and look at natural language and then we'll look at why processing it with a computer is so hard. I had an interesting quote that I saw earlier this week from General Hayden, who's the retired director of the CIA and the NSA. Talking to his staff and saying that you're not just responsible for what you say, you're responsible for what people hear. And I think that's an important thing to start to think about as we look at natural language processing because even for people, there are a lot of issues in trying to understand language. If you're familiar with the old telephone game, the old party gaming telephone, where one person tells a story to someone else and he tells someone else and he keeps going around, stores tend to degrade. The messages tend to degrade. So people are not bad. We're pretty good actually at communicating in natural languages, but we're not perfect. We have to use cues besides the actual words that we hear. We have to understand things in context. We have our own frame of reference, so two people hearing the same message. It's quite likely you'll get different understandings. Natural languages are inherently ambiguous. There's often multiple ways to interpret something that are all legal according to the language, but one will make more sense than another. We also have to compete with different signals and noise and try to figure out what is actually important to process. And it's the same when we're dealing with natural language processing with a computer. There are times when words will drop out. You fill those in. It's kind of like predictive analytics. If you were with us for that webinar where you can generally figure out what was said during this space where no line goes dead if it's brief. But there's meaning there that people often talk about this, that there's meaning in the silence. And that's not something that you're going to get with a computer. So the state today is very positive, and I'll give you the preface here that this is going to end on a very high note. I think that we've got a lot of advances, and I want people to get started. But we're still missing some fundamentals. In the news, this was June, Facebook's head of AI wants to teach chatbots common sense. Well, good luck at this point. It's hard enough to teach people common sense. If you're a parent, you'll know that that's more difficult than a lot of other things. But the issue here is that these things tend to be very simple-minded. And they tend to, when we're dealing with chatbots in particular, they tend to really be built with a predetermined set of responses. And so it's sort of a lookup. You get something in. You parse it. You break it up into the words. You break it up into the parts of speech. And you try and understand what was being said and then select perhaps from a set of options. And I'm going to give you an example here that you probably all familiar with if you use the Apple iPhone. Siri. So I've created these screens the other night. And I've deleted here what I actually said that got the response from Siri. Well, I never or but, but. And what had happened was I was actually driving recently and I stopped at a stoplight to, well, because the light turned red. But I spoke into my iPhone and I asked for directions to a store. And as I was doing that, the light turned green. An impatient teenager hopped the horn behind me and I said something that I probably shouldn't have said. And Siri responded with, hey, I don't think I deserve that, which made me start to wonder what else was programmed in there. And as I experimented with it, there are a lot of things that you can say that it's just, there's some common that the designers of Siri have put in these common responses. It's not really doing any sort of deep search. But let's take a look now at the overall opinion of the market. This was just, what was this, just a couple of days ago. Yeah, August 9th in the MIT Technology Review, an article saying that machines that truly understand language would be useful, but we don't know how to build them. Well, that is true if we go with a strict definition here of truly understand. Because there are a lot of things about language that we don't understand, so we can't put it in the form of data structures and algorithms. But I wouldn't want anybody to read something like that and think, yeah, I'm going to wait for that because it's not useful. The key message you're going to take away today is there's some really good technology out there. It's a big commercial office. I'm going to show you some of the examples. And even if we don't need to build, even if we don't know how to build something that's perfect, that should not be an inhibitor in any way. There's also sort of a very common theme in the press these days where people are trying to disparage the progress. Some of it comes from fear of the unknown. Some of it comes from misunderstandings. But one that I saw this week that also drove me crazy was this is in venture beat, why AI will never truly be useful. And as I started to read that, my blood pressure was going up. But then I happened to take a look at the actual URL for the article. And apparently the author wrote it as why chatbots will never be truly useful, not why AI will never be truly useful. So even here we've had something that was written one way in a natural language and someone, presumably an editor, had changed chatbots to AI and it completely changed the meaning. So this wasn't even a computer doing that. I can't imagine that anybody could write a system that would change chatbots. That could be substituted for AI. It's to change the entire meaning. And that's one of the things that we need to look at. Where is meaning in all of this? So why is this so difficult? If you look at natural language, and I should just say I'm talking natural language. I'm obviously talking language that people speak to each other, so English, French, German, et cetera, as opposed to programming languages. And if you study language, if you see linguistics, if you see the way we understand, getting into language pretty quickly gets into philosophy. But if we stick it with a structure and look at something like grammar, you know, children are taught the grammar for a language, they're taught the parts of speech, they're taught how they go together. But even there, there isn't universal agreement. We have different types of grammars to specify a language. So a generative grammar, which is the one, which is the class of grammars that we use for programming languages, can do a pretty good job of specifying a language like English. So it'll tell you what the structures are, what, you know, parts of speech go together. And I'll give you an example of that in a second. But there are other ways to approach it, which would be, say, a constraint-based grammar. Right now, you're looking at the rules that specify the constraints. So anything that can constrain you with a rule is valid. So in a generative grammar, you can generate everything, and you can work backwards to see if something fits that. So any string, any sentence to be correct, to be strictly speaking correct, could go back to that generative grammar. Whereas with a constraint-based grammar, you're only saying things that you're only specifying in the grammar, things that are invalid. And the last one I think is kind of interesting, particularly if you listen to politicians these days as a stochastic grammar, which says that you can look at speech. You can look at the narrative view of speech. You can look at the text that represents the speech. And the correctness of an utterance or a string or a sentence or a paragraph, that correctness is based on a probability. It's kind of like a fuzzy set theory. Is something correct or not? You may say, well, it's 80% correct. And a lot of times people, particularly speaking in a hurry, leave things out. They mangled grammar a little bit. And so with a stochastic grammar, you could actually capture that. But some of the other things that make it difficult, the languages are ambiguous. There are cultural differences. Sarcasm is one of my favorites. It doesn't translate very well between people, even in the same language sometimes based on your experience. It doesn't translate well between languages. And it's very difficult to deal with in machine translation. So some other things. This is one of my favorites. So here is it a ruler? Is it a guideline? In English we say that two negatives always make a positive. So if you say, this is not an uncommon occurrence. Not is a negative. An uncommon is a negative. So that translates where it is equivalent to that is a common occurrence. And in English it's generally rule that two positives never make a negative. But the first time I heard this, it was given in an English class. Somebody in the class said, yeah, right. Well, yeah, it's positive. Right is positive. And now we've got two positives together that form sarcasm. And that's really hard. So as we're dealing with this, I said that inflection really matters. That's true when you're dealing with voice to text to understanding. The other thing is that usually when someone says something is literal, it isn't. And so all the things that we go through, all the machinations that we go through to understand language as people, there has to be an analogous process, if you will, to do it automatically with NLP. And NLP is generally just referring to the machine processing of natural language. One quick example here. We talked about grammar. You may have heard the old infinite monkey theorem that says that if you had enough monkeys and enough typewriters, they just kept pounding away. Eventually, there's a good chance that one of them would produce one of the words of Shakespeare. And from a probability standpoint, that's true. You could speed it up if they weren't typing letters, if they were typing words. And so if we had a generative grammar, like one here where our postal address is defined to be a name followed by a street followed by a zip, you could create an application that would generate every possible string, every possible sentence. And if you had a generative grammar for English, you could generate every possible combination. But the combinatorics are just enormous because for each of these, there are multiple parts. So you're very quickly getting into the thousands, hundreds of thousands, the millions of possibilities. So it's not something that you're going to use just to generate, although we'll see in the second part of the talk how we do generate speech. But if you have a syntax like this expressed in a grammar, you can map the words that you see to the grammar to see if it's structurally correct. And that's kind of the first part in natural language understanding. For those of you that have worked with compilers, compilers are generally written with context-free grammars or regular grammars that use lookahead. So you may be looking at a statement in a programming language. And if you can't resolve what it means, if you look ahead, then with that context of looking ahead, you can figure out what the proper meaning is. Because with a programming language, the language itself should be unambiguous. So once you have enough context, you can always resolve it with a single meaning. Okay, so just to give sort of the structure for the rest of the talk here. When I say NLP, I'm talking about the combination of natural language understanding and natural language generation. A lot of times when people say NLP, they're really just referring to natural language understanding. So it's important to make sure that we're using the same terms here. If you look at applications and people say, oh, yes, I have NLP, I would say probably 70 or 80% of the time what they're talking about is just natural language understanding. They don't generate it. So they can take natural language text in. They can take it from voice, if you will. But generally going from voice, the first step is to change it to text. So we're really going to focus on natural language expressed in text understanding. Once it goes into that big question mark, then we can have natural language generation. And the key concept here is what is understanding. I'm going to get to that actually at the very end and say we don't have a general definition that everybody's going to agree on for what's understanding. But there's enough working definitions that we can make some progress and we can build some real systems. So if we start with the overall idea of understanding, you've got some voice which will be translated to text or you have text directly. It may be coming from a person or it may be coming from a machine. It may be text from a medical journal. But to understand it, to be able to use it, and I'm very big on the idea that this is all about pragmatics and use, you start by looking at the syntax, which is expressed in one of the grammars that I talked about. So that gives you the structure. You look at the structure and see if it's correct. And you look at the semantics, which help you understand the meanings. So you may have a grammar that says, okay, I'm going to have a noun and a verb and a clause. You can say, the boy hit the ball. Great. That's easy. You could substitute any noun and it would still be syntactically correct. You could say the boy hit the moon. That's a noun just like the ball is, but it's unlikely. And so you have to look at the syntax first and then the semantics and make sure that it makes sense or resolve the ambiguities or see if it's a metaphor or something. You know, the cow jumped over the moon. Okay. In English-speaking countries, that's generally understood to be part of a story for children. There's a metaphor involved there. But just reading that, you're not going to get that. So once you've gone through these steps and done the syntax and the semantics, and I'll talk a little bit about some of the ways we can do that with statistics, what you get is what I'm calling modeled understanding. And in a couple of the other webinars that we've done, I've talked about models and it's really important here to understand that when we're talking about a model in a modern AI system or a cognitive computing system, in general what we're talking about is some data that is represented in a way that fits with our assumptions. And we always have to have some assumptions here. So if you're reading a medical journal, you know that it's a medical journal. When you see a term that could be used in more than one way, let's say tissue. If we're talking about self-tissue in biology, that has one meaning. If you're talking in medical journal that somebody has already knows and they need to use tissue, it's the same word, it has a different meaning. You have to be representing those in different ways. And that's one of the things that we need to be aware of. So we'll get into that a little deeper now. I just want to throw out one term that you may see. It's not as popular as it was years ago, but computational linguistics is really what we're talking about when we're talking about natural language processing using rules or statistical models to represent our understanding of concepts. And it's that whole idea of understanding the concepts. It gets very fuzzy and it's still a huge debate when somebody says, oh yeah, I built a system that understands this. Well, does it understand it or does it represent it well enough that you can do something useful even if the system doesn't understand anything? And we'll sort of make the distinction between what a lot of chatbots are, whether it's static performance, you preprogram the logic, you preprogram the responses. The same input is always going to give you the same output or it's going to give you something from a set of chosen outputs or maybe probabilistically, but over time the output is a function of the input and the model that was in place when the system was built. I want to contrast that with a learning system that will improve its performance either in understanding or generating based on feedback. And that in general comes from the use of machine learning, which gets into neural networks and deep learning. And one of the things that I pointed out last month on the webinar if anybody was with me is that this is really where I get into the idea of modern AI. Because if you look at the way natural language process was done in 20, 30 years ago, there was very little that was done with this deep learning, machine learning. So the techniques have changed and as a result the performance has really improved. So to net it out to give you kind of a simple view here, we have some system and that's what's in the middle, and in the middle of the system or somewhere in the system we have to have this model of the world and the representations. And when I say representations, I'm talking about the way we represent concepts and relationships and that's hugely important because some concepts are related to each other based on the way we use them in language, the way they represent things. And it's kind of like having a map. A map represents the terrain. There's an old saying that when the map and the terrain don't agree, believe the terrain. Here, the key is to build the best model and the best representation you can to begin with. But if you want the system to evolve and continue to improve its performance, you have to have a way that there's feedback and the model itself gets updated. So that new instances, when the system has been trained and when we discovered that it's made some assumptions because again, we're dealing with probabilities here, we're not dealing with a perfect one-to-one correspondence between a sentence in a correctly structured sentence and one meaning. Many times we are, things are very simple. If you say that is a red card, okay, unless you happen to need to know what shade of red card is, that would distinguish it from all other color cars. But if you start to get into things that are a little more subtle, you're going to make mistakes in your assumptions and those need to be corrected. So on the understanding side, that's really analysis. We're breaking down what we get as input. We're trying to understand the syntax. We're understanding the semantics. And then we can do some statistical modeling on that before it goes into the updated model. All right, and the statistical modeling uses things like Markov chains to predict what's in there, predict the meaning as you're doing the translation, if you will. Going to the output side, using this same view of the world or at least the same slide, if you have a model in the representation, which after all is at the end of the day, the model in the representation is expressed in data, you can generate natural language generation or synthesize natural language from the data. And the data doesn't have to be any natural language. What's in that model, what's in that representation, frankly, at the end of the day is ones and zeros and how you represent a concept doesn't need to be a one-to-one representation. So if your concept is a large machine, that doesn't mean that you're going to have something in there that looks like it. And this is probably a good time to just mention that one of the issues that people who criticize the way we do natural language today, one of the issues that comes up is, well, we're not really representing it the way it's done in the human brain. Shannon mentioned that I have my undergraduate degrees in psychology. I spent a lot of time looking at different models of psychology and really focused on behaviorism. This is very similar. I actually don't care if we do our natural language processing using simulated neurons and synapses and connections and reinforced links between concepts. I care about how useful is it on the output. And that's what we're going to really get to in just a second here. So once we have something like this, I'm going to put it in the context of the commercial model. And so here, if you have any sort of a cognitive system or a modern AI system that's using natural language process, you have this NLP layer on it. What we're looking at today is really text NLP and narrative generation. So anything on the left-hand side for input is coming from some source that needs to be put into a machine-readable form. And so if it's coming in as the human voice, something like Siri or one of the others, then that gets translated into or converted into some form of text or machine representation. And so we're really just looking at it once that's happened. On output, if you're creating machine output from an actual language system for machines, that can be anything in terms of the output. But if you're creating it for people to read and you're creating a narrative, then that's what we're going to be looking at today. So it's really from the bottom left to the top right are the areas. And each of these, the company names in black are representative of the market. And I will give some examples from each category in the next couple of minutes. So here's one. Google has, this is the end of last year, Google Smart Reply, an email responder. So the idea of this system, and you may have tried it, certainly a lot of people have been experimenting with it, is to create a set of rules that will read your email in the inbox and either automatically respond if you can figure out what the right rule would be, or give you a set of responses and you just choose one. And the way Google has done this here is with machine learning. It's got what we call recurrent neural networks on both sides. So one translates it into this thought vector, which is a mathematical representation of what are the concepts, what are the issues. And then on the outside, the decoder, it says, what are the likely responses? And as you use it, the idea is that it will get better at recommending appropriate responses. If you use one of the apps on the iPhone when you get a phone call, it will give you a choice if you want to send a text message with two or three recommended messages. But something like this is in order of magnitude more complex because the possible responses range very widely. This is a pretty good example of one of the systems that's out there today that's doing really machine, sorry, natural language understanding. The generation is primitive because the generation is you select from a set, and it's generally a small set. It's not really generating the way we think of as a data-driven generation system. So that brings me to this whole idea of proximity modeling and a natural language understanding. I said that what actually gets stored is not like the way you store it in your brain, but there is an analogy to that. So if you have a mapping of words in memory and your system needs to figure out if these things are related, if the purpose of this mapping is to do an autocorrect when you're typing, then you want things that are structurally similar. So you can look at some of these and say, oh yeah, okay, well, they and they are related because they're very similar word structures, and N and M are next to each other on the keyboard. So depending on what you're writing, if it says nay, it could be nay, it could be nay. These things have to be together so that those are the things that would be recommended, but there's no logical meaning. So if you look at the words in green, think about that each of these words represents a thought, an idea, a concept. And again, one of the issues is that the same word can have multiple different meanings. It can even be different parts of speech, but whereas similar structure between two entities or objects generally means that there's a reason to put them together if you're doing vision processing or image processing, things that look alike are more likely to be similar. But just because words look alike, boy and man are closer in concept than boy and bae, even though alphabetically and in any structural ways boy and bae would be closer. So the issue is creating an algorithm or set of algorithms that will identify the properties of the words or the word phrases or the structures that create a vector that can put them in this n-dimensional space so that when you see one, you're trying to do the translation, you're trying to do the understanding, you can interpret it in the most meaningful way. And so that's sort of a very simplified way of looking at statistical modeling to represent it. Now, within natural language understanding, right now there's a lot of options out there. If you're interested in getting started and adding NLU to an application, I'm just going to mention a couple during the talk. This one is Google where they have a number of APIs for natural language to go in and take what you have and get some entity recognition to figure out what it's talking about, sense and analysis, syntax analysis, and this is typical of what's out there today. I'll give another couple of examples at the end, but up to 5,000 units a month are free. It's really a good time to be experimenting with these things. Here's an example from a company called Digital Reasoning. Digital Reasoning doesn't sell like pieces like Google does with a SAS model for doing it yourself. Digital Reasoning has a system that is widely used in government applications and other domains, but I wanted to use a couple of their slides because they've got a really nice layout here to show the process for this understanding. So you've got ingestion, you've got re-resolve and reason, and this is the issue where they refer to natural language processing. What they're really referring to is natural language understanding, and the understanding part is this entity and fact extraction. And for the types of applications that they're building here, they do a lot of things for security and for fraud detection and things like that. They're looking for categorization and entities that fit with those domains and what it says, reason for the uncovering and comparing, etc. These things are stored as graphs, and we did a webinar, I guess it was in May on graph databases. But if you think about it, a graph is being a hierarchy where you can find relationships based on their distance from each other within the graph. This is a pretty cool way of looking for relationships. So if you think about a graph for people, you can use that to find out if A knows B and B knows C. That's two degrees of separation and used to look at patterns, and that's a very interesting thing from a concept standpoint. But here you're doing it from the actual language, so you're starting to look and say, well, I'm deducing that these people know each other based on what I'm reading in it. Just a couple more from that. So we talked about the idea of syntax, and here the digital reason for the example is they've gone from the actual language and now it's annotated with the tokens. And you can see it's breaking it apart, looking at the parts of speech, how it's used, whether it's first person singular pronoun. Everything gets categorized, and then you can start to figure out what are the relationships. So you first, as I said, doing this syntactic analysis, you're getting into concepts. So here is the same thing at a higher level of abstraction. Now we're looking for the concepts and seeing where these things fit together. So really what we've just seen is sort of a mathematical approach to representing the structure and then the semantics. The semantics being how it's interpreted that enables us to do the translation. But more importantly here, we're not just doing translation. We want to capture this information in a way that it can be used for some sort of problem solving or some sort of analysis. So the output of this, as I showed in an earlier slide, is that model of the understanding that you get from reading it. And again, I have to focus on this word understanding because it is such a point of contention with people. It doesn't mean that the system is understanding in the same way. It doesn't mean that the system is representing in the same way that a human does. But it means that it's putting these concepts together and storing them digitally in a form that can be used productively. And I think that's an important distinction. So this is the last one in this series. Looking at this, now we're taking something at a higher level. And again, think of this as a graph. You can start to look at the distance. You can look at the relationships. You could create weights for the edges and do a lot of things because graphs have formal mathematical properties. But you can't get to that until you've done that syntactic and semantic analysis of the text, the input string itself. And so this represents really what we're talking about as the end product of natural language understanding. You've got this. And now you can do something with it. You could produce insights, which would be based on the relationships that you find. You can update it based on external sources. You may create some natural language generation. You may create a report out of it. But this, as you see it, is really sort of a quick graphical view of the end product of natural language understanding. It's a representation that has some meaning in the context of the domain that can be used to create another review. So that was all on the input side, the natural language understanding. Now I want to quickly go through natural language generation. This is another hot area. So now we've taken the data, and this data could be the data that we just saw in the previous slide. So it's all the data about something. Or it could be data that you have from an application. It could be something in a spreadsheet. It could be something in your ERP system. You have some data that is now going to have a filter or a structure applied to it. And again, this is a model. It has your assumptions about what that data means and how it's related to each other. And from that, you're going to generate a narrative. And this can be used from the simplest chat box to a structured response. And I left it on here because I typed from a simple chat box three times before I gave up. And I realized that the system that I was using in trying to help me and correct me didn't recognize chat bot until I put it in quotes. So this is a good example, may not be perfect, of the way natural language processing today often makes simple but wrong assumptions. And if you're not paying attention, that gets propagated and things just get worse. But here's the general structure. You have some data. You have a set of rules or a statistical model for taking that data and creating a generated narrative. And here's a couple of news examples. So this was from August 5th. The Washington Post experiments with automated storytelling with the Rio Olympics. And here what they've done, you find that sports is a hot area for natural language generation because there's a lot of statistics. There's a lot of data. And it's generally pretty clear. So the Washington Post is using a system that will take that data and write a narrative. So a lot of their reports, oddly, the events has actually been created today by a system called Heliograph. And what's interesting about this, we'll talk about a few of the commercial companies, is that the Washington Post did it themselves. They built this Heliograph system to augment, if you will, the writing staff. And in general, all these types of systems tend to do is replace existing reporters for the most part. I mean, that certainly can happen. But it helps them give coverage of things that otherwise they wouldn't have time to report on. So minor league baseball games, some types of reports. So this is one. Here's another one in the news. AP sports is using robot reporters to cover minor league baseball. In this case it was somebody called automated insights. I'll show you an example from them. But it's the same thing. There's a lot of data that's already there. It's very structured. And if you have that right template, I'm going to go back to the middle here. If you have the right structure and template and you have the right model for how to interpret that, you can generate a narrative from it. So go ahead. So in this space right now, the data, the model, the generated narratives, there are quite a few companies that are out there. I'm going to mention two. I'm going to mention a third quill. There are also a few that are outside the U.S. that are operating internationally. And I know that we get people, I think at last count we've had folks from 40 countries here. So I don't want you to think that we just look at U.S. products. But ARIA in the U.K., for example, is generating narratives. But they first filter their data in some cases from IBM's Watson. The last one, Easyop is a French company that now has a major presence in the U.S. I believe it's in Dallas. So it's a hot area. And what we're seeing is that as these technologies advance, particularly the top few that I mentioned here, you're likely to be reading things created by these natural language generation systems without even realizing it. So quickly, here's another one from Automated Insights. Their program is called Wordsmith. And here's the Orlando Magic. They had all this data on tickets. And one of the issues for a professional sports team is getting people to renew their season tickets. And they did the analysis and realized that one of the great predictors of whether or not somebody would renew is whether they actually used their season tickets. And so they would track, and if they start to see somebody selling a lot of their tickets, they think, hmm, we should engage more, but that's time-consuming. It's expensive. And so they actually created this app that generates emails to their ticket holders based on what they see. And here's an example. We notice that your tickets are posted for resale. And by the way, it was the analysis. And he's not likely to sell them because other people in the same section have been trying to sell them for a while. So they've created a new way of engaging. They basically buy back or trade in those tickets for something else of value from the team and kind of build this loyalty. Well, that's not something that a regular application would have done. By taking that data and building a template that says, okay, under these conditions, something hasn't sold, something else is selling, this person, you know, we have all the analytics, they can add to it. And my understanding is that they actually deployed this app within a week. So once you understand your data, this is one of the simpler things to do today. Quill is one of the products from Narrative Science. And Narrative Science is an interesting company. I always like to mention them because they were spun out of my alma mater Northwestern. And what they're doing is generating text. They have a very interesting model for not only creating the narrative itself, it's not just selecting from, you know, a predefined set of alternatives. And they started out also with sports because, you know, sports scores are easy to do. But they're getting into, like, fraud reports for Deloitte. I don't know what they're doing for Inkytel. Inkytel is the venture arm of the U.S. defense community. So CIA, et cetera, have invested in them because I think this is an interesting way of generating. But, you know, here it tends to be a pretty detailed report that's being generated. And I know that, I don't have a good slide for it, but they have things that will let you fine-tune the tone or the tenor, if you will, so that it has kind of a point of view, you know. And if you're doing sports scores or a sports report for hockey, you may want to describe something in different terms than you would do in synchronized swimming, just as an example. You've got the data and you want to be able to have it sound authentic. Here's the one from the Washington Post. When I first saw that they were building heliograph themselves, I thought, well, that's kind of interesting because there are these others that are out there that are pretty well established. But what they're actually doing is the Washington Post has developed a new business called Arc Publishing to take the tools that they build in-house and provide them as tools for other publishers or for digital businesses. So the tools that they're using today to power their Olympic coverage, you'll be able to go on with an API and use those to build your own systems. So it's really a fast-evolving market. I'm pretty excited about what I see out there today. Now, before I wrap up and open it up to questions, I wanted to come back to that lingering question of what is understanding. And I guess it was the one webinar last month or the month before that I had just been involved in an interesting exchange looking at this question. And a specific example was Bob Dylan has been doing some commercials with IBM and Watson says, you know, I've read all your lyrics and I see the themes of time passes and love fades. I think that was the themes. One of the early researchers in this area just kind of went off on it and said, well, that's not understanding because that's not really what he was talking about. You know, he's talking about war and he's talking about protest. And the fact is though, if you just take the text and you had someone who didn't have any historical context, you gave all these texts to college freshmen who weren't around in the 60s, they would likely come up with the same thing. So the sentiment and the emotion of the themes and the concepts that we're deriving with natural language understand them and that we're using to create that model that may then drive natural language generation. The understanding maybe at a different level to maybe incomplete, it may in some cases be inaccurate. You know, we're dealing with things that are probabilistic, it's not deterministic. But at the end of the day, what we have today right now is already capable of solving very real problems. So I would say that understanding is still an unresolved question. You know, it's something that can be measured, but there's imprecision there. But if you decide that you're going to wait for all the authorities to agree on that we do have some 90% level of understanding whatever it is, you're never going to get there. So don't let the search for perfection in understanding interfere with your progress. I think it's time to get started today and so I'm going to give you a couple of quick thoughts on that and then we'll open it up to questions. So here, the issue is for both understanding and generation, which together as I said form natural language processing, the commercial technologies are imperfect but they're useful and that's so important. So I think it's time to get started. So the question is, are you looking for something that is natural language understanding? Do you need to derive data or insights from natural language? Do you have a body of data that you need to analyze and understand? If you're in the medical domain, you need to read the journals and hear something else. You'll need to look for patterns. Are you looking to do fault diagnosis? Do you want to allow people to have an interaction? And a lot of times the interaction with these systems is very much more natural language centered on the understanding part than on the generation. So someone working in a diagnostic system maybe asked a lot of questions and they can do it in preform English, but the answers are pretty structured and come out of perhaps a set of choices. So we'll look at, there are commercial technologies, you can do that right now via APIs. Or do you already have data and you want to create the content from it? In general, it's one or the other. The same application is not likely to need the same level of technology for understanding and generation. So let's look at what you can do right now. You can build it yourself. This one was from May of this month. And Google claims that there, and I hate this name, Parsee, like Parseface, is an English language parser. They claim to be the most accurate parser today. And it is now open source. You can go to Google, you can get a copy of this, and you can start analyzing. And as we saw, this parsing for syntax is the first step, so this will get you along the way if you're doing natural language understanding. We'll look today. Again, using APIs, Watson Developer Cloud allows you to go in and put your text in. You can experiment with this today. Just put files in. There's no charge to do that. You can do things like tone analyzer. If you're building a system and you want your customer to be able to type in some question or query, the tone analyzer with reasonable accuracy will tell you information about how that person is dealing with things. So that's another one that's available right now. One that was recently released, you can also experiment with Watson Conversation. This one is more bidirectional for bots. So it's similar in some ways to the technology or similar in scope to the technology that we saw with Google and the email. You can go in and build yourself a system with something like the Narrative Science Quill. They also have a relationship with Microsoft for their Power BI and with Click for Click Analytics. So you can go in and build a system that will take your data from one of those applications or take your structured data from something else that you're doing and create the narrative. Lord Smith, one that the Orlando Magic used. You can go in right now, try it with a free trial, and you can build something pretty quickly that's going to get you started. You'll know if you have a good foundation for building a bigger system. So this is where I always get to the point where is it ready? Yes, it's ready. It's ready to do something. It may not be ready for what you want, but you're not really going to know until you really give it a try like this. So with that, I'm running long. I'm going to open it up to questions. I'll head it back to Shannon and just let you know that we have topics and dates picks for the next several webinars. If you have any questions that don't get addressed right now, I'll be happy to do that offline. Shannon? Adrian, thank you so much for this great presentation. We do have several questions coming in. So to answer some of the most common questions, I will be sending out a follow-up email for this webinar within two business days. So by end of day Monday, with links to the slides, the recording of the session, and anything else requested throughout. Just to get right to it, Adrian, have you ever, have you seen specific successful uses of NLP by healthcare providers or improving healthcare delivery? Healthcare. Yep. If I get out of this app so that I can see things that you're sending them to me, will you just take control back of the screen? Okay, let me do that so I can also see that. Okay, great. In healthcare. Yeah, there's a couple. The ones that I'm most familiar with are the ones where the natural language was used sort of as a front end to the Q&A engine, if you will, or functionality within IBM's Watson. And so they have used that to power a number of applications. They're kind of self-service, and actually I should mention that some of these solutions that I talked about, like Watson Quill from Narrative Science and Wardsmith from Automated Insights, some of them require a pretty significant investment in services to build a big app, but most of them are available as self-service to get started. So having said that, in healthcare with natural language processing, the couple of things I've seen it used for translating patient records. Now just going from speech to text, I mean that's been sort of the reasonably well solved problem for a long time. If you go back to Dragon software, even before it was part of Nuance, you know, they had a medical vocabulary and you could transcribe your notes and it would just turn into text. But what we're talking about here is going further. So if you go to, you know, some of the things that IBM has done with like the Memorial Sloan Kettering, they put a lot of those records in. They digitize them by going from voice to text or from text into this directly into an intermediate representation for knowledge management. So that's one. There's also some apps out there right now that are using it to help. I can't remember. I think it's Healthline that's doing one where you can talk about symptoms and do it conversationally. Conversationally, in text, you know, you type in what's going on, they ask you questions, but it's all done relatively free form. So that is one of the early areas that's quite active. One of the issues there is certainly you have to be careful with these systems and not to be doing diagnosis because there are regulations around that. And it's also the case with healthcare that the same data coming in from a physician may need to be treated differently from whether something is coming in from a patient. So there's a big part of context there. Sorry, that was a long answer. I'm sorry, and we just have a couple of minutes left, but I want to ask, get one more question in. To your knowledge, is anyone trying to tackle or model the issues of colloquial language misconstruction, such as the verbification of nouns in English? Examples, I will text you. Yeah. Yes, how about that? I'll just say yes. But you probably want more than that. Yeah, there are a couple of things. One is that that is being done, particularly in some research things. I'll try and follow up with a citation on that. But it's also true that as words change in the way they're used and interpreted, if you're doing purely statistical modeling, you can look at that word, text, for example. Just as a string, it has a numeric value and then starts to do statistics on how it's used. You don't really know the word text. There's a company called Loop AI in Silicon Valley that their analysis is, to my knowledge, I did an interview with their CDL a couple of months ago, purely based on that statistical and pattern matching. And so if you start to switch between languages, that doesn't make much of a difference, you know, because it's not trying to do that translation first. It's treating it as, I mean, it's finding meaning through usage. So those are two different approaches to the same problem. Perfect. And while we are right at the top of the hour, again, I will send a follow up email within two business days. There's a ton of other questions coming in. I'll get those over to you, Adrienne, so you can see those. And, you know, thanks to everyone for attending and for being so involved in everything that we do and for asking such great questions. And Adrienne, again, thank you so much for such a great presentation and we hope to see everyone next month. My pleasure. Thanks. Take care, everyone. Cheers.