 And I'd like you to all join me in welcoming Catherine because Catherine hates your computer give her hand Start by thanking everybody for coming here thinking you're a python for having me and thinking the yo python social That I'm for hosting a really great event I've had a wonderful time and I keep meeting people way smarter than me. So that's always a good time My name is Catherine Jarmel. I'm known across the internet as K jam I run a company called K Jamis and so you can find me blogging and writing and talking about things at kjamis and calm And I like working with text and data And if you're ever in Berlin come by the pie data meetup group We're a fun group of people and we'd love to meet you So I feel like because I'm talking on machine learning and I don't have a PhD next to my name I need to have this this disclaimer slide, which is I am not an expert in machine learning I can't sit here and tell you about which algorithm Algorithms work better and exactly why I'm a Python developer and I'm interested in applying machine learning to text analysis and sentiment analysis So by all means if you are a machine learning expert, and I say something wrong Feel free to either correct me in the questions or I will buy you a beer later, and you can teach me something So here's some assumptions. I've made about you. I'm not going to be going over NLP basics I'm not going to be going over machine learning basics I assume that we already have those covered if you want to talk about basics later than again, of course Happy to talk about it later, but I'm assuming that you already have done Maybe even a little bit of sentiment analysis But you already have played around with some natural language processing and you have some basic understandings of machine learning and what methods to use So here's a little bit of what we'll cover in the initial description I said that there was going to be live coding and demos in this and that Upon further inspection that becomes a little bit more difficult when we go into deep sentiment analysis and cross language Sentiment analysis, so what I'm going to do is I'm going to present you some code and some repositories that are good to use But we don't have time today for live coding instead. We're going to cover some of the tools We're going to talk a little bit about Watson and how it's being used We're going to talk a little bit about sentiment analysis and we're going to cover a lot of the research that's being done right now I'm seeing a massive gap in terms of the research that's being done around sentiment analysis and the tools We have available in code and I'm kind of hoping to push forward the Conversation in the Python community by giving this talk and kind of covering a little bit of what's happening So what we won't cover is some magical Python library where you can download today and it works in multiple languages for sentiment analysis I must say that if you want to take a look Please send hate mail to Salesforce So in case you wanted to send hate mail somewhere or you want to practice a natural language generation and send hate mail somewhere Send it to Salesforce MetaMind was a really amazing startup using matrix vectorization recursive or sorry yeah recurrent neural tensor networks and they had an open API and it was fabulous and then Salesforce bought them and they shut down the API and they're currently shutting down the paid API as we speak I ran into a guy who works for Salesforce who actually gets to work with this team and I asked him I baked him Please let something be publicly available and he said good luck. Have fun with that so Again all the hate mail and if you work for Salesforce and you can somehow get me inside the internal network that would be great So another one that's trying to do some things and they're one of the only ones that's available via apis is monkey learn And so here you can see that they have these different modules available They have some that are being actively worked on you can also create a publicly available module here So if you absolutely need hey K jam like I need something that works tomorrow I would start at some of these places, but we can see just even looking through the API That we have very different precision levels and so I would say That's why I'm giving this talk is we're gonna talk about some of the methods and the theory behind How to create a sentiment analysis that works for you So to begin with we're gonna talk about what is sentiment analysis and how do we go about it in Python? So what is sentiment analysis really and when I look at this tweet and I Being American some of my tweets in my Twitter feed are very American references So forgive me for being that way, but when we look at this tweet We have a lot of different sentiment right if we look at the emoji we see crying if we look at the Photo that we have here, then we see a different emotion of crying if we look at the text Then we might just see the word like and determine that this is a positive analysis And so when we're talking about sentiment analysis, we have all these different things We have stance. How does the author feel towards Jennifer Hudson? How does the author feel towards the BT awards? These are really complex things and we think we can boil them down into positive negative neutral But I am here to debunk that myth today So sentiment analysis is used in all different sorts of systems And I've had the pleasure of going to a sentiment analysis symposium recently and hearing just how many different places it's used There are even people using it in anomaly detection and they're charting sentiment across reviews or Twitter And they're looking at it via release tags and so there's all different ways that people are using sentiment analysis Including in obviously just simple user satisfaction or brand engagement So when we look at the sentiment analysis steps, there's can basically be broken down into four major steps and They basically go from left to right and top to bottom However, depending on what you choose they could be mixed up. So the first thing is dealing with your corpora Are you building a lexicon? What how are you using your labeling? Next then you're probably going to choose your algorithm or you're going to determine what model or approach to use Then you're doing parsing and pre-processing you might even do normalization and standardization across your data set and Finally, you're testing evaluating and improving and in that improvement step. You may revisit any of these earlier steps It's quite possible that you will need to to improve your model So we're going to go through the talk and we're going to basically cover all of the steps And we're going to talk about what's happening in research and what's happening in code The first step is choosing your lexicon So one of the oldest and most tested and true ways of autotagging a lexicon Particularly if you're using social media data is autotagging and it's this idea of distance supervision So I'm going to go and I'm going to gather tweets or I'm going to go and gather Facebook statuses or whatever I'm going to use and I'm going to map with emoticons or emoji and you can think about this also as a good Opportunity to use at the SARS so in quite a lot of the research. They will also just expand The corpora that they're using by using at the SARS So I want to grab all hashtag happy hashtag joyful etc. And this can also help expand Into new languages if you're working in one language and then you want to apply a different model as long as you have a very good The SARS then you can do that Then one of the other steps that you're going to need so now you have all of this data How do you tag it? We have this very naive idea that everything can be tagged positive negative or positive negative neutral or even positive negative indeterminate neutral that's Really in research being proven wrong and so what we have to think about here is if you really need a simple model That's simply positive negative and neutral Then you're you're going to have a lot of gray area And you're going to have probably more ending up in neutral or more ending up in false positives or false negatives So when you start moving away from that and you start thinking about Positive negative as a scale you start to see that things converge a lot faster and that you have less of these false positives and negatives This also means that you have to build your own lexicon and you have to label it yourself So there's pluses and minuses and depending on the resources of your team. You may not be able to do that There's also quite a lot of people looking at stance detection so this is a mixture of NLP topic identification or Entity detection and then using stance and a lot of that right now is essentially a bag of words models still and People are trying to evolve that to be more advanced and to try and look at things and say okay Can I determine that these set of words are actually applied to this? Organization or this entity and then I can start to detect these stance. So it might be positive towards I really like this restaurant. It has great food, but the service was total crap Then that means that I have one idea of the restaurant one idea of the food and one idea of the service Right and then finally we have this categorization by emotion because we have this wide range of emotions It's really difficult to determine just positive or negative So in terms of tagging a lexicon and it looks like one of my emojis is broken We have a few different methods So I'll actually start from the bottom the simple Boolean is zero one right or maybe negative one zero one Then we have this sliding scale which is going from completely negative to completely positive And I'm asking people then to just rate on this scale what they've actually found with research on that is that people tire over time So if you're sitting there asking me to rate something for five hours I'm eventually gonna just be like neutral neutral neutral and try to get done with it Right and like we know that just from our knowledge of humans. They also have found that Psychologically people avoid the edges. So you'll get much more representative of like it's kind of negative It's kind of positive because people don't want to say it's completely negative or completely positive So one of the best methods that's recently come into practice is this best worst scenario And you have a list of say four or maybe five words or n grams And you can use that and you can ask people say okay Choose the most positive and the most negative and that actually is creating some really interesting Lexicon and what they found is that people agree after about three examples So I can show it to three people and move on to my next sample and I'm in the 90% agreement range Which is pretty massive for tagging lexicons And this is another simple study and this is somebody Saif M. Muhammad who works in the National Resource Council in Canada does quite a lot of different talks on Semival competitions, which is like a sentiment evaluation machine learning competition that happens every year and He has some really great state-of-the-art models What he found is that when he switched to using a scale of negative one-to-one that That 90% of people agree within point four of each other So that was a really interesting revelation to move away from this zero one because we hear constantly Well humans don't even agree 77% of the time or whatever Statistics you look at and what he was able to find is when you give people a scale and you let people Speak on a scale you can actually find this least perceptible difference and particularly then if you even focus on native language Speakers you get an even smaller a smaller score difference there So I had the opportunity to chat and I'll hopefully be posting our chat with William Hamilton Who's currently working at Stanford on his PhD and he's developed a new model called social scent and what he's done is he's taken subreddits and he's Determined that subreddits and different communities use language differently So when I'm hanging out in our sports and I say oh man, he's really soft I don't mean that as a compliment right? I mean that in a really negative way So again, we have this mentality a lot of times in this code that we all use the same words to describe positive and negative And that's just not the case especially if you're doing sentiment analysis on a community and especially a community Like for example us as Python developers. We're gonna have a lot of different ways that we use words He has all of his data available and I took a look at our programming and I pulled out someone the most positive and the most negative I Also found out that Python has a slightly negative definition So I know you're all talking trash about Python on our programming and you need to stop So here I found some interesting things it's funny to me that 200s right in the middle like we can tell that we have web developers in our programming and Spaghetti is really negative, but I love spaghetti But spaghetti code maybe not so much and we find that Minecraft is has a great positive Association and so it's a really interesting data data set. It's only unigrams right now, but he's very open to suggestions So feel free to send him any suggestions or ask him to do a certain n-gram for you So now that we have our lexicon or we've chosen a lexicon we need to choose an approach or an algorithm so how do we do that what machine learning systems can we use and it turns out we can use a whole bunch and I know this is throwing a little bit of a lot on the screen But there are all different ways that people are using machine learning with sentiment analysis And they're achieving really great results with really small data sets So I'll come to this again later, but if you are dealing with movie reviews You're good to go But if you're not using the IMDb model or lexicon You start to enter this space where there's quite a lot of active research and not a lot of agreement on what's good to use Why or why not and people are achieving very different results with very different systems Obviously the finely tuned state of the art systems are still the best performing ones right now And this is these people that spend all of their time Researching and pulling out little details, but we don't have time for that right? We're probably Python developers first and sentiment analysis researchers next so I'll talk a little bit about how these are comparing First I'm going to talk about kind of the older approaches and the newer approaches The old approach used to be bag of words or even continuous bag of words and I would say oh, yeah It has mainly positive words. It must be positive The new idea is to start to use word embeddings and start to be able to use those word embeddings in deep learning models And by doing so I have a little bit of more complexity and I can say hey There are these words bunched together. Maybe they are bunched together because they have a sentiment together That's not always the case The old way was term frequency inverse document frequency, which is basically a nice way of doing bag of words on a document and The new way is using these doc to vet or these document embedding models The old way is kind of unigrams if I say a good job I mean good and I'm talking about job and they're all related and the new way is talking about skipgrams Or even dependency modifications, which is starting to look at the parts of speech and Label the words with parts of speech because I may mean very different things if I use a word as a noun versus a verb and The old way is these supervised state-of-the-art systems and the new way is kind of moving to a semi supervised approach So if you're curious about some of the state-of-the-art sentiment features and I'll post these slides because they reference the papers across the bottom This is a good summarization and this is a summarization of again site East Muhammad's model Which has done really well in competitions And these are all of the different things that he's pulling out as features and then he's fine tuning them and Analyzing them and and pulling out probably more sub features that we don't know about that haven't been released But you can see that there is a starting place if you're interested in tuning your own model But let's talk about what we can actually use as Python developers and what we can use easily We can use word to beck and doc to beck how many people in here have used Jensen Okay, so Jensen is this great library and what it gives you is these word embeds Sorry, how many people know about word to beck and doc to beck? Okay, super. So it's this idea of representing a word or a document as a vector So everything is in a vector space and it can be multi-dimensional It is multi-dimensional and What you're going to do is you're going to use Jensen and Jensen can load in these multi-dimensional vectors and give you a matrix or Vector representation of each of your words and your documents and what this is going to give you is a mathematical way to represent text Right and this is kind of where text was held back for quite some time Is we didn't have a really good mathematical way to represent text? We're using counters and bag of words and that's not that interesting So there's also glove Python. So there's two different ways to word embed Everybody talks really a lot about word to beck and I feel like there's not enough attention given to glove Glove is a Stanford model and it has a slightly different representation It uses a weighted model So word to beck is basically does this word appear in a sentence then it means it's somewhat related to these other words in the Sentence whereas glove is going to say hey these words are closer Therefore they might have more of a proximity in terms of meaning So depending on what word embedding you model you use or method you use you can get really different results So here we're comparing a word embeddings built on bag of words five, which is like a five n gram like a You know five words around count Then we have bag of words two and then we have a dependency which again is this part of speech And what we see is if you take a look at the Florida line What we see is in in Florida with bag of words we get this mixture of other ways to talk about Florida Cities in Florida, but when we move to dependencies we get other states and this is maybe Significant to whatever you're trying to do if you want other entities Likewise entities rather than just related words. You might want to look at a dependency word embedding rather than a bag of words word embedding There's an online comparison tool where you can look at this And so I took a look at Python and it looks like there's some of them mixed together If you use bag of words five two or the dependency you get slightly different results and the really neat thing about this is In this as well They give you a little bit of a visualization of how those word embeddings are used And so if you want to just play around and start to make decisions on what you're using for your word embeddings Then I recommend going to this site So another thing that I feel like needs to be raised is it was recently released There was a paper referenced at the bottom where word embeddings are not neutral They're based on human language and they have the same biases that you would expect to see in humans and To come up with some of these vectors. I was using the Google news Google news vectorization the 300 one and that one is used by places all over the place And I found some pretty atrocious stuff in there Maybe not in the first ten examples like for example here at the bottom There was some intense racism going on if I looked at the top 30 words that were within that space So I forewarned you to especially if you're using it for something like sentiment, right? Which were we may have some words mixed in there that are very biased and or if you're using it for natural language Generation that you need to be very aware that these biases exist one interesting thing However, is the authors of this paper were able to find they could find vectors Showing the actual like misogyny so they specifically focused on the misogyny in it And they were actually able to reverse it and reverse the wreck vectors by finding these vectors that were That were pushing things in a direction where like women had lesser professions So let's talk about what that actually means Here I have some great examples used it is too much code to go over in a short talk so I Would recommend looking at them. These are great examples for simple machine learning. So this is using naive buys or Simple linear approaches linear SVM and what it's doing is it's going to take the vector and it's going to chart it to Whatever it is whatever you're using as labels And if you want to do it on a document approach You're essentially taking a vector of the document model or the sentence model it depends and they have different results and Then you're passing it as a labeled sentence a labeled sentence is actually a model in Jensom So if you're using Jensom, you can label your sentences and then pass them directly in let's say Logistic regression or whatever it is you'd like to use. This is like a simple approach, right? These are the shallow machine learnings Another thing to be aware of when you're making these decisions is you're choosing your n-grams, right? So you have to choose unigram or bi-gram or how do you want to represent these and when you're choosing word embeddings? You're generally making those decisions as you're choosing the word embedding and what they found This is the research by Stanford which enabled them to come up with the matrix vectorization recurrent neural tensor network and what they found is The longer you let your n-grams the more mixed sentiment you have which is kind of logical When you think about it, and I'm going to go on and on and on and have a big long sentence I'm probably going to express maybe differing viewpoints or a mixed emotion and so if you're choosing Unigrams or bi-grams you can see you're way over here and There's some words that are obviously negative and positive But there's probably a lot of words that are just filler words and the further you go towards exploring the n-grams The more complex sentiment you get and the more people are able to say okay Yeah, this phrase is definitely negative and the whole phrase is negative So this is based on the Stanford sentiment word word bank, which is available in Java another approach that people are using for this that's going pretty well is LSTM, which is long short term memory networks And these are deep learning networks and what they're using is they're generally using these word embeddings or these document embeddings And they're pushing them into these deep learning models and the nice thing about LSTM And why it's been doing some really amazing stuff for also Other natural language processing is that it can forget things So it can learn things and it can forget things So because it has that ability what it can do is it can change as it sees more or less of a Representative model so language we know changes and trends come and go and because of that this approach has been really Powerful and you can use it with both the ono and tensor floor. I have some examples There's also the ability to use convolutional neural networks And so initially the idea is that convolutional neural networks are really great at images because you can chunk and label Images and then move accordingly through the pixels right you can crop them But now people are using it for sentences or for documents And they're able to use the word embeddings for each of those and chunk and move through them And then through the same principles of pooling and softmax you can then reach a conclusion and say okay According to my sigmoid softmax. This is positive or negative or neutral And again, I will be posting these so you can take a look at the links It's a little bit different when you're during dealing with deep learning labeled text So what a lot of times you'll have is you'll have the document and then you'll be passing an array And a lot of times people are just using simple arrays like okay This is good is positive. So it's going to point up and this negative is going to point down There's different things happening in this so I would keep an eye on this space in terms of how it how it performs because again Right here. We're having this everything is either positive or negative I'd be curious to see and I know that some of the academics are working on more complex Representations of the labeling and because of that that this might become an interesting space sooner rather than later And then here's where a lot of people are kind of using this dual system Maybe I have a symbol classifier and I have a lexicon and I have auto-tagging and I'm feeding them both into a deep learning system So maybe I have a constant Twitter feed or this or that and I'm feeding it into a symbol classifier And I'm allowing my deep learning system to learn from that classifier This is where a lot of people are starting to experience some big jumps without having to do a lot of work So if you're like me and you're lazy, this might be the best approach There's still obviously some open research But it's an interesting theory this idea that I can just use my simple state of the art classifier and feed it into my deep learning system And my deep learning system will eventually become more intelligent and have a much wider lexicon over time So depending on your system, you might need to handle regularization Pre-processing parsing. This is like the classic NLP problems, right? Is Pre-processing essential? It really depends who you ask. There have been some really interesting research on Sentiment analysis and saying doing too much pre-pre, sorry, doing too much pre-processing Actually is a problem and what ends up happening is you get rid of some of this way that we speak in slang to one another Or the way that we use colloquialisms and by getting rid of those you actually are removing part of how we're expressing our selves So what I would recommend and what generally the literature agrees upon is do minimal pre-processing like maybe just some simple limitization or some simple tagging and then Try it and then do more pre-processing and then try that and see where you hit a nice accuracy in your model And this is kind of where sense to Beck and spacey are really making some cool inroads spacey is a startup based in Berlin and They are doing essentially Parsi McParser face in Python And what they're able to do is they they trained on or they have the English language and the German language Corpus available trained on reddit data and what they're able to do is they're tagging each of the vectors So sense of that is you have this tagged vector So when I talk about Google the organization and Google the verb I mean different things and I might feel differently about them in terms of sentiment So they're making some great inroads when I pass in this is using spacey when I pass in the vector for the winky face Winky face old-school emoji Then I get back some of these other things and I can see that I'm getting back interjections Which is probably a lot more of how we use the winky emoji in the way we express sentiment Like I might say something that I that I don't mean like I hate to computer and then use winky face to say I'm just kidding. So here we can see that he heaps It's spacey is picking up on the fact that I may actually be negating what I'm saying and I'd be really curious to see how spacey gets Incorporated into a lot of these deep learning models. It hasn't been incorporated in most of them yet Finally we get to kind of like what IBM Watson is doing and we're talking about emotions here So there's this famous Diagram that you see almost at every sentiment analysis talk and it's this idea that we have all of these emotions but our emotions can be Made down into just these eight core emotions and then everything else is just some combination of those eight core emotions. I Am no Psychologist so I can't comment on this, but you know it makes somewhat sense So if you're looking into sentiment analysis and you want to start moving away from just positive Native and you want to move into emotions? Let's say you're in charge of like customer service channels and you're trying to say I really want to know when the rage and loathing are activated Because we need to work on that first, but I don't care so much about the positive emotions. I care about more of these angers and IBM Watson has the tone analyzer that's available on the blue mix API and it tries to classify into anger joy fear distrust and sadness It also has the social tendencies and the language style So one of the interesting things that I was asking some researchers when I was able to chat with them is hey Could you maybe say hey? I'm angry, but I'm saying something positive that means maybe I'm being sarcastic or ironic and What what the response I got is that nobody's really doing that yet, but that would be interesting to hear about So I will hopefully be building something that does something similar to this and taking a look at can we start to? Talk about when our words say one thing But our tone or maybe our tone over time says another and by building these models by even Necessarily saying like you as one person have a lexicon and you speak a certain way Can we actually start to understand sentiment at a much deeper level? We'll get into that as we talk about The next portion which is what is still unsolved which is sadly quite a lot So apologies if you live at the UK and are offended, but I found this to be very funny But yeah humor humor is lost on sentiment analysis models We don't know how to tell when somebody's being funny And we a lot of times can't even tell what they're being funny about So here I have this mixture of this you know social media Presence which is this mixture of images and text and how can we start to you know? We don't have to understand images now and we know how to understand text But there's very few that try to do both and in sentiment This is a really key part because how we're talking online is with this combination so negation is still a really big problem and Here's a unigram and by Graham study that says when you negate a unigram or a by Graham The red line is the assumption that you simply create a negative value for that unigram or by Graham And what we see here is the blue dots have the actual representation. So when I say Yeah, it was not that bad Then I don't actually mean that it was good, right? And so we can start to look at these and talk about hey It's not so simple to say just because even somebody negates an emotion It is some opposite and there's been some really interesting studies about Modifiers and how maybe they can point towards a mathematical representation of negation of a sentiment so back to Stanford sentiment trees, which are these MVR NTN networks and They're actually have gotten pretty advanced at determining negation and overall sentiment of a complex sentence And what they are doing here is they're creating smaller n-grams and they're creating sentiment vectorization for that and then they have a series of logic rules and the logic rules state, okay well, this is more negative than this is positive so overall it is negative and They're doing some really really interesting research again. They have their Java jar that you can download and play around with it's still mainly trained on these movie reviews Which we'll get to in a second. So we still have this problem that it's trained on movie reviews So there's still quite a lot of things that are hard We have mixed sentiment. We have complex motions sarcasm irony humor lost, right? We also have speaker intention and personality and there's been some interesting research in that in terms of trying to determine Who you are as a speaker and if I study you over time? Can I determine who you are as a speaker and then can I get a better idea of what you mean? We also have slang and like new and older phrases So part of William Hamilton study with social scent was looking at sentiment over time And he was pointing out that if you look at historical documents You can't use any normal sentiment analysis because in 1920 people were talking a lot differently about things And we have these new phrases too where you know cool AF means something very different than just cool, right? And so then we also have just general NLP issues So I feel like I would be left out of your Python if I didn't have a Pokemon reference in my slides So I put one in here Which I really hope people in America do anyways because I'm very fearful watching from Berlin about what may or may not happen in November And so yeah, let's put some Pokemon lures and and we shouldn't be pulling places But here's an idea. This is obviously exploring a sentiment but No sentiment analysis would understand this so when we're talking about cultural references We also need an ability to say hey right now Pokemon means emphasis We also have speaking with images and Gifts right so now that Giffy is part of Twitter This is important, right? I would not Understand the sentiment of this tweet if I couldn't also look at the image and then determine sentiment of the image Right and the really interesting thing is a lot of these are tagged, right? So these are in a tagged database somewhere and maybe somewhere. There's like hashtag hate or hashtag Or whatever it is and I can apply sentiment to these and then I can pull them out Pull pull out the visual representation and put it back into words and use those words to make a decision Another really big problem with these is speed speed and memory So if you ever try to make your own word embeddings like I hope that you have a Server somewhere set up with a lot of memory and you have a lot of time on there because sometimes compiling these word embeddings Glove is actually a little bit faster a lot faster almost half the time But word to back if you're making your own word embeddings it can take 1415 hours and if you're using glove it can end up eating up gigabytes and gigabytes of RAM So if you're making these embeddings yourself it can be become a real problem and a performance bottleneck both for how do I keep? Feeding my model and also how do I you know apply it? Let's say in a real-time situation so what some folks in Munchen have been studying is the ability to create dense word vectors and dense document vectors and they created this idea of Densify and what they were able to see I don't know if you can see over in the corner But they were able to get it from English words for executions per second to the densified model at 178 with only a slight difference in accuracy so the more solutions we find for sentiment analysis the more problems we have and Depending on especially if you're working in different languages this can get even more difficult What some people are doing that is interesting is attempting to do aggregation within lexicon So if I aggregate everybody speaking English in this particular city what they're finding is maybe there's more agreement within that community on sentiment and words So that's something to look at I also haven't seen a lot of studies on ensemble methods And I think that that's coming into vogue as well as character level embeddings So character level embeddings are the idea that we can predict and attune ourselves on a character level given a language So here is a graphic. I made it's very tongue-in-cheek. It's sorry if it's a little small But this is like my own version of Andreas mules scikit-learn flowchart Which basically says if you have movie reviews or any reviews you're set and if you have anything else and you don't have Money and time and samples. You're pretty much screwed. So sorry about that And I'll So have we solved sentiment and else Not really But if you have time or are willing to work on building a lexicon particularly for your model what they've found is that there can be some really great Winds so I found a lot of papers and also a lot of companies that are building their own Corpora and their own lexicon and they're achieving in the 90th percentile So I want to say thanks you can reach out if you have any questions I'll be posting the slides so you can read all the papers and Look at the code that I referenced for the LSTM as well as the CNN and the simple Simple model architecture All right everyone we have a time for two or three questions. So who would like a first one? Do-do-do-do if you want to leave desperately and go and get coffee, please do so very quietly and respectfully Okay, here we go Hi, thanks for the talk. It's great. I wonder if you had a chance to look at Google sentiment analysis API. They just released this week Look at that. I do have a bookmarked, but I'll let you know when I do the problem really is is that a lot of these really depend on the lexicon and Google is likely using their word embeddings and their word embeddings have problems because everybody else is also using their word embeddings for sentiment So I would doubt it's massively better than what's available from Stanford Which is generally in the high 70s percent low 80s But I'd be curious to see and and if you have done any have you used it yourself? Yeah It was a little there's so much happening. It was hard to keep up with everything and also prepare the talk Thanks very much for the talk. I'm a linguist by training So a lot of the examples you mentioned Really rely like the being able to accurately tell but the sentiment is really relies on knowing a lot about the context So how how much do you think we can actually expect? How accurate do you think we expect to be just based on the text because it seems like people are really really pouring a lot of effort and time into Making the text model is really great and this is wonderful But maybe there's just a cap at how well we can do only based on text I agree to a certain degree to that statement and basically what I'm curious about is more when we start to look at these dependency models and we're start to able to detect stance and phrases that if I'm using a series of phrases Directed at a particular object and particularly if I can do that in a multilingual situation Which people are starting to prove that you can do that that's gonna take sentiment analysis really to the next level because if I can start to say this cluster of words applies to this object and It's obviously negative or it's obviously positive Then I can start to make a little bit of these leaps and bounds and moving away from just this idea that Somehow words around other words So the better our parsers get and the better the more we incorporate the parsers into our models the better Sentiment analysis has gotten over time This question way up at the top Sorry, thank you amazing talk Catherine. That's that's really interesting It seems to me it's a bit like the previous question there this sentiment analysis is really a problem of hard AI and I wonder whether It's going to be the thing that Helps hard AI or whether hard AI is going to Give you the the ultimate goal in in sentiment analysis. Does that does that make sense? Yeah, it does and I was really really impressed with some of the deeper learning models that have been coming out of this and the Fact that they're actually having really really good accuracy with very little training So there's actually some great papers and I'm happy to reference them and post them There's some great trainers of saying. Ah, yeah Well, we just decided to play around with it for a month And we were able to nearly get to state-of-the-art fine-tuned systems So I really think that deep learning is likely the answer here and as deep learning evolves I think sentiment analysis will too because we've seen what it's done in terms of natural language generation and if it's making these inroads in natural language generation and we're getting these networks that are able to Start to understand what we mean or what we're trying to say then we can start to maybe predict how we feel But the problem is the sentiment analysis field is still like very very much a Closed model most of the places that are doing it are owned by places where they can't open source their code metamined and So I think that there's this pressure on us as like open source developers to try and keep up with these things That are happening behind closed doors And that's all we have time for folks. So I think you'll join me in that thank you in Kjam once more