 Thanks very much, Cronin. Hi, everyone. Thanks for coming. Yeah, so this is going to be on a similar topic, similar application area to what Shane just discussed there in his excellent talk. But going to go a little bit more talking about lessons learned for building machine learning products based on our journey through this application area. So a little bit about me. Thanks, Cronin, for his generous introduction. I'm a technologist. I've kind of a degree in computer science and PhD in machine learning. I really have always been interested in technology. But over the course of my career, I've always found myself sort of sucked more and more into the product management side of things. And maybe this talk, I'll be able to explain a little bit why that is as I've tried building chip successful machine learning products. A little bit about Intercom. Just want to make sure everyone has heard of Intercom. Maybe you haven't. Intercom is a company based in Ireland here. And Intercom makes those kind of little widgets that you see in the bottom right-hand corner of the website. You can click on that, and you can open up a messenger. And writing is sort of to the business. You're kind of your customer support problem and your sales problem. So this talk, I'm really trying to talk here about pitfalls and lessons learned. Pitfalls have fallen into lessons I've learned building a couple of different machine learning products in a few different companies. But through the vehicle of answerbot, because a talker where I just kind of worked through a checklist of all the things that have gone wrong is probably less interesting than talking about how we built this product for Intercom. And the context I'm really speaking about here is software products, not big mega projects. If you're building this huge, big research machine learning product, some of these lessons may not apply. If you've got tens of millions of dollars of funding. But for everybody else, I think this is interesting. And I'm really reacting here that I've been in several conversations with people where they read some of these papers produced by these excellent companies like Google or IBM, talking about all the things you need to do to ship a machine learning product. You need to have scalability. And you need to back test all your features. And you need to make sure things aren't drifting. And these huge, big, long checklists and capability maturity models. And I've looked at these papers, and I kind of go, that's kind of alien. When I'm starting to build a new product, the thing I'm worried about is not whether it can scale to tens of millions of users or in a year's time, whether it'll be too much technical debt. It's like, how do I ship anything? Does anyone want this? And so I want to talk a little bit more about the more pragmatic side of shipping machine learning at a smarter scale than this. True talking about Answerbot. So Answerbot is a product we built, perhaps not a million miles away from some of the things that Shane was talking about, to attempt to answer people's questions. So Intercom has a whole customer service help desk. But actually, we believe that there are some. We have evidence that there are a percentage of questions that every business has that do make it true to their customer support that sometimes can be answered with a bot. And Answerbot is our attempt to build a bot that will fully automate those things. So the user comes along, they ask the question, the bot jumps in with an answer, and hopefully that helps. And the end user can say, yes, my question's been answered. So why build Answerbot? Automate drudgery. Again, in a lot of our data, not all companies, but a lot of companies, they're not proactively catching every help query before it comes true. There's a certain amount of queries that do come true in a well-oiled support context that might be a small number. But customer service agents are still getting asked these kind of drudgery questions, the questions that they have to answer again and again. And the second reason is to just help people scale their support effort. There's many risks when we set out to build Answerbot. One of the big risks with a project like this, I think, is always end user experience risks. And so this is something that I think when you're starting out to build any machine learning product is something you should really obsess over. I want to give an example here of an application that, hopefully, some of us are familiar with. This is the Google Android keyboard and the suggestions for the Android keyboard. And I love the design of this application. I love the way from a machine learning product perspective. I love the way that the replies, the suggestions that come in, the type ahead suggestions that come in as you type your keyboard, they appear above your keyboard. And someone spent a lot of design effort here to make these subtle. These are not typically annoying if they're irrelevant, right? And so you don't get that for free. Someone spent a lot of design time to make a suggestion that was displayed in a way that if it was helpful, you could kind of notice it. But when it was unhelpful, you could kind of ignore it. And that's really the best place, the most fruitful place you're looking for when you're building a machine learning project, because you can ship a first version of the machine learning here. That's not amazing, but doesn't really damage the end user experience. So basically, the cost of false positives are low. And so when we were building Answerbot, Answerbot was going to have to interject its way into conversations. And we were a little bit nervous at the start. What's going to be the cost of a false positive there? It's going to get it wrong, like the examples Shane mentioned. It's going to be times it doesn't understand things. It's going to be times it comes in when it's irrelevant. How much does that hurt the end user experience? There's obviously technology risk. We knew natural language processing is getting better and better. But was it good enough to produce a good user experience there? And finally, there was a huge worry for us was customer set up experience. So Intercom has tens of thousands of customers. They needed to be able to set up and configure Answerbot for each of their businesses without any intervention from us to make this work as a product. So that was a risky thing for us. And so I would suggest, and I guess one of my core thesis in the talk is that for any machine learning product you're building, you're trying to find the path that cheaply reduces these different types of risks in a balanced fashion. So something is probably not going well if you find yourself six months into your machine learning project and you're still working on the machine learning and you haven't shipped things contact the customers. Now that's sort of my thesis. You instead want to find the simplest possible feature you can do that you can ship and that you can start creating value and kind of get that flywheel spinning. So with that in mind, I want to talk through how we built Answerbot in the timeline of Answerbot. So Answerbot took us just under a year from first prototype in November 2017 to launch. And hopefully it'll be interesting. Some of the things that went really well over this time that we're proud of and some of the things we messed up. So we started off in the first sort of six weeks of the project. We did a wave of prototypes. Prototype one was just something just hacking together in a week, Jupyter notebooks and regular expressions if you built a bot as a college project or something like this. That this might be how you did it, really old, old tech. But this enabled us to put something in front of stakeholders internally and make sure we were communicating, make sure we were speaking the same language, making sure that the people in the design department weren't looking at what we were building and going, no, you're going completely up the wrong tree. That was the weekend. Prototype number two, create a whole bunch of content for this bot by hand. Teach it like 20 or 30 different questions that people might ask Intercom using the finite state machine and using the search engine. We didn't build any machine learning or any proprietary algorithms or deep learning. We just took an off the shelf Python search engine and wrapped that into the bot. We weren't going to build the machine learning if we don't have to because we're trying to reduce all the different risks in a balanced fashion. The question there is, well, will this search tech work for us at all? How does the user experience feel? That was maybe two weeks in. The next prototype, let's take this thing that's moving out of a Jupyter notebook just showing it on a laptop to people and let's actually put it on a single EC2 instance running Python back end. No scalability concerns, no months of prototyping with Spark or anything like that. A single machine, but now in the Intercom messenger on an internal test site that we can show to internal people. And it looks like this at this point. So here's the bot. It can come in. You can sort a dialogue with it. But look, it's within the Intercom context. So our designers can start a look at that and go, hey, you're not going crazy. That's a reasonable way to proceed. Loads of stuff isn't implemented yet. There's no buttons. There's loads of unimplemented things, and that's fine. And the real question there is, does this feel plausible? Are stakeholders OK with that? Stakeholder management is huge on any sort of machine learning product. So yeah, after that, prototype 4, it's looking a little bit like this, maybe six weeks into the product. And we had kind of been going breath first, like building the smallest little piece of each bit of the product that we might eventually need. And we were really running into this stumbling block, which is, how does the bot get content? And by sort of proceeding in this breath first way, we hit this stumbling block early. So we prototyped a tool, a really terrible JavaScript tool, which would take people to historical questions, put them through a simple off-the-shelf and non-negative matrix factorization clustering algorithm, grouped them into clusters and showed us to people and enabled you to start training the bot that way. And we're like, yes, there's really something here. So it started in alpha. Actually, that's the version there on the left. So maybe this is maybe two months into the product. And it's six weeks to two months. And it's live answering questions for real customers on intercom.com. Got a load of learnings by that march. Engages just seemed happy. We shouldn't be too worried about the false positive risk. Curation is a very iterative process and is hard. While we shouldn't be too worried about the false positive risks, we had no way of solving it with search. So we decided to add keywords where a human curator would manually have to type in a word that the bot would require to be present for the bot to fire. And this was a really hard decision, but we decided to do it to see if we could get away with search. Product magic and the bot, when it perfectly answers a question, it was just a great experience. People got their answer immediately without having to wait. And using historical questions when configuring the bot was great. We moved on to full production, just answering simple single questions. That was the envelope we had carved out. And it was kind of working well for us. We were carrying risks, the risk of false positives. We had no way of mitigating that. And the risk of just the setup experience, what are customers, not us, but what are customers be able to set up the bot for themselves? And it started to move forward with the bigger product team to kind of build this kind of curation interface like this. So this is how you're training a specific question. You're saying, yes, but these are examples of people asking this question or no, they're not. An interface where our customers could do that. Kept running into this problem, we had kind of said, OK, we'll solve the false positive issue by having people curate keywords. And it just wasn't working. Everything was going wrong. We had months of design iteration. Our customers just couldn't get their heads around manually curating keywords. We had made it too hard for them. And we tried so many different design experiences. It just wasn't working for people. It was a no on risk when we entered the product, but it was still very dangerous. And the wheels almost came off the entire thing at that point. But at that point, we started our own machine learning product to replace this off the shelf search engine with something more modern, using word vectors and deep learning and so on. And it started the machine learning product to do that. And at least we knew that when we nailed it, if we nailed it, if we got this done, it would be useful. And so there's a way in which this is a bad part of the story where things got a bit dangerous first, but also where it's good, because at least we knew now this investment in machine learning we were making, it had a drop in place in the product ready to go. That worked successfully after two or three months and we're feeling pretty good about things. We hit our second problem. You remember back during curation, we talked about using topic modeling to extract questions historically. And it's very difficult to take an academic algorithm or an algorithm like NMF and ship that because frequently it gets things wrong. And it's one thing to, if you're a data scientist hacking yourself when you get bad clusters, but if you want to put this live in front of customers in an unsupervised way, you really need a more conservative algorithm. And so we ended up doing a network-based approach where we kind of would find groups of clusters using a network-based clustering approach. And this showed up in the product like that in the end. And I guess that the lesson there is that sometimes these academic algorithms or the more standard algorithms you might use, they're really great algorithms, but they need adaptation. They need to be made more conservative and more predictable to put them into production contexts. We launched in October, the beta felt good. It's giving many people their answers and seven figures, numbers of answers at this point delivered. Great metrics and still the scope, you still have a lot of work to do on this. I guess my key kind of takeaway points, machine learning products are always risk-eater, like much riskier than CRUD apps because you've got this unknown technology risk. And so you have to balance product and technology risk. Don't go and spend six or 12 months and build a beautiful design that just needs an amazing machine learning feature that can be delivered or the other way around. Start prototyping as soon as possible with minimal machine learning, that certainly worked for us. Test with as many types of users as you can as soon as possible. And, you know, it's always difficult to manage expectations or prototypes. Maybe what one taught to leave you with is, you know, the scariest thing I found a stakeholder can say is when I talk to a stakeholder and they say, this version is not quite good enough to ship. This is not quite accurate enough to ship yet, but I'm sure it will be later. And whenever I hear that now, alarm bells ring because I'm always like, gosh, why can't we ship it now? We want to ship something simple as soon as we can. We have to hit some accuracy threshold or not yet and that's bad news for the product. And then the last thing is, you know, it's always about how machine learning performance, the performance of your model, interplays with the product envelope. If you carve out a good product envelope, an envelope that fits the machine learning model well, you can cut a lot of corners and you can get away with a lot. And I guess the final optimistic thought is, technology here is like way ahead of application. And, you know, we've so much cool technology to use. I think we're just looking for techniques like this to apply it across lots of domains. So, huge potential to build products here. I look forward to seeing what we see in the next few years. Thanks very much. Thanks, Virgil. So, Sue, you actually are letting AI talk to your customers. Say that again. You actually are letting AI talk directly to your customers. Yeah, we're letting AI talk directly to our customers and that's something that Intercom was reluctant to do for a long time. But we feel the technology is there now Answerbot very carefully selects which questions to come in for, which is why we had to build ourselves. Excellent. I really like the journey you went on and the rapid prototyping. It's probably a really good example of how to do this, how to do a product in the right way. So, thank you so much. Thanks. Thanks very much, Grona. I'll take the... Well done. Thank you.