 Um, very full room. Hi everybody. So my name is Charlie Mosco. This is actually my first DrupalCon, which I'm super excited about. Um, so I work at my planet, and I've been working on Drupal projects for about seven-ish years now. So I've been benefiting from a lot of the work that the Drupal community has put out. And I've, um, I'm not actually a developer. I've been working in a product management role. So this has been my first opportunity to actually contribute back something that I've learned, which is pretty exciting for me. Um, so what I'm going to run through with you is a piece of conceptual work that my team has been doing over the course of the last few months. That's very, very cool. Um, where essentially we have decided to look at how we can apply some of the emerging AI technologies in different ways. Um, in the context of content heavy websites uh, like the kind that I'm sure many of you built. Um, specifically we're looking at are there new interesting ways that we can learn things about our users in order to tailor their experience to their needs. Um, so one way we're looking at doing that is through uh, recommendations. So we're going to try and build the world's fanciest recommendation engine. Um, and the main thing we're actually going to be looking at today is how people make decisions, how people choose the kinds of things they like, how people choose the kinds of things they need, and how we as people strike a balance between those different things. Um, and we're going to be looking at how we can use machine learning. Both some off the shelf options and some role your own to deliver those kinds of experiences to users. Um, so I'll talk about what we did. I'll then go through why it's really hard and may not work right now but could probably work pretty soon. Um, and hopefully at the end of this presentation you would have basically a two to three week head start over where you would be if you just tried to pick this stuff up from scratch. So the case study I'll be walking you through is a part of a series of case studies we've been doing. Um, and it is looking at how we can reduce the burden of choice on, uh, on people in terms of decisions that they're making. In this case, fashion decisions. Um, we've been trying this on a lot of different kinds of decisions on decisions about where to travel, on decisions about, uh, where to go to school. Um, I'm going to focus on fashion and I'll get to why in just a second. Uh, but for a little disclaimer, not really a Drupal expert. I've worked on a lot of Drupal projects but if any real Drupal questions come up, throw them at Jeff with the camera. Um, I'm also not really a machine learning expert. My educational background is in developmental psychology and computer science. So I definitely roll in those circles but I've not, uh, been applying it heavily. Uh, and definitely not a fashion expert. Um, so we were leaning on, um, a lot of people with expertise in industry for this project, not me. Um, so if I am not a fashion expert, why fashion for this particular case study, fashion is a really, really cool thing. Um, where it is primarily an emotional choice where you see an article of clothing and you make a decision about whether you like it or not. Um, and that's sort of your primary guidance on whether or not you're going to go ahead and purchase that thing. Price will play into it, size will play into it, like with anything there's a bunch of restricting factors but it's primarily a means of preferences more than requirements, which makes it a really, really interesting thing to try and search for and a very interesting thing to try and recommend. Um, it's also a thing that people really struggle to describe. So this, we talked to a lot of people who do fashion recommendations in department stores. Um, and in higher end boutiques as a way to sort of suss this out, it's very difficult to get someone to describe what they are actually asking for. Um, and a lot of that is because fashion is very relative. What is formal to a developer might be casual to a banker. Um, and also it's very culturally relative for different people of different ages and different cultural backgrounds are into very different things. Um, and what makes it cool is that apparel on the sort of sliding scale in terms of how we make decisions skews emotional but still with a bunch of requirements thrown in. So for example when you're shopping for something like jewelry, uh, or entertainment, music or video products, it's almost exclusively an emotional decision where the users are thinking about what they like or in some cases what their significant other might like. Um, at the far end of the spectrum on the other side we have things like insurance and education where you're going to buy what suits your needs the best and then we have this wide scope of things in the middle where users have some wants and some needs and we're going to try and fit those together. So apparel skews emotional, um, but it's very much in the middle of the spectrum and some fantastic recommendation work has already been done kind of across the board here where a lot of the big players in the media industry have been doing a lot of heavy lifting on emotion based recommendations for a long time. Spotify is very good at figuring out what your tastes in music are and recommending things. Uh, Netflix has been doing the same thing with video for a long time and now they're really cranking it up to the next level where they're actually deciding what content to produce based on what their viewers would like. Um, and then, uh, Amazon has clearly been able to show you products that are like the things you need for a very long time. We've done a lot of great work, uh, with Drupal Commerce historically on some of our e-com stores and we're now playing around with Aquia Lift a lot which is also showing a lot of promise and the question for us now becomes for things that sit in the middle where we want to be able to recommend things that a user will both like and that suits their gamut of needs, how do we do that? Um, few more little last words on fashion before I stop talking about it and start talking about AI, I promise. Um, we interviewed a lot of people about how they dress and most of the feedback we got back was along these lines where it's a representation of their self so it's a very personal choice. And there's some additional complicating factors that make this particular challenge a lot of fun. One is that trends change over time. Uh, so something maybe very stylish today may not be in the future so any sort of learning model that we're building here needs to evolve over time as fashion senses do and as trends come and go. Um, and as I mentioned earlier the descriptors are relative so what one person says is casual may not be universally casual, one person says is chic, may not be universally chic. This last bullet point that people want to stand out while fitting in is actually sort of the crux of how we managed to pull off some of this AI implementation. Um, so if you look at a user's digital persona, all the things they post online, you get a very good sense of who they are, you also get a very good sense of the kinds of circles they roll in which we can then use to learn a lot about that person. Um, so our goals with this project is we wanted to see what we could do to gain a better understanding of user's preferences so by looking at some portion of someone's online persona, the kinds of things they post to assorted social medias, what can we learn about them. Um, and then we wanted to provide some of our retail customers ways to better serve their customer needs, to make better recommendations, to do things that might surprise or delight their customers. Um, and the way we were looking at doing this is by applying some new AI tools and we sort of had two tracks going at the same time so we were rolling our own neural networks and spinning up our own TensorFlow instances which was a bunch of fun. Um, but we also decided that to accelerate the process we would lean on some of the existing AI tools out in the wild just to sort of jump start the process and let us try things without having to have a full infrastructure already built out. Um, which by the way I highly recommend. To get started with any of these platforms is a couple hours to days of work and to get them to start producing useful things is also pretty quick. Um, although I will also get to later why some of these are kind of problematic in production, especially in an open source context. The reason why I'm talking about our fashion example and we have a bunch of great ones is there's actually a really incredible data set that you can use to play with this yourself. Um, Liu et al did some fantastic work for the IEEE COCV last year where they did some great pre-work on actually building out a fashion identifier. Um, so they've built some really incredible tools that can look at a picture and tell you what kinds of clothing are in it. Um, they can for example tell the difference between a maxi dress and a romper, which is something I can't do. So this went a long way and the most important thing is that they then decided to share their data set for non-commercial uses. Um, so if you want an awesome data set to play around with, I recommend these guys. You can download the whole thing. Oh, and it's pre-tagged, um, which is really helpful as well. So they've put it through a bunch of their learning already so you know what is in each of those pictures. So here's what we were trying to do. Um, we wanted the user to be able to state a goal, like I am looking for a dress. Um, then we wanted to grab a social media input from them and we tried a bunch. So we would get access to their Instagram feed or look at a particular Pinterest board or get access to some of their recent Facebook photos. We would then hit those images with a computer vision AI. So we're going to do some magic under the hood. We tried a bunch of different implementations that I'll get too shortly. And then magically at the end it would spit out some fashion recommendations. Um, and what we did was essentially validated this with two kinds of groups of people, people who would be shopping for fashion and see whether or not they liked what was put in from the system. Uh, and also with people who do this for a living. So is this the kind of thing you would recommend to this kind of person with, uh, people whose job it is at a retail store to make these kinds of recommendations. What we built ended up looking something like this. Uh, so the user would input a statement. They would then get prompted to connect to one of an assortment of social media feeds. This is the Pinterest one. Um, it's actually pretty easy to get started with these APIs. Both Instagram and Pinterest make it relatively easy to get at the user's photos if you give them the permission to do so. Uh, if they give you permission to do so. Um, and then we had basically a set of 30 to 100 images per user. And we're going to try and recommend things for them to buy based on things that they've uploaded online in the past. So how do we go about doing that? We tried three things, uh, with very varying levels of success. So the first thing we wanted to try to look something like this. We would get the most recent 30 images from someone's Instagram feed, which is what Instagram, uh, provides sort of their standard API level access to. We would then run them through an, uh, a visual API. In this case we used the Clarify API and it would give us back a series of tags. We would then basically input those tags as a search into our existing series of products that we had for the user as if the user had searched themselves for those things. And then we would spit out the recommendations to the user, here are some things that you can buy. Um, here's what that looks like. So these results are from Clarify, which tends to be sort of the current front runner in terms of these things. There's actually a great, um, let's see if I can pull this up here. There's some great comparison work that's been done on, um, sort of all of the different APIs and who has the best outputs. It's extremely variable for different kinds of visual recognition. Um, where's my corgi? I like that one. Uh, there's a collie. It's pretty good. Um, so this compares recognition, Microsoft's Open API, IBM's Watson API, Google view, Cloud site, and Clarify. Um, and depending on the kind of thing you're going for, it's sort of variable who has the best data set, kind of based on who has used that API historically, because these things are learning as users upload images, so whoever's uploaded the most images, whoever's had the most images uploaded is the best. Um, so that is the, we decided to go with, we tried a bunch, but we decided to go with Clarify for this tagging process. Um, and even with the best of the bunch, on this image as an example, what the Clarify API spat back to us was fashion urban city auto woman sunglasses sweater, which is a pretty good description of what's in that photo, um, but functionally useless in terms of making recommendations. The fact that someone has bought a sweater does not mean that they will buy another sweater, and it definitely does not mean that out of our series of 800,000 images of sweaters, we can pick the right one for them. It's time to try something else. Um, so another service that a lot of these public APIs offer, uh, is the ability to do direct image to image visual comparison. Um, so you feed it an image, and it gives you an image with similar characteristics from your dataset along with how similar it believes those images to be. Um, disclaimer, we didn't think this would work going in. It sounds like a pretty bad idea, but we wanted to try it anyway. And here is basically what you come back with. So you can see how there are similarities in these images. They're both people, they're both wearing white and beige. Um, but beyond that, if you were to actually try and make recommendations based on the output from this sort of simple, untrained what you get back from IBM without any qualification, also not useful. Um, so then a designer on our team showed me something really interesting. Um, so this is British millennial fashion, juggernaut, shop. Um, and they have built a style quiz on their website, which is basically attempting to get the users of their site to describe what kind of thing they're looking for. On this page you basically say whether you're more of a Beyonce posh glam style, or a Florence Welch eccentric cool look. Um, I think we can all agree that I am a Beyonce. So we decided maybe there's a way to do this without having to have the user automatically define themselves. Um, and what we came up with was something like this. So we were going to pull a user's Pinterest feed. That one actually came from feedback from people we were testing these things with who said uh, you know, my Instagram really has a lot of personal stuff on it. I may not want to share that with you, but I have this Pinterest feed of dresses I really like. Maybe you can look at that and then make a dress suggestion. That sounded like a good idea. Um, so we went with that. We then did a visual pre-screen for noise. So we hit basically every image with the Clarify API once, and only processed images that had people or clothing in them. Um, which is something that these APIs are really good at telling you. And then we did a visual corpus comparison to make recommendations. Um, so here's where it gets fun. We had our big data set of images. Some of them were pre-tagged. And what we essentially did was we started to build up knowledge of some fashion terms and descriptors that can apply to specific kinds of entities. So we tried to basically teach the Watson API what Cheek looks like, and what Glamorous looks like, and what Sporty looks like. And we had about 30 of these descriptors, which we had enough of a data set going in that we could hit the API and sort of teach what kind of thing we were going for. Um, and what that does is it essentially builds up a corpus of knowledge around what a specific kind of pattern is. Uh, we get back to Creepy Cats again, which is always fun. Um, this is sort of a famous image from 2013 where Google decided to have its visual engine, its visual recognition engine, sort of spit back out what it had learned. Um, this is what Google thinks a cat looks like. This is the prototypical corpus cat. Um, you can see it looks kind of cat-like maybe. Um, but this is sort of the visual representation of cat, and what we did was essentially train the Watson API on what a visual depiction of several different kinds of fashion were. We then validated those with some scraping of some social media sites. So our idea of what was Glamorous basically matched what people on Instagram had tagged with Glamorous in the last few months and that sort of thing. Um, and then we were ready to try and use that based on these Pinterest inputs to make some suggestions. And here's how this tends to work. So we, uh, IBM Watson has a similarity beta, uh, which is a visual API that they're getting close to publishing, which does basically exactly this. You train it up on separate corpuses of knowledge and then you give it a new series of photos. It compares series A with series B and tells you how similar those photos you've uploaded are to different entities from your preset series of knowledge. So in this case, we've added just this single image to make it a bit clearer what was going on. Um, but what we got back is from this one image, you can see that this is considered primarily casual fashion with elements of sporty, chic, and classic. I probably could have told you that a bit, um, where you can see it's a semi-casual dress. She has the sporty shoes going on, the sweater might be a bit of a classic look. We did test these sorts of input output chains, um, with people who worked in department stores and also with some friends who we had who we thought were just relatively fashionable people. And the descriptors came back pretty good. So what we have here is actually um, at least nominally functional. Um, to the point where we can use this as the basis to start making recommendations. So if we compare what, uh, different articles of clothing within the, uh, the store we're working with or selling look like and compare sort of how they line up with these indicators to the inputs from a social media feed, we can start to make pretty useful predictions. Um, and also interestingly predict things that a user may not have thought they wanted. Um, so this is where we're starting to bring back a little bit of the human touch to these recommendations that you can get in a store where someone goes, well, if you like that sort of thing, maybe you'll like this kind of item, even if it wasn't directly what that person was asking for. Um, so this is really cool. Um, and we've now been putting this in front of some of the, the commerce merchants who we work with and asking if they would like this sort of thing. Uh, and the response has been relatively positive. So to run through what we learned so you don't have to learn it yourselves, um, there's some good news and some bad news. So these APIs are pretty awesome. Um, you can, as opposed to spinning up your own neural networks on your own hardware or buying some pretty expensive cloud hardware to do all this processing for you, hit some of these public APIs to validate the concepts and then either keep using them or do it yourself subsequently. The, the barrier to entry on this is really low. Um, there are a lot of pretty nice JSON APIs. So as long as you can, uh, feed in JSON and images and deal with JSON coming back at you, you should be good to go. Um, it's really good to validate these concepts. And another important one, and this is sort of the classic machine learning learning, is that the further away you move from something that's human readable, the better the results tend to be. So when we were asking for tags back from the machine where we could see what was being searched for, it didn't work so well when we let the visual to visual algorithms more just do their thing, we tended to get significantly better results. Um, so some bad news, which would make this application not super useful in a lot of, uh, actual production deployments. In fashion, a lot of people do their selection by merchant as opposed to by item. So people choose where they're going to shop as a way that matches their personality versus, um, the actual items that they're picking in stores. So, uh, this could almost be a good tool for a fashion store recommender as opposed to a fashion item recommender. Um, the biggest barrier to deploying something like this in the wild is that if you're going to use social media images as an input, they're quite accessible, but the terms on them are quite restrictive. Um, and on the flip side, if you're using these public machine learning APIs, the terms tend to be pretty open in terms of what those services can do with the images that you've uploaded. So there's, uh, there's a pretty significant legal challenge in terms of pairing up the source of the images that you're using with whatever you're using to do the processing. If you roll your own, that's less of a problem, but you still have to make sure that you're compliant with, uh, all of these social media APIs. And some of them are intentionally restrictive because this is the kind of thing that a Pinterest, for example, wants to be able to do themselves and they need to maintain their strategic advantage. So a lot of the times are kind of hard to get at. Um, this stuff is still expensive today. So to hit these public APIs, you're looking at anywhere from half a cent to five cents per call. Um, and that really adds up both in the training stage when you're hitting it with hundreds of thousands of images to build up corpuses of knowledge, but then also on the learning side, if you have to feed many images in, it's expensive to get, uh, it's expensive per user, where you're looking at cents per user to run the system. So you need to have a pretty good return in order to make it work. If you roll your own, it's also still expensive because at this point that typically involves powering some GPUs, one night we accidentally left an AWS GPU running all night and came back to like a $600 bill in the morning just for the one computer. So right now the costs are high, but with Moore's law they should come down, we hope. Um, and then we'd be able to do these things all the time. Um, and there's also a significant challenge right now making, um, these sorts of systems mesh with an open source mentality where a lot of image data is very closely protected both by the producers of that information and also by the different repositories where it's stored. And there isn't really a concept of open data today. Um, so there, there aren't really collaborative, uh, pools that we can use for training. Like we couldn't actually deploy this directly into production because our dataset had a strictly non-commercial license on it. Um, so you're either working with someone who has a big pool of data already like a fashion retailer with a whole bunch of tagged images um, or we need some way to collaboratively source these things which is more difficult. Um, so key advice, you have to sort of trust the systems. The more you try and monitor them the harder it is to actually get anything back. And things for all of you to pick up if you're interested. Um, one thing we're looking at is sort of living corpuses of knowledge. So things that evolve over time where our definition of what is stylish uh, may be different in two years than it is now. Um, ways we've looked at adapting that are either datasets that change over time or potentially matching a user's input to a real live human with similar tastes, some kind of taste maker uh, where we could then pull their results. Um, another big next step is machine only trends and patterns. So right now we tried to teach it some human classifiers of fashion, things like chic and sporty. Um, those probably don't mean much so it's possible that if we instead let a smart deep learning system pull out trends on its own it could potentially make even better recommendations even though we would understand them less. Um, and a big one with all these things is sort of reducing the, the ick factor where a lot of people still want a human touch in their sales systems. Um, and we'll need to work on that along with the collaborative dataset piece. So this was a bunch of fun. Uh, I learned a lot and I hope you learned a bit too. So thank you. And I think we do have a few minutes for questions if anyone has questions. Looks like we're good. Thanks guys. Oh wait, one question.