 So this talk is how AI gives your project superpowers. It was written as, it gives your site superpowers, but being that we're all digital experience creators now, not just website creators, this seemed like a better title. Quick reminder, code sprints are on Friday. If you haven't participated, you're interested in participating more in the Drupal community. That is a great way to get some kind of hands-on help get your first core contributed patch over the week. So I'm Frank Currie. I'm the CEO of HeliCore, all those business cards you got around this company that I'm partnered with. I've been a Drupalist for 10 years, so I've been around the block seeing how Drupal and the web has changed over the last 10 years, quite a bit. It's crazy to think like 10 years ago was right around the time. I don't think the iPhone was debuted just yet. I think it was a little bit later in the year. So 10 years ago, there was no iPhone. Something to think about. So 2007 was interesting, but 2012 was a big year too. You guys might remember this movie. I don't remember if I ever saw it. I think it was terrible. But the whole idea was that in 2012, the Mayan calendar was running out and it was going to be some sort of apocalypse. People started taking this pretty seriously. I remember there were some things in the subway of like, come join our cult. So 2012 was a big year for me. I've left my position, lucrative position at Sony Music to go back to school, get a degree in machine learning and AI. And in the middle of my grad program, there's a program called ImageNet. It's a competition where different researchers try to best each other by taking millions of images across thousands of categories. Categories like different types of dogs, cats, a bottle of beer, and classifying them correctly using whatever technique they want, but generally machine learning. So what was big about 2012 was previously, people were using hand-coded techniques. Techniques were used to where you go in and you write some if-else statements. I'm exaggerating a little bit, but that's basically what they were doing. And this was a body of work that had been going on since the 50s to figure out what image features to use, how to segment them, and things like that. But at the end of the day, it effectively was not machine learning, but it was more statistics. They knew how to cut these things up and they'd shown what worked over time, but it was sort of human labor figuring these things out, not machines. And then in 2012, the winner dropped, this is the error rate on the left-hand side, the winner dropped the error rate significantly. So this has been going on for years prior to this and things were kind of plateauing. Humans were only able to improve this thing so much. In 2012, everything you see in purple is this new method. This new method was called deep learning. How many people have not heard of deep learning? Okay, so deep learning is a new methodology that uses a, it's inspired by the brain, let's say. It's not using neurons, but it's inspired by the way the brain works or the little bit we know about it. And it's a technique in machine learning that allows us to deal with very, let's say, noisy, difficult problems that have some sort of hierarchy. And it's proved to be very effective. So in this case, 2012, you can see how much the rate dropped. It dropped from around what, 30% down to 15%. And then each subsequent year, including now, it keeps dropping. What you'll notice is that the human performance is in red. So once we become superhuman at this task, and some of these tasks are hard, where it's like different types of dogs. I'm a dog lover, but I don't know all these different types of dogs. The AI was able to correctly predict the difference between a poodle and a dot sender, something like that. So it continues to be better. So these, I would say are superpowers that we're gaining. When it's better than human, it's, say, a human superpower. But even if it's not a human, and it's just a machine, I think of these as superpowers. Because even if it's not as good as a human at the task, it still can alleviate a lot of work that the humans would have had to do otherwise. So what are these superpowers that I'm talking about? So one thing to keep in mind is that you may already be using machine learning right now. And the goal of my talk is to say, hey, we should be exploring these things. You should be interested in these things because they're gonna become tools that you're gonna be using very soon. But you may already, most of you, how many are familiar with this logo? So it's Malim, right? Malim uses machine learning behind the scenes to figure out what's spam and what's ham. And it does it pretty effectively, and it learns over time. So just the idea that you may already be using some of these tools. So some of these superpowers, you could categorize them in a large swath, like interpreting images, or categorizing images, like we talked about with ImageNet. Natural language processing, so if you're working on some sort of bots and you want people to ask a natural question, they may ask it in a few different ways, and you wanna be able to figure out what are they really asking for it. Being able to listen, so right now, Google has technology where for the hearing impaired, I could speak, it could start typing, and people could use that right away. Speaking, it can generate speech, so you've used a Garmin or something, Google Maps, and the person speaking. That technology is getting much, much, much better than it was just a few years ago. And then translating, the traditional way Google's translating was taking two pieces of text that had been translated by humans and just sort of mapping them one-to-one, roughly. What the new techniques are doing that we'll get into in a minute is using deep learning to actually translate end-to-end. So it's able to translate between languages that it didn't have examples for. So maybe going from Swahili to Hungarian, or maybe that's a very uncommon pairing, but now it's able to do that because it's sort of like learning a meta-language in between. And then content creation and augmentation. I think this is a really exciting space, especially for the Drupal world because we sort of live with content all the time, that we're used to user-generated content, but what about AI-generated content, things like that? So we'll explore all these. So how do we create this? This Frankenstein monster. So we're gonna keep it very simple. We're not gonna dive in. If you wanna dive in, please come to the bof. We'll talk a little bit more in detail. But basically what you want is, you need to collect some data. And what you want is labeled data, meaning that you have sort of, here's a bunch of images, say an image net. Here's what the right category is for that image. So you sort of need this pair, right? The input data that you wanna end up being able to predict in the future and the correct label that it's supposed to be labeled. And when you have the correct labels, you can know when the model goes and predicts something that it predicted it incorrectly. And you can say, how incorrectly did it predict it? And that's how it's learning. It's learning based on just like a child saying, this is a cat and this is a dog, right? And Drupal's great at this because it has all the structured data. So in this case, we've got a set of taxonomy tags. So this is a great example where you could maybe have a site that exists where you, let's say it's articles, news articles, and they're supposed to go in some category on your site. You can use the existing data on your site. The more data you have, the better. But you can use the existing data on your site to actually predict in the future when someone say creates a new article, what is that article supposed to be tagged with? So you can recommend tags that they should consider, for instance. So Drupal's really good at having this data in a structure that we can export. And you may export it through like a SQL query or something like that to get it out into a format that you can use to process by some other system. So you take these two sets of data, right? Your X is your inputs and Y is the, what are the correct outputs? You put it into a model, right? And this is, we're black boxing it. I'm not gonna get into details. But you put it into the model and you do this basically over and over and over again. And the model just like a toddler, just talking with Josh Kay about this, sort of like a toddler learning to walk or learning to speak. It gets it wrong first. It's not like it goes from zero to 100 right away. But it slowly improves over time as it figures these things out. So once you've built your model and you get it to a state where you like, you like how it's working, you're looking at the predictions it's making. It's say, making it 80% is good enough for your use case and you move on and put it into production. So how does this work in production? You can think of it as like a black box behind a REST API. It's probably how you're gonna do this, which Drupal. So you're going to say upload a body of a JSON, throw the body of your node into the model and then you're gonna get back a prediction, maybe here's a bunch of recommended tags. And that's how you would use this for Drupal. Right now, PHP is not a great language for machine learning. Python and Scala and some others are very popular. So what you're probably going to be doing in this case is writing this as a separate, service oriented type thing that you're gonna connect with. And it gets a little bit more complicated, but there's this idea of having different sets of data. So you take your bulk set of data, you're training on that data. You don't want it to just blindly remember the data, which can happen, which is called overfitting. You want it to actually figure out the data in a generalistic way. You want it to generalize about where these categories should be. So you keep aside a subset of your data and then that's the data that you're testing with. So the easiest way to do this is to forget everything I just said, right? And just use something off the shelf, right? So just like Mullum, it's a service that you have to use. You don't have to think about the machine learning going on. So we have face detection. Face detection's been around for a while. It works very well. So we'll move on. That's not that impressive of a superpower. Image classification we talked about. This is an example with Clarify. Clarify uses a deep learning model behind the scenes and they have a lot of categories. And so they come back and it's hard to see with the predictions on the right, but these are almost all in like almost 100% predicted. And we've got things like sunset, dawn, boat, reflection. It even says at the bottom that this is, there's no people in this photo. So that can be helpful. We've got not-safe-for-work photos. There's an interesting documentary on Vimeo that's called The Moderators, which looks at the people, has anyone seen it? It's basically the folks in India that have to look at all this smut that gets uploaded every day at all these sites and that it's really a human-driven thing right now. So with a not-safe-for-work model, you could think about having a lot of your content moderators job be handed off to an AI, but then they don't have to do with the obvious cases of pornography or whatever. They can just handle these sort of in-between case. So in this case, it's predicting sort of 50-50 accuracy between the not-safe-for-work and safe-for-work, which is probably good, right? It's borderline. I don't think Arnold could lose too much clothing here and still be safe-for-work. We can also do this with video. So in this case, it's looking frame-by-frame across this video and it's saying, what are the different categories in those frames? So it's very similar to image prediction, but in this case, we sort of get a continuous line of this. And this can be really useful. You can imagine where, if I'm a media company and I want to find what was the video that we had that had baseball in it? And it may have only been like a two-second clip inside a larger video, but I can search across my videos in this way. This is Watson. So this gives you text categorization. So I would upload an article. In this case, it was an article about sort of how AI is gonna take over everything. And it would give me back predictions on different hierarchies of categories. So in this case, they're, like I said, these are off the shelf. So you could maybe think about mapping some of these categories into your own taxonomy if that was appropriate for your site, or maybe just throwing them out your existing taxonomy and just using the sort of standard stock ones that they provide. Text summarizing, so this can be really useful. We have oftentimes a lot of legacy content, or we've imported importing content from other system. Drupal has a summary field that can be really useful or a pain in the neck, depending on your case. And what we may want to do is even have sort of a dynamic summary where maybe 500 characters was okay for our previous summary, but now we want to have it on Twitter or something like that, and we want to provide a 100 character version of it. This is an example of a site that allows it to do it. So I uploaded the same article that was all about how AI is gonna take over, and it cut it down by 98% and just gave me a one sentence output. And if you can read it, it's very impressive, very accurate, what it cut it down to. We talked about speech to text. There's still running gag of how stupid Siri is that doesn't understand what you're saying. But believe me, we're at say 90, 95% predictability rate and we're getting better and better. So those last like nines, we all work on the web. You know how many nines maybe you have to worry about. The getting those last few inches can be some of the toughest to do. So in this case, I've just read something on the back of a book. I was pretty far away. I didn't have a very good headset or anything. And it picked it up pretty well. Some companies are starting to get smart with the environment. So they're taking their X, their input data. They're actually mixing in the audio with conference noise or outside noise or restaurant noise. And they're actually maybe able to make it handle those situations better because they're able to augment the data. We talked about translation. Let's talk about language detection. So this could be really useful if you say we're taking queries and you want to send it off to solar or elastic search. It might be helpful to know if even if people landed on your .com site and they didn't switch the language that you could detect the language beforehand to automatically provide the right queries. So that could be useful. So those are all kind of off the shelf you are available right now. Here's some ones that are a little bit hybrid. So you're using your own data or at least you can start with their model and then expand on it with your own data but it's still much easier than starting from scratch. In this case, it's a text classification. So if you're making say a chat bot or something like that, people may ask the same question multiple ways and you could sort of just think out or maybe you could look at your existing conversation data or search data and you could sort of hand categorize, hand label, well what are they asking for? Are they asking for movie recommendations or are they asking for product information and you have all these different permutations of the questions that they would ask. You can upload that data that X and Y data we talked about up to the Watson service and it will actually process do the learning for you. So it's this nice sort of a hybrid solution that is also available from the others, Google, et cetera, where you can basically just give them your data and they already have the model and they handle the training and then once it's trained you just hit up a REST API like you would with something else. So this can be handy. Here's an example with image classification. Now these are two different types of dingy sailboats or whatever. I'm not a sail person so they don't make a lot of difference to me. You can see they look very similar. In this case someone uploaded and labeled and what's cool is like because most of the training was done they only had to hand label maybe 10 different examples for each and they made their own custom classifier that said this is a Opti or whatever brand and this is a laser brand. So you can think about like product if you're doing like product related things or maybe you're automatically importing products maybe this would be something handy for you. So we get back to sort of the DIY the harder machine learning do-it-yourself but it opens up a lot of cool things like here's style transfer and I forget if I'm not an Instagram or a Snapchat person but I don't know if they have these things yet but if they don't they will soon. So in this case we're able to take the Mona Lisa and we're able to reimplement it in the style of other things, other paintings. Face detection isn't available as far as I know from a service but it's something that you could implement yourself. It's actually very good. You've noticed Facebook and other programs of how good they are at finding who you are and what faces are what. So this is basically how they're doing it. They find a face, they throw it into a neural net and they get some clustering and they say okay this person's likelihood is between this person, this person, this person and obviously if you have their social graph it makes it a little easier to figure out exactly who it is. Image captions. This is something else we could use with Drupal I think for visually impaired folks. It could be very useful for readers to be able to look at an image and say well what image is this? Give me an example of this caption. So here a man in a black shirt is playing the guitars on the far left and it's not perfect, right? If you look on the right hand side it says boy is doing backflip on a wakeboard. Well he's not on a wakeboard, it's like a trampoline but you can see the water in the background and you could see how it makes sense, right? So it's interesting when it gets it wrong, it gets it wrong in an interesting way, generally. And so you could see using this when people upload to the site it's always a pain to get them to put in the alt text. Maybe you just say screw it, I'm gonna have it, we'll still recommend them to do the alt text but I'll just provide it automatically and that'll cover our accessibility issues. Then we get to generation which I think is really interesting. So we think of user generated content but we don't think necessarily yet of machine generated content. This is an example where what's called a recurrent neural net was given Shakespeare and it just learned on Shakespeare over and over and over again and it was able to create this. Now all of these are words. It sort of gets the idea of IMB pentameter. It even predicts the capitalization for the names of the characters. It knows to put two carriage returns before the next person speaks. It learns how to put the semicolons. So it learns all of this just by learning how to predict one letter in front of it. So you can think of other ways that this could be interesting. Some people are using it for instance for image generation. So how many people have seen this? It's like took over the internet for a day and a half. So you're able to draw a picture of a cat on the left hand side and the model basically does its best to say okay this is what a cat would look like if that drawing was real and it can lead to some of these very scary, strange looking cats. These are actually some two that are, so I would recommend it. I think it's a, it's not cat to cat but something like that. So you can think about this in other ways. It's doing similar things with music where you can sort of give it a little riff or something like that and it's starting to put together other more nuanced predictions of what other pieces or motifs that you could put together. We're doing things with voices. So there's some pretty realistic singing voices that are happening. You can imagine where that could go. Like auto tune on steroids. We don't even need to pay the artist anymore. So here's another one, maybe a little more applicable to Drupal. It allows you to upscale images. So this can be really useful. The displays that we're using are going up and up and up in display size but you have a bunch of these old images and maybe you don't have access to the originals anymore to re-crop them. So in this way, what it does is the original images are on the right hand side, the down sampled images on the left hand side and you can see sort of how bad it gets. And then the one just to the right of it is just sort of averaging across. So you get this blurry image. But if you go one more to the right, so the third from the left, that is what it predicts. So it's able to go from an original image and you can see there's some differences, right? But it's predicting the reasonable outputs for these. So you could see this used for good for just say, okay, this movie or whatever was in VHS when it was originally made and we want to upscale it and make it look better. You could also see some sort of nefarious things happening, right? Where you see an image and it looks realistic, but maybe it's been altered in some interesting ways. So here's an example of something I did. We're often used to using search, solar, et cetera. This is implementation of an image search. So what I did was fed about 200,000 images from Yelp through a system and then got some features and then I was able to search by uploading a new image and saying, what are the closest images based on the image that I uploaded? So you could do this in Drupal with the help of some AI. So going beyond where we're at now, there's this idea that we are developers, we're used to constructing things, putting together if statements, where the AI is really different mode of thinking. You're growing something. So you may be tweaking the nutrients that you're giving it. You're maybe breeding two things together. And so it's a different mindset and I think it's a helpful mindset for us to start getting into now to at least know it's available, maybe start implementing those things on our sites and so that we're ready for the next things to come, which could be significant. So again, we're in the machine learning bof is gonna be today at 5 p.m. in room 312. Hope to see you there. If you wanna go into some more detail, we could talk more about like what is really going on with deep learning and so yeah. Follow me on FrankKerry and don't forget to do the session review. I'm sure you're getting that from everyone. So thank you. So we've got a few minutes. Any questions? There's a mic here too. I think we'll try to record it. I think that's the idea. Oh hi. So I was just wondering what... And don't go anywhere. There's another good talk right after this. All right. What kind of tools would you recommend that connect to Drupal the best? Would it be like some deep learning Java thing? As far as I know, there's sort of two types of modules. There's like specific modules that maybe work with Google's image search for instance. But there's no sort of like search API and that's something that I'm actually interested in talking to some people about at the BOF of being able to sort of create a standard API to do some of these predictions especially. Okay. Thank you. Could you both describe an average data set that one might be able to learn off of and then also what computational power like let's say in hours an average computer would take to process that data? Sure. So the first question was like what does a typical data set look like? And the second was sort of how long does it take? So a typical data set, you're gonna want quite a few instances. So if you've got a website that has five pieces of content it's very hard to learn from those, right? Just like a kid. If you only show them five pictures of a dog they might not know what a Chihuahua is because they've never seen something like that. So the more data the better. And the types of data you're looking at images, a lot of text data, you can get into more nuanced things but this sort of text and image stuff seems to be the most common. And how long does it take to train? You can train a model in as quick as, I'm talking like half a million images or something like that. You can train it pretty quickly. What's interesting, what helps to have is like a GPU. Big GPU helps. It maybe will save you like a fifth of the time. But it's not necessary. You could run this on your laptop. It just might take a little longer. So it depends on the model really. It's a very wide gambit. So the ImageNet competition, those guys were training it for a week, like 24 seven. What's nice is there's a thing called transfer learning where let's say I wanna do something image related. I can actually just steal their model and you can think of it as figuring out this hierarchical representation of the images. So low level, it's like edges and lines and circles. And then it starts to get more high level as you go sort of up the stack of the deep part of the deep learning. And with that you can train a new model very quickly with much less data as well. Because it's sort of already figured out vision. It just needs to figure out boat versus sail, you know, different types of sailboats like you saw. In that case, I think they did with like 10 examples. It's pretty impressive. There are some good course materials available for getting started. But I was curious if you could recommend some books where people would say, particularly people that have math stats background, what it takes to get from there to a working knowledge of the speed of learning algorithms. Sure, and there's quite a few. So like deep learning is its own sort of niche, but there are other more traditional learning algorithms that are available. I posted, if my buffer thing worked correctly, I scheduled it to post something to my Twitter account at Frank Carry. And that should, it's a blog post that I just wrote this morning with like some additional material, blogs that you could read. Some of the companies that we mentioned, some of the things that we mentioned in the slides. So if you go to at Frank Carry, I'm sure if anyone's gone there and doesn't see it, let me know. But it should be there now. It looks like deep learning is very different from the development, but what are debugging techniques are there? They're not easy. So it's just like, how would you debug a four year old learning to walk? Or I guess younger than four, hopefully. It's tricky, right? Just to think about it, it's tricky. Like maybe there, you come up with some hypotheses just like you do with regular debugging. You say maybe they're not getting the right information. Maybe their balance is off or something, right? You start to figure out why, why aren't they learning when I think they should or something like that? Maybe their brain is wrong or something, right? I mean, see, the trick is though, because you don't have like a traditional debugger, right? Or you can step through the code. But some of that's changing. Like the, if you use a framework like TensorFlow, I think they just released like an actual debugger recently, but other things you can look at are sort of as it's learning, you think of as like, all these neurons and you can see the levels of the neurons, like what values they are. And if they all go to zero, then something's wrong. If they go to a billion, then something else is wrong. So you can sort of see the results over time. And when you're doing these called epochs, you're learning, learning, learning, what you'll see is like you're getting better and better and better. And based on how, maybe you're getting better and better and better, but you're not learning, like you're not generalizing, then you're overfitting. So you start to learn these techniques. And unfortunately, they're like nothing like traditional development. So it's not, it's not trivial to jump into it, but it's also not that scary. I mean, I guarantee you, if you have reasonable programming skills, you could get a deep neural net running in an hour on your laptop with a blog post. Thank you. Yeah. Do you want to jump up and I'll field a couple more questions while you're, you start in two minutes, right?