 So, welcome to not hot dog, hot dog, or hot dog, not hot dog, artificial intelligence with Drupal. My name is Rob Loach. What we'll be talking about today, we'll be talking about artificial intelligence, we'll go through machine learning, computer vision, Google Cloud Vision API, and then get into some demos and then wrap it all up. But before we get into that, a little bit about me. I work with Kalamuna. I've been around the Drupal verse for about 12 or so years maybe. My mother found an essay that I had written back in 1999, and I thought I'd post it up here. This is about what I'd like to see on a website. Much of it has changed over the past few years since 1999. You can see me in my dorm room there, along with working with Jeff Noyes from Acquia. And today, I'm working with Kalamuna. Kalamuna has a focus on making the world a better place. We work with mission-driven organizations to transform their digital presence. This includes Fair Trade, USA, Linux Foundation, the American Foundation for the Blind. All of these organizations are trying to look at the world and see how it can be improved. We're based in Oakland, but we are all over the place. I myself am in Toronto. There's people in New York. One of my co-workers is in Romania, so it makes coordinating stand-ups. It can be a challenge sometimes, but we make it happen. But today, we will be talking about artificial intelligence. Ever since computers have been a thing, we've been wanting to imbue them with artificial intelligence. Artificial intelligence is our way of mimicking human behavior within a computer. That is what makes speech recognized by your phone. It allows computers to drive cars, Tesla. It even allows you to talk to it, ask it questions, give you recipes in the morning. Wish you a happy good night. It's also the reason why people are questioning if this is a good thing or whether it's heading to judgment day. We are recreating human behavior. Ethics has always changed over time, and artificial intelligence has dramatically questioned what humanity actually means. It is all done through machine learning. Machine learning is a subset of artificial intelligence, and it is the most exciting part of why it exists today. By allowing computers to solve their own problems, artificial intelligence is the breakthrough that brought it to where it is today, all through machine learning. Machine learning is all handled through neural networks. Neural networks, neural net models, learn through example. By training a neural net, you provide it a whole bunch of labeled examples, and these labeled examples will recognize patterns and then pass those patterns into recognizable outputs. When you pass it a unlabeled example, it will look at the patterns that it recognizes through the neural net and then provide understandable results with context. This is all done through statistical analysis. One way of thinking about it is if at first you don't succeed, try a billion more times. Artificial intelligence was used to beat one of the most incredible go players in the world recently. What I found interesting about this was at one point during the match, the artificial intelligence, the AI, made a move on the go board that made no sense to any of the professional players there, but the AI ended up stomping on the world's most incredible go players. I thought that was really, really cool. Yeah, the subset of artificial intelligence we'll be talking about today is computer vision. I see some people standing up in the back there, so feel free to come up and take a seat, get comfortable. There's a bunch of seats on the right, right at the front as well, thanks so much. This is computer vision. It uses machine learning to recognize images and give context behind these images. Realtor.com uses a mobile app. You can take a picture of a real estate listing and it will give you information about the listing. This isn't a QR code that it recognizes. This is reading the listing and then providing you information directly on your phone. Your bank likely has an automatic deposit app. You can take a picture of a check deposited into your bank account. Very cool stuff. Patch.com also uses machine learning. They use artificial intelligence to curate some of the content that is posted on their site. Patch is a community-driven website, so they have a lot of content. There has to be a way to manage and curate that content. Patch uses artificial intelligence for that. Tumblr has been in the news recently as well. They did something similar. They engaged artificial intelligence computer vision to curate content on their website, ending up removing a whole bunch of content that was probably not meant to be removed. So they got a lot of publicity around that. Pretty interesting stuff. Tesla as well and Boeing. People have found out ways to trick some of the computer vision behind the Tesla cars, leading them into an incorrect direction. And Boeing as well has been in the news about their reliance on automated systems. We are talking about people's lives. People's lives are at stake because of this. So we have to decide where our reliance lies and how we use it. It's not all doom and gloom though. This is a screenshot of Google Photos. Google Photos looks at people's faces that you take on your phone and sticks them in a photo album. You can see all of these incredible human beings. It creates photo albums for all of them. One of the most interesting things that I found about this is that it was also able to recognize my friend's dog's butt. So if you want a photo album of my friend's dog's butt, it's right there. But how does this work on applications that we work on today? How can we make use of it? This is all done through a Google Cloud Vision API. This is one example. There's many different examples out there, but we'll be talking about Google Cloud API today. Google Cloud API has a whole bunch of different features. Some of them include label detection. It's able to give context into items that it recognizes in photos that you provide. It also goes through face detection, gives you ideas into where the eyes line up. It gives you information about where the nose is on a picture. Optical character recognition. If there's text on the image, it'll provide that. Explicit content detection. This is what Patch used to make sure that no adult content gets uploaded onto their site. Landmark detection. If there's the Eiffel Tower, it'll recognize the Eiffel Tower. We're in Seattle right now. If there's a space needle in a photo, it'll say, hey, there's a space needle. Logo detection. It'll likely recognize the DrupalCon logo, Disney logo. It'll tell you if that's in the photo. If we're to try this out today, you can just visit cloud.google.com. And upload a photo directly onto the website, and it'll give you the results. So let's experiment with this a bit. Does anyone know who this gentleman is? Yeah. Yeah? So one person does. Does anyone else know who this is? Yes. So who is it? Alan Turing. Alan Turing. Science. Yes. And largely thought of as the godfather of artificial intelligence. I thought Turing machines on my nails. Turing machines on the nails. Wow. Incredible. She wins the prize at the end. So let's pass Alan Turing through Google Cloud Vision. You can see that it not only recognizes where Alan Turing's eyes are, his mouth, his nose, but it also gives some emotional status into Alan Turing. You can see he's quite joyful in this picture. And he's not wearing any headwear, which I found interesting. One thing to note is that you'll see a confidence meter at the bottom there. It shows that it is 100% confident that this image provides joy. Very cool stuff. Does anyone know who these three boys are? No? This is my brothers and I. I am on the right-hand side. You can tell where Teenage Mutant Ninja Turtles fans. Yeah, this is a while ago, though. When we pass this through Google Cloud Vision, we see that there is some adult content here. That is probably because our shirts are off. It's very racy. It also says very likely to not be violent, which I have concerns about because I can tell you there was much violence amongst my brothers and I. There's also, this demonstrates some of the landmark detection. This is in Toronto where I'm from. This is the CN Tower. You can see when we upload it to Google Cloud Vision, there is a high confidence that the CN Tower is there, along with Scotiabank Arena. It detects landmarks and provides more information about what is in the picture through the web. Outside of that, there's a JSON feed. This is all through web APIs, so this is a JSON feed that it provides. There's a whole bunch of crazy information that it gives you when you upload any kind of image here. It's quite fun just to read through this JSON API and see what you can find. There is more information that it gives. It provides crop hinting as well. Just an incredible wealth of information that it can gleam from these images, so I highly recommend you check it out. Many people consider this somewhat of a black box, and it kind of is. We have no control over how these neural networks detect patterns in the image. We just give it some kind of input, and it will tell us what it thinks. How we use that data is up to us. All right, so let's get into some of the demos and see how we can use this. The first demo I have for you today is a PHP application. This is hotdog, not hotdog. The joke behind hotdog, not hotdog is from Jin Yang on Silicon Valley. Jin Yang, has anyone watched Silicon Valley? Yeah, so Jin Yang, he developed this not hotdog web application. And the joke is that it ended up being better at recognizing something that kind of resembles a hotdog, so his application ends up being bought out and he becomes super famous. So let's see how this works using Google Cloud Vision API. So I'll find a hotdog image here and upload it. And now, when I upload this image, it goes through this PHP application and it grabs the data from the image and uploads it to Google Cloud Vision. Google Cloud Vision responds back to the PHP application and gives us the result. In this case, it is 93% confident that this is a hotdog. But what happens when we upload something that is not a hotdog? What does that look like? Let's upload Alan Turing. There we go, so this is not a hotdog. Alan Turing is not a hotdog. It's 83% confident that Alan Turing is a man. And you can also see that it found that there's a tie that he's wearing as well. Pretty cool, pretty cool stuff. Let's try another hotdog just for fun. How about this one? Let's be a hotdog. This is apparently not a hotdog. So this is a good case. So while this is a dancing hotdog, it's more confident that there is a footwear and that it's a hotdog person. So this is a good example that there can be false positives and inaccurate responses from Google Vision API. Another demo, how does this work with Drupal? We're at DrupalCon today. So how can we use this with a Drupal website? So this is Catogram. Many of my friends upload pictures of their cats to Instagram. But I want just a feed of just cats. I want to see what's going on with my friends' cats. So here is Bobo, my brother's cat. Yeah, so yeah, pretty awesome cats, Lola and Elliot. When we go to upload a cat, I'll grab this cat here. So you can see we can grab a cat and there's a cat. So fantastic I can upload a cat. But what if we upload something that is somewhat inappropriate? Like I can't trust my friends to just upload cats. If I grab, here we go. So this is the Statue of David. When we try to upload the Statue of David, you'll see that it sends the image to Google Vision API and responds back, resulting in racy content. This means that I can curate my content and ensure that there's no adult content being uploaded to my website. So when my friends try to upload their not cat pictures, I can be sure that there's some good images being uploaded. We can go even further with this though. You'll see if we go to the next page of Catergram on page two. You'll see that there's a dog on here. Like why is there a dog on Catergram? Yeah, I'm asking that question myself. This is something that we could do with Google Vision API. We can check what the context is that it provides and see if there's a dog or a cat and just make sure that only cats get uploaded. Very cool stuff. So how was this done on our Drupal website there? This was all done through a Drupal module that we had put together. This was a partnership put together through Pantheon in patch.com. Pantheon reached out to us, was like, hey, there's this new Google Cloud Vision API, what can we do with it? Let's explore some possibilities. So we curated a Drupal module, making use of it, and had patch.com tested out. We added an umami, and we added a Drupal module. We added an umami demo install, curating recipe websites. So when people uploaded a recipe, no adult content was uploaded. You can see here, when we go and add a recipe, it will allow images that are not racy onto the site. And then when images are uploaded that are racy, it goes through a content moderation system. There you can see we get a racy content when we upload the racy image. Since this is all done through Drupal, we could leverage content moderation. So content moderators can go through the content workflow and approve images which were deemed as racy. Yeah, so suspect images can be approved by the content moderators. This module is open source. Yeah, check it out. It's at google.com slash kalamune slash google underscore cloud underscore vision. We hope to do a few things to add to it. Because right now, when you set up, there's a configuration page. All you have to do is set a key name for your Google vision project ID. And on the, it integrates with the key module as well. So you provide your key to the module. The key module will grab this JSON API key from Google Vision API and then have it connect. How do you get that JSON API key though? It is all done on the cloud platform. When you visit the cloud platform, you can enable the Vision API and then grab that JSON API. You go through and grab the service account key when you log in. This JSON file is just a straight up file. So you can grab the data from the JSON file and upload it to the key module. There is a, since this is a Google API, there is a pricing model behind this. But you can see that it's actually pretty reasonable. I believe the first month is also free too. So it's good to try out and see how it works with your users. It is just the beginning too. You saw that we checked on racy content and patch.com is using it. But there is a lot more that we could do to this module. It would be really cool to see it turn into an API module where you could hook into it and make use of it on your Drupal site. I would love for this to be used in the accessibility space when people upload images, detect what is on the image, and then inject the alt text so that content editors are not confused about what to write into alt text. It would be really neat to see it used for taxonomy as well. Automatically tag images. I mentioned explicit content detection, content curation as well. One of the things that it also outputs in this JSON feed is suggested crop hints. So you could upload an image and have Google Vision API automatically crop the image based on its cropping suggestions. So in conclusion, where does this all bring us today? Yeah, artificial intelligence has drastically revolutionized how we think about solving problems. It recognizes our faces. It understands when we're happy and we're sad. It can drive our cars. It can fly our drones. It's talked at the UN and it even beats us at Jeopardy, Go and Chess. Some people think that this is very scary and it can be. We are changing how we think about human behavior. We're recreating AI in our own image and it is now an extension of ourselves. It is an opportunity. Let's figure out how we can use this and make the lives of everyone around us much better. But first, before we do that, hey, Google, are you going to destroy all of humanity? Help, and that's the opposite of helping. You can count me out. All right. Thanks so much. So we have about nine-ish minutes available for questions and comments, ideas. I would love to hear how you're using AI and how you think it could be put to good use. Yes? So when Google gets it wrong, is there a way to provide feedback and say no? Yes, that is a cat. That is a hot dog. David is not racy. I'm sure there is a way to get in touch with the Google Brain team and provide that feedback, but all of these false positives are just results of what it sees in the image. All it can do is recognize patterns and then output that result. So you have to be deterministic on how you act on what it provides. Thanks. Great question. So I'm not using AI specifically, but I am curious. When Google Cloud Vision gets a new image, does it then learn that image for future reference, or is it just looking in its existing database to see if there's a match? That's a great question, and I'm honestly not entirely sure. Sure. It recognizes all of these patterns in the images and bases it off of its neural network. Whether or not it houses that information, that is beyond me. And something that I would certainly be curious about. Would love to see how that works. Hi, thanks. I was curious. Like, obviously the centralized services are going to have the best trained AIs available, but I was wondering if you knew of any models that are being regularly produced that can perform edge computation of AI? Like, I know that a lot of models are being exported for use directly on phones, so that you can use them disconnected and stuff. Yeah, so I mostly talked about Google Cloud Platform. Which is available to use as an API. But there are other neural networks out there that you can take advantage of. I've listed two up here. TensorFlow is a very popular open source neural network that you can implement. And OpenAI also has some very interesting use cases of this. Yeah, so definitely check them out. I would love to see a TensorFlow Drupal module. That's definitely on my wish list. Yes, another question. Do you know of any APIs that recognize text? One of my projects requires me to take the text and have it automatically tag the content, which clearly I haven't figured out. Yes, yes, great question. So recognizing text and giving context behind content that is uploaded, like a paragraph of text. Yeah, so there's a, the Google Cloud Platform also has a, I think it's called the natural language API, and that will take a piece of content and read it and output keywords to that. And there's also a really cool demo with four kitchens right now. You should go check it out. You give them a moment in your past, and then it looks at the content, and then generates images based on the content that you uploaded. And we'll be talking about that more in depth at our BOF, at 230, BOF Room 4. Yeah, so that's today. Yeah, so check that out. There's also a book here that I wanted to list, New Dark Age. It covers a lot of the open-ended questions of our humanity. Definitely good. It's a good read. Just a question, like have you tried using the new library from OpenAI? I have not, have you? I learned about it, but I didn't get the time to try it. But I think it would be something useful for all drooplers, because it basically takes a keyword or something, and it generates text for you. So it can help. It can be used to help content editors and authors to come up with interesting ways to write their content. Oh, very cool. If you're doing something like that, I'll be definitely like to see what you can do. Do you explore in the Drupal context around OpenAI? Yeah. Oh, what did you do? I tried to just like the same thing. I'm not really that much into that, but I tried to just do image classification. But I have other projects that I work on that are just analysis strategy. Yeah, I'd love to see more use cases for this crop up, because content editors can only do so much when it comes to content moderation and making sure that their content is accessible. So I'd love to see more use cases of this crop up. Yeah. Yeah, so definitely come to LeBof and we'll talk through it. I'll try to do it. So most of what you've been talking about is online and kind of plow-based. Are there things that you can do offline when you have no network available? I'm working with ZoneMinder to kind of do some stuff, but I'm wondering if they're making some other tools out there for offline? Yeah, so the question is around like this is an online API. Your website has to reach out to the web service and then get the content back. But what happens when you don't have access to upload that content? This is where TensorFlow really interests me because you can host your own neural net. Yeah. Very cool. So I think that's all for the question. So I wanted to thank you all for coming out here. I've loved talking with you all about this. It really interests me, so I'm glad that it interests you as well. Thank you so much. Oh, yes. Yeah, they're playing StarCraft and Dota. Hopefully, you can make a Drupal website that plays StarCraft. There are other applications that I like to share. You make it for like game critic. Yeah, yeah. Very cool. Yeah, I loved it. I haven't watched all of those matches.