 All right. It's Jeff Fritz and I've got John Gowley. Hello. This is exciting. We're on for the for us midnight till 6am shift. Oh, yeah. Happy to be here with you. Of course. Of course. It's great to be here with all of our friends around the world. But we're going to introduce our next speaker. Yes. So we have Louis Beltranon, and he's speaking on building real-time image classifiers for mobile apps with Azure Custom Vision. Oh, man. I love the AI stuff. Yeah. Louis, are you there? There we are. There's Louis. Hello. Hello. Greetings from Czech Republic. Oh, it's great to see you. The floor is yours. Sorry? The conference is yours. I'll share your screen and away you go. Awesome. Thank you very much. Hello, everyone. Thank you for joining.net.conference. It's 2019, day three. My name is Louis Beltran. I am quite excited to be part of this world of.net conference. I am a Microsoft MVP in Artificial Intelligence and Developer Technologies, and today I'm going to talk about building real-time image classifiers for mobile apps with Azure Custom Vision and also Summary. I hope that you find this session interesting and useful for your project. So let me know if you bother your thoughts about it. So let's get started. Microsoft has a wide selection of AI services included in its platform. For deep learning and neural network scientists, you can create models from scratch by using the Cognitive Toolkit, CNTK, or create a virtual machine with TensorFlow or CAFE installed. You can also develop machine learning models with ML.NET, Azure ML Studio, and Visual Tools for AI. But if you are neither a data scientist or an artificial intelligence expert, no worries. Microsoft has you covered with the Azure AI services, which contain pre-built AI to develop powerful smart applications with cognitive services, conversational bots with both framework and services, and you can also roll on your own AI with Azure Machine Learning. In case your AI needs to rely on huge amounts of storage, you can also use Cosmos DB, SQL DB, and other options. Now, let's talk a bit about Azure Cognitive Services, which allows you to infuse your applications, websites, and even bots with intelligent smart algorithms which are able to see, hear, speak, understand, and interpret your user needs through natural methods of communication. In the digital transformation world, I believe that this is a great benefit for us, since we can create these powerful applications without any background in AI and just a few lines of code. So with Azure Cognitive Services, there are some areas such as decision-making or knowledge. You can build applications which give recommendations for decision-making. Regarding vision, you can recognize, identify, moderate your pictures, videos, and digital content. You can also use the speech for converting text into speech or vice versa into natural language speech. You can use translation services or even identify the sentiment of text. Same goes with language, which is used for bots, for instance. As you can see in some of these Cognitive Services, you can create your own custom services regarding your own needs. We can talk about custom vision, which is the visual intelligence made easy. You can easily customize your own computer vision models. You just need to upload a level of image, let custom vision service to do the magic for the artificial intelligence. With just one click, you can export the trained models into a mobile application, which uses TensorFlow, Onyx, or Event Core ML for iOS applications. And you can also deploy it to a Docker container. We will see the process in a few minutes. But basically, this is what you have. You send some images for training. You set, okay, these images are cats. These are other images for dogs. Then send it to the custom vision service and create a model. After this model is trained and published, you can send new images to perform, to see what is the result if it is correctly classified or not. In case there is an error, it means that we need to train it with more examples or even polish our previous images. The custom vision functionality can also perform object detection. Object detection is similar to image classification. The difference is that object detection also returns coordinates where this object of interest is detected within the image. In case of image classification, only the tag or the label is returned. In order to build a classifier, it is very easy. There are some steps that you need. First, you create a project within your Azure subscription. Then you select a domain. I will talk a bit about it. What is it? Then we add images, assign labels or tags to these images. We train a classifier and then we perform the evaluation. Let's explore the Azure custom vision portal and see how we can create our projects and also add images for training. The website is customvision.ai and of course you need an Azure subscription. When you create a new project, you enter the information, versus cat, I will say number two. We can create a resource in case we have not created one on the Azure site. In this case, I can select this one. I told you that there are two types of projects, classification or object detection. We will look for the simplest one, which is classification. In one image, you can have several labels. Maybe your image contains tags such as dog, animal, even the color can be different labels. In this case, I will use multi-class, which means one single tag per image. The domain means or it helps the classifier to optimize or to focus on a specific element. There is a domain for food, for instance, which is useful if we are building a restaurant application or we want to identify between oranges, apples, fruits, and so on, landmarks for open spaces, retail for shops, shirts, products, or if there isn't any domain, we can use the general domain. There is also a compact version for each of the previous domains. Compact versions, we use them when we are exporting the model, for instance, for a mobile application. In our case, since we are building a real-time classifier, yes, we are using the compact version, which also has the advantage that it is offline, so we don't need to send requests, HTTP requests. There are some capabilities, which are TensorFlow, CoverML, Onyx, I will use that one, and we create a project. After that, we can add the images that we are going to use for building our model. In this case, well, I can use this dataset, it's a huge dataset of cats. I will add 100 images. It is very fast. And after they are here, we can add a tag, OK, this is a cat, and we upload the images. All right, that process is done. Now we can maybe add more images. I have more examples for dogs, OK. Again, let's add around 100 images. It is important that our images or different tags are, let's say, equal, or equally distributed. So if I add 100 cats, OK, it is first to add also the same amount, the same number of images for dogs. Otherwise, our model will be one-sided or biased. All right, so after we add this, let's say we can train the model, we can perform fast training for demo purposes. Or if we want to boost the performance and the accuracy, of course, we can use advanced training, but this takes more time. So we will go for fast training, but yeah, that's another possibility. Every time that we add new images, we are creating an iteration. The iteration, of course, you can say it is a version of your model and you can switch between iterations if you wish. Maybe you can have the production one, which your users are calling every time, but in the meantime, you can have a staging iteration. All right, so in this case, we already built a model and we can perform a quick test. We can either submit an online image or we can choose another one. So in this case, I will select an image that I did not use for training and let's see if it works. Sorry, sorry, test. Yes, test. This images, I did not use any of these for the training, so I will add and let's see if it is correctly identified. Yeah, it says that it is a cat, so it works. And as you can see, in maybe five minutes, we built a smart image classifier. So let's continue with our presentation because the main purpose of this talk is of course talking about mobile and real-time classifiers and that's where the export function comes in hand. So you can train your model, you can add images, you can add these tags in the Custom Vision Service. There is also an API if you want to run a script you can add automatically the images if you don't want to add it manually. Maybe you have a service that is generating images from your users every five minutes, every hour, so you can also use the Custom Vision API or SDK in order to develop these models automatically and train them of course. But in the end, if you want to use this service, you can publish it and send HTTP requests online to Azure, but it is also possible to deploy your model to TensorFlow for Android and Python applications to an Onyx model for Windows, WinML, CoreML format for iOS and also Docker for on-edge scenarios. So now let's explore what it takes to build a mobile image classifier with this service. By the way, this project will be available in maybe 10 minutes after the presentation ends in my GitHub. So what we need to do is to export the model. We can do that. Okay, so we select the project. Okay, I can find the export option. Okay, well, I will try to find it in a few seconds. Okay, yes, we select the creation. Okay, yeah. Okay, yes, it is here in performance. Yes, we have export button. Sorry for that. And then we can select. So I select the TensorFlow format for Android and Python applications. And after that, we get a zip file which contains two files, the labels, which actually has the tags such as cat and dog and the model, which is a TensorFlow model, of course. These two files, we add them into our Android project. We can create a Summary Enforms application and then add them to the specific project, Android in the assets part, or we can directly create a Summary Android project. All right, so you can see here that I have the labels and also the model. Of course, this was previously done. And from this project, if you have built mobile applications, well, there are three important elements. First, we have our view. In this case, I just have a button which will classify the image. You will see the application in a few seconds. We call this command, classify photo local command, which I have on a Custom Vision model. And this is the method. I am using Nougat Package Custom Vision Local Service project, sorry, this cross image classifier, which takes the image and takes into consideration both the labels and the model that we created. You can see it in main activity. When we start the application, we say, okay, the image classifiers should use this model and labels file and also the domain we establish as general. And we will obtain the top prediction for this test, all right? So for the test, I have this video. So let's take a look. So this is a test of our application. We click on take photo button. I use one of the test images. And we will see the result. Okay, this is the same. We click on classify the image button and you will see the result. It is a dog. All these calls, this is another test, are local, offline, no internet required. We save this picture. We test it just as we did in the portal, but now with a mobile application, of course. The result is a cat. And I have a final test. In the meantime, I can tell you, okay, this model only works for this system, for this operating system, Android. But you can export to other models, formats, if you want to include the solution for Core ML, iOS, and also Windows, sorry, with the onyx format. But as you can see, it is very easy. You just need some few lines of code. And we can have our own classifier. So I will just wait a few seconds if you want to maybe screenshot this image for the URL or just visit my GitHub in 10, 15 minutes. It will be ready for you. So you can test it as well. So just for conclusion, sometimes when we build our classifier, we might encounter some problems. So in order to improve, in order to optimize this classifier, it is better if we add a variety of images, especially if we add real images, not only very beautiful pictures that we find on internet, because maybe we are building a classifier for products, and we take online images, but when we are testing the real application, it doesn't work because maybe in the office or in the store, there is not so much light, so the pictures are too dark, or maybe they are taking in one orientation, vertical or landscape, sorry, and we are using in real time in another orientation. So it is better to consider all these kind of attributes, different backgrounds, different angles, the object size also, that's important. Every time we need to train the classifier after we add the images, and we can retrain our model with the images that we use for testing. It means, for instance, the ones that I use for the mobile application, I can add them to boost or to improve our classifier. And of course, it's better to have photos in context. And yeah, that's it from me. Are there any questions? It looks like we don't have any questions. That was really cool. I love seeing that type of image classification. It's simple though, and I love the way that you presented that, Louis. I mean, dogs and cats. I think I can. I forgot to unmute us. So, Louis, I love seeing how you're able to boil that down to something that's clear and easy for us to understand. Comparing dogs and cats is really something that we all kind of know how to do, but to teach a machine to do it really nice. In a half hour. In a half hour. Really great stuff there. Oh my gosh. Just in the nick of time, as we lost our network connection to Louis. All right. Well, our time was up anyways. Yeah, wow. Okay, let's move on to the next. Oh, wait, hang on. Louis is calling back. He's calling on yours. Hang on, hang on. Let's get Louis back real quick. Hang on, hang on, hang on. There we go. It's totally going to work. Totally going to work. It seems that the connection was lost for a second. Sorry. Not a problem. There we go. There's Louis. Yes. So, we really like the way that you were to take that complex topic and boil it down, make it something a lot easier for all of us to understand. There was a question actually of what other kinds of images can a machine identify. Are there specific sorts of images that will work better or worse? You mean different scenarios? Yeah, I don't know. It was just a question of... Yes, well, you can identify whatever you want. It can be used even to detect if there are some illness in plants because of color or if like, okay, we can train it with green or specific color of the plant when it is healthy and if it's dry or if it's like yellowish, you can say, okay, this is a sick or ill plant, so we can test it on real environments in order to detect maybe, okay, this plant needs some current, some water or something. We can use it for, as I said, with the stores to identify or to help customers to add to the cart product, they just place it in the camera and then, okay, this is added to your cart. Very cool, very cool. Well, thanks a bunch. All right, we're on a tight schedule. We're going to move on to the next. Yeah, thank you so much for your time and thank you for the opportunity. All right, Louis, take care. Okay, you too. Goodbye. All right, goodbye. Thank you, everyone.