 computing power that we wouldn't think was possible like I think half a decade ago. I think everybody knows about that and these sophisticated self-learning algorithms coupled with these two things gives us machine learning. Okay, so I want to narrow down our subject from machine learning to machine learning on mobile because that's what we are interested in right now. Typically, machine learning on mobile uses pre-trained models. Yes, on board level three, it is possible to train a model on device but you still need some sort of pre-trained model in there. And along with machine learning on mobile you get security and speed. This is because your data never leaves the device. So things like images and stuff, user wants it to be secure and not to send it over the wire. So when you do machine learning on mobile your data will never leave the device. It's more secure and it is fast because no API calls, right? So it is more seamless experience. Alright, let's just get this out of the way. You do not need to be a machine learning expert to understand artificial neural networks, all the complicated algorithms to develop apps using Core ML today. Of course, if you're interested, you can dive deeper, understand more, teach me a little bit about it. And you can definitely add some value but the point of this is it is optional. You do not need to understand all these complex algorithms to use Core ML on iOS today. Alright, so machine learning is easy. How do I use it in my app? Another point I want to make is you do not need to come up with a new idea of an app to use machine learning. You can already do in your existing apps. How do you ask? Let's take the inspiration from Apple's default apps. First app I'm showing here is Photos. So obviously the main feature of Photos is to show the photos that you previously taken using the camera. But it doesn't stop there. What the Photos app does is it uses classification technique and it will categorize all your photos into different categories. It doesn't work like only docs, stats and people. It will also differentiate your photos according to the setting, like photos taken at the beach or photos taken at a restaurant. So I think that's a very classic example of how Photos app is using machine learning. Next one is my favorite, it's the Keyboard app. So what it does is it will optimize the touchy area based on how you type each letter. This is the reason why when you type on your friend's phone, you can't really get it right. Because every phone optimizes the keyboard for its user. I think this is a very perfect example of how little enhancements can significantly impact users using machine learning. Alright, so I want to introduce a new app to you guys. Let's say we are building an app for Singapore Zoo. So I thought that we will have a new feature where if people take pictures in the zoo, we can classify the image and find the species of that animal species and type basically. So let's look at the steps on how we can achieve that. So I've got four steps for you. First one is finding a core ML model. So let's go into each of the steps later. Next one is importing the model into your app. Executing, taking the input from the user and executing it in the request and handling the result. How do you find the model? You can use transfer open source models available online. But basically you have two things to keep in mind. First one is the required model should be in dot ML model format. That's the one that core ML needs. Another one is you should, you need to understand the inputs and outputs of the model. For example, if your input is going to be an image, does the word format, does the model take? Is it very easy to change that format from a UI image to this one? You also need to consider the outputs like what format is it? How can you use it in your use case and stuff like that? Okay. Quick mention for core ML tools. So of course there are tons of rich models available by other platforms. You must have heard about TensorFlow by Google. So you can take advantage of these models and convert it into core ML. There are Python utilities which are called core ML tools. And Apple is the one who invested in them. So basically Apple is themselves giving this utility for you to use other models on iOS platforms. All right. So once you have the model, you want to initialize it. This code is pretty standard. It's basically copy and paste. Most of the times two things are happening here. We're creating a request with the model that we want to use. And we are supplying it a completion handler. This completion handler is called when the request is done processing. So notice that here we are still not executing the request. This is just the initialization part. This is how you run the request. Again, very standard piece of code. You need to provide the orientation because if you mess that orientation, it's not going to be able to detect what it is if the image is flipped out or something like that. So in this case, I'm providing an array of requests. Of course, we just have one request that we created in the previous steps. So we're sending that. Notice that the input type is CI image. It is from the core image framework of Apple. So it's very easy to convert your image to CI image. So we're good. Next one is handling the output. So after this request is successful, it will call the handler that we passed it in the first step. This is the completion handler. It can be called on success and failure both. First thing is checking if it was able to classify the image. So basically, if that's the case, then we have results in the request. And it gives you an array of classification observation object. This is a very generic response on any classification. It basically just tells you what it has found in the image. And it will give you some sort of confidence on how confident the model is on that classification. All right. So now that we have this, let's try to build out our zoo app. Bring up my Xcode. So I've already done the basic things. Let me run this app and show it to you. Okay. So when you have this zoo app, you basically need to need some sort of an uploader. You pick an image and then it's going to classify it. So I've already added the upload functionality. So I'm going to choose something and it just says processing. Is it cutting off? Yes. Okay. So right now it just says processing because there's no machine learning yet. So what did we learn? So the step one was finding a model, right? So let's go and find a model. So for us, it was pretty easy. We just went to Apple's website and they have featured some of the models. And I decided to use this one. It says the architecture is trained to classify dominant objects in a camera frame or an image. And I've been told that it is trained with lots of images of animals. So I think this should suit our purpose. I've already downloaded it. I'm just going to drag and drop it here. On this super time, can you guys see? It's hard for me to see. Okay. So importing is easy using drag and drop. All right. So when you click on the model, what do you see? It shows you the inputs, outputs, and it automatically creates a class. The class is the same name as the file. So it will create a class with the name of that model. You basically do not need anything else than this name, mostly. It tells you the inputs and outputs. The input is image and the output is class label. So it's perfect. We just need the label of what it has identified and we should be done. All right. So let me show you a little bit about how the app is working. I'm using SwiftUI because why not? We have multiple views. The first one which says processing is detection result, the text box. And we have the image where we display the image. I should show this first, yeah. And there's the upload button and the image picker, of course. So the one that we need to focus on right now is the image process, the image picker. So whenever you click an image, what it does is it will call image processor or process image. Let's go to this method. So, of course, this one is empty because there's no machine learning yet. Let me try, let's try and try to do those steps we discussed earlier. So if you remember, step one was about initializing the model. This is what it is. Again, we're going to, oh, I forgot to import stuff. So import vision and code ML. Okay. So what this does is it is creating a code ML model. And sorry, it is creating a code ML request and sending your model in. So this is the model that we just imported, MobileNet v2. This is the only place where you need to specify the model. This is particularly easy when you're switching models. It will work flawlessly. You just need to switch this file that you're using. And this request has a completion handler. And that's all for now. Let's move on to step two, where we perform the request. So we don't need this anymore. So step one is where the user will upload the image and you need to process it. In the step two, we are converting our UI image to ACI image from core image framework, creating a request and just performing it. Pretty standard code, nothing else going on. All right. Next step, what happens once this request is not successful is complete. One, two, step three. Yeah. So let's focus here. I have two word statements to handle two different errors. So two problems can happen. Either it is not able to classify, there is some error in the classification, or it was able to classify, but it did not detect anything. This is the reason I have two guard statements. First one is checking if the results are present. And the next one is taking if the results are empty. Okay. Once we go through the guard statements, the next step is for us to look at the output. As I discussed before, it is going to be the end classification observation. This guy has an identifier. So it says this is the label or identifier of the classification request. So this is exactly what we need. Let's just print this out for now and run the app and see what happens. So this is going to, okay, let's just print the confidence as well along with this so that we can see both the things. So the confidence is a first object or confidence. Ignore the force and drag. This is just a print statement. Let's try the cheetah. So it printed. You can see. So it printed. Did it print anything? Oh, because I have commented the completion. All right. Let's try again. So it prints cheetah and the confidence is 0.86, which means 86%. So when it works, next thing I'm going to do is process the results by that. I mean, I'm going to print out first two detections. So what exactly is this model is giving us? It's giving us a list of predictions and it is all sorted based on how much confidence it has on each of the prediction. So the first one is obviously the most confident one. You can do many things here. So typically what you can do is you can have a threshold level of confidence. I only want to show the output to the user with more than 50% of confidence so that you don't see any junk to the user and if the model is not confident enough. I'm not worried about that. So I'm just going to print out first two detections here and see what happens. So this is pretty straightforward. It is taking first and second elements and it's creating a string out of the detections. Pretty straightforward implementation. I'm formatting the identifier and the confidence in percentages. Let's try this guy. This image was actually taken in the Singapore Zoo. So it says it's an iceberg polar bear, 97%. So that is pretty cool. It also says it could be a brown bear and it has 0.4% of confidence. So you see what I mean. Usually you want to only show values to the user where the model is more confident. I'm going to try it out with different animals now. Golden retriever seems about right. This is the rabbit, yes. So interesting fact, head and rabbit. So heads generally have longer ears than rabbits. So you can see this guy has long ears and the other one is a rabbit. So it has smaller ones. Here's another one. This looks like a hare as well. So this is actually not a real dog and it is able to identify that it is indeed a teddy bear. So that is quite impressive, my opinion. This guy, comic book. This model was not trained to identify comic book characters. So let's try this. I don't really know a lot about cats. Come tell. So it's very tricky to make up the difference between dogs and fox for a model. So let's try with this guy. Red fox seems about right as well. So I'm pretty happy with how the model works. I just want to look at a few things and correct a few things in our processor. So what exactly is happening here is we are running everything on mean thread. As you guys know, machine learning models might take long time to process. So it is not advised to work on mean thread. One of the other apps I was working on, when you pick an image, it stays on the image speaker for seconds. That is exactly not a good experience. So because the model is pretty fast, it is going down quickly. So for that, what you can do is go to the processed image and run this on background. So let's use the initiated. Let's ignore the indentation. So this is wrong because there is a UI update happening right here. So this object detection result guy is interesting. This is the one that I'm publishing on the UI. So that needs to happen on mean thread. And yes, we still have the problem in SwiftUI. So let's fix that. Okay, let's see if it works. Okay, that's okay. It's a dog. It looks weird. It's in the car. Cool. Okay, now that the app is working, let's get back to our presentation. So I want to talk a little bit about vision frameworks. So of course we used core ML, but why did we use vision framework? As you can see, it sits right on top of core ML. So how I think about it is it's like a pipeline for us from your app to core ML. It does all the heavy lifting sorts of things like it converts your image to image buffer. So image buffer, by the way, is the ideal type which core ML consumes and expects for you to send if you want to operate on images. But you don't have to think about it anymore. You just send in an image and vision will take care of everything. As you guys saw, it also sent us like a very genetic response type. It was called VM classification observation. This is a pretty generic class where it tells you about the results of its classification models. It also has similar ones for feature detection and stuff like that. So even getting outputs from model was hard before this. But vision makes everything easier. It also performs phase and line-up detection, mark-hold recognition, etc. All of this is out of the box. You don't even need a machine learning model for this stuff. All right. So to end this talk, I want to leave you guys with this. Machine learning has become so easy on Mughal nowadays that soon it's going to be normal and people are going to expect all of these things from your apps by default. And I think we should just solve interesting problems, have fun, and who knows, you might just find a huge gap and disturb the industry. Thank you. Thank you, Prana, for the talk. Does anyone have any questions? If you have any questions about how you can find her later at the end of this session. Yeah, you can contact me on Twitter. Okay. Thank you so much.