 So today, as you mentioned, we're going to be talking about how to build a pet detector in a notebooks environment. So just to give you a bit of a context, we're going to start off by doing a high-level overview of the deep learning mechanisms we'll be using and the machine learning workflow. And then we'll dive deep into the spend most of the time in the demo notebook and talk about the actual code. So just to give you a bit of context, I'm Catherine Kampf. I work for Microsoft. I'm a program manager on a product called Azure Notebooks, which is our free Azure hosted Jupyter Notebook service. So we'll be seeing it shortly. And here's some of my contact information. I'll be showing it later, so don't worry too much about taking photos or anything just yet. The most important thing to know about me is I really love dogs. And even me knowing a bunch of dog breeds, it's still sometimes difficult to tell them apart. If we look at Alaska Malmoots, Siberian Huskies, they can look super similar. And especially when you're then trying to train a machine to understand this, it can get difficult and you need a ton of data to get an algorithm that can successfully distinguish between these different breeds. So the way we're going to approach this today is using a technique called deep learning. So this is often how deep learning is viewed where you have an input. So that's the dog photograph and the sort of black box where we don't really know exactly what's happening. But then we get these outputs of either a dog or cat or other. Or in this demo, we're actually going to go a bit farther to say if it's a dog or a cat, which breed do we think it is. And so this differs a bit from traditional machine learning where you'd often be doing manual feature extraction. So this requires a bit more hands-on work to say try to discern which features of the data are most important, whether it's size or color, et cetera. And it also requires domain expertise. So this is easier to think about in the pet classification topic space. But if you try to apply this same thinking to a really specialized field, it can get a bit more difficult to do this manual feature extraction. And then you'll also need to run through different classification algorithms to try to distinguish which will work best and get you the highest accuracy. And this differs again from deep learning where we're going to be doing a layered approach trying to understand different features pixel by pixel of our images. And it's going to be the machine doing the heavy lifting of that work rather than manual data science work tuning the features. So like I said, deep learning will require a ton of data. But once you have that data, it becomes a lot easier for the machine to try to understand the data set and generate predictions. Specifically within deep learning, today we're going to be using what's called a convolutional neural network. So this is a really popular network to use specifically for image classification. And that's because this works by preserving the RGB channels in the first layer. So this will take each pixel value, what value they are of red, green, blue, and preserve that in the first layer. And then we'll use a technique, and this is at a very high level, but we'll use a technique to filter on the images themselves and try to extract which information is most important in the convolution layer. And then next we'll move into pooling where we'll try to aggregate that data and reduce the amount of information because of course deep learning is super computationally expensive. So if you can have a bit less of information going into the fully connected layer, it can save you a lot in time and compute power. And then in the end we'll get our prediction for whether or not the animal is a dog, cat, or something else if the network is a bit confused. So if we take a step back and look at the general machine learning workflow that we're going to be walking through today, you're often going to want to start off with data exploration and the data itself in a machine learning workflow. This is going to be where all of the power for your deep learning network comes from and it's going to be the most important part of the process. So this will involve finding a data set, possibly transforming the data set into a particular format you need, cleaning any data, and running some visualizations, trying to understand the basic attributes of the data. And once you have a sense of that you can move into training. And with training this is where we'll actually be developing our algorithm. So there's three main concepts that we like to think about with the training script, compute, and tuning. So to start off with our training script, this is where we'll try out different algorithmic approaches. And of course, your local box can be pretty powerful, but compute-wise, depending on the size of your data, you might need to scale out to a larger VM in the cloud or on-prem or a cluster computing environment. And once you have that algorithm you might be fully satisfied with the accuracy you're seeing or you might want to do some tuning to try to refine that algorithm to see a bit higher accuracies. And once you're happy with your model, then you'll move into the inferencing stage. And this is where you're actually using your model in an application. So this involves three different components where you'll have productization. So this can often mean refactoring your code. So you might have started in a Jupyter notebook environment, but you need to output a Python module. So you'll have to do some work there to refactor it, and we'll talk a bit about that later. And then deploying your model to a web service so you can use it in your applications or other folks across your company or organization can as well. And once your model is deployed, then you can write a test application to send a photo of a dog and get returned back with your breed prediction. So that's the high-level overview of what we're going to be walking through today. We'll spend most of our time in these first two stages just for the sake of time, but I'll point you to a get-over repository at the end of this through the entire life cycle. So we're going to start off talking about the data we're using. So today we're using the Oxford Pet dataset. So this is a pretty common dataset which contains 37 categories of different pet breeds. So this is cats and dogs and around 200 images per each breed. And here's a link to it. I definitely encourage you to, especially if you're getting started with machine learning or image recognition, this is a really great and well-labeled and well-documented dataset to use. And then once we get into wanting to explore and understand our data, there's a bunch of different great tools you can use depending on what types of data you're working with or the scale of your data. And today we're going to be using notebooks. So if you're not familiar with Jupyter notebooks, they essentially let you combine markdown text, images, visualizations, et cetera, alongside executable code. So it's super useful in data exploration and data science in general to tell a story around how you got to a specific graph or how you got to a specific model so you have the context either to present to others or look back on your own work and understand each step you went through and get strong visualizations of your data. Specifically today we're going to be using what I mentioned earlier, Azure notebooks, mainly for the free setup and scale out. So my local box is fairly powerful but for something like this I'd rather use a really big, beefy GPU machine in Azure. So Azure notebooks lets you connect from their free compute to a remote VM in Azure. So super useful for scale out scenarios and so I can make it public for all of you to go and play with on your own as well. So I'm going to switch over to the demo to show you exactly what this looks like. So this is the GitHub repository. I'm going to link... Oh, this isn't showing up. Did the IT people go? All right, let's see if we can fix this. Ah, okay, we're good. All right, so this is the GitHub repository I'll link you to just to give you an overview. If you scroll down here you'll see these little launch badge. If you click this it'll automatically clone it into your Azure notebooks account as a new project. So super easy to get started. I've already cloned in which I've already done. You'll get an overview of all the project files here. And this is the compute picker I mentioned. So there's free compute offered as well as... This is my camp VM that I'm connected to. So it's a NC6 GPU machine in Azure. If we move into our notebook I'll just start off and run all of these. So let's first try to understand what the data structure we're working with is. So we know that we're using the Oxford Pets dataset but I'm just going to do a quick ls to see the folder structure we're working with. So it looks like we have 37 folders and presumably those contain the 200 images that we're going to be analyzing today. And just to get a quick visual sense of these I created a little plotting function so we can see exactly the breeds we're going to be working with. And holler in the back if this font's not big enough to make it big. And I think it's very useful to always get this visual sense of your data. For instance here we can see all these 37 breeds and it'd be easy to, once we have our model deployed, submit a breed that's not covered by this dataset and then we confuse if we submitted a golden retriever which isn't reflected in this data and saw relatively low accuracy. It's because it's not reflected in this original dataset. So as I mentioned we're working with 200 images per breed which may seem like a lot but for something like deep learning it's really not enough. That's relatively pretty small data and we'd likely end up with an overfitted model that wouldn't scale out to the data we'd see in the wild. So instead of just training on that we're going to be using a technique called transfer learning. So with transfer learning we're going to take a pre-trained model, so this is the mobile net model that we're using today which has been trained on thousands of general images and then what we'll do is retrain that last layer specifically to our 37 pet breeds. So it'll use all the power of someone who trained this massive, massive network but specify it down to our dataset. So when I run this training job we can see here it takes a bit and has a ton of output. So we can see it took around 26 seconds and we saw an accuracy of almost 80%. So this is from doing that initial transfer learning not tuning any of our hyperparameters which we'll get into later just using a flat learning rate we are still able to get to 80% in 26 seconds which is pretty impressive. If you look back this dataset was first released in 2012 so seven years ago and even with a lot more compute power a lot more time data scientists were still only able to get to around 59% accuracy so it's pretty impressive how far we've been able to come in just a short amount of time. So 79, 80% is pretty great but I want to see if we can improve this all so now we're going to be working with what are called hyperparameters and these are attributes of your network you can determine beforehand so specifically we'll be looking at what's called learning rate and the learning rate is essentially how much you'll let the weight vary on a node from iteration to iteration so how quickly you're letting the network learn and oftentimes in data science you find yourself trying out a bunch of these values and just for looping through randomly because it's often difficult to determine which value might be the best for your specific network so instead of doing that by hand and taking hours to do it we're going to use something called Azure Machine Learning Service which lets you distribute this work across a cluster so I have a four node cluster in my Azure subscription and basically I'm going to send the training script to each of the worker nodes and it'll try out a bunch of so if we see here it'll try out a bunch of uniform random values for the learning rate and it'll tell me which gets the best accuracy so then I can treat that as my best model without having to do as much work and we're going to do a couple of things to make this more efficient so I need to update my calls but we're going to use this early termination policy and basically this lets us if you see this 0.15 what's that saying is if we're seeing a run and the accuracy is less than or more than 15% away from what our current best accuracy we've seen is then we'll just cut that run short and free up that compute resource to be used with a new value so as this runs we can see a bunch of output it's not loaded yet but sometimes it takes a bit depending on wifi but we can see all these jobs running so and a bit of information about our cluster so we have four nodes running and it'll run through a different job tell us how long it took what its run ID is so I'll come back up in a bit so we can see the visualizations it'll start giving you but even though it's pretty efficient to distribute it across a cluster it'll still take around 25 minutes so this is a 30 minute talk I don't really have time for that so here you can see some of the validation accuracy starting to come in as well as different learning rates but I'll just skip ahead to a run we've already done in the past so this was a run I did yesterday and I feed it the specific run ID and now I can see a bit of the information see another graph of the validation accuracy as it went through training and see that the final accuracy was around 93% so with just an additional 25 minutes of training it'll increase our accuracy by 13% which is pretty exciting and 93% is a really great accuracy especially for something like an image recognition task so now that I have that treated as my best run I'm going to register it with the Azure Machine Learning Service and this will basically let me use this model from anywhere or deploy it easily so if I wanted to access this model from BS Code etc. or in a future notebook it's registered and available to me as well as anyone who's working inside my workspace so I know we just covered a bit so I'm going to flip back to the slides to review some of the topics so again we just went through training and we were looking specifically at trying to do deep learning with small data and by nature deep learning is going to require huge amounts of training data because it's doing that feature extraction and trying to figure out the best network structure on its own and so you need as much data as possible to learn from for that so 200 images isn't going to be enough which is why we decided to use transfer learning and specifically transfer learning with the mobile net where we'll be taking the existing mobile net model and retraining the last layers specifically to our 37 pets and then since that was pretty good accuracy around 80% we decided to see if we could do any better by doing some hyper parameter tuning using azure machine learning service so just to call out a couple more exciting things if you're just getting started with machine learning AML can be super useful it's got experiences where you can just do a drag and drop automated machine learning and it tries out a bunch of different classification algorithms for you or use it for hyper parameter tuning like we just saw an automated compute scale up scale down so I have a four node cluster in my subscription right now but I've set the min nodes to zero so whenever I'm not using it and not running my jobs it'll scale down and won't cost me money which is super great so once we have this model like I mentioned you might want to do some refactoring so I'm going to move into VS code for that and this is the same demo notebook that we just had the IPYMB file but when I open it in VS code the JSON dump of what a raw notebook file looks like and I also see this option to import it so I'm going to go ahead and click that and VS code will turn this into a .py file with a bunch of cells so here I can see the mark down has been turned into comments my Python code is still here and I can visually see that I have these little cells with this run cell option that'll bring up an interactive window and run a cell as you would see in Jupyter side by side and this essentially works as a Python console so you can type code in here etc so when you're into refactoring this can be super useful if we just highlight a snippet of code we'll have all the refactoring capabilities you're used to with an editor so you can see we can change all occurrences extract method etc all from what started as a Jupyter notebook so you can now refactor that into whatever form fits your workload best and I have an example in the GitHub repository of what a refactored Python module might look like and once we've refactored it to what we're happy with and deployed our model we'll want to go ahead and test it so this is a testing script I've written so we'll do the same action and run this cell to see what it looks like and what we'll be doing here you can either we have some code to access a random pet to try it out or in this example I'm specifically trying with this little toawa I found on the internet and we can see here it loads the toawa image as we would expect and something new that we just introduced is actually the ability to debug cell by cell so we just ran this first cell successfully and now I'm going to hit this debug cell and we can see at this point here so once I hit debug cell it'll open the different tools I'm used to in my debugger I can step through etc or just continue on and once I continue on I can see that it gets it right toawa with a pretty accurate probability so this is super useful to be able to do this debugging bit by bit and we just introduced it this week and we're going to learn more about it come by our booth and we can talk more but this is going to be super powerful when you're working in a data science space and trying to debug cell by cell and so now that we know our pet detector is working pretty well and we have a great probability we're good to go so just to rehash a bit about working with python in visual studio code we just saw that we have debugging and refactoring capabilities there's also IntelliSense so auto completion as well as the ability to import export Jupyter Notebook so as you saw we were importing that Jupyter Notebook it transformed it into a .py file with different cells and you can continue to work and refactor that into a python module or you can re-export it back as a Jupyter Notebook if so you want to present the information etc there's a variable explorer data viewer a bunch of full-fledged data science tools and if there's anything you don't see that you would love to see please come talk to us we are heavily investing in making this a great experience so now that we've covered most of that workflow what's next so here's a link to the GitHub repository where you can build your own pet detector as well as links to try out Azure Notebooks and the data science tooling as well as the visual studio code and then I have some resources on the next slide as well but I'll let people take photos of this as they wish and I think we have a few minutes we'll have a few minutes left for questions and then I'll hang around outside in the hallway as well or I'll just be outside and at the booth thank you for the talk I have a question usually in this kind of example so we talk about classification to the small amount of categories like brews of dogs what what shall we do if what can I train something to classify let's say several thousand categories or maybe one hundred thousand let's say I just want to create some classifier that tells me what is on the picture so how can I accomplish this task so shall I try a lot of classifiers for each separate category or maybe there are some approaches to do it just out of box yeah so it depends sort of on this how big the size of each category you have so if you have hundreds of thousands of categories and five images per each category then you'll have to try to use some pre-existing models but if you're able to do if you have a hundred thousand categories and you have a hundred thousand images within those then you can employ a lot of different techniques whether you just want to do traditional deep learning on your full data set or try out bit by bit depending on the topics you're working with in the image you could try different classification algorithms on each set as you mentioned okay and another question is there some you know software as a service in Azure that provides classifier as service just for me not to write it by my own but just call some API and yeah so we have something that's called Azure Cognitive Services which basically are suite of exactly that APIs for speech to text, search image recognition etc so if you search just Azure Cognitive Services that's exactly what it does hopefully that helps in the demo you are using an image folder locally is that correctly understood in the GitHub repository you have images there and they are then uploaded to your model the images are not coming from somewhere else yeah so these images I actually put into Azure storage they are in the GitHub repository as well so the notebook refers to the Azure storage so you need to upload it to the Azure storage and then so I loaded them on to the specific VM I was working with but you could there's a limited amount of free storage in Azure notebooks as well you can upload a subset of the data and work with it there or upload the full data set to Azure and you could also use you also have a service for uploading the images right on Azure if I remember correctly yeah there's a couple services you can either do it from the Azure portal or use the Azure Data Explorer which will let you upload data to a variety of stores depending on what your end store is in Azure if it's blob or Azure Data Lake it's the possibility of marking up where in the image the object is that you are so this data set is nice because it actually it gives you the full image and then it also has a highlight around a box highlight around the face of the pet which makes it a good sample data set for cases like this but do you mean in general or specific for this data set for specific yeah yeah does that help that yeah no that's good it'll like box the face for you to make it a bit easier so if you're looking at a full dog or just a dog space I just didn't see it in the GitHub repository but it's somewhere yeah if you the link I provided to the like Oxford pet data set earlier in the slides if you read there it'll have the boxed images thank you yeah this one or the one okay and I'll tweet the link to the slides as well so if there are no more questions is there a question left if there are no more questions let's thank the speaker again