 Welcome to day two for.NET Conf. I'm Ankitas Tanna, and I'm joined here by. I'm Gala Oshry. We're going to be talking about what's new with machine learning.NET or ML.NET. So as you know, .NET is a great tech stack for building a variety of applications. You can build web apps, cloud apps, gaming apps, IoT apps, and what specifically we're going to be talking about in this session is how we're making .NET great with ML.NET. So let's get started. Before we get there though, I know that a lot of you folks perhaps are just getting started with machine learning. So Gala, perhaps can we give it a little bit an overview about what machine learning is? Absolutely. So one way to think about machine learning is that it helps you program the unprogrammable. So if I wanted to write a function that takes an image as an input and tells us whether there's a face in it or not, it's not clear how you would start writing that function. Or if I wanted to write a function that takes the description of a shirt and outputs the price of the shirt, you might look for some key words within the description, but it's not clear if that would work very well, or how you would scale that to more items. Now, even though we don't know how to write that function of inputs and outputs, we have many examples of these inputs and outputs as data sets. So we can take a large number of images and actually label them as having faces or no faces in them. Now machine learning lets us take this data set and create this function, a model that can take the input of an image and predict whether there's a face in it or not. Now there are many tasks in machine learning. To give a few examples, one is classification where you're trying to decide whether something is A or B. In this case, we looked at is there a face or no face in this image. Another example is regression, where you're trying to predict how much or how many. So if you're trying to predict the temperature tomorrow or the price of a shirt, that's a regression problem. Another is clustering, where you're trying to look at what are the groups in this data, what are maybe the different topics in these set of news articles. Today, essentially with ML.NET, we're going to be showing you how we solve three real-world ML problems. So let's give you a taste of that for the first problem that we have here, which is GitHub issue classification. So let me just switch here from my PowerPoint presentation here to the screen here, and what you're seeing here is essentially the.NET CoreFX repo. This is essentially an example of how the.NET team is already using ML by building a classification model. So here what I provided here is the title of an issue and then description of an issue. I'm actually going to go ahead and submit this issue here. So this is a new issue that's been open yet, and what we're going to do with ML.NET here is, we're going to go and classify the label tag on what this issue really belongs to, using the description and the title that's here. So in the background here, I'm just going to cheat here a little bit and run this little app here, and what that's going to do is, it's going to use the GitHub APIs to essentially go and tag this CoreFX issue. So you'll see here that as soon as I ran this particular app in the background, it went ahead and labeled this as an area system.NET. When you look at the title of this issue and the description of the issue, you will look at it and you'll be like, you know what? This is actually really good. This actually works well. So this is one example that we're going to get into detail on how we build this issue classification using ML.NET. This is something that the.NET team is already using today as a part of the CoreFX repo for issue tagging and so on. The other two examples that we will also get into is how do we actually build a movie recommender with machine-learning.NET. This is one of the new features that we've added as a part of ML.NET.NET.3, and then GAL is going to get into how we can do image classification now with machine-learning.NET using deep learning with the TensorFlow Edition as a part of the ML.NET.5 story. Before we go further, given that some of you folks might be new to what ML.NET is, it's fairly new. We released it as built, we released it at build. So we want to give you a little bit of an overview on what ML.NET is and how this currently positions itself against the other Microsoft products that are also in the AI ML space. So if you're new to machine-learning, one great way to get started with machine-learning is to take a look at pre-built ML. Microsoft as a part of Azure Cognitive Services, as a part of the Microsoft platform, provides a great way for you to be able to do pre-built machine-learning. So in this code snippet where you're looking at here is that I'm using the Text to Analytics API, which is a part of the Cognitive Services set, and I am providing it a product review. In this case, this product review is for a vacuum cleaner. It says that this is a great vacuum cleaner, and then this API essentially returns a sentiment whether this review is positive or not. So as you can see here in this case, this is very, very simple to do. You're using an API which is basically backed by a pre-trained ML model, and it's returning you a sentiment. For a lot of scenarios, this just works very well, you should use it. Having said that, let's say we actually change the product review now to be instead of being that this is a great vacuum cleaner, to now that this vacuum cleaner sucks so much dirt. So now what's happened is we've changed the product review here, and in fact, this is a great product review for a vacuum cleaner. But what happens with pre-trained ML is that the sentiment that this is going to return, now is going to be only 9 percent positive. So it's actually negative sentiment here. This highlights some of the problems that there are with pre-trained ML. Pre-trained ML works well for a good number of scenarios, but in a lot of other scenarios, you do need to build custom ML models where you bring in your own data, you train with your data, you have your own algorithms, and then you run and deploy these models in production. That's really what ML.NET is about. ML.NET is a framework for building your own ML models, custom ML models essentially. Now, as this is a framework, this has various components as a part of it. The first component that it has is it has APIs. So what we've done here is we've taken a look at the APIs that CNTK had, a couple of other offerings had, and what we've come back with is one set of developer-friendly APIs for model consumption and model training essentially. Something else that the base ML.NET Nuget package comes with is a list of transforms. So when you're doing machine learning, one of the most important activities you'll do when you spend your time with is doing data preprocessing, data transforms, feature engineering, and so on. For these kind of tasks, you require these transforms that come with ML.NET essentially. So things like categorical transforms, text transforms, feature selection, these are all built into the framework, so you can use them for data preprocessing, processing, and feature engineering. Something else that this framework comes with out of the box is a set of classical machine learning algorithms, and you can use these algorithms for doing various ML tasks that Gauss just talked about like regression and classification. So we have linear learners, we have pre-learners, we have SVM, we have learners for clustering like K-means, and so on. In addition to this, ML.NET also has the base package also comes with some other stuff. So as you can imagine, machine learning data has certain characteristics. And for these characteristics, we have heterogeneous data types and homogeneous data types that ML.NET comes with, which can help you with data messaging, data massaging, and transforming, which eventually you'll use as a part of the model building phase. And then ML.NET also comes with some other aids or utility aids for, let's say, loading from a CSV, from a T-S-V, and so on. So all of this is essentially what makes up the ML.NET framework today. The last thing that I also wanna touch on briefly before we go further is that ML.NET is extensible, which means that as popular first party or third party frameworks come into being, we will wire them up as a part of ML.NET, for example, TensorFlow, which we're gonna talk about later, and you can then use TensorFlow and other popular first party and third party machine learning frameworks to the same set of consistent developer-friendly APIs that we've built for you in providing you as a part of the ML.NET package, essentially. So that should give you a little bit of an idea about what this framework is, what are some of the components there, what it comes with, and as we do the demos, you'll see these different things in action, essentially. One important thing perhaps to also call out here is that Python and R are great in wiring from machine learning and data science, and if you're familiar with them, that's actually a great choice. We provide great tooling and experience for them in Visual Studio and our products. But if you love .NET and you love C-Sharp and you wanna stay in C-Sharp and .NET, ML.NET is a great way for you to get started in machine learning. In terms of products that you might've already used, like Cognitive Services, ML.NET complements this experience. For example, Cognitive Services, as I mentioned, has pre-trained machine learning, for the most part, whereas ML.NET provides the ability to build custom machine learning models, entirely in .NET. Azure ML Studio is another product that we have in this space, which provides you a very graphical way of dragging and dropping components to compose models, machine learning models. And ML.NET complements that by providing a very nice code-first experience which will extend and complement your experience as you use these products together. The other thing to point out here is that while ML.NET is only about four to five months old at this point externally, ML.NET has been used heavily within the company for the last decade. ML.NET is used by a lot of the iconic Microsoft products like Bing. For example, Bing uses ML.NET for click-add predictions. Excel uses ML.NET for chart recommendations, a feature that you might have used. PowerPoint uses ML.NET for design ideas feature, another feature that you might have used. Windows 10 Hello uses, Windows Hello uses ML.NET, Windows Defender uses ML.NET. So what we're trying to get across here is that ML.NET has its roots for the last one decade within the company and has powered some of the most iconic products. And even though you might not be filled in with ML.NET from an external perspective, you probably have used a feature which is basically being built by ML.NET or by using ML.NET. So in a nutshell, what we're going to summarize here, if you're just looking at ML.NET for the first time, ML.NET is a machine learning framework for .NET developers. It allows you to build your own models. The part of view we've taken with designing and developing ML.NET is that we want this to be developer focused, which means that we're going to optimize it for certain ML scenarios like recommendation, sentiment analysis, classification scenarios, predictive maintenance, regression, forecasting, things that are common ML tasks that you run into every day. ML.NET is proven and extensible, which means that we've had this within the company for the last 10 years and what we're doing externally now is we're redesigning the APIs as a part of the community. But ML.NET is, the base of ML.NET is extremely strong. And the last thing there, just how we roll in .NET these days is that it's also open source and class platform. And you can check us out at our machine learning repo, which is the .NET machine learning repo here. You will see that we already have about 3,500 starters here, a bunch of forks, there's a lot of activity on this repo. In addition to this, we also have a samples repo that you can check out. And with the samples repo, we're growing this repo right now. But currently you can see it has C-Shop and F-Shop samples and you can explore them as you wanna get exposed or as you learn, as you start to get started with learning machine learning. You can check out these samples here. Getting started with machine learning with ML.NET is also easy. If you land on the .NET website, the easiest way to find or getting started docs is by going to the ML.NET Getting Started page, which is this landing page right here. When I click Get Started, you can follow the instructions here. And this will help you get started with ML.NET by installing the console app, by installing the ML.NET New Get Package, and then taking a look at the IRS data set, and you'll go through and walk through a problem here, which will allow you to do a regression model for building and predicting, classifying IRS flowers, essentially. So that's some of the stuff that we wanted to mention as a part of what is ML.NET. Next, we wanna get into some of the demos that we have. But before we get there, we also wanna briefly touch on what we've shipped as a part of ML.NET releases over the last four months. Every month, the first Tuesday, roughly, we ship a version, a point version of ML.NET. We've already shipped four versions at this point, point two, point three, point four, point five. And as a part of these different versions of ML.NET that we've shipped, we've added a lot of capabilities. So the first thing that we've added there is the ability to do additional ML scenarios, like if you wanna do clustering, we've added learners and transforms for that. We've added recommendation as a scenario. This is the preview of recommendation in ML.NET. And with ML.NET 5, we've also added support for TensorFlow, which will allow you to do some deep learning scenarios like image classification. In addition to this, we've also added some other data science capabilities here. So we've added cross-validation as an option for training and testing your data, a variety of new transform and additional learners, support for ONIX. We've done some work in ML.NET.4 for improving our support for F-sharp by supporting F-sharp records. And lastly, we've also authored the ML.NET samples repo, which has some end-to-end and examples that you can use to get started with ML.NET. So I hope that gives you a little bit of an overview of what ML.NET is, what are the different components in the framework, what we've been shipping every month for the last four months as a part of this. And next, let's get into how we can use ML.NET to, let's say, build the GitHub issue classification model, an example that I just demoed a few minutes ago. Gal, perhaps can you walk us through that? Absolutely. So if we go to the next slide. So just to quickly remind ourselves, this is a classification problem because we have the issue coming in with the title and the description. And we're trying to assign it to one of these classes, one of these areas that the engineering team at .NET has decided. So this is an example issue where we see the title, the description, as well as the labels and the bottom right. The label that we're going to try to predict is area-system.io. It's where in the code is this issue relevant? So the features, which is the input to our model and the information available to us when making the prediction are the title and the description. This is what the engineering lead would look at to make that decision. The label in this case is area-system.io. So when we want to train our model, we're going to take these features and the labels, all these examples, and feed them to our learning algorithm to output a model. Now this model at prediction time can then be used to take in the features when the new issue comes in and predict which label is most relevant for that issue. Taking a step back and looking at the end-to-end machine learning workflow, we start by loading data. So we bring in the data set from all the issues and the core effects repository. We then extract features from it and convert everything to numbers that the learning algorithm can understand. So we need to turn all the text that we have into numeric vectors that have the information available for the learning algorithm to use to make good predictions. That's the extract feature stage or feature engineering. We then train our model using this data and the output of this, the model itself needs to be evaluated. We need to understand is this model actually good? Will it perform well on new issues that come in? If we're happy with the results of this model, we can then deploy it into our app and start consuming it, using it to make predictions as new issues are coming in. So let's take a quick look at the code and let's switch over to my laptop. So one thing I want to note is that this is the existing APIs that are available in ML.NET 0.5, the learning pipeline APIs. This is what we released with ML.NET at build a few months ago, but we knew from the beginning that this API will not cover all of the scenarios that ML.NET needs to enable. So while I want to review all the concepts that are shown through these APIs, later in the session, we will actually give you a sneak peek into what the new APIs look like. So what I have here is a console app. I have the Microsoft.ML Nougat, version 0.5. I also have a few data sets downloaded from the core effects repository. So issues train has all of the issues and their area, their title, as well as the description. The ID does not matter to us right now. What I'll do is start running this app and we'll get to the breakpoint right after we've loaded in the data into our learning pipeline. So this learning pipeline will include the text loader which brings in the data, some transformations to extract the features and turn them into numeric vectors that the learning algorithm can understand. We will then have the learner itself and finally we can do the training step. But let's look at this step by step. So over here I can actually open the pipeline and look at the top 10 rows within it. So if I pick one of these rows, I can actually get a preview of how the data was loaded in. Right now we're separating the columns with this vertical bar. So you see the ID, the area, the title, and the description of the issue. If I go one step forward after this dictionarizer, I've converted the area to a label column, which is numeric, it's a type key, which will be meaningful to the learning algorithm once we get to that stage. So you can see that we added this new column over here in the preview. As a next step, I'm going to featureize the text, again turning it into a numeric vector that the algorithm can understand. So previewing this again, we see that the title in place has changed into numbers. One step forward, we've featureized the description as well. So we have all of these as numbers. Now what we need to do as a final step is concatenate these two columns into one features vector. So the learning algorithm can just take the features and the label columns and use them to train the model. So taking one final look, we see the label here and the features column which concatenated the title and the description. Now I can train the model. Let me actually start running this. But now I can train the model by giving it an input and output class. So this input class represents the columns in the data set that we want to read. And the output class tells us what do we want to keep out of the model at the end. So this training will take a while, but what happens after the training is that we can save the model as a .zip file to load it into a different app. Or we can evaluate on a different data set. Now in machine learning it's important to evaluate the model on data that it hasn't seen yet to make sure that it performs well in a realistic scenario where data is coming again in the future and you didn't have it available when you trained the model. So over here we can evaluate it and look at the metrics like micro accuracy. And finally we can actually take in a new issue. So we've created one here with a title and description and we can bring it into the model to get a prediction. Now one thing that's important is, so the program actually finished running and you can see over here the micro accuracy and the area. So one thing that's important is not to pay attention right now to these exact APIs but to the concepts that we covered in terms of bringing in the data, transforming it, training the model, then evaluating it and using it to make new predictions. So let's switch back to the slides. Cool, thanks Cal. So that was kind of like a quick intro into how you can go about building get-up issue classification model with ML.NET. If you wanna follow the example there, that sample is actually checked in to the ML.NET samples repo. So you can clone that and you can follow the code there. One other capability that we've released as a part of ML.NET.3 is the preview experience for being able to do recommendations as a part of ML.NET. So let's take an example of how we will use the ML.NET capabilities to build a movie recommender. As you can imagine building a movie recommender, you can have various approaches there. So let's cover a few popular ones that come to mind. The first approach there that you could take is population averages. So what we're doing in this approach here is that we pick a particular movie and we take a look at that rating for that particular movie. So let's say that in this case we have The Princess Bride, great movie to watch. And we take a look at the rating for this movie which is about 8.1 on IMDB. And then what we come up with is a threshold and we say that, hey, if the rating of a movie is higher than a particular threshold, we will go ahead and recommend this movie. So all we're basically doing is we're using the population averages and coming up with a threshold and saying that, hey, if the rating of this movie is higher than this threshold, let's recommend it. If it's not, let's not recommend it. This is a very simple approach and for a number of cases, this actually might work very well. But in reality, as you can imagine, there might be a scenario where someone doesn't like romantic and fantasy movies and hence doesn't perhaps like this recommendation. So this approach might not work very well in those scenarios. Another approach that you can take towards movie recommendation is content-based filtering and content-based filtering is looking at some user preferences. For example, the movie genre and then coming up with recommendations based on that. So for example, if a particular person likes Iron Man and likes Captain America, there's a chance or a stronger probability that this person might also like Avengers, essentially. Now again, there's a fall through in this approach as well as it could happen that while the genre-based prediction or the genre-based recommendation is correct, the movie by itself might not be a great movie. The third approach that we're gonna talk about in detail here is collaborative filtering. Now, collaborative filtering as a recommendation approach is becoming extremely popular and is used behind a lot of the popular recommendation systems that you see. So let's learn a little bit more about this. So what collaborative filtering essentially says is that if person A, let's say Gal, has the same opinion as person B, Czar, for example, on a particular issue, it's likely that Gal, person A again, is gonna have a similar opinion as person B on a different issue than a randomly selected person. And to illustrate this point, let's take a look at this example here, right? So what you're seeing in this table here are three users on the left, Ankit, Gal and Czar. And then on the right, you're seeing different movies that we've seen. The tick marks here or the check marks here suggest these are recommended movies or these are movies that we liked. So in my case, for example, I liked Home Alone and Terminator 2. And the crosses here mean that I did not like Heat, Mission Impossible or Casino Royale in this case. So this is an example of data that you can collect as a part of, you know, or movie recommendation system that you're building. Now the question comes up here is that, given this data set, what is the probability of Gal liking Casino Royale? And this is something that Collaborative Filtering can really help us with. So in ML.NET.3, we added support for Collaborative Filtering and what Collaborative Filtering tells us in this case is two things. The first thing it tells us is that Gal and Czar have similar taste in movies and that given Czar liked Casino Royale, there's a higher likelihood that Gal also might like Casino Royale. So let's go ahead and start building this model in code. And I just wanna call out here that there's various approaches for doing Collaborative Filtering. What we've released in ML.NET.3 is Facturization Machines, which is a binary classification learner that we've added, which we're gonna use for this demo. Now once we've established the approach for building a movie recommender, in this case that being Collaborative Filtering, the next thing for us to figure out is which dataset to use. So Movie Lens, for example, is a very popular dataset for movie ratings. The dataset has close to 20 million ratings there across 27,000 movies by about 130,000 users. If you wanna go in and check this dataset out, there's a link here. This dataset comes with two attributes here, two CSVs here, the rating CSV, and the rating CSV provides things like user ID, movie ID, weighting, and the time stamp at which the review is provided. The second dataset here that it comes with is movies.csv, which has the movie ID, the title of the movie, and the genre of the movie. So in our case, once, now that we know the dataset, let's try to model this problem, right? So what kind of ML task is this? So we know we're doing recommendation, but there's a catch here. In ML.net.3, we release factorization machines, which is a binary classification learner. What that means is that it can only tell between A versus B. So what we're gonna do with our data here is that we're gonna take our ratings dataset, and we're gonna do some massaging of data here, and we're gonna transform this dataset into, if the rating of a movie is greater than three, we're gonna put a one there, which means the user liked this movie. And if the rating of the movie is lower than three, the range is one to five, is lower than three, we're going to go ahead and mark the rating here as zero, which means the user did not actually like this movie. Modeling this problem this way as a binary classification model will help us being able to succeed. So let's go ahead with that. So now that we know it's a binary classification problem, the next phase is feature engineering, which means that I wanna pick up the features that I wanna pick. These are the features which will help me predict the rating of the movie. We're gonna use two features here in this case, the user ID and the movie ID. And then we're gonna go ahead and use the rating field as a label, which is what we're trying to predict. Once we have the features and the labels, we can go ahead and train our model, and that's what this picture is showing you. So our features again are the user ID, the movie ID, and the label what we're trying to predict is the rating. And then once our model is ready, at prediction time, we will input into this model user ID and movie ID, and what this is gonna return us is the predicted label, which is the rating of the movie to zero or one along with the scores field. I hope that makes sense. So let's see this app in action. I'm going to just flip here for a second. Let me run this app here and then we'll get into how we build this. The app that we built here is essentially an HP.NET app. It's an MVC app there. So as you can see, this app has a set of profiles here. So I have three users here. I have Ankit, I have Gal, and I have Czar. So let's click on Gal and see what we see here. So in case of Gal, what this Gal's profile shows us here is the recently watched movies that Gal has seen like Heat, Terminator, Mission Impossible. The thumbs up or the thumbs down tells us whether Gal liked these movies or he didn't like these movies for that matter. The next film link here shows you a list of movies which are popular hits on the box office, like Face Off or Titanic or Casino Royale again. And if you look at this little button here called Recommended, when I click this button, what's gonna happen is that this is actually gonna call into the ML.NET machine learning model, the recommendation model that we've built and come back with a zero to 100 rating for each movie. The rating here is represented by this percentage that you're seeing here with this burning icon. So a higher rating here, a higher percentage here means a higher likelihood of Gal liking this movie. So in this case, you'll see that Casino Royale and Gladiator, which are two action movies have much higher rating, close to 80% here each, which means that there's a stronger likelihood that Gal might actually like these movies over some of the other movies mentioned here. And if you take a look at the predictions here based upon some of the data that we used here, which is Gal's profile and Gal's recently watched movies, like Mission Impossible and Terminator, which he upvoted, it will sort of make sense that makes sense in terms of why these two movies, which are also action movies and popular movies, were strongly recommended for Gal. So something else I also wanna show you as a part of this app is how collaborative filtering is helping with some of these Gal's predictions here. So let's take a look at Cesar here and Cesar's profile here. So if you remember in the slide when we showed you, Cesar and Gal have a very similar taste in movies here. So what you're seeing here is that Cesar also has seen some of the similar movies that Gal has seen, and he's also upvoted or liked Heat, Mission Impossible, and let's say not liked Home Alone here, very similar to what Gal had also provided feedback on as he saw these movies. So let's go ahead and do the same thing for Cesar and click the recommended button here, which is gonna bring in the zero to 100 predictions. So you're gonna see here that Casino Royale again is predicted or recommended really high with the close to 85% and Gladiator here is recommended about 80%. Now if you look closely here, we cheated a little bit here because in case of Cesar, Cesar has already seen Casino Royale and he upvoted it and we've used this data for training. But the important thing to note here is that since Cesar is very close in movie taste to Gal and they roughly have the same cohort of choices, this prediction that data that was provided as part of Cesar's profile with him watching and upvoting Casino Royale is what's really helping Gal's prediction being so high for Casino Royale. And that's really an example of how collaborative filtering here is an action with ML.net. To wrap this up, you can also see my profile here, which is almost a polar inverse of Cesar and Gal. So in my case, I actually like Home Alone and pretty much nothing else. And you're gonna see here when I click recommend, it's gonna come back and say that my ratings or my preferences here for some of these movies is much lower. So you'll see here, even though these are still blockbuster and very popular movies that almost everybody liked, it did drop a lot by about 10 points here essentially. So let's quickly go over code here and show you how we build this. So in this solution here, we have two projects. The first project here is where we actually build the model. And just to cover this in a little bit of depth here, you're using the Microsoft ML namespace, you're using the ML.net new get. These are things that you wanna start with if you're getting started with ML.net. Once you've done that, the first thing we do here is if you remember, we wanna convert our data set which currently is providing user ID, movie ID rating and ratings from one to five to be a binary data set which is zero or one for likes and dislikes. So we call this wrangle data or massage data piece of code here. And what that does is that it actually goes through, reads the rating CSV file, converts the ratings greater than three into a one and converts the rating less than three into zero. Once we've massaged our data, we're pretty much ready to go here. We can start again by creating a learning pipeline which is this first line of code. We can then go into adding to this learning pipeline using the text loader API or data. As we add this data, we need to provide the input class here. So that's this is rating data class which takes in the user types, user ID, movie ID, strings and then the field that we try to predict as a label. Once we've created our pipeline and added our data using this line here, the next thing is to do a couple of transforms. So if you remember earlier during the start, I mentioned that ML.net is a framework and it comes with a lot of transforms. These are transforms you need to massage your data into a feature vector, which is eventually what you learn is going to use. So in this case, because this is categorical data, we have to perform this categorical hash one hot vectorizer transform. And what this does is that it takes this categorical data and converts that into a numeric feature vector. Once we've added, once you convert this into feature vectors, this next line of code basically takes the two feature vectors, one for user ID, the second for movie ID, combines them together to create one feature vector, which is what we're going to pass into the pipeline trainer API later. The last thing here to do before we can call pipeline.train is to add the factorization binary classifier learner. This is the learner that allows us to do collaborative filtering and we've added this since ML.net.net.3. Once the training is done, so this data set uses about 20 million records so you can imagine it takes some time. So I'm not going to run this here, but you can follow this code. We're going to be checking this sample sometime today. And once the model is trained, we can then write it to disk and start using it in our app. So I hope that gives you a little bit of clarity in terms of how we build this model. And the app here that we're adding this model to is your classic ASP.net MVC app. So maybe we take a look at this movie's controller here very quickly and bring in the recommend function. Give me one second. So here's my recommend method here. And let me just start this in debugger mode here very quickly. So let's go ahead and choose Gal again, click the recommend button here, which is going to break into the Visual Studio Debugger. Oops, it didn't do that. Oh, I probably didn't start debugging it, so let me try that again. It's going to take a minute to do that. So here we go, click Gal. We bring up Gal's profile, hit the recommended button again. And what that's going to do is it's going to break into the debugger here, into the recommend method. So as a part of the recommend method, the first thing we do is we pass the profile ID. So this is essentially the profile ID for Gal, which has been passed here. The next thing we do is we actually load the machine learning model that we built as a part of the other project. And this also allows us to talk about one of the other value props that ML.NET has. With ML.NET, you can write the model to disk and then you can use that model app locally in your app if you like, as we're trying to show you here. Or you can hide this model behind a web API, publish this to Azure or any other cloud environment and use it that way. But ML.NET will give you the flexibility for being able to do that. Loading a model is really easy. You would use this prediction model type and use the read async API. The read async API takes the input class and the output class, just like how Gal was talking about them. So let's take a look into that very briefly here. So the rating data class is exactly the same class you had before, user ID, movie ID and the label. And the rating prediction is the output class which stores the prediction as we make these calls to the model.predict API. So you can see in the rating prediction here, that has two fields, it has a Boolean field for predicted labels that tells us true or false. Whether the recommendation was good or bad. And then we've also added another column here called scores, which gives us a numeric value which tells us numerically what the score for this recommendation was. The higher the score, the better the recommendation. Once we've loaded the model via this prediction model async API, we can go ahead to the next piece of code that I wanna show very quickly. Which is, I wanna go through all these trending movies here which you're seeing here, face off, Titanic, Home Alone. And one by one, get the rating or the recommendation prediction for them. So I have this for each loop here. And I'm just gonna show you one prediction here very quickly. So this model that predict is what takes in the features that we added, which is the ID and the movie ID and the user ID. And what it returns is a prediction. You can see the prediction in the debugger here. It's gonna return two fields. It's gonna return a predicted labels field and it's gonna return a score. The score is, you can think of it as an numeric value. The higher the score, the better it is. We use a function here called sigmoid. And what that rule is doing is that it's normalizing the score between a 0 to 100 percentage value. So we can represent that to our user. So in this case, when I run through that, you're gonna see the normalized score for this particular movie, which in this case is I believe face off, is about 53%. So what this is telling us is that there's about a 53% likelihood that Gal will like face off. So in this for each loop, we're gonna go traverse through all the trending movies. And once we've done that, we would have gone through and created our ratings array, which we essentially then passed to our view for viewing as a part of this experience right here. So I hope that gives you an idea on how you could use ML.net from a perspective of loading it in an ASP.net app and then using it. We will check in the sample, the recommendation sample soon in the repo as well. The machine learning samples repo, so you can actually follow this as well. So that brings us to our next demo. I'm just gonna move back to slides here very quickly. Give me one second here. And we're gonna talk about the next thing we had in mind, which is, here we go. So just to summarize here, for recommendation systems, we added the field of verification machine in ML.net.3. It's a very popular learner. You use it in click prediction and recommendation system competitions. It performs really well. Here's a piece of code that we use for creating this learner if you wanna follow along. A couple of other things here is that the model accuracy is about 72% for this model. And in order to get better results, what you wanna do is you wanna use more features. For example, we didn't actually take movie genre as a feature in our model, you can use that. And then also, as ML.net matures and goes, moves towards ML.net 1.0, we'll add more recommendation learners and so on, which will help with improving this model accuracy even further. The other thing that I wanna very briefly touch on, and I'll come back with a demo for this later, is that you wanna make sure that for F-sharp, ML.net is a great environment. So if you wanna do machine learning with F-sharp, ML.net is there for you. And as a part of ML.net.4, we did a bunch of work on fixing the ML.net APIs for making F-sharp records work. So the next example that I wanna show you, and I'll come back to this towards the end perhaps, since we're running out of time here perhaps, is how can we do sentiment analysis with F-sharp and ML.net? Gal, do you perhaps wanna go with the new API first and then come back to F-sharp? Absolutely, so if you go to the next slide. So I mentioned earlier that we released ML.net with a learning pipeline API at build. This is a really good API for getting started with machine learning, and gives you one concept, the learning pipeline, where you add your text loader, your transforms, and your learner, and then you can train that learning pipeline to get a model. However, this API had very obvious limitations. So one example is that you always have to end the pipeline with a learner or you have to have a learner in the pipeline. So if you wanted to just do your data processing, save that data as a separate file and come back to it later, you wouldn't be able to do that. Or if you wanted to add multiple files to use for training in the pipeline, that's also not very easy right now. Another area that's really important is feature importance and model understanding. So if you want to look inside your model and understand what are the ways or coefficients to understand why it's making the predictions that it's making, it's not really possible to do that right now with the learning pipeline API. And we also had a lot of different suggestions from the community in terms of how to make the API better, how to make it easier to use. And we knew from the beginning that we would want to make some updates before we reach ML.net 1.0. So this new API for ML.net will be actually released in the next few weeks as an early version within ML.net. And it will enable all of these different scenarios that I just mentioned and more. So one other example that came up is taking a model that you've already trained and using it as a starting point to train a different model. So you don't have to bring in all of your data set again and start from scratch. This is a really common scenario in many cases like click prediction. Now we also want to use parallel terminology to existing ML frameworks like scikit learn. So estimators and transformers and you'll see that over here. We won't go into in detail today, but it's something that you can learn about a bit later. And finally, and I think one of the most exciting things about this new API is that it takes advantage of strong types in.net. And you'll see how it can help guide me through which columns are available at each step when I'm training, when I'm creating my model and what is available to me. So if we, and one thing to mention is that all of this information is covered in a blog post that we posted yesterday that introduced ML.net 0.5 and also discusses the upcoming API changes. You can also learn a lot more in the GitHub repo where all of this is being discussed in various issues. So if we switch over to my laptop, we're going to look at another console app again with the GitHub scenario. So if we switch over to the other laptop, awesome. So you'll see over here that we start with Microsoft.ML Nougat 0.6. So this is a preview of the next release and it's already publicly available. So you can actually try it out and use it today. Now we have the same data set again, issues train that has the GitHub issue label, the title and the description. Now we'll only pay attention to some of the areas of this code for now. Now the first thing that I want to mention is this reader that we're creating. So before we always created that input class where we defined which columns we want to use. Now we actually bring in this text loader and we say that we have the area in column index one, the title in column index two and description in column index three. We're ignoring the ID because we don't need it. Now what I'm going to attempt to do is create the estimator. This defines the pipeline and the steps that we're going to take to transform this data and turn it into a model. And what you can see later in this app is that we actually read in the data and provide it to the estimator through the dot fit function to get the model. So what I'm going to do is create an estimator and I'm going to start with the reader and I can actually see that I have make new estimator available to me on this reader. So I'm going to try that. And then what I can do is now append as many steps as I want to take the input and transform it into to get the predictions from the model. So what you can see is that actually because this is strongly typed, I can see that I have the area, the title and the description as the columns that are available to me. So what I'm going to do is first of all, do the same featureization that we did before where we have to turn everything into numbers. If I just took the area and tried to do something like predict, I see that I have nothing available because it's the wrong type, it's a string. So what I can do is turn it into a key and then I see that I have predict SDCA classification available, which is the linear learner. Now I don't want to do that yet because I want to actually do some additional featureization first. So what I can do is create this label and then again for the title and description, I'm just going to do featureize text for both. And you can see that IntelliSense is actually helping me here to understand which columns are available to me at each step and what are the appropriate transformations and components that I can use on each of them. So at this point, everything is a numeric, but what I want to do is again, concatenate the title and description into one features column. So I can take the title and I see that I have concat available here. And one thing that's important because we're using a linear learner is that we want to normalize the features. So I'm typing that here now, but you can imagine that in the future we'll have some helpers that can identify that you're using a linear learner and that you should use this normalize component. Now I'm also going to keep the label and propagate it down to the next step. Now I'm ready to start training. So what I can do is first of all keep the label and then I'm going to create a score column that takes the label, predict with SCCA classification, I'm going to provide it the features and I'm going to provide it a loss function that I defined above, but in the future this will be a default. Now I want to change the data a bit so that it can be used for evaluation and I can actually, instead of just taking the key that is the output from this model, I'll be able to look at, convert it back to the string that we can attach to the GitHub issue. So first of all, I'm going to propagate the label, I'm going to propagate the score, but I also want to create a predicted label by going to the score and it has this predicted label field. Now this is a key, so it's going to be the integer that represents the class, the issue from GitHub. What we actually want is the string. So I can go back here and do two value and see that it will give me back the string that I want. So it converts the key column to a column containing the corresponding value. So now I'm done. Actually this is the estimator that I'm going to use and when I use the fit method and give it the data, I get the model back. Now maybe I'll start training this while we go through the rest of the code. So the first thing that we might want to do is evaluate the model on some test data. So this is a separate data set that we have available and what we're doing here is just calculating the metrics on this data set. So here we're going to print the micro accuracy which is how many of the issues in this test data set that we predict correctly. Over here we have make prediction function which again takes an input and output class that have the input fields that are available and the predicted label which is the thing that we care about seeing at the end. And we can do predictor.predict on a new GitHub issue and then output the predicted label. So this will actually take a few minutes. So what I think we want to do is actually switch over back to the slides and go over one more scenario that we have available. So Ankit mentioned earlier deep learning and in ML.NET 0.5, we added a transform that does TensorFlow model scoring. So if you go to the next slide, one thing to just mention really quickly about deep learning and we don't have enough time to really explain it in detail but it's an area of machine learning that's really revolutionizing areas like computer vision and speech recognition. The methods in deep learning like neural networks have been around for many years but it's only in the last decade or so that it's been able to take advantage of the huge amounts of data and compute that are not available. So we can train image classifiers that detect whether there's a dog or a face or other objects within an image with really high accuracy. So we added TensorFlow to TensorFlow model scoring to ML.NET. Now, if you're not familiar with TensorFlow, it is one of the most popular frameworks for doing deep learning. Now, when I say we added model scoring, what I mean is that if you have a TensorFlow model like Inception, you can bring that into your ML.NET pipeline and give it to your input and get a score out of it, get a prediction out of it. Now, with the old APIs, you would have to always add a learner on top of that transform. With the new APIs, you're able to look at that score directly and use it. So that's why we're saving this demo. Even though this transform is available in 0.5, we're going to show it to you with the 0.6, the new APIs. So if we switch over back to the other laptop, what we see here is again a console app with ML.NET, Nougat version 0.6, and what we're doing is using this TensorFlow model that was downloaded from this location, very similar APIs in terms of the reader and the estimator, but now we have this image path as the input. So we have a file in this folder, so a couple of images, and what we can do is load it as an image, resize it, extract the pixels from it with a few parameters that match how the TensorFlow model was trained itself, and then we can do apply TensorFlow graph and give it the model location. So the model in this case is a frozen model in the TensorFlow terminology. Now, when we downloaded this model, we also got the inception labels file. So this tells us for each class that the model might predict, what is the corresponding item that it has found? Maybe it's a golden retriever, another type of dog or a car or something else. So after I have my estimator, I fit it in the same way that I did before, I create the prediction function, and I give it the name of an image or the file name. So I can really quickly run this, and in this case, it will detect the type of dog here. Now it's not always going to be accurate, but inception is actually a really good model, and it works for a large number of the images that we have tried over here. So this is a really sneak peek into how you could use the TensorFlow model scoring API or transform with the new ML.NET APIs. So I think if we switch back to the slides. Great. So I just want to call out there, thanks Gal for demoing TensorFlow and the new API. So one thing I just want to call out here is that this is a very, this is the start of us adding deep learning in ML.NET. So some of the code there is going to look very complex and we're going to work on making it easier for you to be able to score existing TensorFlow models, pre-trained TensorFlow models or deep learning models into ML.NET. It's going to take us a little bit of time, but this is just a start for those folks who already know what TensorFlow is and know how to use TensorFlow, and they want to use TensorFlow as a part of ML.NET. So I just want to make that clear. The new APIs. 15 minutes for questions as well. Yeah, so the new API is going to be available as a part of the .6 ML.NET package that Gal just showed you, which ships sometime next month. So just one thing I just want to clarify, mention here as well is that a lot of the code that we've shown you here, we're currently composing the model training code by ourselves, whether the new API or the old API. One of the things we're also working in parallel is on a UI that will generate or to generate the model training code for you and the model consumption code for you as well, which will really help you get started with ML if this looks a bit complex or looks too much to get started with currently. You want to take a couple of questions here, but we just want to go over this one last slide here, which is about what's next with ML.NET. And we're working on improving the API as Gal just showed you. We really want your feedback there. We're trying to bridge into worlds there. We're trying to generalize over the machine learning terms. We're also trying to give you a very powerful machine learning API that gives you all the flexibility for you to be able to support and carry on with ML tasks. We're going to go ahead and add deep learning with TensorFlow. We're going to add and provide a UI in some time, which will allow you to simplify some of these tasks and we'll go and innovate both the language, C-Sharp, F-Sharp for machine learning, but also improve the tooling that we have in VS here. Just one last thing I want to show you before we take questions here is that I skipped over one of the demos that we had here, which was the F-Sharp one. So let me just go over that very quickly and point out one thing here. Doesn't seem to come up. Let me try another thing here very quickly. Oops. Can't seem to find it. Let me start taking a few questions while I keep pointing. So there's the one question from Radu. So can TensorFlow be fully used via ML.NET? Does it mean I do not have to use Python? So we mentioned that the TensorFlow capabilities that have been added to ML.NET so far are for model scoring. So if you have a trained TensorFlow model that you found online or that you trained through TensorFlow and Python, you can bring that in and use it. Now we want to expand these capabilities over time, but that's not available right now. So if you're training your own model with your own custom data, you would still want to use TensorFlow directly. Right. Does Azure Cognitive use ML.NET? So not for all models or not for many models, I think there are some where we're discussing things with the Cognitive Services team, but I don't think most of them are based on ML.NET right now. Right. So I think, yeah, so we'll have to go and check in this sample there. It is the F-Sharp sample is available in our samples repo. The one thing I want to call out is that if you're using records, you do have to decorate with the CLI mutable attribute, which will make it mutable for the ML.NET API. So just keep that in mind. But I think this is what we had to show you today. You can reach out to us on our GitHub machine learning issues repo and ask us all these questions that you're asking us here. Some of the great questions got already covered. You can also reach us on Twitter and via email. But thanks so much for having us here. We'll have Cesar next who's going to get into providing you a little bit more background about what machine learning and AI is and how you can use different kinds of machine learning AI components that Microsoft provides along with ML.NET. So thank you for watching us today and hopefully you enjoyed our session. Thank you very much for your time.