 Hey, good morning. Welcome to this session about AI and dot-net. My name is Tisar Daltore and I work as principal program manager in dot-net team I'm ML dot-net team. So in today's session, we're gonna we're gonna talk about AI and what you can do With dot-net and multiple components. So it's kind of a broader Components than previous session about ML dot-net. So we will be including Connected services as well and positioning the different technologies even bots So let's start with a very briefly for people that just came to the session. What is machine learning and basically machine learning is is a way to create functions that It was very difficult to do with traditional programming. So for instance You could create a function to identify objects If this is the surface or not if you were doing that in a traditional way it would be very complex like trying to identify if it has a mouth or or Or a IS etc, but instead of that or another example would be you want to Say what's gonna be the the price of the shirt so in order to do that with artificial intelligence you need to train a model and so therefore you You provide a bunch of data About faces. So you say this is a phase. This is not a phase and then Based on that training the the model would work okay Something important is that you need a lot of data to train machine learning models So let's go to the next thing. Okay, so what problems can you solve with machine learning? there are many many problems and And also those problems are related to more technical ML tasks and this is kind of a Another world for for developers like many many things like regression clustering anomaly detection and so on and What we want to do with Microsoft development technologies is to simplify these from a development perspective and Instead of you know, like having to learn the low-level details about machine learning Even like those formulas for logistic logistic regression or or or whatever We just want to democratize so it is easy to use AI in general for developers. So so Let's first position the the multiple technologies that we have for AI and first of all you could have Technologies that you can just consume like pre-built AI and for that You could use Azure Connective Services or other pre-trained models like onyx Which is an standard or core ML or Windows ML in your program So basically you are taking up pre-trained model and you're just running it or predicting What we call the scoring and and just use it in your application. So you're infusing machine learning to your current applications, but you can go further because You can also create your own AI your own machine learning models So for instance, we have multiple choices You could use Azure machine learning studio or you can use machine learning dot net ML dot net Or you can also use other deep learning Technologies like TensorFlow, CNTK or Torch, right? So for today's session, I'm going to be focusing on County services and also on ML net and and of course and then from ML net you can also Integrate with TensorFlow as you saw in the previous session about ML net from Ankit and Gull and Finally, of course, you can use those models From any client app so it could be a bot could be a web app could be mobile apps or could be IOT edge devices, right? So let's start with pre-built AI using connected services Probably you saw this slide If you saw any presentation about connected services, basically We are providing here a higher level of services that you can just use and consume like computer vision or a speech recognition or language understanding and each of those kind of pillars Have multiple sub services So for instance, I'm today. I'm going to be focusing on computer vision in computer vision service on the on the custom vision service and for the demo so Let's do a demo about that and using connected to service computer vision and custom vision for this demo, I'm using a a Reference application forked from eShop on containers So it's a microservice architecture based application but then we are extending eShop on containers with AI features or machine learning features and This is so you can also reuse or surface all these features from multiple clients the web or bot or Or even mobile applications, right? So let's move to the to the sample In this case, we're gonna So you see that I have here running The application in Docker because it's based on all these micro services are running on as locker containers Right and then most of them are just like regular micro services accessing data and some of them are Are using AI features? So this is the application You can see that in the solution you have many services, but just one is the one that we need to focus now Which is the one about? Community services, so we'll come back in to the code in a minute, but I just want to show you what we can do So this is the eShop You could buy things and in the store, but the demo is let's say you would want to search but you Let's say the scenario would be a mobile application You take a picture for of something and then you want to search if the catalog has something like that or In a comparable way for for the demo. I could be searching in the internet Maybe for Frisbees right like here, but maybe I'm searching a different language like in Spanish This goes voladores and then I don't really know I have no clue about the name of this article in English, which is frisbee So then I might save this into into my hard drive and I could search using object Identification right so in this case you can see that I'm using computer vision first so If I provide that image But first of all I'm gonna provide like an umbrella and you'll see that it's working It's gonna be connecting to computer vision in Azure Which is a service that you can just use out of the box with a generic model and then it's just calling to that service and Identifying that picture and then searching in the catalog in my application and you see that it is searching for other umbrellas as well but if I search for The frisbee like this one you can see now that It doesn't work. It doesn't find any any frisbee and that's why because that generic model is not finding the That object was it's identifying that object as a dishware instead of a frisbee, right? So let's go to the code and see a little bit about it So well first of all I want to show you why when using a Computer vision you see that the name of that object was identified as dishware And that's why it didn't work when searching in the catalog. So for that you can use a custom vision So before getting into custom vision, I want to I want to show you The code about computer vision, which is just using that model in the internet So it's it's basically this is the microservice. I can just provide the api keys and the URL in Azure and then just Consume that and provide provide a picture and it will respond with all the tags that That object was identified and that is that simple, but if we want to to use custom vision Custom vision the difference is basically two difference two differences So one is you can train with your own images and secondly it can be online or it can be offline You can generate a model a frozen model in a file and run it in your in your program. So What you can do in this application each open containers AI is available in github In dotnet architecture each open containers AI here you can see The architecture extending each open containers and so on and then in the in the wiki You see how to set it up and one of the points is about custom vision where the difference here is that you can Create your service and then upload a bunch of images that you want to train in this case about Frisbees And after you train it in in Azure in the internet Then you have two choices you can you can run it online Like providing the key and the URL and then just doing a similar way than you use computer vision or the other way is that you? generate or export a file like a TensorFlow Frozen model file and then you can use it from your code. So for instance in in this case if I use the custom vision I Just online then I would I would provide the prediction key and then I will call it with HTTP But if I'm using custom vision offline What I need to provide is the model file name like in this case I would download a defile which is this one model PB as you can see here below Model PB so I just need to load that file in my code and then Provide the image and gen and then run it. So it's offline. So it could be running in this web app could be running a mobile app offline, etc, right? So that's the difference and So I'm gonna change and take advantage of microservices and Unchange variable in the containers so to use computer vision Sorry custom vision instead of computer vision. So you can see here. We are changing this variable to to use Sorry, this one to use custom vision offline Right, it's gonna use custom vision offline and then just restarting the Docker containers using these Using custom vision so I'm gonna run it here. You can see Run and restart the containers We might have like 12 containers in this case We just need to restart a couple of them and then once it is restarted if I go back to the application You can see that I refresh the application and then It's using now and we're showing it custom vision offline So if we search now With a frisbee image And run it you can see that it's identifying This as a frisbee and then we get the right results in the application, right? And so about the code As I mentioned, you just need to provide the file image and then run it locally and it can be completely offline So let's move on to the next section okay, so now let's talk about Custom machine learning and specifically using ml.net So the first question is okay, so you have pre-trained pre-built machine learning Is that enough for me like quantity services? Well, as you might know The answer is it depends it might fit your your needs, but sometimes you need something else so for instance if you use quantity services sentiment analysis service and then you provide like this This is a great vacuum cleaner. Then that's that's a positive, right? But if you provide another sentence like these vacuum cleaners suck so much dirt that that will be identified as not positive and You might want to train these in a different way, right? So this is just one example But there are many cases where you want to know just train with your data that you can do with custom vision but also there are other services in quantity services that you can do custom training and You might also use different algorithms So for that in custom vision you typically have different phases. You prepare your data. You need to work with the data Message of data a lot and then you need to be able to train your own model. So this is the difference compared to pre-built AI and then just run it or consume it or or Use it in your application, right? so in a little bit more detail in The process is you have a bunch of historic data, then you will be Building your model and training with that data and then you test it you test you evaluate the accuracy of your models And when you think it's good, then you generate a file a model file Which in the case of ML.net is a zip file and then you run it or predict or score in any application, right? so You just saw a full session about ML.net from Ankit and Gal I just want to do a very very quick intro about it in case you just came to the session So ML.net is a framework first made for download developers is open source is especially made for developers and It's proven and extensible because we've been using these libraries internally Microsoft for quite a few years I know we are making it public and Simplifying it with a new API and then currently we are in v0.5 in September. So we will be releasing version one in next year and Basically, if we go back to the process that we previously saw you can build and train your models on dotnet core or dotnet framework and by simply adding the Nougat packages of dotnet ML.net in in your application and In the future, we will also provide a UI application model builder So you can use it graphically as well and then that zip file with your model Then you can use it in your application which can also be dotnet core or dotnet framework and simply by adding the Nougat package of ML.net and Loading that zip file and then scoring or predicting right so that's kind of the high-level process Vision and then what what can you do with ML.net? Usually ML.net? Focuses most of all on the traditional and machine learning problems With data, which is very useful for the enterprise for instance So you can you can do you can identify if some data is a or b which is binary classification or you can predict how much about something like forecasting or or Clustering and so on right so there are many problems that we can solve with machine learning like predictive value It would be a regression or something is a or b binary classification or detect issues or problems that might happen In some for instance in devices coming a bunch of events Then you might detect anomaly detection and so on right so we want to simplify this So showing you problems instead of ML tasks in all the documentation and samples and so we make it simpler to use So I want to do a quick demo Which is about regression or a forecast of e-shop dashboard in this case Here the code you see we have two projects Because now we need to first Build and train a model and then when we export the model or we generate a zip file Then we will use it in the web application. So that's why we have two separate Projects for applications, right? So first of all, let's use the The console application to build and train a model. It's a console application here. You can see it's a It's a Regular console app and then we are going to be generating two models in this case one is for predicting forecast of sales in about Specifically about products and then the second model is about Predicting or forecasting the sales for countries. So that's why we have two of them. I'm going to just go ahead and Debug and Took a little bit about it. You see this is the console app. So we're gonna create a model and train it for the first model And then we will write the zip file after that So so you can see in here. This is kind of the important code It's using API. This is the learning pipeline might change in in a few weeks But the important thing here is about the concept. So first of all, you will load the file Like you can see here the CSV So file with data with all the data related to sales about the products And then we in that CSV we load it into the pipeline and then we do a bunch of transformations We we need to concatenate data. We need to convert all the data into numeric vectors and then you we need to say what What's gonna be the label or what we want to predict in the regression and so it's just like a concatenator vectorizer and and then concatenating all the Features and finally we provide the algorithm that we want to use in this case a fast-tree Twitter regressor you could test and try different algorithms and see which one is more accurate and finally we train, right? So just It's training now you can see Here Training and then once it is trained then we generate the Defile, right? So just finishes is gonna go for the for the second model, but basically it will finish now And then we have the two models Okay So these two models are generated here If we go to being debug Then you see the two models one for the countries one for the products and these models You can just copy those files into in in this case like a web application We have the the models copied here and And then I'm gonna just Run it so you can see how it works Right. So in this case, it's just a monolithic web application, but we also have the same thing to each open containers AI Okay So here's running the application. So let's try first Forecast for any specific product like Jumbo Searching now the products. Let's select this one And then here you can see here all the historic data And then this last piece of the chart is the prediction of the Salesforce forecast, right? Let's do the same for country and then see some code Of what we're doing. So first of all We just need to provide what is the zip file of the model for the country And then we provide the data that we need to evaluate and predict or to use for the prediction, right? then Let's go and predict You see that we are providing this instance of this sample and based on that it will predict the forecast for for for this country And here we are it's for the country and then this would be the prediction, okay? Cool, so let's move on So something we really want to highlight is that ML.net is a framework first and so it's basically Libraries no good packages, but we will also provide tooling for it And and is for custom machine learning so you can build and train or you can then run any model It supports on the core or the other framework and then here are kind of the elements that you use with this API you use transforms to make transformations to concatenate data within Dataset that you are going to be using for the training and then Or convert to numeric vectors Futurized and so on then you we have learners or algorithms the ones you want to use depending on the problem and Finally, we are also integrating or extending ML.net with deep learning Frameworks like in this case tensorflow in the previous session. I think I used so a demo about that. You can also go and See the blog post that I published yesterday in the dotnet blog About the features in ML.net 05 and there's quite a few information about the integration with ML.net and tensorflow, okay? I want to really highlight that this is not something just new even when it is in preview We've been using internally ML.net in Microsoft in products like Bing or Excel or PowerPoint or Windows 10 and Yeah, just a final highlight about ML.net is that it's also we want to democratize Custom AI or custom machine learning for dotnet developers. So the difference is that we are targeting Dotnet developers with ML.net Either for training models and also we will provide in that we'll provide in This democratization based on code APIs, but also on a UI tool that will be really simple soon And then finally I also Want to highlight that usually in the in the enterprise You might have Like the enterprise might might be researching not just AI ML But also maybe microservices and bots and so on so that's why We are also releasing this sample of this reference application that I was showing so You can see that in each upon containers AI is what You see in the whole picture, but each upon containers. So just the reference application for microservices would be the application that is Kind of disabled here or great and then we are extending that with AI with the demos that I showed And also with the bot which is the demo that I'm going to be showing now. So Before doing the demo I want to come back to the Site of eShop on containers AI right so In dotnet architecture in GitHub eShop containers AI you can see the wiki all the multiple AI features so for computer vision or pre-trained model and And in this case you also have a section on how you can create a bot with bot framework and Extend this application or kind of surface all the information All the AI features that we have in the app and with a bot so for that You and then this bot is not just using our own services. It is also using Lewis So language understanding in Azure and here you have all these all the Setup steps like how you can create register in Azure and create your Lewis account for that and And then create and register your bot in bot framework service and Finally how you can run it and test it with the bot emulator that I'm gonna demo now and a few Highlights about the about the code, right? So it's ready for you to test this demo I'm gonna show it running to you. So basically you can see that we have all these services Running in in Docker in this case all the micro services. We're gonna be in reality using this service that are using the says the the bot and and Then the one about quantity services. So let's let's try it out So here I have the bot emulator. So bot emulator is an app that you can try locally and you can Provide the information about your bot. The bot is nothing else than another web API that you can add and deploy As a service or in this case we are deploying this web API in a Docker container and here you provide the endpoint to that web API for your bot And the application ID and finally you just try it here So Now it's talking to the bot to the web API here is responding and then because it is already integrated Into Luis we can we can talk to it and it can understand Depending on what how we configure Luis, but for instance, let's say I'm saying hello And it is responding Okay, so let's say I want to I want to see products of the catalog Okay, so I don't know like show me so many products Well, and then here we can see the categories. So I we could be kind of navigating across the the catalog just using the typical the typical flow of the of the web form but also here we are surfacing AI features like the quantity services custom vision that we That we use and if we go back to the Web app we see that it's using now custom vision offline. So the frisbees should work so I can also provide through the Debaut the image and then is searching Products about image and here we can see the frisbees again and you could enter into them or buy It's area, right? So this is a demo about how you can also surface AI or any feature in your services Through a bot. Let's see the code very briefly I'm gonna stop this So if we go to the bot services And the bot is this So we have two versions right now we have bot using dotnet core framework and a bot using full dotnet framework because the Bot framework for done that core is still in preview. So a Few months ago, we we were developing with the full dotnet framework But we really want to use the bot with with done that car because we want to deploy it as a container in Linux So that's why we won't we want the dotnet core version So and then the most important code here about the code about the bot is this method in on turn Where you will get the message from from the bot client and then depending on the message it will redirect that into Lewis and In Lewis will send back. What is the the category of the message? and then we will redirect that into into our Dialogs and then for instance the dialogue the catalog dialogue is the one that it's also talking to the Custom vision for the image and then querying the This service with the product catalogs in the system and returning okay, okay, and then something else that I want to highlight is About Azure stream analytics so This is right now generally available and you can You can integrate machine learning and also with ML net with stream analytics So steam analytics is made so you can you can gather thousands millions of events from devices from and then through IOT hub and then in real time you could also Create queries and those queries could be also talking to your models and Depending on what you're predicting you could do different different things, right? That's one way of using Stream analytics integrated to machine learning a very typical thing would be for predictive maintenance and And then in real time you could be you could be predicting what's going on on the devices if there's an issue And and then the other possibility is and this is new This is in private preview right now, but you can request access in that URL above You you can also use Azure a stream analytics within an IOT device run it on the edge and and gather the events from the devices in like let's say you have a Raspberry Pi doing that or any other IOT edge device and You run Azure a stream analytics internally in that device and you can also and this is new You could also because this is a C sharp as well You could create a module in C sharp Which is a C sharp function and then use ML net for scoring models So any model that you created you train Outside then you could Reuse it in into within the IOT edge device and then on the fly you could be predicting if there's an issue or whatever Whatever the problem you're solving or you're predicting and then finally communicating with with the with the cloud and That's almost all that I wanted to cover So you have more resources in the dotnet machine learning page also if you want to if you want to see kind of simple hello world very Easy to get started samples go to github.net machine learning samples You can also go to the ML net page if you want to see the internals about ML net and Collaborate with the open source project and also finally you can also see these Broader sample based on microservices extending each upon containers With machine learning and AI in each upon containers AI in github as well I also want to highlight the other sessions in dotnet com about ML net you Might also have seen it The session about ML net from Ankit and goll then there's another session about quantity services Using summary in apps from Veronica and then of course this session that I'm just presenting and Yeah, so let's Let's answer a few Questions Yeah, that's a good question. So it's saying so when using When using Azure custom vision Do you need? The custom model in the file in the offline in order to you know Score for the frisbee the demo that it did or or can it also be online? Yeah, so I think I said that but yeah, it can be offline or online So when you will train all those images in Azure in the project and Azure about with Azure custom vision You have both approaches you can extract or export the zip file and use it offline or you can just Use the key in in Azure and the URL and then from your service or your web application You provide the key and the URL and you just connect to Azure to the service and then Use a model next question Can we run ML.net on Raspberry Pi with Windows 10 for IoT or Raspin OS? I think you can you can Use both. I mean basically the answer is Whatever you can run C sharp and dotnet core You can run ML.net. That's the answer. So in any device that you can run dotnet core you would be able to run ML.net because it's just a libraries or a set of libraries and and use it through a new package Another question is what kind of AI ML knowledge should I have in order to work with ML.net? Okay, that's a good question because this is precisely Kind of our Related to the goals of ML.net. So our goal is that any dotnet developer researching Or yeah with interest about AI and ML.net. I mean and machine learning could learn and do Like solve problems with ML.net, of course It will depend like you you without any knowledge about What is machine learning? You'd be able to do basic things and solve basic problems. Maybe Would be a lot about retry and See what works better for you what algorithms and so on If you have more knowledge About machine learning problems and algorithms and approaches and working with data Then of course it will be a lot easier But our our our goal is to provide also the UI so That UI tool will kind of help you or drive you when you don't know what to use because So you will just submit initial data your data set and then it will make Suggestions about hey this data looks like you want to You could use it for a regression. Maybe this is the value that you could be predicting and then if you want to do that we suggest this algorithm or this Three or four algorithms and then you can play with that you can try different approaches and then see An evaluate what's going to be better and then just use it another question is Will ML dot network on the client side of a web app with laser similar to tensorflow j s So we haven't tested that but initially it's a similar question that the previous one about IOT so On any place that you can run dot.net core initially you could run also ML.net so potentially yes another question is About do you need to define a graph statically before a model can run like tensorflow So I'm not sure about that question if you mean about tensorflow If you just send me an email and then we can get them more details. I would need more information about that What else Can you see the other question? Yeah Is there a way to for ML.net to leverage on GPUs for distributed training? So GPUs are mostly used or you can take advantage of GPUs most of all on on deep learning. So and then Initially in ML.net we target more traditional machine learning problems. So we are not using GPU for that but if you are using tensorflow for instance and and you You train tensorflow. Let's say in in Python, then you would be using GPU but you would be using just the Frozen model in ML.net. So initially we are not using GPU right now Okay, so with that we finished the session. Thanks a lot for for coming