 Hey everyone, we are back with .NETConf 2018. Veronica, how's it going? I'm good. Hi. Hi everyone. Yes. Well, we have Veronica talking about Cognitive Services and Samarin. Tell us a little more about that. Sure. Oh, hello everyone. Today, we're going to talk about Cognitive Services and Samarin applications, and I'm Veronica Gilles Nicola. So, here's a little bit about me. I got my master's recently and my master's in information technology. I started my career, yes? Oh, I'm sorry. So yeah, back to information about me. I started my career as a quality assurance engineer, and then I moved to development. Now I have more than six years of development experience. Mostly, I'm working with Microsoft technologies like C-Sharp, .NET, SQL. Recently, I started playing with Samarin. But for my day job, I'm working with .NET, the CMSs like Episerver and Sitecore. At some point, I was using PHP, MySQL, Drupal, and Ruben Rails. I'm not working with those technologies right now, but maybe I'll get back to them at some point. And my hobbies are dancing, traveling, and aerial yoga. So, here's the agenda for next probably 45 minutes. I'll start with artificial intelligence and machine learning, and I'll tell you some basic information about them. Then we'll move to basics of Microsoft Cognitive Services, and then I'll tell you about groups and individual services there. Then we'll move to specifics of integrating Microsoft Cognitive Services with Samarin and Samarin forms applications, and then the best part, custom vision service demo. So, definitely stay till the end. So, let's talk about artificial intelligence and machine learning. All big tech companies are investing lots of time at my inter-artificial intelligence and machine learning. Those are big things currently, and we all kind of understand what's going on there. We have lots of tools available, but sometimes it might be confusing how artificial intelligence relates to machine learning. Are they working together? Are they completely separate? Let me actually clarify that for you. And I'll start with artificial intelligence. So, technically, artificial intelligence has more than 50 different definitions. And in general, it's something that we as humans are good at, but machines are not. So, we are trying to mimic human intelligence, using logic, e-stand rules, decision trees, and other cool stuff. Next, we have machine learning, and machine learning is a subset of artificial intelligence. And basically, it's a combination of statistics and algorithms. So, when we use those techniques, we enable machines to improve our tasks with experience. And next, we have deep learning. And deep learning is a subset of machine learning. So, when we have lots of data, we are trying to organize it and create neural networks with data nodes. And using those neural networks, we enable our machines to perform more complicated tasks like speech and image recognition. And cognitive services are based on deep learning. So, what are Microsoft Cognitive Services? I know yesterday we had a couple of sessions about machine learning and cognitive services. But if you didn't have a chance to connect, then let me tell you a little bit about them. Microsoft Cognitive Services are sets of APIs and SDKs that you can use to make your applications smarter and make them more interactive for your users. They are really easy to use. You just need to get a key and write a couple of lines of code. And you are getting access to all those cognitive services through REST APIs. All those cognitive services are really flexible. So, it doesn't matter if you are a hardcore iOS developer or you can't leave without Python. You can still use Microsoft Cognitive Services and use all their available functionality. Also, all those Microsoft Cognitive Services are tested and created by super smart people from Microsoft Research, BIN, and Azure Machine Learning. Those people actually got their degrees in machine learning and I'm sure they know what they're doing. So, we don't have to get our degree in machine learning and spend years digging those topics. So, thanks to those people, they are great. Also, online, you can get quality documentation about cognitive services. They have sample code, huge community support, really awesome examples online. You can check MSDN, you can go to GitHub, and if you notice that something is not right or you want to improve something, you can actually get the code, fix it, and submit a pull request. I'm sure Microsoft team will be super happy to have your code there. Now, let's move to groups and individual services. There are officially five groups of cognitive services, but I personally think there are six of them and I'll explain you why a little later. Currently, if I'm not mistaken, there are 24 individual services available. Unfortunately, I don't have enough time to tell you about each and every individual service, but you can go ahead, check the website, and see what services you are interested in and what services you would like to use. We'll start with vision services group, and it's all about visual information. First one here, really interesting, computer vision API. This service actually can detect objects on your images, and also it can perform optical character recognition. If you have written text somewhere, you have a picture of it, you can just upload that picture to the service and it will translate it to actual text. Next one here is face API. And recently, they integrated emotions API into face API, so now it can detect specific people on your images, and at the same time, check their emotions. So you see how happy or sad they are. That is really useful. If you check all those vision cognitive services, you didn't find anything that can be suitable for you, or you have specific set of images where you want to search and get specific information from those images, you can go ahead and use custom vision service. And using custom vision service is really interesting. You can feel yourself like you're actually machine learning an expert, because you can load your images, you can train the model yourself, you can expert it, and we'll dig deeper into that later during the demo. And I'm sure it will be really straightforward and really easy to use. And here is the example from cognitive services website. And this example of computer vision API. There you can just go to cognitive services website, find that specific cognitive service computer vision API, you can either upload your own image right on the page, or you can use images that are provided there on the page. And then you can see the output, you can see JSON, and you can see what computer vision API returns to you. So you don't have to actually get the key, you don't have to write any code, you don't have to do anything, you don't have to even use postman. You can just go to the website, see the JSON that the service returns. You can see lots of information there, most of applications don't need that much, but if you want, you can just go crazy and use all that information. Next group here is speech, and it's all about spoken information. There are several of them that I'm really interested in for me personally. First of them is speech to text. This one is pretty straightforward. So your users can use your application, maybe it's a bot, can talk to your bot, and then your bot is gonna use speech to text API to translate speech to text, and then to maybe pass it along so other tools can analyze the text. Next one here is speech translation. So this service can actually translate speech to other languages, and actually the whole speech suite was used in new add-on for PowerPoint. It's really convenient if you travel into a foreign country and you don't know language, but you still need to present your PowerPoint to have your slides. So that add-on can actually translate your slides or you can set up subtitles. So that add-on will show translated subtitles right on top of your presentation. So I think these can see what you are talking about, or they have an opportunity to scan a QR code with their mobile devices and see those subtitles right there. They can also ask your questions on their device, from their devices, and you'll see those questions, you can answer them. Really useful tool. I definitely recommend to check it out. And here's an example from Cognitive Services' website, an example of speech to text. Here you have options either to play samples that are provided on the website or you can record your speech and then you can see how the service works. You also have an opportunity to select language there. Really useful. It's all on the website, all online. You can go and check it out there. And next group here is language. And that is all about language analytics. And first Cognitive Service here is Lewis. I'm sure most of you heard about it. It's really popular right now. It's widely used in bot design and development. Also it's used in virtual assistant skills creation. Language understanding intelligence service or Lewis can understand intent and can parse those little variables of maybe time and place from your user's preferences. So your user can talk to your bot. Then you use speech to text, you're translating, spoken word to text. And then pass the text to Lewis and then Lewis will understand what your user wants and based on that it will return information. You can train Lewis, you can upload sample intents and sample variables. So that is really useful and definitely check it out. I personally used it for Cortana skills creation. It works great. Then next one here is bin spell check API. This service is really useful. I think everyone should use that. I have lots of spelling mistakes all the time. So definitely go and check it out, especially if your users need to write a lot and it's important for them to spell everything correctly. Next one here is text analytics API. And this Cognitive Service provides basic analytics for your text, like number of letters or number of words in your text. That can be really useful if your users work with text a lot. Okay, and here's the example of Lewis. It's also from the website, Cognitive Services website. You can either type a command or you can pick one of predefined commands there. And it's really interesting here because you are not only seeing the adjacent response from the service, you also can see how it actually works. So here I picked switch all lights to green and you see that on the left the light was turned green, right? And also Jason here. So you know what you're working with, what kind of data you're getting from the service. Okay, so next is knowledge group. And here we have only one cognitive service right now and it's Q&A Maker. Q&A Maker is also widely used for bot design and development and also virtual assistant skills creation. Q&A Maker you can use when you know that your users mostly ask you about same questions over and over and they want to get same answers again and again. You probably already have a Q-page on your website or you have a PDF document somewhere with all those questions and answers. So you can just upload the PDF document with questions and answers or provide a link to your page, your FAQ page and then this cognitive service will parse it, bleed it into questions and answers and voila, your users can go talk to your bots, type their questions and get their answers right away and you don't have to do a lot there. And here's an example from cognitive services website. They have questions here, some examples and on the right you can see preview of the answer. Also you can get Jason's response so you see what kind of information you're going to work with. Okay, next group is search and that's the biggest group because all cognitive services, they probably started with Bean Research team and then they grew into actually cognitive services as we see them right now. And here you have all kinds of searches. You have Bean Image Search and News Search and Web Search. If you need some specific information in the search or you wanted to search through specific data set or want to create some kind of customization there, you have access to Bean Custom Search there. Definitely go and check it out. Okay, so we went through all five main groups of cognitive services and now that's the sixth group. It's called Labs. I personally think it's official group. It should be an official group because that's the future. Going there and checking Labs, you can see where cognitive services are going, what can be next step there, what researchers are currently working on. All those labs are experimental so I don't recommend you to use it in production environment, but definitely go and check them out. Definitely provide feedback and maybe submit pull request with your fixes and your code. So one day one of those labs or maybe all of those labs can become actual cognitive services. And there are a couple of those I really like. For example, Project Gesture. If you're using Project Gesture, your users don't have to use keyboard or mouse anymore. They don't have to use the touch screens anymore. That is insane. They can just use the camera and using gestures. They can interact with objects on a screen. They can enlarge those objects. They can move them around. That is real cool and they have awesome video on the website so you can see how it actually works. Next one here is Project Event Tracking. And this lab actually provides information about events based on Wikipedia entries. Another really interesting and really, really useful one is Project URL Preview. So when you use that service, you're enable preview for pages that your users are trying to load. So they are not loading any malicious content. They are not loading any adult content. They can preview the page before actually loading it. So this one is really useful too. And there are 13 labs available currently on Cognitive Services website. And I think six are available at ailabs.maxhole.com. Definitely go and check them out. So now you know everything about Cognitive Services. And I'm sure you're ready to just start and go crazy. Check them all out. Work with all of them because they all are awesome, right? Hey Veronica, there's a question on the chat that I think it's a good place to ask about it. Okay. But wording 94 is asking, are Cognitive Services provided by Microsoft maintainable? What if, for example, speech to text service needs more learning on the Arabic language? How can we maintain that? Is that something that Microsoft needs to fix or add? Or is that something as the end developers we can do and cooperate? So as the services reach a certain point, how do we, do we wait for Microsoft to add things? Or as something as a community, do we add them? So actually it's a good question. There are several options, but definitely use them all. A little trick there is the more you're using the service, the better it gets. So if you're integrating it with all your applications and you're using it a lot and you have a specific group of users, definitely those services will learn from experience and they will get better. Also, if you have some questions or some issues with code there, then definitely submit your questions on the website or submit your full request if you want to contribute. Microsoft is really responsive to comments and concerns. So that's also an option there. Thank you so much. That's a great answer. Any other questions? No, no questions right now. Go ahead and continue on. Well, this is really, really great. Everybody's engaged. Okay, awesome. I'm glad you liked it. Oh, and by the way, McGuarty94 says thank you so much for the answer. No problem. Okay, so all those cognitive services are available in Azure. So if you have Azure subscription, you can go to Azure portal right now and just search for specific cognitive services or check AI and cognitive services group and see all those services there. There is also a free tier available for all those cognitive services. It depends on each service. How many transactions allowed there for free? For example, for computer vision API, free tier includes 5,000 transactions per month. But before signing out, definitely go and check the documentation. We have a calculator there so you can see how much you're going to spend on this or that service. And actually, I want to tell you that 5,000 transactions per month are more than enough for testing and playing. Definitely not enough for production environment. Once your users start using your app, then all those transactions are flying just like that. And if you for some reason don't have Azure subscription, then go get it. But seriously, you can try cognitive services without Azure subscription. You can go to cognitive services website and try all those services for free. And free option is available for a month. So you can play with those services. You can get the same keys from the website as you can get from Azure portal. And another cool thing here is custom vision service. It actually has mobile model experts. So you can export your model as TensorFlow file or Coremail file. So you can either access custom vision service with your custom train model using API like any other service, or you can download your pre-train model and put it on your server, for example. So even if you don't have internet connection, your model will still work and it will return results to your users. Okay, and here's a big picture of Microsoft AI platform. You can see that cognitive services is just a small part of it and it pre-builds AI. Also have access to conversational AI like bot services. But if you're really experienced with machine learning, you know a lot about it. You are ready to build your own models, ready to train them, play with them, go crazy. You can use custom AI and it's Azure machine learning. Also, Azure provides the whole infrastructure for you to work with like AI on data, all kinds of databases, Cosmos DB, Data Lake, also AI computer power, like Spark and IoT Edge, different kinds of CPU, GPU, everything that you might need. They also have pre-build machine on Azure with everything you might need. So you don't need to set up anything yourself, just spin up virtual machines and work with models, build your machine learning models, train them, work with them, everything. Also, you can have access to coding and management tools, like real tools for AI and Azure Machine Learning Studio. And you have access to deep works like C&TK and also to some frameworks like TensorFlow and Coffee. So that is all about cognitive services specific part. Now, I'm going to move to specifics of integrating Microsoft Cognitive Services with Xamarin and Xamarin Forms applications. So, I hope you all know the difference between Xamarin and Xamarin Forms. In Xamarin Forms, we're trying to share as much code as possible, but sometimes we need to tweak it here and there. If you want to tweak it specifically for a device and if it's a front end change, then you can use device class and here I have an example of it. If you want to access platform or environment-specific functionality, then you can use dependency services. And dependency services give you access to all kinds of environment-specific functionality for Android, iOS and Universal Windows platform. You can access camera and location and settings, all that awesomeness there. Pretty much, you need to connect to camera and microphone and other device-specific parts in order to work with cognitive services, because usually you are providing images or you are providing some information from microphone like user speech or any other information, implication-based information to the service. Okay, where to start? So, now if you're using, if you're building your app with Xamarin Forms and you're trying to share as much code as possible, now you don't have to use dependency services for everything. Lots of really talented developers went out there and created common APIs, so you can have access to microphone and camera and settings and all other stuff. You have all a list of common APIs online, they're all on GitHub, you can talk to developers, you can maybe submit your pull request with your code changes and updates, so that is really awesome. I used Xamarin plugin media that was created by James Montemagno. This specific plugin gives you access to camera and also gives you access to your user's media folder on their device. I used that one when I worked with computer vision API, but you can also use it with custom vision services and whenever you want to connect to camera. But now, we have Xamarin Essentials and I hope you checked James Montemagno's session two days ago. He was talking about Xamarin Essentials, they're still on preview, but I heard that they're releasing them soon, so stay tuned and Xamarin Essentials, they're just one day allow one new get package for all your device specific functionality. You can go to website, you can check what Xamarin Essentials library actually has and what kind of functionality you can access using Xamarin Essentials. If you are using cognitive services and you try and connect them to your application, you definitely need to get a new get package, right? And in order to find those new get packages, you need to search for Project Oxford. Project Oxford is a regional name of cognitive services and here you can see that I used Project Oxford dot vision for computer vision API, but you have speech recognition and common and emotions, they don't have emotions anymore because they're part of face API, but I'm sure you have Project Oxford face API there. Okay, that's it and let's move to the demo. So here you can just, I'm going to show you custom vision example and how you can use custom vision service with your Xamarin applications. I already have a couple of projects here, but when you go first to that website, you can create new project, type name of the project, type description, and then you can select resource group. It's actually, for me, it's connected to my Azure subscription, but you can create new resource group. You can get limited trial. Limited trial is perfect for playing with the service and seeing how it all works. You have two project types here and project detection is on preview currently. You can create different classification types like multi-label and multi-class and all kinds of domains here. If you are planning to import your custom pre-trained model, you need to select compact because if you have all of those domains without compact, then you won't be able to export those models, but there is a little secret there. You can always change your domain if you want to. And here I really have my project setup. It's called Kats, so if you are not a Kats fan or you are more like a dog person, I'm sorry. I'm a big Kats fan, so I created Kats example here with all kinds of Kats. The only part is that I'm not really good with their breeds, so I just classify them based on their appearance. For example, here I have bicolor orange and you click it. You can see all of them and I uploaded all those images to the project. You can do it with that button and images and you have your images here. I have the whole folder. For example, this one, you open it and then you can tag it. You can write a couple of tags if you're trying to classify them by a couple of options there. I have only one tag and here I have predefined tags because I already used them. I tagged it with Angora and I'm uploading that one file here. Done, so now it's part of my set. You can see all your tagged images or your untagged images that maybe some images got left out and you didn't tag them, so you can filter them and tag them here. You can select images, you can add tags here and when you actually upload all the images and tag them, you can train your model. My model is already pre-trained and then you can quick test it. You can get an image here and it will return you the information, so most likely it's Angora. Here you also have performance overview. You see precision information and recall. Hey Veronica, there's a really good question as you're demoing this from Hegar Delgada. He's asking will it be possible to add images that we don't have in our PC? Do we have to download the images and then upload them or can we pull them from an API or from some other data source to be able to process them? So I believe you can only get them from your PC. I'm not 100% sure but looks like that's only from your PC. Gotcha, so right now as the way that the product stands, if we need to do any of those checkings or anything like that, we have to manually do it at least and train the system so we can then get the information that we need, right? Yes. Perfect, perfect, thank you. No problem. Okay, so moving forward, the performance. So here you can set up your threshold, you can create your iterations, then you get information about precision and recall and also some analytical information about performance per tag. And here you can see all your predictions when you actually tested your model. And here we're going back to performance tab and here you have expert button. So you expert your pre-trained custom model as CoreML for iOS, as TensorFlow for Android, as Onyx for Windows ML, and as Dockerfile if you want to use it on Azure IoT Edge, Azure Functions, and Azure ML. I already downloaded my TensorFlow model. So I'm going to jump into my visual studio here and here I have my Xamarin custom vision solution and actually that is Xamarin Android application. Again, I want to just make it clear you can either expert your model so it can be stored on your user's device or on your specific server or you can just access your custom vision model through API. Since I downloaded the model as TensorFlow, I created here Xamarin application for Android. And here in assets folder, I actually put those two files for TensorFlow, model.pbe, and labels.txt and all came from my custom vision model. And here I wrote really simple, really straightforward code. I'm creating a structure here with label and confidence because that's what we're going to display. And then I am connecting to camera here. So you'll see that there'll be a screen, really simple screen with just pretty much one button. You click, you take a picture, and then you actually send in a picture to your model. And here I'm actually using, let me show you here, I am using Android TensorFlow library. There is also a couple of other TensorFlow libraries for csharp developers like one created by Miguel de Casa. It's really popular. I don't think it currently supports Xamarin, but I'm looking forward when it actually starts supporting that. And here I am getting back to my code. I'm actually getting the model and opening the labels file. And I am working with the image that I got from my camera. Here we are normalizing the image. Actually, TensorFlow requires images to be normalized in a specific way. So with specific size and some specific values here, you can resize the image, get those values. That is pretty straight forward. Then you are actually feeding the values to your model. You are feeding your image to the model. And then you are analyzing it. And from the model, you are getting back labels, meaning those tags that we saw in the portal. We're getting labels and we're getting a level of confidence there. And then in order to get the one and only answer there, we actually sort those results by level of confidence. And we're returning only the one that the model is more confident about. And actually, let's see how it all works. Let me start it here in my Android M leader. Unfortunately, I don't have any cats right now next to me, but I prepared. So I bring in a couple of pictures of cats and we'll see how it all works. Okay. Okay. Hi. If you forgot how I look like, that's me. And that is my cat today. And I'm gonna take a picture. Okay. It's better when you have actual cat. I don't have one. So we are working with whatever we have for it. Okay. So the model thought that that was by color orange and the confidence is pretty low, 3.3%. It's not a real cat. It's just a picture. Now we have, I don't have enough light probably here. So the confidence is pretty low, but the model returned the information that that was by color orange cat. And that is pretty accurate, right? Even when the confidence is not that high. And I think I'm ready for more questions here. Do we have any questions online? No, we do not. Go ahead, continue. This is really great. We were just talking about the obstinance of the demo, how, you know, you brought the picture up and it picked it up and it had all the confidence and everything. This is really, really awesome. Thank you. Let me ask you this. As someone who doesn't do Xamarin, how can I quickly get started this outside of these Nougat packages? You know, how can I, is there something that I can just download or clone from GitHub and be able to get started on this? How about, you know, go about doing something like that? Yeah, it's a good question. So actually there are lots of examples online, lots of articles, lots of GitHub examples. I have my personal GitHub account. You can go and get that project that I currently showed, just showed a couple of minutes ago, a couple other projects also. There are projects available from people who actually work with it with Xamarin every day. They're really awesome. Also, there are lots of courses online, like in Xamarin University courses or plural site courses. So lots of information available here. I have a couple of useful links. So you can see that is Collective Services website. And you can see documentation and an article about optical character recognition. Also, if you want to do more mobile development, even if you do it with Xamarin, you can take iOS and I enjoy specific development courses so you know more about environment and just some information about me. So follow me on Twitter, send me all your questions and reach me through LinkedIn. Yeah, so I started answering the question and then I just went crazy. Sorry. Sorry, I was just unmuting, pushing all the different buttons to make the magic happen with this streak. Thank you so much, Veronica. This has been really great. And all of you out there who are watching us live on Twitch, we got more stuff coming up, tons of sessions. So please get engaged, ask your questions on the Twitch channel so that we can monitor them here and relay them back to the speakers. Any closing thoughts? That was great. Thank you for the opportunity. I hope everyone liked my session. And if you guys have any questions, definitely find me on Twitter. Send me your questions and we'll chat more about that. Thank you so much, Veronica. And thank you guys for watching. Thank you. Right back.