 All right, I think it's time to get started. Are we the last time between now and lunch? Yeah, probably. Well, I mean, just not answer. Maybe. We promise we won't go long then. We don't want to hold you for before the lunch. Well, we have cool demos and we have what I hope is going to be an interesting talk on running AI at the Edge. So I'm Benjamin Cabet. I'm a program manager at Microsoft and I work on all things at Developer Evangelism as part of the Azure IoT group, right? So all things IoT, IoT at the Edge, IoT in the Cloud, IoT on country devices, more beefier devices, that would be me. And Pamela, maybe you want to quickly introduce yourself as well. So my name is Pamela Cortez. My past life, I was a hardware engineer still doing development on IoT devices. But now I work at Microsoft, focusing on IoT on how we can really move forward on a intelligent edge. So really excited to go through kind of the challenges and things to think about when building these new type of solutions that we've encountered and hopefully that could help all of you as well. Yeah, so yeah, we're going to talk basically about some of the use cases of running AI at the Edge, some of the best practices, some of the technology that's out there, some of the open source technology that you can use to basically run AI at the Edge and what it means to do some processing at the Edge and then find ways to still operate your solutions in a scalable way. Still find ways to make sure that you move your data and all your insights from the Edge to the Cloud, right? So that quickly going through the agenda but basically just talking about Edge, what is Edge computing? And in particular, what does it mean to run intelligence at the Edge? Like I said, talking about some of the technology that's out there. So we're going to talk about the Azure IoT Edge framework, which happens to be an open source project and an open source ecosystem for running IoT and AI at the Edge. Diving into some of the scenarios around computer vision, image recognition, sound analysis, predictive maintenance, that kind of stuff and demos. Like I said before. So just making sure that we're talking about the same thing here but what is Edge computing, right? And looking at those scenarios, doing predictive maintenance. You have an old rig with lots of data, probably actually lots of legacy devices like 10 or 20 years old devices producing lots of data. What if you could use this data to predict when something somewhere is going to fail? What about doing speech recognition? What about smart retail and figuring out when you have a shelf that's about to like, I mean there's a particular product, a particular skew for which you're really low in inventory and you need to order more. What about smart city and figuring out like traffic and all that kind of stuff? Well, those scenarios, there's some things that I see that are pretty common to those scenarios. There's lots of data involved, right? When it comes to like the oil rig and predictive maintenance, like I said, there's really tons of data and it's probably going to be pretty complex to predict whenever something like a particular part is going to fail. So if you need to move all the data that you're collecting on your super remote oil field all the way to the cloud to run your machine learning algorithms to just like figure out that there is this particular oil machine whatever that's going to fail, then it's not really going to work, right? What about security as well, right? What about like running, like yeah, making sure that the data that you move between your old rig and the cloud, you move only the data that's basically safe to move or if you do move and have some data flowing between your edge devices and your field and the cloud that making sure that it's secure basically. So that's when we start thinking about edge computing, starting thinking about running some of the stuff that typically these days you would be running in your cloud, start moving this closer to the edge, closer to where that stuff actually happens, right? But then it's hard, like it's hard in terms of making sure, and in no particular order, making sure that all that you're going to do at the edge, you're still going to have the same level of security and trust that you would expect from a private or public cloud. Like in terms of, I don't know, like on my oil rig and in my plant, I might have different kind of algorithms and different software, pieces of software that I want to run, maybe coming from different vendors. So how do I make sure that all those vendors maybe run on the same piece of hardware and same piece of server if you will on my factory floor, but like do it in a secure way? What about connectivity? If I am to run software closer to the edge, think again, remote old rig, the connection is going to be and the bandwidth is going to be at a premium, right? It's going to be expensive. It's probably not going to be super reliable. I don't know, maybe you use satellite. So it's not like you're going to have the same level of, and the same reliability that you would have within a cloud environment. The hardware, just like when you have a server farm at the edge, it's even more like the heterogeneity of the hardware is even more of a concern. You have, like I mentioned before, you have some devices that might be legacy devices running some funky operating systems. You may have different flavors of whatever is your operating system of choice. So how do you orchestrate all that software that eventually you want to move to the edge? How do you have that flow into your build pipelines, right? And whatever software you're going to build and create for and for running at the edge, how do you actually push that to the edge, right? So the idea at the end is, I mean, the typical IoT pattern is pretty simple, right? It's typically you have those things, those physical devices that are able to connect to the internet. So fine, they can use some kind of cloud gateway. And then all the data that you're moving to the cloud, that's pretty easy. You can just extract all the insights and all the information that you need. But what happens when, for whatever reason, you're remote, like an old rig, or you have some security constraints, you don't really want to rely on the cloud, then you want to move all or part of your intelligence to the edge. Essentially, how would you do that? Taking Azure IoT Edge as an example, that's really just one example. Think containers basically, right? All the code and all the science, if you will, that will go into doing your predictive maintenance, doing your image recognition, etc. You will basically build these as containers, right? And so you can use proprietary services or commercial services, like what Azure is providing in terms of Azure cognitive services for image recognition and so on. You can build your own containers with custom code that's going to be processing basically data streams for doing predictive maintenance and so on. And you build those containers, you put them in your container registry of choice, and then what's basically happening is that everything that you would have typically done by relying on those server-side services, you just move to the edge. And you have the Azure IoT Edge runtime, which is basically a Docker runtime, if you will, right? That's just going to take all those containers and making sure that they run at the edge, making sure that for some reason they need to be restarted. It's basically going to orchestrate all that, take care of all the confidentiality that can go in making sure that all the containers are really isolated from one another. And then you can start thinking about offline scenarios and basically moving some of the processing at the edge and just pushing only the relevant bits to your cloud for effectively making the world aware of all the data insights that you've extracted from the field. So Azure IoT Edge, and again, we're just going through this as an example of how to build and how to run AI at the edge using Azure IoT Edge as an example. It's open source. So in practice, it's effectively what I just did. You can create workloads, including really pretty complex AI machine learning kind of stuff. You create those workloads, and you have IoT Edge that orchestrates them, that runs them locally on the gateway, takes care of monitoring the health, restarting the containers that for some reason might need to keep them up to date. If that's what you want to do, any time that you will have an update deployed to a container on your registry, it would do the update. And it's open source, right? Security-wise, it also takes care of the security, like I mentioned, you only want your Edge device, your Edge gateway, especially if you're in an oil rig or on the shop floor, you only want to run those modules, those containers that are effectively trusted. So with IoT Edge, you can leverage hardware, root of trust, you can leverage secured intercommunication between all your containers so that, effectively, you have a solution for securely running your software and your IP at the Edge. So yeah, IoT Edge is, in terms of hardware, it really depends, the kind of hardware you're going to use really depends on your workload. It can be as small or as simple as a Raspberry Pi kind of device, but it can be a more like a beefier kind of Edge server, something even maybe deployed in the server, like in a data center local to the factory floor. Or it can be anything really, all the way from pretty constrained devices to really super powerful devices, including GPU-capable devices, like what Pamela is going to cover, running Linux, running Windows, it doesn't really matter. And any OCI-compatible container that you may have, IoT Edge can run, right? So again, if you've already gone through the work of packaging some of your IP into Docker containers, you can start thinking about pushing this software and this IP at the Edge. So there's lots of devices that are available out there, all the way from Raspberry Pi to NVIDIA, GPU devices, to all kinds of servers that are already IoT Edge capable, so you can check out the link. And yeah, I mean, feature-wise, you have everything you need for running whatever container you'd like. Security-wise, in terms of provisioning devices onto your solution, IoT Edge can leverage whatever hardware modules or X509-based PKI infrastructure you may have to make sure that only trusted pieces of gateways can join your IoT solution. There's lots of services from basically the AI ecosystem at large and the container ecosystem at large that can already run on IoT Edge all the way from machine learning to custom vision to all kinds of SQL or no SQL databases that just run. And of course, you will need to have hardware that can support whatever requirements, Redis or Mongo, whatever we will have. But you can leverage all this community and ecosystem. And yeah, it's enterprise-ready. You can use whatever language you'd like, of course. That's what's nice with container-based solution. I mean, you just use whatever is your language of choice. We have all the tooling, and we're going to be showing some of that as part of the demo. So what happens when beyond just Edge use cases, what happens? What is AI at the Edge, really? I think it's actually interesting to look at the different or some of the use cases and basically what category they belong to in terms of what data you manipulate and what data you use to basically do your magic and your artificial intelligence. With all the telemetry that you collect from all the IoT sensors that you have in your oil rig and your factory floor, basically one of the typical use cases for AI at the Edge would be predictive maintenance. I'm collecting the temperature and the torque and whatever on all my equipment, and I can start figuring out when something is going to fail. All things sound I can use for doing speech recognition or also for predictive maintenance. Any idea what I'm thinking about when I say predictive maintenance with sound? Think elevators or escalators. I've heard actually a story of escalators in Germany in the winter time because they put so much salt on the roads, et cetera. People just walk on the escalators and they bring lots of salt and lots of dirt, really. And so putting some microphones near to the escalators, you can listen to how the escalator sounds. And when it starts sounding pretty bad, then you can maybe send a technician to just clean it up. So that's a use case. And typically you don't want to send megabytes or gigabytes of sound streams to the cloud to do your sound recognition. Vision, even more so, it's going to be maybe even terabytes with computer vision. So for smart retail, smart city, optimizing traffic. Yeah, that's another use case as well. So predictive maintenance, how would you do predictive maintenance at the Edge using something like Azure IoT Edge? Well, typically, you can use and run any stream analytics solution that's available out there. You can use Apache Spark Streaming and Azure Stream Analytics. Essentially what you have when you start using something like Azure Stream Analytics or Spark Streaming is you can just stop writing queries. Like you have your, at the Edge, you will deploy some containers, essentially, that are extracting data from your sensors, extracting data and telemetry from all the equipment that you have at the Edge. And you can feed this data into your stream analytics algorithms, basically, to be like, A, I see that there is a trend and something's going to fail. So you can leverage IoT Edge to orchestrate how the data is going to flow from your data producers to your stream analytics algorithms. And literally, in just a few lines of code and queries, you can start doing predictive maintenance, really. And anomaly detection. Sound, you can use, I mean, there's lots of solutions out there for doing and running sound-based AI at the Edge. But you can look at the Cortana SDK, for example, or some examples that we've published, but essentially, same principle. You have some containers. And UV can have a modular solution there. Some containers that are producing sound and set sound, you would route into the containers that are in charge of extracting information, doing speech-to-text, text-to-speech. And then once you've extracted the data that's meaningful to you, maybe you actually decide that that data is going to be sent to the cloud, which actually brings us to more that you can do with things like Azure Cognitive Services, you can actually start looking into video analysis and custom vision. And yeah, I think, Pamela, you have lots to say about how to do the video analysis and custom vision at the Edge. All right, so we'll go to vision. So one of the top scenarios, Benjamin covered two of them. But vision, adding cameras as a sensor is actually a really, really hot topic and hot scenario, especially in retail, even manufacturing, fleet management. There's so many solutions and scenarios that wants to have cameras at the Edge and be able to do machine learning at the Edge. And there's a couple of reasons for that. One is because imagine that you have tons and tons of video that's being taken by this camera. And you're your gateway device. Do you want to send all of that data to the cloud? Some solutions you might. Maybe other solutions you don't because it's a lot of money to be sending all of that data, especially all the video if you're taking high res or anything like that to send all of it to the cloud. So a lot of people kind of want to save money that way and only send the important bits that they were able to maybe they were doing anatomy detection as one of their solutions. And they had an anatomy detection AI model that lives on the Edge. They're actually able to pick what data they want to send. And so that's really important for cost saving, but it's also important for latency. So a really great example that I really like is Shell, they came up with a solution with Microsoft where at a gas station, they were thinking about gas station safety. And they had all of these security cameras. And you can imagine at a gas pump, hopefully no one in this room does it. If you're smoking next to a gas pump, obviously that's a bad, bad thing. Or even if you're talking on your cell phone and you're pumping your gas, there's a lot of things that can happen. And if you have to wait for that data to go to the cloud, do your machine learning on the cloud side, and then come back to tell, let's say, the worker who's in the gas station, hey, hurry up and stop that, you might actually run into issues because every split second counts. And so that's the other reason why people are seeing vision and doing intelligent edge solutions becoming more and more popular. So Benjamin gave kind of, you know, from small devices to big devices, but when it comes to vision on the edge, one thing I've noticed is there's still kind of confusion on which devices to use. Especially if you're doing a proof of concept, doing a proof of concept to going into production is a whole new story. And it's, you know, if you're into hardware engineering or you're on deal with the hardware side, you have all of your basic things. You have to think about end of life or all of those things. But since this is still kind of a new space, you're seeing more and more devices pop up into the market. And then maybe you won't even see a device in the market yet that fits your needs because this is still a really new space. So we call out Intel, we call out NVIDIA where they are building these ecosystem of products to make it easier to do some intelligent at the edge. But I'm seeing a lot more hardware providers be able to bring that. A next question is, okay, great. I wanna do, let's say, anonymity detection in my factory floor. And I want to have high-res video that's going into my gateway device. I wanna do intense machine learning on the edge. And can I do that on a Raspberry Pi? So that is something that I hear a lot because there's a still trying of understanding of this world of AI when you're on the hardware side of understanding how big are these models. And that's something that I think of it as this new way of teams coming together and working together. I'm interested in this room, who focuses on kind of the AI side? Do we have any kind of data folks over here? Okay, I see a couple over there. Who focuses on the hardware side? Okay, a little bit more there. What I've been noticing in my previous job is when I was coding with partners and customers, those teams rarely talk to each other. So when you bring them together and say, hey, we're gonna build on this device, the machine learning experts are super excited. They build this beautiful model that cannot fit on the device. Or the device folks might not have the same terminology or be able to communicate effectively to those teams. So when, if you're at your company, start to think about how those teams can come together and work better together. I'm even seeing a huge increase in Python support because there's a lot of AI models being built in Python and C. And because of that, there's a common language that both the hardware folks and the machine learning experts can come together and kind of have a way where they can have the same language even. But there's different toolings, different mindsets of how to engage an issue or a challenge. Other thing when selecting a hardware is thinking about storage. Storage is huge. If you are a company where you're on an oil rig, we always use that one because there's a lot of those partners and customers. But if you're on an oil rig and you deploy this edge device, has this beautiful machine learning model on it, it's, you have that deployed, but you have only one connection through satellite once every two weeks. Where's that data gonna be stored? So thinking about that, and that's one reason why I put the data box edge. There's plenty of other devices out there where it has a lot of storage and also a lot of processing power to do these ML models or machine learning on the edge. So I kind of talked about bringing the two teams together, but on the AI side, there's a lot of companies right now, a lot of startups and even the big tech companies that are noticing that we have a shortage. We have a shortage of hardware engineers where we have a shortage of these AI experts as well. So what about these teams who maybe not have a large team to be able to develop these solutions? So now you see these tech companies being able to provide services like Cog Services where it actually has the AI model and you can take it, deploy it on the edge and you don't have to worry about the upkeep of that model because you have would say the Microsoft research team or you have folks who've been working on the AI for a long time do that for you. There's also the option of creating your custom model, taking models that maybe you find through the different communities and making it and applying it to your solution instead of reinventing the wheel. And that's really popular and I think that's gonna be more and more as this community builds, it's gonna continue to be a lot of support there or just build your own. And that's always great, but it's things of that lifecycle of that model and upkeeping it as well that you need to think about. So Benjamin mentioned about containers and what's awesome about putting your AI model in a container and then deploying it on the edge is the fact that would say I have a container that has anatomy detection on it. I just deployed it on one of these devices. This is actually a Justin Nano. I'm not being sponsored by NVIDIA, I wish I was because I would love to get all of their products. But this is a small, small device. It's only like, I think $99, don't quote me on the price, is really cheap and then there's this big, big hardware device. This is a PX2, it's meant for autonomous vehicles. A container that can run on this device can run on the same device. And that's really important when you have multiple gateway devices. Also, if you wanna use that same AI model, I would say your company does something as a retail solution, but it's an anatomy detection and it also can deploy in a other solution in a different scenario. You can still use that container that has that type of information that be useful in the other solution. So it's a way for you to not have to rewrite code a bunch of different times. So I think that's really important. This slide is showing there's really two architectures you can think about. One, separate containers. If you just wanna run all of your solution into one container, what are sort of the benefits of that versus kind of separating out the containers. It really depends on your solution. Some people like separating out the containers because they can reuse bits and pieces of it in multiple solutions. So it's fully up to your different solution. All right, so I'm gonna go ahead and start on the demo. So this is image recognition using IoT Edge which is an open source project with Nvidia Jetson. So what I'm gonna do, first find the mouse. Perfect, all right, I got the camera here. You can see there's tons and tons where it's picking up as person, person, person, chair, chair, chair, chair. And it's able to look at the AI model which is the YOLO AI module that we have container-wise that's running on this device. Yeah, so I don't maybe just set this cup here and it can identify that it's a cup. And so what you're actually able to do is do all of that machine learning on Edge. So it's just running on this device, on this particular device. You can actually visualize that data and send it to the cloud, the important data that you want. So that important data was to say, hey, I actually was able to see a bunch of people. I was able to see a cup and identify how many times you saw that and where in a point of time. So you can send the important data up and I think that's really, really powerful. One thing that will also a challenge that people are going to face when it comes to a tellage at Edge, especially with privacy, I'm a big privacy nerd. So I'm going to just insert this in really quick is that I just took a bunch of the camera, looked at that person, person, person, but I didn't spend time identifying who you are. And so for example, if you're in Europe and would say you are doing an example where you're doing video training and you're walking in the public streets and you're capturing people's faces, if that data lives on the device and you're thinking about GDPR, those are things you need to think about. And how do you protect that data or how do you, before the data is even stored on that device, blur out the faces or how you store that data. That is something I think is going to really be a big challenge that a lot of companies are going to face when they start thinking about privacy and ethics when it comes to the AI models. So that was a demo there. That same container could be deployed on this device over here. Let me go back to my presentation again. So this is, so this looks great, you saw a cup, but what is the actual real world scenario for that? Cause I mean, I think we've all seen plenty of models that show it's a banana or a cup or anything like that. But a great scenario would be actually a shipment company where they are looking at their different packages and you could see that you have the same container run on all of these cameras, which is really nice because you didn't have to flash the camera with all different code or anything like that. You can actually, since you have the container on the cloud, Benjamin mentioned the container registry. If you have that container in the cloud, you could deploy it to mobile devices without having to sit there and have every device individual, and that's super powerful. So you can see with this video that there's actually one of the belts, belt three, that the package is not actually correct. So it's actually telling you this is the wrong package and all of these cameras are running the same container. So then that goes back to cloud, then you can actually put that into your business integration and let someone know on the factory floor that it's the wrong package. And that $100 device that Pamela was talking about, it can actually process multiple video streams, like 1080p video streams at the same time, like one IoT Edge instance, one set of containers, several video streams, all that processing can be done basically with just a couple hundred bucks at the edge, that's pretty cool. And one of the things you'll start seeing in that hardware ecosystem is those powerful devices, but you're also gonna see really tiny, tiny devices, like TinyML, I know we're working on stuff with Microsoft Research around that, you're also seeing even TensorFlow being ran on smaller microcontrollers, and so that is gonna be even more powerful where Intelligent is gonna be any part of those devices. So I kinda mentioned this before about your storage on the Edge, thinking about that, like for example, if you work with Azure before you might have heard of Blob Storage, but as a type of storage, you can actually run that, same thing that you're doing on the cloud, you can run it on the Edge. So being conscious of time, I wanna leave some time for Q&A, some of the resources for you is if you want to actually run this demo, here's the hands-on lab that you can run if you get a Nano device. We also have a workshop that's running two more times today that you don't have to register, so if you didn't register on the site, don't worry about it, it's the AI dev kit where we'll be over there so you can learn how to make a model, deploy it on the Edge, and it's workplace safety, and it's really fun to actually play with those cameras. And then we have plenty of other resources too. I'll do a really quick plug, sorry Benjamin, I'm part of the IoT show, so I do a lot of videos. So if you're curious about IoT, definitely check out the IoT show, there's Olivier and myself, we try to educate people about IoT all the way from the hardware side, how to do provisioning of your devices all the way to, what is kind of the new tech that's coming out as well? All right, is there any questions that folks have that, did you? No, I got way too excited because you moved your arm. All right, well if you have any questions, feel free, we'll be here next door and then at the booth to answer any questions. So it's a really exciting space and being able to work across teams and a company, it's just really exciting to have different types of minds come together to build these solutions. So thank you so much for attending. Thank you. Thank you.