 Hi everyone. I'm Ryan Erickson from Laird Connectivity. I am in our products group at Laird Connectivity, and what we do there is simplify wireless connectivity. I know there's a lot of connectivity there. I got a lot to cover today. I'm going to breeze through some of the simpler slides. We're going to get deep down into this, so I'm going to get going. Hopefully have some time for questions at the end. Today we're going to talk about how we use Zephyr to solve the problem of device management of an IoT solution. How do we manage gateways and end devices from the cloud? Configuring those devices from the cloud so you have a zero physical touch of the devices and then remote monitoring and debug to ensure your fleet is operating the way it should. Our Canvas Device Manager is our solution that we developed. It consists of firmware and cloud software. As part of this solution, devices need to be factory provisioned. You need to be able to software update them, configure them, monitor their health, be able to diagnose them on the fly, and scale your fleet. None of this is useful unless you have telemetry enablements. You want to see some data for your application. What are the pieces of the system here? We have a gateway and we have end devices. Along with the cloud side, we partnered with our partner at JQ to give us our platform in the cloud to manage your fleet. The firmware running on the gateway needs a device management connection to get your configuration, your health monitoring, all your diagnostics. Alongside of that, call it your application firmware. It is one firmware image, but in this sense the application firmware is enabling that telemetry, separate telemetry connection to get your data up to the service of your choice. Again, just stressing the fact that devices ship factory provisioned. You can out of the box connect to the cloud. Real quick, the hardware pieces of this system are our Centrius MG100 gateway. It is an LTE and BIOT gateway with Bluetooth. So bridging BLE data to the cloud. Our end devices in the system are our Centrius BT610 IO sensors. It is an IP67 BLE sensor. It has four different sensor ports, quite configurable for different kinds of sensors, analog inputs, digital inputs, outputs, or serial bus type applications. So when developing this device management platform, obviously there's some of the high level requirements here are security number one. Using open standards and protocols, this is important because it gives us flexibility in transitioning to other systems if needed in the future. No touch deployment and management of the fleet and health monitoring. So in order to get into how we did the gateway all the way down to the end device, I need to talk about the protocols that we used. So as sticking with part of our requirements to have standard protocols, we chose to use the Lightweight MDM protocol. The Lightweight MDM protocol is an application layer protocol built on top of co-app and then built on top of UDP. It has a schema based object model that represents objects and underneath those objects resources associated with your device. So in the overall solution you have your gateway connecting or managing all the end devices. So Lightweight MDM is representing objects or objects being important things you care about on your device. Underneath those objects you have resources. The objects and resources have defined IDs per the specification and it has flexibility to create your own if you find something that you don't, that the spec has or doesn't have and you want to do. The standard objects include stuff for connectivity monitoring, firmware updates and what not. The base schema of Lightweight MDM has four levels. You have your object ID at the top. Per object ID you have a number of instances of that object in certain cases. Then per instance of object one or more resources and then underneath that one or more instances of those resources. So you can think of it like a directory structure going from most generic and getting more specific. So going along to show an example of this. In this case we'll look at object three. Object three is the device object per the specification. It is a required object when you implement a Lightweight MDM device. So object three has a number of resources underneath it. In this case we're showing how you would represent resource two. So at top level slash three slash zero for the first instance of object three and per the spec it is specified that only one instance of object three is allowed on your device. So it will only ever be three slash zero and then whatever resource you want to access underneath that. So in this case we want to look at resource two. Resource two happens to be the serial number. So in this case from the cloud you would query hey give me three slash zero slash two. It will send you back the serial number of the device. At the bottom here is the link to the OMA registry. This is where you can go to look at all the official objects in the specification. So now that we know how Lightweight MDM is structured we can talk about Lightweight MDM gateway as part of the spec. Lightweight MDM gateway takes the same structure but adds a prefix on the front. So the Lightweight MDM gateway can use these prefixes to identify nodes that the gateway is talking to. So everything works the same. The node just gets prefixed on the front and the gateway will see that and know what end device to talk to. So object 25 is the object for Lightweight MDM gateway. So each there will be multiple instances of object 25 representing each end device that the gateway sees. The server can then query these object 25 instances to discover the end nodes and know what their capabilities are. So resource zero under object 25 is the unique idea of the device. Resource ID one defines the prefix for that device and that prefix is what gets added onto the front in order for the cloud to be able to interact with that device through the gateway. And then the third resource is the device objects which tells the server what the capabilities of the end device are. So those device capabilities are specified in a certain format and this format is called the core link format. So this is defined by the spec. There's a specific, it's a string and there's a specific format to it that lets the gateway then query the end device and construct that string, throw it in that resource, and now the server knows what that end device is capable of. So let's see here. So when a end device registered with the gateway at that point, that's when the gateway queries all of the capabilities of the end device and fills that in as kind of the part of the registration process. So here's a, okay that's big enough, here's a quick diagram kind of showing where we're at here now. So we have our gateway up at the top. It itself is running its own lightweight MDM client so it has its own objects and resources so the gateway itself can be managed. So then to diagram kind of the stack here of how we're going to talk to our end device, we have lightweight MDM co-app and then we're going to have some sort of RF protocol to talk to our wireless sensor. In this case it's going to be BLE and we're going to go into how that's done. So on the end device it is running its lightweight MDM client as well and it has its own objects and resources to represent what it's capable of. So in order to make all this useful, the first piece you need is a server. The server needs to provide smart support for Object 25. This allows applications to interact with the end devices just as a user would interact with the gateway itself. So the gateway instantiates dynamically instances of Object 25 based on end nodes it sees. Then the platform is going to provide similar functionality for each device that you want to manage so it's kind of a seamless experience. In the end you don't care that you're managing a gateway or an end device although you do want to know that but in the end you want to be able to firmware update each of these devices and do it reliably. So I'm going to take you through some of the scenarios of how the devices are talking to the cloud. So first off it all starts with the gateway registering. This is registering with the server so the server knows it's out there. So the gateways obviously usually are an IP enabled device. In this case we're talking over cellular connection so it can talk directly with the server. Again the lightweight MDM messaging is at the highest level underneath the co-op protocol is used for security we're using DTLS with UDP down at the lowest level. So the gateway comes online it gets on the network it sends up its registration that's the term used in lightweight MDM to connect to the server and let the server know it's there. The lightweight MDM server will then of course act that and now the device is registered and connected. Keep in mind that we're UDP connection so much of the protocol is handshake send receive type responses. So then if you want to do something with the gateway of course you're going to initiate a request from the server the gateway will respond to that request and give you its information. Now we move on to the end nodes so the gateway has to have some smarts and we're going to get into that but at a high level you have your end device and each time a new end device is seen by the gateway we got to register it. This is a little bit different so the end node has its own lightweight MDM client as I mentioned and its own capabilities. So when it comes online it's going to use the same protocol here right we're trying to keep use standards. So the end device is still going to use lightweight MDM and we're going to get into how then we send that lightweight MDM packets over an on IP interface. First the end node end node is going to broadcast hey I'm here I it has a flag in its advertisement saying you know I need you to do something I have data for you. Then the gateway will say okay yeah this is a new device I don't have you in my list I'm going to request your core link string so I can get your capabilities. The end device will respond and that then completes the registration of the end device because now the gateway has all the information it needs it has the device's unique ID. It can then generate a unique prefix for this device and it has its core link string so we can fill in all the properties of the object 25 instance that we just created. And then once that object 25 instance is created a notification gets sent up to the server so none of the server knows oh you have a new end device there's a new instance of object 25 here and now I know about it. Once your end device is registered interacting with it is for all intensive purposes transparent to the user it works the same way as you would talk to the gateway you're going to send the request down like I want to read your serial number. The gateway is then going to say oh well I'm not connected right now right this is a wireless device it's sleeping to save battery so it's going to say hey. I need to connect it initiates a connection response and we get connected and then the data gets sent down to that end device the end device responds the gateway receives that response and then relays it back up to the server. After a certain amount of time of inactivity the gateway will disconnect and let the end device go back to sleep obviously that's really important for battery life. And the last case if there's a notification that needs to be generated from your end device your end device is going to use its broadcasting mechanism say hey I'm here I have set a flag that I have data for you. This then triggers the gateway OK I got to spin up this connection so we can do some data transfer. And then once the connections established that end device can send its notification and again the gateway will relay it up to the server. And again after a certain time of inactivity we disconnect and let everything go idle. So how are we going to do this. This is where the spec does not define how to do this. It gives you object 25 and specifies how to use object 25. So in order to do this we developed what we call our lightweight and gateway proxy. So it's going to be responsible for that secondary RF connection and and managing all of that. So again request coming down. It's a lightweight message which is effectively co-app. So we need to be able to take that co-app message as is. We need to be able to send that over in this case Bluetooth. So we're we chose to use MCU manager SMP which is a protocol in Zephyr call it standard standard for Zephyr. But we chose this because we didn't want to invent some other way to do this right. There's already something there and we could reuse it. So what happens right we just take we add that SMP header on top of the lightweight MDM message. So that way we can send it over BLE. Of course the end device knows OK this is a SMP message. I can just take that header off. I have now the raw lightweight MDM message. That message should get sent down to the lightweight MDM client that's already in Zephyr and it gets handled. Then with our response a co-app message is generated from the lightweight MDM stack. We got to throw our SMP header back on there send it over Bluetooth and then the gateway again will strip that off send it back up to the server. So how do we how do we do this with SMP. So SMP stands for simple management protocol right you can get confused SMP is used for other abbreviations. So in this case we're talking about MCU manager simple management protocol. It's defined by an 8 byte header. So the important parts of this are you got an operation code for read or write then a group. I'm sorry I clicked on that. We got a group which specifies your top level command and then a command ID underneath that group in case you have multiple different messages you want to implement for that group. So in this case we all we had to do is define a new group ID and we called it our lightweight MDM BLE transport. So we just came up with a number that was outside of the standard number they have a reserved number area so we just picked some large number. We defined a new command ID then underneath that we only needed one we just called it our command tunnel. And so you just use this header as is and our lightweight MDM co-op buffer just gets inserted into the data data spot. So just to summarize we use we use the SMP service with GAT over BLE. It's used just to wrap up lightweight MDM co-op messages between the gateway and the end device. The gateway has to inspect these co-op requests that come down from the server to know oh there's a prefix on here so that means you want to talk to this device. It has to then strip that prefix off package up the message to send over SMP and send it to the end device. So then when the gateway and the end device are connected the gateway writes the SMP characteristic over BLE to send the message and then notifications over BLE are used to send the co-op responses back the other way. And then as part of how the gateway knows to communicate with the end device we use Bluetooth low energy advertisements and set a flag in there to indicate you know hey I got something you need to do gateway talk to me. So in order to accomplish this the first thing we had to do is make some lightweight MDM stack modifications. The first thing was to separate out dependency on UDP sockets. The lightweight MDM engine is heavily dependent on UDP so we needed to remove that dependency and that was important obviously for the end device and the gateway. And then the other major piece is adding in hooks to the stack for received messages and this is important for the gateway so we can hook in and know hey we have a prefix. We got to strip that off and then do something to connect and send to the end device. So here's a quick snippet of the changes and I provided a link here you can go look at later all the changes we made but basically we pulled the socket implementation out of the lightweight MDM engine. We added a transport ID to the main lightweight MDM context in the stack and then we created a new lightweight MDM transport and a lightweight UDP transport files to abstract that all away. So now we have a version of the stack that works exactly the same but we abstracted out the transport so this is paving the way for us to add in a new BLE transport for lightweight MDM. Double switch alright next we needed the application hooks in order for the gateway to inspect messages coming down from the server. This is a relatively small change as you can see only a few files you can go look at this but basically this is what paves our way for implementing the actual object 25 work. So then first we needed to implement object 25 itself and that's there's nice APIs in the lightweight MDM stack for defining objects you define the object and the resources in there and that you know the variable types of the resources whether it's a byte or a string all that right so. And then after implementing the object we needed some common APIs that could be reused from from another layer so we needed to be able to create and delete instances at runtime. We needed to allow and block list support in case you came across sensors let's say you have a gateway in this room and a gateway in that room but I don't want to talk to the sensors in that room right so allow this block list is very handy thing to have. And you know some I other APIs to look up a device by ID to see if it's in the list if I've already talked to it before. And then after implementing object 25 and those APIs that allowed us to then implement our proxy and the proxy is what's doing all the all the work here right. If it discovers an end device it creates an object 25 instance it connects to the device transfers data over BLE as we discussed packaging it up over SMP. It manages the inactivity timeouts that's important for sensors being able to roam between gateways. It takes care of processing that those lightweight and requests from the server and inspecting if there's a prefix and then obviously knowing OK strip that off send the message to the end device and then. If a device hasn't been seen for a certain amount of time we want to remove that object 25 instance from the gateway altogether this again is part of being able to support roaming of devices across gateways. A little bit small but this details what lightweight and objects are using the gateway so the top the top two are required you got lightweight and server which is details about you know what server you're going to connect to in the cloud. Device is required at a minimum it requires you to make a reboot function you get access to firmware version battery levels and whatnot and device the device object connectivity monitoring object for is used for information about the LTE connection. So you can get signal strength IP address all that good stuff object five firmware update pretty obvious that's what is used to update the gateway itself. It's image it's effort image object nine because we have a separate cellular modem in the gateway is object nine is used to provide firmware update capabilities to that modem. Then obviously object 25 is the star of the show that's what we needed to add in in order to make a gateway be able to talk to end devices keep switching on me. And then in order to facilitate configuration we we actually had to create two custom objects we couldn't find anything in the spec that did what we wanted so we came up with a file system object which allows you to. Read directories and see what files are in a directory in the file system and then a file object to be able to interact with files. So that that plays really well with having a very generic way to configure a device everything's done with configuration files in a file system on the device so very flexible on the end device. A lot of the same same objects you got object one the server object object 3D because again those required by the by the client and then. Object four for monitoring BLE connection statistics and then firmware firmware update object again for being able to firmware update the end devices and then we include three telemetry based. Three telemetry based are actually for telemetry based objects for being able to talk to the different sensors on the end device you got temperature current pressure and fill level. So levels used with our ultrasonic sensor for kind of tank monitoring. And then again the file system and file. Objects. So here's a quick snapshot of the device management portal. This is our partner edge IQ so this is your your portal to be able to manage your fleet kind of standard stuff showing how many devices online offline that kind of stuff and then you know you can go in per device and interact with each device. I talked at the beginning about being able to remote monitor and and in the health of your device. We have another partner mem fault which I'm sure everyone has probably come across them at the conference here. They're fantastic great partner to have. They provide remote debugging capability so your firmware crashes. A core dump is saved the next time it connects sends up that core dump they analyze it you could figure out what what crashed in the in the firmware. So that's obviously very important along with knowing what firmware versions are deployed out there and all that kind of stuff is a quick screenshot of the core dump analysis. All right and obviously it's it's fun to device manager device but if you can't send telemetry that's a problem so inherently built in the firmware we have support for a separate telemetry connection. We have a few options here we actually did another modification lightweight MDM stack to support a second lightweight MDM connection simultaneously which you may think is a little strange but we did it just because OK we have our device management lightweight MDM connection but maybe a customer wants to use some other provider for lightweight MDM and just do telemetry over that and just do the device management operations over the other connection. So we had that and then go Ryan I just want to let you know you've got about 10 minutes left. Thanks. All right and then our other option for telemetry is MQTT obviously this enables a wide support of cloud partners so the customer can choose whoever they want to use and you know send their telemetry for their telemetry data there. Other features in the firmware device configuration I touched on it slightly a little talk about a little more here so for development everything can be configured over the shell with shell commands. We call this our attribute system we develop this kind of cool system where you define attributes with YAML and then when you build the firmware it automatically code generates everything you need. So all you really have to do is just add an entry in YAML for a new attribute is it readable writable saveable what type it is as a string is it a UN 8 and then it code generates it so you can get and set all these attributes right from the shell. Then SMP we talked about using SMP over Bluetooth but SMP actually has a protocol over shell so we use SMP over shell for in manufacturing to automate, automate setting up device configurations on the manufacturing line. And then all settings are stored in file system so this makes it super generic and easy to configure devices from the cloud using our file and file system objects over lightweight M2M. We then also provide a configuration template repository for common configurations for for the end devices in this case right the end device sensors are the most complicated thing to configure based on oh well I want this sensor or whatever. So we have a bunch of combinations that that you can build off of. Touch on some of the security features of the firmware. Thanks to NRF Connect SDK we they have a really nice hardware unique key API that is really nice because when the device boots the first time it generates its own hardware unique key. Only the device knows this key no one else ever can get access to it it can't be read out of the device. That key can then be used for encryption. So we have a dual boot dual boot loader secure boot which is nice because using the dual boot loader approach allows us to be able to swap out signing keys of the firmware if something gets compromised. The firmware is signed obviously with that. For the shell system we built a login system so like if you get physical access and you try and do stuff on the shell you have to log in first obviously that's that's really nice. A really nice thing we we worked on is we we came up with a way to encrypt all secrets at rest. Unfortunately at this time Zephyr doesn't have encrypted file system but what we did is we're using little FS. And we designated a folder and everything in that folder is encrypted so anytime there's a request coming through the system for something in that folder. The firmware intercepts that knows well that's been encrypted with that hardware unique key. And so the gateway itself or the end devices is on both can encrypt and decrypt that on the fly and only it can. So then if that key is ever compromised or someone tampers with the device and erases flash that key is gone and now all those settings are still protected. And you have to start from scratch and reconfigure the device. And lastly all our BLE communication is encrypted. We actually encrypted a section of the advertisement. And then when we get in a connection that whole data sent over is encrypted at the application layer. The reason we did this is we built in a PKI system so you can have public public key infrastructure for those don't know the acronym. Device certificates private keys for each device. And then they you know when they when they start a connection they do effectively a TLS exchange of their credentials in order to facilitate this. So using again all standards here right that's that's important. And finally all this firmware is open source because right it's Zephyr. We want to make it open source and one of the main reasons for that is we don't know what a customer wants to do right. You start here build on do it and do exactly what you want to do in your application. And that's it. Open up for questions. Yeah there is one question came in virtually. Do you use six low pan between the IOT end node and IOT gateway. No we do not. We are using just just BLE and that SMP characteristic. The reason for this is we wanted to keep it a little bit simpler and not deal with IPv6 routing in the gateway there. I believe there's actually some pieces in Zephyr missing yet to do full proper routing of gateway and end device. So the question was how do you update firmware in the IOT nodes. The same way you would the gateway that object five is represented on the server. And what you do is how object five works is you write a URL in this case firmware update supports being able to push or pull and pull me from from the respect of the device. So we support pull. So what the server does is you write down a URL saying hey my your firmware image is here. And as soon as you write that URL object five the device will then start downloading that. And there are resources in object five that will tell you the status of OK it's downloading it's been downloaded. So now now you can execute and install it. Yeah so the question is is making comment about how we disconnect from the end devices if there's no activity. Yes we could stay connected but there are a few reasons why in the end for device management really when you think about it you're not going to be talking that much right. So OK I want a firmware update. I want to read a serial number. Well now I have that up in the cloud so I don't need to read it again right. So you're doing stuff very infrequently so actually disconnecting saves saves more power. And another reason for that is if we stay connected to one device we can't service any other devices. So we made the gateway we could support simultaneous connections that's that's more taxing. So we chose to support one connection and do the disconnect and then service who needs servicing. Yes so when you open up that connection to the end device everything works exactly the same. That was the benefit to tunneling lightweight MDM and co-app over that connection. So how firmware pull works is it actually we're using HTTP to co-app proxy in the cloud. So we actually are pulling the image with co-app. For those of you not familiar with co-app co-app is very similar to HTTP protocol. It's done over UDP obviously but it's got the same functions like get post and all that. So you basically do a co-app get and it has block wise transfers. So because we're just tunneling co-app and lightweight MDM the end device then is asking for that. So it's doing all the gateways just passing it through with with no smarts at all. But we did add in firmware caching. So that way if you have 10 end devices that need to update the same firmware image and they they're all asking for the same thing. The gateway has we added some smarts in so that way we cash the firmware image we look and then we can just send that down and we don't have to download it 10 times. Thanks Ryan we're we're at the end of time. Thanks everybody.