 Hey everybody, I'm Jared, a.k.a. Circuit Dojo. I'm going to be talking a little bit about provisioning. It's nice to actually follow Kevin on this subject because he's much more knowledgeable on the subject, so I'm going to try to hit some of the important things, reiterate some of the things that he said, and we'll go from there. That's how I got started with Zephyr. I made that board, and I went down this crazy rabbit hole with Zephyr and haven't stopped since. I definitely took the red pill on that one. I also do live sessions on Zephyr, so if you haven't seen them already, if you should check out YouTube, if you just search for Zephyr, I'll probably, besides the Linux Foundation, you'll see some of my videos. If you're for me, I try to do them fairly often, so I'd love to see you there. But enough of that, let's jump into it. The idea here is to make your provisioning, or your setup for your devices as easy as possible, so we're going to be talking about making them secure, making them easy to maintain, and we're going to try to save some time in the process, so all these things in a nutshell is to essentially make it easier for you to set up devices that are going to go out into the field. We're going to focus on getting provisioning certificates, we'll touch on getting also just a little bit on configuration, because I think that's also just an important part of provisioning for devices, and we'll talk a little bit about open source tools, things that are out there that can make your life a little bit easier for this on the subject. I was also talked about in the last talk, so if you missed it, definitely go check out that one as well. And some important security implications that you might, you know, just important things to think about because we're talking about security, we're talking about provisioning devices, so they're secure with communication between device and server, so, and I won't be touching on anything like JITP or just in time provisioning, so if you came for that, I'm sorry. So let's talk about generating certificates. There are a bazillion different ways to do this with a scientific way of saying it, but you know, for example, if you're using AWS, I'm sure many of you in this room have gone to their IoT, created a device, and one of the first things that they suggest is, hey, let's create a certificate for your device. So that is kind of one thing that you can do to get a certificate for your device. You can also do it kind of the old-fashioned way using the open SSL CLI, easy RSA is also kind of a wrapper around that that kind of at least kind of organizes everything, puts your CA cert here, puts your server cert here, per device, certs kind of go in their own little place, so it's also an option that's something that I was playing with early on when I was doing this. A lot of these things that apply to kind of web world where you're generating certificates for there, but you can also use it for generating certificates for devices. And one thing that I did a while back was that wasn't really working for me, so I actually created a kind of a CLI that worked directly with a server that had written in Rust. It actually is a MQTT broker plus some other fun stuff, and some of the things that I've done on the CLI is actually integrate directly with one of these guys to pull off that hardware ID, pull off that unique identifier, and then generate certs and then push them back onto the device in one step. So here's that screen that I was talking about with AWS. I'm sure you've all seen this generator cert. One thing you can also do, and this is something that I had just happened to be doing this past, let's see, last week, was a customer had to move from Google Cloud to AWS, and their devices, they actually had to send the certs back up to AWS because their compatibility wasn't there where you could generate certs. Some of these devices are already out in the field, so you kind of have to deal with what you got kind of thing. So here is an example of EZRSA, just creating, initializing, building your CA, building your server cert, and then also just building the cert for one of your devices. So that's kind of like the general process of it, but that's the idea here. Perious, kind of what I was talking about before is the CLI uses the RCGEN crate for actually cert generation, and then also the serial port crate for writing over UR to actually write the certs from your device, from your computer to your device. So the nice thing about that is that it does keep all your certs kind of organized in one place. This can be a bad thing or a good thing, ideally like what Kevin had mentioned before, ideally you're not storing it all, but we'll get to that in a second. Generate server and CA certificates, so you'll also be able to do that in just one fell swoop, and it's also facilitating loading to the device. So it's able to talk over a custom shell implementation where you can actually issue either AT commands or if you're using a device that might not have a secure key store, you can also use things like not ideally, but using the setting subsystem to kind of store things so you can do it on devices that might not have that capability. Here is kind of the output of that cert device command there. So there are a couple options here, one for kind of what the key format is. We're talking over what port tags are also kind of an important concept for setting up any type of TLS or DTLS connection. We'll get into that in a second. So this is just kind of giving you an idea of what it does. Here's kind of the output of what you would expect. These are just all the things that are required generally for any type of secure connection to a server. You have your CA cert, you have your server cert. So if you're actually hosting your own server, that's also something that's important to have and in place. And then these guys are actually kind of unique ID, and then also the idea here is the different sets of certs for each device. You can see here, this is actually, if anybody's working for Nordic here, this might look very familiar. This is the format that they actually use for NRF Cloud, but I kind of adopted it for my purposes, mostly because it was really easy to take that output and then if I wanted to kind of debug it or pop it into NRF LTE link monitor or now it's called cellular console or cellular monitor, you can pop this in there and it'll program it over AT commands. So it's just another option or just the way of kind of storing that information in an easy digestible format. For unique identifier, so there are a couple of different things that you can do for a unique identifier. Generally, this is kind of like the client ID that you're going to be working with. And for the NRF 9160, you can use something like an IMEI, or you can use the other thing that I wrote here, which is the FICR register, which is a kind of OTP register that is a random number that's generated during manufacturing. So those are kind of some of the things that you can use for anything else. You can also use MAC addresses or anything like that. As long as it's kind of immutable and random enough, so you have, it's unique across your fleet. And you can also, for anything, most likely it's going to have some type of random ID that gets generated. So that's just something you got to check on your end. I'm mostly talking about Nordic because those are the chips that I use. So on the cloud side, I'm just going to bring this up real quick for them, is that they have an easy way of generating certificates that are used for provisioning the NRF 9160 or any other chips that might be connecting to their cloud. So they have that API. For this little guy, every time I test and kind of set it up, every single one gets a set of certificates written to it over using the key management AT commands. And what it does is it will write the search to the secure key store on the device. There's no really way to get that information back off. It's in a place where it's secure enough unless you want to actually tamper with the device and do some destructive tests, I guess you could say. Here's the API for getting that information. You might be seeing that the output or the payload there is very similar to the JSON before because it is. So there's some interesting caveats here. It really depends on your device and also your server. So the idea here for RSA versus EC is I ran into this problem in the past where I was generating EC certs, and either the server did not support it. It was just the way they had written the server. Eventually, it did a PR where it actually said, hey, let's also support this. But in the meantime, it's like, oh, I've got to generate RSA certs. So it's something to keep in mind when you are generating certificates, you just want to make sure that they're supported by that server. In most cases, in all cases, as long as you're using a standard kind of TLS or DTLS implementation, it's not going to be a problem. Another thing about this is make sure that, so for these, you see that I'm storing things on the file system, not ideal. If you have a situation where you're going to be generating these certs, at least put it on a machine where it's air gapped, where it's not connected to the internet, at the very least. Ideally, what Kevin was saying in the last talk is you don't have to store it. Don't try not to store it. But another kind of pro tip here is just make sure that whatever you're using to generate those certs, it's open source. It's ideally audited, something that has enough entropy that when just basically randomness to generate those certificates, so someone can't come through back and regenerate certificates, you wouldn't want that to happen. Storing certificates, so this is kind of two ways of doing it. I kind of mentioned them before. The one thing that you could do is put things in the setting subsystem. Not ideal, because we'll get into that in a second. But for here, you can see the hardware info. The module is actually turned on, and this is what we're going to be using to get the unique ID off of the device. And we'll see that in a second. But it's essentially a shell command that you just run to get the device ID. We're using the shell, but we're also turning off the while card. And this is important for actually that I've run into. And maybe everybody here might have a better idea. But for writing large strings or large payloads of data, you've got to turn that off, because otherwise it'll ruin that, so you don't want to keep that off. Then turning on some other things here, so the custom implementation for the AT shell, so you can actually, instead of using the way Nordic does it, is that they have an AT console, which you can communicate directly with your Tiet director to your UART. But in this case, I want to use the shell. I want to be able to use the shell for other parts of the system. So that is also just an important piece here, and then also credential shell for. So for anything else that might be using, that doesn't support AT commands, like it doesn't have, it's not an NRF 916C. So it's for any other device, you can actually use that guy instead to actually write your certs. So storing certificates is also tricky. And it depends on the hardware, really. So anything that has that, you know, you get the Cortex M33s, the NRF 91s, the 53s, they all have some type of secure key store. And that's kind of an ideal situation. The third option down here is also an ideal situation where you're actually using some type of external secure element. Now the support for that in Zephyr is not quite there yet. I know there's some discussions in some issues on the repo itself that this is something that's kind of a long time coming. But once that's kind of there, that would be kind of the ideal situation for this to remove this item right here. So right now it's kind of like, OK, well, the best thing we can do is store it in the file system. It is what it is. One thing that you can just think about though is you can at least avoid putting it on the same flash where your program lives. So if you accidentally maybe erase the device, at least you're preserving your certificates. So just something to keep in mind. Here is the output of actually writing to the same device I have here. And the idea here is that it's pulling that device ID from the device using the shell, the custom shell implementation. Or in this case, it's just a hardware device information shell. And then it is pulling it, and then it's generating the certificates. Where by default, it is already using the tag that is associated with the client that's on the device. So in this case, it's just an MQTT client that's connecting to the server. But let's say if you're using a different tag, or maybe you're using multiple tags, and we'll get into that in a second. But you can change those things. So we're generating private public key, and we're writing it back. We're also writing the CA cert also. So this is all writing over the AT commands here. And one thing that happens here, if you ever do write these type of certs to a device like this, is that you do need to kind of give it some time between each one, or else it will not work for you. And that's just the way the modem firmware is on these NRF9160s. So the commands we're using, they're under the hood. These are the CMNGAT commands. To kind of modify and update the secured key store. And we're using a custom AT shell implementation. So you can check out what I've done here, is I've provided some links and some info for you, if you want to go and check it out. All of this stuff is Apache 2, so you're more than welcome to take it and run with it if you want. For devices that, so for the devices that aren't, they don't have the secure key store. You can do something like this, where you can actually, you can use the settings subsystems actually to store the certificates. Again, not ideal, because you're basically storing them in clear text that anybody can read out. But what I'm doing here a little bit differently is, I'm still sending them using the AT client, but I'm sending in more of a binary format rather than a string format. So it's just a kind of differentiation. That's one thing I believe the site on Zephyr is actually expecting it in more of a binary format. Again, downsides, you're basically putting your search on there, so anybody who has access to the device can read them, so keep that in mind. Here's the code, we're diving into the code a little bit. So we're loading the credentials, so this is just a subroutine that we'll go and we'll get. We're setting, we have the security tag, that's basically a K config value that we're just making sure that we're writing to the correct one. And then we're also, we're kind of been putting the cloud credentials here, so you'll see in a second. So what I've done is here, you can see this line right here, it's actually pointing to the path that I've actually written those search earlier, and then we'll be reading those back out and then putting them into the device. So you can see settings, we're loading right from the settings and making sure that it's valid. And then you can see also, this is kind of out of date because I'm still using the log serve here, but you get the idea. And then we're using the TLS credential add. So in Zephyr, if you wanna add any credentials, this is the way you do it. So I've already read it in, we already know what the type is, or the tag, the type is whatever, if it's a CA cert, private, public key, and then we're writing that from a buffer. And what this does is, it allows your TLS or DTLS implementation to grab that information and use it for your TLS or DTLS handshake and for any type of your authentication and also for encryption purposes. So that's the part of, it's a little bit more complicated. So for the devices that already support this or a secure key store, it's kind of hands off from your perspective because all you really need to know is, did I write it to the correct tag and am I using the right tag? Request for encryption and signing, it's all done for you so you don't really have to worry about it too much. You also have the choice of whether or not you want to use, say, the TLS implementation on the modem side or if you wanna use something like Nbed TLS, really up to you in many cases. You're gonna get a little bit more flexibility with Nbed but for the most part on the modem side, you're gonna get enough support for most of these cases here. And of course, it's used to set up the TLS and DTLS connection so. And the nice thing here is once you actually, from before, if you go back here, we've already written the search using that TLS credential add. This can be done at any point in time in your application if maybe if you don't have control over your client, you can do this before the initialization of that client. So maybe you're using AWS or something like that and you wanna add it before the fact if maybe your device does not have a security key store, just an option. Let's see, make sure I didn't skip it. So we're gonna use them. So the interesting thing here about this is that you can use multiple tags. So if you have multiple sets of certificates, you can actually use multiple of them. So maybe you have some that are set for, you know, you have a set of certs, maybe with a staggered set of expires. You can actually add them in here. So it just gives you flexibility. You can hear it, there's not really much difference here but I just wanted to show that there's a lot of guys in here so. This, we're doing the same thing. They're actually using DTLS. This one's using TLS. There's no real big difference. And you can also see, so this is the setup for MQTT. Some of you might have seen this because this is kind of the main thing for a while now that most InterF9 160 implementations kind of use this but you can see that we're getting the tag count and then we're setting a pointer to that list. Once it's taken in, you get the TLS configuration set in. It's kind of hands off. You don't really have to worry about it. And this is their example here. So they're setting their protocol. They're setting their tag list. They're setting the number of tags that they're using and then hands off. So they're, account reiterating the point. Yes, they're provided as a list so you can actually set a couple or whatever. For the most part, I'm only using one. So you, and it's provided to the TLS implementation, and then everything else is handled for you. I'm not going to reiterate the point here. So let's change gears a little bit here and let's just talk a little bit about device configuration. There are many kind of different ways to attack this. And one thing that I've done in the past and this is kind of what I do currently is what I'll do is I will encode the data payload to the settings subsystem. So key value store, really easy to use. You can, I typically use Seaborr but you can use JSON, they're interoperable, so it's kind of nice. You can set it to it, then you can retrieve it from the settings subsystem usually on boot. And then you can decode it and then also apply it and just check for differences and you'll see what I'm talking about in a second. Now one cool thing that I've, what I typically do is I'll set up a struct, a configuration struct, that I will provide a kind of default macro. And the macro is basically, I have this device, here are the settings that are ideal for this device and no matter what, if something happens to that configuration that's read in, it'll at least set those defaults when, so say in the instance of device boots and then you try to read that configuration, it's invalid or maybe it's not even there, at least you have those. So most likely your device is gonna keep trekking along, maybe not in the state that you want it to be, but at least it's still working. If it's complicated, you actually can layer these structures and you can set these macros, so you can actually set them up in a staggered format. So maybe you have a struct for subsystem for GPS or maybe you have a struct for kind of general connection information. You can kind of set those up in separate macros, it kind of makes them a little bit more clean. And so here's an example of actually getting the configuration. So you can see here, I'm actually setting that temporary config and I'm setting all of the standard defaults. So, Jared, why don't you just use, just set it to zero and just not worry about it? Well, some of the defaults might not be zero, so you just wanna be able to set everything to that. And then what we're gonna do is we're going to try to decode into that struct and then if we can't find maybe a value, maybe something has changed, maybe you have an OTA, your device has been updated, at least it's gonna use that same default value if it's no longer, if it's not available and the configuration that's on the device. So we're decoding it, this could be anything, this could be JSON, this could be Seaboard, this could be message pack, this could be whatever of your choice. And then what we're gonna do here is we're gonna compare and apply the configuration. And what's happening here is instead of kind of applying the configuration for everything and then kind of like resetting everything every time that you get a new config, at least you can see the differences in the configurations and maybe only tap, hey, GPS, new configuration, let's reload and restart and keep going. It's gonna keep your device a little bit more stable over the long term and it'll probably keep you sane as well. So we set the same defaults, the configuration is decoded, if it's missing, like what I was saying before, if it's missing any values or maybe if it's completely missing from the device for whatever reason, at least you have something there that's gonna keep the device running and it's robust enough where you're gonna keep this device as stable as possible. Now there's a bunch of different ways you can do it. You can, and it really depends on your application. I have had some customers come and say to me, hey, I really want to be able to update this thing over BLE. It's like, okay, cool, we'll make it happen. Some really want it so they can update and change the device functionality while it's in the field. So you could have something like an MQTT topic that you subscribe to or collab if you're observing as a particular endpoint just to get that updated configuration. And then for this, a lot like coming back to provisioning, I like provisioning where I have it in my hand, I can write the certs and I can write the configuration right there. I'm doing it over the shell. And essentially, I'm writing the certs using those AT commands most likely, especially if I'm using one of these. And then for anything else, I'm probably gonna be using the setting subsystem unless something is different. And on the back end, for the setting subsystem, this could be anything from little FS, usually it's a little FS, if anything. And that's my preferred choice. And another, I'll add another tidbit here that I've started doing is for device provisioning, I actually set up gates inside that kind of boot initialization or that boot process. So it'll actually be looking for certain things. It'll make sure that, hey, do I have certs? Do I have a device ID? Do I have all the information that I need to actually boot and be operable? And if it doesn't, it's gonna say, hey, I'm gonna turn on an LED, I'm gonna spit some stuff out on the console and be like, hey, I'm gonna stop right here because I'm not ready yet. So there's tons of resources out there on cert generation. This is just like the tip of the iceberg. I hope you found it useful. There are tons of open source tools out there. I will provide, feel free to check out some of this information because I hope it's useful for you. And one of the nice things is once these things are loaded, you don't really have to worry about it. Just keep in mind the security implications of dealing with certificates and keeping them safe because otherwise you're giving away the keys to the castle. And the last configuration, you can handle a very similar way to how you can use it for provisioning certificates. Here are some resources here. If you'd like to take a quick picture, but I do have the slides available on the schedule so you can check it out there and you can also provide the QR code here and I'll email you. Make sure everybody's done taking pictures. Does anybody have any questions? So you can check this out. This is the link to get up. If you'd like to stay in touch, I have an email responder that's set up. You can sign up and I'll send you both of the slides for the topics. But do you have any questions? Feel free to ask. I think we're pretty close to done. That's something I haven't really played with. Kind of like just in time kind of thing. So I haven't played with that actually. I'm not discounting that at all. Let's just know their options. So the question was like, have I used the ability of the NERF-1960 to generate certificates on the device? And that's also a great option if you have that ability. So I like that. Okay. Yeah. There are a lot of smarter people in this room and there's a lot of things that I don't know. So I'm really glad you brought that up. That is a good question. So the question was how do you handle the expiry of those certificates? I haven't dealt with that yet, honestly. But one thing that you can do, one thing that maybe you can do is something like that where you can regenerate the certificates on the device and maybe only send the pertinent information back to the server. Whatever it is, you're gonna have to work out on the client side, on the device side, and on the server side to handle that regeneration, whether or not that happens on the server side or on the client. So it's a little bit more complicated, for sure. But it's a bit of a problem, difficult problem, for sure. Sorry, I didn't quite answer it. I haven't had to deal with it quite yet. So if you're really strict about it, yeah, it's another problem to solve, for sure. Well, thanks for everybody for being here. I really appreciate it.