 So my name is Michael Scott. I work for a company called Foundry's I.O. It's kind of a think of it as an easy button for product developers to kind of get a head start in the embedded development space. They can bring their own hardware or software, and we provide a lot of this sort of CI, test loop, security, infrastructure like that. You would have to hire several engineers to provide. My talk today is going to center around AT-based modems and their integration into the Zephyr RTOS. We're going to start with what's there currently, what was there prior to current. The stuff that mainly we're focusing on is going to be released with the 2.0 release. Just to show hands, who's familiar with Zephyr? All right, good, good. Who's familiar with AT-based modems? Oh, yeah, that's painful. All right. So without further ado, we're going to talk about several things here, and I'm going to try to get through it so we save some time at the end to kind of talk about your particular questions or something you might be interested in. So I'm going to start with a survey, we'll kind of cover the basics of the networking stack in Zephyr. Two key features there in how they offload data that comes into that stack, because that's where modems kind of come in. You don't want to use all of the underpinnings of stuff that are already provided by the modems. We're also going to cover what was the original modem implementation with Zephyr 114 and how we've changed it for Zephyr 2.0 coming up here. I think it's going to be released August 30th. We're going to talk about that new implementation has several components. Why we did it that way, what we hope to gain, and how we hope to kind of ramp that out in the future. Then we're going to talk about which modems are currently covered by Zephyr as far as either in the mainline or a mainline-based SDK, as well as modems that are in products that are currently either going to be shipped soon or out there. And then we're going to talk about what comes next, and we'll take some questions. So before we, again, I'm going to tell a little story because I'm always fascinated as to how people get involved with various projects and why they come up with the idea to actually submit something like a modem layer or some helper driver for something like Zephyr. And for me, my engineering manager came to me one day and said, you know, this AT&T modem thing that happened to be a shield, it kind of went on an XP part. He said, that thing looks really cool. This LTEM, NBIOT, this thing looks really neat. We need to investigate it and see if it's something that we can sort of present as options for our products. And so I downloaded the software and I think it was Mbed OS or one of those other ones. Not to throw anybody under the bus, but every single sample that I went to load had the entire modem driver and was totally customized to do that one thing. And I'm like, you got to be kidding me. What if I want to do MQTT or LWM to M or all these other things? I've got to like magically put together all these pieces and craft this thing. And I think it came down to the fact that they didn't have a central way of taking the network stack and kind of intercepting it with the driver and leaving that as sort of core to Mbed OS. And so I started thinking I was already an implementer. I had brought in the LWM to M subsystem for Zephyr and I was thinking to myself, well, maybe we could do something. This is, maybe we could submit something that would kind of make life better and maybe unify it a little bit and see what we can get. And so that was my start in the Zephyr modem layer. And just to highlight the network stack here, you can kind of see how samples live on the very top. They access most of the protocol layers directly so you can initialize MQTT. You can start the LWM sort of client. And then those protocol layers actually act through the sockets layer that's been implemented fairly recently down through the network context layers. And you get into the core of the TCPIP UDP down onto the actual hardware type driver, whether it's Ethernet, 802.15.4, BLE6, low pan, et cetera. Now, I highlighted two options on the right there. And these are called offloaded APIs. And so for instance, what you're seeing a lot lately is that on the NCP, the network, whatever it is, if it's a Wi-Fi or an AT-based modem, they do that stack for you. So you don't necessarily want to let a lot of that stuff go down into this lower layer because you're just going to hand it off the modem and make a call and it's going to give you a response. And that's going to be the end of it. So they have two different methods of offloading that data. One is at the top, you have a socket-based offload API. And on the bottom, you have sort of this net offloaded API. And we're going to cover those in just a little bit of detail once we talk about how the original implementation was done versus the new one. So here is how the modem implementation looked in 114. You've kind of got the net offloaded APIs on the left. That's where the injections are. We intercept our calls for send, receive, connect, those sort of basic network calls. And you've got sort of the core modem driver in the middle managing everything. And then you have your hardware is on the right. You're writing the pins. You're reading off of, say, the GPIOs or you aren't to get responses. And then you can see how this happens to be a Sarah R4 modem driver utilizes its own command parser to take that data, parse it up, and then do something on it. What you ended up with is kind of a driver that does everything. It's very hard for people to say, hey, where can I contribute? Oh, I noticed you got a bug in your command parser. Do you submit it to each driver? Is it just gets messy? Because in the beginning, I think I talk about this a little later, but it was really only one or two drivers. And so it's not really very scalable or sustainable as far as really growing that subsystem. So what we did was we kind of rearranged things. This is sort of the new model for 2.0. At the heart of everything is something called a modem context driver. And this is a helper layer that just sort of keeps the public shareable portions of the modem functions, read, write, processing. And it's able to share that out with some of the other subsystems. Right now, it really only works with sort of the modem shell, so you can actually send commands to the modem. But it doesn't know anything about that modem other than the fact that it exists. It has an ID number, and this is an API I can send a command to. Everything else happens kind of under the wheels. So you'll notice that in this other version, we used the net offload here. That's actually been changed. We now use the socket offload. And I'll cover those APIs and why we made that choice. Not only does the actual modem driver is kind of off to the left here. The main goal for the modem driver now is to configure everything underneath that modem context, hand it off, and just get callbacks. As commands come in, they get handled, there's a lot of surface area where the community can contribute. And you'll notice that those sort of four things around the center off of the modem context are really what we're going to be focusing today. There's like modem socket kind of abstraction. There's modem pins. You've got the command parser. And then hopefully power control, which would get into like EDRX or low power modes on the modems in the future. Because that's really what I think that's most interesting right now, is if you're going to use a low bandwidth, low resource usage modem, you kind of want to put it in the lowest power mode you can. Turns out we didn't quite get to everything. So the modem pins and sockets aren't really attached very well to the context. So in this first version that I'm going to talk about, that's kind of what it looks like right now. They really are just bolt-ons onto the driver. They set them up. We're going to probably move more and more into the center and hopefully get a better look there. But we are moving forward and hopefully it'll be better for contributions and everything. Next thing I'm going to talk about is this concept of offloaded APIs. And what I wanted to kind of start with is this is the past. This is the kind of access you had in these net offload APIs. You had things like get, bind, listen, connect, accept, send, receive. These are very basic network commands. They don't give you access to much else. And the kind of other things that we're talking about, I'll cover in a sec, but they also used kind of these cryptic parameters. So it's a net context. And you had to have this net package structure that you passed in. And that thing's like a linked list of buffers. It all made for very bloated drivers, very hard to handle. Iterating through these things and trying to sort it was really kind of a pain in the butt. This is where we move to the socket APIs. So this is a POSIX-based. I don't want to say it's compliant, because that's kind of a wiggly word. But you'll notice that if the functions that are offloaded, you actually have kind of some interesting other options. And a lot of modems or AT-based command sets, they provide things like TLS setup or DNS calls. Or these are things like, before you even make a connection, you actually want to do something. Whereas in the other layer, you almost couldn't do it. And so it was really frustrating at times for to implement certain modems. The thing you're handing around is not some cryptic net context. It's just a file descriptor, just like any other POSIX system. You hand that thing off, and it just makes sense when it enters and leaves the modem driver. That you turn around and hand that off to all of these offloaded APIs, and everything kind of stays in sync. The kind of network data that you're handing around is just buffers. These are just char pointers. They're very easy to handle. There's no sort of iteration through linked lists or trying to do funky allocation and freeing up of all this stuff. And so I think this is actually going to simplify the drivers over time. You're going to get a lot less bugs. And in many cases, you might even be able to port existing code because of the way it sort of POSIX-based. The other thing I noticed is a lot of these AT-based command modems really try to mimic the socket workflow. So when you create a connection, it gives you kind of a number back, and that's your socket number. And then every time you write to that connection, you've got to reference that number. And it's a lot like using sockets. So it just mirrored that workflow a lot better. We're going to talk about what the original idea of having a modem helper layer was. What the heck is a modem receiver? This is the first thing that got submitted with the first modem driver. And it was done back in August of 2018, way, way back. And it was ancient for Zephyr. And it was submitted along the time, the very first modem driver that I did for that Wistron modem that we got from AT&T. It's just very simple. It's a tight loop. It handles an ISR. It offloads the data off the UART as fast as it possibly can into kind of a thread safe ring buffer. And then it's up to the modem driver to call back in and say, hey, do you have more data? When he has time, can I just receive something? And if nothing's there, fine, move on. If so, I'll grab it. And then basically once the modem driver got that data, everything else was handled in the driver. Everything was about the modem receiver was very UART specific. It didn't allow much abstraction. Like you had public and private data. There's semaphores in there. It's all kind of messed up. And unfortunately, now that has to live in there a while because there are other drivers that are using it. So now we have our problem. I want to submit a new driver. Maybe I want to change that implementation a little bit. The community started kind of like, hey, how would I do this? And maybe my thing needs to be tweaked a little. And I realized we kind of have a problem. Nowhere to land bug fixes. It's confusing as far as, what if I want to change a parser? Who do I copy paste everything? How do I set up a new one? There really was no future with that system. So let's do something better. Earlier this month, I actually finalized the contribution that landed in Zephyr for 2.0. And it centers around this concept of a modem context. You're taking a series of these underlying objects. You're configuring them. You have several different implementations. They have both public and private structures there. So in a modem context, it's what's public. So you get a basic read and write abstraction for the modem interface. You have a really basic process abstraction. For once you get that data, you hand it off to your command parser. And then you have some kind of helper layers in the pin config. And that's where I was talking about pin config. It's not really. We're getting there. It'll be better and better. Basically, what that did was it stopped all this boiler code from people having to import the GPIO layer and do all these other things. It just makes it a little easier. And then when you compare the new modem interface with what was the modem receiver, the biggest thing was what happens if we don't have a UART-based modem? How would we treat that? So if you have the public read and write APIs and you leave the private data up to each implementation, then you have the option of adding more. And that allows the community to really go crazy on it. The first implementation is clearly a UART-based implementation. So all it does is assigns itself the read and write functions. It has all of its own private little semaphore ring buffer. It's very similar to what was the modem receiver, but it allows other things to coexist. And what happens, as the modem drivers get more fully fledged, it's almost going to be like a buffet table of mix and match objects here. And as I described the other parts, you have to imagine I'm writing a modem driver. I'm going to take that piece, that piece, and that piece. I'm going to link them all up to the modem context and hand it off now. And everything just runs. We'll talk about the idea of centralizing command parsing and what it actually is. It's really tricky when you get into AT modems. You've got to deal with these carriage returns. And so you're sending a command. And you kind of need to wait for the right response. And so when I sat down and I tried to abstract this out and I wanted to make sure this is something the community can kind of help, I came up with that sort of generic sort of process. So at its heart, every driver has kind of a receive loop. And once they get data back from the interface, they just hand it off to the command parser that they've chosen. The initial command parser deals with very straightforward AT based commands. It uses carriage returns. But it does a lot of nice things. And I'll get into, how does this look? Basically, there's a define called modem command define, which is basically establishing this is going to be a callback. You then create three kinds of responses. So one response is a generic response. It's a OK. I had an error. Something maybe even extended error. And that's what I'm defining right here in the response commands. And then you have unsolicited commands. These are like, hey, there's some data ready to read. Or hey, you're now registered on the network. Or these are kind of things that you can't expect or wait for. And then there's a third kind of responses. That's the response to the command you're actually sending at the moment. And that's the one you're actually waiting on. So in a way, you can handle your unsolicited. I call the response commands at the top kind of those are the enders. Those are the ones that say, yep, I got what I wanted. Hopefully, in the middle, you were able to get some data out of that and use it. Because that's what triggers. Say, for instance, you're going to send a command to read the manufacturer data. In the middle of that, you're going to get that manufacturer data. It's going to hit the command callback. And then it's going to hit the OK. And the OK says, give up the semaphore. Let's move on. That command has been successful. Or it had an error. You'll notice that there's some interesting parameters here. The modem define defines the handler. The modem command itself matches what it's looking for with how many parameters you're going to get back from it and what the delimiter is. And that's why you're seeing this string of data there. And what that does is it actually handles kind of like an argv pointer, an array of strings back. So you don't have to sit there and iterate through that data looking for things. If you have your right delimiter, you get the right array of commands that you're looking for. You just grab it and copy it. It actually chews up a lot of the boilerplate that was ending up in the modem drivers that each one was implementing specifically. This is just going to talk about the config helper a little bit. Because every modem seems to have like a power, a reset, or some kind of generic. Sometimes there are GPOs to put it in low power mode. What I found was a lot of people had a lot of different code in there doing different ways to control the modem. And what I found was we want to encourage the right behavior. So in the example that I'm the one that we set up for this first iteration, we use DTS. And so the DTS portion here is actually coming from the binding. It's set up for that modem. It generates this sort of standard define. And that ended up with your controller number, the pin number, flags potentially. You put that in a nice big array. You can use it in a num. And then you can make generic modem pin writes. You turn things on, turn things off. If you have to wait for an interrupt, it could get a little more complicated. But at least this means you didn't have to deal with all the GPIO stuff that you end up setting up in Zephyr. Am I going too quick? No? Good? All right. Let's talk about what those socket offload APIs look in actual code. And so remember how I said, these are sort of positive space. You have a socket call, which allocates the FD. You've got a socket close where you pass the FD back and you're giving it up. These actually end up as functions with sort of these parameters where you then translate that into the AT modem command. And so in some point, you're probably you're busting out that text back on. OK, here's my, you create, sock, whatever. It seems like every modem has a different set of commands to manage this part of it. There's no 3GPP standard for all the socket-based network communication. But then what you do is you assign those functions to this offload structure. And you do have to register it at the bottom there. So somewhere in your modem in it, you're going to call register. It then says, ah, I need to intercept all this stuff. And instead of pushing it down the stack, I can deliver it over here. And it's actually really cool. You can turn off the stack at that point. And it makes your code size small. It's kind of neat that you can actually put these modems on very, very tiny devices. The next thing I'm going to talk about is how to get started. I know a lot of people on Slack and on IRC have like, I'm confused. I know I could probably build this modem driver. And I could connect it right up to your LWM time client. And that would be great. But I mean, I want to do that. I don't want this thing to all of a sudden try to get and receive data. I want to sit and maybe just tinker with it a little bit. And so here's some helpful tips. You're going to have to get your AT command reference, first of all, and get real familiar with it, especially the networking portions of it. How to create a socket, how to send data. Do you have limits on that data? Does it need to be hex encoded? All that sort of stuff you're going to have to know. The next part is really get familiar with the Sarah R4 modem driver. That's the first modem driver that really uses this modem context setup. There's not going to be much that you can copy and paste from it, but you can certainly set your driver up the same way. As far as here's your handlers, here's how you squash them down into an array and hand them off. That driver is probably going to get you on a good start. Here's some helpful config items. I love to enable modem shell. I think it's a great way to play with the modem. And I like to play the hello. So I'll compile the hello world sample, which literally prints hello world and stops. But meanwhile, behind the scenes, the modem shell is still running. You can send AT commands. You can watch for responses. You can really play with the modem driver a little bit. And what I'm illustrating at the bottom here, if you're familiar with Zephyr and the West Build System, you can actually tack on config items on the fly after the double hyphen. So I'm actually not running an overlay config to enable the modem. I'm literally just enabling the modem with all these commands. And you can do that with any sample in the Zephyr library. And there's a link to the Sarah R4 modem driver there at the bottom. So we're going to step away from the technical stuff a little bit, and we're going to back up. Here is basic lists of just which modems are covered in Zephyr right now. This is the original AT&T IoT starter kit. It's a WishDrawn modem. The AT commands are a little sketchy at times. There's a lot of workarounds. I'm not recommending people necessarily jump into this one. If you already had some investment there, you might want to take a look at this. The one thing I will point out is the first two are actually shields. And you can, any piece of Zephyr that has Arduino headers, and it has the DTS defined for them, these shields also have matching defines so that you can mate them via the Arduino headers. And it will just work if you specify it in the shield parameter there. We spent a lot of time really trying to abstract the shield layers out. And I think you can really see the benefit on these. This one and the next one, this is the one I kind of liked the toy with. And I would recommend if somebody does want to jump in and have a good time with it. This one is a SparkFun product that they took a Sarah R4 modem. They slapped it on our Arduino header. You only really have two pins plus the UArts. It's very easy to play with. It doesn't matter whether you have a Nordic dev kit that has Arduino headers. You can have the Freedom board which has Arduino headers, or you could have an SDM part that has Arduino headers. It'll pretty much work with almost anything, as long as you get the DTS. Actually, in this case, it's all taken for you. This one in particular, if you literally run it for any of the networking samples, it should just work. If it doesn't, it can hunt me down. And you lied to me. I hate you. This is an interesting one. And I wanted to talk about this one a little bit. This is a brand new product from Nordic or fairly new. It's a system on a package, the 9160. They use a Zephyr-based SDK. It's one of the first companies that actually has taken Zephyr, adapted it into their software development kit directly. They have all of their own samples written for this that are literally going to work out of the box. It's a really neat, they're really leveraging their SDK plus really well-built network samples. They have a cloud-based connector. I helped them get the LWM-TEM client up and working for this. It's kind of fun. I mean, if you guys have worked on Nordic dev kits, this is just like a dev kit style, but with the modem part included. It doesn't have BLE or any of the other things included. But it's that sort of same form factor. And the last one I'm talking about, so this is not a modem that's actually supported in Mainline. This is an actual product where they chose to implement it using Zephyr as the driver. And they hand that as the software piece out for people that are looking to customize the modem in some way or run some sort of custom software. I thought it was interesting because I communicated with this guy for a quarter of a while on the current modem driver and how we were going to make it better. And he was super excited. He was like, I'm not going to touch that because we're going to ship this thing and we're going to get it out the door. But I want to talk to you later. So I did want to mention it. Hopefully he's going to submit the actual driver to Mainland Zephyr here pretty soon. Maybe it'll get in for 2.1. But this is an interesting product. And again, they're going to hand Zephyr out as sort of the out-of-the-box demo that you can code on this. So now that we've covered all that, what's next? Clearly, I had a couple of fibs in my diagram there that we're going to try to work on. We should really abstract that network layer, the socket-based connections to it. So it's easier. There's a lot of boilerplate stuff in there. It just doesn't need to be in the drivers. We need to move more and more into the context so that the hard work of building the driver that shouldn't be there isn't there. We need to support the low power modes. One of the guys had a really interesting idea of actually being able to, one config, enable a bunch of standard 3GPP out-of-the-box modem commands, set up as handlers already. So for instance, a lot of reading the manufacturer data, the model, all of that stuff is set in stone. It's part of the standard. It should just work. So if we were to have a 3GPP layer, you would actually cut your probably modem driver in half. All those handlers would just exist. There's some other really interesting ideas of creating a new interface called a dummy interface. And then you would have a dummy modem driver. And what you're doing there is you're feeding data in and making sure that you get the right responses from command parsers. Or we would use it as more of like CI, make sure that we didn't break the modem layer. We need to work on that. There's some storage configuration for APN data and things like that. We need if you could just bolt that on the bottom. I know a lot of modems have internal storage for that. But if you wanted to do something special in this effort to kind of feed configurations into that, it would be interesting. And now we're open for drivers. I mean, it's more about as people get interested, you pick a driver, and it's not as hard as you might think. Once you get used to the code, it should be easier. It could not at least you have a place to post bugs. If you see problem, you can actually post it on the mailing list or post it up on the Git site. And with that, I'm going to open it to questions. Let's see if somebody has something specific as far as something you were interested in. Right here. That's right. They have an add-on. It's interesting. I think it's AT-based commands. So what would be fascinating for me is to someone that would pick the GPS module up, they do the add-on, and then perhaps write a subset of commands to work with GPS, and then you can enable it via config. And it would almost work with any of the CERA modules. The CERA R4 module happens to work with the U2 module as well, which was an interesting contribution. But that's a great idea, I think, is enabling GPS via that. Several of the modems have those type of add-ons as well, not just the CERAs. Was there any other questions? It really is. And as soon as you say, oh, no, it's POSIX compliant. And then they'll be like, wait a minute, you don't do this, and you don't handle this. So I hesitate to use POSIX compliant as a word. It is POSIX-esque. It is POSIX, and you should be very comfortable using those commands. They're going to give you a file descriptor. They're going to do the things they should do. I just didn't want to step on any. You can get into arguments real, real quick. It's almost as bad as tabs versus spaces once you step in the POSIX space. Absolutely. And there's code out there. And Zephyr has made a huge effort to become more and more POSIX compliant. And I think it's definitely stepped in the right direction because you're getting code reuse. You're not trying to reinvent the wheel. And there's so many of the drivers are already out there. Why not leverage them? It was more about implementing the reference. You're correct for the socket type stuff. Actually, what I meant was the 3GPP has a standard set of AT commands that almost every modem implements. And so I felt it might be interesting if we had a layer that you could just turn on, and it would automatically pre-query like the ops that you're connected to. It might actually pre-configure some of the past in commands and stuff and automatically be able to query the manufacturer info, the software version, things like that. It gets a little weird when you're implementing the OK error parts that are so specific to your driver, depending on are you waiting on something back and things like that. But I thought it might actually reduce the size of the drivers quite a bit. That was the 3GPP reference that I was talking about. I think the idea is that you're going to clear it out any time you do a receive call. So at that point, the driver's ready to handle data. The neat part is so if you get into the details here, the implementations have their own buffer sizes. So the ring buffer is one thing just to get the data off the york, because it doesn't stay there forever. And if you leave it on too long, you're generating ISRs. And so you want to get out of that context very, very quickly. The driver needs to then have its own processing buffer that's specific to the command parser. And that can be oftentimes a lot bigger than the ring buffer, because you might get half an AT command response. You might get a whole one. And whatever doesn't get processed in the current implementation actually gets left in there so that it gets picked up the next time. But that's a great question. Every driver might need different size buffers. And that's why we allowed the driver itself to configure all of the sort of private data for each component. And then when it registers it with the modem context, it's accessible then. So the command buffer has several different configurable items. And it's something that we're going to have to do a better job, I think, of documenting. But for now, you're going to kind of have to look through the code and go, oh, there's that big fat receive buffer that you want. And typically, it revolves around the max length size. In the particular in the SARA R4, I think it's 1024 is the max sort of receive size. And that's what you'll see in the buffers. It reflects that. You have to, right? I love the fact that you brought this up, because the idea behind splitting out a UART specific implementation for the interface was that you might encounter exactly that. There might be another way to get the data. And in my head, I was thinking, I don't know, spy or something crazy sort of implemented. But USB is actually a really great way, because you do see that implemented on a lot of the modems. And if you have sort of that sort of read and write methods that still apply, I don't know if they actually still apply when you get into USB, you might actually be doing more Ioc-tile type calls in the Linux side. Maybe it does end up being like a send and receive. It is a read. So you get one at a time, you get packets. OK, I love it, great. I would love to see somebody implement a USB interface for this particular, because that would just open up a whole another wave of drivers. So that just lets me know maybe we did the right thing. Was there a specific modem where you mentioned? Did they make a recommendation on which one to use if you're going to have certain use cases? It's usually the bands, too, that they support sometimes. I was thinking actually, I don't know how easy it is to do low power on USB. Sometimes it's harder with the teardown and the bring up. Sometimes they go, if you're going to do low power, recommend UART. It's easy to turn the UART, like IP, off in many cases. But that brings us to what's next. I mean, there's things like, so I can tell you right now there are about four modem drivers in works. The one that I mentioned for the pinnacle is the one that's probably the closest. There was another modem driver written for, I think, one of the Qualcomm modems, which I thought was really interesting. Hopefully, he gets that up and running pretty quickly. And there were two others where I think they mentioned it, but I may have forgotten. So we'll see how quickly they can adopt sort of that. Because I really wanted to see new submissions using this. I don't want people to keep writing these sort of copy-paste drivers and really hard to manage. So we'll see how quickly they come up. But I think the more example drivers using this, it's actually going to get the snowball rolling. People are going to have an easier time implementing them. There'll be a lot of really good examples. I would love to see a USB implementation. Any other questions? All right. Well, with that, have a good afternoon.