 So, hello, my name is Panthele Santoneu, I work for Consulco Group, we are a small consulting company we are Linux experts we think and we're lately starting to do stuff with Zephyr This is a site project for me, so it's a it's my 20% project so be kind so I Think everyone have heard what's going on with IoT a Couple of years ago It exploded everyone started talking about it this is the standard definition Which sounds exactly like the stuff that most of us have been doing for decades before so exactly what changed well, what changes that now there's a big market and Stuff have gotten cheap enough so that it's now open for other people besides geeks Or people working for big companies and having big budgets I'm not going to talk in this presentation about any kind of IoT protocol because there's so many and frankly I have no idea which one is going to be The winner there's so many they're competing and Have no clue who's going to be the winner, but I know that we need to figure out a way to make Linux and Zephyr or whatever small real-time operating system work together, so IoT on Linux is not a new thing has been around for many decades We've been doing that stuff before we have it had the name But we used to do stuff with the custom protocol custom protocols Standard servers no no clouds, but Everything is on Linux already, so we have all join MQT we've xmpp everything everything is on Linux but we cannot use it on a Big class of IoT devices because it's just so big Yeah, so It's it's that's correct, so Well, if you have a general purpose Linux distribution, it's going to be a lot so and people like to use it. So if you're going to use Node.js or all that stuff, it's not going to be Something small. It's going to be huge. So if we're honest with each other Who would use anything else than Linux if he can use it on an IoT device? We would all use Linux because it's what we know it's we know we have all the software support we need It's it's not small. That's the problem. We cannot fit on those small Socs that are targeted for the IoT space, so we cannot use it for small devices and It has other features as well like it's been it is quite secure. We have Very very well scrutinized IP stack. We have security experts going over it and I'm pretty sure that If you had a choice between a homegrown IP stack or a Linux IP stack You could use the IP Linux IP stack. You could use all the servers or the all the demons that you have on Linux You wouldn't want to write your own SSH server. You wouldn't want to write your own SSL library. You would want to use what's what's ever out there Now if we take a look at the IO capabilities, there was a presentation a little bit earlier by Matt Anostey about the IO subsystem, which pretty much covers every kind of Analog to any kind of sensor that you could possibly hook on a IoT device but you don't have that on a real small device and The the big thing again is price you can get an IoT SoC To be as cheap as $1. You cannot get a Linux system with enough RAM and enough peripherals even close to that range so We have to run IoT stuff on a half a dollar part and Yeah, we could get by with $1 the the target price is for stuff that it's like dust so let's say you have a Feel that you want to You want to put a seat seat there instead of Having to do expensive sensors. What if you had really small expendable Disposable IoT devices that you could use in the seat and just spread them around and then you could read them and see whether you need to water the field or put The fertilizer there, but you cannot do that if the device is too expensive. You need something that's You don't even think about if it gets lost or whatever Again, take a look at light bulbs. So, you know, there are connected light bulbs out there, but they're expensive and Okay, so you can buy a 15 or 50 dollar light bulb, but It's it's a choice. You have to think that I want to buy this Cool gizmo that costs $50 for its light bulb. I won't use but what if The price of the light bulb was just $1 more than the normal light bulb and it would be You wouldn't have to think about it. You just buy the more fancy one without thinking about it and so Since you can't run Linux on those kinds of devices You need another OS operating system for that so I set out and said, okay, I need Real-time operating system for these small kinds of small Devices and we have a lot of options right now So these are the most relevant ones in that space Yes, it is and it's it's it's the mostly used one actually, yeah For really small devices, you don't put anything there So you have free RTOS embed OS, NatX Magenda or little Canon from Google. This is a new thing. It just came out with it You got you have none and then you have Zephyr so I went through it's a nerd one and List started listing all its plus and minuses so free RTOS. I don't know how many of you work with free RTOS or know about it Anyone yeah, so it's pretty popular It has a dual license, so it's CPL version 2 but with a Lincoln exception or you can buy a commercial license But it's API. It's really sparse. It just doesn't have a lot of functionality It's just a scheduler and maybe some memory management. Everything else is an add-on There's not really a community community around it. It's mostly at the company that makes it It has a pre-empty thread model The networking is an add-on as I said, so there's no high-level frameworks. It's a really bare bones operating system For people that are moving on from bare metal which might not be what we need. We might need something more powerful so Embed OS how many of you have heard about it used it so When I started looking at it I did I'm bed 5 wasn't out yet So there wasn't an even an RTOS option for for it. It was just An event-based API based on C++ with the right really nice build system It has an Apache 2 license It pretty much has everything you need for IoT, but it has It tries to push you into using the the cloud of that arm provides so it's really constricting for An open source guy like me, so There are a few Problems with it It's since it's cute based on C++. It's all also uses the most memory. So I think the minimal Application for it's something like 80k Which might not be enough too big but for really small parts that are like 16k or 32k, it's too big for it. I know it's really popular, but it wasn't good enough So NuttX how many of you have used NuttX? Okay, thank you Matt It has a BST license So that means it you can use it wherever you want and you don't have to release anything. It's pretty convenient It really strives to be like Linux, but lighter so it has Pretty large community But if you take a look at the commit logs, 80% of the commits is coming from Mr. Gregory Nutt himself So it's pretty much in one month. So It's positive compatible, so it's easy to port stuff from Linux The problem is it's it is like Linux. It's only the kernel you might get stuff for it But there's nothing else IoT related. So you just get an IP stock nothing else and it's quite After embed OS, it's quite large. So It has more functionality, but it gets to be like 50 or 68 just for the minimal application So Magenta magenta is part of fuchsia. Don't ask me what fuchsia is because I couldn't figure it out It has a MIT license And it's really little kernel. How many of you have heard of little kernel? Yeah, you're on Android. So it's used in bootloaders. I don't know who else where else you use it Yeah, yeah, it's It works, it's really popular. I think it's really popular with Colcomb Yeah, I have no idea but It's popular Yeah, I know I heard NVIDIA does it too, but I Don't know. It's just it's just a bare bones real-time operating system. It works. It's small. Maybe that's all they need It has a little bit of Scaling issues. It doesn't have too many priority numbers. The primitives are pretty primitive. I Tried to figure out whether there was a networking stack and Google supplied sources, I couldn't figure it out. So there must be a networking stack But I couldn't find it and then this is a Google project. So There's a problem with the infant mortality. You never know When Google will pull the plug and then we'll be left with nothing because all of this is Google Engineers and when they're gone the project is gone. So you can see what happens with project error, for instance None so It works and it's the fastest one that you can get but you have to be pretty hardcore. I'm not that hardcore anymore Yep So now we're up to Zephyr The last license is standard Apache 2 license It has a network stack. Well, actually it has two network stacks. It has one for IPv4 v4 and one for IPv6 It's got a nice Real-time operating system API with a nano and a micro option which is pretty useful because if you don't do much and You don't need the micro option You can get by just the nano option and the nano option is excellent for fitting into very small Parts. So with a nano option you can get it down to 8k of flash which is Pretty decent for a full-featured operating system it's really Familiar if you're familiar with Linux because it has a Linux K config build system and That's what I used so I had some experience with it so It's a nice fit for what we want to do. So when we try to do IOT on Zephyr we see that they are Quite enough of libraries already integrated and more are coming So we have libraries for co-op Bluetooth low energy We have Contiki and TinyDTLS TinyDTLS is pretty important It can be really small We can port whatever else we need We know that IOT vendors will provide IOT libraries for it eventually and Now the question is this presentation is about Linux and Zephyr. So where do both of them fit in? so There are some problems when trying to do IOT stuff on Zephyr own and on every real-time operating system that's really small and not very Wide-spread like Linux Security so let's say you have an IOT node and you put a Wi-Fi chip there and you use it for your home application So that means you will have to store Your Wi-Fi password there. You might need to store credentials or whatever If this device is so small that you might have to put it outside the house or whatever that means someone can steal it and if it if he can read the memory he can get your Wi-Fi keys and whatever else is there. So You need to put enough stuff there so that you can communicate on the network, but you shouldn't put too much Because then you have a security problem just because of the nature of the devices to be disposable Then you have convenience so When you have just a couple of devices, it's easy to well It's not easy, but you can considerably update software up perform software update for each and every one but when you have let's say 50 or 60 devices on your house or in the Comple for On the field when you might have a few hundreds or 200 1000 devices, how do you update the software you might need to update it? So It's going to be a pretty painful even if you have an automated method like software update or S3 or whatever Then The next problem is future proofing so You might have enough space to put one a single IoT protocol in there, but Who knows what's going to happen in 10 years? Would you expect to change your light bulbs if In five years from now a new IoT protocol is out and you had supported Probably not and you would be really pissed if you had to throw away Light bulbs that cost like 60 or 50 dollars so The company might go out of business and this happened Which company was it that? had an IoT enabled light bulb and then went out of business and then The cloud stopped working and then the light bulb stopped stopped working It's not it's not very fun, but you have to think about them. So you might need to Make sure that your investment in whatever you buy survives for 10 years and then You have the warring tribes problem When you have you don't have enough space in the flash you don't have enough space to support any kind of Protocol so let's say you have an iPhone and your wife has an Has an Android phone would be nice if you could only control half of the lights in your house It would be easier if you're trying trying for a divorce, but not not really so Yeah, but that's a low-tech solution. We're all geeks here Well, if you can make your wife do whatever you want you're a happy man, but So I think that the solution would be to have a Linux gateway, so you could run whatever IoT protocol you want on the Linux device and Then have slaves if you're IoT devices You have no problem supporting any IoT protocol at the same time on a Linux device because things have gotten really cheap right now you can buy a chip for I don't know 15 dollars and are buy for 20 I don't know how how low it got so it's got enough RAM for any kind of Protocol it's got network interfaces you can easily update it and It's secure. It's ideal, but how do you talk to the sensor on the IoT device? Do you have will you have software running on the IoT device well, maybe there's a solution We have to look at what an IoT SOC Contains and what an IOC sensor looks like if we were to address this problem, so What is an IoT device it's just a cheap SOC typically running an arm core or a peak or or x86 Intel came up with a few on a br Anyway, the idea is it's a very small low end part Not enough CPU. It has a few GPIOs PWM serial some standard interfaces and A way to get on the network whether it's that a Wi-Fi Six low-pan or Could be something custom. So it all depends on what you use What else is there that are some analog glue interface? So you might if you have a Sensor that breaches humidity you might need Just an I2C bus so that you can plug the sensor in and then read The data it's very price sensitive because If you have something too expensive, no one will buy it Especially if they're they're parts in China, even if the software sucks If they're if it's if you're 10 times more expensive than the other guy No one cares whether you run Linux or whatever else. The price is really important you don't have enough a large enough battery you just have a very small power budget and Less is more so that means you you should get by with this With a smaller system that you can you can find and you should really design for it so let's go back to Linux so Linux has the full IO capabilities that an IoT device requires you can read the humidity sensor you do have GPIO devices Device support on Linux. You do have PWMs. You have everything that an IoT device Uses but your sensors are not connected to your Linux box. There are somewhere else so instead of running a Full IoT protocol on the sensor Why not just get rid of everything and make all these effort devices simply Linux peripherals That that means you get your software running on Linux. Your sensors are really small They're cheap because you don't need anything else besides a networking stack The software load on the devices is not different. So let's say you have Sensors that feed that are based on us on a similar SOC You don't really need software on the sensor because everything is running on Linux and it will just access the devices directly without having different software load for its different sensor type and You can abstract all of that with by using Linux kernel interfaces so you can present a Remote GPIO as a local GPI on the Linux device We can present the remote humidity sensor as a local humidity sensor so everything works as if The sensors were connected on Linux and the applications on Linux work exactly as if the sensor for that has no No, no, so no software difference at all. You just need a way to transport the peripheral Commands from the IoT device to the Linux security problems we have Much less of an attack surface there because all the sensors are completely dumb. They don't contain anything that's important You don't have to use the same communication protocol that you use for IoT you can Engineer something that's really small. It's secure enough for your needs for transporting Samples, but there's no big importance to the sample. So you don't have to have have great security You just have to have good enough security for for that so you don't have much software there and If you can get by by program pre-programming The keys for the communication in its device Without using the network interface that means it's more secure. So I Don't know how many of you were in a presentation that some Disney guys had with They made a lead based communication network They were cheap leads and the lads can also work as a as a receiver, so You can get by creating a very very cheap communication meeting using standard leads You can program them with an iPhone iPhone while smartphone and you can put the Pre-program key in the device without using the network interface. So when you start when you go to provision the device you just flash it with Smartphone you set the keys for the communication the communication keys with the Gateway and then when you're done, you don't have to use anything else You just pre-program the keys like you were if you if there was a serial port and you Program it by by hand. It's much more secure than having to manage keys and the device then is Disposable you don't care about it. We've even if you lose it. There's nothing important there so Now let's go to some implementation details So there are two ways to go about it. So you have you can go bare metal which means you just You directly have access to the Small ASOS resources like GPIO registers PWM registers and You Essentially have enough software in the IoT device to boot present itself on the network and then you directly poke registers and Have access to the ice-cold sea buses as PI buses from the Linux Gateway That means you don't have any software at all on the device. It's you can even put it in Rome and It's time you use You use it from the Gateway performance is not very good though because instead of Transporting assembles or the data you might need to perform the register accesses over the network That's lower and you might need more power because the radio would be turned on for for a larger amount of time well The second way would be to have class drivers So you have a thin class layer Where you present a GPO GPIO class driver for the remote peripheral? A sensor class driver for its sensor you might connect there So that means you might have a different software load on its IoT device, which is It's not very desirable, but you get better performance and less power Well, I don't have a demo yet. This is my 20% project I have a habit I Do my work on a big bone and a little Galileo I have something that provides the remote GPIOs, but no i-square COS PI to come I Use device 3 overlays to load and unload the sensors and the GPIOs from the bus where it's useful I do that over Bluetooth low energy. I use pre-pro configured keys with DTLS and I hope I'll have a demo on the next embedded Linux conference in Portland, so I Think that's it. So Any questions? Yeah, because that's not a Google project. It's It's Greg gates products project I could there's not nothing that precludes me, but I can get by with device 3 overlays as well It's it's an employment day. It's an implementation detail if you can present remote peripherals Other way would work so you can you can support both if you want Okay, this this is getting philosophical I Don't know the the target is something that you don't care about if you lose it So if there is any other impact, I don't know environmental or something. I haven't thought about it, but I don't know Maybe there's a biodegradable biodegradable electronics in a few years, maybe Yep Yeah, but it's the same As far as the upper layers are concerned It's the Linux device. That's the IoT sensor node. There's nothing The upper layers don't know anything about also. It's the same thing as having a failure of an IoT device But it has more interfaces You can have two gateways There's nothing that precludes you of having a failover, but I haven't thought about that yet. So Yeah, most of the stuff is in kernel of the user space to see is the user space interface as if the peripheral was local so Yeah, you have a helper that goes out of the kernel Communicates and then fits it but fits it back The idea is so that you don't have any difference between a local peripheral because if you do have a gateway You might be tempted in putting a few Sensors there as well. So you don't want to use it to lose it Know the whole idea is that this is fixed if the IoT node gets up and communicates and Authenticates because you have to know the pre-configured Key then it will show up. There is no no authentication mechanism. That's the point so that You don't want it to be complicated. I don't I don't want to reinvent other protocols. It's just one something for transporting peripheral data out in and out anyone else It's not tight at all. You have a You will have a well-defined kernel interface something like WPA supple gun or something Which uses a netlink socket and then you can use whatever you want to aggregate stuff so there's no No tying to any kind of networking technology if you can push packets in and out it should work Nope Sounds like that might be Yeah, it's still an Intel open-sourcing hell. So the code is not public yet, but soon. Well, I didn't know about that But it does make sense because even for the user space Approach I would use a Crypt dev interface so that I could offload a few stuff out of user space if I can because You're you're gonna be running on an embedded part and you don't have enough too much power Because I want a very hard interface to be exactly the same as a local resource on Linux So the application developer doesn't know anything about all that stuff He's just working on as a Linux application accessing Linux peripherals So you don't need another interface. It's The interface is what the kernel provides So if you have an application that works on Linux it should work Without any changes just point it to the remote peripherals and get it work and that's it Well, there is Zephyr running on the IoT node because I need the networking stack No, no, there is logic on the endpoint, but it doesn't have to do anything with collecting data That are IoT related It doesn't need anything but it does need a networking stack because you need to If you can supply the implementation because I will only do the Zephyr implementation I won't do anything else because We do open source stuff if you can there's there's gonna be Normally, yeah Yeah Yeah, but I don't want to tie it to Bluetooth Bluetooth is just a single Yeah, yeah, if you can get it to work on anything else No But it's easier if you just use Zephyr because we will provide An implementation that works and then it's easier if you just use Zephyr It's gonna be based on Nano Zephyr, which is really small to state K So it's not that much of an overhead for what you want to do because okay You might want to do more stuff. I don't want to be tight exactly to that model You mean how many devices? Yeah, because it's overlays are typically like 200 bytes 300 bytes Yeah, it could work, but you increase the storage requirements and you need to program it on the device Yep In theory you could just have it programmed from the factory with ROM without any flash whatsoever there's some other ideas about using Things like eBPF fragments, so if you need to do some kind of processing if then like PLC stuff in theory you could do an eBPF Program on the host jit it into the target Target Compile it and then transmit it and have it execute But that's a bit further off, but it should work as long as you're willing to Develop it, but the idea is that you don't have any any kind of software that you don't need on the note. It's just Standard stuff that will always work. Yeah Whatever you just need a way to transport packets Using a beer Lee but No, I don't need it anything else, okay, that's the case. Thank you for