 Hello everyone. Sorry for the small delay. I'm Igor Stopa. I work for Intel. Today I would like to talk with you about my experience advice, survival course of how to build IoT device. I hope we can have it as an interactive session, so if you have comments, if you disagree violently with me, feel free to just press your hand and let's discuss about it. So, couple of monetary disclaimers. What you're gonna hear is my mostly in a sense or opinion. I do not claim to own any trademark or copyright that I will be using. So, what are we gonna talk about? What is an IoT device? Why do you want to build one? And what it's gonna do? Where it's gonna live? What are the constraints you have to deal with when you create a IoT device? What are the hardware software options? The idea is that here you want to create something maybe for yourself, maybe because you want to sell it, it doesn't matter. But the idea is that you want to be as fast as possible, as effective as possible in delivering something which makes you or your customer happy. So, there's also part of selecting the tools. And then how to integrate it. And then some final considerations about what could be done on top of that. So, what is an IoT device? It's a semi-autonomous agent producing and or consuming information, mostly by interacting with other IoT agents. Doesn't say much, but that's what it is. You have this thing and it's supposed to go on mostly by itself, unattended, chitchatting with something else like that, possibly also with something on the cloud. For those who like me start to have some gray hair, this is not in particular different from embedded device. We used to call them SCADA or with other words, now the buzzword is IoT so we go by that, cloud is the other one. The difference is that what previously used to be SCADA costing north of thousands of dollars, now you pay a few bucks and it fits in your pocket and the battery probably lasts forever. So, you can finally unleash all your dreams of becoming a spy or monitor, whatever you feel like. And actually what I'm not gonna talk about it here, but I find even more scary is that you can correlate it. I mean, that's what we saw in the introduction with all these cloud services. There's a really unprecedented computational power available. So, not every IoT device looks the same. If you look at them with, yes, like a tree. So, you have a cloud which is high above and then you have nodes and leaves. And some devices in the IoT world are kind of smart. Some others are less smart but probably cheaper. On some you can run Linux. On something else you cannot or do not want to run Linux because, for example, it's more convenient or appropriate to run a real-time operating system because maybe they are microcontrollers. And they also get to operate in very different environment. If you had them on the internet at wild, they are more likely to be exposed to certain types of threats. Depending on what sort of services they are providing, they are more or less subject to certain types of attacks. And depending on what sort of information they deal with, they might be subject to tampering or not. If you are making some set of box which can play movies, it might be that your user is actually also the one who feels like tampering with it. But that's not necessarily the case. From my perspective, what is more interesting is the first of these three points. You might think that your local LAN at home is safe, but how many of you have some box that you got from your internet provider or some Android player, something like that? I have one. And it's already a couple of Android versions behind. In practice it's something which I do not fully control because I bought it. I do not have any idea what software it's running and it's exposed to the internet so potentially malicious applications can touch it. So even if you think that you are deploying your IoT device in a friendly environment like your local network at home, that's not necessarily true. What do we want to do and what is going to limit us? No matter what's the type of device that we are trying to build, we want to do it fast. And we also have a limited amount of resources. Think about it as your hobby project so it might be something that you do in the spare hour you have in the night or the weekend or it might be that you have a small team of people and you really want to focus on what you think is going to make a difference for your customers. You want to waste your time and money doing something which someone else has already done. Constrains, certain types of use cases are not standard so you might have to deal with, for example, ways to make your device interoperable with others and there are several competing standards. They are trying to unify them but you might have to make a choice which means that in a sense you are betting your money on the fact that the one of the standards that you will choose or more will be the winning one if you are in for the long run. So the type of device we want to build is a high-end leaf compared to the description I gave earlier in which it means that it runs Linux, it has a certain amount of computation and power on board. It can be considered almost like a PC or almost like what in the past we would have thought to be a PC. From this perspective and from the fact that we do not want to overoptimize, the idea is we want to be able to treat this device in a standard as possible way. Either we would be able to even develop over it to minimize the amount of tools that we have to deal with. My advice in that sense is it's not so important to choose this or that distra as long as it's something that you are familiar with, something which you can command fluently. Almost every major distro, actually every major distro nowadays is a good quality, much better than what someone might try to achieve by working on it by himself. Then there comes also the question how long you want your device to be around. As I said, the box I bought was very cheap, it was running Android, it was version 4.something, and now it's not getting any love from its manufacturer. I cannot run Netflix on it anymore, I cannot install a certain application anymore simply because things have moved on. I'm under the assumption here that you are trying to build a device where it's providing a useful service and you want it to continue to be usable in the future, which means that if you rely on some hardware manufacturer to be your software vendor of BSP, for example, you might get stuck with some old kernel and that's gonna be a problem in the long run. This is the sort of consideration that I would recommend to have when choosing the hardware. Ideally, you would not need to have any special patch to maintain yourself or the distro you choose. Ideally, your device would run on a kernel from mainline. Why? Because most likely then it will not be used by one single distro, but by several, which means that the amount of people using this device is much larger and therefore you have much better chance of having a software base which has been tried and tested and debugged compared to the single DM vendor. The last point is how critical is the data you are gonna have on your device? If you are doing a media player, you might have some legal obligation with content owners. In other cases, if you are dealing with people's sensitive data, then you have a different type of obligation that you have to fulfill. In other cases, it might be security. You don't want to give out to easily personal data of your users. So my recommendation is the approach to the security solution you choose should be proportional to the criticality of the data that is gonna live on the device. And why am I saying all of this? Because what we want is to minimize the total cost of ownership. So that's a bit of material, of course, but that's just the most obvious part of it. You are gonna spend time learning new tools if you have decided that you have to use something that you don't know. So in that sense, my question is, do you really want to or have to do it? You might have decided that this is a good learning opportunity for you. Then so be it, but it must be a conscious decision. The development time, how painful will it be for you to go through the iteration of developing new software, deploying it, testing it and then going back to the development board? Maintenance. This is life. Things will be taught for sure. So you might have your nice application and then after a while you find out that you have to update some of the other components and over time slowly or not so slowly your API or ABI change and you have to maintain what you've written. So my personal conclusion is the more the device we are talking about looks like a PC, the more we can apply methodologies which are typical of the PC world and they have been refined over the years and you have a wealth of resources that you can leverage. You are far less on your own so there is a much bigger community than if you choose some custom niche board which might be the sexiest of the moment or it might be slightly cheaper but on the other hand the ODM might disappear in one year or two. So when does this make sense? When your resources are constrained. In reality they are always constrained but in some cases you might actually have ability to afford a larger spending larger investment. In other cases you might not. Typical example is if you are doing something for yourself as a hobby project it's just you. And do you want to waste your time playing with the distro or do you want to focus on working on the use case that started it all? For this perspective the idea is to reuse whatever is available and try not to reinvent. The hardware cost as I said typically is not that critical. Nowadays most of the boards they are almost equivalent cost wise and performance wise. At least unless you are doing some really high performance application there is going to be very little difference between one board and the other. The simplification is the user is not a threat which might not be true for some cases as I said if you are doing a media player something which is playing some protected content but if you are going to be the user then I hope you are not going to hack into your own device. And similarly when you really want to leverage what many others have done other case which can happen in certain situations you don't want to waste your time waiting for packages that are not part of your core application to rebuild. In some cases you end up having to rebuild a multitude of packages. To make a practical example I've picked the Minoboard Toolbot. It's one of the other devices that I know best developed by my same organization OTC open source technology group in Intel. It's small it behaves like a PC because it has a UFI BIOS it doesn't have a fan it has low power you can out of the box boot I've tried a few distro I've tried Debian, I've tried Wound they all work it has a USB 3 port so it provides speed and expandability. Other alternative that I have not tried but should work are the Raspberry Pi 2 and 3 Nowadays I am told that they are supported out of the box by Ubuntu. What I do not know is if Ubuntu is using some out of kernel patches or not. That part I haven't checked but you might try or you might not care. In my case just to pick one I've chosen Debian, but really you could take whatever you are comfortable with even Arch Linux if you really like. The idea is really do not waste time relearning the basics use some environment that you are familiar with unless you have some gap to fill and what you know is not capable of filling it. So just get it installed in the case of the minoboard there is no internal storage. What it has is two USB ports and one SD card which is located I think it is located underneath. Sorry it is here so the idea is to proceed with the installation put installer on a USB key installed on the SD card and choose the most minimalistic configuration available. After that basically all you need in most of the cases is just enable SSH services enable a firewall make sure that the only service that can access it is SSH and that is your starting point. I really want to stress the part about enabling the firewall if you are not familiar with it there are websites where you can kind of specify what you want and it will generate some rules. It is not perfect but it is better than nothing it is a starting point even if you want to play with it by yourself it is already something that should work. At the end of the presentation I have collected a set of links you can refer to those. One thing that I like about the minoboard is that since it does not have internal storage but it uses SD card once you have created the base image you can just duplicate that and then you can happily go and hack without carrying if you are ruining some of the installation because you can just restore a previous image then your application I haven't defined it yet in the example but the idea is do not necessarily try to use some standardized protocol unless you need to you probably have to exchange some data so whatever comes familiar to you it can be use data stream archive, http page just go with that in many cases when you embark yourself on this sort of adventure there are surprises waiting for you which you have not well they are surprise so they are unexpected so what is preferable is to get somehow at the end get your use case somehow running and then go back and optimize parts where you might have intentionally used some hack because at least this is a more controlled way of refining your creation, your IoT device premature optimization can be caused for delays or even cancellations in time of a product which another way might mean that you just get bored or frustrated and that is not what you are going to do other easy way out of one of the typical problems with IoT with IoT you are typically using some sensor actuator and most of them come over some standard buses, they can be I2C, SPI, camera buses, one wire whatever they are nice, they are optimized but my advice is go with USB if you can USB is old it is not particularly optimized bus but exactly because it is old it has been debugged quite thoroughly if you have chosen proper hardware you also have good kernel drivers and the upside of it is you do not need to develop on the target device because your main PC has a USB port so you can even develop your application directly on that the major drawback of USB is gonna make your device to draw higher power but as long as you are doing some application which is ok to deal with the world world so if you are not running on battery, USB is the cheapest and easiest way out of your problem and most of the sensors are available either as a USB device or as whatever bus they really have USB is a convenient way to hide the complexity of the sensor behind some more standard bus at least the board I choose has one USB 3 port so throughput is not even a problem in some cases you have USB 2 port you might have issues with throughput but with USB 3 no as very simple example I choose to create an IP camera so in this case all you need if you are following my advice to use USB is just webcam you don't need much more than that you need a webcam you already have support for it with Linux nowadays I haven't found a single webcam even this 50 cents one that you can find on ebay that wouldn't work with Linux and then you need gstreamer so the lot of gstreamer tools on the SD card that you have or rather on the copy of it put couple of comments that show how to set up a simple streaming server and that allows you to stream video from inside the device now of course you are not connected directly to the device so what you want to do is to stream this video elsewhere again I'm gonna use a hammer like solution and secure proof just establish in the way you prefer SSH tunnel to the device and forward the port where the streaming is available and you have already created your streaming IP camera this might be suboptimal because I think it generates twice the encoding so there are ways to do it better but there are ways I would be more concerned about keeping the device safe secure it's not uncommon to read in the news of some C device which has some security hole so that random people can log into it actually I think there is even a search changing for these cameras so my main message is no matter what you do avoid growing the attack surface when you are exposing your sensitive data to the network be overkill in that sense do not try to optimize it rather make sure that you keep low the number of ports you are opening we already have opened the SSH port so this solution is basically reusing that it doesn't require you to open more holes in the firewall and then I've put reference 11 to the command that is used to see the stream this is the part that I said is optional that's why I call them pain points you might need them if you are making a C device something which will be produced in many units something that you will not be managing yourself so for example for a typical PC installation you are probably happy to just use the software editor that comes with your distro but if you are making a real C device then that's not enough you will need some form of software editor you might want to add additional security features and you might want to add some standardized APIs for interoperability coming to the software as I said the first simple option the one that comes for free is to use the package manager of your distro it's well tested the problem is that typically they do not provide the atomicity to the update so you might have something that goes wrong and your device is in some intermediate state some packages have been updated so if you are managing the device directly that's probably non-issue if your user is competent enough to just flash a new SD card and replace it, that's again non-issue if your user is supposed to be blissfully unaware of this problem that can occur then you cannot rely on the distro package manager, you need something else something else is typically fully in two camps one is you just you have to develop some means to replace the entire image I think there was a speech about it yesterday it's reliable in the sense that it's a simple operation you just stream the entire content of the new image possibly some small optimization if you have zeros you can compress those bandwidth intensive in general the other approach is to use one of the many solutions that are currently being proposed there's oyster with flatback snappy, they are based on containers there's a clear Linux approach which instead uses bundles they are far more optimized to be frank they have a smaller use base so when you use one of those you are basically deciding to be part of the guinea pigs so if you feel like guinea pig go with that security there's a lot of stuff that you can do I just listed few of the options that are the more recurring ones and each of them comes with some advantage if you know how to use it and that's a big if I don't think to be one of those people who know how to use it the question is first of all do you really need it I mean try to think about what data is gonna go on your device what's the worst scenario that can happen and then decide if it's worth the fourth also you might actually do something bad when you're using one of these solutions and that's just about the outcome the other part that will happen almost for sure is that it will make your life more miserable when you develop on it because this security solution per se they tend to make it harder to replace stuff on the device so it means that during your development cycle you will have to develop means to somehow cope with this additional difficulty so really the question is is it worth I'm not saying that it is not depends really on what sort of device you are doing and that's why I was tunneling the video stream through SSH because in that sense it's something that is I think more likely to be achievable by random linux guide using proficiently one of these security features that I've listed here and finally interoperability we saw a nice demo about Node Red during the starting keynotes there's OCF Intel, my employer participates, so let also is another flow base solution they're nice again the question is what do you need them for because you will have to create bindings for this so do you have a use case where for example you want to do something with your phone or with some other IoT device which will interact with it that might be an answer it's up to you again I'm not saying that it's good or bad but it should be a conscious decision what I can say is that at least in the IP camera example that I whip together you do not necessarily need any of this for example you can just set it up as web page secure web page log into it with your browser your phone and that's it so the question is do you want to do something fancy do you want for example to run face recognition on the camera that might require some cloud service and that might justify creating the bindings for it but you really need to understand your use case because there's gonna be a cost in creating the bindings and maintaining them so this is my final message if there's anything I would like you to take away from this top is be opportunistic don't go for the nicest cleanest solution go for the most effective one the one that you are more comfortable with something that you feel you will be able to debug because there will be a time where the thing falls apart and you don't know why and you have to debug it so be nice to your future self is there any question thank you for listening