 Linux and there's no need to think of any other operating systems. There are, of course, alternatives. I can't think of one from the top of my head. It had something to do with things you can look through, but I don't remember. The left part of the figure, so we see how I can change color in my very innovative approach of displaying information. The left part of the figure, so this one here. The network node is the class of system that interests me in this talk. So that's systems that still are 32-bit capable. Of course, there are also users of eight or 16-bit systems. And maybe there are even operating systems that support such eight or 16-bit systems. But these, I firmly believe, will really be part of the past in a not so distant time in the future. We're talking about clock speeds well below 500 megahertz. We're talking about megabytes of non-volatile storage. And for M, I've said here it's less than eight megabytes. There are arguments in the Linux kernel community that you can actually trim down a kernel to less than eight megabytes, which you comfortably can do. But at some point, the question rises. You need to strip away so much stuff of Linux that it basically becomes something fundamentally different than Linux. And so there's not much point in using it in the first place. So for me, the boundary between the non-Linux and the Linux domain is this eight megabytes of RAM. So stripping down Linux, there is, of course, some opinions in the community. If you ask the network maintainer that came up in the thread like one and a half or so years ago, he was asked, when people discussed if we should provide Linux for the internet of things or not, what parts of Linux would you remove to get the footprint down for a two megabyte single purpose machine? And his answer was very simply, I wouldn't use Linux and a story maybe two decades ago, but now those days are over. There's, of course, some disagreement about this in the kernel community. But I tend to agree with this statement. If you strip down Linux beyond a certain size, that's not necessarily two megabytes, but four or eight megabytes. It simply stops becoming Linux. So your maintenance efforts will be so high with the stripped-down versions. Your programming model will be so different. You don't have all the device drivers you're used to. You don't have all the core kernel infrastructure that you're used to that. It's maybe called Linux, but it's just something different than Linux. So that's a reasonable point of view. Now, if we say we need a different operating system than Linux, of course, the question is what to pick. And the Linux community also has some opinion in that regard. So from the Linux kernel tignification efforts, frequently asked questions, there's the statement, Linux has a lot more longevity and generality than most embedded operating systems. Most such operating systems are proprietary. All of them lack the range of capabilities, drivers, and general level of code quality and revenue found in Linux. So being modest has, I guess, never been one of the defining properties of the Linux kernel community. And they also say most of them have far smaller communities or no communities at all, which is quite a statement. So let's spend a few minutes looking at, oh, and actually, so Weltanschauung, I really like the world Weltanschauung. It's your point of view that you take or that you, how you think the world does look like. Of course, Germans have a word for everything and Weltanschauung is one of these words that are barely and that are basically untranslatable into anything different unless your mind runs in German. But Weltanschauung need not necessarily be true. So let's compare Weltanschauung and veracity of these statements. So of course, many parts of Linux. And I guess I don't need to bring all the other ones here are indeed of very high quality. That's without no debate. But this also comes at the price. If you remember how hard it was to build Linux systems 10 years ago or 15 years ago embedded systems, if you wanted to attach another input source to a system that was basically very easy 10 years ago, these days you have to deal with layers and layers and layers of abstractions. You have to go to input abstraction layers. You have to go through devices. You have to configure and reconfigure your kernel. You have to deal with things like system D. You have to deal with automatic device generation that's supposed to work most of the time but then doesn't most of the time. So we're seeing a tremendous complexity that comes with this very high quality and generality of systems and the question is, is this really necessary? Plus, the Linux community may be big but if you look into corner cases, corner cases in inverted commas like real time that are quite crucial to many internet of things and data acquisition solutions, then the sizes of the community actually are not that large. So I guess it's safe to say that for instance the pre-empt RT extensions to the Linux kernel may be understood by three or four people in the world in their full generality but not much more if you look into approaches extensions like Xenomyon and so on and it's a community of two or three people that drives these efforts that is actually not much more than or even less than many proprietary or competing operating systems have. Another perception is according to LWN.net which usually reports quite true and interesting stuff is that a growing kernel makes it hard for people who are trying to build tiny system, forcing them to go to a proprietary real time operating system instead. So again, we have our black and white, that's quite popular Linux. So if it's not Linux, then it's proprietary and if it's proprietary, then it's of course bad. I do agree with the second part of the statement but I do not agree with the first part of the statement. So if you look beyond your Linux realm, you realize there's actually a lot of alternative real time operating systems, embedded operating systems that are available in open source form. That's Artemis, Ecos, Contiki, Riot, M.Bud, Free Outer, Smuzy, Linux, Fredx and tons more funder would not and of course still there are some proprietary operating systems but I'd say it's much harder to actually find a proprietary operating system, a small operating system these days than it is to find an open source one. So there's quite some misperception in the Linux community in this respect. Also the statement that these systems have no communities at all, I wouldn't say couldn't be further from the truth but it's not quite what you see when you look at the numbers. So I did this comparison a while ago, you see the numbers go to January 2015. Actually I wanted to update the numbers for this talk but I was comparing postings on mailing lists over time and when I wanted to update the numbers, of course the G main archive went down and worthed all the glorious statistics that you could get from them. So maybe things have changed during the last year but I don't expect that in any case, sorry for providing you with old numbers. What I just compared very quickly is the amount of postings on the RTEMs mailing lists, on the mailing lists of the relatively new riots, IOT operating system and on specialized mailing lists for preempt RT and as you can see, RTEMs, so the green line and the riot, the blue line actually have much higher volume of discussion on the list than preempt RT. So in fact, you can't really say there are no communities at all, they are of course smaller than the Linux communities but there are quite substantial communities and these operating systems are really alive. Okay, so after this debunking some myth section about non-Linux operating system, let me come to some of the conceptual differences between the internet of things operating systems and traditional Linux based embedded systems and what you see here is not an empty slide but that's a template slide to adapt the content of the talk to the audience's expectations. So I heard that many of you have already quite a lot of experience in developing non-Linux embedded systems so I guess I will shorten this technical part a bit what I wanted to show is basically is the basics of non-MMU, non-process supporting operating systems. Well, let me go through that quickly as you all know in Linux we have our beloved address spaces so we have on a, we especially don't have a working back mechanism on this non-Linux internet of things device, but well, so we have an address space and a 32 gigabyte system that provides us with four gigabytes of RAM and Linux of course we have our distinction between kernel space, here roughly speaking in the top gigabyte of the virtual address range and we have all our user space payload code in the bottom three gigabytes of the address space for multiple applications for multiple tasks that run on the system we have thanks to the capabilities of the memory management units multiple of these virtual address spaces each with its own range for the kernel but this and its own range for the application payload so these two applications are separated by the processor as I guess you all know and cannot interfere with each other so it's a very good virtualization mechanism in a sense that every process sees its own copy of the hardware that's usually not the case in deeply embedded operating systems and IOT operating systems for one thing the distinction between embedded the distinction between kernel and user land of course gets less important both kernel code and user land code run in the same privileged level of the CPU so you have the kernel lying around in RAM somewhere you have one user land application lying around in RAM somewhere you have another user land application lying around in RAM somewhere and the user land application basically could touch the kernel memory whenever it wants the user land application can touch the memory of other applications whenever it wants which of course brings some changes to the way how you think about your software architecture your software design but when basically this sounds more more disturbing than it actually is if your applications behave well if they only touch memory then they're supposed to touch it they have allocated if they don't try to meddle with the hardware details of the system too much then basically you won't see any difference except that of course you need to think more about what resources you use in the first place instead of just allocating as much as you can get and then use it okay moving on from that here's my next template test slide when it comes to small embedded systems they're usually real time operating systems and that's one of the most fundamental differences between IoT operating systems and embedded Linux as you all know embedded Linux is an interrupt driven system which means when you look at how time progresses in the system some application is running application A then it's something some external event happens to the system an interrupt is raised the CPU is notified that some external activity has happened then Linux transitions from user modes to kernel mode processes the interrupt in the kernel maybe a network packet has arrived maybe a key was pressed on a keyboard attached to the system maybe some sensor delivered data the kernel somewhat satisfies the needs of the devices of the device I'm raising the interrupt and then goes back to executing the user land application let's say here we switch from application A to application B because its time slide has expired when application B has used up its time budget its stand time to execute an application C then another interrupt may come in triggers a transition to kernel mode and so on guess not much need to tell you more details about that since you all seem to be very well aware of it the fundamental difference for most IOT systems is that an interrupt driven system is generally very hard to get right so there's in the throughput oriented system no real alternative to that but the much simpler way of looking at systems is to assign the computational resources upfront and not use this interrupt driven design where basically things can happen in some completely unpredictable order that cannot be planned ahead but you can use a deterministic schedule you can say okay even before running the system I know that this application needs about 30% of the CPU time this needs 20% this needs 10% and then you partition your available time in a fixed roster you say okay we have here certain temporal intervals and I always give these first two intervals to application A then application B gets the next time interval then application C gets the next time interval and so on which simplifies scheduling actually quite a lot because you don't need to dynamically keep track of dependencies between processes you don't need to keep track of different priorities of processes you don't even need to care much about external events in the most extreme case because if your application cyclically will get CPU time anyway you can just then at this point check if there was any external interruption if any data has been supplied by some external device and then do the processing then it doesn't really matter if this is immediately coupled to the cause of the interruption or not why small embedded system take this approach is clear the whole scheduling code which is tens of thousands of line in Linux and requires quite extensive data structures basically is boiled down to a few very simple lines of code okay moving on to my what I think guys my final template slide another major difference between embedded Linux and IOT type operating system is the way how you build appliances with an embedded Linux box as you all know you install some sorts of distribution on the device that can be some embedded Debian that can be provided by the Yocto project that can be based on Elbe so there's a tremendous amount of possibilities for doing that but in the end of the day you have some storage on the embedded system that contains a kernel your base distribution you can access the system via some communication mechanism be that SSH be that a simple terminal connection and the software that you actually upload to the target is quite variable so you upload a binary you run that you exchange a binary you run something different that makes updating very easy that makes dynamic provisioning of the systems very easy but the crucial point is you build or building your system is very much decoupled from building your actual applications even if you do that as part of the same distribution mechanisms it's two completely different entities with internet of things operating systems what you usually build and deploy on your machine is one big binary blob so one application file or app as I have learned this is called this day in the app contains of course your payload code maybe plus the description of how many tasks you want on the device of how you're scheduling time should be distributed and so on and that's packaged together with the kernels so that forms one entity that you've then put on your device that runs and then gives you a very much more static system setup that you get with most embedded Linux appliances and there's one quite interesting legal points to that type of setup when you bind the kernel and your userland payload code together which is of course quite obvious to anyone who is aware of the problematics of OSS licensing who is aware of the problematics of linking and so on but what we've seen quite often in the wild is that people do not really explicitly take care of do not really explicitly consider that or are so used to embedded Linux development that they don't see any coupling between the kernel and between their payload application code and that's because of course in Linux when you transition from kernel to the userland you do that with a SUS call you don't do that with static linking so there is effectively a license barrier involved in every Linux system that's decouples the license the GPL license of the kernel from the license that you want to put your payload code under be that a proprietary license be that something different now if you go to these IoT type operating systems and combine userland code or payload code you actually can't really call the userland code because it runs in kernel mode combine your payload code with the kernel code then this license barrier this system call barrier goes away and in the worst case depending on the actual license that the embedded operating system has depending on the license obligations that the system imposes on you that means that you could even go up to inheriting the code of your operating system for the code of your payload applications and I've seen instances of cases where companies wanted to keep their code proprietary but were actually forced to publish their source code because of the linking issues because the GPL like licenses of the embedded operating systems required them to do so that's of course the worst case but if you haven't considered that if you're coming from a strict Linux background and haven't considered that then I do really pay attention to that that's of course something you don't want to have coming from the difference to the communalities if you're doing development for these IoT type operating systems then there's still a lot of stuff that's very similar to when you do Linux application development so you use the same tool chains for most of the IoT class systems GCC is the tool chain of choice you do cross build your application so you don't build on your device that sometimes happens for embedded Linux but that's in my opinion not really the same choice you use the same build systems you use the same version control mechanisms you even use the same debugging mechanism so it doesn't really make much of a difference if you use an ADB debugger and ADB bridge and pass it to GDP or if you use a traditional GDP server on your device or if you use GDP server on a deeply embedded device that's everything in the end of the day in the front end the very same thing what's also important is most of these IoT class embedded operating systems use C or C++ as the major programming language there are of course exceptions you can also get Lua based operating systems in the device range I'm talking about that most of them are based on CZ++ and even if these systems don't support POSIX, a POSIX API and many of them in fact don't that's not so much of a difference than one would initially expect because if you look at your code and think about how much is really POSIX and how much is standard C then standard C outweighs the POSIX elements usually the orders of magnitude plus POSIX is by far not the only library the only interface specification that you use in moderately complex applications you have anyway tons of libraries that have nothing whatsoever to do with POSIX and in a way if you see the API of your IoT operating system is just another library then programming the system is then not much else than picking up learning a new library for standard embedded Linux development. Okay so from that moving on to different properties of different qualities of IoT type operating systems of course in a 45 minutes talk there's not enough time to go into detail for any of these operating systems and the selection I'm presenting you is of course based on a fairly lengthy evaluation that we did with the different systems we looked at them we looked at their source code quality we looked at their maturity we looked at how many architectures how many drivers they support and so on but in the end of the day it's a bit more subjective my subjective insight into these I've compared three RTOS, RTEMs, MusiLinux, Mbed and Sapphire and yeah basically there's not so much to say about this table I guess it's quite self-explanatory when you pick an operating system for your projects you know yourself what's most important for you do you want a really mature operating system is it important for you that you have comprehensive drivers are put because your hardware varies how much emphasis do you put on documentation quality because this varies quite wildly and most importantly how many resources in terms of RAM, in terms of storage can your device provide and then this table may be a good starting point for you to look into more detail into a system in more details a few things about the candidate systems that we've surveyed so RTEMs is the real-time executive for multi-processor system that's a decade old system stemming from from military use cases the most aligned points that can be said about this system is that it has a really comprehensive support for CPU architecture so we're talking about similar support that the Linux kernel provides even if it's of course much less unknown but basically there's no architecture no reasonable architecture on the market that you won't find support in RTEMs where it also has fairly comprehensive support for different development boards it's actually sometimes a bit hard to judge if they say that the board is supported and if a board is really supported so that documentation regarding what maybe has been supported in some ramshackle way or another 10 years ago and what is really supported is bad but still overall their list is quite impressive and if you come from a more traditional background there are commercial vendors at least two commercial vendors involved in the development of RTEMs you don't hear this argument about geeks and hippies building Linux system much these days anymore but now people start the same argument with the non-Linux systems that's all just geeky stuff and so we cannot use that in our precious commercial development but at least in this system you really have good solid commercial vendors who can provide support so it's also very easy to transition from embedded Linux to RTEMs because you have POSIX support up to two minor differences you don't have a virtual address basis so you of course don't have a fork, exec and so on systems caused provided by POSIX but 99% of the POSIX standards are covered by RTEMs you have Berkeley socket support so if you have a traditional network application then it's also just a matter of recompiling it for RTEMs you have different networking stacks in the system one of free BST that's a little bit larger and the lightweight IP networking stack that's a very compact networking stack and you even have support for memory protection units so which is kind of in between between full MMU full task support and having a single address space but making sure by processor mechanisms that applications cannot interfere so much with each other. Free Artus is another more or less hybrid between the architecture of a standard embedded Linux system and an IoT system you do have virtual address space support in the system so it supports MMUs you can have tasks that are separated by the very same mechanisms as they are in embedded Linux the major difference is that you can really strip down the systems that's a highly configurable system you can strip it down to a few kilobytes of RAM so if you look at their source code at the details they count the amount of bytes they count the amount of memory that you need to support another process on your system in bytes so in Linux that's we're not exactly talking about megabytes but quite some substantial amount of memory just for keeping the data structures in RAM that are used to represent a task in free Artus you're talking about bytes it also has a number of TCP IP stacks Berkeley it has a Berkeley sockets API the major drawback I see with Free Artus is that it has a very small community which is quite astonishing because it's fairly popular in many embedded projects but if you look at the community closer I cannot discuss the details here for time reason but it's astonishing with few people and so that's of course your choice if you want to trust such a small community or not in that case the argument about Linux being so much more stable in terms of community does apply coming to embed that's a very nice alternative when you're doing first time deeply embedded development so they have an SDK that's as far away from POSIX as could be that can be a good thing or can be a bad thing depending on how you see it it's also mostly based on C++ so it's a very nice opportunity to get all these developers that don't have the natural habitat in the kernel area that don't routinely use ISORGEN instruments to shift bits around in CPUs to embedded programming because they can work in an environment that they are used to they can work with languages that more represent the train of the school of thought they have and especially I learned that it's very popular for students that maybe have been born after the 1990s because these guys all want web stuff they want web browser support they want ideally programmed devices in Facebook and you need to take care of this somehow so embed has a very nice web based development environment where you don't need to fiddle around with all the cross compiler issues with all the debugger issues and so on you just compile your programs in a web based environment and what's the most fun part about ARM embed is you have the embed hardware development kit so that provides schematics for reference devices that these devices basically have a USB connectivity you attach them to your computer and then they show up as more storage components or as a USB flash drive you drag and drop your application to your more storage device then it's flashed onto the device and when you unplug it you have reprogram the thing that is of course very scary to a traditional embedded guy that wants to use this 25,000 euro Lauterbach debugger to flash his stuff exactly where he wants it to be but once you've gotten used to that it's actually quite interesting thing of equipping deeply embedded systems with code Good, Sapphire final thing on my list that of course has not the ARM embed logo as I just realized maybe I should exchange that it's a comparatively novel approach by the Linux Foundation novel in terms of when it was published the source code history if you look into the repository it's much further that's been based on an old proprietary operating system and that's a really nice alternative if you're used to the Linux kernel style of programming if you're used to the Linux kernel build system so it reuses all that from the Linux kernel has a cable based build system you feel immediately at home when it comes to coding standards and so on plus it's a good choice when you want to go to a really, really small system so talking about a few kilobytes, eight, 16, 32 kilobytes of RAM it has two kernels a nano kernel and a micro kernel the nano kernel is basically this simple scheduling loop that you usually write with your own custom operating systems that I was talking about in the beginning you have a single task you have interrupt service routines but not much more you have non preemptive cooperative multitasking and the micro kernel then goes a bit further you can actually specify multiple tasks get a lot of preemptive multitasking but there's still much, much away from standard embedded Linux okay so my time basically is already up what I wanted to show you in the last part of the talk but that doesn't make much of a difference is how to do an exemplary embedded development process with ARTEMs I'm going to skip quite quickly over that the main argument that I wanted to make with ARTEMs is this is of course just a hello world program but it shows the essential things you write your normal C code you don't have a main function you have some different initialization function but you can write C code as you're used to it you use a standard POSIX environment more or less you use standard libraries you then specify in contrast to a standard embedded Linux systems what the actual properties of your system will be do I need a console do I need networking how many tasks do I need and so on and with a bit of pre-processor magic then the system knows what to generate so I've listed quite explicit steps that you can re-bike at home how to build an embedded appliance based on ARTEMs I'm not going into details here of course the executive summary is that this feels exactly like embedded Linux programming failed 15 or 20 years ago you really start from the scratch you really build everything yourself for your appliance but that in some instances is much more convenient in my opinion because you really know what you did so you don't face this tremendous complexity that you see with embedded systems these days but you go back to really getting full control over your devices and knowing what you actually run and the final argument that I wanted to make is ARTEMs is a very good support for a target PC386, a standard x86 target and this target works very well in combination with QMU so once you build your system images you get a file some .exe files which is of course not a Windows executable but an ARTEMs executable that combines the kernel and the userland portion and using QMU you can run that basically like any other userland application that you run on Linux the best thing about that is QMU has very good integrated debugging support for operating systems so you're not debugging the application in the case of ARTEMs binaries you're debugging the operating system but that as I've shown you since this is all in one single package is actually the application you specify that minus s command line argument to QMU and then can just open the combined operating system payload executable with GDB as you would every other user space a regular user space operation you attach to your remote target that's running on your system of course but that doesn't make much of a difference and then you can debug your application in very much the same way as you do debug your standard embedded Linux application so there's essentially not much difference these days if you use systems like ARTEMs in developing IoT class application and Linux class applications and the commonalities between these two worlds are much more pronounced than typical IoT vendors typical IoT systems would usually make you think so I did a little experiment with the students and if you don't really tell them what they're doing if you just give them a make file if you just tell them don't run the binary directly use this and that command line switch to run it in QMU and then do your debugging then they don't some of them don't even realize that they are not running an application but they're running basically a complete virtual machine with an operating system with a completely different environment than they're used to so with that time up I'm not gonna talk about how to build appliances or how easy that is with these operating systems and do thank you for your interest any questions maybe? I guess we have time for one or so actually I had Contiki on my list but I removed it because when I was doing the comparison you come across so many apples and peas because Contiki is so much focused on extremely low memory consumption and doing everything different than other systems on using a completely different approach to networking that I was afraid that I wouldn't have enough time to talk about these differences and do an unfair comparison and that I guess has actually turned out to be true. Okay good then thanks again and I'll be around if you wanna talk to me.