 Hello. Okay. Welcome to introducing ResinOS, the opening system for the embedded world and containers. So, who are we? I'm Peter Sanjalatos, and my colleague here, Andre Gerson will do the presentation together. He's the lead developer of ResinOS, and also the maintainer of meta Raspberry Pi, and the meta chip for the chip board. And I'm the founder of Resin, the company behind ResinOS, and I was involved with the original work of porting Docker to ARM. So, today we'll talk about what is our mission, and why did we even build an operating system, some history, and the architecture of our operating system, some notable features we implemented, and development tools that a developer can use for developing on ResinOS with containers, and how do we see the future of that operating system. All these slides will be uploaded, and we have GitHub links all over the slides that will point you to the right repos for a particular component that we're talking about. So, what is our mission? So, we believe that containers have some unique capabilities in development. So, you can have really fast iteration, embedded developers are used to changing their code and rebooting the board. So, containers are a lot easier to use, and another thing is that if the projected numbers of Internet of Things devices that will be around in a few years is correct, then a lot of developers that are not currently embedded developers will start using and developing for the embedded world, and they're used to a lot more different tools. So, we're trying to bridge the two worlds, the worlds of modern DevOps and Docker with the embedded world. So, containers is a very well understood concept in the Cloud world, but there are some unique challenges when you try to apply them into the embedded world. So, for example, a container in the Cloud deals with some virtualized resources like CPU and memory and storage, and it doesn't really care if that's a virtual CPU or a normal CPU if it's in a KVM or not, but on an embedded platform, you might want to interact with hardware, and so the container abstraction kind of breaks there, so you have to take special care to sometimes allow hardware access when it's needed. Also, containers, well, that's generally the case that networking on embedded devices is far more weirder than in the Cloud, so you can have devices in poor network conditions, you can have a device with 3G and all this stuff, so again, we had to take special care there to make sure Docker works under these conditions, and another big part of our mission is making the operating system a friendly place for a developer to work on, so we're also providing some tools that will make that easier. So, the history of ResinOS, so ResinIO is a company, so whenever I say ResinIO, I mean the company in ResinOS is a project that we're releasing today, we didn't open source it today, so we started four years ago, and the reason we're involved with the embedded stuff was we were managing 200 devices in the city of London for two years, and we found most of the tooling available for that job to be pretty primitive and a pretty hard problem, so we were spending a lot of time developing for the infrastructure rather than our product, so at some point we said, let's try and fix it, and so this is how ResinIO, the platform was born, and to make ResinIO, the platform a thing, we needed to have an operating system that we would run our devices, so in the beginning we were wondering how we should go about that, and we tried various things, we tried modifying in our cleanings distribution, we tried modifying TinyCore, we had the requirement was to have a minimal thing that we could port to a lot of devices, but both of these approaches were limited and we figured it out pretty soon, so we finally settled on the Octo on January 2004, we open sourced the code a bit more than a year ago in July, 2015, and it has been in very active development, and we've been running that in production for 2.5 years, and it's been running on thousands of devices, so this presentation is, so today we are more or less detaching ResinOS from being a component that's being purpose built for using it in the ResinIO platform and making it a separate component that somebody can run on his own without having anything to do with the ResinIO platform, so all of the knowledge we've learned, all the problems we've solved that can be leveraged for other applications, so I'll start with going through the architecture, actually, Andrea, if you wanna go through this part, so we'll go through the architecture of our platform and how we build the operating system, what the runtime looks like, some challenges that we had to overcome. Hi, everyone, I'm Andre, so first of all, why do you choose Yocto? Because our build system of choice right now is Yocto, and we are using it basically because it's minimal, it has low footprint, and it allows easy patching that basically we can stack layers and then we can advertise them as being hardware specific or not, and then based on it, we can add patches that are specific to boards or they are just generalized across all the boards. Board vendors usually supply Yocto BSP, they tend to be kind of a unified or accepted way of delivering BSPs nowadays, FreeSkill does it all the time, all droids do that, Raspberry Pi has the community across BSPs or Yocto BSPs and so on, so device support comes pretty easy, obviously we still have to patch it, we still have to configure, but we have kind of out of the box experience with the board supports. How do we structure this? We basically have for each device a resin board repository, which is a collection of sub modules and it gives you a setup of a build, I don't know how many of you are familiar with Yocto builds, I'll have you done one before. Can you raise your hands? Oh, basically everyone. Okay, so we define a resin board repository, we create sub modules that are basically the dependencies of the build and then we stack different layers, we have the meta-resin, which is the co-operating system and then we have a meta-resin board, for example, meta-resin Raspberry Pi, which are changes specific to that board. And obviously we have Pokea and Metalware, so the dependencies that are for the operating system. So one Rippup board, why do we do that? Because we want to make sure that we don't affect builds and we don't, by mistake, change or append recipes from another layer because there is a bug or something in the BBR pen, for example, something like that. So we want them separated, sub-modules for dependent layers, how we set that, and each board can move independently. Okay, so meta-resin, it's our main resin OS layer, it's basically the common part of all our supported boards. It has a lot of features, I will mention here, automatic AUFS patching, that is, we develop a mechanism where we pool and match the current version of kernel with patches of AUFS, we pool them, we apply them after the patch task, and we roll the build. We obviously detect if the kernel supplied by the BSP already has AUFS support, so we just use it if it's there. Can pre-populate Docker images, that means you can pre-load from Yocto directly an image on your device. So for example, you wanna create a build that has, let's say, Ubuntu on it as Docker image, you'd be able to do it in the build system so you don't have to waste time downloading on the target afterwards. So pre-loaded Docker images, kernel had us for out of the three module development. Obviously, when you have a device, you wanna develop stuff and some of the things kernel modules, so you need the simverse, you need configuration of the kernel, you have kernel headers, so you can build your modules over that. So it looks like that, meta-resin has a meta-resin common, which is the part that is common across all the versions of Yocto, and then specific layers for each supported Yocto version that we have right now. We support Daisy, probably, some of you have seen laughing, I've seen laughing, but we do that because we have a board called Edison, Intel Edison, and they tend to be pretty bad in support. I hope I don't hurt anyone, but that's how it is. And then Jettro and Effido, which are kind of, well Jettro is used on probably all the other boards right now, and Effido is just the one that we used to have before, and we're keeping it that for, in case some other boards that come and want support come with BSP for that specific version, so we just keep it for future support. Yeah, build system, as a build system, we use an environment predefined in a Docker file, so what we do, we run all our builds in a container, so we have all the requirements, host configuration predictable, we don't have to redeploy a new server and reconfigure everything, we just have a new container and everything goes inside. Docker image artifacts, what we do, we use our OS as an operating system as a container, so we push this container to Docker, to Docker Hub, under that link handle, and basically you will have the entire user space pushed to a Docker image. Okay, partition layout, we use a separate root FS and root state partitions, so in this way we can catch any right root FS and basically, we know exactly which service is right to disk, we have a dual root partition, we do that for updates, so we'll be able to switch AB, mechanism, green, blue, whatever you wanna call it, we use that for that, data partition auto expands on first boot, you don't wanna burn eight gigs of disk, you just wanna burn whatever you need and the first boot will expand to the end of the disk and not only that, sometimes you don't know exactly what are you going to use as a size of the device, so it will just automatically increase to the limit. Read-only root, one of the biggest struggles that we had to overcome over time was mutation of the root FS and that is try to keep things as pristine as possible so we can do updates easily, we don't have conflicts, we don't have to transfer things across updates and so on and so forth, so what we do, we started to use read-only root FSs and we catch all the writes and in order, well, the system still needs to write things on the root FS, so we have two different solutions, if it's a state, if it's a state write, then everything goes through a bind mouth to the state partition, if it's not a state or it's a transition state, we use TMFS and configures stored in a state partition, those are persistent ones, we list this here, network configuration, random seed, the system D random seed, we want to persist it across updates, clock a shutdown and for the other, for the other transition temporary state, we list here the ACP lists and limited logs. Yeah, read-only root gives us a couple of features, it gives us a cleaner separation, it gives us OTA updates which are super simple now because we don't have mutations, it enables us the possibility of this diff based updates so on the server offline, we can compute the binary diff in between two images and then push and apply on the device and it doesn't give us the problem of leaving the state behind, that is, we have to transfer configuration across updates, we have to handle all the files that were modified and transfer them on the new update. So the architecture of the partition layout has some other interesting properties, so one of them is the compartmentalization of failures. In the system that we presented, the data partition is where docker containers are installed and run and so most of the IO activity will be happening on that partition which is separate from the root partition and therefore the likelihood of a failure happening would be in that partition and the device can definitely survive file system corruption there. The root partition is never written to while in use, but also when we didn't update, not only we will write the new update to disk and sync it, but we'll also read it back and verify it to Exams just to make sure that it's been written correctly. Also, whenever we have to do changes in the state partition, we make sure we do that in atomic fashion so we will always do a temporary file and rename thing whenever we have to change stuff. So this is another view of the build system and the partition layout. Now I'd like to go through the runtime, how the runtime looks on that operating system. So in this diagram, generally the developers will be working on the blue squares which is the container and ResinOS is what takes care of everything and the green boxes. So the goal of the operating system used to be fully updatable and by having this separation, you essentially have two different speeds and two different methods of updating. So in the container, you have a full user space and you can do updates as fast as you want as often as you want without really caring if they work or not. Even if you push a container that it's completely broken, the host will still be there for you to send the next update that fixes the problem. And so the user space below that is, we're purposely making it do as less as possible. So the goal for the user space is to bring the board up, connect to the network and enable containers and that's all it does. And so while we have the fast updates for the containers, we still have the ability to update the user space using the double partition method but that happens a lot more frequently than container updates. So this is the list of ingredients. It's not an exhaustive list obviously. These are just some highlights. So we're based on system D, network manager and modern manager for networking, modern manager for 3D support. Dropware for, as I say, it's DNS mask for handling DNS. Dogger obviously for containers and Avahi for the development tools which we'll talk later. So again, why did we choose system D? Excuse me. So initially we were based on CSV in its, which is the defaults in its system of Yachto but we quickly found ourselves having a hard time managing dependencies sometimes. Restarting a service would leave processes behind and then its program was trying to manage its own logging. And so when we should system D, we had a lot of these problems solved and then we started taking advantage of some other features that we can get so we can easily define out of memory score for critical services. So if your container gets out of memory, the system will not kill your network manager, for example, or running some services in separate mountain namespaces. We do that particularly for Docker. We don't really need to, dependency management is handled very nicely and we're also using some of the utility features that it has so NTP, we're using the system D time sync D for that. Suck activation was also something we took advantage of so while you can do assets on your board it will only run when you actually connect to it. And there are some, there are a lot of more features that we're planning of taking advantage of in the future. So on the networking side, the things are a bit more interesting. So I mentioned that in the beginning that networking in a data center is pretty predictable but we found out that the networking is pretty broken when you go outside the data center. And so the normal resolver of libc is very limited so it supports three name servers. It will try to resolve one by one what you have in your resolve confile and we've seen networks where the DHCP DNS will be broken or the Google DNS would be blocked or the DHCP DNS will take forever to resolve and it will be blocking forever or that we have more than three DNS servers available and all these kind of stuff. And so for these reasons, we included DNS mask in our operating system which has a very, very intelligent way of handling DNS. So we'll try things in parallel and choose the best DNS server and periodically re-establish which DNS server to use. And we also integrated Docker to direct all the container DNS requests to that host resolver. On the network manager, we were really interested in the D-Bus API. So everything you can do, everything you can do with configuration files on the network manager you can do through the D-Bus API. Same goes for a modern manager. And this is really important because when you're running inside a container, sometimes you want to interact with the system on the host and for example ask what are the available wifi access points and so having a D-Bus API which you have access to just through a socket, it's a very clean way of doing that through the container. So Docker, which is the main component that enables running containers. So we mentioned that we are automatically passing the kernels with AUFS. This was a hard choice to make because AUFS is not a mainline file system. However, the choices we had were pretty limited. So the other options for Docker would be overlay FS but that would require fairly new kernel 3.18 or newer. And the overlay storage driver of Docker uses a large number of inodes which was fixed with the overlay to driver but that requires an even newer kernel. And the other options were bitter FS, device mapper or ZFS which pretty much rule out any non-based device and we wanted to be able to support non-based devices. So this is the reason we are now on AUFS. We have back ported some stability patches of Docker. So these are already on the Docker source code but not on version 110 but basically they allow you to have atomic operations on the file system level so that if you pull the power plug it will not end up in a weird state. And the journal driver is also another important thing. So Docker by default will run your container and will catch the standard out of your process and anything you write to your standard output will be logged in on JSON file on disk. So that was really bad for SD card where or flash memory where and by integrating the journal D back in driver everything goes to journal D and we have a central place there where we can control if the logs will stay in RAM how much of them will stay in RAM or if it will have much more control on the logs there. And another thing is that we also enable sec-comp on any board that is built with meta-resin and Docker will take advantage of that and will run the container with a default sec-comp profile. So log management so we try to don't allow any program to write its own logs. We send everything to journal D and by default we have an eight megabyte buffer in memory. So that means that if your device has an issue and you pull the power out there's no logs to inspect on the SD card or if you put it back up. So we also offer an option to persist the logs if you want to do back something. And also all the logs that go to journal D from Docker are annotated with the container they came from so you can do pretty complex queries and find out where the logs are coming from. And another important thing is by having a central place where all the logs go if you, and that is particularly important in use cases that we see from our customers when you have a fleet of devices deployed you probably want to be sending a stream of logs back to four stores and maybe run some analytics on them. So having a central place where everything goes is very helpful to accomplish that. All right, so based on these components on the operating system we've built some features. Actually the first feature I want to talk about is part of meta-resin. So some boards like the Beaglebone or an Intel Nuke will have internal storage and the normal approach of Yachto is that you build your image and it produces a root file system but then it's your problem to find a way to flash it on the internal storage. And so with meta-resin you can, we define an image class which basically creates and creates a bootable image that will copy itself into the internal storage so you don't have to do that manually. And it would provide feedback through leads on the device. Another important thing is hostess updates and there are a large amount of options and a lot of people are talking about it and in this conference as well. So it's an area where we spend a lot of thinking and engineering on and resin hub is our current approach. However, it's definitely still under development so don't expect a production ready policy solution. So our current approach basically is taking advantage of the partition layout that we've chosen in meta-resin and do the updates by putting the new version in the inactive partition, validate everything, change the boot loader and reboot the device. Some, so this method is used by a lot of projects. However, it's costly storage-wise so you have to have double the space to do the updates. And so we're experimenting with some more advanced techniques which will reuse space for files that haven't changed in the case of OS 3 or it will reuse space for common layers in the case of Docker. And specifically the integration with Docker and resin hub. So when you do an operating system update where you essentially have to download a new user space and put it in your inactive root partition and doing that with a tar archive when you have Docker was felt wrong. So Docker is a tool where you can, I mean a container image is a user space file system. And this connects to the build artifact that Andre mentioned in the beginning. So when we do a build of a meta-resin device we push to Docker hub the version of the operating system as a Docker image which we then utilize at the time of updates to pull on the target. And so resin hub will use Docker to pull the new update. It will unpack it and do the updates. Now this opens up the door for utilizing other Docker features. For example, you can have signed images and you don't have to implement it yourself. We also have a pretty good programmatic API for managing fetching the images, managing which are installed. And an open question, but we are again thinking a lot of is can we unify the containers and the host operating system? And so if you think about it in the data partition you basically have multiple sys routes whenever you pull a container. So it should be possible to boot into one of them which is similar to OS 3 as well. Other features which have to do with how we develop the operating system is that we have a lot of automated testing around the board. So we're moving fast, but then we also have to test. So we've taken two approaches here. One is the automatic emulated testing. And so we have two QEMU targets that we used to test on every pull request that the new version of meta version will boot, that the network is working and it's integrated with our Jenkins system. And the other is that the automated hardware testing and we've built this small piece of hardware which you can find schematics for in that repo. And it basically allows you to instrument a board and let's say a Raspberry Pi. And so you can have a programmatic API that says, write these things into the SD card, now power on the board, now pull up the GPIO, now check this GPIO. So you basically have a programmatic API on a device and with this one we can do a lot more advanced testing. We can see if the provisioning works, we can see if the Wi-Fi works and all kinds of stuff. These are the current, this is the current list of devices we support. They're currently 22, I think. And most of them are on V7, but we have at least one candidate for all these architectures. And the important thing which was also mentioned in the beginning is that by using Yachter, but also the way we've structured MetaResin and its abstractions, it's fairly easy to add a new board and we really, really try to make the board specific chain is required to support the board as few as possible. So MetaResin will handle all this stuff for you. And again, this is important for, I guess, bigger projects that will prototype on a Raspberry Pi or will prototype on another board and at some point will move on and build their own custom board. Now, for development tools, so normally again, if you build an image, there's a question of how do you develop on that image? So for example, how do you put the Wi-Fi credentials in your image or how do you provision it? How do you run your code in there and get back the logs? So we had to build some tools around that. When you build a ResinOS image in development mode, it will do these things for you. So it will run an S8 server with no password so you can connect on your local network. It will also expose the Docker socket over TCP so you can run Docker commands remotely and will also use MDNS to publish its existence under the hostname.local domain name. And we've built some tools that are based on these things. The tool is called ResinDevice Toolbox and it basically allows you to do three categories of things. One is configuring the image. The second one is flashing the image and the third one is helping you do it during development. And so in this example, I've run RDT configure and I've passed the path to a Resin image and it will interrogate me on the terminal for the Wi-Fi credentials, for example, or if I want to set the hostname and then it will persist these changes in the image. As the next step, I will ask it to flash the image to the SD card or the USB drive. Again, obviously you can do that with DD as well but the good thing about this method is it will try really hard not to select a hard drive and will also validate the, I see somebody laughing but this has happened to me for sure and I'm pretty sure a lot of people killed their hard drives with DD but it will also validate these decored bugs. So some of these decards will corrupt the thing as you write it and you will find out much later so this is also an added bonus. And another thing we want to integrate into the process is beamups and so the write process will become faster by skipping unrelated sectors of the image. And during development, and this is I guess the most important command, the RDD push, you have, who has done Docker development here? Oh, a few people. So if you have your source code which generally will include a Docker file and some other source code, by running this command on your source directory it will discover the ResinOS device on your network. It will connect to the Docker socket and build your Docker file and then it will run your container and give you back the logs but at the same time it will start an iNotify watcher on your repo and whenever you change files it will automatically rsync them directly into your container and restart the container so you get a fast iteration cycle and as a special case if you change your Docker file it will figure it out and it will rebuild your Docker file. Now a question is where do you find all these images to develop for a ResinOS device? So we're also having a fairly big library of base images. So there are the ResinOS documents that we mentioned in the beginning but these are pretty primitive in terms of developments because you get a Yocto user space so we're also providing base images for all these devices that have a DBM Fedora or Alpine user space and for each one of them we have and Node.js, Python, Golang, and Java variants which basically include all the development tools for that particular framework and they're mostly based on the Docker base images and they follow the conventions but we've also added some features there that have to do with running containers on embedded devices. So the future of this is so the future as we see it has a lot of things this is just a list of things that we're thinking about but it's definitely there are a lot of more things that have to be done so compressed mom and security boot and watchdog integration newer Docker versions all these things are needed and there is lots of room for innovation I think on Satsun operating system and we're definitely interested in your thoughts and what would you expect from Satsun operating system and obviously everything we are presenting so far that also have these GitHub links underneath this open source so you can find builds of ResinOS in the website it was released, it was online two hours ago so it's really fresh you can find all our codes all the layers the other hats in our GitHub ResinOS and you can also chat with us in the ResinOS Gitter channel and we're using Apache 2 license for all our code and yeah, we'll be more than happy to talk to you right do you have any questions? yes so the fleet management is what we do in the ResinIO platform so we're using the operating system to manage that the ResinIO platform is not open source currently but we are planning to open source that part as well so today what I would do is run an open VPN server on a ResinOS device and have it connect to a server of mine and then you could take it from there yes, I think I can upload them in their online yeah, I'll put them on the page of the event yes so for our purposes we have done that for some components so when we use ResinOS in our product there is one container that we built and we use Yocto to build a container so it's definitely possible and actually it produces a really, really small container there are many motivations, there was not custom leverage where it was the size of the container and it was the best option we had and it's definitely a viable path so for a container it depends on your base image so if your base image is a Debian base image and you do a lot of up-get installs in there you will fairly quickly go and then hundreds of megabytes so doing that becomes a big user space your other option and that depends also on the language you want to run so if you have for example a Golang project you can build your Golang binary and just have a Docker image with the Golang binary so that will be really small also you can do what he said about creating a Docker image based on Yocto so these numbers were for containers our operating system currently has 300 megabytes size for its partition there are definitely ways of shrinking that I mean it's being on purpose over provisioned one of our biggest challenges is Docker so the Docker binary currently if you want to include Docker 112 all the binaries are 60 megs I think in total so this is the biggest contributor in size in our file system but we have ways of getting that down to 20 so it will be a lot smaller and again the fact that we do dual-root partition updates means that we get double users from that so finding a way forward on that front will also reduce the space requirements we actually had to look on a lot of stuff so there are a lot of ways of running containers we've looked even in using Docker to build a container and then translating that into an OS-3 deployment and then run the container with system.en spawn or the recent run C the problem with all these is that when a developer uses such an operating system he will almost certainly expect the Docker API to be there and so if you want to provide the Docker API you have no other option other than running the Docker daemon I'm not sure if we're going to ever abstract that in the future so you can interchange container monitors but currently it's Docker specific so SquashFS was a... we thought about that the main issue with SquashFS that we have is that we want to do Diff updates and if you compress a file system with SquashFS then if you change part of the file system because of the compression the bitstream changes a lot so the Diffs become basically nonexistent and so we will lose the benefit of bandwidth reduction on that front if that makes sense yeah, you can have it as a stack but then where is the second okay maybe that was the rationale behind that alright, thank you very much