 Okay. Hi everybody. Welcome to this session. My name is Diego and this is about the challenges of using containers to run graphical embedded systems. So a few words about who am I. I am embedded engineer at Kinetics. I have worked on lots of different OSs built from Yachto or Android operating system. So my experience is building everything from the kernel to everything in user space. What this session is about, to look around and see the possible approaches to use containers, which are technology very, very used in, for example, web services and web servers in general, also unembedded. And in this case, also with graphical applications. So QT application, GTK applications and so on. So first, quick poll. How many of you are using containers, for example, Docker? Okay. So most of you, right? And how many are already using them on embedded devices? Okay. So not that many. So we think that we can enjoy the benefits of using containers also on embedded boards. So we are not trying to show the products that are available because there are products trying to solve this problem. But instead, we wanted to evaluate all the possible options that are easily available. It's not that difficult, but you need to have a look and evaluate what's best for you. So here's what we'll see today. So why we want to do that. The options are available. The impact of having the drivers outside of the container. So you need to have the kernel on the host, which is running the Docker demo. And especially, we want to have a look at 3D graphics, so OpenGL applications. We want to have a look if there's a performance difference between running natively or if Docker doesn't make a difference, which is what we expect. And also, of course, when you select an option to run your graphical application in a container, you want to take into account security. Okay. So why? First of all, we have experience with building Yachto operating system. And a lot of times people just want to install a new package and you probably need to build everything from sources. And every small difference in the sources requires compiling everything again. So we wanted to avoid the hassles of building everything from sources if that's not needed. Of course, building an application inside a container has packaging advantages. So for example, you can build all that you need and describe your requirements in a Docker file and quickly build an image. And also, you can easily move your application from an architecture to a different architecture if the base image in your container supports that. For example, if you are using Ubuntu Docker image, you will have support for x86 for arm 32 bits and arm 64 bits. So your Docker file will not change only the image that is pulled by Docker changes. So you already know what containers are. But this is what we need to pay attention when using containers. So the difference with respect to virtual machine is that containers don't have hardware virtualization. So the kernel is running on the host operating system. It's not part of the container. On the other side, in CEH root, you don't have the process and network isolation you have in Docker. So you are running in the same process space when using CEH root. So here's a simple diagram about what we want to do. So again, we have an operating system. We call it the core OS, which is running on the host. The kernel is part of this operating system. And the user space, for example, also the Docker demon, the Docker runtime is part of the core operating system. But then we want to run our graphical application in the container. So how can we do that? In general, graphical applications are clients in a server and client architecture. So if you're running X11, you have an X11 display server. And your application are X11 clients. And the same goes for Wayland. So how can we actually run a graphical application in a container? There are different ideas. One of them is the simplest one, I think, is to use the network remote protocols that are already available. For example, think of BNC. You can already run remotely a graphical application in a different machine. But of course, that means that in the middle you have a network protocol. You can use local IPC. For example, sockets sharing. We'll see a bit more about that later. Or you can use a display server on the host and the client window in the container. And the last option, which is the most radical, is to move everything in the container. So you just have the kernel and the drivers and the rest in the container. So you can run an X11 display server in the container or a Western compositor. Okay. So in the previous slide, we had the high-level options. And we are now going into the details. If you want to follow along, the slides are already uploaded. So this part is kind of difficult. But I've named the options, so you will see references to this list of options later. So specifically for X11, I wrote this list of possible options. The first one is, again, to use network-reacted remote protocol. For example, BNC or XPRA to connect from the container to the X display server, which is running on the host. The second option, which has security implications, which we'll discuss later, is to run the X display server on the host again. But just share the X11 socket to the container. So the container has direct access to the X11 socket. And your window is not that different to the other X11 windows that are running locally on the host. This is not a great idea because it doesn't provide isolation. A better idea is to use something called X1X, which is a full X display server running inside the main display server. So in this case, your application will not see the parent X display server, but a new one created, for example, by Xaffer. This provides good isolation. But as we were talking about, we will not have GPU 3D acceleration. So it has some drawbacks. A good idea, I think, is the D option. So to actually run a Wayland compositor on the host, so for example, like Weston, and have your X11 application be a X Wayland client. So basically, your application will talk the Wayland protocol thanks to the X Wayland implementation. So you will not have an X11 display server on the host, but only a translation between the two protocols. The last option is to have everything in the container. So you just have a shell, nothing else, on the console on the host. And in the container, you have, for example, XARC, X11 display server implementation, and also your X11 application. So let's see how we can run Wayland applications now. So if you're using GTK or QT to develop your application, you will likely have the possibility to switch easily between using an X11 backend or Wayland backend. But if you don't have that option and you still have an X11 application, you will have to use the X11. So in case you can run a Wayland native client, again, there are similar options. So Wayland compositor running on the host and a Wayland application client in the container. Or, again, move everything inside the container and have, for example, waste them inside the container. The last options, but I don't really like it, is to have X11 display server on the host and then run Wayland as X11 client, Wayland compositor as X11 client. And inside you have the Wayland application. I don't see any good reason to do things like that. Okay, so let's see some details about the options that we described before. So XPR is a good example of using a remote protocol. The advantage is that it has good isolation and it's called also screen for X11. So you can also detach from sessions and reattach like you would do with a screen on the common line. So it is a good solution if you don't require 3D graphics. So if you don't need GPU acceleration, it's a good solution. Another option that we mentioned before is running on X-on-X implementation like Safer. And this one provides you isolation because you are having two different X11 display servers. So your application is isolated in the child X11 display server. Again, you cannot use the GPU with this solution. Okay, so let's have a look at one of the solutions that I like the most, which is having a Wayland compositor running on your host. So again, we see that the host OS, the core OS, let's call it, is running the kernel and some user space on the hardware. And then we have our graphical applications inside the container. In this case, we're using the Wayland protocol between the host and the container. And because the Wayland protocol is much more secure than X11, it is actually a good solution to run both a Wayland client, a Wayland graphical application, and an X11 graphical application. Because if you have support for X-Wayland, you can run your X11 application inside a Wayland compositor. Okay, so one of the best tools I have found to play and start to understand how things work when running graphical applications in containers is called X11 Docker. Despite the name, which I think is somehow an historical name, it can be used both to run X11 application and Wayland applications. So disregard the name, you can try almost every option that I have mentioned before with this tool. It's a very simple script. It's actually just one script, shell script, that allows you to run graphical application or entire desktop inside Docker Linux containers. It's very simple to try, so you just download the script, have a look at the command line options that are available, you select the one that you prefer, and the best thing about the project is that it's very, very well documented. So when you have played with what is available, you can look behind the curtain and just understand what it is doing. Basically, it's just generating environment variables, copy some very important files, and then preparing a Docker command line for you. So when you already, when you have selected the option that fits your use case, you can basically disregard the tool just understand what it does and run the same command line that this tool is actually creating for you. So very good to try things out. So we wanted to play with hardware, and so we wanted to have a look at some of the issues that you may find when using an open source graphic stack or a proprietary graphic stack. So we have two examples here. We work with NXP devices the most, so NXP chips. We are trying the AtnaViv driver on the IMX6 and the Vivante proprietary stack on the IMX8M. We had to add an M SATA disk because we wanted to try general available distributions like Fedora or Ubuntu. We didn't want to create our own Yachto OS to do this kind of tests because we wanted to stay as general as possible. So to fit the Fedora that we were going to install, the space on EMMC was not enough and we installed an M SATA disk to have better performances and the space that we needed. So again to test the open source graphic stack we tried the AtnaViv driver on Fedora 30 on R. I will go a bit quickly here because basically it's what you need to do to install Fedora on an APALYS IMX6 board. Again, you install the disk and copy Fedora on the board. You adjust Uboot to tell hey, boot from the SATA disk. And then we had to tweak some kernel parameters because there were some bugs we had to work around or some option that we didn't need. Then we installed the 2DX11 graphics driver and then so at that point we had X11 display server working correctly with 2D and 3D graphics acceleration and we were close to have the same also on Wayland but there's a small bug in Waystone 6 that prevents the session to actually start. So I had to build from master, build Waystone 7.0 and then everything was working perfectly basically. And Fedora already had the Mesa 3D drivers to provide OpenGL 2 and OpenGL ES 2 acceleration. So this is a quick recap of the installation instructions to build Waystone from sources. Let's have a look now at how to... So now we have X11 and Wayland working correctly on our host but we didn't touch Docker yet. So now we want to actually test if everything is working also inside a Docker container. And in this case, so with the open source graphic stack, it's being very, very easy because we used an Ubuntu Docker image, a very small and basic Ubuntu image as a starting point and installed the applications, so the 3D benchmarks that we needed. And these common lines, so apt-get install GL Mark 2 already provided a Mesa version with EtnaViv drivers. So we didn't have to copy anything special inside the container. We already had 3D drivers and we just built the Docker image from this Docker file and we had our graphical application running inside the container. So here's a brief recap of how to use the tool, the X11 Docker tool that I mentioned before. So here are three examples of running the GL Mark 2 benchmark. In the first case, we are running X11 on the host and have an X11 client running inside the container in Docker. This one is not very isolated, so it's not secure, but if you want performances, of course, in this case, you can use 3D graphics. The other two are using Wayland exclusively. So in the second one, we have Waston running on the host, so in the core part of the core OS and a Wayland client inside the Docker container. And in the last one, instead, we have everything inside the container. Okay, so let's see how we can do the same thing with a proprietary graphic stack. It's a bit more difficult when you have to write the Docker file, because, again, we wanted to test the proprietary Vivante graphic stack on a nitrogen 8M, so basically, AMX8. So as you can see here, there are some steps you need to do, because you will not find the Vivante proprietary graphic stack inside Ubuntu basic Docker image, so it's not in the Ubuntu repositories, so I had to basically copy the apt-get configuration and repositories from the host and install, as you can see, the AMX GPU Vivante proprietary graphics driver. A couple of sim links to tell Ubuntu look in the right places for 3D graphics libraries, and then it was working as expected. So in this case, we only have support from the proprietary drivers for Waston, so again, here's an example of running X11 Docker to run the GLMark benchmark. As you can see here, we had to share this additional device here, so the slash dev slash guard core is something that the Vivante kernel drivers creates, and of course, is needed also to be available inside the Docker container. So as you can see here, there are two options related to the GPU. The first one is GPU, which was also in the open source graphic stack, which means share. You can see that in the Docker command line that X11 Docker generates, it means share slash dev slash DRI, so the container can access the graphics card device, but in this case, we had also to share this specific device. So we now have created some examples. We have evaluated the options that we can have, but we want to be sure that what is running, the setup that we have on the host doesn't have any specific advantage over running in Docker. So the question is, does it make any difference running 3D graphics inside a container with respect to using host libraries? And the answer is no. So as you can see here, I have three different examples with Wayland, with X11, with the open source graphic stack, or with the proprietary graphic stacks. It doesn't make a difference. After all, it's just that you are isolated in a dedicated process space when you are using Docker, but you are actually using the resources provided from the same kernel. So the driver in the kernel is the same. What can change, of course, is the 3D graphics drivers. So as you can see here, when you are running in the core OS on the host, you are using the Mesa implementation provided by the host. So Fedora 30 in our case. So we had Mesa 19.1. But if you are running inside the Docker container, you are, of course, running the Mesa implementation of Ubuntu, so of the user space in the container. So in this case, Mesa 19.0. So this is the case for the open source graphic stack on X11. You can see the results are the same. Also, when using Wayland on the open source graphic stack, and as well on, of course, the proprietary graphic stack. So it doesn't make a difference. Of course, we need to take care of security when selecting the best option for us to run a graphical application. The first thing that we need to keep in mind are the X11 protocol security flows. So it was designed many years ago and it has relevant problems like you are able to capture the whole screen, the whole contents of the screen from an X application. So basically, you can grab everything that is on the screen from an application, or you can do key logging. So X11 is not a secure protocol, whereas the Wayland protocol has been designed with security in mind. This is the first thing that you need to keep in mind. Then you need to remember that you don't want to be root inside a container, and you also want to start the container in background, not from the shell. Because for example, if you are adding your host user to the Docker group, you can actually run Docker command lines from your user on your PC. And then you can do nasty things like sharing the whole root of the host and be root inside the container. So inside the container, you are root and you can modify the content of the host, so of your PC or embedded board. So don't do that. And then again, sometimes you need to make compromises. For example, sharing the GPU devices is some kind of security violation. But at the moment, there are not many options you can use to work around that. So share as least as you can. So don't share what you don't need to share. And of course, start with the highest security possible, and then if you really need to disable the security options that causes you troubles or issues. Okay. So we have seen how to experiment with running applications inside the Docker container. We have seen some considerations on what you need to think about. So security, of course, is one of the most important things. But of course, for example, if you are running just a Kiosk application, which is your only applications, you don't need to isolate with something else because something else doesn't exist. So thank you very much, everybody. If you have questions, I'm open to questions.